tcmlabs / hexagonal-architecture-python-sparkLinks
Hexagonal (ports and adapters) architecture applied to Spark and Python data engineering project
☆33Updated 2 years ago
Alternatives and similar repositories for hexagonal-architecture-python-spark
Users that are interested in hexagonal-architecture-python-spark are comparing it to the libraries listed below
Sorting:
- A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects☆223Updated 6 months ago
- Code snippets for Data Engineering Design Patterns book☆256Updated 7 months ago
- Possibly the fastest DataFrame-agnostic quality check library in town.☆223Updated last week
- A curated list of awesome blogs, videos, tools and resources about Data Contracts☆180Updated last year
- My first attempt at a rough ETL pipeline; technologies include spark, GCS, prefect orchestration, and terraform☆14Updated 3 years ago
- Example repository showing how to build a data platform with Prefect, dbt and Snowflake☆107Updated 2 years ago
- Pythonic Programming Framework to orchestrate jobs in Databricks Workflow☆221Updated 3 weeks ago
- The Data Contract Specification Repository☆383Updated last month
- A simple and easy to use Data Quality (DQ) tool built with Python.☆50Updated 2 years ago
- (project & tutorial) dag pipeline tests + ci/cd setup☆88Updated 4 years ago
- The easiest way to run Airflow locally, with linting & tests for valid DAGs and Plugins.☆257Updated 4 years ago
- Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate☆116Updated 2 years ago
- Example repo to create end to end tests for data pipeline.☆25Updated last year
- Ingesting data with Pulumi, AWS lambdas and Snowflake in a scalable, fully replayable manner☆71Updated 3 years ago
- PySpark test helper methods with beautiful error messages☆723Updated last month
- Resources for video demonstrations and blog posts related to DataOps on AWS☆181Updated 3 years ago
- Just starting your DE journey or along the way already?. I will be sharing a short list of DATA-ENGINEERING-CENTRED books that covers the…☆34Updated 3 years ago
- Template for a data contract used in a data mesh.☆479Updated last year
- Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes☆64Updated 3 years ago
- A guide for leading a data (engineering) team☆63Updated last year
- 🏃♀️ Minimalist SQL orchestrator☆290Updated this week
- ☆120Updated 3 months ago
- Quickstart for any service☆165Updated this week
- A Series of Notebooks on how to start with Kafka and Python☆152Updated 8 months ago
- end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck & Evidence☆226Updated 3 weeks ago
- ☆23Updated 4 years ago
- ☆74Updated last year
- A Python package that creates fine-grained dbt tasks on Apache Airflow☆74Updated 2 weeks ago
- Example Repo to have full end to end pyspark testing via docker-compose☆31Updated 2 years ago
- Template for Data Engineering and Data Pipeline projects☆114Updated 2 years ago