rodalbuyeh / pyspark-k8s-boilerplate
Boilerplate for PySpark on Cloud Kubernetes
☆33Updated 3 years ago
Alternatives and similar repositories for pyspark-k8s-boilerplate:
Users that are interested in pyspark-k8s-boilerplate are comparing it to the libraries listed below
- Pyspark boilerplate for running prod ready data pipeline☆28Updated 4 years ago
- PySpark data-pipeline testing and CICD☆28Updated 4 years ago
- Delta-Lake, ETL, Spark, Airflow☆46Updated 2 years ago
- A repository of sample code to show data quality checking best practices using Airflow.☆74Updated 2 years ago
- Delta Lake Documentation☆49Updated 9 months ago
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio…☆53Updated last year
- Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested da…☆111Updated last year
- Spark data pipeline that processes movie ratings data.☆28Updated last week
- ☆20Updated 3 years ago
- A repository of sample code to accompany our blog post on Airflow and dbt.☆170Updated last year
- Data lake, data warehouse on GCP☆56Updated 3 years ago
- Code snippets for Data Engineering Design Patterns book☆74Updated this week
- Simple stream processing pipeline☆99Updated 9 months ago
- (project & tutorial) dag pipeline tests + ci/cd setup☆86Updated 4 years ago
- Sample Airflow DAGs☆62Updated 2 years ago
- Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes☆63Updated 2 years ago
- Learn how to add data validation and documentation to a data pipeline built with dbt and Airflow.☆167Updated last year
- New generation opensource data stack☆65Updated 2 years ago
- A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.☆38Updated 3 years ago
- Materials of the Official Helm Chart Webinar☆27Updated 3 years ago
- Docker with Airflow and Spark standalone cluster☆252Updated last year
- Resources for video demonstrations and blog posts related to DataOps on AWS☆172Updated 3 years ago
- An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR☆174Updated last year
- Airflow Providers containing Deferrable Operators & Sensors from Astronomer☆146Updated last week
- Rules based grant management for Snowflake☆40Updated 6 years ago
- Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python☆42Updated 2 years ago
- Jupyter notebooks and AWS CloudFormation template to show how Hudi, Iceberg, and Delta Lake work☆48Updated 2 years ago
- Pythonic Programming Framework to orchestrate jobs in Databricks Workflow☆212Updated 3 weeks ago
- Spark app to merge different schemas☆23Updated 4 years ago
- Open source stack lakehouse☆25Updated last year