rodalbuyeh / pyspark-k8s-boilerplate
Boilerplate for PySpark on Cloud Kubernetes
☆33Updated 3 years ago
Alternatives and similar repositories for pyspark-k8s-boilerplate:
Users that are interested in pyspark-k8s-boilerplate are comparing it to the libraries listed below
- ☆19Updated 3 years ago
- Trino dbt demo project to mix and load BigQuery data with and in a local PostgreSQL database☆72Updated 3 years ago
- Delta Lake helper methods. No Spark dependency.☆22Updated 5 months ago
- ☆43Updated 3 years ago
- A repository of sample code to show data quality checking best practices using Airflow.☆74Updated last year
- Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes☆63Updated 2 years ago
- Rules based grant management for Snowflake☆40Updated 6 years ago
- Delta Lake Documentation☆48Updated 8 months ago
- Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python☆42Updated 2 years ago
- Pythonic Programming Framework to orchestrate jobs in Databricks Workflow☆199Updated last week
- Spark style guide☆257Updated 4 months ago
- Pyspark boilerplate for running prod ready data pipeline☆28Updated 3 years ago
- A repository of sample code to accompany our blog post on Airflow and dbt.☆169Updated last year
- (project & tutorial) dag pipeline tests + ci/cd setup☆86Updated 4 years ago
- PySpark data-pipeline testing and CICD☆28Updated 4 years ago
- New generation opensource data stack☆65Updated 2 years ago
- Great Expectations Airflow operator☆159Updated last week
- Delta Lake examples☆217Updated 4 months ago
- Delta-Lake, ETL, Spark, Airflow☆46Updated 2 years ago
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio…☆53Updated last year
- An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR☆174Updated last year
- Astronomer Core Docker Images☆106Updated 8 months ago
- Code snippets for Data Engineering Design Patterns book☆68Updated 2 weeks ago
- Solution Accelerators for Serverless Spark on GCP, the industry's first auto-scaling and serverless Spark as a service☆64Updated 9 months ago
- Airflow Providers containing Deferrable Operators & Sensors from Astronomer☆144Updated last week
- A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.☆39Updated 3 years ago
- Spark data pipeline that processes movie ratings data.☆28Updated 3 weeks ago
- Make simple storing test results and visualisation of these in a BI dashboard☆41Updated last week
- spark on kubernetes☆105Updated 2 years ago
- For a series of posts on Amazon MSK, Amazon EKS, and Amazon EMR☆65Updated 3 years ago