rogeriomm / labtools-k8s
Complete data engineering pipeline running on Minikube Kubernetes, Argo CD, Spark, Trino, S3, Delta lake, Postgres+ Debezium CDC, MySQL,Airflow, Kafka Strimzi, Datahub, OpenMetadata,Zeppelin, Jupyter, JFrog Container Registry
☆29Updated last week
Alternatives and similar repositories for labtools-k8s:
Users that are interested in labtools-k8s are comparing it to the libraries listed below
- Demo DAGs that show how to run dbt Core in Airflow using Cosmos☆53Updated 3 months ago
- Delta Lake Documentation☆48Updated 7 months ago
- A repository of sample code to show data quality checking best practices using Airflow.☆74Updated last year
- ☆23Updated last year
- Delta-Lake, ETL, Spark, Airflow☆46Updated 2 years ago
- A Python package that creates fine-grained dbt tasks on Apache Airflow☆63Updated 4 months ago
- To provide a deeper understanding of how the modern, open-source data stack consisting of Iceberg, dbt, Trino, and Hive operates within a…☆30Updated 10 months ago
- An ETL Orchestration using Apache Airflow to extract CSV files from a Google Drive, validate, transform, and load into a PostgreSQL datab…☆23Updated 7 months ago
- Code snippets for Data Engineering Design Patterns book☆53Updated 3 weeks ago
- ☆15Updated 9 months ago
- dbt package for monitoring airflow DAGs and tasks☆29Updated this week
- Spark development environment for kubernetes, spark-submit and jupyter notebook☆19Updated 3 years ago
- Library to convert DBT manifest metadata to Airflow tasks☆47Updated 10 months ago
- Spark data pipeline that processes movie ratings data.☆27Updated this week
- Full stack data engineering tools and infrastructure set-up☆48Updated 3 years ago
- ☆18Updated 5 months ago
- Covid19 and Iowa Liquor Sales analysis at BigQuery using dbt, Airflow, Marquez, Google Cloud and other modern data stack tools☆14Updated 2 years ago
- ☆73Updated 3 months ago
- Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python☆43Updated 2 years ago
- ☆106Updated 6 months ago
- A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects☆206Updated this week
- Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes☆63Updated 2 years ago
- Data Engineering com Apache Spark☆43Updated 3 years ago
- A modern ELT demo using airbyte, dbt, snowflake and dagster☆26Updated 2 years ago
- An example dbt project using AutomateDV to create a Data Vault 2.0 Data Warehouse based on the Snowflake TPC-H dataset.☆45Updated 10 months ago
- Utility functions for dbt projects running on Spark☆31Updated last week
- The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and …☆82Updated 9 months ago
- ☆22Updated 3 years ago
- CICD pipeline that deploys a dbt image on a GKE cluster☆11Updated 3 years ago
- Execution of DBT models using Apache Airflow through Docker Compose☆113Updated 2 years ago