This project showcases how to integrate the world of DevOps, focusing on Continuous Integration (CI) and Continuous Deployment (CD) with the realm of modern data engineering using Terraform and Azure as the case study
☆15Dec 27, 2023Updated 2 years ago
Alternatives and similar repositories for cicd_for_data_engineering
Users that are interested in cicd_for_data_engineering are comparing it to the libraries listed below
Sorting:
- This repository contains the necessary configuration files and DAGs (Directed Acyclic Graphs) for setting up a robust data engineering en…☆25Jan 26, 2024Updated 2 years ago
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆11Nov 18, 2023Updated 2 years ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆39Dec 18, 2023Updated 2 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆44Jan 4, 2024Updated 2 years ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆45Dec 11, 2023Updated 2 years ago
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆12Oct 11, 2023Updated 2 years ago
- This project shows how to capture changes from postgres database and stream them into kafka☆41May 17, 2024Updated last year
- An end-to-end data engineering pipeline that fetches real-time YouTube analytics and streams them through Kafka for processing with ksqlD…☆16Sep 19, 2023Updated 2 years ago
- Repo contains the materializations for Data Engineers DataOps Framework☆35Feb 24, 2026Updated last week
- An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…☆316Feb 14, 2025Updated last year
- An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Az…☆32Oct 2, 2023Updated 2 years ago
- Repository for the dbt Semantic Layer course☆11Nov 13, 2025Updated 3 months ago
- Beyond Vibe Coding. Code, Planning, Documentation and Product Management agents.☆70Feb 20, 2026Updated last week
- ☆24May 13, 2025Updated 9 months ago
- DBT and clickhouse test project with dagster☆12Aug 29, 2023Updated 2 years ago
- This project sets up a real-time data pipeline utilizing Change Data Capture (CDC) to stream changes from a PostgreSQL database to a Clic…☆12May 9, 2024Updated last year
- Code to help generate SQL for stakeholders. Code at https://www.startdataengineering.com/post/data-democratize-llm/☆13May 24, 2024Updated last year
- GitHub Action for use with python package interrogate☆11Nov 12, 2024Updated last year
- Rust And Delta Demo. Explanation and walkthrough on delta-rs☆10Aug 21, 2023Updated 2 years ago
- ☆10Sep 9, 2021Updated 4 years ago
- Building a poor man's data lake: Exploring the Power of Polars and Delta Lake☆11Dec 6, 2025Updated 2 months ago
- End-to-End ELT data pipeline with Postgres, Airbyte, dbt, Dagster, Snowflake and Metabase☆11Jul 13, 2023Updated 2 years ago
- This construct builds some elements for you to quickly launch an EMR Serverless application. After submitting the Emr Serverless job, you…☆11Nov 18, 2025Updated 3 months ago
- dbt-databend adapter plugin☆10May 30, 2024Updated last year
- ☆10Oct 9, 2021Updated 4 years ago
- This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…☆48Mar 14, 2024Updated last year
- Trying out the Dataframe Polars library with Delta Lake ... feat Python.☆12Jan 29, 2025Updated last year
- Automation of Databricks workflows☆13Nov 9, 2025Updated 3 months ago
- Data Observability for Data Engineering, published by Packt Publishing☆11Jan 24, 2025Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆48Dec 4, 2023Updated 2 years ago
- ☆11Sep 16, 2021Updated 4 years ago
- dlt-dagster-demo☆13Nov 6, 2023Updated 2 years ago
- dbt package for EDU's Ed-Fi data warehouse☆17Feb 17, 2026Updated last week
- ☆15Dec 11, 2023Updated 2 years ago
- Code for Apache Hudi, Apache Iceberg and Delta Lake analysis☆10Feb 2, 2024Updated 2 years ago
- Undefined yet.☆10Jan 5, 2023Updated 3 years ago
- Example of how to build machine learning training workflow on AWS by Prefect☆12Nov 2, 2022Updated 3 years ago
- ☆12Apr 17, 2023Updated 2 years ago
- CS886: Graph Neural Networks☆12Mar 28, 2025Updated 11 months ago