This project showcases how to integrate the world of DevOps, focusing on Continuous Integration (CI) and Continuous Deployment (CD) with the realm of modern data engineering using Terraform and Azure as the case study
☆14Dec 27, 2023Updated 2 years ago
Alternatives and similar repositories for cicd_for_data_engineering
Users that are interested in cicd_for_data_engineering are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This repository contains the necessary configuration files and DAGs (Directed Acyclic Graphs) for setting up a robust data engineering en…☆25Jan 26, 2024Updated 2 years ago
- This repository contains an end-to-end data engineering project using Apache Flink, focused on performing sales analytics. The project de…☆12Nov 18, 2023Updated 2 years ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆39Dec 18, 2023Updated 2 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆44Jan 4, 2024Updated 2 years ago
- This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark cluste…☆12Oct 11, 2023Updated 2 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆47Dec 11, 2023Updated 2 years ago
- An end-to-end data engineering pipeline that fetches real-time YouTube analytics and streams them through Kafka for processing with ksqlD…☆16Sep 19, 2023Updated 2 years ago
- This project shows how to capture changes from postgres database and stream them into kafka☆42May 17, 2024Updated last year
- An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Az…☆32Oct 2, 2023Updated 2 years ago
- ☆10Sep 9, 2021Updated 4 years ago
- ☆12Apr 17, 2023Updated 3 years ago
- ☆10Oct 9, 2021Updated 4 years ago
- ☆29May 13, 2025Updated 11 months ago
- ☆14Oct 17, 2023Updated 2 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- ☆11Sep 16, 2021Updated 4 years ago
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆50Dec 4, 2023Updated 2 years ago
- ☆14Nov 10, 2022Updated 3 years ago
- Automation of Databricks workflows☆13Nov 9, 2025Updated 5 months ago
- An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…☆323Feb 14, 2025Updated last year
- This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python…☆48Mar 14, 2024Updated 2 years ago
- Example of how to build machine learning training workflow on AWS by Prefect☆12Nov 2, 2022Updated 3 years ago
- Undefined yet.☆10Jan 5, 2023Updated 3 years ago
- This is a simple iris flower classification model deployment project as flask app on Docker or Kubernetes.☆13Feb 16, 2022Updated 4 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- This repository showcases a collection of machine learning projects in various domains, demonstrating my skills and expertise as a data s…☆11Nov 20, 2023Updated 2 years ago
- This is an end to end MLOps system☆34Nov 27, 2025Updated 5 months ago
- Demoing how to use Matrix and Each definitions in Azure DevOps YAML pipelines.☆19Apr 1, 2026Updated last month
- Repo contains the materializations for Data Engineers DataOps Framework☆35Mar 16, 2026Updated last month
- CS886: Graph Neural Networks☆14Mar 28, 2025Updated last year
- GitHub Action for use with python package interrogate☆11Nov 12, 2024Updated last year
- Superstore Sales with Streamlit is a data visualization and analysis project that uses the Streamlit framework to create an interactive w…☆23Aug 24, 2023Updated 2 years ago
- Data Engineering Essentials☆29Jan 7, 2025Updated last year
- Course notes for selected courses at the University of Waterloo☆18May 21, 2022Updated 3 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- This formatter which is for handling parameters and file uploaded to Web API controller.☆26Dec 7, 2022Updated 3 years ago
- ☆26Jul 9, 2023Updated 2 years ago
- This demo shows how to deploy infrastructure into Azure using Terraform and Azure DevOps Yaml pipelines.☆29Feb 7, 2021Updated 5 years ago
- dbt-databend adapter plugin☆10May 30, 2024Updated last year
- Nishiki is an app for tracking and sharing food inventories within groups for better pantry management.☆23Nov 1, 2024Updated last year
- This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data wareh…☆214Oct 23, 2023Updated 2 years ago
- ☆33Feb 5, 2024Updated 2 years ago