DucAnhNTT / bigdata-ETL-pipelineLinks
The Data Pipeline and Analytics Stack is a comprehensive solution designed for processing, storing, and visualizing data. Explore a complete data pipeline with all components seamlessly set up and ready to use
☆17Updated last year
Alternatives and similar repositories for bigdata-ETL-pipeline
Users that are interested in bigdata-ETL-pipeline are comparing it to the libraries listed below
Sorting:
- A portable Datamart and Business Intelligence suite built with Docker, Airflow, dbt, PostgreSQL and Superset☆46Updated last year
- A Postgres data warehouse for processing synthetic data using IAC principles☆19Updated 2 years ago
- Full stack data engineering tools and infrastructure set-up☆57Updated 4 years ago
- ☆44Updated last year
- Cost Efficient Data Pipelines with DuckDB☆60Updated 7 months ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆43Updated last year
- ETL to scrape a real estate website, process house prices and data, and build an ML model of the house prices.☆16Updated 3 years ago
- Delta-Lake, ETL, Spark, Airflow☆48Updated 3 years ago
- A self-contained, ready to run Airflow ELT project. Can be run locally or within codespaces.☆79Updated 2 years ago
- Content related to Mastering Postgresql along with videos.☆18Updated 4 years ago
- Deploying a Machine Learning model streaming application with Apache Kafka☆11Updated 3 years ago
- To provide a deeper understanding of how the modern, open-source data stack consisting of Iceberg, dbt, Trino, and Hive operates within a…☆44Updated last year
- Processing TfL data for bike usage with Google Cloud Platform.☆46Updated 3 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆34Updated last year
- ☆21Updated last year
- Data engineering project using UK Bus Open Data Service (BODS) to calculate late buses in real-time for any selected region in England. P…☆30Updated 2 years ago
- Building Data Lakehouse by open source technology. Support end to end data pipeline, from source data on AWS S3 to Lakehouse, visualize a…☆34Updated last year
- ☆32Updated 2 years ago
- Building a Data Pipeline with an Open Source Stack☆55Updated 5 months ago
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆46Updated 2 years ago
- Apache Spark Guide☆34Updated 3 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆42Updated 2 years ago
- Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO☆64Updated 2 years ago
- Open Data Stack Projects: Examples of End to End Data Engineering Projects☆91Updated 2 years ago
- Project for real-time anomaly detection using Kafka and python☆58Updated 3 years ago
- The Python fake data producer for Apache Kafka® is a complete demo app allowing you to quickly produce JSON fake streaming datasets and …☆85Updated last year
- This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. Ther…☆24Updated 3 months ago
- Data pipeline that scrapes Rust cheater Steam profiles☆54Updated 3 years ago
- New generation opensource data stack☆76Updated 3 years ago
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆139Updated 5 years ago