PritomDas / Real-Time-Streaming-Data-Pipeline-and-Dashboard
Building Real Time Data Pipeline using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexomonster on Docker to track status of Servers in the Data Center across the Globe.
☆20Updated 3 years ago
Related projects: ⓘ
- ☆14Updated 4 years ago
- Project for real-time anomaly detection using Kafka and python☆55Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆33Updated 9 months ago
- ☆36Updated 4 years ago
- ☆35Updated 2 months ago
- Final Project for IoT: Big Data Processing and Analytics class. Analyzing U.S nationwide temperature from IoT sensors in real-time☆65Updated 7 years ago
- Data pipeline project☆22Updated last year
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆24Updated 9 months ago
- Building a Data Pipeline with an Open Source Stack☆36Updated 2 months ago
- Delta-Lake, ETL, Spark, Airflow☆42Updated last year
- Kafka variant of the MLOps Level 1 stack☆22Updated 2 years ago
- ☆59Updated this week
- The demo of using Kafka, Spark, Hive, Cassandra, etc by using Docker. It produces the production ready environment for any kinds of big d…☆30Updated 4 years ago
- This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenA…☆25Updated 8 months ago
- Spark, Airflow, Kafka☆27Updated last year
- A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…☆29Updated 4 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆26Updated 7 months ago
- Projects done in the Data Engineer Nanodegree Program by Udacity.com☆83Updated last year
- PySpark Tutorial for Beginners on Google Colab: Hands-On Guide☆16Updated 4 years ago
- This repo gives an introduction to setting up streaming analytics using open source technologies☆21Updated last year
- ☆12Updated 2 years ago
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆19Updated 9 months ago
- End to end data engineering project with kafka, airflow, spark, postgres and docker.☆54Updated last month
- ☆30Updated last year
- Airflow Tutorials☆24Updated 3 years ago
- The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Pos…☆48Updated last year
- Creation of a data lakehouse and an ELT pipeline to enable the efficient analysis and use of data☆37Updated 9 months ago
- A Postgres data warehouse for processing synthetic data using IAC principles☆16Updated last year
- Here I will be exploring various tools and methods that are used in data engineering process with Python.☆22Updated 3 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆127Updated 4 years ago