PritomDas / Real-Time-Streaming-Data-Pipeline-and-DashboardLinks
Building Real Time Data Pipeline using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexomonster on Docker to track status of Servers in the Data Center across the Globe.
☆20Updated 4 years ago
Alternatives and similar repositories for Real-Time-Streaming-Data-Pipeline-and-Dashboard
Users that are interested in Real-Time-Streaming-Data-Pipeline-and-Dashboard are comparing it to the libraries listed below
Sorting:
- Project for real-time anomaly detection using Kafka and python☆58Updated 2 years ago
- Docker compose and Google Colab demo to build a CDC with Delta Lake☆15Updated 2 years ago
- To provide a deeper understanding of how the modern, open-source data stack consisting of Iceberg, dbt, Trino, and Hive operates within a…☆37Updated last year
- ☆15Updated 5 years ago
- Creation of a data lakehouse and an ELT pipeline to enable the efficient analysis and use of data☆47Updated last year
- ☆37Updated 5 years ago
- Challenge Data Engineer☆25Updated 3 years ago
- Delta-Lake, ETL, Spark, Airflow☆47Updated 2 years ago
- This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and …☆31Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆44Updated last year
- Recohut - Learn data engineering, data science☆97Updated last year
- The demo of using Kafka, Spark, Hive, Cassandra, etc by using Docker. It produces the production ready environment for any kinds of big d…☆34Updated 5 years ago
- ☆12Updated 3 years ago
- Synthetic data generation for graph ML experiments☆22Updated 4 years ago
- Here I will be exploring various tools and methods that are used in data engineering process with Python.☆22Updated 4 years ago
- This repo gives an introduction to setting up streaming analytics using open source technologies☆25Updated 2 years ago
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆137Updated 5 years ago
- ☆14Updated 3 years ago
- Kafka-ML: connecting the data stream with ML/AI frameworks (now TensorFlow and PyTorch!)☆189Updated last month
- End-to-end Kafka Streaming Examples on Databricks with Evolving Avro Schemas.☆9Updated last year
- Using a feature store to connect the DataOps and MLOps workflows to enable collaborative teams to develop efficiently.☆56Updated 2 years ago
- Making Machine Learning Simple and Scalable with Python, Jupyter Notebook, TensorFlow, Keras, Apache Kafka and KSQL☆96Updated 6 years ago
- Data pipeline for extracting, transforming, and visualising Covid-19 data☆14Updated 2 years ago
- ☆17Updated 11 months ago
- This research goal is to build binary classifier model which are able to separate fraud transactions from non-fraud transactions.☆14Updated last year
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆37Updated last year
- "1 config, 1 command from Jupyter Notebook to serve Millions of users", Full-stack On-Premises MLOps system for Computer Vision from Data…☆46Updated 11 months ago
- A data pipeline with Kafka, Spark Streaming, dbt, Docker, Airflow, and GCP!☆12Updated 2 years ago
- Simple alert system implemented in Kafka and Python☆96Updated 7 years ago
- Kafka variant of the MLOps Level 1 stack☆25Updated 3 years ago