Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO
☆65Jul 21, 2023Updated 2 years ago
Alternatives and similar repositories for streaming_data_processing
Users that are interested in streaming_data_processing are comparing it to the libraries listed below
Sorting:
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆144Jul 27, 2023Updated 2 years ago
- Create a chatbot that provides responses in Vietnamese, focusing on the products offered by a flower shop☆11Nov 14, 2024Updated last year
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Aug 26, 2023Updated 2 years ago
- Get Crypto data from API, stream it to Kafka with Airflow. Write data to MySQL and visualize with Metabase☆17Oct 2, 2023Updated 2 years ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆37Sep 1, 2023Updated 2 years ago
- Business challenge that requires building a data platform for retailer data analytics.☆17Feb 19, 2023Updated 3 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆43Sep 26, 2023Updated 2 years ago
- Repository for Data Engineering Zoomcamp 2024☆14Mar 25, 2024Updated last year
- Comparison between label, one-hot, target, and cross-fold target encoding☆13Mar 5, 2019Updated 6 years ago
- A website that provides fundamental analysis of the stock market. Built with Vue.js, Vite.js, Vuex, Quasar, TypeScript, and using Yahoo F…☆17May 24, 2023Updated 2 years ago
- Stock Market predictions with Prophet and FastAPI☆17Dec 22, 2021Updated 4 years ago
- Nuxt.js + Tailwind CSS Admin starter☆18Jan 3, 2023Updated 3 years ago
- End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API…☆21Jul 26, 2024Updated last year
- A package to run DuckDB queries from Apache Airflow.☆21Jun 17, 2024Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆48Dec 4, 2023Updated 2 years ago
- Fully dockerized Data Warehouse (DWH) using Airflow, dbt, PostgreSQL and dashboard using redash☆25Nov 12, 2022Updated 3 years ago
- Data Engineering Bootcamp☆30Aug 5, 2025Updated 6 months ago
- A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apa…☆27Jun 7, 2023Updated 2 years ago
- ☆43Feb 20, 2016Updated 10 years ago
- ☆33Feb 22, 2022Updated 4 years ago
- This repo contains a spark standalone cluster on docker for anyone who wants to play with PySpark by submitting their applications.☆38Jun 9, 2023Updated 2 years ago
- This repo gives an introduction to how to make full working example to serve your model using asynchronous Celery tasks and FastAPI. 🔥 …☆30May 21, 2024Updated last year
- Ghi chép về snort, suricata, SIEM, OSSEC ...☆11Dec 4, 2018Updated 7 years ago
- Use MobileNet SSD and openCV to detect and count car on road☆12Jan 13, 2020Updated 6 years ago
- http://archive.ics.uci.edu/ml/index.html☆11Jan 25, 2020Updated 6 years ago
- Wingolfsplattform. AK Internet des Wingolfsbundes.☆14Dec 31, 2022Updated 3 years ago
- ASM-HEMT is industry standard compact model for GaN RF and power devices. This repository is the source of the open source version of the…☆15Mar 28, 2021Updated 4 years ago
- ฝึกนักสร้างเว็บไซต์ จาก ผู้เริ่มต้น ไปเป็น มือโปร☆15Nov 26, 2023Updated 2 years ago
- This repo is for the Linkedin Learning course: Terraform: Managing Network Infrastructure☆12Mar 29, 2024Updated last year
- ☆17Feb 8, 2026Updated 3 weeks ago
- A VS Code Extension to make it easier to manage and develop Spark jobs on EMR☆39Feb 17, 2025Updated last year
- This project introduces PySpark, a powerful open-source framework for distributed data processing. We explore its architecture, component…☆42Sep 26, 2024Updated last year
- ☆10Jun 22, 2022Updated 3 years ago
- ☆10Feb 12, 2026Updated 2 weeks ago
- proxy that read from redis(or ssdb) write to both use for redis <=> ssdb migration on production☆12Aug 22, 2016Updated 9 years ago
- Admin general base for development boosting☆12Mar 22, 2025Updated 11 months ago
- ☆12Feb 23, 2026Updated last week
- Coupon System project: SpringBoot & AngularTS☆12Jan 3, 2021Updated 5 years ago
- a B2C modern react template☆13Nov 19, 2025Updated 3 months ago