Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO
☆66Jul 21, 2023Updated 3 years ago
Alternatives and similar repositories for streaming_data_processing
Users that are interested in streaming_data_processing are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra☆145Jul 27, 2023Updated 3 years ago
- Get Crypto data from API, stream it to Kafka with Airflow. Write data to MySQL and visualize with Metabase☆17Oct 2, 2023Updated 3 years ago
- Glue ETL job or EMR Spark that gets from data catalog, modifies and uploads to S3 and Data Catalog☆13Aug 26, 2023Updated 3 years ago
- Writes the CSV file to Postgres, read table and modify it. Write more tables to Postgres with Airflow.☆38Sep 1, 2023Updated 3 years ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆43Sep 26, 2023Updated 3 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Docker Apache Airflow☆13Mar 1, 2023Updated 3 years ago
- Create a chatbot that provides responses in Vietnamese, focusing on the products offered by a flower shop☆11Nov 14, 2024Updated last year
- This repository contains an Apache Flink application for real-time sales analytics built using Docker Compose to orchestrate the necessar…☆53Dec 4, 2023Updated 2 years ago
- Repository for Data Engineering Zoomcamp 2024☆14Mar 25, 2024Updated 2 years ago
- Change Data Capture (CDC) from PostgreSQL to ClickHouse☆16Jul 15, 2024Updated 2 years ago
- End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API…☆23Sep 23, 2026Updated last month
- ☆11May 11, 2023Updated 3 years ago
- This project shows how to capture changes from postgres database and stream them into kafka☆44May 17, 2024Updated 2 years ago
- Data Engineering Projects☆12Mar 30, 2024Updated 2 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Az…☆31Oct 2, 2023Updated 3 years ago
- This repository contains the code for a realtime election voting system. The system is built using Python, Kafka, Spark Streaming, Postgr…☆50Dec 11, 2023Updated 2 years ago
- Series follows learning from Apache Spark (PySpark) with quick tips and workaround for daily problems in hand☆56Sep 30, 2023Updated 3 years ago
- ☆37Jul 18, 2025Updated last year
- ☆45Jul 6, 2024Updated 2 years ago
- Business challenge that requires building a data platform for retailer data analytics.☆18Feb 19, 2023Updated 3 years ago
- Awesome Data Engineering☆21Feb 3, 2025Updated last year
- ☆30Aug 18, 2026Updated 2 months ago
- Library to convert DBT manifest metadata to Airflow tasks☆51Dec 23, 2025Updated 10 months ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Data Engineering Project in GCP☆23Mar 29, 2023Updated 3 years ago
- ☆11Mar 28, 2022Updated 4 years ago
- commit-linter helps you lint your commit messages according to commit conventions☆10Aug 17, 2023Updated 3 years ago
- Code for dbt tutorial☆188Jun 4, 2026Updated 4 months ago
- This is a collecton of Amazon CDK projects to show how to directly ingest streaming data from Amazon Mananged Service for Apache Kafka (M…☆16Jul 28, 2026Updated 3 months ago
- End to end data pipeline to extract and analyze submissions from any subreddit using Pushshift, python, dbt and BigQuery.☆13Jul 17, 2023Updated 3 years ago
- A VS Code Extension to make it easier to manage and develop Spark jobs on EMR☆39Feb 17, 2025Updated last year
- Streaming Data from Kafka to Postgres with Kafka Connect, AVRO, Schema Registry and Python☆14Jun 11, 2018Updated 8 years ago
- Toolset for detecting reflected xss in websites☆16Oct 6, 2018Updated 8 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Basic ChatBot using CTransformers, ChromaDB and Gradio. Configured for CPU.☆13Nov 23, 2023Updated 2 years ago
- Sentiment analysis for Ekşi Sözlük with R and Python.☆11Sep 1, 2023Updated 3 years ago
- Google Flan T5☆11Feb 5, 2023Updated 3 years ago
- The goal of this project is to analyse the impact of Covid-19 on the Aviation industry through data engineering processes using technolog…☆13Jun 26, 2022Updated 4 years ago
- Code for 'Contrastive Multi-Document Question Generation'☆11Oct 16, 2022Updated 4 years ago
- Check, compile, upload, run and schedule Vertex Pipelines in a standardized manner.☆14Jun 15, 2026Updated 4 months ago
- An AWS Data Engineering End-to-End Project (Glue, Lambda, Kinesis, Redshift, QuickSight, Athena, EC2, S3)☆17Sep 20, 2023Updated 3 years ago