I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations
☆29May 2, 2023Updated 3 years ago
Alternatives and similar repositories for real-time_crypto_data_pipeline_using_kafka
Users that are interested in real-time_crypto_data_pipeline_using_kafka are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This project involves an ETL (Extract, Transform, Load) process to analyze sleep data exported from Apple Health☆29Apr 29, 2023Updated 3 years ago
- Data Engineering & Analysis Project- San Francisco Eviction Data ETL Pipeline An end-to-end batch data pipeline for performing ETL on San…☆10Oct 2, 2025Updated 11 months ago
- ☆19May 27, 2023Updated 3 years ago
- In this project, we will build and ETL(Extract,Transform,Load) pipeline using the Spotify API on AWS. The pipeline will retrieve data fro…☆25May 6, 2023Updated 3 years ago
- Kafka Connect: How to create a real time data pipeline using Change Data Capture (CDC)☆13Jan 24, 2021Updated 5 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆146Jan 31, 2023Updated 3 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆25May 14, 2022Updated 4 years ago
- Apartments Data Pipeline using Airflow and Spark.☆24Mar 28, 2022Updated 4 years ago
- A big data project to develop a real-time data pipeline for analyzing the popularity and sentiments of trending topics on Twitter.☆24Jun 21, 2022Updated 4 years ago
- Data pipeline that scrapes Rust cheater Steam profiles☆53Feb 13, 2022Updated 4 years ago
- 😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS☆51Aug 23, 2019Updated 7 years ago
- This project focuses on building a robust data pipeline using Apache Airflow to automate the ingestion of weather data from the OpenWeath…☆22Feb 3, 2026Updated 7 months ago
- Capstone Project for the IBM Data Engineering Professional Certification.☆13Mar 7, 2022Updated 4 years ago
- A golang and graphql/restapi boilerplate build for fast and quick build.☆13Apr 28, 2024Updated 2 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- ☆12Oct 22, 2022Updated 3 years ago
- 모두의 AI 케인의 Agent로 완성하는 RAG 강의 레포지토리입니다.☆20Dec 16, 2025Updated 8 months ago
- A highly scalable microservice to handle WhatsApp, SMS and email-based notifications.☆20Mar 29, 2021Updated 5 years ago
- This repo contains all iNeuron Full Stack Data Science Assignments☆12Jun 6, 2023Updated 3 years ago
- A dead simple Java REST API(without Spring) to transfer money between accounts☆15Aug 29, 2019Updated 7 years ago
- #Mentor4Nepal Initiative is for recent graduates and early career professionals focusing on the enhancement of one’s skill set, support f…☆18May 12, 2025Updated last year
- An interactive platform that enables individuals to share their unique experiences across various stages of life and emotional journeys. …☆14Feb 29, 2024Updated 2 years ago
- Insight Data Engineering project: A platform built in HDFS, Spark and Airflow to help you to find social influencers from GitHub Net…☆16May 21, 2024Updated 2 years ago
- A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!☆28Nov 8, 2022Updated 3 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Winners solutions for [WNS Analytics Wizard 2018](https://datahack.analyticsvidhya.com/contest/wns-analytics-hackathon-2018/)☆25Dec 13, 2018Updated 7 years ago
- Monoscope's Golang client SDK.☆19Aug 27, 2026Updated last week
- Data Engineering YouTube Analysis Project by Darshil Parmar☆253Dec 8, 2023Updated 2 years ago
- A real time analytics dashboard to analyze the trending hashtags and @ mentions at any location using kafka and spark streaming.☆21Oct 15, 2024Updated last year
- Business challenge that requires building a data platform for retailer data analytics.☆18Feb 19, 2023Updated 3 years ago
- Demonstration of using Apache Spark to build robust ETL pipelines while taking advantage of open source, general purpose cluster computin…☆24Aug 11, 2023Updated 3 years ago
- ☆18Oct 31, 2020Updated 5 years ago
- ☆349Aug 13, 2024Updated 2 years ago
- ☆22Jul 10, 2025Updated last year
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- ☆216Aug 13, 2023Updated 3 years ago
- Outlier Detection with AI + ML☆16Sep 12, 2025Updated 11 months ago
- The project focuses on the drowsiness of IT employees, drivers, pilots, crane operators, student etc. These people need a system which ca…☆14Sep 13, 2018Updated 7 years ago
- The project aims to analyse different types of attacks using the Bot-IoT dataset and also apply & compare different classification algori…☆13Oct 19, 2021Updated 4 years ago
- Stream processing guidelines and examples using Apache Flink and Apache Spark☆46Apr 21, 2023Updated 3 years ago
- In this web scraping project, my goal is to extract real-time stock market data from the renowned Yahoo Finance website. By leveraging we…☆13Jun 12, 2023Updated 3 years ago
- Simple examples of serving HuggingFace models with TensorFlow Serving☆16Oct 21, 2023Updated 2 years ago