I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations
☆29May 2, 2023Updated 3 years ago
Alternatives and similar repositories for real-time_crypto_data_pipeline_using_kafka
Users that are interested in real-time_crypto_data_pipeline_using_kafka are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This project involves an ETL (Extract, Transform, Load) process to analyze sleep data exported from Apple Health☆29Apr 29, 2023Updated 3 years ago
- sql-for-data-engineering-course☆18May 12, 2023Updated 3 years ago
- ☆19May 27, 2023Updated 3 years ago
- In this project, we will build and ETL(Extract,Transform,Load) pipeline using the Spotify API on AWS. The pipeline will retrieve data fro…☆25May 6, 2023Updated 3 years ago
- Data pipeline that scrapes Rust cheater Steam profiles☆53Feb 13, 2022Updated 4 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Kafka Connect: How to create a real time data pipeline using Change Data Capture (CDC)☆13Jan 24, 2021Updated 5 years ago
- ☆147Jan 31, 2023Updated 3 years ago
- *****PROJECT SPECIFICATION: Machine Learning Capstone Analysis Project***** This capstone project involves machine learning modeling and…☆15Mar 28, 2018Updated 8 years ago
- A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from loc…☆25May 14, 2022Updated 4 years ago
- reating a modern data pipeline using a combination of Terraform, AWS Lambda and S3, Snowflake, DBT, Mage AI, and Dash.☆15Jun 26, 2023Updated 3 years ago
- My solutions for the Udacity Data Engineering Nanodegree☆33Oct 14, 2019Updated 7 years ago
- A Hiphop v. Literature project to demonstrate using NLP that Hip-Hop is a form of literature and rap artists are literary geniuses.☆13Nov 13, 2020Updated 5 years ago
- Qs from other and my answer on them☆11Jan 27, 2024Updated 2 years ago
- YouTube tutorial project☆112Oct 17, 2023Updated 3 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- Analyse Spotify playlists, albums and artists.☆35Nov 15, 2022Updated 3 years ago
- Capstone Project for the IBM Data Engineering Professional Certification.☆13Mar 7, 2022Updated 4 years ago
- ☆16Feb 20, 2026Updated 8 months ago
- A simple cli tool that deletes files matching an extension within a given directory structure.☆12Aug 23, 2026Updated 2 months ago
- Pipeline that extracts data from the Spotify API to build a more detailed version of Spotify Wrapped☆50Sep 8, 2026Updated 2 months ago
- An LLM-based playlist generator connected to Spotify☆10May 16, 2023Updated 3 years ago
- How Will Your Tweet Be Received? Predicting theSentiment Polarity of Tweet Replies☆12Aug 29, 2021Updated 5 years ago
- Data Engineering YouTube Analysis Project by Darshil Parmar☆256Dec 8, 2023Updated 2 years ago
- A Discord Bot for distilling papers, GitHub repos, Blogposts, and much more using the power of LLMs and vector search.☆13May 3, 2023Updated 3 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- This project focuses on building a robust data pipeline using Apache Airflow to automate the ingestion of weather data from the OpenWeath…☆22Feb 3, 2026Updated 9 months ago
- ☆11Aug 7, 2022Updated 4 years ago
- A dead simple Java REST API(without Spring) to transfer money between accounts☆14Aug 29, 2019Updated 7 years ago
- A highly scalable microservice to handle WhatsApp, SMS and email-based notifications.☆20Mar 29, 2021Updated 5 years ago
- Data warehouse implementation for an e-commerce website “Infibeam” that sells digital and consumer electronics.☆23Jan 28, 2018Updated 8 years ago
- ☆218Aug 13, 2023Updated 3 years ago
- Rust And Delta Demo. Explanation and walkthrough on delta-rs☆10Aug 21, 2023Updated 3 years ago
- Data pipeline for EIA datasets.☆12Apr 8, 2026Updated 7 months ago
- Implementation of Quantum Perceptron: An Artificial Neuron Implemented on an Actual Quantum Processor☆10Mar 30, 2025Updated last year
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- E-Learning Platform using MERN stack☆18Jun 18, 2022Updated 4 years ago
- AlvinToh Learning Repository for The Ultimate Hands-On Hadoop - Tame your Big Data!☆10May 23, 2018Updated 8 years ago
- ☆29Jan 12, 2023Updated 3 years ago
- Kafka Schema Registry Spring Boot demo with Avro producer/consumer and PostgreSQL persistence☆12Aug 30, 2026Updated 2 months ago
- Ship detection in image and videos using YoloV8 and Efficientdet☆13Oct 5, 2023Updated 3 years ago
- RustBuy is a simple command-line interface (CLI) e-commerce application built in Rust, designed specifically to help you understand and p…☆20Sep 26, 2025Updated last year
- A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!☆29Nov 8, 2022Updated 4 years ago
Are these results useful?