I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations
☆29May 2, 2023Updated 3 years ago
Alternatives and similar repositories for real-time_crypto_data_pipeline_using_kafka
Users that are interested in real-time_crypto_data_pipeline_using_kafka are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- sql-for-data-engineering-course☆18May 12, 2023Updated 3 years ago
- Data Engineering & Analysis Project- San Francisco Eviction Data ETL Pipeline An end-to-end batch data pipeline for performing ETL on San…☆10Oct 2, 2025Updated 11 months ago
- ☆19May 27, 2023Updated 3 years ago
- ☆145Jan 31, 2023Updated 3 years ago
- Apartments Data Pipeline using Airflow and Spark.☆24Mar 28, 2022Updated 4 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- A big data project to develop a real-time data pipeline for analyzing the popularity and sentiments of trending topics on Twitter.☆24Jun 21, 2022Updated 4 years ago
- Running ECS task for ML prediction orchestrated by Airflow☆14May 4, 2023Updated 3 years ago
- Data pipeline that scrapes Rust cheater Steam profiles☆53Feb 13, 2022Updated 4 years ago
- ☆10Jan 5, 2022Updated 4 years ago
- A Hiphop v. Literature project to demonstrate using NLP that Hip-Hop is a form of literature and rap artists are literary geniuses.☆13Nov 13, 2020Updated 5 years ago
- ☆16Feb 20, 2026Updated 7 months ago
- This project focuses on building a robust data pipeline using Apache Airflow to automate the ingestion of weather data from the OpenWeath…☆22Feb 3, 2026Updated 7 months ago
- Pipeline that extracts data from the Spotify API to build a more detailed version of Spotify Wrapped☆50Sep 8, 2026Updated 3 weeks ago
- Capstone Project for the IBM Data Engineering Professional Certification.☆13Mar 7, 2022Updated 4 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- A golang and graphql/restapi boilerplate build for fast and quick build.☆13Apr 28, 2024Updated 2 years ago
- Data warehouse implementation for an e-commerce website “Infibeam” that sells digital and consumer electronics.☆23Jan 28, 2018Updated 8 years ago
- A highly scalable microservice to handle WhatsApp, SMS and email-based notifications.☆20Mar 29, 2021Updated 5 years ago
- This repo contains all iNeuron Full Stack Data Science Assignments☆12Jun 6, 2023Updated 3 years ago
- A dead simple Java REST API(without Spring) to transfer money between accounts☆15Aug 29, 2019Updated 7 years ago
- An interactive platform that enables individuals to share their unique experiences across various stages of life and emotional journeys. …☆14Feb 29, 2024Updated 2 years ago
- A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!☆28Nov 8, 2022Updated 3 years ago
- Winners solutions for [WNS Analytics Wizard 2018](https://datahack.analyticsvidhya.com/contest/wns-analytics-hackathon-2018/)☆25Dec 13, 2018Updated 7 years ago
- Stream/batch system with Hadoop, Spark on NYC taxi data | #DE☆26Apr 10, 2026Updated 5 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- E-Learning Platform using MERN stack☆18Jun 18, 2022Updated 4 years ago
- Monoscope's Golang client SDK.☆19Aug 27, 2026Updated last month
- Data Engineering YouTube Analysis Project by Darshil Parmar☆254Dec 8, 2023Updated 2 years ago
- ☆177May 20, 2022Updated 4 years ago
- ☆18Feb 9, 2023Updated 3 years ago
- Business challenge that requires building a data platform for retailer data analytics.☆18Feb 19, 2023Updated 3 years ago
- This repository contains analysis of IMDB data from multiple sources and analysis of movies/cast/box office revenues, movie brands and fr…☆31Jun 1, 2020Updated 6 years ago
- ☆18Oct 31, 2020Updated 5 years ago
- Simple Python wrapper for ABBYY Cloud OCR☆17Feb 5, 2024Updated 2 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- ☆217Aug 13, 2023Updated 3 years ago
- Deployed on expo go☆23May 22, 2022Updated 4 years ago
- Outlier Detection with AI + ML☆16Sep 12, 2025Updated last year
- The project focuses on the drowsiness of IT employees, drivers, pilots, crane operators, student etc. These people need a system which ca…☆14Sep 13, 2018Updated 8 years ago
- A real-time reddit data streaming pipeline for sentiment analysis of various subreddits☆147Aug 23, 2023Updated 3 years ago
- Classwork projects and home works done through Udacity data engineering nano degree☆10Jun 6, 2021Updated 5 years ago
- A simple to use python script for Automatic License Plate Recognition using Google Cloud Vision API.☆15Apr 29, 2018Updated 8 years ago