Joshua-omolewa / Stock_streaming_pipeline_project
Built a real-time streaming pipeline to extract stock data, using Apache Nifi, Debezium, Kafka, and Spark Streaming. Loaded the transformed data into Glue database and created real-time dashboards using Power BI and Tableau with Athena. The pipeline is orchestrated using Airflow.
☆21Updated 11 months ago
Related projects: ⓘ
- End to end data engineering project☆49Updated last year
- ☆35Updated last year
- ☆35Updated 2 months ago
- Sample project to demonstrate data engineering best practices☆156Updated 6 months ago
- A data engineering project with Airflow, dbt, Terrafrom, GCP and much more!☆21Updated last year
- velib-v2___an ETL pipeline that employs batch and streaming jobs using spark, kafka, airflow, and other tools☆17Updated last week
- ☆123Updated last year
- Data Pipeline from the Global Historical Climatology Network DataSet☆24Updated last year
- ☆124Updated 2 years ago
- Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow☆127Updated 4 years ago
- Data pipeline that scrapes Rust cheater Steam profiles☆50Updated 2 years ago
- Pipeline that extracts data from the Spotify API to build a more detailed version of Spotify Wrapped☆29Updated 9 months ago
- Data Engineering Project to Extract and Process Solana Reddit Data☆18Updated 7 months ago
- ☆16Updated last year
- In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our …☆19Updated 9 months ago
- Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.☆191Updated last year
- Code Repository for my 3rd Data Project.☆12Updated last year
- Code for "Advanced data transformations in SQL" free live workshop☆54Updated last month
- Code for my "Efficient Data Processing in SQL" book.☆47Updated last month
- This repo will guide you step-by-step method to create star schema dimensional model.☆25Updated 3 years ago
- ☆27Updated 10 months ago
- Produce Kafka messages, consume them and upload into Cassandra, MongoDB.☆36Updated 11 months ago
- DataTalks.Club's Data Engineering Zoomcamp Project☆19Updated 2 years ago
- The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such …☆111Updated 2 years ago
- Generate synthetic Spotify music stream dataset to create dashboards. Spotify API generates fake event data emitted to Kafka. Spark consu…☆66Updated 9 months ago
- Repository for Data Engineering Zoomcamp 2024☆13Updated 5 months ago
- Simple stream processing pipeline☆89Updated 3 months ago
- 😈Complete End to End ETL Pipeline with Spark, Airflow, & AWS☆39Updated 5 years ago
- ☆22Updated 5 months ago
- Code for blog at https://www.startdataengineering.com/post/python-for-de/☆47Updated 3 months ago