In this project I have built etl pipline which scraps the trending repository based on month,week and day LIVE extract other related information using API and transform it into star schema and load to postgresql then analyzed it over PowerBI . Deployment repo link:
☆12Sep 9, 2023Updated 2 years ago
Alternatives and similar repositories for dataengineering-github-data-pipelineline
Users that are interested in dataengineering-github-data-pipelineline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This repository contains code and resources for abstractive text summarization (TS) using a novel framework that leverages knowledge-base…☆14Sep 29, 2023Updated 2 years ago
- End-to-end ELT data engineering project☆23Dec 24, 2022Updated 3 years ago
- NoSQL extract, transform, load (ETL) toolkit with Python☆16Jun 9, 2026Updated last week
- A mental health chatbot built using NLP techniques that provides responses from verified psychologists.☆54Nov 26, 2020Updated 5 years ago
- DataTalks.Club's Data Engineering Zoomcamp Project☆24Jul 14, 2022Updated 3 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Run an open-source data LakeHouse locally using Docker Compose☆12May 31, 2024Updated 2 years ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 2 years ago
- Simple ETL pipeline using Python☆29May 22, 2023Updated 3 years ago
- Data Pipeline that utilizes GCP, Python 3.10, Prefect, and more.☆10Jan 23, 2023Updated 3 years ago
- an end-to-end data pipeline extracting music listening habits and producing an insightful dashboard☆18Mar 31, 2024Updated 2 years ago
- Files for the Docker and Kubernetes on Google Cloud Hands-On labs☆11Mar 14, 2023Updated 3 years ago
- Spark Structured Streaming data pipeline that processes movie ratings data in real-time.☆14Apr 15, 2026Updated 2 months ago
- GUI project of Library Management System in Python using Tkinter and SQL☆18Jul 6, 2019Updated 6 years ago
- A PowerBI dashboard to analyze raw sales data from a multinational pharmaceutical manufacturing company and get insights into the perform…☆20Aug 30, 2023Updated 2 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- 🌟 An end-to-end full-stack data science project, including modelling, MLOps, and data storytelling. ✨☆16Aug 30, 2025Updated 9 months ago
- Portfolio of projects and studies conducted in data engineering.☆34Feb 22, 2025Updated last year
- KnetBuilder data integration platform for building knowledge graphs. Previously known as ondex.☆15Apr 2, 2026Updated 2 months ago
- StarCraft 2 Data Pipeline with Airflow, DuckDB and Streamlit☆16Mar 14, 2024Updated 2 years ago
- ☆56Jun 7, 2026Updated last week
- A Firebase Cloud Function and a Firebase hosted web app to treat weather data collected by Cloud IoT Core☆18Mar 10, 2019Updated 7 years ago
- trino monitoring with JMX metrics through Prometheus and Grafana☆17Aug 14, 2024Updated last year
- Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testin…☆78Sep 2, 2023Updated 2 years ago
- Short Range Ultrasonic Radar - A simple radar using the ultrasonic sensor, this radar works by measuring a range from 3cm to 40 cm as non…☆19Nov 11, 2024Updated last year
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Spark-based pipeline to extract and parse monthly games from the Lichess database.☆22Sep 22, 2025Updated 8 months ago
- velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…☆21Aug 12, 2025Updated 10 months ago
- Automate data collection from Spotify's worldwide ranking in 50+ countries☆25May 3, 2020Updated 6 years ago
- Skooldio: Data Pipelines with Airflow☆23May 24, 2025Updated last year
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆23Nov 19, 2024Updated last year
- End-to-End deployment of E-commerce customers segmentation using Clustering Machine learning algorithms in Google Cloud Platform and MLOp…☆19Jun 5, 2024Updated 2 years ago
- Welcome to my data engineering projects repository! Here you will find a collection of data engineering projects that I have worked on.☆25Apr 27, 2023Updated 3 years ago
- Sample code and documentation for very basic things that I can't remember but want to aggregate in one place☆13Nov 7, 2021Updated 4 years ago
- My AI Stand. Realtime by day, rewriting itself by night. Summon my AI superpower.☆344Jun 9, 2026Updated last week
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- This is a real-life, high throughput streaming ELT data pipeline for ecommerce☆15May 22, 2023Updated 3 years ago
- Some exercises to learn Spark. Solved in Python.☆21Oct 15, 2024Updated last year
- ☆25Dec 18, 2020Updated 5 years ago
- Modern Data Engineering Project☆12Jun 3, 2022Updated 4 years ago
- This is the end to end MLOps project I built through participated the MLOps Zoomcamp☆10Sep 11, 2022Updated 3 years ago
- Open episode of the data engineering practice course☆32Jul 2, 2024Updated last year
- The Data Pipeline and Analytics Stack is a comprehensive solution designed for processing, storing, and visualizing data. Explore a compl…☆18Dec 26, 2023Updated 2 years ago