In this project I have built etl pipline which scraps the trending repository based on month,week and day LIVE extract other related information using API and transform it into star schema and load to postgresql then analyzed it over PowerBI . Deployment repo link:
☆12Sep 9, 2023Updated 3 years ago
Alternatives and similar repositories for dataengineering-github-data-pipelineline
Users that are interested in dataengineering-github-data-pipelineline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This repository contains code and resources for abstractive text summarization (TS) using a novel framework that leverages knowledge-base…☆14Sep 29, 2023Updated 3 years ago
- A mental health chatbot built using NLP techniques that provides responses from verified psychologists.☆52Nov 26, 2020Updated 5 years ago
- End-to-end ELT data engineering project☆23Dec 24, 2022Updated 3 years ago
- Conduktor Helm Chart Repository☆11Oct 23, 2026Updated last week
- Run an open-source data LakeHouse locally using Docker Compose☆12May 31, 2024Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- DataTalks.Club's Data Engineering Zoomcamp Project☆25Jul 14, 2022Updated 4 years ago
- 🌟 An end-to-end full-stack data science project, including modelling, MLOps, and data storytelling. ✨☆16Aug 30, 2025Updated last year
- GUI project of Library Management System in Python using Tkinter and SQL☆18Jul 6, 2019Updated 7 years ago
- NoSQL extract, transform, load (ETL) toolkit with Python☆16Oct 19, 2026Updated 2 weeks ago
- Built a stream processing data pipeline to get data from disparate systems into a dashboard using Kafka as an intermediary.☆29Aug 14, 2023Updated 3 years ago
- ☆14Feb 23, 2025Updated last year
- Simple ETL pipeline using Python☆29May 22, 2023Updated 3 years ago
- Spark Structured Streaming data pipeline that processes movie ratings data in real-time.☆14Oct 23, 2026Updated last week
- Portfolio of projects and studies conducted in data engineering.☆34Feb 22, 2025Updated last year
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Files for the Docker and Kubernetes on Google Cloud Hands-On labs☆11Mar 14, 2023Updated 3 years ago
- Collection Of All Python Notebooks Shared On Twitter☆11Apr 10, 2023Updated 3 years ago
- A data pipeline for the ingestion & visualisation of U.S Geological Survey Earthquake data☆14Jun 24, 2024Updated 2 years ago
- KnetBuilder data integration platform for building knowledge graphs. Previously known as ondex.☆16Sep 30, 2026Updated last month
- Data Engineering & Analysis Project- San Francisco Eviction Data ETL Pipeline An end-to-end batch data pipeline for performing ETL on San…☆10Oct 2, 2025Updated last year
- StarCraft 2 Data Pipeline with Airflow, DuckDB and Streamlit☆17Mar 14, 2024Updated 2 years ago
- A Firebase Cloud Function and a Firebase hosted web app to treat weather data collected by Cloud IoT Core☆18Mar 10, 2019Updated 7 years ago
- Short Range Ultrasonic Radar - A simple radar using the ultrasonic sensor, this radar works by measuring a range from 3cm to 40 cm as non…☆19Nov 11, 2024Updated last year
- trino monitoring with JMX metrics through Prometheus and Grafana☆18Aug 14, 2024Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…☆20Aug 12, 2025Updated last year
- an end-to-end data pipeline extracting music listening habits and producing an insightful dashboard☆17Mar 31, 2024Updated 2 years ago
- A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Doc…☆25Nov 19, 2024Updated last year
- Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testin…☆81Sep 2, 2023Updated 3 years ago
- Cross-provider market data validation for OHLCV prices, adjusted returns, discrepancies, and consensus datasets.☆16Sep 28, 2026Updated last month
- A PowerBI dashboard to analyze raw sales data from a multinational pharmaceutical manufacturing company and get insights into the perform…☆27Aug 30, 2023Updated 3 years ago
- Patterns and concepts for building resilient data pipelines in Python and Scala☆17Aug 27, 2024Updated 2 years ago
- Skooldio: Data Pipelines with Airflow☆23May 24, 2025Updated last year
- Automate data collection from Spotify's worldwide ranking in 50+ countries☆25May 3, 2020Updated 6 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- The Data Pipeline and Analytics Stack is a comprehensive solution designed for processing, storing, and visualizing data. Explore a compl…☆18Dec 26, 2023Updated 2 years ago
- Spark-based pipeline to extract and parse monthly games from the Lichess database.☆22Sep 22, 2025Updated last year
- End-to-End deployment of E-commerce customers segmentation using Clustering Machine learning algorithms in Google Cloud Platform and MLOp…☆22Jun 5, 2024Updated 2 years ago
- ☆29Sep 29, 2026Updated last month
- Get historical dividends, splits, capital gains distributions, and prices for a set of symbols from Yahoo Finance☆12Dec 23, 2025Updated 10 months ago
- Sample code and documentation for very basic things that I can't remember but want to aggregate in one place☆13Nov 7, 2021Updated 4 years ago
- Pine Script v5 indicator that detects BOS/MSS structure, marks a stop and TP when a reversal breaks a swing inside the prior zone, and ca…☆15Sep 30, 2026Updated last month