The Data Pipeline and Analytics Stack is a comprehensive solution designed for processing, storing, and visualizing data. Explore a complete data pipeline with all components seamlessly set up and ready to use
☆18Dec 26, 2023Updated 2 years ago
Alternatives and similar repositories for bigdata-ETL-pipeline
Users that are interested in bigdata-ETL-pipeline are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This project aims to move the data from a Relational database system (RDBMS) to a Hadoop file system (HDFS)☆11Apr 29, 2022Updated 4 years ago
- A data pipeline moving data from a Relational database system (RDBMS) to a Hadoop file system (HDFS).☆15Jun 3, 2021Updated 5 years ago
- velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docke…☆20Aug 12, 2025Updated last year
- Building Data Lakehouse by open source technology. Support end to end data pipeline, from source data on AWS S3 to Lakehouse, visualize a…☆42Dec 15, 2025Updated 11 months ago
- Conduktor Helm Chart Repository☆11Nov 9, 2026Updated last week
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Files for the Docker and Kubernetes on Google Cloud Hands-On labs☆11Mar 14, 2023Updated 3 years ago
- React-flows brought over to the dash framework.☆13Nov 1, 2026Updated 2 weeks ago
- Playground for Lakehouse (Iceberg, Hudi, Spark, Flink, Trino, DBT, Airflow, Kafka, Debezium CDC)☆73Sep 23, 2023Updated 3 years ago
- Data Engineering & Analysis Project- San Francisco Eviction Data ETL Pipeline An end-to-end batch data pipeline for performing ETL on San…☆10Oct 2, 2025Updated last year
- A data pipeline for the ingestion & visualisation of U.S Geological Survey Earthquake data☆14Jun 24, 2024Updated 2 years ago
- Hands-on workshop with Apache Iceberg☆14Mar 13, 2024Updated 2 years ago
- Short Range Ultrasonic Radar - A simple radar using the ultrasonic sensor, this radar works by measuring a range from 3cm to 40 cm as non…☆19Nov 11, 2024Updated 2 years ago
- ☆10Aug 17, 2021Updated 5 years ago
- FleetTrack is a data collection, ingestion and visualisation application which operates on pings from vehicles in a fleet.☆23Jun 26, 2024Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Patterns and concepts for building resilient data pipelines in Python and Scala☆17Aug 27, 2024Updated 2 years ago
- trino monitoring with JMX metrics through Prometheus and Grafana☆18Aug 14, 2024Updated 2 years ago
- This project implements an ELT (Extract - Load - Transform) data pipeline with the goodreads dataset, using dagster (orchestration), spar…☆45Apr 22, 2023Updated 3 years ago
- Spark-based pipeline to extract and parse monthly games from the Lichess database.☆22Sep 22, 2025Updated last year
- Jupyter notebooks for the teaching of mechanics☆11Oct 8, 2024Updated 2 years ago
- Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testin…☆81Sep 2, 2023Updated 3 years ago
- Sample code and documentation for very basic things that I can't remember but want to aggregate in one place☆13Nov 7, 2021Updated 5 years ago
- ☆14Mar 24, 2021Updated 5 years ago
- Jena Plugins for Pentaho Kettle to transform data into RDF☆12May 31, 2023Updated 3 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Training intrinsically motivated, independent Q-learners to play Tic-Tac-Toe☆11May 12, 2021Updated 5 years ago
- Automate data collection from Spotify's worldwide ranking in 50+ countries☆25May 3, 2020Updated 6 years ago
- ☆23Feb 5, 2024Updated 2 years ago
- A simple Perceptron in Python☆10Feb 11, 2022Updated 4 years ago
- Some exercises to learn Spark. Solved in Python.☆21Oct 15, 2024Updated 2 years ago
- ☆16Dec 13, 2023Updated 2 years ago
- Completion of three Deep Q-Networks : Deep Q-Network (DQN), Double Deep Q-Network (DDQN), Double Dueling Deep Q-Network (D3QN)☆12Jun 26, 2021Updated 5 years ago
- HypercrawlTurbo is a turbocharged web scraper for extracting URLs from a webpage.☆11May 26, 2024Updated 2 years ago
- Spark and Hive docker containers sharing a common MySQL metastore☆26Apr 17, 2020Updated 6 years ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- Skribify is a powerful transcription and summarization tool that leverages the power of OpenAI's GPT-4 and WhisperAI to generate concise …☆12Oct 21, 2026Updated 3 weeks ago
- Installer for Nexus IQ Server☆10Jul 8, 2026Updated 4 months ago
- Hands on workshop "Refactor your Jupyter notebooks into maintainable data science code with Kedro"☆18Jan 22, 2025Updated last year
- Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE☆11Dec 19, 2022Updated 3 years ago
- 参考 Chat2DB 的效果,使用 chatgpt 进行自然语言翻译,然后对数据库进行操作,使用 rust 语言实现的 web 应用。☆10Jan 13, 2025Updated last year
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio…☆56May 6, 2023Updated 3 years ago
- Doing sql in notebooks.☆15Aug 14, 2023Updated 3 years ago
Are these results useful?