📈 A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.
☆88Mar 7, 2026Updated this week
Alternatives and similar repositories for End-to-End-Data-Pipeline
Users that are interested in End-to-End-Data-Pipeline are comparing it to the libraries listed below
Sorting:
- SDLC Copilot is an Agentic AI system designed to streamline and automate the Software Development Lifecycle (SDLC). From requirement gath…☆23Jun 14, 2025Updated 8 months ago
- ☆12May 28, 2024Updated last year
- ☆22Feb 11, 2026Updated 3 weeks ago
- Create an Anime database containing all the Anime currently available on the website, which includes: 'Anime Title', 'Description', 'C…☆12Jun 10, 2020Updated 5 years ago
- Using data from IBM Watson, descriptive and predictive analytics using Python and tableau☆12Dec 23, 2017Updated 8 years ago
- Documentation and guides for twdps working examples and reusable code☆14Aug 29, 2025Updated 6 months ago
- Middleware that provides libraries, GUI, and code generator to design multi-node (clustered) applications that are highly available, redu…☆26Updated this week
- SonarQube Scanner for MSBuild to .NET Core projects in Docker☆12Feb 16, 2018Updated 8 years ago
- ☆11Apr 20, 2024Updated last year
- EDA, manipulating raw data, drawing conclusions from plots on Netflix data.☆11May 25, 2021Updated 4 years ago
- Tools and resources to support Chaos Engineering☆11Sep 6, 2018Updated 7 years ago
- Web-Service for plato - JavaScript source code visualization, static analysis, and complexity tool☆19Dec 2, 2013Updated 12 years ago
- ☆12Jun 24, 2021Updated 4 years ago
- ☆12Feb 28, 2026Updated last week
- Collection of AWS .NET (and possibly Scala) Samples☆12Dec 10, 2018Updated 7 years ago
- An obsidian plugin to post to linkedin directly from the obsidian app.☆10Apr 27, 2024Updated last year
- This repository shows how to setup Centralized CloudWatch Observability Manager using Terraform☆17Nov 28, 2024Updated last year
- My solutions to exercises from various SQL learning courses and platforms☆12Jun 23, 2021Updated 4 years ago
- Performed Exploratory Data Analysis(EDA) on the global COVID-19 dataset. Used Geopython to get a worldwide view of COVID-19 cases.☆13Aug 27, 2020Updated 5 years ago
- Google cAdvisor, InfluxDB 0.9, Grafana☆10Feb 14, 2016Updated 10 years ago
- Multivariate adaptive shrinkage methods used for analysis GTEx data in Urbut et al (2017).☆12Sep 25, 2017Updated 8 years ago
- How to have private platform serverless APIs communicating securely internally within your organisations without needing to traverse the …☆11Dec 15, 2021Updated 4 years ago
- ☆11Aug 29, 2023Updated 2 years ago
- 👀 An all-purpose eye tracking web application and API for Alzheimer's disease research (3 tasks, <3 mins). 1st place in the 2021 CNT hac…☆13Jun 17, 2021Updated 4 years ago
- Sample AWS Batch project to read CSV files☆11Oct 22, 2017Updated 8 years ago
- ZSTD bindings for Node.js☆14Feb 9, 2015Updated 11 years ago
- Collection of AI Assistants for Investment Research☆17Apr 21, 2024Updated last year
- Samples about using Prism in Universal Windows Apps☆15Mar 24, 2015Updated 10 years ago
- Datasets for CS109☆28Oct 8, 2013Updated 12 years ago
- Nyc_Taxi_Data_Pipeline - DE Project☆139Oct 21, 2024Updated last year
- SQL Server 2016 Developer's Guide by Packt☆14Jan 30, 2023Updated 3 years ago
- ☆10Sep 11, 2017Updated 8 years ago
- The code for my talk in building a recommendation engine using Spring and Hadoop☆20Sep 15, 2014Updated 11 years ago
- Demo project showing how to deploy MongoDB on the Azure Container Services (ACS) with Kubernetes☆13Jan 11, 2018Updated 8 years ago
- Converts Trading 212 CSV files to different formats such Yahoo Finance CSV☆21Nov 10, 2022Updated 3 years ago
- Sample use of Amazon Personalize for a recommender system for car searches☆15Jul 17, 2019Updated 6 years ago
- A highly available kong cluster running on ECS backed by highly available Cassandra running on containers☆12Nov 24, 2017Updated 8 years ago
- A project that can be deployed to an Azure Kubernetes Cluster and will allow each node to obtain a Public IP address.☆17Mar 13, 2019Updated 6 years ago
- Bootstrap a VPC with BOSH and Concourse to run PaaS☆14Oct 1, 2025Updated 5 months ago