My first attempt at a rough ETL pipeline; technologies include spark, GCS, prefect orchestration, and terraform
☆14Oct 12, 2022Updated 4 years ago
Alternatives and similar repositories for data_engineering_project_1
Users that are interested in data_engineering_project_1 are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Batch Final Project for Data Engineering Zoomcamp Course☆11Apr 6, 2023Updated 3 years ago
- ecommerce GCP Streaming pipeline ― Cloud Storage, Compute Engine, Pub/Sub, Dataflow, Apache Beam, BigQuery and Tableau; GCP Batch pipelin…☆10Mar 9, 2022Updated 4 years ago
- Pipeline for Automated Updates of Kaggle's "Formula 2 Dataset"☆11Aug 15, 2023Updated 3 years ago
- Data Pipeline from the Global Historical Climatology Network DataSet☆27Dec 20, 2022Updated 3 years ago
- reating a modern data pipeline using a combination of Terraform, AWS Lambda and S3, Snowflake, DBT, Mage AI, and Dash.☆15Jun 26, 2023Updated 3 years ago
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- This is the final project that after participated the Data Engineering Zoomcamp☆11Apr 4, 2022Updated 4 years ago
- This is a real-life, high throughput streaming ELT data pipeline for ecommerce☆15May 22, 2023Updated 3 years ago
- keras implementation of A Discriminative Feature Learning Approach for Deep Face Recognition based on MNIST☆10Mar 1, 2019Updated 7 years ago
- Data Engineering Project in GCP☆23Mar 29, 2023Updated 3 years ago
- dbt docs but windows 95☆16Jun 7, 2022Updated 4 years ago
- ☆46Oct 14, 2026Updated last month
- In this project I used apache airflow to scrape website periodically. This is for the tutorials I do on youtube.☆10Nov 21, 2022Updated 3 years ago
- ☆19Jun 28, 2023Updated 3 years ago
- A Covid-19 data pipeline on AWS featuring PySpark/Glue, Docker, Great Expectations, Airflow, and Redshift, templated in CloudFormation an…☆24Nov 21, 2023Updated 2 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Classwork projects and home works done through Udacity data engineering nano degree☆10Jun 6, 2021Updated 5 years ago
- Get started setting up infrastructure as code on Google Cloud Platform☆11Jun 13, 2021Updated 5 years ago
- Built a real-time streaming pipeline to extract stock data, using Apache Nifi, Debezium, Kafka, and Spark Streaming. Loaded the transform…☆29Oct 13, 2023Updated 3 years ago
- ☆12Oct 8, 2017Updated 9 years ago
- An example implementation of the FabricNet paper.☆10Oct 23, 2023Updated 3 years ago
- Collection Of All Python Notebooks Shared On Twitter☆11Apr 10, 2023Updated 3 years ago
- Here is my implementation of Center Loss with Keras☆11May 2, 2018Updated 8 years ago
- Automate ETL pipeline, build a data warehouse.☆10Dec 29, 2022Updated 3 years ago
- Resources and projects from Udacity Data Engineering with AWS nano degree programme☆30Apr 12, 2023Updated 3 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- Snowflake-specific dbt package for account-wide query workload and table- and column-level usage analytics☆21Jul 17, 2026Updated 4 months ago
- Detection of oil spill located area in the Satelite images of Ocean☆16Aug 26, 2019Updated 7 years ago
- An efficient PyTorch library for Global Wheat Detection using YOLOv5. The project is based on this Kaggle competition Global Wheat Detect…☆14Mar 13, 2023Updated 3 years ago
- Full-stack OpenTelemetry observability for Apache Spark☆20Oct 12, 2026Updated last month
- Docktor is a Web App that deploys an easy-to-use kit of analysis and scanning tools.☆13Nov 1, 2023Updated 3 years ago
- Oil Spill Detection Based on Deep Convolution Neural Network using Polarimetric Scattering Information from Sentinel-1 SAR Images☆12May 13, 2021Updated 5 years ago
- ☆11Apr 9, 2022Updated 4 years ago
- A Telegram bot that uses Google's Agent Development Kit (ADK) to generate and upload YouTube Shorts based on a title and description.☆14May 22, 2025Updated last year
- It is One of the Easiest Problems in Data Science to Detect the MNIST Numbers, Using a Classification Algorithm, Here I have used a csv F…☆16Aug 19, 2019Updated 7 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- A Python Snowpark CLI for loading the TPC-DI dataset into Snowflake. Additional dbt models for building the data warehouse.☆11Sep 4, 2025Updated last year
- Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard. The dashboa…☆270Jan 1, 2023Updated 3 years ago
- Chatbot for BI☆37Feb 21, 2023Updated 3 years ago
- Instructions and code for the workshop "From Big Data to NLP Insights: Unlocking the Power of PySpark and Spark NLP"☆12May 9, 2023Updated 3 years ago
- Notebooks for the PyTorch course by @deeplizard.☆16Dec 3, 2019Updated 6 years ago
- ☆11Dec 28, 2020Updated 5 years ago
- Multi-Agent AI Application(Python) that uses Semantic-Kernel along with Azure AI Agent Service in Azure Ai Foundry☆15Mar 6, 2025Updated last year
Are these results useful?