jacopotagliabue / paas-data-ingestion
Ingesting data with Pulumi, AWS lambdas and Snowflake in a scalable, fully replayable manner
☆71Updated 3 years ago
Alternatives and similar repositories for paas-data-ingestion:
Users that are interested in paas-data-ingestion are comparing it to the libraries listed below
- Joining the modern data stack with the modern ML stack☆196Updated last year
- A PaaS End-to-End ML Setup with Metaflow, Serverless and SageMaker.☆37Updated 4 years ago
- Recommendations at "Reasonable Scale": joining dataOps with recSys through dbt, Merlin and Metaflow☆237Updated last year
- Playground for using large language models into the Modern Data Stack for entity matching☆107Updated last year
- Kedro Plugin to support running workflows on Kubeflow Pipelines☆53Updated 6 months ago
- Sample project that use Dagster, dbt, DuckDB and Dash to visualize car and motorcycle Spanish market☆57Updated 2 years ago
- A write-audit-publish implementation on a data lake without the JVM☆46Updated 7 months ago
- Tutorials for Fugue - A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark and Dask withou…☆113Updated 11 months ago
- (project & tutorial) dag pipeline tests + ci/cd setup☆86Updated 4 years ago
- Example repo to kickstart integration with mlflow pipelines.☆76Updated 2 years ago
- Supporting materials/code examples for my course in data engineering for machine learning.☆38Updated 2 years ago
- Sample configuration to deploy a modern data platform.☆88Updated 3 years ago
- A GitHub Action that makes it easy to use Great Expectations to validate your data pipelines in your CI workflows.☆80Updated 10 months ago
- Data pipeline with dbt, Airflow, Great Expectations☆161Updated 3 years ago
- ☆43Updated 3 years ago
- Feast AWS guide using Redshift / Spectrum / DynamoDB to build a credit scoring model☆63Updated 3 years ago
- A SQL port of python's scikit-learn preprocessing module, provided as cross-database dbt macros.☆183Updated last year
- 🛠 Python project template with unit tests, code coverage, linting, type checking, Makefile wrapper, and GitHub Actions.☆145Updated 11 months ago
- Demo repository to lambda-fy your dbt runs☆11Updated last year
- Snowflake Guide: Building a Recommendation Engine Using Snowflake & Amazon SageMaker☆31Updated 3 years ago
- A simple and easy to use Data Quality (DQ) tool built with Python.☆49Updated last year
- Pythonic Programming Framework to orchestrate jobs in Databricks Workflow☆207Updated 3 weeks ago
- Example Dagster Cloud code for the Hooli Data Engineering organization.☆85Updated last week
- Demo Project for Open Source MDS☆168Updated last year
- Code snippets for Data Engineering Design Patterns book☆74Updated last month
- Materials for my 2021 NYU class on NLP and ML Systems (Master of Engineering).☆96Updated 2 years ago
- Delta Lake helper methods. No Spark dependency.☆23Updated 6 months ago
- Open Data Stack Projects: Examples of End to End Data Engineering Projects☆78Updated last year
- 🚕 Self-contained demo using Redpanda, Materialize, River, Redis, and Streamlit to predict taxi trip durations☆47Updated 2 years ago
- A Python Library to support running data quality rules while the spark job is running⚡☆180Updated this week