moj-analytical-services / dataengineeringutils3Links
Fully unit tested utility functions for data engineering. Python 3 only.
☆18Updated last year
Alternatives and similar repositories for dataengineeringutils3
Users that are interested in dataengineeringutils3 are comparing it to the libraries listed below
Sorting:
- A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects☆224Updated 7 months ago
- A modern ELT demo using airbyte, dbt, snowflake and dagster☆28Updated 3 years ago
- This repository contains the dbt-glue adapter☆138Updated this week
- A command-line interface for packaging, deploying, and running your EMR Serverless Spark jobs☆45Updated last year
- Data-aware orchestration with dagster, dbt, and airbyte☆30Updated 2 years ago
- A best practices guide for using AWS EMR. The guide will cover best practices on the topics of cost, performance, security, operational e…☆109Updated 2 months ago
- pytest plugin to run the tests with support of pyspark☆86Updated 6 months ago
- Enforce Best Practices for all your Airflow DAGs. ⭐☆106Updated this week
- A flake8 plugin that detects of usage withColumn in a loop or inside reduce☆28Updated 5 months ago
- A simple and easy to use Data Quality (DQ) tool built with Python.☆50Updated 2 years ago
- Fake Pandas / PySpark DataFrame creator☆48Updated last year
- Pipeline definitions for managing data flows to power analytics at MIT Open Learning☆45Updated this week
- Cost Efficient Data Pipelines with DuckDB☆60Updated 7 months ago
- A Python package to help Databricks Unity Catalog users to read and query Delta Lake tables with Polars, DuckDb, or PyArrow.☆27Updated last year
- Spark runtime on AWS Lambda☆113Updated 3 months ago
- Supporting materials/code examples for my course in data engineering for machine learning.☆39Updated 3 years ago
- Delta Lake helper methods. No Spark dependency.☆23Updated last year
- Delta Lake Documentation☆51Updated last year
- Make simple storing test results and visualisation of these in a BI dashboard☆52Updated 2 months ago
- Pytest plugin for dbt core☆63Updated 11 months ago
- Python project template for Snowpark development☆79Updated 2 years ago
- Pythonic Programming Framework to orchestrate jobs in Databricks Workflow☆222Updated last week
- ☆11Updated 3 months ago
- A portable Datamart and Business Intelligence suite built with Docker, sqlmesh + dbtcore, DuckDB and Superset☆55Updated 2 months ago
- ☆23Updated 4 years ago
- Delta Lake examples☆235Updated last year
- Possibly the fastest DataFrame-agnostic quality check library in town.☆229Updated last month
- Build DataOps platform with Apache Airflow and dbt on AWS☆59Updated 4 years ago
- JumpSpark - A modern cookiecutter template for pyspark projects with batteries included.☆10Updated 2 years ago
- A Python Library to support running data quality rules while the spark job is running⚡☆193Updated this week