moj-analytical-services / dataengineeringutils3Links
Fully unit tested utility functions for data engineering. Python 3 only.
☆18Updated last year
Alternatives and similar repositories for dataengineeringutils3
Users that are interested in dataengineeringutils3 are comparing it to the libraries listed below
Sorting:
- A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects☆223Updated 5 months ago
- This repository contains the dbt-glue adapter☆135Updated this week
- A simple and easy to use Data Quality (DQ) tool built with Python.☆50Updated 2 years ago
- pytest plugin to run the tests with support of pyspark☆87Updated 4 months ago
- dbt (data build tool) projects targeting AWS analytics services (redshift, glue, emr, athena) and open table formats☆30Updated 2 years ago
- A flake8 plugin that detects of usage withColumn in a loop or inside reduce☆28Updated 3 months ago
- ✨ A Pydantic to PySpark schema library☆107Updated last week
- Enforce Best Practices for all your Airflow DAGs. ⭐☆104Updated this week
- A Python package to help Databricks Unity Catalog users to read and query Delta Lake tables with Polars, DuckDb, or PyArrow.☆26Updated last year
- Amazon Managed Workflows for Apache Airflow (MWAA) Examples repository contains example DAGs, requirements.txt, plugins, and CloudFormati…☆116Updated 3 months ago
- Example Dagster Cloud code for the Hooli Data Engineering organization.☆14Updated this week
- Delta Lake helper methods. No Spark dependency.☆23Updated last year
- Possibly the fastest DataFrame-agnostic quality check library in town.☆220Updated this week
- Spark runtime on AWS Lambda☆111Updated last month
- Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes☆64Updated 3 years ago
- Pythonic Programming Framework to orchestrate jobs in Databricks Workflow☆218Updated last week
- Airflow Providers containing Deferrable Operators & Sensors from Astronomer☆149Updated this week
- Companion repository for the "Streamlining AWS Glue CI/CD — A Comprehensive Blueprint" blog post☆12Updated 11 months ago
- Delta Lake examples☆229Updated last year
- A command-line interface for packaging, deploying, and running your EMR Serverless Spark jobs☆45Updated last year
- Build DataOps platform with Apache Airflow and dbt on AWS☆59Updated 4 years ago
- dbt / Amazon Redshift Demonstration Project☆34Updated 2 years ago
- Data-aware orchestration with dagster, dbt, and airbyte☆30Updated 2 years ago
- A VS Code Extension to make it easier to manage and develop Spark jobs on EMR☆39Updated 8 months ago
- Pytest plugin for dbt core☆62Updated 9 months ago
- Make simple storing test results and visualisation of these in a BI dashboard☆47Updated 3 weeks ago
- A modern ELT demo using airbyte, dbt, snowflake and dagster☆28Updated 2 years ago
- A Python Library to support running data quality rules while the spark job is running⚡☆189Updated this week
- Run dbt serverless in the Cloud (AWS)☆43Updated 5 years ago
- Fake Pandas / PySpark DataFrame creator☆48Updated last year