adidas / m3d-engine
M3D Engine is a Spark application for the development of scalable data transformations and ingestions in data lakes.
☆18Updated 3 years ago
Alternatives and similar repositories for m3d-engine:
Users that are interested in m3d-engine are comparing it to the libraries listed below
- Metadata Driven Development (m3d) is a cloud and platform agnostic framework for the automated creation, management and governance of dat…☆31Updated last year
- Yet Another (Spark) ETL Framework☆20Updated last year
- Python code that will collapse structured columns separating out the attributes into new columns☆11Updated 3 years ago
- Data Profiler for AWS Glue Data Catalog application as described in the AWS Big Data Blog post "Build an automatic data profiling and rep…☆19Updated 4 years ago
- Unity Catalog UI☆40Updated 7 months ago
- Sample code to collect Apache Iceberg metrics for table monitoring☆26Updated 8 months ago
- Example project using DBT, Databricks and AdventureWorks sample database☆11Updated 2 years ago
- The sane way of building a data layer in Airflow☆24Updated 5 years ago
- Rules based grant management for Snowflake☆40Updated 6 years ago
- Spark app to merge different schemas☆23Updated 4 years ago
- Shunting Yard is a real-time data replication tool that copies data between Hive Metastores.☆20Updated 3 years ago
- Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python☆43Updated 2 years ago
- Data validation library for PySpark 3.0.0☆33Updated 2 years ago
- Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes☆63Updated 2 years ago
- Databricks Azure DevOps Tutorial☆17Updated 6 years ago
- Pipeline library for StreamSets Data Collector and Transformer☆33Updated 2 years ago
- ☆96Updated last year
- Useful scripts, utilities, and tools for Snowflake☆13Updated 4 years ago
- ☆11Updated 5 years ago
- Road to Azure Data Engineer Part-II: DP-201 - Designing an Azure Data Solution☆19Updated 4 years ago
- Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pi…☆94Updated this week
- 📆 Run, schedule, and manage your dbt jobs using Kubernetes.☆24Updated 6 years ago
- Nested Data (JSON/AVRO/XML) Parsing and Flattening in Spark☆16Updated last year
- Delta reader for the Ray open-source toolkit for building ML applications☆45Updated last year
- This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenario…☆23Updated 5 months ago
- An Azure Function which allows Azure Data Factory (ADF) to connect to Snowflake in a flexible way.☆26Updated last year
- Utility functions for dbt projects running on Spark☆32Updated 2 months ago
- PySpark for ETL jobs including lineage to Apache Atlas in one script via code inspection☆18Updated 8 years ago
- Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested da…☆111Updated last year
- A DBT package to perform DataOps & administrative CI/CD on your data warehouse.☆16Updated 3 years ago