adidas / m3d-engineLinks
M3D Engine is a Spark application for the development of scalable data transformations and ingestions in data lakes.
☆18Updated 4 years ago
Alternatives and similar repositories for m3d-engine
Users that are interested in m3d-engine are comparing it to the libraries listed below
Sorting:
- Metadata Driven Development (m3d) is a cloud and platform agnostic framework for the automated creation, management and governance of dat…☆31Updated 2 years ago
- Unity Catalog UI☆40Updated 9 months ago
- Data Profiler for AWS Glue Data Catalog application as described in the AWS Big Data Blog post "Build an automatic data profiling and rep…☆20Updated 5 years ago
- Spark app to merge different schemas☆23Updated 4 years ago
- Delta reader for the Ray open-source toolkit for building ML applications☆46Updated last year
- 📆 Run, schedule, and manage your dbt jobs using Kubernetes.☆24Updated 6 years ago
- Yet Another (Spark) ETL Framework☆21Updated last year
- DataHub on AWS demonstration resources☆10Updated 2 years ago
- Example project using DBT, Databricks and AdventureWorks sample database☆12Updated 2 years ago
- Python code that will collapse structured columns separating out the attributes into new columns☆11Updated 3 years ago
- Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python☆44Updated 2 years ago
- ☆10Updated 3 years ago
- dbt adapter for Azure Synapse Dedicated SQL Pools☆73Updated 2 weeks ago
- AWS Quick Start Team☆19Updated 8 months ago
- dbt package for monitoring airflow DAGs and tasks☆29Updated 3 months ago
- Sample Airflow DAGs☆62Updated 2 years ago
- Demos for Nessie. Nessie provides Git-like capabilities for your Data Lake.☆29Updated this week
- Shunting Yard is a real-time data replication tool that copies data between Hive Metastores.☆20Updated 3 years ago
- Pipeline library for StreamSets Data Collector and Transformer☆33Updated 2 years ago
- Spark data pipeline that processes movie ratings data.☆28Updated this week
- Databricks Azure DevOps Tutorial☆17Updated 6 years ago
- This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenario…☆25Updated 6 months ago
- Databricks Migration Tools☆43Updated 4 years ago
- Hadoop/Hive/Spark container to perform CI tests☆11Updated 4 years ago
- Data Mesh Manager (Community Edition)☆38Updated 2 months ago
- ☆11Updated 5 years ago
- Rules based grant management for Snowflake☆40Updated 6 years ago
- CI/CD for Snowflake using Jenkins and Sqitch☆8Updated 6 years ago
- The sane way of building a data layer in Airflow☆24Updated 5 years ago
- Utility functions for dbt projects running on Spark☆34Updated 3 months ago