adidas / m3d-engineLinks
M3D Engine is a Spark application for the development of scalable data transformations and ingestions in data lakes.
☆18Updated 4 years ago
Alternatives and similar repositories for m3d-engine
Users that are interested in m3d-engine are comparing it to the libraries listed below
Sorting:
- Metadata Driven Development (m3d) is a cloud and platform agnostic framework for the automated creation, management and governance of dat…☆31Updated 2 years ago
- Data Profiler for AWS Glue Data Catalog application as described in the AWS Big Data Blog post "Build an automatic data profiling and rep…☆20Updated 5 years ago
- Yet Another (Spark) ETL Framework☆21Updated last year
- Unity Catalog UI☆40Updated 9 months ago
- PySpark for ETL jobs including lineage to Apache Atlas in one script via code inspection☆18Updated 8 years ago
- Delta reader for the Ray open-source toolkit for building ML applications☆46Updated last year
- Data validation library for PySpark 3.0.0☆33Updated 2 years ago
- Spark app to merge different schemas☆23Updated 4 years ago
- AWS Quick Start Team☆19Updated 8 months ago
- An Ansible collection for lifecycle and management of Cloudera CDP Private Cloud resources on bare metal, IaaS, and PaaS.☆34Updated last week
- 📆 Run, schedule, and manage your dbt jobs using Kubernetes.☆24Updated 6 years ago
- Sample code to collect Apache Iceberg metrics for table monitoring☆28Updated 10 months ago
- Example project using DBT, Databricks and AdventureWorks sample database☆12Updated 2 years ago
- ☆96Updated last year
- 💻 CLI for reporting events to Faros platform☆14Updated last month
- Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pi…☆95Updated last week
- A K8s-based infrastructure for analytics☆24Updated 5 years ago
- Shunting Yard is a real-time data replication tool that copies data between Hive Metastores.☆20Updated 3 years ago
- The Internals of Spark on Kubernetes☆71Updated 3 years ago
- An open specification for data products in Data Mesh☆60Updated 7 months ago
- Demonstrations of DBT☆16Updated 5 years ago
- Nested Data (JSON/AVRO/XML) Parsing and Flattening in Spark☆16Updated last year
- ☆16Updated 4 years ago
- Data Mesh Manager (Community Edition)☆41Updated 3 months ago
- dbt package for monitoring airflow DAGs and tasks☆29Updated 4 months ago
- DataHub on AWS demonstration resources☆10Updated 2 years ago
- Aiven's S3 Sink Connector for Apache Kafka®☆70Updated 9 months ago
- ☆10Updated 3 years ago
- Terraform / NiFi on the Google Cloud Platform☆28Updated 7 months ago
- A tool to create Airflow RBAC roles with dag-level permissions from cli.☆13Updated last year