adidas / m3d-engineLinks
M3D Engine is a Spark application for the development of scalable data transformations and ingestions in data lakes.
☆18Updated 4 years ago
Alternatives and similar repositories for m3d-engine
Users that are interested in m3d-engine are comparing it to the libraries listed below
Sorting:
- Metadata Driven Development (m3d) is a cloud and platform agnostic framework for the automated creation, management and governance of dat…☆31Updated 2 years ago
- Superglue is a lineage-tracking tool built to help visualize the propagation of data through complex pipelines composed of tables, jobs …☆158Updated 2 years ago
- Support for generating modern platforms dynamically with services such as Kafka, Spark, Streamsets, HDFS, ....☆75Updated last week
- Egeria's Guidance on Governance as well as large media files such as presentations and movies☆105Updated 2 years ago
- Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pi…☆96Updated 2 weeks ago
- Sparglim✨ makes PySpark App Configurable and Deploy Spark Connect Server Easier!☆37Updated 5 months ago
- Multi-stage, config driven, SQL based ETL framework using PySpark☆26Updated 5 years ago
- The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that sup…☆128Updated 3 years ago
- A Table format agnostic data sharing framework☆38Updated last year
- Accompanying code examples for webinar and blog post "three ways to run airflow on kubernetes"☆15Updated 4 years ago
- ☆96Updated 2 years ago
- Spark on Kubernetes infrastructure Helm charts repo☆203Updated 2 years ago
- Spark on Kubernetes samples☆20Updated 4 years ago
- Sample Airflow DAGs☆62Updated 2 years ago
- FADI - Ingest, store and analyse big data flows☆46Updated last year
- Yet Another (Spark) ETL Framework☆21Updated last year
- EverythingApacheNiFi☆113Updated last year
- Example for article Running Spark 3 with standalone Hive Metastore 3.0☆100Updated 2 years ago
- Unity Catalog UI☆42Updated 11 months ago
- Dremio Container Tools☆162Updated 3 months ago
- Data Profiler for AWS Glue Data Catalog application as described in the AWS Big Data Blog post "Build an automatic data profiling and rep…☆20Updated 5 years ago
- Auto-generated Diagrams from Airflow DAGs. 🔮 🪄☆345Updated this week
- Airflow support for Marquez☆31Updated 4 years ago
- Fybrik☆132Updated last year
- ☆40Updated 2 years ago
- Apache Ranger Plugin for S3☆20Updated 2 years ago
- Setup for running Trino with Hive Metastore on Kubernetes☆102Updated 2 years ago
- Data Mesh Architecture☆80Updated last year
- Performance optimization for Spark running on Kubernetes☆89Updated 4 years ago
- Terraform / NiFi on the Google Cloud Platform☆28Updated 8 months ago