adidas / m3d-engineLinks
M3D Engine is a Spark application for the development of scalable data transformations and ingestions in data lakes.
☆18Updated 4 years ago
Alternatives and similar repositories for m3d-engine
Users that are interested in m3d-engine are comparing it to the libraries listed below
Sorting:
- Auto-generated Diagrams from Airflow DAGs. 🔮 🪄☆352Updated this week
- Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pi…☆96Updated last month
- Data Profiler for AWS Glue Data Catalog application as described in the AWS Big Data Blog post "Build an automatic data profiling and rep…☆20Updated 5 years ago
- Airflow on Kubernetes Operator☆86Updated 2 years ago
- Support for generating modern platforms dynamically with services such as Kafka, Spark, Streamsets, HDFS, ....☆77Updated last week
- Superglue is a lineage-tracking tool built to help visualize the propagation of data through complex pipelines composed of tables, jobs …☆159Updated 2 years ago
- The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that sup…☆128Updated 3 years ago
- A Helm chart to install Apache Airflow on Kubernetes☆290Updated this week
- Curated list of resources about Apache Airflow☆19Updated 4 years ago
- Egeria's Guidance on Governance as well as large media files such as presentations and movies☆106Updated 3 years ago
- Terraform / NiFi on the Google Cloud Platform☆28Updated 11 months ago
- Prometheus Exporter for Airflow☆161Updated last year
- Oozie Workflow to Airflow DAGs migration tool☆88Updated 8 months ago
- Performance optimization for Spark running on Kubernetes☆89Updated 5 years ago
- Setup for running Trino with Hive Metastore on Kubernetes☆103Updated 3 years ago
- Support files for Kublr Demo Scenarios☆14Updated 2 years ago
- Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested da…☆113Updated 2 years ago
- 🚀 Deploy Kubeflow on AWS EKS with Terraform 🤖☆65Updated 2 years ago
- Spark on Kubernetes infrastructure Helm charts repo☆203Updated 3 years ago
- Use SQL to build ELT pipelines on a data lakehouse.☆288Updated 3 years ago
- Deploy a secured, clustered, auto-scaling NiFi service in AWS.☆50Updated 3 years ago
- ☆97Updated 2 years ago
- Data validation library for PySpark 3.0.0☆33Updated 2 years ago
- Example for article Running Spark 3 with standalone Hive Metastore 3.0☆102Updated 2 years ago
- Airflow declarative DAGs via YAML☆133Updated 2 years ago
- Grafana dashboards and StatsD exporter config for Airflow monitoring☆289Updated last year
- Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.☆125Updated 4 years ago
- DBND is an agile pipeline framework that helps data engineering teams track and orchestrate their data processes.☆267Updated 7 months ago
- Repository of helm charts for deploying DataHub on a Kubernetes cluster☆196Updated last week
- Trino dbt demo project to mix and load BigQuery data with and in a local PostgreSQL database☆77Updated 4 years ago