This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.
☆77Oct 30, 2018Updated 8 years ago
Alternatives and similar repositories for aws-concurrent-data-orchestration-pipeline-emr-livy
Users that are interested in aws-concurrent-data-orchestration-pipeline-emr-livy are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Serverless function to automate enforcement of Multi-Factor Authentication (MFA) to all AWS IAM users with access to AWS Management Conso…☆13Oct 30, 2018Updated 8 years ago
- A Serverless function for posting to a Slack Webhook in response to a Mailgun route☆11Oct 12, 2016Updated 10 years ago
- Practice real voice interviews with an AI interviewer, get coding problems evaluated live, and receive a scored performance report.☆21Jun 29, 2026Updated 4 months ago
- Turbine: the bare metals that gets you Airflow☆377Oct 10, 2021Updated 5 years ago
- A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.☆348Mar 29, 2024Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Builder session on securing your EKS☆12Dec 3, 2019Updated 6 years ago
- Spark development environment for kubernetes, spark-submit and jupyter notebook☆18Nov 30, 2021Updated 4 years ago
- Serverless sign-up to Slack (and other services) with Serverless.com☆30Nov 2, 2018Updated 8 years ago
- ☆11May 16, 2022Updated 4 years ago
- Implementations of open source Apache Hadoop/Hive interfaces which allow for ingesting data from Amazon DynamoDB☆228Apr 8, 2026Updated 7 months ago
- Big Data Engineering & Analytics Project☆36Nov 6, 2020Updated 6 years ago
- Concat multiple files in s3☆40May 15, 2025Updated last year
- ☆11Mar 24, 2015Updated 11 years ago
- A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator☆73Sep 20, 2019Updated 7 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- The open source version of the Amazon EMR Management Guide. You can submit feedback & requests for changes by submitting issues in this r…☆62Jun 15, 2023Updated 3 years ago
- AWS Glue code samples☆1,539Updated this week
- Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks☆159Oct 24, 2024Updated 2 years ago
- ☆12Oct 12, 2018Updated 8 years ago
- Airflow Deployment on AWS ECS Fargate Using Cloudformation☆206Jun 22, 2022Updated 4 years ago
- Docker image to submit Spark applications☆38Jan 15, 2018Updated 8 years ago
- Pipeline to build, test and deploy Serverless Framework Projects with CodeBuild and CodePipeline on AWS using Terraform.☆42Mar 12, 2019Updated 7 years ago
- Code supporting Data Science articles at The Marketing Technologist, Floryn Tech Blog, and Pythom.nl☆71Mar 17, 2023Updated 3 years ago
- This is a fork of the Apache Flink Kinesis connector adding Enhanced Fanout support for Flink 1.8/1.11 on KDA.☆24Mar 1, 2026Updated 8 months ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- A toolset to streamline running spark python on EMR☆20Nov 16, 2016Updated 10 years ago
- AWS Quick Start Team☆40Oct 3, 2024Updated 2 years ago
- This GitHub project provides a series of lab exercises which help users get started using the Redshift platform.☆53Mar 31, 2021Updated 5 years ago
- Repository for hosting models on AWS blog post☆14May 6, 2019Updated 7 years ago
- DynamoDB data source for Apache Spark☆93Sep 2, 2021Updated 5 years ago
- An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR☆177Apr 13, 2026Updated 7 months ago
- This repository hosts sample pipelines☆471May 8, 2020Updated 6 years ago
- Quickstart PySpark with Anaconda on AWS/EMR☆53Jan 9, 2017Updated 9 years ago
- Amazon EMR on EKS Custom Image CLI☆32Sep 26, 2024Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆35Dec 12, 2022Updated 3 years ago
- Lighthouse is a library for data lakes built on top of Apache Spark. It provides high-level APIs in Scala to streamline data pipelines an…☆64Sep 6, 2024Updated 2 years ago
- Example AWS Cloud Development Kit app that deploys the greeter microservice stack☆19Mar 27, 2026Updated 7 months ago
- Repository for AWS DBS Reference Architectures - Enterprise Data Warehousing☆35Jan 17, 2019Updated 7 years ago
- Combined Utilities for River Routing Nested Together (CURRNT)☆10Jan 31, 2025Updated last year
- An extension for Jupyter notebooks that allows running notebooks inside a Docker container and converting them to runnable Docker images.☆28Aug 14, 2023Updated 3 years ago
- Collection of Cloud Formation Templates, Lambda Scripts and sample code required to provision an AWS Data Lake for a ReInvent Lab Exercis…☆26Apr 9, 2019Updated 7 years ago
Are these results useful?