danielblazevski / airflow-pyspark-reddit
Example of using Airflow to schedule downloading data form S3 and launching spark jobs
☆15Updated 8 years ago
Alternatives and similar repositories for airflow-pyspark-reddit:
Users that are interested in airflow-pyspark-reddit are comparing it to the libraries listed below
- This repo demonstrates how to load a sample Parquet formatted file from an AWS S3 Bucket. A python job will then be submitted to a Apach…☆19Updated 8 years ago
- Setup Apache Airflow on Kubernetes☆10Updated 6 years ago
- Using Luigi to create a Machine Learning Pipeline using the Rossman Sales data from Kaggle☆33Updated 8 years ago
- ☆28Updated 4 years ago
- A curated list of all the awesome examples, articles, tutorials and videos for Apache Airflow.☆96Updated 4 years ago
- Docker compose files for various kafka stacks☆32Updated 7 years ago
- A Getting Started Guide for developing and using Airflow Plugins☆93Updated 6 years ago
- A cookiecutter template for Apache Spark applications written in Scala☆10Updated 6 years ago
- ☆26Updated last year
- Quickstart PySpark with Anaconda on AWS/EMR☆53Updated 8 years ago
- Skeleton project for Apache Airflow training participants to work on.☆16Updated 4 years ago
- Airflow workflow management platform chef cookbook.☆71Updated 5 years ago
- CLI tool to launch Spark jobs on AWS EMR☆67Updated last year
- Twitter sentiment analysis using Spark and Stanford CoreNLP and visualization using elasticsearch and kibana☆20Updated 7 years ago
- Documentation and resources for deploying JupyterHub on Hadoop☆18Updated 5 years ago
- An example PySpark project with pytest☆16Updated 7 years ago
- Blog post on ETL pipelines with Airflow☆23Updated 4 years ago
- Slides produced by Engineers and Data Scientists of Blue Yonder☆50Updated 5 years ago
- A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator☆73Updated 5 years ago
- Terraform module for a PostgreSQL-backed Apache Airflow instance☆24Updated 6 years ago
- Spark and Python (PySpark) Examples☆39Updated 3 years ago
- Python Streaming Pipelines with Beam on Flink - Demo☆14Updated 2 years ago
- Collection of dask example notebooks☆58Updated 7 years ago
- Use Airflow to move data from multiple MySQL databases to BigQuery☆100Updated 4 years ago
- Multi docker container images for main Big Data Tools. (Hadoop, Spark, Kafka, HBase, Cassandra, Zookeeper, Zeppelin, Drill, Flink, Hive, …☆34Updated 4 months ago
- Docker container to make running Luigi tasks real easy.☆11Updated 8 years ago
- Mastering Spark for Data Science, published by Packt☆47Updated 2 years ago
- Cubes OLAP Examples☆74Updated 6 years ago
- A DockerSwarm Jupyterhub setup, which uses a NFS Server running in a Docker Container for persistent storage☆20Updated 7 years ago
- Deploy your Spark Production Cluster on Kubernetes☆47Updated 4 years ago