danielblazevski / airflow-pyspark-reddit
Example of using Airflow to schedule downloading data form S3 and launching spark jobs
☆16Updated 8 years ago
Alternatives and similar repositories for airflow-pyspark-reddit
Users that are interested in airflow-pyspark-reddit are comparing it to the libraries listed below
Sorting:
- Setup Apache Airflow on Kubernetes☆10Updated 6 years ago
- This repo demonstrates how to load a sample Parquet formatted file from an AWS S3 Bucket. A python job will then be submitted to a Apach…☆19Updated 8 years ago
- ☆28Updated 4 years ago
- ☆26Updated 4 years ago
- Terraform module for a PostgreSQL-backed Apache Airflow instance☆24Updated 7 years ago
- Using Luigi to create a Machine Learning Pipeline using the Rossman Sales data from Kaggle☆33Updated 8 years ago
- An example PySpark project with pytest☆16Updated 7 years ago
- Spark Application UI extension for JupyterLab☆10Updated 3 years ago
- A curated list of all the awesome examples, articles, tutorials and videos for Apache Airflow.☆96Updated 4 years ago
- Docker compose files for various kafka stacks☆32Updated 7 years ago
- Apache Airflow CI pipeline☆19Updated 5 years ago
- Real time and offline time series analysis with Spark, Spark Streaming and Storm☆21Updated 4 years ago
- Source code for 'PySpark Recipes' by Raju Kumar Mishra☆25Updated 5 years ago
- A Getting Started Guide for developing and using Airflow Plugins☆93Updated 6 years ago
- A toolset to streamline running spark python on EMR☆20Updated 8 years ago
- CLI tool to launch Spark jobs on AWS EMR☆67Updated last year
- Sentiment Analysis of a Twitter Topic with Spark Structured Streaming☆55Updated 6 years ago
- Docker container to make running Luigi tasks real easy.☆11Updated 8 years ago
- Documentation and resources for deploying JupyterHub on Hadoop☆18Updated 5 years ago
- Mastering Spark for Data Science, published by Packt☆47Updated 2 years ago
- Puppet module to provision Airbnb's Airflow☆19Updated 2 years ago
- A data engineering pipeline for harvesting top author data from Medium☆16Updated 6 years ago
- A simple introduction to using spark ml pipelines☆26Updated 7 years ago
- Some AWS EMR examples☆16Updated 7 years ago
- Airflow workflow management platform chef cookbook.☆71Updated 5 years ago
- Build and run Spark Structured Streaming pipelines in Hadoop - project using PySpark.☆13Updated 5 years ago
- A plugin for Apache Airflow that allows you to manage the users that can login☆14Updated 5 years ago
- Realtime social media data analytics with Apache Spark, Python, Kafka, Pandas, etc☆51Updated 8 years ago
- Ansible roles to deploy Kubernetes, JupyterHub, Jupyter Enterprise Gateway and Spark on Kubernetes cluster☆39Updated 4 years ago
- This is a simple streaming application that utilises Kafka and Python☆46Updated 6 years ago