danielblazevski / airflow-pyspark-reddit
Example of using Airflow to schedule downloading data form S3 and launching spark jobs
☆15Updated 7 years ago
Related projects: ⓘ
- ☆28Updated 3 years ago
- ☆39Updated this week
- An example PySpark project with pytest☆17Updated 6 years ago
- This is a simple streaming application that utilises Kafka and Python☆45Updated 5 years ago
- Using Luigi to create a Machine Learning Pipeline using the Rossman Sales data from Kaggle☆33Updated 8 years ago
- A curated list of all the awesome examples, articles, tutorials and videos for Apache Airflow.☆96Updated 3 years ago
- Docker compose files for various kafka stacks☆33Updated 6 years ago
- This repo demonstrates how to load a sample Parquet formatted file from an AWS S3 Bucket. A python job will then be submitted to a Apach…☆19Updated 8 years ago
- A Getting Started Guide for developing and using Airflow Plugins☆94Updated 5 years ago
- Sentiment Analysis of a Twitter Topic with Spark Structured Streaming☆54Updated 5 years ago
- Quickstart PySpark with Anaconda on AWS/EMR☆53Updated 7 years ago
- Code supporting Data Science articles at The Marketing Technologist, Floryn Tech Blog, and Pythom.nl☆69Updated last year
- Realtime social media data analytics with Apache Spark, Python, Kafka, Pandas, etc☆52Updated 8 years ago
- A cookiecutter template for Apache Spark applications written in Scala☆10Updated 5 years ago
- Code to build a simple analytics data pipeline with Python☆102Updated 7 years ago
- Multi docker container images for main Big Data Tools. (Hadoop, Spark, Kafka, HBase, Cassandra, Zookeeper, Zeppelin, Drill, Flink, Hive, …☆32Updated last year
- Just a boilerplate for PySpark and Flask☆35Updated 6 years ago
- CLI tool to launch Spark jobs on AWS EMR☆67Updated 11 months ago
- Airflow workflow management platform chef cookbook.☆67Updated 5 years ago
- ☆54Updated 5 years ago
- A luigi powered analytics / warehouse stack☆87Updated 7 years ago
- Example for an airflow plugin☆49Updated 8 years ago
- REST-like API exposing Airflow data and operations☆61Updated 5 years ago
- Airflow training for the crunch conf☆105Updated 5 years ago
- 🚨 Simple, self-contained fraud detection system built with Apache Kafka and Python☆81Updated 5 years ago
- event-triggered plugins for airflow☆21Updated 4 years ago
- ☆26Updated 3 years ago
- Use Airflow to move data from multiple MySQL databases to BigQuery☆99Updated 4 years ago
- Spark and Python (PySpark) Examples☆39Updated 3 years ago
- ☆11Updated 5 years ago