danielblazevski / airflow-pyspark-reddit
Example of using Airflow to schedule downloading data form S3 and launching spark jobs
☆15Updated 8 years ago
Related projects ⓘ
Alternatives and complementary repositories for airflow-pyspark-reddit
- ☆28Updated 4 years ago
- Docker compose files for various kafka stacks☆33Updated 6 years ago
- This is a simple streaming application that utilises Kafka and Python☆45Updated 5 years ago
- A curated list of all the awesome examples, articles, tutorials and videos for Apache Airflow.☆96Updated 3 years ago
- This repo demonstrates how to load a sample Parquet formatted file from an AWS S3 Bucket. A python job will then be submitted to a Apach…☆19Updated 8 years ago
- Apache Airflow CI pipeline☆18Updated 5 years ago
- Terraform module for a PostgreSQL-backed Apache Airflow instance☆24Updated 6 years ago
- Airflow workflow management platform chef cookbook.☆68Updated 5 years ago
- Sentiment Analysis of a Twitter Topic with Spark Structured Streaming☆55Updated 5 years ago
- CLI tool to launch Spark jobs on AWS EMR☆67Updated last year
- Quickstart PySpark with Anaconda on AWS/EMR☆53Updated 7 years ago
- Multi docker container images for main Big Data Tools. (Hadoop, Spark, Kafka, HBase, Cassandra, Zookeeper, Zeppelin, Drill, Flink, Hive, …☆33Updated this week
- An example PySpark project with pytest☆17Updated 7 years ago
- Data validation library for PySpark 3.0.0☆34Updated 2 years ago
- Just a boilerplate for PySpark and Flask☆35Updated 6 years ago
- Cubes OLAP Examples☆76Updated 6 years ago
- PySpark Code for Hands-on Learners☆114Updated 5 years ago
- A plugin to Apache Airflow to allow you to run Spark Submit Commands as an Operator☆75Updated 5 years ago
- Realtime social media data analytics with Apache Spark, Python, Kafka, Pandas, etc☆52Updated 8 years ago
- Use Airflow to move data from multiple MySQL databases to BigQuery☆99Updated 4 years ago
- Ingest tweets with Kafka. Use Spark to track popular hashtags and trendsetters for each hashtag☆29Updated 8 years ago
- Example code for building your own MemSQL Streamliner Pipelines☆23Updated 7 years ago
- ☆26Updated 10 months ago
- Docker image for Jupyter notebooks with PySpark☆27Updated 6 years ago
- Source code for 'PySpark Recipes' by Raju Kumar Mishra☆25Updated 4 years ago
- A Getting Started Guide for developing and using Airflow Plugins☆94Updated 6 years ago
- Documentation and resources for deploying JupyterHub on Hadoop☆18Updated 5 years ago
- A repository for a PySpark Cookbook by Tomasz Drabas and Denny Lee☆60Updated 6 years ago
- AWS Big Data Certification☆25Updated last year