redapt / pyspark-s3-parquet-exampleLinks
This repo demonstrates how to load a sample Parquet formatted file from an AWS S3 Bucket. A python job will then be submitted to a Apache Spark instance running on AWS EMR, which will run a SQLContext to create a temporary table using a DataFrame. SQL queries will then be possible against the temporary table.
☆19Updated 9 years ago
Alternatives and similar repositories for pyspark-s3-parquet-example
Users that are interested in pyspark-s3-parquet-example are comparing it to the libraries listed below
Sorting:
- Ingest tweets with Kafka. Use Spark to track popular hashtags and trendsetters for each hashtag☆29Updated 9 years ago
- Sentiment Analysis of a Twitter Topic with Spark Structured Streaming☆55Updated 6 years ago
- Repo for all my code on the articles I post on medium☆107Updated 3 years ago
- scaffold of Apache Airflow executing Docker containers☆86Updated 2 years ago
- Airflow workflow management platform chef cookbook.☆71Updated 6 years ago
- A toolset to streamline running spark python on EMR☆20Updated 9 years ago
- An example PySpark project with pytest☆17Updated 8 years ago
- Docker container for Kafka - Spark Streaming - Cassandra☆97Updated 6 years ago
- Data validation library for PySpark 3.0.0☆33Updated 3 years ago
- Quickstart PySpark with Anaconda on AWS/EMR using Terraform☆47Updated 10 months ago
- AWS Big Data Certification☆25Updated 10 months ago
- PySpark phonetic and string matching algorithms☆39Updated last year
- Some class materials for a data processing course using PySpark☆52Updated 2 years ago
- A curated list of all the awesome examples, articles, tutorials and videos for Apache Airflow.☆96Updated 4 years ago
- A simple introduction to using spark ml pipelines☆26Updated 7 years ago
- Composable filesystem hooks and operators for Apache Airflow.☆17Updated 4 years ago
- Snowflake Guide: Building a Recommendation Engine Using Snowflake & Amazon SageMaker☆32Updated 4 years ago
- A Getting Started Guide for developing and using Airflow Plugins☆93Updated 7 years ago
- Creating a Streaming Pipeline for user log data in Google Cloud Platform☆22Updated 6 years ago
- Tutorial repo for the article "ML in Production"☆30Updated 2 years ago
- Code supporting Data Science articles at The Marketing Technologist, Floryn Tech Blog, and Pythom.nl☆71Updated 2 years ago
- An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR☆175Updated 5 months ago
- A Scalable Data Cleaning Library for PySpark.☆29Updated 6 years ago
- How to manage Slowly Changing Dimensions with Apache Hive☆55Updated 6 years ago
- Scripts and instructions to facilitate running Deep Learning Tasks on Amazon EMR☆63Updated 2 years ago
- Repository with examples and smoke tests for the GCP Airflow operators and hooks☆152Updated 8 years ago
- ☆16Updated 2 years ago
- The open source version of the Amazon Athena documentation. To submit feedback & requests for changes, submit issues in this repository, …☆84Updated 2 years ago
- CLI tool to launch Spark jobs on AWS EMR☆67Updated 2 years ago
- Helpers & syntactic sugar for PySpark.☆62Updated 2 years ago