redapt / pyspark-s3-parquet-exampleLinks
This repo demonstrates how to load a sample Parquet formatted file from an AWS S3 Bucket. A python job will then be submitted to a Apache Spark instance running on AWS EMR, which will run a SQLContext to create a temporary table using a DataFrame. SQL queries will then be possible against the temporary table.
☆19Updated 9 years ago
Alternatives and similar repositories for pyspark-s3-parquet-example
Users that are interested in pyspark-s3-parquet-example are comparing it to the libraries listed below
Sorting:
- Airflow workflow management platform chef cookbook.☆71Updated 6 years ago
- A simple introduction to using spark ml pipelines☆26Updated 7 years ago
- AWS Big Data Certification☆25Updated 6 months ago
- scaffold of Apache Airflow executing Docker containers☆85Updated 2 years ago
- Quickstart PySpark with Anaconda on AWS/EMR using Terraform☆47Updated 6 months ago
- ☆16Updated 2 years ago
- Airflow code accompanying blog post.☆21Updated 6 years ago
- Business Data Analysis by HiPIC of CalStateLA☆20Updated 6 years ago
- Code supporting Data Science articles at The Marketing Technologist, Floryn Tech Blog, and Pythom.nl☆71Updated 2 years ago
- An example PySpark project with pytest☆16Updated 7 years ago
- Sentiment Analysis of a Twitter Topic with Spark Structured Streaming☆55Updated 6 years ago
- Mastering Spark for Data Science, published by Packt☆47Updated 2 years ago
- This service is meant to simplify running Google Cloud operations, especially BigQuery tasks. This means you do not have to worry about …☆45Updated 6 years ago
- Real-Time Data Processing Pipeline & Visualization with Docker, Spark, Kafka and Cassandra☆85Updated 8 years ago
- Docker container for Kafka - Spark Streaming - Cassandra☆98Updated 6 years ago
- ☆17Updated 6 years ago
- Data validation library for PySpark 3.0.0☆33Updated 2 years ago
- A Getting Started Guide for developing and using Airflow Plugins☆93Updated 6 years ago
- Basic tutorial of using Apache Airflow☆36Updated 6 years ago
- Demonstration code for MLeap, both Jupyter notebooks and projects☆24Updated 5 years ago
- Kafka Connect playground☆10Updated 5 years ago
- Code examples for the Introduction to Kubeflow course☆14Updated 4 years ago
- CLI tool to launch Spark jobs on AWS EMR☆67Updated last year
- A curated list of all the awesome examples, articles, tutorials and videos for Apache Airflow.☆96Updated 4 years ago
- 🚨 Simple, self-contained fraud detection system built with Apache Kafka and Python☆88Updated 6 years ago
- This repository contains recipes for Apache Pinot.☆30Updated 4 months ago
- A single docker image that combines Neo4j Mazerunner and Apache Spark GraphX into a powerful all-in-one graph processing engine☆46Updated 5 years ago
- ☆111Updated 6 months ago
- The sane way of building a data layer in Airflow☆24Updated 5 years ago
- The open source version of the Amazon Redshift Cluster Management Guide.☆48Updated 2 years ago