chuqiaoshen / Git-Influencer
Insight Data Engineering project: A platform built in HDFS, Spark and Airflow to help you to find social influencers from GitHub Network.
☆16Updated 5 months ago
Related projects ⓘ
Alternatives and complementary repositories for Git-Influencer
- 🚨 Simple, self-contained fraud detection system built with Apache Kafka and Python☆83Updated 5 years ago
- Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validatio…☆53Updated last year
- The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on…☆25Updated 2 years ago
- Data engineering interviews Q&A for data community by data community☆61Updated 4 years ago
- Udacity Data Pipeline Exercises☆15Updated 4 years ago
- Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database☆13Updated 3 years ago
- A real-time event pipeline around Kafka Ecosystem for Chicago Transit Authority.☆29Updated last year
- Basic tutorial of using Apache Airflow☆35Updated 6 years ago
- Project files for the post: Running PySpark Applications on Amazon EMR using Apache Airflow: Using the new Amazon Managed Workflows for A…☆41Updated 2 years ago
- Use Kafka and Apache Spark streaming to perform click stream analytics☆76Updated 4 years ago
- Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…☆89Updated 2 years ago
- Public source code for the Batch Processing with Apache Beam (Python) online course☆19Updated 4 years ago
- Design/Implement stream/batch architecture on NYC taxi data | #DE☆26Updated 3 years ago
- Jupyter notebooks for pyspark tutorials given at University☆104Updated 2 months ago
- AWS Big Data Certification☆25Updated last year
- Learning from multiple companies in Silicon Valley. Netflix, Facebook, Google, Startups☆16Updated 6 years ago
- Code examples for the Introduction to Kubeflow course☆13Updated 3 years ago
- A code-based tutorial for production level data streaming with PySpark plus Optimus for data cleaning, Confluent Kafka, & Apache Drill u…☆26Updated 5 years ago
- Sample Airflow DAGs to load data from the CovidTracking API to Snowflake via an AWS S3 intermediary.☆16Updated 3 years ago
- Sentiment Analysis of a Twitter Topic with Spark Structured Streaming☆55Updated 5 years ago
- Apache Spark Interview Question and Answers☆21Updated 4 years ago
- ☆23Updated 5 years ago
- Because its never late to start taking notes and 'public' it...☆60Updated 3 weeks ago
- How to build an awesome data engineering team☆99Updated 5 years ago
- Full stack data engineering tools and infrastructure set-up☆43Updated 3 years ago
- PySpark phonetic and string matching algorithms☆35Updated 8 months ago
- A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for …☆132Updated 4 years ago
- Amazon Redshift Cookbook, Published by Packt☆15Updated last year
- Blog post on ETL pipelines with Airflow☆23Updated 4 years ago
- Challenge for those applying to the Software Engineer, Big Data position☆34Updated 13 years ago