dbusteed / spark-structured-streaming

Related projects: ⓘ

• dbusteed / kafka-spark-streaming-example
  ☆36Updated 4 years ago
• LearningJournal / Spark-Streaming-In-Python
  Apache Spark 3 - Structured Streaming Course Material
  ☆120Updated last year
• pran4ajith / spark-twitter-streaming
  A real-time streaming ETL pipeline for streaming and performing sentiment analysis on Twitter data using Apache Kafka, Apache Spark and D…
  ☆29Updated 4 years ago
• dsynkov / spark-livy-on-airflow-workspace
  A workspace to experiment with Apache Spark, Livy, and Airflow in a Docker environment.
  ☆39Updated 3 years ago
• kroudir / Data-Engineer-Nanodegree-Projects-Udacity
  Projects done in the Data Engineer Nanodegree Program by Udacity.com
  ☆83Updated last year
• rodrigo-arenas / kafkaml-anomaly-detection
  Project for real-time anomaly detection using Kafka and python
  ☆55Updated last year
• confluentinc / coding-in-motion
  Source code for the "Coding in Motion" series.
  ☆24Updated 2 years ago
• mahmoudparsian / data-algorithms-with-spark
  O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian
  ☆202Updated last year
• airscholar / e2e-data-engineering
  An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Ka…
  ☆177Updated 11 months ago
• josephmachado / beginner_de_project_stream
  Simple stream processing pipeline
  ☆89Updated 3 months ago
• sdesilva26 / docker-spark
  Tutorial for setting up a Spark cluster running inside of Docker containers located on different machines
  ☆121Updated last year
• yennanliu / NYC_Taxi_Pipeline
  Design/Implement stream/batch architecture on NYC taxi data | #DE
  ☆26Updated 3 years ago
• shravan-kuchkula / udacity-data-eng-proj-1
  Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation,…
  ☆89Updated 2 years ago
• HamzaG737 / data-engineering-project
  End to end data engineering project with kafka, airflow, spark, postgres and docker.
  ☆54Updated last month
• KentHsu / Udacity-Data-Engineering-Nanodgree
  Udacity Data Engineering Nanodegree Program
  ☆51Updated 3 years ago
• immu0001 / Udacity-Data-Engineer-nanodegree
  Classwork projects and home works done through Udacity data engineering nano degree
  ☆74Updated 9 months ago
• simardeep1792 / Data-Engineering-Streaming-Project
  ☆35Updated 2 months ago
• dogukannulu / kafka_spark_structured_streaming
  Get data from API, run a scheduled script with Airflow, send data to Kafka and consume with Spark, then write to Cassandra
  ☆125Updated last year
• Aiven-Labs / python-notebooks-for-apache-kafka
  A Series of Notebooks on how to start with Kafka and Python
  ☆151Updated last year
• tatwan / airflow-spark-aws-emr
  ☆11Updated 3 years ago
• indiacloudtv / structuredstreamingkafkapyspark
  Apche Spark Structured Streaming with Kafka using Python(PySpark)
  ☆41Updated 5 years ago
• indiacloudtv / pyspark_on_google_colab
  PySpark Tutorial for Beginners on Google Colab: Hands-On Guide
  ☆16Updated 4 years ago
• cordon-thiago / airflow-spark
  Docker with Airflow and Spark standalone cluster
  ☆239Updated last year
• MBKraus / incremental_training
  Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and MLFlow'
  ☆120Updated last year
• SatadruMukherjee / Data-Preprocessing-Models
  ☆59Updated this week
• guoliveira / data-engineer-zoomcamp-project
  ☆35Updated last year
• danieldiamond / data-engineering-capstone
  Data Engineering Capstone Project: ETL Pipelines and Data Warehouse Development
  ☆20Updated 5 years ago
• zhujun98 / data-engineering
  Spark, Airflow, Kafka
  ☆27Updated last year
• moj-analytical-services / etl-pipeline-example
  An example of an ETL pipeline that lays out generic DE processes. This is now out of date but still provides useful information
  ☆26Updated 2 years ago
• dogukannulu / streaming_data_processing
  Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO
  ☆56Updated last year