aws-samples / aws-glue-streaming-ingestion-from-kafka-to-apache-iceberg
This is a collecton of Amazon CDK projects to show how to directly ingest streaming data from Amazon Mananged Service for Apache Kafka (MSK) and MSK Serverless into Apache Iceberg table in S3 with AWS Glue Streaming.
☆11Updated 6 months ago
Alternatives and similar repositories for aws-glue-streaming-ingestion-from-kafka-to-apache-iceberg:
Users that are interested in aws-glue-streaming-ingestion-from-kafka-to-apache-iceberg are comparing it to the libraries listed below
- Streaming ETL job cases in AWS Glue to integrate Iceberg and creating an in-place updatable data lake on Amazon S3☆22Updated 6 months ago
- ☆69Updated 9 months ago
- dbt (data build tool) projects targeting AWS analytics services (redshift, glue, emr, athena) and open table formats☆29Updated last year
- Sample code to collect Apache Iceberg metrics for table monitoring☆25Updated 7 months ago
- ☆14Updated 2 years ago
- Build, Test and Deploy ETL solutions using AWS Glue and AWS CDK based CI/CD pipelines☆42Updated 2 years ago
- ☆31Updated last year
- Framework to enforce long term health of your AWS Data Lake by providing visibility into operational, data quality and business metrics.☆17Updated 3 years ago
- ☆30Updated last year
- The data product processor is a library for dynamically creating and executing Apache Spark Jobs based on a declarative description of a …☆13Updated 10 months ago
- Stream CDC into an Amazon S3 data lake in Apache Iceberg table format with AWS Glue Streaming and DMS☆30Updated last month
- ☆16Updated 8 months ago
- ☆34Updated 2 years ago
- This repository provides the resources required for the Amazon Redshift Streaming workshop☆12Updated last year
- ☆26Updated 7 months ago
- A VS Code Extension to make it easier to manage and develop Spark jobs on EMR☆31Updated last month
- Operational Data Processing Framework developed using AWS Glue and Apache Hudi. This framework is suitable for Data Lake and Modern Data …☆21Updated last year
- ☆12Updated 7 months ago
- This repository contains ready-to-use notebook examples for a wide variety of use cases in Amazon EMR Studio.☆50Updated last year
- ☆21Updated 6 months ago
- A CLI to manage and monitor permissions in AWS Lake Formation☆26Updated 2 years ago
- ☆16Updated 2 years ago
- An open-source framework that simplifies implementation of data solutions.☆128Updated this week
- Kinesis Data Analytics Blueprints are a curated collection of Apache Flink applications. Each blueprint will walk you through how to solv…☆16Updated last year
- ☆12Updated 5 months ago
- Demo code to illustrate the execution of PyTest unit test cases for AWS Glue jobs in AWS CodePipeline using AWS CodeBuild projects☆44Updated 3 months ago
- A collection of examples built with AWS DataOps Development Kit (DDK)☆41Updated last month
- ☆19Updated 5 months ago
- ☆15Updated last year