aws-samples / iceberg-streaming-examplesLinks
This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenarios using best practices. The code can be deployed into any Spark compatible engine like Amazon EMR Serverless or AWS Glue. A fully local developer environment is also provided.
☆27Updated 3 months ago
Alternatives and similar repositories for iceberg-streaming-examples
Users that are interested in iceberg-streaming-examples are comparing it to the libraries listed below
Sorting:
- Sample code to collect Apache Iceberg metrics for table monitoring☆28Updated last year
- ☆11Updated 10 months ago
- Spark runtime on AWS Lambda☆110Updated last month
- This repository contains ready-to-use notebook examples for a wide variety of use cases in Amazon EMR Studio.☆52Updated last year
- Streaming ETL job cases in AWS Glue to integrate Iceberg and creating an in-place updatable data lake on Amazon S3☆25Updated last year
- ☆16Updated 2 years ago
- Example code for running Spark and Hive jobs on EMR Serverless.☆168Updated 9 months ago
- Amazon EMR Notebook to show how to read from and write to Delta tables with Amazon EMR☆17Updated 5 months ago
- dbt / Amazon Redshift Demonstration Project☆34Updated 2 years ago
- dbt (data build tool) projects targeting AWS analytics services (redshift, glue, emr, athena) and open table formats☆30Updated 2 years ago
- This repository contains the dbt-glue adapter☆135Updated this week
- Sample solution to automate tedious regulatory compliance processes using multi-agent systems☆17Updated 5 months ago
- ☆15Updated 2 months ago
- ☆73Updated last year
- Operational Data Processing Framework developed using AWS Glue and Apache Hudi. This framework is suitable for Data Lake and Modern Data …☆23Updated 2 years ago
- ☆16Updated last month
- Describes the concepts of lambda architecture and the actual deployment process with an example of building a serverless business intelli…☆15Updated 4 months ago
- Unity Catalog UI☆43Updated last year
- ☆54Updated 2 months ago
- Jupyter notebooks and AWS CloudFormation template to show how Hudi, Iceberg, and Delta Lake work☆47Updated 3 years ago
- Build DataOps platform with Apache Airflow and dbt on AWS☆59Updated 4 years ago
- ☆25Updated last year
- Using DuckDB with AWS Lambda to process Delta Lake data☆30Updated 8 months ago
- A Python package to help Databricks Unity Catalog users to read and query Delta Lake tables with Polars, DuckDb, or PyArrow.☆26Updated last year
- Amazon Managed Workflows for Apache Airflow (MWAA) Examples repository contains example DAGs, requirements.txt, plugins, and CloudFormati…☆116Updated 3 months ago
- ☆10Updated 6 months ago
- Cost Efficient Data Pipelines with DuckDB☆57Updated 4 months ago
- Demonstrating the capabilities of DuckDB as a transformation engine for data lakes☆30Updated last year
- ☆18Updated last year
- JumpSpark - A modern cookiecutter template for pyspark projects with batteries included.☆10Updated 2 years ago