dvryaboy / idl_storage_guidelines
This document attempts to capture useful patterns and warn about subtle gotchas when it comes to designing and evolving schemas for long-term serialized data. It is not intended as a guide for how to best represent a particular dataset or process.
☆13Updated 7 years ago
Related projects ⓘ
Alternatives and complementary repositories for idl_storage_guidelines
- ☆21Updated last year
- Spooker is a dynamic framework for processing high volume data streams via processing pipelines☆29Updated 8 years ago
- kafka-connect-s3 : Ingest data from Kafka to Object Stores(s3)☆95Updated 5 years ago
- An application that records stats about consumer group offset commits and reports them as prometheus metrics☆14Updated 5 years ago
- A Kafka-Connect Sink for S3 with no Hadoop dependencies.☆57Updated last year
- Dione - a Spark and HDFS indexing library☆50Updated 8 months ago
- Spark Structured Streaming State Tools☆34Updated 4 years ago
- Avro Schema Shredder is a REST API that enables storage of Avro Schemas in Apache Atlas. This API enables an organization to use Apache A…☆13Updated 7 years ago
- Building Scio from scratch step by step☆20Updated 5 years ago
- Experimental mechanism for forwarding data regardless of B3 sampling☆10Updated 10 months ago
- Cascading on Apache Flink®☆54Updated 9 months ago
- A library for strong, schema based conversion between 'natural' JSON documents and Avro☆18Updated 8 months ago
- A small project to report offset lag for Kafka Consumer Groups via Burrow.☆31Updated 5 years ago
- A small project to allow publishing data to Apache Kafka, Apache Pulsar or any other target system☆14Updated 4 years ago
- Spark stream from kafka(json) to s3(parquet)☆15Updated 6 years ago
- Graph Analytics with Apache Kafka☆101Updated last week
- Circus Train is a dataset replication tool that copies Hive tables between clusters and clouds.☆86Updated 8 months ago
- Set of tools for creating backups, compaction and restoration of Apache Kafka® Clusters☆18Updated this week
- Playbook to provision a Confluent Cluster☆10Updated 7 years ago
- Use cases built on SnappyData. Use cases contained here: 1. Ad Analytics 2. Streaming data ingestion from RabbitMQ.☆32Updated 2 years ago
- ## Auto-archived due to inactivity. ## Simple JVM Profiler Using StatsD and Other Metrics Backends☆15Updated last year
- Shunting Yard is a real-time data replication tool that copies data between Hive Metastores.☆20Updated 3 years ago
- Demonstration of a Hive Input Format for Iceberg☆26Updated 3 years ago
- Automated rack-aware assignment of Kafka partitions to brokers☆62Updated 4 years ago
- Use SQL to transform your avro schema/records☆28Updated 6 years ago
- Kafka Connect Tooling☆118Updated 3 years ago
- ☆22Updated 5 years ago
- Data Sketches for Apache Spark☆21Updated last year
- Simple Samza Job Using Confluent Platform☆15Updated 8 years ago