Includes notes on using Apache Spark, with drill down on Spark for Physics, how to run TPCDS on PySpark, how to create histograms with Spark. Also tools for stress testing, measuring CPUs' performance, and I/O latency heat maps. Jupyter notebooks examples for using various DB systems.
☆464May 19, 2026Updated 6 months ago
Alternatives and similar repositories for Miscellaneous
Users that are interested in Miscellaneous are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simp…☆832Oct 16, 2026Updated last month
- Spark-Dashboard is an open-source monitoring solution for Apache Spark that provides real-time performance dashboards using containers an…☆141Sep 18, 2026Updated 2 months ago
- Code and examples of how to write and deploy Apache Spark Plugins. Spark plugins allow runnig custom code on the executors as they are in…☆96May 11, 2026Updated 6 months ago
- This is a mirror of https://github.com/LucaCanali/sparkMeasure - sparkMeasure is a tool for performance troubleshooting of Apache Spark w…☆16May 21, 2026Updated 5 months ago
- Qubole Sparklens tool for performance tuning Apache Spark☆593Jun 26, 2024Updated 2 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Spark metrics related custom classes and sinks (e.g. Prometheus)☆185Aug 2, 2022Updated 4 years ago
- A library that provides useful extensions to Apache Spark and PySpark.☆240Sep 10, 2026Updated 2 months ago
- Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.☆1,627Updated this week
- Hadoop Profiler, or hprofiler, is a tool which is able to analyze on- and off-CPU workloads on distributed computing environments.☆24Jul 7, 2016Updated 10 years ago
- Custom Alerts for Ambari server☆12Jul 27, 2015Updated 11 years ago
- The Internals of Delta Lake☆185Jun 18, 2026Updated 5 months ago
- Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.☆255Feb 21, 2023Updated 3 years ago
- Material for the course "Introduction to Apache Spark APIs for Data Processing" https://sparktraining.web.cern.ch/☆21May 13, 2025Updated last year
- Intel® Performance Counter Monitor (Intel® PCM)☆3,361Nov 13, 2026Updated last week
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.☆431Jan 14, 2022Updated 4 years ago
- Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange☆129Dec 19, 2024Updated last year
- The Internals of Spark SQL☆492Updated this week
- Provides a set of benchmarks that can be used to measure the memory bandwidth performance of CPU's☆95Apr 8, 2024Updated 2 years ago
- Bulletproof Apache Spark jobs with fast root cause analysis of failures.☆73Mar 14, 2021Updated 5 years ago
- Scripts for generating Grafana dashboards for monitoring Spark jobs☆238Mar 26, 2015Updated 11 years ago
- Tool for gathering blocks and replicas meta data from HDFS. It also builds a heat map showing how replicas are distributed along disks an…☆55May 9, 2017Updated 9 years ago
- Rocksdb state storage implementation for Structured Streaming.☆17Oct 21, 2020Updated 6 years ago
- The Internals of Apache Spark☆1,553Oct 10, 2026Updated last month
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Jupyter magics and kernels for working with remote Spark clusters☆1,369Updated this week
- Monitor Apache Spark from Jupyter Notebook☆172May 16, 2022Updated 4 years ago
- A Spark Reliability Testing Suite☆13Jan 10, 2017Updated 9 years ago
- Helpers & syntactic sugar for PySpark.☆61Sep 11, 2026Updated 2 months ago
- A re-implementation of Hadoop DistCP in Apache Spark☆47Dec 20, 2023Updated 2 years ago
- Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark☆1,371Aug 22, 2023Updated 3 years ago
- Base classes to use when writing tests with Spark☆1,555Oct 31, 2026Updated 3 weeks ago
- JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter☆1,800May 21, 2026Updated 5 months ago
- Collection of oracle scripts☆25Mar 3, 2020Updated 6 years ago
- Proton VPN Special Offer - Get 70% off • AdSpecial partner offer. Trusted by over 100 million users worldwide. Tested, Approved and Recommended by Experts.
- Takes AWR reports as input and produces 1-page summary of key instrumentation data to identify and help reason about inconsistencies☆16Feb 22, 2019Updated 7 years ago
- ☆312Nov 26, 2018Updated 7 years ago
- Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.☆3,160Updated this week
- Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.☆1,081Updated this week
- A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.☆345May 31, 2024Updated 2 years ago
- NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs☆1,021Updated this week
- Allows reading ROOT TTrees into Apache Spark as DataFrames☆12Jun 14, 2023Updated 3 years ago
Are these results useful?