Apache Spark is an open-source, distributed computing system designed for big data processing and analytics, offering an alternative to traditional MapReduce models with improved performance and ease of use. It provides a unified analytics engine capable of handling large-scale data processing tasks efficiently by leveraging in-memory computation and a resilient distributed dataset (RDD) framework. Spark supports a variety of programming languages such as Java, Scala, Python, and R, enabling developers to build sophisticated data pipelines with ease. Additionally, it integrates seamlessly with a variety of data sources like Hadoop Distributed File System (HDFS), Apache HBase, and Apache Cassandra, among others. The ecosystem includes libraries for SQL (Spark SQL), streaming data (Spark Streaming), machine learning (MLlib), and graph processing (GraphX), making it highly versatile in addressing a wide range of data processing and analytic workloads across various industries. For application developers, Apache Spark provides the tools to scale applications efficiently and handle complex data transformations, ultimately enabling faster insights from data-intensive applications.
View the most prominent open source Apache Spark projects in the list below. Click on a specific project to view its alternative or complementary packages. Make comparisons and find the best package for your app.
- Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼☆45,829Sep 15, 2026Updated last week
- Apache Spark - A unified analytics engine for large-scale data processing☆44,070Updated this week
- Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce,…☆29,359Mar 20, 2024Updated 2 years ago
- Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.☆28,816Updated this week
- 最新Docker容器技术，从真实案例中学习最佳实践！| Learn and understand Docker&Container technologies, with real DevOps practice!☆26,271Updated this week
- 大数据入门指南☆16,970Jan 5, 2024Updated 2 years ago
- List of Data Science Cheatsheets to rule the world☆16,353Jul 18, 2024Updated 2 years ago
- GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.☆15,277Sep 16, 2026Updated last week
- flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Ta…☆15,098May 6, 2026Updated 4 months ago
- 【大厂面试专栏】一份Java程序员需要的技术指南，这里有面试题、系统架构、职场锦囊、主流中间件等，让你成为更牛的自己！☆14,740Jul 21, 2025Updated last year
- Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.☆14,682Sep 17, 2026Updated last week
- Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and …☆14,265Updated this week
- 专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...☆10,553Aug 7, 2023Updated 3 years ago
- Python SQL Parser and Transpiler☆9,639Updated this week
- An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Tr…☆9,021Updated this week
- 🧙 Build, run, and manage data pipelines for integrating and transforming data.☆8,826Sep 11, 2026Updated 2 weeks ago
- H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random F…☆7,510Updated this week
- Alluxio, data orchestration for analytics and machine learning in the cloud☆7,245Sep 1, 2026Updated 3 weeks ago
- A Flexible and Powerful Parameter Server for large-scale machine learning☆6,793Sep 18, 2026Updated last week
- Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.☆6,661Updated this week
- macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime …☆6,270Feb 27, 2023Updated 3 years ago
- Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark☆5,247Updated this week
- cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek…☆5,076Jul 11, 2026Updated 2 months ago
- State of the Art Natural Language Processing☆4,158Updated this week
- The Hunting ELK☆3,929Jun 1, 2024Updated 2 years ago
- A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others☆3,935Sep 10, 2026Updated 2 weeks ago
- TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.☆3,846Jul 10, 2023Updated 3 years ago
- Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.☆3,648Sep 16, 2026Updated last week
- 酷玩 Spark: Spark 源代码解析、Spark 类库等☆3,473May 18, 2022Updated 4 years ago
- 🔨 用 JSON 来生成结构化的 SQL 语句，基于 Vue3 + TypeScript + Vite + Ant Design + MonacoEditor 实现，项目简单（重逻辑轻页面）、适合练手~☆3,417Jan 17, 2024Updated 2 years ago
- Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications…☆3,411Sep 13, 2026Updated 2 weeks ago
- Koalas: pandas API on Apache Spark☆3,372Mar 20, 2024Updated 2 years ago
- Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch …☆3,408Updated this week
- DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitizati…☆3,268Nov 4, 2025Updated 10 months ago
- LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, …☆3,254Updated this week
- Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.☆3,398Updated this week
- 大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料☆3,241Jan 20, 2026Updated 8 months ago
- Interactive and Reactive Data Science using Scala and Spark.☆3,139May 16, 2023Updated 3 years ago
- Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.☆3,156Sep 18, 2026Updated last week
- REST job server for Apache Spark☆2,835Mar 3, 2026Updated 6 months ago