Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
☆1,890Jan 2, 2026Updated 7 months ago
Alternatives and similar repositories for petastorm
Users that are interested in petastorm are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.☆14,693Jul 29, 2026Updated last week
- Koalas: pandas API on Apache Spark☆3,371Mar 20, 2024Updated 2 years ago
- AI Infra / AI Orchestration / AI Control Plane☆3,718Updated this week
- Low-code framework for building custom LLMs, neural networks, and other AI models☆11,748Updated this week
- Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.☆43,497Updated this week
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- The Open Source Feature Store for AI/ML☆7,205Updated this week
- High performance model preprocessing library on PyTorch☆641Mar 29, 2024Updated 2 years ago
- The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, a…☆27,466Updated this week
- An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models☆4,768Mar 23, 2026Updated 4 months ago
- 🦉 Data Versioning and ML Experiments☆15,807Updated this week
- Simple and Distributed Machine Learning☆5,236Updated this week
- MLeap: Deploy ML Pipelines to Production☆1,540Jul 21, 2026Updated 3 weeks ago
- Build, Manage and Deploy AI/ML Systems☆10,207Updated this week
- Modin: Scale your Pandas workflows by changing a single line of code☆10,391Feb 10, 2026Updated 6 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- PyTorch extensions for high performance and large scale training.☆3,408Apr 26, 2025Updated last year
- Distributed Computing for AI Made Simple☆1,047Mar 19, 2023Updated 3 years ago
- PyTorch elastic training☆726Jun 15, 2022Updated 4 years ago
- Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per s…☆8,509Apr 1, 2026Updated 4 months ago
- Parallel computing with task scheduling☆13,886Updated this week
- A uniform interface to run deep learning models from multiple frameworks☆943Jan 3, 2024Updated 2 years ago
- Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics☆17,004Updated this week
- A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.☆3,158Feb 9, 2026Updated 6 months ago
- cuDF - GPU DataFrame Library☆9,726Updated this week
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Dynamic, resilient AI orchestration. Coordinate data, models, and compute as you build AI workflows.☆7,195Updated this week
- Machine Learning Toolkit for Kubernetes☆15,810Jul 10, 2026Updated last month
- Data-Centric Pipelines and Data Versioning☆6,303Feb 3, 2025Updated last year
- Deep universal probabilistic programming with Python and PyTorch☆9,036Aug 4, 2026Updated last week
- TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.☆3,846Jul 10, 2023Updated 3 years ago
- A system for quickly generating training data with weak supervision☆6,003Jun 8, 2026Updated 2 months ago
- Open Source ML Model Versioning, Metadata, and Experiment Management☆1,749Jul 23, 2024Updated 2 years ago
- 📚 Parameterize, execute, and analyze notebooks☆6,465Jul 6, 2026Updated last month
- Deep Learning Pipelines for Apache Spark☆1,989Mar 30, 2023Updated 3 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Production infrastructure for machine learning at scale☆8,013Jun 12, 2024Updated 2 years ago
- Read and write Tensorflow TFRecord data from Apache Spark.☆300Apr 22, 2024Updated 2 years ago
- An open source python library for automated feature engineering☆7,665Jul 27, 2026Updated 2 weeks ago
- Hydra is a framework for elegantly configuring complex applications☆10,590Updated this week
- A low-latency prediction-serving system☆1,420Apr 26, 2021Updated 5 years ago
- AIStore: scalable storage for AI applications☆1,912Updated this week
- Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more☆36,140Updated this week