Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
☆1,895Jan 2, 2026Updated 9 months ago
Alternatives and similar repositories for petastorm
Users that are interested in petastorm are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.☆14,674Sep 17, 2026Updated last month
- Koalas: pandas API on Apache Spark☆3,371Mar 20, 2024Updated 2 years ago
- AI Infra / AI Orchestration / AI Control Plane☆3,747Updated this week
- Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.☆44,171Updated this week
- Low-code framework for building custom LLMs, neural networks, and other AI models☆11,791Updated this week
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- The Open Source Feature Store for AI/ML☆7,351Oct 25, 2026Updated last week
- High performance model preprocessing library on PyTorch☆639Mar 29, 2024Updated 2 years ago
- An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models☆4,785Mar 23, 2026Updated 7 months ago
- Distributed Computing for AI Made Simple☆1,049Mar 19, 2023Updated 3 years ago
- MLeap: Deploy ML Pipelines to Production☆1,545Oct 5, 2026Updated 3 weeks ago
- Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per s…☆8,518Updated this week
- Modin: Scale your Pandas workflows by changing a single line of code☆10,391Feb 10, 2026Updated 8 months ago
- The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, a…☆28,609Updated this week
- Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark☆5,251Updated this week
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- 🦉 Data Versioning and ML Experiments☆15,955Updated this week
- A system for quickly generating training data with weak supervision☆6,026Oct 13, 2026Updated 2 weeks ago
- PyTorch extensions for high performance and large scale training.☆3,404Apr 26, 2025Updated last year
- Deep universal probabilistic programming with Python and PyTorch☆9,072Oct 5, 2026Updated 3 weeks ago
- Build, Manage and Deploy AI/ML Systems☆10,331Updated this week
- Parallel computing with task scheduling☆13,941Sep 29, 2026Updated last month
- The Cloud Native AI Platform☆15,942Sep 30, 2026Updated last month
- Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics☆17,264Updated this week
- Data-Centric Pipelines and Data Versioning☆6,313Feb 3, 2025Updated last year
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- PyTorch elastic training☆726Jun 15, 2022Updated 4 years ago
- A uniform interface to run deep learning models from multiple frameworks☆942Jan 3, 2024Updated 2 years ago
- TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.☆3,848Jul 10, 2023Updated 3 years ago
- A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.☆3,212Feb 9, 2026Updated 8 months ago
- cuDF - GPU DataFrame Library☆9,786Updated this week
- An open source python library for automated feature engineering☆7,693Sep 11, 2026Updated last month
- Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also…☆7,841Updated this week
- 📚 Parameterize, execute, and analyze notebooks☆6,498Oct 5, 2026Updated 3 weeks ago
- Production infrastructure for machine learning at scale☆8,008Jun 12, 2024Updated 2 years ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- A low-latency prediction-serving system☆1,420Apr 26, 2021Updated 5 years ago
- Open Source ML Model Versioning, Metadata, and Experiment Management☆1,752Jul 23, 2024Updated 2 years ago
- Deep Learning Pipelines for Apache Spark☆1,988Mar 30, 2023Updated 3 years ago
- Read and write Tensorflow TFRecord data from Apache Spark.☆300Apr 22, 2024Updated 2 years ago
- Hydra is a framework for elegantly configuring complex applications☆10,727Updated this week
- Automated Machine Learning on Kubernetes☆1,707Updated this week
- Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more☆36,551Updated this week