Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
☆1,894Jan 2, 2026Updated 10 months ago
Alternatives and similar repositories for petastorm
Users that are interested in petastorm are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.☆14,675Sep 17, 2026Updated 2 months ago
- Koalas: pandas API on Apache Spark☆3,369Mar 20, 2024Updated 2 years ago
- AI Infra / AI Orchestration / AI Control Plane☆3,747Updated this week
- Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.☆44,340Updated this week
- Low-code framework for building custom LLMs, neural networks, and other AI models☆11,792Updated this week
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- The Open Source Feature Store for AI/ML☆7,362Updated this week
- High performance model preprocessing library on PyTorch☆641Mar 29, 2024Updated 2 years ago
- An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models☆4,781Mar 23, 2026Updated 7 months ago
- Distributed Computing for AI Made Simple☆1,049Mar 19, 2023Updated 3 years ago
- Modin: Scale your Pandas workflows by changing a single line of code☆10,389Feb 10, 2026Updated 9 months ago
- Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark☆5,257Updated this week
- MLeap: Deploy ML Pipelines to Production☆1,544Nov 9, 2026Updated last week
- Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per s…☆8,518Nov 6, 2026Updated 2 weeks ago
- Deep universal probabilistic programming with Python and PyTorch☆9,078Oct 5, 2026Updated last month
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, a…☆28,899Updated this week
- 🦉 Data Versioning and ML Experiments☆15,985Updated this week
- A system for quickly generating training data with weak supervision☆6,028Oct 13, 2026Updated last month
- PyTorch extensions for high performance and large scale training.☆3,404Apr 26, 2025Updated last year
- Parallel computing with task scheduling☆13,959Updated this week
- Build, Manage and Deploy AI/ML Systems☆10,354Updated this week
- A uniform interface to run deep learning models from multiple frameworks☆941Jan 3, 2024Updated 2 years ago
- The Cloud Native AI Platform☆15,973Sep 30, 2026Updated last month
- Data-Centric Pipelines and Data Versioning☆6,311Feb 3, 2025Updated last year
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.☆3,848Jul 10, 2023Updated 3 years ago
- PyTorch elastic training☆726Jun 15, 2022Updated 4 years ago
- A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.☆3,222Feb 9, 2026Updated 9 months ago
- Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics☆17,320Updated this week
- cuDF - GPU DataFrame Library☆9,801Updated this week
- Deep Learning Pipelines for Apache Spark☆1,986Mar 30, 2023Updated 3 years ago
- 📚 Parameterize, execute, and analyze notebooks☆6,499Oct 5, 2026Updated last month
- Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also…☆7,814Updated this week
- An open source python library for automated feature engineering☆7,696Sep 11, 2026Updated 2 months ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Production infrastructure for machine learning at scale☆8,005Jun 12, 2024Updated 2 years ago
- Open Source ML Model Versioning, Metadata, and Experiment Management☆1,750Jul 23, 2024Updated 2 years ago
- A low-latency prediction-serving system☆1,416Apr 26, 2021Updated 5 years ago
- Hydra is a framework for elegantly configuring complex applications☆10,757Updated this week
- WARNING: Katib project is DEPRECATED, users should migrate to OptimizationJob and Kubeflow Trainer☆1,707Updated this week
- Read and write Tensorflow TFRecord data from Apache Spark.☆300Apr 22, 2024Updated 2 years ago
- Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more☆36,657Updated this week
Are these results useful?