awslabs / s3-connector-for-pytorchLinks
The Amazon S3 Connector for PyTorch delivers high throughput for PyTorch training jobs that access and store data in Amazon S3.
☆195Updated 3 weeks ago
Alternatives and similar repositories for s3-connector-for-pytorch
Users that are interested in s3-connector-for-pytorch are comparing it to the libraries listed below
Sorting:
- TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and sup…☆411Updated this week
- ☆111Updated last year
- Container plugin for Slurm Workload Manager☆407Updated last week
- Module, Model, and Tensor Serialization/Deserialization☆283Updated 4 months ago
- Scalable and Performant Data Loading☆362Updated this week
- Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)☆467Updated 2 weeks ago
- ☆63Updated 3 weeks ago
- EFA/NCCL base AMI build Packer and CodeBuild/Pipeline files. Also base Docker build files to enable EFA/NCCL in containers☆43Updated 2 years ago
- Training and inference on AWS Trainium and Inferentia chips.☆254Updated this week
- ☆58Updated last week
- Example code for AWS Neuron SDK developers building inference and training applications☆154Updated 3 weeks ago
- ☆274Updated this week
- The Triton backend for the PyTorch TorchScript models.☆170Updated this week
- Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and i…☆571Updated this week
- ☆72Updated 11 months ago
- A helper library to connect into Amazon SageMaker with AWS Systems Manager and SSH (Secure Shell)☆258Updated 6 months ago
- Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.☆382Updated last week
- CUDA checkpoint and restore utility☆403Updated 4 months ago
- NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the …☆247Updated this week
- JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs wel…☆398Updated last week
- A tool to configure, launch and manage your machine learning experiments.☆214Updated this week
- A library to analyze PyTorch traces.☆454Updated last month
- A performant, memory-efficient checkpointing library for PyTorch applications, designed with large, complex distributed workloads in mind…☆162Updated 3 weeks ago
- A CLI tool that helps manage training jobs on the SageMaker HyperPod clusters orchestrated by Amazon EKS☆34Updated last week
- PyTorch Single Controller☆948Updated this week
- Speed up model training by fixing data loading.☆568Updated last week
- Kubernetes Operator, ansible playbooks, and production scripts for large-scale AIStore deployments on Kubernetes.☆118Updated this week
- Deploying EFA in EKS utilizing GPUDirectRDMA where supported☆36Updated last year
- Create and manage Amazon SageMaker HyperPod clusters, run distributed model training☆25Updated this week
- Create, List, Update, Delete Amazon EKS clusters. Deploy and manage software on EKS. Run distributed model training and inference example…☆64Updated last month