webdataset / tarp
Fast and simple stream processing of files in tar files, useful for deep learning, big data, and many other applications.
☆128Updated last year
Alternatives and similar repositories for tarp:
Users that are interested in tarp are comparing it to the libraries listed below
- A small demonstration of using WebDataset with ImageNet and PyTorch Lightning☆74Updated last year
- Examples for the WebDataset PyTorch Dataset Library☆47Updated 3 years ago
- gpu tester detects broken and slow gpus in a cluster☆69Updated 2 years ago
- CLOOB training (JAX) and inference (JAX and PyTorch)☆71Updated 2 years ago
- A python library that provides common I/O interface across different storage backends.☆145Updated 2 months ago
- An unopinionated replacement for PyTorch's Dataset and ImageFolder, that handles Tar archives☆77Updated 2 years ago
- JAX implementation ViT-VQGAN☆82Updated 2 years ago
- Simple and efficient RevNet-Library for PyTorch with XLA and DeepSpeed support and parameter offload☆127Updated 2 years ago
- Contrastive Language-Image Pretraining☆142Updated 2 years ago
- A JAX implementation of the continuous time formulation of Consistency Models☆84Updated 2 years ago
- Simple python template☆41Updated last year
- Optimized library for large-scale extraction of frames and audio from video.☆204Updated last year
- Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI☆87Updated 3 years ago
- Simple large-scale training of stable diffusion with multi-node support.☆131Updated last year
- ☆27Updated 4 years ago
- Implementation of the video diffusion model and training scheme presented in the paper, Flexible Diffusion Modeling of Long Videos, in Py…☆84Updated 2 years ago
- ESGD-M is a stochastic non-convex second order optimizer, suitable for training deep learning models, for PyTorch.☆56Updated 2 years ago
- ☆12Updated 3 years ago
- ☆158Updated 2 years ago
- Named tensors with first-class dimensions for PyTorch☆320Updated last year
- Framework for writing deep learning training loops. Lightweight, and retaining full freedom to design as you see fits. It handles checkpo…☆110Updated last year
- Utilities for sequential processing of tar files.☆24Updated 3 years ago
- ☆36Updated last year
- AdaCat☆49Updated 2 years ago
- v objective diffusion inference code for JAX.☆213Updated 3 years ago
- Re-implementation of 'Grokking: Generalization beyond overfitting on small algorithmic datasets'☆38Updated 3 years ago
- ☆25Updated 4 years ago
- A small repository demonstrating the use of Webdataset and Imagenet☆16Updated last year
- Aggregating embeddings over time☆31Updated 2 years ago
- Implementation of fused cosine similarity attention in the same style as Flash Attention☆213Updated 2 years ago