KastanDay / video-pretrained-transformer
Multi-model video-to-text by combining embeddings from Flan-T5 + CLIP + Whisper + SceneGraph. The 'backbone LLM' is pre-trained from scratch on YouTube (YT-1B dataset).
☆53Updated last year
Alternatives and similar repositories for video-pretrained-transformer:
Users that are interested in video-pretrained-transformer are comparing it to the libraries listed below
- [ICLR2024] Codes and Models for COSA: Concatenated Sample Pretrained Vision-Language Foundation Model☆41Updated 2 months ago
- [CVPR 2023] Official code for "Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations"☆52Updated last year
- VideoLLM: Modeling Video Sequence with Large Language Models☆156Updated last year
- Code release for "EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone" [ICCV, 2023]☆95Updated 8 months ago
- [ICCV2023] EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understanding☆75Updated last year
- Code for “Pretrained Language Models as Visual Planners for Human Assistance”☆60Updated last year
- Code release for the paper "Egocentric Video Task Translation" (CVPR 2023 Highlight)☆32Updated last year
- [CVPR 2023] HierVL Learning Hierarchical Video-Language Embeddings☆45Updated last year
- Language Repository for Long Video Understanding☆31Updated 8 months ago
- FunQA benchmarks funny, creative, and magic videos for challenging tasks including timestamp localization, video description, reasoning, …☆96Updated 2 months ago
- [PR 2024] A large Cross-Modal Video Retrieval Dataset with Reading Comprehension☆24Updated last year
- Official Repository of VideoLLaMB: Long Video Understanding with Recurrent Memory Bridges☆65Updated this week
- Graph learning framework for long-term video understanding☆59Updated last month
- ☆162Updated 4 months ago
- ☆72Updated 9 months ago
- LAVIS - A One-stop Library for Language-Vision Intelligence☆48Updated 7 months ago
- EILeV: Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties☆119Updated 3 months ago
- Hierarchical Video-Moment Retrieval and Step-Captioning (CVPR 2023)☆99Updated last month
- Supercharged BLIP-2 that can handle videos☆117Updated last year
- ☆19Updated 11 months ago
- Learning to cut end-to-end pretrained modules☆30Updated 7 months ago
- Code for paper "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos"☆95Updated this week
- ☆31Updated last year
- [CVPR 2024] Data and benchmark code for the EgoExoLearn dataset☆54Updated 6 months ago
- [CVPR 2024] Official PyTorch implementation of the paper "One For All: Video Conversation is Feasible Without Video Instruction Tuning"☆30Updated last year
- ☆31Updated 5 months ago
- Democratization of "PaLI: A Jointly-Scaled Multilingual Language-Image Model"☆88Updated 11 months ago
- [ICLR 2025] CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion☆40Updated last month
- ☆70Updated 2 months ago
- [NeurIPS2023] Official implementation of the paper "Large Language Models are Visual Reasoning Coordinators"☆104Updated last year