KastanDay / video-pretrained-transformer
Multi-model video-to-text by combining embeddings from Flan-T5 + CLIP + Whisper + SceneGraph. The 'backbone LLM' is pre-trained from scratch on YouTube (YT-1B dataset).
☆51Updated last year
Related projects ⓘ
Alternatives and complementary repositories for video-pretrained-transformer
- Supercharged BLIP-2 that can handle videos☆116Updated 11 months ago
- Multimodal Video Understanding Framework (MVU)☆24Updated 6 months ago
- Code release for "EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone" [ICCV, 2023]☆90Updated 4 months ago
- [CVPR 2023] HierVL Learning Hierarchical Video-Language Embeddings☆44Updated last year
- Codes and Models for COSA: Concatenated Sample Pretrained Vision-Language Foundation Model☆39Updated last year
- [ECCV 2024🔥] Official implementation of the paper "ST-LLM: Large Language Models Are Effective Temporal Learners"☆125Updated 2 months ago
- Language Repository for Long Video Understanding☆29Updated 5 months ago
- EILeV: Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties☆117Updated 2 weeks ago
- Data-Efficient Multimodal Fusion on a Single GPU☆47Updated 6 months ago
- [ECCV2024] Official code implementation of Merlin: Empowering Multimodal LLMs with Foresight Minds☆82Updated 4 months ago
- [ACL 2024 Findings] "TempCompass: Do Video LLMs Really Understand Videos?", Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, …☆89Updated last week
- VideoLLM: Modeling Video Sequence with Large Language Models☆154Updated last year
- Code for paper "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos"☆82Updated 3 months ago
- ☆120Updated last month
- ☆72Updated 6 months ago
- ☆210Updated 7 months ago
- Code release for the paper "Egocentric Video Task Translation" (CVPR 2023 Highlight)☆31Updated last year
- [CVPR 2023] Official code for "Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations"☆51Updated last year
- [ICCV2023] EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understanding☆75Updated last year
- Pytorch Implementation of the Model from "MIRASOL3B: A MULTIMODAL AUTOREGRESSIVE MODEL FOR TIME-ALIGNED AND CONTEXTUAL MODALITIES"☆24Updated 2 weeks ago
- Official implementation for "A Simple LLM Framework for Long-Range Video Question-Answering"☆85Updated 3 weeks ago
- [CVPR 2024] ViT-Lens: Towards Omni-modal Representations☆164Updated 4 months ago
- PG-Video-LLaVA: Pixel Grounding in Large Multimodal Video Models☆245Updated 10 months ago
- ☆146Updated last month
- ☆90Updated 6 months ago
- A simple reproducible template to implement AI research papers☆23Updated 2 months ago
- FunQA benchmarks funny, creative, and magic videos for challenging tasks including timestamp localization, video description, reasoning, …☆96Updated 4 months ago
- [PR 2024] A large Cross-Modal Video Retrieval Dataset with Reading Comprehension☆22Updated 10 months ago
- [NeurIPS 2024] Official implementation of the paper "Interfacing Foundation Models' Embeddings"☆112Updated 3 months ago
- ☆39Updated last year