Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
☆649Feb 24, 2025Updated last year
Alternatives and similar repositories for Craw4LLM
Users that are interested in Craw4LLM are comparing it to the libraries listed below
Sorting:
- This repository contains the resource introduced in the paper: "Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasis"…☆25Oct 15, 2025Updated 4 months ago
- [ACL 2025 Demo] Repository for the demo and paper: ReasonGraph: Visualisation of Reasoning Paths☆513May 23, 2025Updated 9 months ago
- The first open-source agent skills builder. Define skills by vibe workflow, run on Claude Code, Cursor, Codex & more. Build Clawdbot 🦞· …☆6,845Updated this week
- NeuraPress 是一个现代化的 Markdown 编辑器，专注于提供优质的微信公众号排版体验。响应式设计，支持移动设备。搭配 DeepSeek和微信公众号助手使用，碎片时间也能用手机发有排版的文章了。☆1,737Dec 19, 2025Updated 2 months ago
- TrendPublish: 全自动 AI 内容生成与发布系统 | 微信公众号自动化 | 多源数据抓取 (Twitter/X、网站) | DeepseekAI、千问、讯飞模型 | 智能内容分析排序 | 定时发布 | 多模板支持 | Node.js | TypeScript |…☆2,796Jan 8, 2026Updated last month
- OpenSource Production ready Customer service with built in Evals and monitoring☆1,437Jan 12, 2026Updated last month
- Speech to Text but with all the bells and whistles and most importantly AI! AI will clean up your filler words, edit and will refine what…☆328Feb 9, 2025Updated last year
- Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, advanced RAG, advanced summaries, scriptable, …☆509Updated this week
- PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generation☆2,372Sep 10, 2025Updated 5 months ago
- [AAAI 2025] StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization☆227Jan 13, 2026Updated last month
- ☆302Aug 23, 2024Updated last year
- Query and Summarize your chat messages.☆1,032Dec 4, 2024Updated last year
- AI reads books: Page-by-Page PDF Knowledge Extractor & Summarizer. script performs an intelligent page-by-page analysis of PDF books, met…☆1,577Jan 20, 2025Updated last year
- AI ContentCraft is an all-in-one content creation suite that helps creators generate stories, podcast scripts, and multimedia content usi…☆385Jul 4, 2025Updated 7 months ago
- RAG Web UI is an intelligent dialogue system based on RAG (Retrieval-Augmented Generation) technology.☆2,796Dec 8, 2025Updated 2 months ago
- [ICLR 2026] A Training-free Iterative Framework for Long Story Visualization☆947Feb 18, 2026Updated last week
- enhance any agent's browser use skill☆8,076Updated this week
- 🔍 An LLM-based Multi-agent Framework of Web Search Engine (like Perplexity.ai Pro and SearchGPT)☆6,777Jul 4, 2025Updated 7 months ago
- Perplexity style AI Search engine clone built with Gemini 2.0 Flash and Grounding☆2,059Jan 4, 2025Updated last year
- The Level-Navi Agent, a framework that requires no training and utilizes large language models for deep query understanding and precise s…☆81Dec 27, 2024Updated last year
- Code for ACL25-findings. An LLM-based agent simulation framework that simulates human behavior and generates dynamic, text-based social g…☆90Oct 23, 2025Updated 4 months ago
- Open Source Deep Research Alternative to Reason and Search on Private Data. Written in Python.☆7,645Nov 19, 2025Updated 3 months ago
- ☆4,613Feb 13, 2026Updated 2 weeks ago
- Keep searching, reading webpages, reasoning until it finds the answer (or exceeding the token budget)☆5,099Dec 13, 2025Updated 2 months ago
- Toolkit for linearizing PDFs for LLM datasets/training☆16,947Feb 19, 2026Updated last week
- E2M converts various file types (doc, docx, epub, html, htm, url, pdf, ppt, pptx, mp3, m4a) into Markdown. It’s easy to install, with ded…☆1,270Sep 8, 2024Updated last year
- Netflix-level subtitle cutting, translation, alignment, and even dubbing - one-click fully automated AI video subtitle team | Netflix级字幕切…☆16,048May 18, 2025Updated 9 months ago
- "AutoAgent: Fully-Automated and Zero-Code LLM Agent Framework"☆8,616Oct 16, 2025Updated 4 months ago
- ☆2,755May 2, 2025Updated 10 months ago
- 使用AI大模型，一键生成高清故事短视频。Generate high-definition story short videos with one click using AI large models.☆2,147Mar 12, 2025Updated 11 months ago
- Multi-modal OCR pipeline optimized for ML training (text, figure, math, tables, diagrams)☆682May 20, 2025Updated 9 months ago
- A visual playground for agentic workflows: Iterate over your agents 10x faster☆5,694Jul 20, 2025Updated 7 months ago
- Dolphin is a multilingual, multitask ASR model jointly trained by DataoceanAI and Tsinghua University.☆696Nov 27, 2025Updated 3 months ago
- Official Repo for "TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding" [ACL 2025 oral]☆1,462Jul 27, 2025Updated 7 months ago
- Transform PDFs into AI podcasts for engaging on-the-go audio content.☆798Jan 30, 2026Updated last month
- [NeurIPS 2024] The official implementation of "Image Copy Detection for Diffusion Models"☆18Oct 1, 2024Updated last year
- 口袋AI，将世界知识装进口袋。pocketpal-ai 中文版☆568Feb 8, 2025Updated last year
- [EMNLP 2025] OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking☆489Aug 23, 2025Updated 6 months ago
- Using GPT to parse PDF☆3,562Apr 17, 2025Updated 10 months ago