paulpierre / markdown-crawlerLinks
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
☆404Updated last year
Alternatives and similar repositories for markdown-crawler
Users that are interested in markdown-crawler are comparing it to the libraries listed below
Sorting:
- HTML to Markdown converter and crawler.☆591Updated last year
- Parse PDFs into markdown using Vision LLMs☆431Updated last week
- Yet another open source Perplexity☆453Updated 11 months ago
- Extract structured text from pdfs quickly☆610Updated 4 months ago
- An experimental UI for text-to-knowledge-graph generation☆781Updated last year
- Visualize Different Text Splitting Methods☆292Updated 9 months ago
- Easily deployable 🚀 API to convert PDF to markdown quickly with high accuracy.☆906Updated 11 months ago
- SearchGPT / Perplexity Pages clone, but personalised for you.☆245Updated last year
- Clone of https://r.jina.ai which is deployable locally☆48Updated last year
- 90% of what you need for LLM app development. Nothing you don't.☆263Updated last month
- ⚡Chat with GitHub Repo Using 200k context window of Claude instead of RAG!⚡☆169Updated last year
- 📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simp…☆253Updated 2 months ago
- ScribeWizard: Generate organized notes from audio using Groq, Whisper, and Llama3☆498Updated 2 months ago
- ☆89Updated last year
- Lightweight, performant, deep table extraction☆513Updated 2 months ago
- Connect and chat with your multiple documents (pdf and txt) through GPT 3.5, GPT-4 Turbo, Claude and Local Open-Source LLMs☆795Updated 8 months ago
- Octogen is an Open-Source Code Interpreter Agent Framework☆258Updated last year
- A cool AI Diagram generator from a given topic, that streams the partial diagrams from the incomplete JSONs during generation. Built usin …☆214Updated last year
- Social and customizable AI writing assistant! ✍️☆253Updated 3 months ago
- No-code ETL and data pipelines with AI and NLP☆316Updated 7 months ago
- Structured information extraction from documents☆317Updated last year
- 💻📚💡 DoctorGPT provides advanced LLM prompting for PDFs and webpages.☆245Updated last year
- Web scraper made for AI and simplicity in mind. It runs as a CLI that can be parallelized and outputs high-quality markdown content.☆528Updated last week
- A simple Python program to implement the search-extract-summarize flow.☆270Updated 3 months ago
- The simplest open-source implementation of perplexity.ai☆318Updated 8 months ago
- Super performant RAG pipelines for AI apps. Summarization, Retrieve/Rerank and Code Interpreters in one simple API.☆385Updated last year
- ☆237Updated 4 months ago
- Detect and extract tables to markdown and csv☆752Updated 8 months ago
- Official implement of paper "AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation" [EMNLP 24']☆476Updated 9 months ago
- Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, advanced RAG, advanced summaries, scriptable, …☆483Updated this week