paulpierre / markdown-crawlerLinks
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
☆394Updated 11 months ago
Alternatives and similar repositories for markdown-crawler
Users that are interested in markdown-crawler are comparing it to the libraries listed below
Sorting:
- Parse PDFs into markdown using Vision LLMs☆409Updated 6 months ago
- Yet another open source Perplexity☆450Updated 9 months ago
- HTML to Markdown converter and crawler.☆583Updated last year
- 90% of what you need for LLM app development. Nothing you don't.☆266Updated last month
- 📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simp…☆237Updated 3 weeks ago
- SearchGPT / Perplexity Pages clone, but personalised for you.☆244Updated 11 months ago
- ScribeWizard: Generate organized notes from audio using Groq, Whisper, and Llama3☆494Updated 6 months ago
- ☆231Updated last month
- Easily deployable 🚀 API to convert PDF to markdown quickly with high accuracy.☆881Updated 9 months ago
- Visualize Different Text Splitting Methods☆284Updated 7 months ago
- Extract structured text from pdfs quickly☆523Updated last month
- A simple Python program to implement the search-extract-summarize flow.☆269Updated last month
- ☆191Updated last month
- Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, advanced RAG, advanced summaries, scriptable, …☆475Updated 2 weeks ago
- Super performant RAG pipelines for AI apps. Summarization, Retrieve/Rerank and Code Interpreters in one simple API.☆381Updated last year
- This project enhances the construction of RAG applications by addressing challenges, improving accessibility, scalability, and managing d…☆146Updated last year
- Clone of https://r.jina.ai which is deployable locally☆47Updated 10 months ago
- ☆88Updated last year
- No-code ETL and data pipelines with AI and NLP☆317Updated 5 months ago
- ⚡Chat with GitHub Repo Using 200k context window of Claude instead of RAG!⚡☆169Updated last year
- Your first AI prompt engineer☆401Updated last month
- Connect and chat with your multiple documents (pdf and txt) through GPT 3.5, GPT-4 Turbo, Claude and Local Open-Source LLMs☆799Updated 6 months ago
- LLM for Long Text Summary (Comprehensive Bulleted Notes)☆580Updated last month
- Web scraper made for AI and simplicity in mind. It runs as a CLI that can be parallelized and outputs high-quality markdown content.☆524Updated this week
- Structured information extraction from documents☆317Updated 10 months ago
- Excel spreadsheet crawler and table parser for data extraction and querying☆150Updated 5 months ago
- An experimental UI for text-to-knowledge-graph generation☆777Updated last year
- The simplest open-source implementation of perplexity.ai☆316Updated 6 months ago
- Self-hosted version of Microsoft's OmniParser Image-to-text model☆71Updated 2 months ago
- Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extr…☆206Updated last month