paulpierre / markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
☆342Updated 5 months ago
Alternatives and similar repositories for markdown-crawler:
Users that are interested in markdown-crawler are comparing it to the libraries listed below
- Extract structured text from pdfs quickly☆383Updated this week
- Structured information extraction from documents☆297Updated 3 months ago
- Parse PDFs into markdown using Vision LLMs☆197Updated 2 weeks ago
- Yet another open source Perplexity☆404Updated 3 months ago
- 90% of what you need for LLM app development. Nothing you don't.☆227Updated last week
- HTML to Markdown converter and crawler.☆507Updated last year
- 🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with PostgreSQL or SQLite☆678Updated this week
- Generic rag framework to apply the power of LLMs on any given dataset☆499Updated this week
- Use OpenAI's realtime API for a chatting with your documents☆306Updated 3 months ago
- A prompting library☆150Updated 3 months ago
- This repository hosts a suite of specialized agents designed to power your brainstorming sessions. Each agent brings a unique perspective…☆279Updated 2 months ago
- SearchGPT / Perplexity Pages clone, but personalised for you.☆229Updated 4 months ago
- Lightweight, performant, deep table extraction☆387Updated last month
- On-premises conversational RAG with configurable containers☆297Updated this week
- Minimalist LLM Framework in 100 Lines. Enable LLMs to Program Themselves.☆211Updated this week
- Your first AI prompt engineer☆357Updated 2 months ago
- ☆259Updated 6 months ago
- Prompt optimization scratch☆555Updated last week
- An experiment in meeting transcription and diarization with just an LLM. Maybe I went a little overboard though☆368Updated 2 months ago
- High-performance retrieval engine for unstructured data☆1,121Updated last week
- Flexible and powerful multi-agent AI framework☆334Updated this week
- LLM for Long Text Summary (Comprehensive Bulleted Notes)☆468Updated this week
- Awesome MCP Servers - A curated list of Model Context Protocol servers☆329Updated this week
- A simple Python program to implement the search-extract-summarize flow.☆234Updated last month
- The simplest open-source implementation of perplexity.ai☆281Updated 4 months ago
- ☆167Updated this week
- ☆152Updated 3 months ago
- No-code ETL and data pipelines with AI and NLP☆275Updated 2 months ago
- An enterprise-grade AI retriever designed to streamline AI integration into your applications, ensuring cutting-edge accuracy.☆279Updated this week
- ☆198Updated last month