paulpierre / markdown-crawlerLinks
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
☆391Updated 10 months ago
Alternatives and similar repositories for markdown-crawler
Users that are interested in markdown-crawler are comparing it to the libraries listed below
Sorting:
- Parse PDFs into markdown using Vision LLMs☆393Updated 4 months ago
- An experiment in meeting transcription and diarization with just an LLM. Maybe I went a little overboard though☆553Updated 3 weeks ago
- HTML to Markdown converter and crawler.☆571Updated last year
- Structured information extraction from documents☆315Updated 9 months ago
- SearchGPT / Perplexity Pages clone, but personalised for you.☆243Updated 9 months ago
- 🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL☆1,021Updated last week
- ☆268Updated last year
- Extract structured text from pdfs quickly☆497Updated 2 weeks ago
- A cool AI Diagram generator from a given topic, that streams the partial diagrams from the incomplete JSONs during generation. Built usin…☆211Updated last year
- Yet another open source Perplexity☆447Updated 8 months ago
- Generic rag framework to apply the power of LLMs on any given dataset☆629Updated last week
- openperplex is an opensource AI search engine☆864Updated 10 months ago
- Super performant RAG pipelines for AI apps. Summarization, Retrieve/Rerank and Code Interpreters in one simple API.☆377Updated last year
- The simplest open-source implementation of perplexity.ai☆313Updated 5 months ago
- Visualize Different Text Splitting Methods☆263Updated 5 months ago
- Easily deployable 🚀 API to convert PDF to markdown quickly with high accuracy.☆870Updated 8 months ago
- Detect and extract tables to markdown and csv☆749Updated 5 months ago
- ScribeWizard: Generate organized notes from audio using Groq, Whisper, and Llama3☆491Updated 5 months ago
- High-performance retrieval engine for unstructured data☆1,419Updated last week
- Infinite Bookshelf: Generate entire books in seconds using Groq and Llama3☆1,310Updated 6 months ago
- ☆225Updated 2 weeks ago
- Local semantic search. Stupidly simple.☆432Updated 11 months ago
- A simple Python program to implement the search-extract-summarize flow.☆269Updated last week
- A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.☆328Updated 2 weeks ago
- An enterprise-grade AI retriever designed to streamline AI integration into your applications, ensuring cutting-edge accuracy.☆285Updated last month
- ☆435Updated 8 months ago
- Multi-agent that helps you organize and write documents.☆333Updated 7 months ago
- An autoagentic AGI that is self-evolving and modular.☆955Updated 9 months ago
- Home of the AI workforce - Multi-agent system, AI agents & tools☆239Updated last month
- HawkinsDB is our take on giving AI systems a more human-like way to store and recall information, inspired by how our own brains work. Ba…☆294Updated 6 months ago