Alternatives and similar repositories for crawling-framework:

Users that are interested in crawling-framework are comparing it to the libraries listed below

• tokenmill / fast-url-access-checker
  Easily run HTTP GET requests against a list of URLs to check their HTTP status.
  ☆12Updated 5 years ago
• tokenmill / beagle
  Beagle helps you identify keywords, phrases, regexes, and complex search queries of interest in streams of text documents.
  ☆53Updated 3 years ago
• accelerated-text / reaction-acc-text-demo
  Integration between Reaction ECommerce and Accelerated Text to provide product descriptions for an e-shop.
  ☆12Updated 4 years ago
• o19s / hello-nlp
  A natural language search microservice
  ☆96Updated 4 years ago
• raindrum / citeurl
  an extensible tool to generate hyperlinks from legal citations
  ☆33Updated 5 months ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆113Updated this week
• kermitt2 / entity-fishing
  A machine learning tool for fishing entities
  ☆262Updated last week
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆168Updated 2 months ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆51Updated 4 years ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 3 years ago
• lfoppiano / grobid-quantities
  GROBID extension for identifying and normalizing physical quantities.
  ☆80Updated 6 months ago
• mkoniari / LegalParser
  This page is a companion for the paper titled Towards Automatic Structuring and Semantic Indexing of Legal Documents
  ☆29Updated 6 years ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆54Updated 2 months ago
• opensemanticsearch / open-semantic-search-apps
  Python/Django based webapps and web user interfaces for search, structure (meta data management like thesaurus, ontologies, annotations a…
  ☆96Updated 2 years ago
• alephdata / synonames
  Trying to generate name synonyms from wikidata
  ☆32Updated 4 years ago
• iipc / jwarc
  Java library for reading and writing WARC files with a typed API
  ☆49Updated 2 months ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆190Updated 2 years ago
• kermitt2 / grobid-ner
  A Named-Entity Recogniser based on Grobid.
  ☆50Updated 6 months ago
• crawler-commons / url-frontier
  API definition, resources and reference implementation of URL Frontiers
  ☆48Updated last month
• algoriffic / lsa4solr
  Document clustering based on Latent Semantic Analysis
  ☆96Updated 14 years ago
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆188Updated 6 years ago
• searchhub / search-collector
  A fast and simple JavaScript library specifically targeted at collecting search and search related browser events.
  ☆40Updated 6 months ago
• DiceTechJobs / VectorsInSearch
  Dice.com repo to accompany the dice.com 'Vectors in Search' talk by Simon Hughes, from the Activate 2018 search conference, and the 'Sear…
  ☆85Updated 3 years ago
• trivio / common_crawl_index
  Index URLs in Common Crawl
  ☆193Updated 7 years ago
• iliaschalkidis / LegalCrawler
  LegalCrawler: A tool for automated scraping of English legal corpora
  ☆53Updated 2 years ago
• nationalarchives / ds-caselaw-data-enrichment-service
  Judgment citation annotations for the National Archives Find Case Law service
  ☆22Updated this week
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆44Updated 7 years ago
• tballison / lucene-addons
  Standalone versions of LUCENE_5205 and other patches: SpanQueryParser, Concordance and Co-occurrence stats
  ☆18Updated 3 years ago
• amyxzhang / wikum
  tool for collectively summarizing large discussions
  ☆143Updated 2 years ago
• srijiths / jtopia
  Java clone for python term extractor topia.termextract
  ☆34Updated 10 years ago