apache / incubator-stormcrawler

Related projects ⓘ

Alternatives and complementary repositories for incubator-stormcrawler

• USCDataScience / sparkler
  Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
  ☆410Updated last year
• crawler-commons / crawler-commons
  A set of reusable Java components that implement functionality common to any web crawler
  ☆237Updated this week
• scrapinghub / frontera
  A scalable frontier for web crawlers
  ☆1,302Updated last year
• Norconex / crawlers
  Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or fi…
  ☆183Updated this week
• istresearch / scrapy-cluster
  This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
  ☆1,182Updated last year
• apache / nutch
  Apache Nutch is an extensible and scalable web crawler
  ☆2,923Updated 3 weeks ago
• internetarchive / heritrix3
  Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
  ☆2,837Updated this week
• jaeksoft / opensearchserver
  Open-source Enterprise Grade Search Engine Software
  ☆500Updated 2 years ago
• apache / samza
  Mirror of Apache Samza
  ☆820Updated last month
• dragnet-org / dragnet
  Just the facts -- web page content extraction
  ☆1,254Updated 4 months ago
• jayzeng / scrapy-elasticsearch
  A scrapy pipeline which send items to Elastic Search server
  ☆327Updated 2 years ago
• DigitalPebble / behemoth
  Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
  ☆281Updated 6 years ago
• commoncrawl / news-crawl
  News crawling with StormCrawler - stores content as WARC
  ☆322Updated 11 months ago
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆214Updated last year
• elastic / elasticsearch-hadoop
  Elasticsearch real-time search and analytics natively integrated with Hadoop
  ☆9Updated last week
• lucidworks / banana
  Banana for Solr - A Port of Kibana
  ☆668Updated 3 months ago
• karussell / snacktory
  Readability clone in Java
  ☆461Updated 4 years ago
• kohlschutter / boilerpipe
  Work in progress transmit from Google Code
  ☆1,109Updated 6 years ago
• apache / gobblin
  A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, orga…
  ☆2,230Updated this week
• apache / drill
  Apache Drill is a distributed MPP query layer for self describing data
  ☆1,948Updated this week
• lucidworks / spark-solr
  Tools for reading data from Solr as a Spark RDD and indexing objects from Spark into Solr using SolrJ.
  ☆445Updated 11 months ago
• VIDA-NYU / ache
  ACHE is a web crawler for domain-specific search.
  ☆454Updated last year
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆181Updated 6 years ago
• OryxProject / oryx
  Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
  ☆1,788Updated 3 years ago
• spinscale / elasticsearch-opennlp-plugin
  Additional opennlp mapping type for elasticsearch in order to perform named entity recognition
  ☆136Updated 8 years ago
• flaxsearch / luwak
  A java library for stored queries
  ☆374Updated last year
• scrapy / scrapyd-client
  Command line client for Scrapyd server
  ☆770Updated last month
• scrapinghub / webstruct
  NER toolkit for HTML data
  ☆257Updated 6 months ago
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆289Updated 6 months ago
• apache / gora
  The Apache Gora open source framework provides an in-memory data model and persistence for big data.
  ☆119Updated 8 months ago