a simple distributed spider in Java. Java编写的一个简单分布式爬虫
☆160Jun 18, 2013Updated 13 years ago
Alternatives and similar repositories for spider
Users that are interested in spider are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- 使用java+httpclient+httpcleaner，多线程、分布式爬去电商网站商品信息，数据存储在hbase上，并使用solr对商品建立索引，使用redis队列存储一个共享的url仓库；使用zookeeper对爬虫节点生命周期进行监视等。☆233Nov 6, 2020Updated 6 years ago
- 新浪微博爬虫，采用Java语言开发，基于HTTPClient 4.0，采用MySQL存储爬取数据，支持多进程并发执行。功能包括：爬取微博、评论、转发、关注列表（层次）。根据数据需求，持续更新...☆353Feb 27, 2014Updated 12 years ago
- 利用HttpClient4+实现网络小说爬虫，可动态添加热门的小说网站☆30Sep 6, 2012Updated 14 years ago
- 利用spring boot + webmagic 开发的java爬虫系统☆60Dec 29, 2016Updated 9 years ago
- DistributeCrawler的Maven版☆10Jun 20, 2022Updated 4 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- java分布式爬虫，主机和从机控制的机制☆14May 21, 2015Updated 11 years ago
- java implementation framework from github☆13Apr 8, 2015Updated 11 years ago
- dubbo-brave集成☆10Mar 16, 2016Updated 10 years ago
- 网络爬虫☆49Mar 18, 2014Updated 12 years ago
- 分布式脚手架框架（总结整理）☆14Aug 27, 2015Updated 11 years ago
- 分布式在线聊天系统☆10Sep 17, 2014Updated 12 years ago
- 抓取各报社报纸信息－采用配置文件形式实现的一个简单的可定制爬虫☆11Sep 1, 2022Updated 4 years ago
- 爬虫项目源码整理，使用redis进行url缓存，hbase进行详细信息的存储。使用zookeeper进行爬虫线程的状态监控。☆17Oct 7, 2015Updated 11 years ago
- ZkConfig是为zookeeper开发的配置服务工具包，能与现有的Java系统进行良好的集成，也可以使用与非java系统以独立进程运行。提供与spring进行集成的插件。采用注解方式对需要动态更新的内存数据对象进行标注。 ZkConfig用于解决在系统集群中配置文件的实…☆27Apr 17, 2015Updated 11 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- Mail Renderer, Mail Queue, Task Manager☆14Oct 19, 2016Updated 10 years ago
- Java无框架实现爬取知乎用户信息、图片和知乎推荐内容并下载到本地或数据库中☆388Jan 21, 2017Updated 9 years ago
- A configurable web spider with a easy-to-use web console☆996Jun 3, 2026Updated 5 months ago
- 基于hadoop思维的分布式网络爬虫。☆85Mar 8, 2016Updated 10 years ago
- Toolkit that can bundle any Spring Boot application into an Apache Ambari Service, enabling Ambari to provision, manage and monitor the s…☆13Oct 9, 2015Updated 11 years ago
- 大规模分布式系统的跟踪、监控、告警平台☆56May 15, 2014Updated 12 years ago
- 定制爬虫工具（sqlserver版），通过正则表达式自定义抓取模版，通过自定义数据模型入库☆10Sep 5, 2017Updated 9 years ago
- jw, short for java web, 模仿spring, 实现一个简单可用的java web框架☆20Jun 20, 2022Updated 4 years ago
- github上一些觉得对自己工作有用的项目收集☆12Apr 22, 2015Updated 11 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- 一个可以实现关键词搜索的网络爬虫☆10Feb 25, 2014Updated 12 years ago
- Java基础服务器底层架构。 -- made by alzq.zhf☆27Aug 11, 2015Updated 11 years ago
- 一个简单易用的爬虫框架,内置代理管理模块,灵活设置多线程爬取☆63Feb 23, 2017Updated 9 years ago
- 华南理工大学高英实验室进行的分布式爬虫项目,除了实验室内部人员外,不得私自传播.☆21Jul 13, 2014Updated 12 years ago
- BigData Workflow Engine for Hadoop, Hbase, Netezza, Pig, Hive, Redis ...☆13Dec 8, 2021Updated 4 years ago
- 基于canal的mysql slave实现☆12Apr 7, 2013Updated 13 years ago
- ⭐️⭐️⭐️java多线程系列教程(从入门到进阶+源码解读+面试题)https://www.qiuchenglei.top☆16Sep 14, 2022Updated 4 years ago
- mybatis-generator-core mybatis自动生成代码工具、ibatis自动生成代码工具，自己定制了下。☆43Mar 5, 2019Updated 7 years ago
- 反网页爬虫系统☆39Mar 10, 2015Updated 11 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- 全国失信被执行人爬取☆11Sep 1, 2017Updated 9 years ago
- 基于 Java 的免费开源跨平台系统运行监控服务应用及引擎框架。A free, open source, cross-platform system running monitoring service application and engine framework ba…☆17Aug 13, 2019Updated 7 years ago
- The world's leading data crawler platform!☆15Nov 22, 2018Updated 7 years ago
- 一个基于微博用户数据的Java爬虫项目☆316Aug 18, 2020Updated 6 years ago
- 基于mybatis + spring + maven + mysql 自动生成的工具☆13Apr 24, 2013Updated 13 years ago
- A simple demonstration of a hot deployment plugin framework, which can dynamically load and unload jar files. 热部署框架示例，可以实现插件的热插拔，动态加载、卸载j…☆13Dec 17, 2016Updated 9 years ago
- 新闻评论观点挖掘系统，粗粒度的分析出新闻网评观点的倾向和走势☆54Jun 1, 2015Updated 11 years ago