GuozhongCrawler的是一个无须配置、便于二次开发的爬虫开源框架，它提供简单灵活的API，只需少量代码即可实现一个爬虫。其设计灵感来源于多个爬虫国内外爬虫框架的总结。采用完全模块化的设计，功能覆盖整个爬虫的生命周期(链接提取、页面下载、内容抽取、持久化)，支持多线程抓取，分布式抓取，并支持自动重试，定制执行js、自定义cookie等功能。在处理网站抓取多次后被封IP的问题上，guozhongCrawler采用动态轮换IP机制有效防止IP被封。另外，源码中的注释及Log输出全部采用通俗易懂的中文。让初学者能有更加深刻的理解
☆103Apr 20, 2015Updated 11 years ago
Alternatives and similar repositories for GuozhongCrawler
Users that are interested in GuozhongCrawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- 读书笔记《自己动手写网络爬虫》，自己敲的代码。主要 记录了网络爬虫的基本实现，网页去重的算法，网页指纹算法，文本信息挖掘☆47Jan 9, 2015Updated 11 years ago
- 华南理工大学高英实验室进行的分布式爬虫项目,除了实验室内部人员外,不得私自传播.☆21Jul 13, 2014Updated 12 years ago
- dubbox 2.8.1版本 在其基础之上进行扩展☆10Dec 16, 2014Updated 11 years ago
- A simple and flexible web crawler framework for java.☆20Apr 22, 2018Updated 8 years ago
- Utmost是一款基于Java的面向业务配置平台,能够迅速搭建企业级应用,集成了业界优秀的开源框架,内置任务调度,多维权限管理,规则引擎,工作流引擎,系统监控,ETL,报表,打印等企业常用组件.拥有一流的用户体验和丰富的界面展现.☆25Mar 28, 2017Updated 9 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- 一个基于trie树的具有联想功能的文本编辑器。采用python和pyqt☆10Sep 7, 2016Updated 10 years ago
- 轻量级的事件驱动和异步框架☆40Sep 1, 2024Updated 2 years ago
- image server base on nosql☆10Aug 24, 2016Updated 10 years ago
- 一个简单易用的爬虫框架,内置代理管理模块,灵活设置多线程爬取☆63Feb 23, 2017Updated 9 years ago
- 利用WebMagic框架进行58同城数据的抓取☆12Oct 13, 2014Updated 12 years ago
- 豆瓣电影爬虫——a crawler which is able to crawl movie detail and short comments, save them to database mysql, also include Sentiment analysis ba…☆68Mar 24, 2019Updated 7 years ago
- 基于 Java 的免费开源跨平台系统运行监控服务应用及引擎框架。A free, open source, cross-platform system running monitoring service application and engine framework ba…☆17Aug 13, 2019Updated 7 years ago
- 使用java+httpclient+httpcleaner，多线程、分布式爬去电商网站商品信息，数据存储在hbase上，并使用solr对商品建立索引，使用redis队列存储一个共享的url仓库；使用zookeeper对爬虫节点生命周期进行监视等。☆233Nov 6, 2020Updated 6 years ago
- 日志注入(字节码增强）☆13Jan 23, 2017Updated 9 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- 一个简易的搜索引擎，采用Java开发☆33Mar 7, 2014Updated 12 years ago
- 阿里云钉钉交流群收集☆13Feb 13, 2019Updated 7 years ago
- Using GroovyClassLoader to hotswap stateless business logic class☆12Mar 27, 2014Updated 12 years ago
- Distributed in-memory cube base on java8 stream☆17Dec 6, 2014Updated 11 years ago
- 每天三分钟的科技新闻聚合阅读☆17May 15, 2018Updated 8 years ago
- ☆20May 16, 2017Updated 9 years ago
- 一个集分布式爬虫,分布式存储,分布式计算统计分析一体的统计分析数据挖掘项目☆14Feb 6, 2018Updated 8 years ago
- jfinalQ对应的代码生成工具☆16Sep 28, 2015Updated 11 years ago
- 数学函数，表达式，公式，坐标，长度，圆，圆弧，方程，计算式的Java语言实现。☆12Dec 28, 2016Updated 9 years ago
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- 租房爬虫，基于flask，采用apscheduler定时任务，通过微信，定时给用户推送想要的租房信息☆15Mar 13, 2019Updated 7 years ago
- Package seimicrawler project so that can be fast and standalone deployed.It is based on maven-war-plugin and modified. 这是专为SeimiCrawl…☆14Jun 30, 2022Updated 4 years ago
- DDPush任意门推送，开源免费实时信息推送服务器，官网http://www.ddpush.net/☆24Mar 11, 2015Updated 11 years ago
- 数据平台(DataPlateform)，最初的设计想法是：当今大数据横行，我们也不能落后。所以就想着写一个这样的平台系统。此项目集爬虫、搜索、Hadoop、Dwr推送、Quartz定时任务于一体的平台，其目的是想通过抓取互联网数据，通过大数据推测人或者某一事物的下一行为。C…☆18Jul 31, 2017Updated 9 years ago
- blog.☆16Oct 8, 2015Updated 11 years ago
- JAVA COMET UserUnicast framework based on Pushlet. 基于pushlet的用户点播JAVA comet实现框架。☆11Aug 13, 2019Updated 7 years ago
- 基于Java语言编写的轻量级分库分表(Sharding)中间件，丰富的Sharding算法支持(2类4种分片算法)，能够方便DBA实现库的极速扩容和降低数据迁移成本。Kratos站在巨人的肩膀上(SpringJdbc)，采用与应用集成架构，放弃通用性，只为换取更好的执行性能…☆38Sep 3, 2015Updated 11 years ago
- 智能客服 基于springboot+swaggger+elasticsearch+mysql☆11Aug 22, 2018Updated 8 years ago
- JAVA开源关键词提取框架☆10Nov 26, 2014Updated 11 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- requests升级版requests-html 爬虫编写及通用爬虫模块搭建☆11Nov 21, 2018Updated 7 years ago
- ☆22Dec 30, 2016Updated 9 years ago
- 采用领域驱动设计（Domain Driven Design、DDD）实现的CMS系统☆11Aug 8, 2022Updated 4 years ago
- TSOFA - The Simple, Offline Flashcard App. Written in JavaScript for browsers.☆15May 19, 2026Updated 5 months ago
- 文字自动生成视频 - 文字生成视频的AI工具软件汇总☆14Apr 17, 2025Updated last year
- Android game using AndEngine☆11Nov 1, 2013Updated 13 years ago
- 一款基于iframe的单页应用组件（single page application）☆58May 19, 2022Updated 4 years ago