peopleindreamdontsleep/SparkanSpider external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

peopleindreamdontsleep/SparkanSpider

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/peopleindreamdontsleep/SparkanSpider)

Close

peopleindreamdontsleep / SparkanSpiderView on GitHubMore

• External links
• Readme badge

java爬虫，反爬虫策略、ETL清洗数据，以及spark离线和实时分析新闻并存入ES

☆19Nov 26, 2018Updated 7 years ago

Alternatives and similar repositories for SparkanSpider

Users that are interested in SparkanSpider are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• crystal-tensor / spide

  View on GitHub
  网络爬虫 主要抓取的是股票数据，外汇数据，股票背景资料，股票及时新闻
  ☆13Aug 13, 2018Updated 8 years ago
• 51bigdata / StockData2Hbase

  View on GitHub
  股票交易数据处理的整个业务流程 数据源--->数据采集--->数据归类--->数据储存--->数据分析--->数据可视化
  ☆31Nov 23, 2016Updated 9 years ago
• seawaylee / spark-rec-v2

  View on GitHub
  Spark混合推荐系统大数据监控平台
  ☆11May 1, 2018Updated 8 years ago
• lxf44944 / sinaNews_crawler

  View on GitHub
  新浪新闻爬虫
  ☆15Feb 14, 2015Updated 11 years ago
• minminmsn / accesslog-analysis-alarm

  View on GitHub
  日志分析产品，该解决方案整合了filebeat、kafka、logstash、elasticsearch、kibana、grafana、elastalert等开源产品，能够实现海量日志实时分析及错误报警，另外还具有日常报表功能
  ☆23Jan 11, 2019Updated 7 years ago
• Deploy to Railway using AI coding agents - Free Credits Offer • Ad
  Use Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
• changge458 / userdraw

  View on GitHub
  用户画像代码，根据算法推算出用户的性别和年龄比率
  ☆11Dec 18, 2017Updated 8 years ago
• zhoufwind / palo-deploy

  View on GitHub
  使用shell脚本部署Apache Doris (incubating) FE & BE
  ☆11Jul 8, 2019Updated 7 years ago
• JasonWiki / dw_etl

  View on GitHub
  dw etl 工具 mysql 增量、全量抽取 to hive. 合并 hive 数据表, 等数据平台清洗工具
  ☆10Dec 21, 2016Updated 9 years ago
• JP6907 / RealTimeLogAnalyze

  View on GitHub
  一个大数据实时流处理日志分析系统 Demo
  ☆30Nov 16, 2022Updated 3 years ago
• peopleindreamdontsleep / StructuredStreamingInSQL

  View on GitHub
  sql实现Structured Streaming
  ☆40Jan 4, 2019Updated 7 years ago
• orangeMask / spider

  View on GitHub
  抖音,淘宝系,常见新闻爬虫
  ☆13Apr 15, 2022Updated 4 years ago
• gitzh750vp520 / OnlineEdu

  View on GitHub
  视频教育网站
  ☆17Sep 25, 2018Updated 7 years ago
• digfound / sinacrawler

  View on GitHub
  第一次编写Python网络爬虫，主要使用beautifulsoup4爬取新浪新闻首页新闻列表。成功获取新闻标题、时间、来源、详情、评论数、编辑信息，使用pandas整理数据，并保存到数据库。
  ☆13Dec 7, 2017Updated 8 years ago
• jiangyuanyuan / lotterySpider

  View on GitHub
  Based on the Scrapy framework, crawling crawlers ------------------ 基于Scrapy 框架开发 抓取新闻的爬虫 -------------
  ☆13Jul 26, 2019Updated 7 years ago
• Managed Kubernetes at scale on DigitalOcean • Ad
  DigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
• madpudding / RelationshipCrawler

  View on GitHub
  知网、万方、专利局爬虫
  ☆11Mar 20, 2019Updated 7 years ago
• chmod740 / BaiduBaikeSpider

  View on GitHub
  百度百科多线程爬虫Java源码，数据存储采用了Oracle11g
  ☆13Feb 23, 2017Updated 9 years ago
• gugug / WeiboNewsProject

  View on GitHub
  系统从互联网爬取资讯，对热点公共事件进行检测、聚合和追踪，多维度分析事件内容，监测时间传播路径，分析用户的观点和和情绪，形成摘要、报告、图表等分析结果，实现公共事件的舆情分析可视化系统，提供专业的舆情监测、分析和预警服务
  ☆102Jul 4, 2018Updated 8 years ago
• goozp / ths-spider-example

  View on GitHub
  完整的 scrapy 爬虫示例，爬取股票和新闻数据
  ☆17Aug 15, 2020Updated 6 years ago
• KeepSorted / PushServer

  View on GitHub
  A high performance push server based on SpringBoot + ConcurrentHashMap + Netty
  ☆16Oct 22, 2019Updated 6 years ago
• CrawlScript / WeiboLoginTool

  View on GitHub
  基于WebCollector的新浪微博爬虫及相关登录工具，如新浪微博Cookie获取
  ☆14Nov 21, 2018Updated 7 years ago
• fullstackyang / weibo-crawler

  View on GitHub
  新型的免登录微博爬虫，自动获取Cookie直接进行抓取和解析微博数据，免去了账号登录的过程，彻底摆脱账号被封的困扰
  ☆37Oct 15, 2017Updated 8 years ago
• Ingram7 / NewsinaSpider

  View on GitHub
  Scrapy 新浪新闻爬虫
  ☆12Aug 26, 2019Updated 6 years ago
• OneGeek / GWT-CompletableFuture

  View on GitHub
  GWT compatible implementation of java.util.concurrent.CompletableFuture and supporting classes
  ☆24Dec 10, 2021Updated 4 years ago
• Managed hosting for WordPress and PHP on Cloudways • Ad
  Managed hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
• linzhouzhi / mix-query

  View on GitHub
  基于 spark 混合查询平台，支持不同源数据库的联合查询，mysql hive presto ...
  ☆14Aug 3, 2017Updated 9 years ago
• MicroCountry / ddy-stock-real

  View on GitHub
  股票行情--来源恒生，新浪，腾讯，网易
  ☆32Oct 16, 2018Updated 7 years ago
• dazheng / SparkETL

  View on GitHub
  Implement a complete data warehouse etl using spark SQL
  ☆14Sep 8, 2022Updated 3 years ago
• wjhwangjinhui / wenshu

  View on GitHub
  ☆12Jan 5, 2019Updated 7 years ago
• free-iot / freeiot-ios

  View on GitHub
  FreeIOT is a open application to interact with multifarious IOT devices.
  ☆10Oct 22, 2015Updated 10 years ago
• a123demi / spring-sparkstreaming-kafka-10

  View on GitHub
  spring+spark streaming+kafka 10版本集成和异常问题处理
  ☆17Jul 21, 2017Updated 9 years ago
• FrankXiong / cqunews-web

  View on GitHub
  利用Java网络爬虫爬取重庆大学新闻网站数据，依据解析的数据构建的新闻网站
  ☆11Mar 7, 2016Updated 10 years ago
• Harhao / toutiao

  View on GitHub
  今日头条科技新闻接口爬虫
  ☆17Sep 26, 2017Updated 8 years ago
• andyzsf / PyTuShare

  View on GitHub
  TuShare是一个免费、开源的python财经数据接口包。主要实现对股票等金融数据从数据采集、清洗加工 到 数据存储的过程，能够为金融分析人员提供快速、整洁、和多样的便于分析的数据，为他们在数据获取方面极大地减轻工作量，使他们更加专注于策略和模型的研究与实现上。
  ☆39Mar 19, 2016Updated 10 years ago
• Virtual machines for every use case on DigitalOcean • Ad
  Get dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
• feizhiwu / play

  View on GitHub
  一些后台开发中常用的活动算法，大转盘，翻牌，刮刮卡，抢红包，洗牌 and so on ...
  ☆12Dec 27, 2019Updated 6 years ago
• BigDataScholar / User-Portrait

  View on GitHub
  大数据【企业级360°全方位用户画像】标签开发部分源码
  ☆20Dec 18, 2020Updated 5 years ago
• 344399160 / sparkForDB

  View on GitHub
  使用spark对hive、hbase、ES的读写， 实现一次配置可对不同数据库进行导入导出，并对ES、hbase进行封装
  ☆32May 6, 2017Updated 9 years ago
• jiaozhu / weixinSDK

  View on GitHub
  Java版微信机器人
  ☆14Oct 9, 2016Updated 9 years ago
• ex34toway / uw-log-es

  View on GitHub
  一个基于ElasticSearch的业务日志记录工具
  ☆10Nov 5, 2018Updated 7 years ago
• liushuishang / JobSchedule

  View on GitHub
  基于TBSchedule开发的一个分布式任务调度框架，可以解析任务间的依赖，并执行任务（执行Shell、bat脚本）
  ☆12Aug 5, 2016Updated 10 years ago
• vectorsss / news_classification

  View on GitHub
  卷积神经网络&&爬虫 实现网易新闻自动爬取并分类
  ☆13Dec 8, 2022Updated 3 years ago