zpoint / idataapi-transform
Full async support toolkit for IDataAPI for efficiency work, read data from API/ES/csv/xlsx/json/redis/mysql/mongo/kafka, write to ES/csv/xlsx/json/redis/mysql/mongo/kafka, provide CLI and python API
☆44Updated 3 weeks ago
Related projects ⓘ
Alternatives and complementary repositories for idataapi-transform
- fetchman is a simple crawler system/简单好用的爬虫框架☆76Updated 2 years ago
- 通过 airtest + mitmproxy 抓取手机端微信的公众号信息☆38Updated 5 years ago
- MitmProxy and Appium to Crawl Comments in JD APP☆31Updated 7 years ago
- A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.☆89Updated 2 years ago
- 基于httpx的一个大型项目 ，爬取黑胶唱片网站 Discogs☆101Updated last year
- Distributed crawling/scraping, Kafka And Redis based components for Scrapy☆46Updated 4 years ago
- scrapy-monitor，实现爬虫可视化，监控实时状态☆108Updated 7 years ago
- Ajax Hook Demo☆30Updated 4 years ago
- 在scrapyd基础上新增权限验证、爬虫运行信息统计、界面重构、 ，并增加排序、筛选过滤等多个API☆111Updated 6 years ago
- 方便的从浏览器复制浏览器头☆44Updated 4 years ago
- ☆23Updated 6 years ago
- Scrapy Universal Spider☆56Updated 7 years ago
- wrapper around aiomysql easy to use for sanic☆34Updated 3 years ago
- Scrapy Pyppeteer Demo☆23Updated 6 years ago
- Scrapy Redis with Bloom Filter，support redis sentinel and cluster☆23Updated last year
- Web-Scraping for Humans!☆142Updated 2 years ago
- 知乎登录☆22Updated 5 years ago
- Tinepeas，我们自己的爬虫框架。☆62Updated 3 months ago
- 🕷some website spider application base on proxy pool (support http & websocket)☆110Updated 2 years ago
- Crack Weibo Slide Captcha☆55Updated 6 years ago
- 通用新闻类网站分布式爬虫☆72Updated 6 years ago
- 爬取微信公众号评论、点赞等相关信息☆44Updated 6 years ago
- Dynamic configurable crawl (动态可配置化爬虫)☆87Updated 6 years ago
- openlaw数据爬虫v1.1 更新日期:2017.12.16 解决新版openlaw多种加密问题。引入celery轻松异步分布式，爬取速度再次翻倍！！☆58Updated 5 years ago
- 基于mongodb存储，redis缓存，celery 实现的分布式爬虫。☆12Updated last year
- Distributed task redisqueue(最简单python分布式函数调度框架)☆63Updated last year