rivermont / spidy
The simple, easy to use command line web crawler.
☆340Updated 3 months ago
Related projects ⓘ
Alternatives and complementary repositories for spidy
- CoCrawler is a versatile web crawler built using modern tools and concurrency.☆187Updated 2 years ago
- Official python API for Phish.AI public and private API to detect zero-day phishing websites☆170Updated 6 years ago
- next generation web crawling using machine intelligence☆329Updated last year
- Hunt domain names using DNSDumpster, WHOIS, Reverse WHOIS, Shodan, Crimeflare☆295Updated 5 years ago
- A project to attempt to automatically login to a website given a single seed☆123Updated 2 years ago
- Hamburglar -- collect useful information from urls, directories, and files☆315Updated last year
- 🕷 A lightning fast multithreaded network scanner framework with modules.☆320Updated 5 years ago
- Extracting URLs of a specific target based on the results of "commoncrawl.org"☆271Updated last year
- a class that uses scraped proxies to make http GET/POST requests (Python requests)☆388Updated 3 years ago
- An Extensible Image Crawler☆158Updated 7 years ago
- A security tool for grabbing screenshots of many web hosts☆312Updated 7 years ago
- Formasaurus tells you the type of an HTML form and its fields using machine learning☆116Updated 4 months ago
- raven is a Linkedin information gathering tool that can be used by pentesters to gather information about an organization employees usin…☆777Updated 4 years ago
- Simple Python script for fetching "some" (usable) proxies☆576Updated last year
- The Zipru scraper developed in the Advanced Web Scraping Tutorial.☆426Updated 7 years ago
- Web application/technology detection tool☆205Updated last year
- Correlate data between domains, IPs and email addresses, present it as a graph and store everything into Elasticsearch and JSON files.☆674Updated 4 years ago
- detectem - detect software and its version on websites.☆153Updated 3 years ago
- Python Pastebin Webcrawler that returns list of public pastebins containing keywords☆427Updated 3 years ago
- Google Search Scraper☆562Updated 5 years ago
- An OSINT tool that allows you to draw out relationships between people on LinkedIn via endorsements/skills.☆327Updated 3 years ago
- The Hacker's ToolBox☆181Updated 7 years ago
- Phishing on Twitter☆251Updated 6 years ago
- A framework for creating semi-automatic web content extractors☆498Updated last week
- Find emails of Github users☆554Updated 5 years ago
- Scrape the Google search result with Scrapy.☆98Updated 4 years ago
- Finds public elite anonymity proxies and concurrently tests them☆250Updated 7 years ago
- a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine☆96Updated 7 months ago