arbuckle / python_crawler

Python-driven web crawler and scraper. Uses BeautifulSoup to gather all URLs from a target page, and initiates a crawl from a start URL, considering Whitelist/Blacklist criteria that are populated in crawl.py
20Updated 13 years ago

Alternatives and similar repositories for python_crawler:

Users that are interested in python_crawler are comparing it to the libraries listed below