Showing 1 of total 1 results (show query)
ropensci
robotstxt:A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Maintained by Jordan Bradford. Last updated 5 months ago.
crawlerpeer-reviewedrobotstxtscraperspiderwebscraping
68 stars 10.43 score 414 scripts 7 dependents