A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker


[Up] [Top]

Documentation for package ‘robotstxt’ version 0.5.2

Help Pages

%>% re-export magrittr pipe operator
get_robotstxt downloading robots.txt file
get_robotstxts function to get multiple robotstxt files
get_robotstxt_http_get get_robotstxt() worker function to execute HTTP request
guess_domain function guessing domain from path
is_valid_robotstxt function that checks if file is valid / parsable robots.txt file
parse_robotstxt function parsing robots.txt
paths_allowed check if a bot has permissions to access page(s)
paths_allowed_worker_robotstxt paths_allowed_worker for robotstxt flavor
paths_allowed_worker_spiderbar paths_allowed_worker spiderbar flavor
print.robotstxt printing robotstxt
print.robotstxt_text printing robotstxt_text
remove_domain function to remove domain from path
robotstxt Generate a representations of a robots.txt file
rt_cache get_robotstxt() cache