%>% | re-export magrittr pipe operator |
get_robotstxt | downloading robots.txt file |
get_robotstxts | function to get multiple robotstxt files |
get_robotstxt_http_get | get_robotstxt() worker function to execute HTTP request |
guess_domain | function guessing domain from path |
is_valid_robotstxt | function that checks if file is valid / parsable robots.txt file |
parse_robotstxt | function parsing robots.txt |
paths_allowed | check if a bot has permissions to access page(s) |
paths_allowed_worker_robotstxt | paths_allowed_worker for robotstxt flavor |
paths_allowed_worker_spiderbar | paths_allowed_worker spiderbar flavor |
print.robotstxt | printing robotstxt |
print.robotstxt_text | printing robotstxt_text |
remove_domain | function to remove domain from path |
robotstxt | Generate a representations of a robots.txt file |
rt_cache | get_robotstxt() cache |