Parse robots.txt rules, test URL against User-Agent. See Allow/Disallow for Googlebot. Essential for SEO crawling.
Related tools