Robots.txt & Sitemap Parser

See what a website hides from search engines and map its structure.