Robots.txt checker and tester
Enter a website address to read its robots.txt. Add a path to test whether Googlebot may crawl it.
What this checks
Loads the robots.txt file, checks that it exists, that it does not block the whole site, and that it lists a sitemap.
How to fix common problems
- Never leave Disallow: / on a live site you want in search.
- Add a Sitemap line pointing to your sitemap.xml.
- Use robots.txt to block crawling, and a noindex tag to keep a page out of results.
Common questions
Does robots.txt remove a page from Google?
No. It only asks crawlers not to crawl. A blocked page can still be indexed if other sites link to it. Use a noindex tag instead.
Where must robots.txt be?
At the root of the site, for example https://example.com/robots.txt.