Robots.txt Tester
Live fetchFetch a site's robots.txt, lint it for errors, and test whether Googlebot, Bingbot and AI crawlers (GPTBot, ClaudeBot, PerplexityBot…) may crawl a URL. Edit and re-test before you deploy.
- Live data, fetched now
- Results in seconds
- Shareable result link
- Read-only, safe check
Who is allowed to crawl this URL?
Enter any page URL. We fetch the site's robots.txt, show exactly which rule decides access for Googlebot, Bingbot and 8 AI crawlers, and let you edit the file and re-test before you deploy.
Try an example
What you get
- Allowed / blocked verdict for 17 bots, with the deciding line
- AI visibility: which AI assistants can read your pages
- Syntax linting (unknown directives, bad sitemap URLs, 500 KiB limit)
- Edit-and-test sandbox — nothing changes on your site
Need it fixed for you?