Robots.txt Checker & Validator

Check, validate, and test robots.txt files for your website

Check Mode:

🔍 Fetch from URL📋 Paste Content

Enter any website URL to fetch and analyze its robots.txt file. The tool will automatically locate robots.txt at the root of the domain.

The tool automatically validates robots.txt syntax, identifies errors, and warns about potential issues in your file.

Test specific paths against robots.txt rules to see if they're allowed or disallowed for different user-agents.

Robots.txt files are located at the root of a domain (e.g., example.com/robots.txt)
Use wildcards (*) in paths to match multiple URLs
The order of rules matters - the first matching rule wins
Test paths to verify your robots.txt rules work as expected
Check for errors and warnings to ensure proper syntax
Use specific user-agents (like "Googlebot") to test different rule sets