Robots.txt Checker & Validator
Check, validate, and test robots.txt files for your website
How to Use
π Check Robots.txt
Enter any website URL to fetch and analyze its robots.txt file. The tool will automatically locate robots.txt at the root of the domain.
β Validate Syntax
The tool automatically validates robots.txt syntax, identifies errors, and warns about potential issues in your file.
π§ͺ Test Paths
Test specific paths against robots.txt rules to see if they're allowed or disallowed for different user-agents.
π‘ Tips:
- Robots.txt files are located at the root of a domain (e.g., example.com/robots.txt)
- Use wildcards (*) in paths to match multiple URLs
- The order of rules matters - the first matching rule wins
- Test paths to verify your robots.txt rules work as expected
- Check for errors and warnings to ensure proper syntax
- Use specific user-agents (like "Googlebot") to test different rule sets