Welcome

Use this tool to check if robots are being blocked from crawling your website.

Enter in your website address above. Optionally, you can specify a User Agent to test against as well.

Robots.txt

Robots.txt files assist you in controlling what pages/files on your website search engine (and other) robots are allowed to see by defining a set of rules for robots to follow. This tool will check your robots.txt file to see what is being blocked.

Learn More About Robots.txt

Robots Meta Tag

Robots meta tags are used to define page-level rule sets for robots to follow. These are less commonly used, but are still very useful for situations when using the traditional robots.txt file isn't possible. This tool will also check for robots meta tags.

Learn More About Meta Tags