Skip to main content

Robots.txt

Check Overview​

What is this check?​

The Robots.txt tells search engines what to crawl on your website, it's such an important file that any changes to it should be monitored.

Rel Canonical Check Basic Interface

Why is this important?​

Did you know that with two lines of code you can remove your entire website from Google? The robots.txt file is such an important file that often gets forgotten about.

Advanced Usage​

Default Rules​

The default rules for this check in Little Warden are to send you a notification every time a change happens with your robots.txt, we will tell you all the lines that have been added and all the lines removed.

Alert on text​

Rel Canonical Check Basic Interface

If you want to you can be more specific and look for the appearance (or disappearance) of words or URL's within the robots.txt, this will ensure you get less notifications whilst still looking out for the important issues.

But I want more!​

Sure thing! Send us a message on littlewarden.com with what you want to be able to configure and we will see what we can do!