What is the robots.txt file

The robots.txt file is used to instruct search engines which files they're allowed to access, the main purpose of this is to prevent a crawler from overloading a website.

The problem with the robots.txt file is many people use it as a method of security, they feel that if it cannot be found on a search engine then people cannot find it. However, as the file can be accessed through any web browser it can potentially lead to sensitive files or locations. See below for a sample robots.txt file.

User-agent: Googlebot
Disallow: /secret_uploader.php
Disallow: /admin/

User-agent: *
Allow: /

Sitemap: http://www.example.com/sitemap.xml
Make a suggestion +

Notice something not quite right? That's fine we're not perfect. Why not make a suggestion on our community forum, you should be able to see the correct formatting below.

Page Title:
Make suggestion

Still need help?

Are you having trouble using arctil? Why not try reaching out to our Community Forum.

Alternatively, you can try contacting us through the Contact page.