What is the robots.txt file
The robots.txt file is used to instruct search engines which files they're allowed to access, the main purpose of this is to prevent a crawler from overloading a website.
The problem with the robots.txt file is many people use it as a method of security, they feel that if it cannot be found on a search engine then people cannot find it. However, as the file can be accessed through any web browser it can potentially lead to sensitive files or locations. See below for a sample robots.txt file.
Notice something not quite right? That's fine we're not perfect. Why not make a suggestion on our community forum, you should be able to see the correct formatting below.
Still need help?
Are you having trouble using arctil? Why not try reaching out to our Community Forum.
Alternatively, you can try contacting us through the Contact page.