Adding robots.txt to SharePoint

Friend of mine asked me about this today and I had to look it up because its been a while since I have done anything with robots.txt. Turns out its not as easy as just adding the robots.txt file to the web server root. SharePoint likes to block access to the file, so you will need to add a managed path from Central Admin.

Here are the steps:

1. Open Central Admin and navigate to Application Management > Define Managed Paths
2. Select your specific web application from the drop down
3. Add a new path for /robots.txt
4. Switch the type to Explicit inclusion
5. Click OK
6. Run IISREST

This is all you need to do, but if you still get a permissions problem, right-click on robots.txt and give "Everyone" Read permissions to the file.