Adding robots.txt to SharePoint 2

Friend of mine asked me about this today and I had to look it up because its been a while since I have done anything with robots.txt. Turns out its not as easy as just adding the robots.txt file to the web server root. SharePoint likes to block access to the file, so you will need to add a managed path from Central Admin.

Here are the steps:

  1. Open Central Admin and navigate to Application Management > Define Managed Paths
  2. Select your specific web application from the drop down
  3. Add a new path for /robots.txt
  4. Switch the type to Explicit inclusion
  5. Click OK
  6. Run IISREST

This is all you need to do, but if you still get a permissions problem, right-click on robots.txt and give “Everyone” Read permissions to the file.

2 thoughts on “Adding robots.txt to SharePoint

  1. Reply CSU Web Analyst Sep 11,2009 2:49 pm

    I am having difficulties accomplishing this task. Did you use SharePoint Designer to place the file or just drop it on the file-system?

  2. Reply Rahul Soni Nov 22,2009 1:39 am

    I have checked this and it appears to be working fine even without adding the path to Managed path.

    It is strange that it requires IISReset to read any static file in the root. Thankfully, the updates are getting picked!!

    Regards,
    Rahul

Leave a Reply