Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

This is a draft of a section of the new Testing Guide v3

Brief Summary
This section describes how to test the robots.txt file.

Description of the Issue
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the Robots Exclusion Protocol of the robots.txt file in the web root directory.

Web spiders/robots/crawlers can intentionally ignore the "Disallow:" statement[s] of the robots.txt file. Hence, robots.txt should not be considered to control access to web content not intended to be stored or published by external parties.

Black Box testing and example
The robots.txt file is retrieved by from the web root directory of the web server. For example, the URL "http://www.google.com/robots.txt" is the robots.txt file for www.google.com

Google provide an "Analyze robots.txt" function as part of its "Google Webmaster Tools" which can assist with testing.

Gray Box testing and example
The process is the same as Black Box testing above.