Testing: Review Webserver Metafiles for Information Leakage (OTG-INFO-003)
This article is part of the OWASP Testing Guide v4 (the current status is:DRAFT).
OWASP Testing Guide v4 Table of Contents [DRAFT] At the moment the The entire OWASP Testing Guide v3 can be downloaded here.
This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009
1. Information Leakage of the web application's directory/folder path(s).
2. Create the list of directories that are to be avoided by Spiders/Robots/Crawlers.
How to Test
Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the Robots Exclusion Protocol of the robots.txt file in the web root directory .
robots.txt in webroot
As an example, the beginning of the robots.txt file from http://www.google.com/robots.txt sampled on 11 August 2013 is quoted below:
User-agent: * Disallow: /search Disallow: /sdch Disallow: /groups Disallow: /images Disallow: /catalogs ...
The User-Agent directive refers to the specific web spider/robot/crawler. For example the User-Agent: Googlebot refers to the spider from Google while "User-Agent: bingbot" refers to crawler from Microsoft/Yahoo!. User-Agent: * in the example above applies to all web spiders/robots/crawlers  as quoted below:
The Disallow directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:
... Disallow: /search Disallow: /sdch Disallow: /groups Disallow: /images Disallow: /catalogs ...
Web spiders/robots/crawlers can intentionally ignore the Disallow directives specified in a robots.txt file , such as those from Social Networks to ensure that shared linked are still valid. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.
<META> tags are located within the HEAD section of each HTML Document and should be consistent across a web site in the likely event that the robot/spider/crawler start point does not begin from a document link other than webroot i.e. a "deep link".
If there is no "<META NAME="ROBOTS"" ... >" entry than the "Robots Exclusion Protocol defaults to "INDEX,FOLLOW" respectively. Therefore, the other two valid entries defined by the "Robots Exclusion Protocol are prefixed with "NO..." i.e. "NOINDEX" and "NOFOLLOW".
Web spiders/robots/crawlers can intentionally ignore the "<META NAME="ROBOTS"" tag as the robots.txt file convention is preferred. Hence, <META> Tags should not be considered the primary mechanism, rather a complementary control to robots.txt.
Black Box testing and example
robots.txt in webroot - with "wget" or "curl"
The robots.txt file is retrieved from the web root directory of the web server.
For example, to retrieve the robots.txt from www.google.com using wget or "curl":
cmlh$ wget http://www.google.com/robots.txt --2013-08-11 14:40:36-- http://www.google.com/robots.txt Resolving www.google.com... 126.96.36.199, 188.8.131.52, 184.108.40.206, ... Connecting to www.google.com|220.127.116.11|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: ‘robots.txt.1’ [ <=> ] 7,074 --.-K/s in 0s 2013-08-11 14:40:37 (59.7 MB/s) - ‘robots.txt’ saved  cmlh$ head -n5 robots.txt User-agent: * Disallow: /search Disallow: /sdch Disallow: /groups Disallow: /images cmlh$
cmlh$ curl -O http://www.google.com/robots.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 101 7074 0 7074 0 0 9410 0 --:--:-- --:--:-- --:--:-- 27312 cmlh$ head -n5 robots.txt User-agent: * Disallow: /search Disallow: /sdch Disallow: /groups Disallow: /images cmlh$
robots.txt in webroot - with rockspider
"rockspider" automates the creation of the initial scope for Spiders/Robots/Crawlers of files and directories/folders of a web site
For example, to create the initial scope based on the Allowed: directive from www.google.com using "rockspider":
cmlh$ ./rockspider.pl -www www.google.com "Rockspider" Alpha v0.1_2 Copyright 2013 Christian Heinrich Licensed under the Apache License, Version 2.0 1. Downloading http://www.google.com/robots.txt 2. "robots.txt" saved as "www.google.com-robots.txt" 3. Sending Allow: URIs of www.google.com to web proxy i.e. 127.0.0.1:8080 /catalogs/about sent /catalogs/p? sent /news/directory sent ... 4. Done. cmlh$
<META> Tags - with Burp
Based on the Disallow directive(s) listed within the robots.txt file in webroot, a regular expression search for "<META NAME="ROBOTS"" within each web page is undertaken and the result compared to the robots.txt file in webroot.
For example, the robots.txt file from facebook.com has a "Disallow: /ac.php" entry and the resulting search for "<META NAME="ROBOTS"" shown below:
The above might be considered a fail since "INDEX,FOLLOW" is the default <META> Tag specified by the "Robots Exclusion Protocol" yet "Disallow: /ac.php" is listed in robots.txt.
Analyze robots.txt using Google Webmaster Tools
If you are the owner of the website you want to analyze, Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools" (https://www.google.com/webmasters/tools), which can assist you with testing  and the procedure is as follows:
1. Sign into Google Webmaster Tools with your Google Account.
2. On the Dashboard, write the URL for the site you want to analyze.
3. Choose between the available methods and follow the on screen instruction.
Gray Box testing and example
The process is the same as Black Box testing above.
- Browser (View Source function)
-  "The Web Robots Pages" - http://www.robotstxt.org/
-  "Block and Remove Pages Using a robots.txt File" - https://support.google.com/webmasters/answer/156449
-  "(ISC)2 Blog: The Attack of the Spiders from the Clouds" - http://blog.isc2.org/isc2_blog/2008/07/the-attack-of-t.html
-  "Telstra customer database exposed" - http://www.smh.com.au/it-pro/security-it/telstra-customer-database-exposed-20111209-1on60.html