Review Webserver Metafiles for Information Leakage (OTG-INFO-003)

Revision as of 23:27, 10 August 2013 by Cmlh (talk | contribs) (1st DRAFT for v4, might need to update OWASP-IG-009 reference in FINAL)

Jump to: navigation, search
This article is part of the new OWASP Testing Guide v4.
Back to the OWASP Testing Guide v4 ToC: Back to the OWASP Testing Guide Project:


This section describes how to test the robots.txt file for Information Leakage of the web application's directory/folder path(s). Furthermore the list of directories that are to be avoided by Spiders/Robots/Crawlers can also be created as a dependency for OWASP-IG-009[1]

Test Objectives

How to Test

Web spiders/robots/crawlers retrieve a web page and then recursively traverse hyperlinks to retrieve further web content. Their accepted behavior is specified by the Robots Exclusion Protocol of the robots.txt file in the web root directory [1].

As an example, the robots.txt file from taken on 24 August 2008 is quoted below:

User-agent: *
Allow: /searchhistory/
Disallow: /news?output=xhtml&
Allow: /news?output=xhtml
Disallow: /search
Disallow: /groups
Disallow: /images

The User-Agent directive refers to the specific web spider/robot/crawler. For example the User-Agent: Googlebot refers to the GoogleBot crawler while User-Agent: * in the example above applies to all web spiders/robots/crawlers [2] as quoted below:

User-agent: *

The Disallow directive specifies which resources are prohibited by spiders/robots/crawlers. In the example above, directories such as the following are prohibited:

Disallow: /search
Disallow: /groups
Disallow: /images

Web spiders/robots/crawlers can intentionally ignore the Disallow directives specified in a robots.txt file [3]. Hence, robots.txt should not be considered as a mechanism to enforce restrictions on how web content is accessed, stored, or republished by third parties.

Black Box testing and example

The robots.txt file is retrieved from the web root directory of the web server.

For example, to retrieve the robots.txt from using wget:

$ wget
           => 'robots.txt'
Resolving,,, ...
Connecting to||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]

    [ <=>                                 ] 3,425        --.--K/s

23:59:26 (13.67MB/s) - 'robots.txt' saved [3425]

Analyze robots.txt using Google Webmaster Tools
Google provides an "Analyze robots.txt" function as part of its "Google Webmaster Tools", which can assist with testing [4] and the procedure is as follows:

1. Sign into Google Webmaster Tools with your Google Account.
2. On the Dashboard, click the URL for the site you want.
3. Click Tools, and then click Analyze robots.txt.

Gray Box testing and example

The process is the same as Black Box testing above.


  • Browser (View Source function)
  • curl
  • wget