Difference between revisions of "OWASP Hatkit Proxy Project"

From OWASP
Jump to: navigation, search
Line 30: Line 30:
 
Before the proxy actually starts, you need to make some settings. These are all documented within the application, if you click the ?-button, you will see more information about the setting in question. '''Tip:''' Most of these settings can be modified later, so you don't have to restart the proxy to e.g. redefine the filters determining what is captured and what is ignored.  
 
Before the proxy actually starts, you need to make some settings. These are all documented within the application, if you click the ?-button, you will see more information about the setting in question. '''Tip:''' Most of these settings can be modified later, so you don't have to restart the proxy to e.g. redefine the filters determining what is captured and what is ignored.  
  
In order to actually store traffic, you also need to install mongodb. Please see [http://www.mongodb.org/downloads MongoDB] for suitable version for your platform. '''Note:''' MongoDB is usually also available through Linux packet managers, if you want to do it the simple way.  
+
In order to actually store traffic, you also need to install mongodb. Please see [http://www.mongodb.org/downloads MongoDB] for suitable version for your platform. '''Note:''' MongoDB is usually also available through Linux packet managers, if you want to do it the simple way:
 +
sudo apt-get install mongodb
 +
 
 +
=== Running the proxy ===
 +
[[Image:hatkit-proxy-stats.png|thumb|The stats-pane contains some basic counters to show what is happening]]
 +
[[Image:hatkit-proxy-interceptsettings.png|thumb|The intercept-pane is where you select to start intercepting data, control whether you want syntax highlightning and if you want to do it in FQ or NFQ mode]]
 +
[[Image:hatkit-proxy-settings.png|thumb|All settings where the 'Apply'-button is enabled can be modified on-the-fly]]
  
 
== Roadmap ==
 
== Roadmap ==

Revision as of 16:15, 2 April 2011

Main

The Hatkit Proxy is an intercepting http/tcp proxy which is based on the Owasp Proxy.

Background

The primary purpose of the Hatkit Proxy is to create a minimal, lightweight proxy which stores traffic into an offline storage where further analysis can be performed, i.e. all kinds of analysis which is currently implemented by the proxies themselves (webscarab/burp/paros etc).

Also, since the http traffic is stored in a MongoDB, the traffic is stored at an object-level, retaining the structure of the parsed traffic.

Hatkit proxy features

The additions which have been implemented on top of Owasp Proxy are:

  • Swing-based UI,
  • Interception capabilities with manual edit, both for TCP and HTTP traffic,
  • Syntax highlightning (html/form-data/http) based on JFlex,
  • Storage of http traffic into MongoDB database,
  • Possibilities to intercept in Fully Qualified mode (like all other http-proxies) OR Non-fully qualified mode. The latter means that interception is performed *after* the host has been parsed, thereby enabling the user to submit non-valid http content.
  • A set of filters to either ignore or process traffic which is routed to the proxy. The 'ignored' traffic will be streamed to the endpoint with minimal impact on performance.

Getting started

To use the proxy, download the zip-file which can be found on the source code repository (you can also use the direct-link on the release-page).

$ wget http://martin.swende.se/hgwebdir.cgi/hatkit_proxy/raw-file/tip/hatkit.zip
$ unzip hatkit.zip 
$ java -jar hatkit/hatkit-proxy.jar 

The proxy window should now pop up. It has one tab for HTTP-proxy mode, and another for TCP-proxy mode. Http setup options Tcp setup options

Before the proxy actually starts, you need to make some settings. These are all documented within the application, if you click the ?-button, you will see more information about the setting in question. Tip: Most of these settings can be modified later, so you don't have to restart the proxy to e.g. redefine the filters determining what is captured and what is ignored.

In order to actually store traffic, you also need to install mongodb. Please see MongoDB for suitable version for your platform. Note: MongoDB is usually also available through Linux packet managers, if you want to do it the simple way:

sudo apt-get install mongodb

Running the proxy

The stats-pane contains some basic counters to show what is happening
The intercept-pane is where you select to start intercepting data, control whether you want syntax highlightning and if you want to do it in FQ or NFQ mode
All settings where the 'Apply'-button is enabled can be modified on-the-fly

Roadmap

Todo

Storage

The Hatkit Proxy is a 'recorder' which (optionally) records http traffic into a MongoDB database. MongoDB is a document-oriented database, part of a group of databases also coined "NoSql" since they do not implement SQL.

NoSQL type datastorage is usually associated with massively parallel distributed systems with high requirements on scaleability. However, for Hatkit project, MongoDB was chosen for different reasons, since there are advantages to using it when storing data which fits the dynamic (schemaless) model. Having no schema enforced by the database does not imply that the database is just a disk-based hash table with unstructured data content. Instead, it can be argued that many NoSQL solutions are a lot like the (currently out-of-fashion) object databases, with the difference that they have more generic API's (json/bson/http) which does not bind the data to any particular framwork, application-specific classes or programming language. Certain kinds of data fit very well into these models.

Http traffic is very dynamic. Some requests are basically "GET / HTTP/1.1" while others contain forms or json and lots and a multitude of headers. Using MongoDB, it is possible to represent the data more at an object-level, e.g.

{ request:
    { method: "GET",
    headers:{ Content-Length: 1233, Host : "foobar.com", Foo: "bar"}
    parameters: {gaz: "onk"}
                    },
    response : {...}
}

Another reason, beside being very dynamic, why a non-relational database was chosen, is that http traffic was perceived as being pretty much non-relational. Each HTTP dialogue is stored as an object with no foreign keys or relation to any other database objects.

This object representation of a http dialogue allows for different requests/responses to contain different amounts of information. For example, it would be possible (but perhaps not desirable) to store the entire html response as a DOM model, which would allow database queries on html tags and attributes. MongoDB has very powerful querying-facilities. Since each object is stored with this structure in the database, it is possible to reach into objects during queries and perform e.g these kind of queries:

  • "give me response.body where request.parameters.filename exists", or
  • "give me request.body.parameters where
  • request.body.parameters.__viewstate does not exist"

It is also possible to create javascript selection filters which are evaluated within the database. Such functionality can e.g be used to perform evaluations using JavaScript to investigate characteristics on the response html source code.

Also, MongoDB has very powerful aggregation mechanisms, where queries like the following can be used:

  • "Organized by request.headers.host give me all unique parameter names.",
  • "Organized by request.url.path, give me all unique response header keys".

The functionality described above is implemented within the sister project Hatkit Datafiddler.

Storage example

This is an example of using the mongo console to check one HTTP dialogue requesting http://www.owasp.org (some binary fields where the unparsed data is stored have been shortened for brevity)

> use 2011-04-02
switched to db 2011-04-02
> db.conversations.findOne()
{
	"_id" : ObjectId("4d978cf9bc3e4e2a391f27db"),
	"request" : {
		"ssl" : false,
		"target" : "www.owasp.org/216.48.3.18:80",
		"time" : NumberLong( 1.30178e+12 ),
		"raw-header" : BinData(2,...),
		"raw-content" : null,
		"headers" : {
			"Host" : "www.owasp.org",
			"User-Agent" : "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.16) Gecko/20110323 Ubuntu/10.10 (maverick) Firefox/3.6.16",
			"Accept" : "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
			"Accept-Language" : "en-us,en;q=0.5",
			"Accept-Encoding" : "gzip,deflate",
			"Accept-Charset" : "ISO-8859-1,utf-8;q=0.7,*;q=0.7",
			"Keep-Alive" : "115",
			"Proxy-Connection" : "keep-alive",
			"Cookie" : "__utma=77342603.1836101885.1265119674.1287038519.1289393382.114; __utmz=77342603.1289393382.114.78.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=appsec%20research%202010%20polyglot; OAID=16f7cd90d05cc276281dde4e3df2d56c"
		},
		"rawdata" : BinData(2,"AAAAAA=="),
		"data" : {
			
		},
		"method" : "GET",
		"startline" : "GET / HTTP/1.1",
		"url" : {
			"raw" : "/",
			"path" : "/"
		}
	},
	"request-raw" : {
		"header" : BinData(2,...),
		"content" : null
	},
	"response" : {
		"rtt" : NumberLong( 1160 ),
		"raw" : BinData(2,"AAAAAA=="),
		"status" : "301",
		"reason" : "Moved Permanently",
		"version" : "HTTP/1.1",
		"startline" : "HTTP/1.1 301 Moved Permanently",
		"headers" : {
			"Date" : "Sat, 02 Apr 2011 20:54:18 GMT",
			"Server" : "Apache/2.2.17 (Fedora)",
			"X-Powered-By" : "PHP/5.3.5",
			"Vary" : "Accept-Encoding,Cookie",
			"Expires" : "Thu, 01 Jan 1970 00:00:00 GMT",
			"Cache-Control" : "private, must-revalidate, max-age=0",
			"Last-Modified" : "Sat, 02 Apr 2011 20:54:18 GMT",
			"Location" : "http://www.owasp.org/index.php/Main_Page",
			"Content-Encoding" : "gzip",
			"Content-Length" : "26",
			"Content-Type" : "text/html; charset=utf-8"
		}
	},
	"response-raw" : {
		"headers" : BinData(2,...),
		"content" : BinData(2,"GgAAAB+LCAAAAAAAAAMCAAAA//8DAAAAAAAAAAAA")
	}
 }

Project About

PROJECT INFO
What does this OWASP project offer you?
RELEASE(S) INFO
What releases are available for this project?
what is this project?
Name: OWASP Hatkit Proxy Project (home page)
Purpose:
  • The Hatkit Proxy is an intercepting http/tcp proxy based on the Owasp Proxy, but with several additions. These additions are:
    • Swing-based UI,
    • Interception capabilities with manual edit,
    • Syntax highlightning (html/form-data/http) based on JFlex,
    • Storage of http traffic into MongoDB database,
    • Interception capabilities of tcp-traffic,
    • Possibilities to intercept in Fully Qualified mode (like all other http-proxies) OR Non-fully qualified mode. The latter means that interception is performed *after* the host has been parsed, thereby enabling the user to submit non-valid http content.
  • The primary purpose of the Hatkit Proxy is to create a minimal, lightweight proxy which stores traffic into an offline storage where further analysis can be performed, e.g. all kinds of analysis which is currently implemented by the proxies themselves (webscarab/burp/paros etc).
  • Also, since the http traffic is stored in a MongoDB, the traffic is stored at an object-level, retaining the structure of the parsed traffic, which enables a user to perform advanced queries later.
  • The proxy should also be a good choice for 'defenders' who wants to (temporarily?) monitor traffic. The proxy itself is, as stated, very lightweight, and the backend MongoDB storage scales very well and should be able to handle extreme amounts of data. This would allow defenders to perform advanced post-mortem or real-time analysis of incoming traffic.
  • Built in Java/Swing + MongoDB.
License: GNU General Public License v3
who is working on this project?
Project Leader(s):
how can you learn more?
Project Pamphlet: Not Yet Created
Project Presentation:
Mailing list: Mailing List Archives
Project Roadmap: View
Main links:
Key Contacts
  • Contact the GPC to report a problem or concern about this project or to update information.
current release
Hatkit Proxy 0.5.1 - April 8 2011 - (download)
Release description:
  • This is the HatKit proxy, built upon the OWASP Proxy.
Rating: Yellow button.JPG Not Reviewed - Assessment Details
last reviewed release
Not Yet Reviewed


other releases