Difference between revisions of "Category:OWASP Proxy"

From OWASP
Jump to: navigation, search
(Implementation details)
(Provide up to date examples)
Line 27: Line 27:
 
Other than that, there is no intention to add major new features to the library above those required to fulfill its purpose as a Listener and a HTTP client implementation.
 
Other than that, there is no intention to add major new features to the library above those required to fulfill its purpose as a Listener and a HTTP client implementation.
  
==Extending the OWASP Proxy==
+
==Using the OWASP Proxy==
  
The basic classes are Listener and SocksListener, which can be customised to add your own functionality. If you just create an instance of Listener (or SocksListener), it will accept connections on the specified port, and relay them to the requested destination.
+
===The Simplest Proxy===
  
    Listener l = new Listener(8008); // listens to localhost by default
+
About the simplest proxy that you can write is as follows:
    Listener sl = new SocksListener(1080);
+
  
or
+
RequestHandler requestHandler = new DefaultHttpRequestHandler();
 +
ConnectionHandler httpProxy = new HttpProxyConnectionHandler(requestHandler);
 +
InetSocketAddress listen = new InetSocketAddress("localhost", 8008);
 +
Server proxy = new Server(listen, httpProxy);
 +
proxy.start();
  
    Listener l = new Listener(InetAddress.getByAddress(new byte[] { 127, 0,
+
Of course, it is not terribly useful. All it does is forward requests and responses.
                    0, 1 }), 8008);
+
    Listener sl = new SocksListener(InetAddress.getByAddress(new byte[] { 127, 0,
+
                    0, 1 }), 1080);
+
  
The first thing a developer might want to do is make some changes to the request or response. This is done by means of a ProxyMonitor. This is an abstract class that can be extended to provide the specific functionality required. I'll show it here as an interface, just to reduce verbosity.
+
===The Message Object Model===
  
    Response requestReceived(Request request)
+
Let's take a look at the message object model, before we try to do something more complex.
        throws MessageFormatException;
+
   
+
    Response errorReadingRequest(Request request, Exception e)
+
        throws MessageFormatException;
+
   
+
    boolean responseHeaderReceived(Conversation conversation)
+
        throws MessageFormatException;
+
   
+
    void responseContentReceived(Conversation conversation, boolean streamed)
+
        throws MessageFormatException;
+
   
+
    Response errorFetchingResponseHeader(Request request, Exception e)
+
        throws MessageFormatException;
+
   
+
    Response errorFetchingResponseContent(Conversation conversation, Exception e)
+
        throws MessageFormatException;
+
   
+
    void wroteResponseToBrowser(Conversation conversation)
+
        throws MessageFormatException;
+
   
+
    void errorWritingResponseToBrowser(Conversation conversation,
+
        Exception e) throws MessageFormatException;
+
  
These methods are called at various stages of a request's lifecycle, if appropriate. Obviously error methods won't be called if there are no errors! Each method that returns a Response object can be customized to return a custom Response to the browser. e.g. overriding requestReceived(Request) can allow a developer to provide a browser-based interface to their program, by returning Response objects for specific requests, e.g. to a host called "proxy":
+
public interface MessageHeader {
 +
    byte[] getHeader();
 +
    String getStartLine() throws MessageFormatException;
 +
    NamedValue[] getHeaders() throws MessageFormatException;
 +
    String getHeader(String name) throws MessageFormatException;
 +
}
 +
 +
public interface MutableMessageHeader {
 +
    void setHeader(byte[] header);
 +
    void setStartLine(String line) throws MessageFormatException;
 +
    void setHeaders(NamedValue[] headers) throws MessageFormatException;
 +
    void setHeader(String name, String value) throws MessageFormatException;
 +
    void addHeader(String name, String value) throws MessageFormatException;
 +
    String deleteHeader(String name) throws MessageFormatException;
 +
}
  
    public Response requestReceived(Request request)
+
This shows the interface for a MessageHeader, and a mutable MessageHeader. These are the foundations for the other message classes. Everything is represented in a single byte[]. If you want to create a message header that uses a plain CR as a line separator, go ahead and construct a byte[] that has the lines separated by CR's, and call setHeader(). Of course, the convenience methods are configured to expect CRLF, and so if you call any of those methods, you should expect to receive a MessageFormatException, and be prepared to parse the header manually.
        throws MessageFormatException {
+
        if (request.getHost().equals("proxy")) {
+
            return handleRequest(request);
+
        }
+
        return super.requestReceived(request);
+
    }
+
  
The library provides a LoggingProxyMonitor class, which simply logs the Requests and Responses that pass through. It also prints out any exceptions that may occur (not shown).
+
===Message Content===
  
    class LoggingProxyMonitor extends ProxyMonitor {
+
public interface StreamingMessage extends MutableMessageHeader {
   
+
    InputStream getContent();
        @Override
+
    InputStream getDecodedContent() throws MessageFormatException;
        public void wroteResponseToBrowser(Conversation conversation)
+
    void setContent(InputStream content);
                throws MessageFormatException {
+
    void setDecodedContent(InputStream content) throws MessageFormatException;
            try {
+
}
                int resp = conversation.getResponse().getMessage().length;
+
                long time = conversation.getResponseBodyTime()
+
public interface BufferedMessage extends MessageHeader {
                        - conversation.getRequestTime();
+
    byte[] getContent();
   
+
    byte[] getDecodedContent() throws MessageFormatException;
                Request request = conversation.getRequest();
+
}
                StringBuilder buff = new StringBuilder();
+
                buff.append(request.getMethod()).append(" ");
+
public interface MutableBufferedMessage extends BufferedMessage, MutableMessageHeader {
                buff.append(request.isSsl() ? "ssl " : "");
+
    void setContent(byte[] content);
                buff.append(request.getHost()).append(":")
+
    void setDecodedContent(byte[] content) throws MessageFormatException;
                        .append(request.getPort());
+
}
                buff.append(request.getResource()).append(" ");
+
                buff.append(conversation.getResponse().getStatus()).append(" - ")
+
                        .append(resp);
+
                buff.append(" bytes in ").append(time).append("(").append(
+
                        resp / (time * 1000));
+
                buff.append(" bps)");
+
                System.out.println(buff.toString());
+
            } catch (MessageFormatException mfe) {
+
            }
+
        }
+
    }
+
  
One thing that a developer might want to do is teach the proxy how to use an upstream proxy to reach the target. OWASP Proxy uses the standard Java ProxySelector mechanism for this:
+
The above interfaces represent the content of an HTTP message, either in a streaming or buffered state. Streaming messages are useful if you only really want to look at the message header, and not do anything with the message body, or if you can process the message body in a streaming fashion.  
  
    final Proxy upstream = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(host, port);
+
For example, you may want to compress a message transferred without gzip encoding. Update the message header to reflect the new encoding, wrap the content stream with a suitable GzipInputStream, and pass the message on to the next layer.
   
+
    final ProxySelector ps = new ProxySelector() {
+
   
+
        @Override
+
        public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
+
            System.err.println("Proxy connection failed! "
+
                    + ioe.getLocalizedMessage());
+
        }
+
   
+
        @Override
+
        public List<Proxy> select(URI uri) {
+
            return Arrays.asList(upstream);
+
        }
+
    };
+
  
Having created the ProxySelector, we need to make use of it. OWASP Proxy has its own Http Client implementation, that specifically makes NO unnecessary changes to the requests or responses. Some libraries try to make sure that the requests it sends are well-formed, for instance, but for a security tool, sometimes it is interesting to send malformed requests.
+
Of course, if you want to do something complex with the message body, you probably want to work with the buffered content. In that case, the BufferedMessage and MutableBufferedMessage interfaces are appropriate.
  
So, we can tell the Listener how to configure its HttpClient implementations, by providing a customised HttpClientFactory:
+
Note: There is a distinction between BufferedMessage and MutableBufferedMessage mainly as documentation indicating whether they should be modified or not in a particular method. See BufferedMessageInterceptor, for example.
  
    HttpClientFactory hcf = new DefaultHttpClientFactory() {
+
===Requests and Responses===
   
+
        @Override
+
        public HttpClient createHttpClient() {
+
            HttpClient client = super.createHttpClient();
+
            client.setProxySelector(ps);
+
            return client;
+
        }
+
    };
+
    listener.setHttpClientFactory(hcf);
+
  
By default, OWASP Proxy does not intercept SSL connections (HTTP CONNECT verb), and if it encounters an SSL encrypted channel, it simply refuses to continue. Developers can enable SSL interception by providing a CertificateProvider implementation.
+
This is what a Request header looks like. Again, there are convenience methods to obtain specific portions of the request, but underneath it all is that byte[] containing the entire header.
  
This is what the interface looks like:
+
public interface RequestHeader extends MessageHeader {
 +
    InetSocketAddress getTarget();
 +
    boolean isSsl();
 +
    String getMethod() throws MessageFormatException;
 +
    String getResource() throws MessageFormatException;
 +
    String getVersion() throws MessageFormatException;
 +
}
 +
 +
public interface MutableRequestHeader extends RequestHeader, MutableMessageHeader {
 +
    void setTarget(InetSocketAddress target);
 +
    void setSsl(boolean ssl);
 +
    void setMethod(String method) throws MessageFormatException;
 +
    void setResource(String resource) throws MessageFormatException;
 +
    void setVersion(String version) throws MessageFormatException;
 +
}
  
    public interface CertificateProvider {
+
Note that the target server and whether the message should be encrypted or not is external to the message header itself. In most cases, where no upstream proxy is involved, sending the request is as simple as opening a socket to the target InetSocketAddress, and calling write(message.getHeader()); Again, the minimum of parsing is performed, to allow for sending non-RFC compliant messages to a server.
   
+
        SSLSocketFactory getSocketFactory(String host, int port) throws IOException;
+
   
+
    }
+
  
A simple implementation is provided in DefaultCertificateProvider that makes use of a single certificate for all connection attempts. A certificate location and passwords can be provided; alternatively, OWASP Proxy's default certificate can be used.
+
There are similar interfaces for Responses, although they do not have an associated target.
  
    listener.setCertificateProvider(new DefaultCertificateProvider());
+
==Intercepting HTTP Messages==
  
Developers may wish to make use of something like the CyberVillains CA library to automatically generate certificates on the fly for each site visited.
+
OWASP Proxy provides a BufferingHttpRequestHandler class which interacts with implementations of the BufferedMessageInterceptor interface to facilitate manipulation of the request and response.
 +
 
 +
This is what the BufferedMessageInterceptor interface looks like:
 +
 
 +
public interface BufferedMessageInterceptor {
 +
 +
    enum Action { BUFFER, STREAM, IGNORE};
 +
 +
    Action directRequest(MutableRequestHeader request);
 +
    void processRequest(MutableBufferedRequest request);
 +
    void requestContentSizeExceeded(BufferedRequest request, int size);
 +
    void requestStreamed(BufferedRequest request);
 +
 +
    Action directResponse(RequestHeader request, MutableResponseHeader response)
 +
    void processResponse(RequestHeader request, MutableBufferedResponse response)
 +
    void responseContentSizeExceeded(RequestHeader request, ResponseHeader response, int size);
 +
    void responseStreamed(final RequestHeader request, BufferedResponse response);
 +
 +
}
 +
 
 +
Note: BufferedMessageInterceptor is actually an abstract class, to save implementation of methods that you have no interest in.
 +
 
 +
The first thing to do is decide which requests and responses your implementation is interested in. The "directRequest()" method is called first, with the RequestHeader as a parameter. Examine the request header to determine if the request is "interesting" or not. If you want the request content to be buffered, return Action.BUFFER. If you want the request content to be streamed to the server, return Action.STREAM. If you are not interested in any part of the request, you can return Action.IGNORE, and no further methods will be called for that particular Request/Response.
 +
 
 +
Note that the RequestHeader is actually Mutable, so if you are only interested in the header, you can make any changes you like in this method, and then return either Action.STREAM or Action.IGNORE, and forget about it.
 +
 
 +
The methods that will be invoked next depend on the Action that was returned.
 +
 
 +
If the Action was BUFFER, the processRequest(MutableBufferedRequest) method will be called, with the buffered request as a parameter. You can then modify it to suit, and when you return from this method, the buffered request will be sent to the server.
 +
 
 +
If the action was STREAM, the requestStreamed(BufferedRequest) method will be called. Note that this request is no longer mutable, as it is only invoked AFTER the entire request body has been streamed to the server.
 +
 
 +
Note: BufferingHttpRequestHandler takes a "max content size" parameter, to avoid buffering excessively large messages, and potentially running out of memory. If the limit is reached, the requestContentSizeExceeded(BufferedRequest, size) method is invoked, with the BufferedRequest containing the bytes buffered up to the size limit, and the size parameter containing the ultimate size of the message.
 +
 
 +
The same process is followed for the Response cycle. Determine if you are interested in the response, return a suitable Action, and handle the response in the appropriate methods thereafter.
 +
 
 +
==Putting an intercepting proxy together==
 +
 
 +
HttpRequestHandler requestHandler = new DefaultHttpRequestHandler();
 +
BufferedMessageInterceptor interceptor = new BufferedMessageInterceptor() {
 +
    public Action directResponse(RequestHeader request, MutableResponseHeader response) {
 +
        return Action.BUFFER;
 +
    }
 +
 +
    public void processResponse(RequestHeader request, MutableBufferedResponse response) {
 +
        try {
 +
            System.out.println(request.getResource() + " : “ + response.getDecodedContent().length);
 +
        } catch (MessageFormatException mfe) {
 +
            mfe.printStackTrace();
 +
        }
 +
    }
 +
};
 +
int maxContentSize = 10240;
 +
requestHandler = new BufferingHttpRequestHandler(requestHandler, interceptor, maxContentSize);
 +
ConnectionHandler httpProxy = new HttpProxyConnectionHandler(requestHandler);
 +
InetSocketAddress listen = new InetSocketAddress("localhost", 8008);
 +
Server proxy = new Server(listen, httpProxy);
 +
proxy.start();
  
 
==Project Contributors==
 
==Project Contributors==

Revision as of 14:38, 18 March 2010


PROJECT INFORMATION
Project Name OWASP Proxy Project
Short Project Description

The OWASP Proxy aims to provide a high quality intercepting proxy library which can be used by developers who require this functionality in their own programs, rather than having to develop it all from scratch.

The library is developed in Java, making it most attractive to Java developers obviously, but also accessible to Python (Jython) and Ruby (JRuby) developers as well.

Key Project Information

Project Leader
Rogan Dawes

Project Contibutors
Add name

Mailing List
Subscribe here
Use here

License
Creative Commons Attribution Share Alike 3.0

Project Type
Tool

Sponsors
if any, add link

Release Status Main Links Related Projects

Alpha Quality
(under review)
Please see here for complete information.

At the moment there is no packaged version of this library. Development is done in a git repository, located here.
An up to date snapshot can be obtained here.


Contents

Overview

One of the priorities of this project is to allow developers to do whatever they choose, without enforcing RFC compliance. This is important for a security testing library, as often the most interesting behavior manifests outside the RFCs! Keep in mind that a lot of the safety nets that exist in libraries that enforce RFC compliance do not exist in this library, and that as the developer, you need to be prepared to deal with the consequences!

Another priority is to accurately deliver whatever is specified by the client, and similarly, to accurately reflect whatever is returned by the server, rather than coloured by the parsing and normalisation performed by the library.

Download

At the moment there is no packaged version of this library. Development is done in a git repository, located here.

Interested parties can download a snapshot of the code at any point using the snapshot link next to each revision, or clone the repository:

 $ git clone http://dawes.za.net/rogan/owasp-proxy/owasp-proxy.git/

Implementation details

In order to achieve byte for byte accuracy with what was sent by the client, and received from the server, OWASP Proxy does the bare minimum of message parsing. The basic storage of an HTTP message header is as an array of byte (a byte for byte copy of what was read from the network), rather than parsed out into convenient pieces. The library does provide convenience methods for accessing interesting parts of the message, such as headers, content, etc, but the message itself is represented as either a byte[] for the header, and an InputStream for the content, or a byte[] for the header, and a (possibly null) byte[] for the message content.

The Request and Response objects that you may deal with also do not decode the message bodies for you. If the message was sent using chunked encoding, the message body will show the individual chunks that were sent. Of course, again, there are also classes which allow you to obtain the actual entity body, with appropriate decoding performed.

Future development

As mentioned above, one objective is correctness. By this I mean correctly handling whatever the major browsers send to it, and successfully retrieving whichever resource was requested. Failure to do so will be addressed as soon as possible.

Other than that, there is no intention to add major new features to the library above those required to fulfill its purpose as a Listener and a HTTP client implementation.

Using the OWASP Proxy

The Simplest Proxy

About the simplest proxy that you can write is as follows:

RequestHandler requestHandler = new DefaultHttpRequestHandler();
ConnectionHandler httpProxy = new HttpProxyConnectionHandler(requestHandler);
InetSocketAddress listen = new InetSocketAddress("localhost", 8008);
Server proxy = new Server(listen, httpProxy);
proxy.start();

Of course, it is not terribly useful. All it does is forward requests and responses.

The Message Object Model

Let's take a look at the message object model, before we try to do something more complex.

public interface MessageHeader {
    byte[] getHeader();
    String getStartLine() throws MessageFormatException;
    NamedValue[] getHeaders() throws MessageFormatException;
    String getHeader(String name) throws MessageFormatException;
}

public interface MutableMessageHeader {
    void setHeader(byte[] header);
    void setStartLine(String line) throws MessageFormatException;
    void setHeaders(NamedValue[] headers) throws MessageFormatException;
    void setHeader(String name, String value) throws MessageFormatException;
    void addHeader(String name, String value) throws MessageFormatException;
    String deleteHeader(String name) throws MessageFormatException;
}

This shows the interface for a MessageHeader, and a mutable MessageHeader. These are the foundations for the other message classes. Everything is represented in a single byte[]. If you want to create a message header that uses a plain CR as a line separator, go ahead and construct a byte[] that has the lines separated by CR's, and call setHeader(). Of course, the convenience methods are configured to expect CRLF, and so if you call any of those methods, you should expect to receive a MessageFormatException, and be prepared to parse the header manually.

Message Content

public interface StreamingMessage extends MutableMessageHeader {
    InputStream getContent();
    InputStream getDecodedContent() throws MessageFormatException;
    void setContent(InputStream content);
    void setDecodedContent(InputStream content) throws MessageFormatException;
}

public interface BufferedMessage extends MessageHeader {
    byte[] getContent();
    byte[] getDecodedContent() throws MessageFormatException;
}

public interface MutableBufferedMessage extends BufferedMessage, MutableMessageHeader {
    void setContent(byte[] content);
    void setDecodedContent(byte[] content) throws MessageFormatException;
}

The above interfaces represent the content of an HTTP message, either in a streaming or buffered state. Streaming messages are useful if you only really want to look at the message header, and not do anything with the message body, or if you can process the message body in a streaming fashion.

For example, you may want to compress a message transferred without gzip encoding. Update the message header to reflect the new encoding, wrap the content stream with a suitable GzipInputStream, and pass the message on to the next layer.

Of course, if you want to do something complex with the message body, you probably want to work with the buffered content. In that case, the BufferedMessage and MutableBufferedMessage interfaces are appropriate.

Note: There is a distinction between BufferedMessage and MutableBufferedMessage mainly as documentation indicating whether they should be modified or not in a particular method. See BufferedMessageInterceptor, for example.

Requests and Responses

This is what a Request header looks like. Again, there are convenience methods to obtain specific portions of the request, but underneath it all is that byte[] containing the entire header.

public interface RequestHeader extends MessageHeader {
    InetSocketAddress getTarget();
    boolean isSsl();
    String getMethod() throws MessageFormatException;
    String getResource() throws MessageFormatException;
    String getVersion() throws MessageFormatException;
}

public interface MutableRequestHeader extends RequestHeader, MutableMessageHeader {
    void setTarget(InetSocketAddress target);
    void setSsl(boolean ssl);
    void setMethod(String method) throws MessageFormatException;
    void setResource(String resource) throws MessageFormatException;
    void setVersion(String version) throws MessageFormatException;
}

Note that the target server and whether the message should be encrypted or not is external to the message header itself. In most cases, where no upstream proxy is involved, sending the request is as simple as opening a socket to the target InetSocketAddress, and calling write(message.getHeader()); Again, the minimum of parsing is performed, to allow for sending non-RFC compliant messages to a server.

There are similar interfaces for Responses, although they do not have an associated target.

Intercepting HTTP Messages

OWASP Proxy provides a BufferingHttpRequestHandler class which interacts with implementations of the BufferedMessageInterceptor interface to facilitate manipulation of the request and response.

This is what the BufferedMessageInterceptor interface looks like:

public interface BufferedMessageInterceptor {

    enum Action { BUFFER, STREAM, IGNORE};

    Action directRequest(MutableRequestHeader request);
    void processRequest(MutableBufferedRequest request);
    void requestContentSizeExceeded(BufferedRequest request, int size);
    void requestStreamed(BufferedRequest request);

    Action directResponse(RequestHeader request, MutableResponseHeader response)
    void processResponse(RequestHeader request, MutableBufferedResponse response)
    void responseContentSizeExceeded(RequestHeader request, ResponseHeader response, int size);
    void responseStreamed(final RequestHeader request, BufferedResponse response);

}

Note: BufferedMessageInterceptor is actually an abstract class, to save implementation of methods that you have no interest in.

The first thing to do is decide which requests and responses your implementation is interested in. The "directRequest()" method is called first, with the RequestHeader as a parameter. Examine the request header to determine if the request is "interesting" or not. If you want the request content to be buffered, return Action.BUFFER. If you want the request content to be streamed to the server, return Action.STREAM. If you are not interested in any part of the request, you can return Action.IGNORE, and no further methods will be called for that particular Request/Response.

Note that the RequestHeader is actually Mutable, so if you are only interested in the header, you can make any changes you like in this method, and then return either Action.STREAM or Action.IGNORE, and forget about it.

The methods that will be invoked next depend on the Action that was returned.

If the Action was BUFFER, the processRequest(MutableBufferedRequest) method will be called, with the buffered request as a parameter. You can then modify it to suit, and when you return from this method, the buffered request will be sent to the server.

If the action was STREAM, the requestStreamed(BufferedRequest) method will be called. Note that this request is no longer mutable, as it is only invoked AFTER the entire request body has been streamed to the server.

Note: BufferingHttpRequestHandler takes a "max content size" parameter, to avoid buffering excessively large messages, and potentially running out of memory. If the limit is reached, the requestContentSizeExceeded(BufferedRequest, size) method is invoked, with the BufferedRequest containing the bytes buffered up to the size limit, and the size parameter containing the ultimate size of the message.

The same process is followed for the Response cycle. Determine if you are interested in the response, return a suitable Action, and handle the response in the appropriate methods thereafter.

Putting an intercepting proxy together

HttpRequestHandler requestHandler = new DefaultHttpRequestHandler();
BufferedMessageInterceptor interceptor = new BufferedMessageInterceptor() {
    public Action directResponse(RequestHeader request, MutableResponseHeader response) {
        return Action.BUFFER;
    }

    public void processResponse(RequestHeader request, MutableBufferedResponse response) {
        try {
            System.out.println(request.getResource() + " : “ + response.getDecodedContent().length);
        } catch (MessageFormatException mfe) {
            mfe.printStackTrace();
        }
    }
};
int maxContentSize = 10240;
requestHandler = new BufferingHttpRequestHandler(requestHandler, interceptor, maxContentSize);
ConnectionHandler httpProxy = new HttpProxyConnectionHandler(requestHandler);
InetSocketAddress listen = new InetSocketAddress("localhost", 8008);
Server proxy = new Server(listen, httpProxy);
proxy.start();

Project Contributors

The OWASP Proxy project is run by Rogan Dawes of Corsaire Security. He can be contacted at rogan AT dawes.za.net

This category currently contains no pages or media.