Commit Graph

248 Commits

Author SHA1 Message Date
Michael Frister
e67dbf6123 Handle Transfer-Encoding header values case insensitive
According to HTTP/1.1 RFC 2616 Section 3.6.
2012-03-08 23:09:19 +01:00
Aldo Cortesi
e1356dd2b6 Create an SSL certificate class. 2012-03-05 10:22:47 +13:00
Aldo Cortesi
8b841bc9e3 Factor out cert operations in to certutils.py. 2012-02-29 13:20:53 +13:00
Aldo Cortesi
0bed5fae27 Rationalise upstream cert flag and variable names. 2012-02-28 11:37:48 +13:00
Aldo Cortesi
00942c1431 Add upstream certificate lookup.
This initiates a connection to the server to obtain certificate information to
generate interception certificates. At the moment, the information used is the
Common Name, and the list of Subject Alternative Names.
2012-02-27 15:05:45 +13:00
Aldo Cortesi
554047da85 License notifications, minor docs. 2012-02-23 15:52:01 +13:00
Aldo Cortesi
1af26bb915 Minor docs and example script fixes. 2012-02-21 12:32:56 +13:00
Aldo Cortesi
dbd75e02f7 Create ODictCaseless for headers, use vanilla ODict for everything else. 2012-02-20 11:29:36 +13:00
Aldo Cortesi
2616f490fe Rename Headers class to ODict
ODict is an ordered dictionary class that will be useful in many other parts of
our API.
2012-02-20 10:39:00 +13:00
Aldo Cortesi
71ad7140be Consolidate palettes somewhat. 2012-02-18 18:48:08 +13:00
Aldo Cortesi
a7df6e1503 Refactor reverse proxying
- Retain the specification from the Host header as a Request's description.
- Expand upstream proxy specifications to include the scheme. We now say https://hostname:port
- Move the "R" revert keybinding to "v" to make room for a reverse proxy
binding that matches the command-line flag.
2012-02-18 14:45:22 +13:00
Heikki Hannikainen
a3509b7f22 reverse proxy mode: small comment clarification 2012-02-16 16:36:49 +02:00
Heikki Hannikainen
a82ac9eaf0 Implemented reverse proxy mode: -R upstreamhost:port makes the
proxy accept a 'GET / HTTP/1.0' request and fill up the destination
host and port from the ones given with -R (for example,
"-R localhost:80").
2012-02-16 16:33:27 +02:00
Aldo Cortesi
5f785e26b9 Add filter for detecting flows with errors.
Also, remove dependency on weird _is_response method.
2012-02-10 15:22:26 +13:00
Aldo Cortesi
d5e3722c97 Fix an issue caused by some editors when editing a request/response body.
Many editors make it hard save a file without a terminating newline on the last
line. When editing message bodies, this can cause problems. For now, I just
strip the newlines off the end of the body when we return from an editor.
2012-01-21 12:43:00 +13:00
meeee
ae79fe1660 Handle missing message/reason phrase in HTTP response status line gracefully by adding an empty one. 2011-09-26 00:44:43 +03:00
Aldo Cortesi
ee71bcfbe8 Fix a rare crash when a new cert is generated during cerdir removal. 2011-09-11 09:06:46 +12:00
Aldo Cortesi
67f2610032 Add HTTP body size limit specification to command-line tools. 2011-09-09 15:27:31 +12:00
Aldo Cortesi
28daa93268 Basic infrastructure for request and response body size limits. 2011-09-09 14:49:34 +12:00
Aldo Cortesi
e5bded7dee Improve robustness against invalid data. 2011-09-05 07:47:47 +12:00
András Veres-Szentkirályi
9abff4f0ac Removed unused imports 2011-08-18 23:30:02 +02:00
Aldo Cortesi
25f12b0e5d Add a basic Flow processor example. 2011-08-13 13:51:38 +12:00
Aldo Cortesi
b51aac8a86 Code cleanliness - appease pychecker. 2011-08-04 10:34:34 +12:00
Aldo Cortesi
7a3b871b33 Request class now has a clean pydoc profile. 2011-08-04 09:26:26 +12:00
Aldo Cortesi
0760607a7d Further interface cleaning. 2011-08-03 23:02:33 +12:00
Aldo Cortesi
57c653be5f Move all HTTP objects to flow.py
That's Request, Response, ClientConnect, ClientDisconnect, Error, and Headers.
2011-08-03 22:41:38 +12:00
Aldo Cortesi
8cc0469ee7 Tweak encoding behaviour
- Don't fail to identity encoding when an unknown encoding is specified.
- Don't constrain encodings. I want to try to modify traffic as little as
possible by default.
- When decoding, delete content-encoding header rather than set it to "identity"
- Refuse to decode/encode when there is an existing but unknown
content-encoding header.
2011-08-02 20:42:46 +12:00
Aldo Cortesi
1ff6a767d0 Unit test++ 2011-08-02 16:52:47 +12:00
Aldo Cortesi
357502fe03 General cleanup.
Cut out unused variables and code, generally shut up pychecker as much as is
reasonable.
2011-08-02 16:14:33 +12:00
Aldo Cortesi
17835b9b78 Fix a rare undefined variable crash in proxy.py. 2011-08-02 15:43:35 +12:00
Aldo Cortesi
ddb5748a76 Add decoding/encoding for requests. 2011-08-01 10:43:01 +12:00
Aldo Cortesi
c89c4361c3 Merge remote-tracking branch 'alts/encoding' 2011-07-28 11:19:07 +12:00
Stephen Altamirano
78049abac1 Changes replace logic to function in both Python 2.6.x and 2.7.x
Tests now only assume Python 2.6.x rather than requiring 2.7.x. This does not preclude the use of flags as a kwarg in replace
2011-07-26 22:47:08 -07:00
Stephen Altamirano
c1eaa9f74c Adds encode and decode methods to Response objects 2011-07-26 22:03:41 -07:00
Aldo Cortesi
0f4ae61e7d Fix a crash in mitmdump event display. 2011-07-23 16:59:48 +12:00
Aldo Cortesi
3648c7953a Extend eventlog information.
Also, squash an SSL-related bug revealed by the extended logging.
2011-07-23 13:37:06 +12:00
Aldo Cortesi
47e1695512 Also replace strings path for requests. 2011-07-22 20:52:13 +12:00
Aldo Cortesi
1b961fc4ad Add utility functions to search and replace strings in flows
This is a common task in pentesting scenarios. This commit adds the following
functions:

utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
2011-07-22 17:48:42 +12:00
Stephen Altamirano
9c24401b18 Removes last_encoding attribute from Response. Prompts for encoding on identity responses 2011-07-21 22:09:48 -07:00
Stephen Altamirano
74d8b18408 Removes should_autodecode attribute from Response. Adds commandline option 'd' to toggle autodecode, adds togglable option 'd' to do the same 2011-07-21 20:22:13 -07:00
Stephen Altamirano
1c5434d72c Adds ability to toggle between encodings in the response view 2011-07-18 21:52:40 -07:00
alts
6dc0f105cc Adds support for content encoding, namely gip and deflate 2011-07-16 02:47:06 -07:00
Aldo Cortesi
76b4c6ba82 Introduce an anti-compression command-line argument.
This is on by default, which means we avoid compressed content unless the -z
flag is specified.
2011-07-15 15:24:56 +12:00
Aldo Cortesi
1c9e7b982a Rewrite Headers object to preserve order and case. 2011-07-14 16:01:54 +12:00
Aldo Cortesi
8e176c2086 Cast some data read from dump files to str, to prevent unicode promotion.
This fixes a bug that caused a traceback when de-serialized requests were
replayed. Also adds unit tests for the problem.
2011-07-01 14:20:42 +12:00
Aldo Cortesi
0a642f2441 Make the certificate wait time configurable.
Since OpenSSL doesn't let us set certificate start times in the past, the
client and proxy machine time must be synchronized, or the client might reject
the certificate. We can bodgy over small discrepancies by waiting a few seconds
after a new certificate is generated (i.e. the first time an SSL domain is contacted).

Make this a configurable option, and turn it off by default.
2011-06-27 16:10:17 +12:00
Aldo Cortesi
b04d074341 Repair a problem that sometimes caused SSL connections to peg the CPU. 2011-06-23 17:00:55 +12:00
Aldo Cortesi
c0bd1a39e4 unit test coverage ++ 2011-03-20 18:52:16 +13:00
Aldo Cortesi
c726519e73 Add a stickyauth option.
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
2011-03-20 17:31:54 +13:00
Aldo Cortesi
e22fd74d06 Revamp key generation.
We now create three different files in the .mitmproxy directory when a dummy CA
is made:

mitmproxy-ca.pem - the CA, including private key

mitmproxy-ca-cert.p12 - A pkcs12 version of the certificate, for distribution to Windows.

mitmproxy-ca-cert.pem - A PEM version of the certificate, for distribution to everyone else.
2011-03-18 16:45:31 +13:00
Aldo Cortesi
fe1e2f16ff Improve responsiveness of request and response viewing.
- Computing the view of a large body is expensive, so we introduce an LRU cache
to hold the latest 20 results.

- Use ListView more correctly, passing it individual urwid.Text snippets,
rather than a single large one. This hugely improves render time.
2011-03-15 13:05:33 +13:00
Aldo Cortesi
8d0152f2ff Add server replay to mitmproxy. 2011-03-13 17:11:59 +13:00
Aldo Cortesi
40e9067b83 Handle invalid bind address specifications gracefully. 2011-03-12 13:47:37 +13:00
Aldo Cortesi
7d85db0da3 Display Errors and killed connections in mitmdump. 2011-03-11 13:06:51 +13:00
Aldo Cortesi
daa9653ebe Add --norefresh to stop refreshing server playback to mitmdump.
Also, make cookie parsing for refreshing more error-tolerant.
2011-03-11 11:56:10 +13:00
Aldo Cortesi
e99b1d1949 Stub out refresh for server-side replay. 2011-03-09 20:05:30 +13:00
Aldo Cortesi
03f1345385 Add an --anticache option to mitmdump.
This removes all headers that might cause a server to return 304-not-modified.

For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
2011-03-09 13:15:31 +13:00
Aldo Cortesi
765871bd11 Store timestamps on flow components as a UTC time tuple.
Format is:

(tm_year,tm_mon,tm_mday,tm_hour,tm_min, tm_sec,tm_wday,tm_yday,tm_isdst)
2011-03-07 13:46:02 +13:00
Aldo Cortesi
6921b9ff2a Add an indicator that sticky cookies have been applied in mitmdump. 2011-02-25 21:23:44 +13:00
Aldo Cortesi
8cade9fbbf Move stringification funcs from proxy to dump. 2011-02-25 21:11:44 +13:00
Aldo Cortesi
57947b328e Start abstracting out sticky cookie state. 2011-02-24 10:33:39 +13:00
Aldo Cortesi
c3e3897071 Fix a subtle Unicode problem in Response.assemble
If msg is Unicode, the proto string is automatically promoted to Unicode.  If
the proto string is promoted to Unicode, then the FMT interpolation is also
done in Unicode. If this happens, then binary data in content will cause an
exception.
2011-02-21 10:11:50 +13:00
Aldo Cortesi
fd4dd8cb6b First pass of playback function for mitmdump. 2011-02-21 09:54:39 +13:00
Aldo Cortesi
7ddba22f51 Certificates are now generated in a temporary per-session directory.
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
2011-02-20 13:29:41 +13:00
Aldo Cortesi
c2ae8285f4 Revamp SSL configuration.
- Move option parsing utiliities to proxy.py

- Don't have a global config object. Pass it as an argument to ProxyServer.

- Simplify certificate generation logic.
2011-02-20 12:53:42 +13:00
Aldo Cortesi
4fc807cedd Clean up certificate generation.
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.

- Split CA and cert generation into separate functions.

- Generation methods provide an error return when generation fails.

- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
2011-02-20 12:17:10 +13:00
Aldo Cortesi
9c5c3c2b1a Implement state loading that doesn't change object identity.
We need this to let us load state from copied Flows returned from scripts.
2011-02-20 09:36:13 +13:00
Aldo Cortesi
d9c6be3d13 Fix serialization when a Request has no associated client connection. 2011-02-19 17:03:44 +13:00
Aldo Cortesi
5da27a9905 Refactor Flow primitives to remove HTTP1.0 assumption.
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
2011-02-19 17:00:24 +13:00
Aldo Cortesi
8a9f20b78c Don't leave dangling browser connections for pipelined requests. 2011-02-16 22:37:04 +13:00
Aldo Cortesi
f009770d4c Fix a bug in HTTP 1.1 pipelining that caused Requests to be over-written.
We use the ClientConnection object to tie requests, responses and errors
together. This is an HTTP 1.0 assumption, but we can fix it by just making
copies of the connection object when we handle multiple requests.
2011-02-16 19:22:19 +13:00
Aldo Cortesi
66349c9783 FlowMaster bugfixes and unit tests. 2011-02-16 16:43:35 +13:00
Aldo Cortesi
27f0765cdd Extract flow-specific Master operations into FlowMaster. 2011-02-16 16:03:22 +13:00
Aldo Cortesi
33dac700fa Switch over to new serialization format.
Remove BSON from contrib.
2011-02-16 15:10:00 +13:00
Aldo Cortesi
0c6f846861 First draft of the new serialization mechanism. 2011-02-16 14:33:04 +13:00
Henrik Nordstrom
89627a702a drop unused protocol parameter from Response 2011-02-15 22:24:38 +01:00
Aldo Cortesi
759496e24e Repair unit test suite. 2011-02-11 11:06:30 +13:00
Henrik Nordstrom
32adee8743 Implement a dummy CA 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
4ffaadd435 Allow specifying the accepted ciphersuites 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
d11dd742d8 Simple record & playback functionality 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
4bae297fbb Basic HTTP/1.1 Support
Adds support for chunked transfer encoding, and a couple other minor
protocol corrections.

Improve HTTP support

- Support intercepted requests with Host header
- Support HEAD requests proper
- Support any HTTP method including extensions, not just a couple known ones

Support expect: 100-continue and 100 Continue messages

Persistent client connections

Generalize ServerConnection a bit in preparation for keep-alive support

Correct HTTP status codes on errors forwarding the request
2011-02-10 02:59:51 +01:00
Henrik Nordstrom
0613321aef Optimize CONNECT responses, sent in a single packet 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
bd01126b2b Move try_del to utils 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
98b1acc51d Add response creation to edit function on intercepted requests 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
fcc39e1aaf Terminate workers when main thread terminates 2011-02-10 02:59:51 +01:00
Aldo Cortesi
44dc3a052e Add option to tell mitmproxy which interfaces to bind to. 2011-02-06 10:52:54 +13:00
Aldo Cortesi
d0c2d9480c Also serialize ClientConnection and flow backups. 2011-02-06 10:28:43 +13:00
Aldo Cortesi
2ad4c5adf3 Get rid of ReplayConnection - we now have only one ClientConnection class. 2011-02-04 10:58:08 +13:00
Aldo Cortesi
2cb7429d38 Change "connection" to the less confusing "client_conn" throughout. 2011-02-03 14:51:32 +13:00
Aldo Cortesi
6c89749f0a Add timestamps to flows.
For now, these are only displayed on the connection view screen, with second
granularity.
2011-02-03 13:30:47 +13:00
Aldo Cortesi
d8cb826361 Test suite rejiggering and cleanup. 2011-02-03 12:16:03 +13:00
Aldo Cortesi
eb93cc22ce Fine-tune threading and fix an exception. 2011-01-27 14:19:48 +13:00
Aldo Cortesi
efe11a0782 Kill deadlock that sometimes occurred on shutdown. 2011-01-27 13:32:24 +13:00
Aldo Cortesi
29d8007678 Add serialization hooks to flows and flow component objects. 2011-01-26 14:52:03 +13:00
Aldo Cortesi
324c175d02 Fix indentation in previous patch. 2010-09-06 11:21:51 +12:00
Henrik Nordström
2e7e9f9759 Wrap read request, concatenating partial reads until whole request have been read 2010-09-06 07:18:11 +08:00
Aldo Cortesi
c5eb3c592d Fix hang when a POST is made with a 0 content length. 2010-02-26 20:07:37 +13:00
Aldo Cortesi
cb0e328709 Initial checkin. 2010-02-16 17:09:07 +13:00