Commit Graph

276 Commits

Author SHA1 Message Date
Aldo Cortesi
b51aac8a86 Code cleanliness - appease pychecker. 2011-08-04 10:34:34 +12:00
Aldo Cortesi
7a3b871b33 Request class now has a clean pydoc profile. 2011-08-04 09:26:26 +12:00
Aldo Cortesi
0760607a7d Further interface cleaning. 2011-08-03 23:02:33 +12:00
Aldo Cortesi
57c653be5f Move all HTTP objects to flow.py
That's Request, Response, ClientConnect, ClientDisconnect, Error, and Headers.
2011-08-03 22:41:38 +12:00
Aldo Cortesi
8cc0469ee7 Tweak encoding behaviour
- Don't fail to identity encoding when an unknown encoding is specified.
- Don't constrain encodings. I want to try to modify traffic as little as
possible by default.
- When decoding, delete content-encoding header rather than set it to "identity"
- Refuse to decode/encode when there is an existing but unknown
content-encoding header.
2011-08-02 20:42:46 +12:00
Aldo Cortesi
1ff6a767d0 Unit test++ 2011-08-02 16:52:47 +12:00
Aldo Cortesi
357502fe03 General cleanup.
Cut out unused variables and code, generally shut up pychecker as much as is
reasonable.
2011-08-02 16:14:33 +12:00
Aldo Cortesi
17835b9b78 Fix a rare undefined variable crash in proxy.py. 2011-08-02 15:43:35 +12:00
Aldo Cortesi
ddb5748a76 Add decoding/encoding for requests. 2011-08-01 10:43:01 +12:00
Aldo Cortesi
c89c4361c3 Merge remote-tracking branch 'alts/encoding' 2011-07-28 11:19:07 +12:00
Stephen Altamirano
78049abac1 Changes replace logic to function in both Python 2.6.x and 2.7.x
Tests now only assume Python 2.6.x rather than requiring 2.7.x. This does not preclude the use of flags as a kwarg in replace
2011-07-26 22:47:08 -07:00
Stephen Altamirano
c1eaa9f74c Adds encode and decode methods to Response objects 2011-07-26 22:03:41 -07:00
Aldo Cortesi
0f4ae61e7d Fix a crash in mitmdump event display. 2011-07-23 16:59:48 +12:00
Aldo Cortesi
3648c7953a Extend eventlog information.
Also, squash an SSL-related bug revealed by the extended logging.
2011-07-23 13:37:06 +12:00
Aldo Cortesi
47e1695512 Also replace strings path for requests. 2011-07-22 20:52:13 +12:00
Aldo Cortesi
1b961fc4ad Add utility functions to search and replace strings in flows
This is a common task in pentesting scenarios. This commit adds the following
functions:

utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
2011-07-22 17:48:42 +12:00
Stephen Altamirano
9c24401b18 Removes last_encoding attribute from Response. Prompts for encoding on identity responses 2011-07-21 22:09:48 -07:00
Stephen Altamirano
74d8b18408 Removes should_autodecode attribute from Response. Adds commandline option 'd' to toggle autodecode, adds togglable option 'd' to do the same 2011-07-21 20:22:13 -07:00
Stephen Altamirano
1c5434d72c Adds ability to toggle between encodings in the response view 2011-07-18 21:52:40 -07:00
alts
6dc0f105cc Adds support for content encoding, namely gip and deflate 2011-07-16 02:47:06 -07:00
Aldo Cortesi
76b4c6ba82 Introduce an anti-compression command-line argument.
This is on by default, which means we avoid compressed content unless the -z
flag is specified.
2011-07-15 15:24:56 +12:00
Aldo Cortesi
1c9e7b982a Rewrite Headers object to preserve order and case. 2011-07-14 16:01:54 +12:00
Aldo Cortesi
8e176c2086 Cast some data read from dump files to str, to prevent unicode promotion.
This fixes a bug that caused a traceback when de-serialized requests were
replayed. Also adds unit tests for the problem.
2011-07-01 14:20:42 +12:00
Aldo Cortesi
0a642f2441 Make the certificate wait time configurable.
Since OpenSSL doesn't let us set certificate start times in the past, the
client and proxy machine time must be synchronized, or the client might reject
the certificate. We can bodgy over small discrepancies by waiting a few seconds
after a new certificate is generated (i.e. the first time an SSL domain is contacted).

Make this a configurable option, and turn it off by default.
2011-06-27 16:10:17 +12:00
Aldo Cortesi
b04d074341 Repair a problem that sometimes caused SSL connections to peg the CPU. 2011-06-23 17:00:55 +12:00
Aldo Cortesi
c0bd1a39e4 unit test coverage ++ 2011-03-20 18:52:16 +13:00
Aldo Cortesi
c726519e73 Add a stickyauth option.
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
2011-03-20 17:31:54 +13:00
Aldo Cortesi
e22fd74d06 Revamp key generation.
We now create three different files in the .mitmproxy directory when a dummy CA
is made:

mitmproxy-ca.pem - the CA, including private key

mitmproxy-ca-cert.p12 - A pkcs12 version of the certificate, for distribution to Windows.

mitmproxy-ca-cert.pem - A PEM version of the certificate, for distribution to everyone else.
2011-03-18 16:45:31 +13:00
Aldo Cortesi
fe1e2f16ff Improve responsiveness of request and response viewing.
- Computing the view of a large body is expensive, so we introduce an LRU cache
to hold the latest 20 results.

- Use ListView more correctly, passing it individual urwid.Text snippets,
rather than a single large one. This hugely improves render time.
2011-03-15 13:05:33 +13:00
Aldo Cortesi
8d0152f2ff Add server replay to mitmproxy. 2011-03-13 17:11:59 +13:00
Aldo Cortesi
40e9067b83 Handle invalid bind address specifications gracefully. 2011-03-12 13:47:37 +13:00
Aldo Cortesi
7d85db0da3 Display Errors and killed connections in mitmdump. 2011-03-11 13:06:51 +13:00
Aldo Cortesi
daa9653ebe Add --norefresh to stop refreshing server playback to mitmdump.
Also, make cookie parsing for refreshing more error-tolerant.
2011-03-11 11:56:10 +13:00
Aldo Cortesi
e99b1d1949 Stub out refresh for server-side replay. 2011-03-09 20:05:30 +13:00
Aldo Cortesi
03f1345385 Add an --anticache option to mitmdump.
This removes all headers that might cause a server to return 304-not-modified.

For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
2011-03-09 13:15:31 +13:00
Aldo Cortesi
765871bd11 Store timestamps on flow components as a UTC time tuple.
Format is:

(tm_year,tm_mon,tm_mday,tm_hour,tm_min, tm_sec,tm_wday,tm_yday,tm_isdst)
2011-03-07 13:46:02 +13:00
Aldo Cortesi
6921b9ff2a Add an indicator that sticky cookies have been applied in mitmdump. 2011-02-25 21:23:44 +13:00
Aldo Cortesi
8cade9fbbf Move stringification funcs from proxy to dump. 2011-02-25 21:11:44 +13:00
Aldo Cortesi
57947b328e Start abstracting out sticky cookie state. 2011-02-24 10:33:39 +13:00
Aldo Cortesi
c3e3897071 Fix a subtle Unicode problem in Response.assemble
If msg is Unicode, the proto string is automatically promoted to Unicode.  If
the proto string is promoted to Unicode, then the FMT interpolation is also
done in Unicode. If this happens, then binary data in content will cause an
exception.
2011-02-21 10:11:50 +13:00
Aldo Cortesi
fd4dd8cb6b First pass of playback function for mitmdump. 2011-02-21 09:54:39 +13:00
Aldo Cortesi
7ddba22f51 Certificates are now generated in a temporary per-session directory.
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
2011-02-20 13:29:41 +13:00
Aldo Cortesi
c2ae8285f4 Revamp SSL configuration.
- Move option parsing utiliities to proxy.py

- Don't have a global config object. Pass it as an argument to ProxyServer.

- Simplify certificate generation logic.
2011-02-20 12:53:42 +13:00
Aldo Cortesi
4fc807cedd Clean up certificate generation.
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.

- Split CA and cert generation into separate functions.

- Generation methods provide an error return when generation fails.

- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
2011-02-20 12:17:10 +13:00
Aldo Cortesi
9c5c3c2b1a Implement state loading that doesn't change object identity.
We need this to let us load state from copied Flows returned from scripts.
2011-02-20 09:36:13 +13:00
Aldo Cortesi
d9c6be3d13 Fix serialization when a Request has no associated client connection. 2011-02-19 17:03:44 +13:00
Aldo Cortesi
5da27a9905 Refactor Flow primitives to remove HTTP1.0 assumption.
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
2011-02-19 17:00:24 +13:00
Aldo Cortesi
8a9f20b78c Don't leave dangling browser connections for pipelined requests. 2011-02-16 22:37:04 +13:00
Aldo Cortesi
f009770d4c Fix a bug in HTTP 1.1 pipelining that caused Requests to be over-written.
We use the ClientConnection object to tie requests, responses and errors
together. This is an HTTP 1.0 assumption, but we can fix it by just making
copies of the connection object when we handle multiple requests.
2011-02-16 19:22:19 +13:00
Aldo Cortesi
66349c9783 FlowMaster bugfixes and unit tests. 2011-02-16 16:43:35 +13:00
Aldo Cortesi
27f0765cdd Extract flow-specific Master operations into FlowMaster. 2011-02-16 16:03:22 +13:00
Aldo Cortesi
33dac700fa Switch over to new serialization format.
Remove BSON from contrib.
2011-02-16 15:10:00 +13:00
Aldo Cortesi
0c6f846861 First draft of the new serialization mechanism. 2011-02-16 14:33:04 +13:00
Henrik Nordstrom
89627a702a drop unused protocol parameter from Response 2011-02-15 22:24:38 +01:00
Aldo Cortesi
759496e24e Repair unit test suite. 2011-02-11 11:06:30 +13:00
Henrik Nordstrom
32adee8743 Implement a dummy CA 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
4ffaadd435 Allow specifying the accepted ciphersuites 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
d11dd742d8 Simple record & playback functionality 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
4bae297fbb Basic HTTP/1.1 Support
Adds support for chunked transfer encoding, and a couple other minor
protocol corrections.

Improve HTTP support

- Support intercepted requests with Host header
- Support HEAD requests proper
- Support any HTTP method including extensions, not just a couple known ones

Support expect: 100-continue and 100 Continue messages

Persistent client connections

Generalize ServerConnection a bit in preparation for keep-alive support

Correct HTTP status codes on errors forwarding the request
2011-02-10 02:59:51 +01:00
Henrik Nordstrom
0613321aef Optimize CONNECT responses, sent in a single packet 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
bd01126b2b Move try_del to utils 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
98b1acc51d Add response creation to edit function on intercepted requests 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
fcc39e1aaf Terminate workers when main thread terminates 2011-02-10 02:59:51 +01:00
Aldo Cortesi
44dc3a052e Add option to tell mitmproxy which interfaces to bind to. 2011-02-06 10:52:54 +13:00
Aldo Cortesi
d0c2d9480c Also serialize ClientConnection and flow backups. 2011-02-06 10:28:43 +13:00
Aldo Cortesi
2ad4c5adf3 Get rid of ReplayConnection - we now have only one ClientConnection class. 2011-02-04 10:58:08 +13:00
Aldo Cortesi
2cb7429d38 Change "connection" to the less confusing "client_conn" throughout. 2011-02-03 14:51:32 +13:00
Aldo Cortesi
6c89749f0a Add timestamps to flows.
For now, these are only displayed on the connection view screen, with second
granularity.
2011-02-03 13:30:47 +13:00
Aldo Cortesi
d8cb826361 Test suite rejiggering and cleanup. 2011-02-03 12:16:03 +13:00
Aldo Cortesi
eb93cc22ce Fine-tune threading and fix an exception. 2011-01-27 14:19:48 +13:00
Aldo Cortesi
efe11a0782 Kill deadlock that sometimes occurred on shutdown. 2011-01-27 13:32:24 +13:00
Aldo Cortesi
29d8007678 Add serialization hooks to flows and flow component objects. 2011-01-26 14:52:03 +13:00
Aldo Cortesi
324c175d02 Fix indentation in previous patch. 2010-09-06 11:21:51 +12:00
Henrik Nordström
2e7e9f9759 Wrap read request, concatenating partial reads until whole request have been read 2010-09-06 07:18:11 +08:00
Aldo Cortesi
c5eb3c592d Fix hang when a POST is made with a 0 content length. 2010-02-26 20:07:37 +13:00
Aldo Cortesi
cb0e328709 Initial checkin. 2010-02-16 17:09:07 +13:00