Grasscutters/mitmproxy
3
0
Fork 0
mirror of https://github.com/Grasscutters/mitmproxy.git synced 2024-11-26 18:18:25 +00:00
Code Issues Projects Releases Wiki Activity
2,400 Commits 1 Branch 67 Tags 50 MiB
55e89865ff
Commit Graph

342 Commits

Author SHA1 Message Date
Aldo Cortesi
0c2d894cea Add the ability to flag content as missing in a request or a response. 
We'll use this in a number of situations. First, we'll soon have response
streaming that directly pipes responses to clients. These will be content-less
from mitmproxy's perspective. Second, we'll be growing new events that fire
after headers are received, but before content is read.
 2012-05-16 15:42:58 +12:00
Aldo Cortesi
2fe54d17df Don't specify Content-Length on empty content. 
Sometimes, mitmproxy would specify a content-length header value of 0 when
content was empty. Some rare servers (like piratebay.org) would barf on this.
 2012-04-25 14:38:20 +12:00
Aldo Cortesi
c8d2b2594b Add a WSGI adapter that lets us serve a WSGI app out of mitmproxy. 
This commit adds:
    - A WSGI App adapter for mitmproxy
    - An app registry in the proxy instance that lets us link WSGI apps with
    (hostname, port) combinations.
    - Fixes for a number of bugs discovered while creating this feature.
 2012-04-24 14:52:29 +12:00
Aldo Cortesi
8c96264304 Serialized data version check. 2012-04-11 10:10:53 +12:00
Aldo Cortesi
79a0334a02 Improve revert model 
- Flows are backed up whenever an interactive, non-script change is made.
- That backup is canonical and never changed - "V" will always revert to it.

This makes more sense than what we had previously....
 2012-04-04 09:47:57 +12:00
Aldo Cortesi
ab0e10e60f Serialize requestcount for ClientConnect objects. 2012-04-03 22:37:24 +12:00
Aldo Cortesi
4979a22d3e Add accessor method for SSLCert object on Response. 2012-04-02 17:02:23 +12:00
Aldo Cortesi
ab1d8fa350 Expand SSL cert support 
- Capture the remote SSL certificate
- Expose the remote cert as an attribute on Response
- Expand the certutils.SSLCert interface to expose more cert info
 2012-04-02 16:21:23 +12:00
Aldo Cortesi
62e51018d0 Refactor pretty view mechanism. 
Also start adding unit tests for this subsystem.
 2012-03-24 14:20:24 +13:00
Aldo Cortesi
ed74ed24a0 Add error indications to GridEditor. 2012-03-23 13:28:33 +13:00
Aldo Cortesi
5690e7c399 Generalize GridEditor to N columns. 
Start adding a replacement rule editor.
 2012-03-18 14:39:21 +13:00
Aldo Cortesi
c8ae1e85b3 Hooks -> ReplaceHooks 
It makes more sense to specialize this, which will let me build a nicer
interface for replacement hooks in mitmproxy.
 2012-03-17 11:31:05 +13:00
Aldo Cortesi
08f410cacc Add a hooks mechanism, based on filter expressions. 2012-03-16 17:13:11 +13:00
Aldo Cortesi
d138af7217 replace() methods now decode and re-encode contents before substitution. 2012-03-16 11:24:18 +13:00
Aldo Cortesi
d51b8cab0c Add a decoded context manager. 
This simplifies a common chore when modifying traffic - decoding the object,
modifying it, then re-encoding it with the same encoding afterwards. You can
now simply say:

with flow.decoded(request):
    request.content = "bar"
 2012-03-16 11:12:56 +13:00
Aldo Cortesi
8d662e6636 Set a "unique" serial number for each generated cert. 2012-03-14 11:20:25 +13:00
Valtteri Virtanen
ed56d67cea Adds no-pop option to server-side replay 2012-03-05 11:05:11 +02:00
Aldo Cortesi
986a41d180 Unit test++. 2012-02-25 12:19:54 +13:00
Aldo Cortesi
25fa596cd6 Fix detection of URL-encoded forms. 
Thanks to Paul Capestany <capestany@gmail.com> for reporting this.
 2012-02-24 13:03:24 +13:00
Aldo Cortesi
2df9c52c09 Refactor filter matching. 2012-02-23 17:03:58 +13:00
Aldo Cortesi
554047da85 License notifications, minor docs. 2012-02-23 15:52:01 +13:00
Aldo Cortesi
4f38b3a9c0 Documentation and screenshots. 2012-02-22 17:17:13 +13:00
Aldo Cortesi
a4270efaf2 Always return an ODict from get_query 2012-02-21 13:00:45 +13:00
Aldo Cortesi
dbd75e02f7 Create ODictCaseless for headers, use vanilla ODict for everything else. 2012-02-20 11:29:36 +13:00
Aldo Cortesi
18029df99c Use ODict for request.get_form_urlencoded and set_form_urlencoded 2012-02-20 11:13:35 +13:00
Aldo Cortesi
fa11b7c9be Use ODict for Request.get_query and Request.set_query 2012-02-20 10:44:47 +13:00
Aldo Cortesi
2616f490fe Rename Headers class to ODict 
ODict is an ordered dictionary class that will be useful in many other parts of
our API.
 2012-02-20 10:39:00 +13:00
Aldo Cortesi
25a06c3ec1 Minor doc fixes and import cleanups. 2012-02-20 10:15:58 +13:00
Aldo Cortesi
77a33c441b Add duplicate_flow and replay_request hooks to ScriptContext. 2012-02-19 11:29:49 +13:00
Aldo Cortesi
d32d6bc5e3 Add "p" key binding to connection list view to copy a flow. 2012-02-19 00:17:47 +13:00
Aldo Cortesi
8ddc3b4ef2 Add API for duplicating flows. 2012-02-18 23:56:40 +13:00
Aldo Cortesi
6ad8b1a15d Firm up reverse proxy specification. 
- Extract proxy spec parsing and unparsing functions.
- Add a status indicator in mitmproxy.
- Add the "R" keybinding for changing the reverse proxy from within mitmproxy.
 2012-02-18 16:27:09 +13:00
Aldo Cortesi
a7df6e1503 Refactor reverse proxying 
- Retain the specification from the Host header as a Request's description.
- Expand upstream proxy specifications to include the scheme. We now say https://hostname:port
- Move the "R" revert keybinding to "v" to make room for a reverse proxy
binding that matches the command-line flag.
 2012-02-18 14:45:22 +13:00
Aldo Cortesi
14def89f50 Fix a problem in deserialization of flows with errors. 2012-02-18 12:25:22 +13:00
Aldo Cortesi
1ad7e91527 Make filter matching act more sensibly. 2012-02-10 15:31:45 +13:00
Aldo Cortesi
5f785e26b9 Add filter for detecting flows with errors. 
Also, remove dependency on weird _is_response method.
 2012-02-10 15:22:26 +13:00
Aldo Cortesi
b14c29b25c Expand test coverage. 2012-02-10 15:04:20 +13:00
Aldo Cortesi
5326b7610a Enable editing of urlencoded form data with KVEditor. 2012-02-10 14:35:23 +13:00
Aldo Cortesi
9c985f2d20 Methods for getting and setting form urlencoded data on Request. 2012-02-10 14:27:39 +13:00
Aldo Cortesi
2709441d5b Add get_query and set_query methods to Request. 2012-02-09 16:40:31 +13:00
Aldo Cortesi
4b9ee4c31e Very basic KV editor mockup. 2012-02-06 09:49:49 +13:00
Aldo Cortesi
3b246f7e27 Simple fix for a unicode error when editing a request URL. 2011-10-26 14:49:15 +13:00
Aldo Cortesi
d9db1cf5b3 Change size limit cmdline flag to -Z, enable size limits for replay. 2011-09-09 17:31:36 +12:00
Aldo Cortesi
67f2610032 Add HTTP body size limit specification to command-line tools. 2011-09-09 15:27:31 +12:00
Aldo Cortesi
4ac59a7859 Fix a rare crash in sticky cookies. 2011-08-26 18:03:03 +12:00
Aldo Cortesi
8fbba59e8d Fix a problem with sticky cookie domain matching. 
Just like everything else cookie-related in the standard library,
cookielib.domain_match is fucked up.
 2011-08-26 17:37:12 +12:00
Aldo Cortesi
45f4768a5c Add attribution and license for tnetstring.py 2011-08-19 21:53:52 +12:00
Aldo Cortesi
a566684e32 Move to typed netstrings for serialization. 
This change is backwards incompatible with the old serialization format!
 2011-08-19 21:30:24 +12:00
András Veres-Szentkirályi
b1dc418a53 Replaced unnecessary lists with generators 2011-08-18 23:29:57 +02:00
Aldo Cortesi
f23818ceea Add a "done" event for scripts. 
Called exactly once after all other events.
 2011-08-05 14:08:03 +12:00
Aldo Cortesi
87623a8d75 Rip out autodecode 
We simplify things as follows:

    - If we're in "pretty" view mode, we autodecode.
    - Otherwise, we display raw data, and the user can manually encode/decode
    with z shortcut.
 2011-08-04 10:54:42 +12:00
Aldo Cortesi
b51aac8a86 Code cleanliness - appease pychecker. 2011-08-04 10:34:34 +12:00
Aldo Cortesi
730c78ac53 Move script.Context to flow.ScriptContext 2011-08-04 10:14:44 +12:00
Aldo Cortesi
1662b8505b Clean pydoc profile for flow.Flow 2011-08-04 09:56:44 +12:00
Aldo Cortesi
8ef208a9e2 Clean pydoc profile for flow.Response, flow.Error 2011-08-04 09:44:48 +12:00
Aldo Cortesi
7a3b871b33 Request class now has a clean pydoc profile. 2011-08-04 09:26:26 +12:00
Aldo Cortesi
0760607a7d Further interface cleaning. 2011-08-03 23:02:33 +12:00
Aldo Cortesi
9042d3f3b9 Clean up interfaces by making some methods pseudo-private. 2011-08-03 22:48:57 +12:00
Aldo Cortesi
57c653be5f Move all HTTP objects to flow.py 
That's Request, Response, ClientConnect, ClientDisconnect, Error, and Headers.
 2011-08-03 22:41:38 +12:00
Aldo Cortesi
e337682d8e Enable "|" command to run a oneshot script on a single flow. 2011-08-03 17:35:18 +12:00
Aldo Cortesi
179cf75862 Add script hooks, enable new engine for mitmdump. 2011-08-03 16:36:20 +12:00
Aldo Cortesi
f7e4e89b12 Move the event notification mechanism into flow.py 2011-08-03 13:33:18 +12:00
Aldo Cortesi
12d2b1f926 Rip out old script interface, start replacing with new stubs. 
Scripts are broken for now.
 2011-08-03 13:20:36 +12:00
Aldo Cortesi
8cc0469ee7 Tweak encoding behaviour 
- Don't fail to identity encoding when an unknown encoding is specified.
- Don't constrain encodings. I want to try to modify traffic as little as
possible by default.
- When decoding, delete content-encoding header rather than set it to "identity"
- Refuse to decode/encode when there is an existing but unknown
content-encoding header.
 2011-08-02 20:42:46 +12:00
Aldo Cortesi
357502fe03 General cleanup. 
Cut out unused variables and code, generally shut up pychecker as much as is
reasonable.
 2011-08-02 16:14:33 +12:00
Aldo Cortesi
f3742f29da We no longer need to track clientconnections. 2011-08-02 14:56:09 +12:00
Aldo Cortesi
675b3133b4 Improve performance of loading flows from a file hugely. 
Fell into the "expensive __eq__ method" trap. Oh, Python, you little scamp.
 2011-08-01 11:26:09 +12:00
Aldo Cortesi
43f1c72511 Refactor the way we calculate views of the flow list. 
The naive approach we used before recalculated the view on every access, and
consequently had serious performance problems.
 2011-08-01 11:17:01 +12:00
Stephen Altamirano
78049abac1 Changes replace logic to function in both Python 2.6.x and 2.7.x 
Tests now only assume Python 2.6.x rather than requiring 2.7.x. This does not preclude the use of flags as a kwarg in replace
 2011-07-26 22:47:08 -07:00
Aldo Cortesi
e6288e2d07 Fix crash when sticky cookies are read from file. 
Cookielib expects strings, not unicode.
 2011-07-24 16:08:27 +12:00
Aldo Cortesi
1b961fc4ad Add utility functions to search and replace strings in flows 
This is a common task in pentesting scenarios. This commit adds the following
functions:

utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
 2011-07-22 17:48:42 +12:00
Stephen Altamirano
74d8b18408 Removes should_autodecode attribute from Response. Adds commandline option 'd' to toggle autodecode, adds togglable option 'd' to do the same 2011-07-21 20:22:13 -07:00
alts
6dc0f105cc Adds support for content encoding, namely gip and deflate 2011-07-16 02:47:06 -07:00
Aldo Cortesi
76b4c6ba82 Introduce an anti-compression command-line argument. 
This is on by default, which means we avoid compressed content unless the -z
flag is specified.
 2011-07-15 15:24:56 +12:00
Aldo Cortesi
1c9e7b982a Rewrite Headers object to preserve order and case. 2011-07-14 16:01:54 +12:00
Aldo Cortesi
2ae7808ca9 Don't redraw the screen more often than necessary. 2011-06-27 14:01:08 +12:00
Aldo Cortesi
0d9e0eac9a Don't backup flows before replay. 
This lets us revert to the original request, even after replaying an edit.
 2011-06-23 14:47:34 +12:00
Aldo Cortesi
e285b17e3f Add -r option to mitmdump and mitmproxy. 
This option reads a set of flows from a file. I've also regularized the
mitmdump and mitmproxy command-line signatures by removing mitmproxy's old way
of specifying flow loads through naked arguments.
 2011-05-15 11:54:12 +12:00
Aldo Cortesi
613e9a298e Add a new flow loading mechanism. 
We now simulate the normal connection flow when we load flows. That means
that we can run scripts, hooks, sticky cookies, etc.
 2011-05-15 11:22:35 +12:00
Aldo Cortesi
c726519e73 Add a stickyauth option. 
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
 2011-03-20 17:31:54 +13:00
Aldo Cortesi
4f877cde6a Reverse order of flows in mitmproxy. 
It matches user expectations much better to have new flows appended to the bottom.
 2011-03-20 09:31:39 +13:00
Aldo Cortesi
89b0ac5195 Server replay shortcut in mitmproxy can now be used to interrupt a current replay. 2011-03-18 09:43:47 +13:00
Aldo Cortesi
f97c144869 Client playback shortcut in mitmproxy can now be used to interrupt a current client playback. 2011-03-18 09:38:51 +13:00
Aldo Cortesi
394bd1d6b0 Interrupt interception when deleting an intercepting flow. 
Prompting the user for this is annoying.
 2011-03-15 17:53:29 +13:00
Aldo Cortesi
fb28e71f0b Do pre-processing of requests before replay. 
This enables scripts, anticache, server playback and sticky cookies for
request replays.
 2011-03-15 15:11:03 +13:00
Aldo Cortesi
324d7c3955 Add client plaback to mitmproxy. 2011-03-13 16:50:11 +13:00
Aldo Cortesi
ef27e2fb29 Rejigger status bar somewhat. Add indicators for various bits of state. 
In particular, we now display limits, sticky cookie expressions and intercepts
in the bar.
 2011-03-13 15:55:47 +13:00
Aldo Cortesi
18d25ec1a1 Fix stickycookie setting, indented view traceback. 2011-03-13 11:43:55 +13:00
Aldo Cortesi
c901cbbada mitmproxy prompted input now display previously set value. 
E.g. if you set a limit, then re-enter the limit prompt, you start with the
currently set value.
 2011-03-13 11:24:49 +13:00
Aldo Cortesi
40e9067b83 Handle invalid bind address specifications gracefully. 2011-03-12 13:47:37 +13:00
Aldo Cortesi
9f16a84a9e Make mitmdump handle invalid serialized data gracefully. 2011-03-11 15:16:31 +13:00
Aldo Cortesi
7d85db0da3 Display Errors and killed connections in mitmdump. 2011-03-11 13:06:51 +13:00
Aldo Cortesi
daa9653ebe Add --norefresh to stop refreshing server playback to mitmdump. 
Also, make cookie parsing for refreshing more error-tolerant.
 2011-03-11 11:56:10 +13:00
Aldo Cortesi
03f1345385 Add an --anticache option to mitmdump. 
This removes all headers that might cause a server to return 304-not-modified.

For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
 2011-03-09 13:15:31 +13:00
Aldo Cortesi
592812467d Move version constants to version.py, add version to serialized flows. 
For now, we don't do anything with the serialized version info, but this should
give us some future-proofing for serialization format changes.
 2011-03-09 12:36:14 +13:00
Aldo Cortesi
7330f5fccf Make mitmdump server playback also exit by default. 
Like client playback, the --keepserving option makes mitmdump keep serving.
 2011-03-06 17:08:56 +13:00
Aldo Cortesi
ec00b5a66e Make mitmdump exit after client replay is complete by default. 
Add an option --keepserving to make it keep serving after replay.
 2011-03-06 16:54:49 +13:00
Aldo Cortesi
e794cbc0d8 Add flow_count and active_flow_count methods to flow.State. 
Use these in unit tests where sensible.
 2011-03-06 16:11:45 +13:00
Aldo Cortesi
42ffded626 Bump unit tests, rearrange mitmdump command-line options slightly. 2011-03-06 16:02:28 +13:00
Aldo Cortesi
4f02480482 Allow client replays to be replayed against server replays. 
Useful for testing.
 2011-03-06 14:48:16 +13:00
Aldo Cortesi
bad77f3470 Add client replay options to mitmdump. 2011-03-06 11:21:31 +13:00
Aldo Cortesi
5da4099ddf Massage namespace to make room for client replay. 
Mostly replay -> server_replay
 2011-03-05 13:03:26 +13:00
Aldo Cortesi
96471fde1d Basic client playback state structure. 2011-03-04 13:08:43 +13:00
Aldo Cortesi
6921b9ff2a Add an indicator that sticky cookies have been applied in mitmdump. 2011-02-25 21:23:44 +13:00
Aldo Cortesi
57f96c5fe0 Minor housekeeping - unused code, nocover directives. 
95% test coverage.
 2011-02-24 15:44:08 +13:00
Aldo Cortesi
79039eb5d2 More mature sticky cookie primitive. Use it in console.py. 2011-02-24 15:26:34 +13:00
Aldo Cortesi
57947b328e Start abstracting out sticky cookie state. 2011-02-24 10:33:39 +13:00
Aldo Cortesi
39207ffdd2 Add a way for users to specify header significance in server replay. 
Also add the --rheader command-line option to mitmdump to let the user specify
an arbitrary number of significant headers. The default is to treat no headers
as significant.
 2011-02-23 10:54:51 +13:00
Aldo Cortesi
7bc913c40d Move script hooks into the flow primitives. 
This lets handle scripts in corner cases like playback situations more easily.
 2011-02-21 11:40:49 +13:00
Aldo Cortesi
fe99871df8 Add --kill option to mitmdump 
If this option is passed all requests that are not part of a replayed
conversation are killed. If the option is not passed, such requests are passed
through to the server as usual.
 2011-02-21 11:08:35 +13:00
Aldo Cortesi
fd4dd8cb6b First pass of playback function for mitmdump. 2011-02-21 09:54:39 +13:00
Aldo Cortesi
deb79a9c5a Add a simple server playback state object. 
We use a loose hash to match incoming requests with recorded flows. At the
moment, this hash is over the host, port, scheme, method, path and content of
the request. Note that headers are not included here - if we do want to include
headers, we would have to do some work to normalize them to remove variations
between user agents, header order, etc. etc.
 2011-02-21 08:47:19 +13:00
Aldo Cortesi
7ddba22f51 Certificates are now generated in a temporary per-session directory. 
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
 2011-02-20 13:29:41 +13:00
Aldo Cortesi
9c5c3c2b1a Implement state loading that doesn't change object identity. 
We need this to let us load state from copied Flows returned from scripts.
 2011-02-20 09:36:13 +13:00
Aldo Cortesi
1549ec8079 Bump test coverage to 100% for flow.py 2011-02-19 17:21:08 +13:00
Aldo Cortesi
5da27a9905 Refactor Flow primitives to remove HTTP1.0 assumption. 
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
 2011-02-19 17:00:24 +13:00
Aldo Cortesi
cd4eea3934 First pass of script hooks for mitmdump. 
Also stub out docs, improve mitmdump error handling.
 2011-02-18 12:40:45 +13:00
Aldo Cortesi
8a9f20b78c Don't leave dangling browser connections for pipelined requests. 2011-02-16 22:37:04 +13:00
Aldo Cortesi
3f0b84bb49 Un-break request replay. 2011-02-16 19:37:40 +13:00
Aldo Cortesi
66349c9783 FlowMaster bugfixes and unit tests. 2011-02-16 16:43:35 +13:00
Aldo Cortesi
27f0765cdd Extract flow-specific Master operations into FlowMaster. 2011-02-16 16:03:22 +13:00
Aldo Cortesi
33dac700fa Switch over to new serialization format. 
Remove BSON from contrib.
 2011-02-16 15:10:00 +13:00
Aldo Cortesi
0c6f846861 First draft of the new serialization mechanism. 2011-02-16 14:33:04 +13:00
Aldo Cortesi
eb15ef7b80 Merge remote branch 'hno/master' 2011-02-06 18:44:35 +13:00
Aldo Cortesi
d0c2d9480c Also serialize ClientConnection and flow backups. 2011-02-06 10:28:43 +13:00
Henrik Nordstrom
b35cbc0760 Fix kill() again, got broken by the change to use None 2011-02-03 23:39:28 +01:00
Aldo Cortesi
2ad4c5adf3 Get rid of ReplayConnection - we now have only one ClientConnection class. 2011-02-04 10:58:08 +13:00
Aldo Cortesi
2cb7429d38 Change "connection" to the less confusing "client_conn" throughout. 2011-02-03 14:51:32 +13:00
Aldo Cortesi
673ff01acc Fix tracking of edited state. 2011-02-03 12:28:40 +13:00
Aldo Cortesi
d8cb826361 Test suite rejiggering and cleanup. 2011-02-03 12:16:03 +13:00
Aldo Cortesi
db99da6af5 Improve script handling. 
- Display output in external viewer when script exits with error.

- Add a "changed" indicator to show if a request can be reverted.
 2011-02-02 11:44:28 +13:00
Aldo Cortesi
405e60215c View script debug output (stderr) in pager. 2011-02-02 10:09:16 +13:00
Aldo Cortesi
b886f808be Add an external script API. 
External scripts can read a flow, modify it, and then return it to mitmproxy
using a simple API.

The "|" keyboard shortcut within mitmproxy prompts the user for a script.
 2011-01-31 13:26:56 +13:00
Aldo Cortesi
152b97fa0b Restructure to make subclassing Flow unnecessary. 2011-01-31 11:44:52 +13:00
Aldo Cortesi
345ac0f2a0 Refine flow saving shortcuts. 
"S" to save all flows.

"s" to save just the current flow.
 2011-01-27 15:03:53 +13:00
Aldo Cortesi
eb93cc22ce Fine-tune threading and fix an exception. 2011-01-27 14:19:48 +13:00
Aldo Cortesi
efe11a0782 Kill deadlock that sometimes occurred on shutdown. 2011-01-27 13:32:24 +13:00
Aldo Cortesi
077272ec97 Switch to BSON for data serialization. 2011-01-27 10:52:42 +13:00
Aldo Cortesi
bd6c0499fb Fix UI sync bugs introduced by flow serialization. 2011-01-26 22:17:42 +13:00
Aldo Cortesi
635f7a971d Add saving and loading of complete flows for later replay and analysis. 2011-01-26 16:50:17 +13:00
Aldo Cortesi
29d8007678 Add serialization hooks to flows and flow component objects. 2011-01-26 14:52:03 +13:00
Aldo Cortesi
7983dbb26a Abstract flow management out of the interactive code. 2011-01-25 15:02:48 +13:00