Grasscutters/mitmproxy
3
0
Fork 0
mirror of https://github.com/Grasscutters/mitmproxy.git synced 2024-11-30 03:14:22 +00:00
Code Issues Projects Releases Wiki Activity
1,336 Commits 1 Branch 67 Tags 50 MiB
d2475e6a14
Commit Graph

996 Commits

Author SHA1 Message Date
Aldo Cortesi
0042687f0d Remove beep functionality from mitmproxy. 
This acts funny (sometimes works, sometimes doesn't) and isn't really useful.
 2011-03-13 11:26:30 +13:00
Aldo Cortesi
c901cbbada mitmproxy prompted input now display previously set value. 
E.g. if you set a limit, then re-enter the limit prompt, you start with the
currently set value.
 2011-03-13 11:24:49 +13:00
Aldo Cortesi
4ce393cc41 Unit tests for cmdline.py 2011-03-12 15:14:25 +13:00
Aldo Cortesi
96b10ca486 console.py 100% test coverage 2011-03-12 15:08:06 +13:00
Aldo Cortesi
fbbc43a5b2 netstring.py 100% unit test coverage. 2011-03-12 14:39:41 +13:00
Aldo Cortesi
2f457e041e Unify mitmproxy and mitmdump commandline 
- Extract common options into cmdline.py
- Change mitmproxy keybindings to fit command line

Some cmdline options and keybindings aren't in operation yet - just stubs
where functionality will be added in the next few commits.
 2011-03-12 14:30:12 +13:00
Aldo Cortesi
40e9067b83 Handle invalid bind address specifications gracefully. 2011-03-12 13:47:37 +13:00
Aldo Cortesi
9f16a84a9e Make mitmdump handle invalid serialized data gracefully. 2011-03-11 15:16:31 +13:00
Aldo Cortesi
7d85db0da3 Display Errors and killed connections in mitmdump. 2011-03-11 13:06:51 +13:00
Aldo Cortesi
daa9653ebe Add --norefresh to stop refreshing server playback to mitmdump. 
Also, make cookie parsing for refreshing more error-tolerant.
 2011-03-11 11:56:10 +13:00
Aldo Cortesi
e99b1d1949 Stub out refresh for server-side replay. 2011-03-09 20:05:30 +13:00
Aldo Cortesi
03f1345385 Add an --anticache option to mitmdump. 
This removes all headers that might cause a server to return 304-not-modified.

For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
 2011-03-09 13:15:31 +13:00
Aldo Cortesi
592812467d Move version constants to version.py, add version to serialized flows. 
For now, we don't do anything with the serialized version info, but this should
give us some future-proofing for serialization format changes.
 2011-03-09 12:36:14 +13:00
Aldo Cortesi
897c4bfc52 Think harder about timestamps. Just save seconds since the epoch as a float. 2011-03-09 12:18:08 +13:00
Aldo Cortesi
765871bd11 Store timestamps on flow components as a UTC time tuple. 
Format is:

(tm_year,tm_mon,tm_mday,tm_hour,tm_min, tm_sec,tm_wday,tm_yday,tm_isdst)
 2011-03-07 13:46:02 +13:00
Aldo Cortesi
7330f5fccf Make mitmdump server playback also exit by default. 
Like client playback, the --keepserving option makes mitmdump keep serving.
 2011-03-06 17:08:56 +13:00
Aldo Cortesi
ec00b5a66e Make mitmdump exit after client replay is complete by default. 
Add an option --keepserving to make it keep serving after replay.
 2011-03-06 16:54:49 +13:00
Aldo Cortesi
e794cbc0d8 Add flow_count and active_flow_count methods to flow.State. 
Use these in unit tests where sensible.
 2011-03-06 16:11:45 +13:00
Aldo Cortesi
42ffded626 Bump unit tests, rearrange mitmdump command-line options slightly. 2011-03-06 16:02:28 +13:00
Aldo Cortesi
4f02480482 Allow client replays to be replayed against server replays. 
Useful for testing.
 2011-03-06 14:48:16 +13:00
Aldo Cortesi
bad77f3470 Add client replay options to mitmdump. 2011-03-06 11:21:31 +13:00
Aldo Cortesi
5da4099ddf Massage namespace to make room for client replay. 
Mostly replay -> server_replay
 2011-03-05 13:03:26 +13:00
Aldo Cortesi
96471fde1d Basic client playback state structure. 2011-03-04 13:08:43 +13:00
Aldo Cortesi
6921b9ff2a Add an indicator that sticky cookies have been applied in mitmdump. 2011-02-25 21:23:44 +13:00
Aldo Cortesi
8cade9fbbf Move stringification funcs from proxy to dump. 2011-02-25 21:11:44 +13:00
Aldo Cortesi
3792b0084e Add -c and -C options to mitmdump to control sticky cookies. 
It's dumb that this needs to be two different options, but optparse doesn't
support optional arguments. It would be much nicer to just have "-c" for "all",
and "-c filter" for a specified filter.
 2011-02-25 17:32:03 +13:00
Aldo Cortesi
57f96c5fe0 Minor housekeeping - unused code, nocover directives. 
95% test coverage.
 2011-02-24 15:44:08 +13:00
Aldo Cortesi
79039eb5d2 More mature sticky cookie primitive. Use it in console.py. 2011-02-24 15:26:34 +13:00
Aldo Cortesi
57947b328e Start abstracting out sticky cookie state. 2011-02-24 10:33:39 +13:00
Aldo Cortesi
3c1db00ebb Remove first iteration playback/record. 
Bonus: unit test coverage goes from 70% to 94% with one commit. ;)
 2011-02-23 12:40:30 +13:00
Aldo Cortesi
39207ffdd2 Add a way for users to specify header significance in server replay. 
Also add the --rheader command-line option to mitmdump to let the user specify
an arbitrary number of significant headers. The default is to treat no headers
as significant.
 2011-02-23 10:54:51 +13:00
Aldo Cortesi
c80214ba55 Minor housekeeping: bump version, clean todo. 2011-02-21 12:27:01 +13:00
Aldo Cortesi
7bc913c40d Move script hooks into the flow primitives. 
This lets handle scripts in corner cases like playback situations more easily.
 2011-02-21 11:40:49 +13:00
Aldo Cortesi
fe99871df8 Add --kill option to mitmdump 
If this option is passed all requests that are not part of a replayed
conversation are killed. If the option is not passed, such requests are passed
through to the server as usual.
 2011-02-21 11:08:35 +13:00
Aldo Cortesi
c3e3897071 Fix a subtle Unicode problem in Response.assemble 
If msg is Unicode, the proto string is automatically promoted to Unicode.  If
the proto string is promoted to Unicode, then the FMT interpolation is also
done in Unicode. If this happens, then binary data in content will cause an
exception.
 2011-02-21 10:11:50 +13:00
Aldo Cortesi
fd4dd8cb6b First pass of playback function for mitmdump. 2011-02-21 09:54:39 +13:00
Aldo Cortesi
deb79a9c5a Add a simple server playback state object. 
We use a loose hash to match incoming requests with recorded flows. At the
moment, this hash is over the host, port, scheme, method, path and content of
the request. Note that headers are not included here - if we do want to include
headers, we would have to do some work to normalize them to remove variations
between user agents, header order, etc. etc.
 2011-02-21 08:47:19 +13:00
Aldo Cortesi
aa16194518 Clean up and strip down netstrings module. 2011-02-20 14:03:32 +13:00
Aldo Cortesi
7ddba22f51 Certificates are now generated in a temporary per-session directory. 
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
 2011-02-20 13:29:41 +13:00
Aldo Cortesi
c2ae8285f4 Revamp SSL configuration. 
- Move option parsing utiliities to proxy.py

- Don't have a global config object. Pass it as an argument to ProxyServer.

- Simplify certificate generation logic.
 2011-02-20 12:53:42 +13:00
Aldo Cortesi
4fc807cedd Clean up certificate generation. 
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.

- Split CA and cert generation into separate functions.

- Generation methods provide an error return when generation fails.

- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
 2011-02-20 12:17:10 +13:00
Aldo Cortesi
9c5c3c2b1a Implement state loading that doesn't change object identity. 
We need this to let us load state from copied Flows returned from scripts.
 2011-02-20 09:36:13 +13:00
Aldo Cortesi
58fc0041fa Stub out doc structure, add screenshots for configuring certs in Firefox. 2011-02-19 19:43:44 +13:00
Aldo Cortesi
1549ec8079 Bump test coverage to 100% for flow.py 2011-02-19 17:21:08 +13:00
Aldo Cortesi
d9c6be3d13 Fix serialization when a Request has no associated client connection. 2011-02-19 17:03:44 +13:00
Aldo Cortesi
5da27a9905 Refactor Flow primitives to remove HTTP1.0 assumption. 
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
 2011-02-19 17:00:24 +13:00
Aldo Cortesi
cd4eea3934 First pass of script hooks for mitmdump. 
Also stub out docs, improve mitmdump error handling.
 2011-02-18 12:40:45 +13:00
Aldo Cortesi
7769e5a898 Whitespace in mitmdump output. 2011-02-17 15:56:54 +13:00
Aldo Cortesi
afa9104005 Refine verbose mitmdump output. 2011-02-17 15:10:50 +13:00
Aldo Cortesi
cbe6982c6d Improve mitmdump text output. 2011-02-17 14:26:50 +13:00
Aldo Cortesi
ca4df062bf Whitepsace and extraneous code. 2011-02-17 11:45:12 +13:00
Aldo Cortesi
f028dc7e1b Add filtering to mitmdump. 2011-02-17 10:44:08 +13:00
Aldo Cortesi
f0f1fb4b55 Add file writing to mitmdump. 2011-02-17 10:18:38 +13:00
Henrik Nordstrom
7758385ac1 Extract common SSL certificate option processing 2011-02-16 15:50:44 +01:00
Aldo Cortesi
d9374ff97b Extract common SSL certificate options into a group. 
Use this only in mitmdump and mitmproxy for now.
 2011-02-16 23:09:42 +13:00
Aldo Cortesi
f5511350eb Tweaks and cleanups for mitmdump. 2011-02-16 23:03:46 +13:00
Aldo Cortesi
8a9f20b78c Don't leave dangling browser connections for pipelined requests. 2011-02-16 22:37:04 +13:00
Aldo Cortesi
0dd1aa7cc7 Initial port of mitmdump to Flows. 2011-02-16 22:10:24 +13:00
Aldo Cortesi
692556cf20 Fix minor display issues in console app. 2011-02-16 21:03:55 +13:00
Aldo Cortesi
3f0b84bb49 Un-break request replay. 2011-02-16 19:37:40 +13:00
Aldo Cortesi
f009770d4c Fix a bug in HTTP 1.1 pipelining that caused Requests to be over-written. 
We use the ClientConnection object to tie requests, responses and errors
together. This is an HTTP 1.0 assumption, but we can fix it by just making
copies of the connection object when we handle multiple requests.
 2011-02-16 19:22:19 +13:00
Aldo Cortesi
66349c9783 FlowMaster bugfixes and unit tests. 2011-02-16 16:43:35 +13:00
Aldo Cortesi
27f0765cdd Extract flow-specific Master operations into FlowMaster. 2011-02-16 16:03:22 +13:00
Aldo Cortesi
33dac700fa Switch over to new serialization format. 
Remove BSON from contrib.
 2011-02-16 15:10:00 +13:00
Aldo Cortesi
0c6f846861 First draft of the new serialization mechanism. 2011-02-16 14:33:04 +13:00
Aldo Cortesi
5692c7359c Import Will McGugan's netstring module. 
Module is in the Public Domain.

I expect to modify and extend this module, so I've imported into main library
rather than contrib. Code has been reformatted to suite our code standard,
tests have been extrated into /tests directory.
 2011-02-16 11:21:06 +13:00
Henrik Nordstrom
89627a702a drop unused protocol parameter from Response 2011-02-15 22:24:38 +01:00
Aldo Cortesi
759496e24e Repair unit test suite. 2011-02-11 11:06:30 +13:00
Henrik Nordstrom
51cdd7b95d Bump version 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
32adee8743 Implement a dummy CA 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
4ffaadd435 Allow specifying the accepted ciphersuites 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
061cea89da Import cache store control into console controller 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
d11dd742d8 Simple record & playback functionality 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
4bae297fbb Basic HTTP/1.1 Support 
Adds support for chunked transfer encoding, and a couple other minor
protocol corrections.

Improve HTTP support

- Support intercepted requests with Host header
- Support HEAD requests proper
- Support any HTTP method including extensions, not just a couple known ones

Support expect: 100-continue and 100 Continue messages

Persistent client connections

Generalize ServerConnection a bit in preparation for keep-alive support

Correct HTTP status codes on errors forwarding the request
 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
0613321aef Optimize CONNECT responses, sent in a single packet 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
4cf3392e50 Sort header names for a predictable result 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
bd01126b2b Move try_del to utils 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
2d8cfbafc3 Automatically update response message when editing response code 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
98b1acc51d Add response creation to edit function on intercepted requests 2011-02-10 02:59:51 +01:00
Henrik Nordstrom
fcc39e1aaf Terminate workers when main thread terminates 2011-02-10 02:59:51 +01:00
Aldo Cortesi
eb15ef7b80 Merge remote branch 'hno/master' 2011-02-06 18:44:35 +13:00
Aldo Cortesi
a4eaafab5a Further fine-tuning for pretty_xmlish. 2011-02-06 16:56:13 +13:00
Aldo Cortesi
7156d1a73a Rip out BeautifulSoup, and use a custom XML-ish prettyprinter. 2011-02-06 14:17:30 +13:00
Aldo Cortesi
44dc3a052e Add option to tell mitmproxy which interfaces to bind to. 2011-02-06 10:52:54 +13:00
Aldo Cortesi
d0c2d9480c Also serialize ClientConnection and flow backups. 2011-02-06 10:28:43 +13:00
Henrik Nordstrom
b35cbc0760 Fix kill() again, got broken by the change to use None 2011-02-03 23:39:28 +01:00
Henrik Nordstrom
5bf6482bd0 Merge remote-tracking branch 'origin/master' 2011-02-03 23:30:03 +01:00
Henrik Nordstrom
2f813fa748 Messages may need to stay in the controller for considerable 
amount of time. Don't use a timeout waiting for the controller response.
Example where this happens is intercepted flows.
 2011-02-03 23:28:24 +01:00
Aldo Cortesi
2ad4c5adf3 Get rid of ReplayConnection - we now have only one ClientConnection class. 2011-02-04 10:58:08 +13:00
Aldo Cortesi
2cb7429d38 Change "connection" to the less confusing "client_conn" throughout. 2011-02-03 14:51:32 +13:00
Aldo Cortesi
7924f07971 Fine-tune completion a bit. 2011-02-03 13:50:57 +13:00
Aldo Cortesi
6c89749f0a Add timestamps to flows. 
For now, these are only displayed on the connection view screen, with second
granularity.
 2011-02-03 13:30:47 +13:00
Aldo Cortesi
673ff01acc Fix tracking of edited state. 2011-02-03 12:28:40 +13:00
Aldo Cortesi
f6c8654f08 "Q" quits without confirmation prompt. 2011-02-03 12:20:05 +13:00
Aldo Cortesi
d8cb826361 Test suite rejiggering and cleanup. 2011-02-03 12:16:03 +13:00
Aldo Cortesi
db99da6af5 Improve script handling. 
- Display output in external viewer when script exits with error.

- Add a "changed" indicator to show if a request can be reverted.
 2011-02-02 11:44:28 +13:00
Aldo Cortesi
df9d8832fb Use last path specified as default prompt. 2011-02-02 10:19:54 +13:00
Aldo Cortesi
405e60215c View script debug output (stderr) in pager. 2011-02-02 10:09:16 +13:00
Aldo Cortesi
b886f808be Add an external script API. 
External scripts can read a flow, modify it, and then return it to mitmproxy
using a simple API.

The "|" keyboard shortcut within mitmproxy prompts the user for a script.
 2011-01-31 13:26:56 +13:00
Aldo Cortesi
152b97fa0b Restructure to make subclassing Flow unnecessary. 2011-01-31 11:44:52 +13:00
Aldo Cortesi
1619d164ff Re-jigger flow view shortcuts so we don't over-ride global shortcuts. 2011-01-31 10:57:33 +13:00
Aldo Cortesi
74aa622fdd Better help page statusbar text. 2011-01-30 15:31:44 +13:00
Aldo Cortesi
cf165bc32e Next and previous flow shortcuts from within flow view. 2011-01-30 15:26:41 +13:00
Aldo Cortesi
2a39387d27 UI tweaks 
- Make flow view state persistent (request/response, body view mode).
- Don't exit flow view mode when viewing help.
 2011-01-30 14:52:51 +13:00
Aldo Cortesi
091db9cdf2 Prompt user on quit. 2011-01-28 12:55:02 +13:00
Aldo Cortesi
0742bc4fd0 Nicer statusbar messages, with timed expiry. 2011-01-28 12:18:01 +13:00
Aldo Cortesi
dfefe3cdda Make pretty-printing more robust. 
Also, since BeautifulSoup is so damn slow, print a statusbar message saying
that we're calculating a pretty version of the response. Maybe I should add
hangman or something, becuase on a 200k document this can take ages.
 2011-01-28 12:08:25 +13:00
Aldo Cortesi
93ef691bad Basix XML/HTML pretty-printing in flow viewer. 2011-01-27 17:26:01 +13:00
Aldo Cortesi
96e714a74c Handle nonexistent load files gracefully. 
Also improve IOError statusbar messages.
 2011-01-27 15:20:07 +13:00
Aldo Cortesi
edab82f7d8 Add command to write request or response body to file. 2011-01-27 15:16:03 +13:00
Aldo Cortesi
345ac0f2a0 Refine flow saving shortcuts. 
"S" to save all flows.

"s" to save just the current flow.
 2011-01-27 15:03:53 +13:00
Aldo Cortesi
eb93cc22ce Fine-tune threading and fix an exception. 2011-01-27 14:19:48 +13:00
Aldo Cortesi
efe11a0782 Kill deadlock that sometimes occurred on shutdown. 2011-01-27 13:32:24 +13:00
Aldo Cortesi
077272ec97 Switch to BSON for data serialization. 2011-01-27 10:52:42 +13:00
Aldo Cortesi
460107589c Add tab completion for save and load path specs. 2011-01-27 10:29:37 +13:00
Aldo Cortesi
bd6c0499fb Fix UI sync bugs introduced by flow serialization. 2011-01-26 22:17:42 +13:00
Aldo Cortesi
635f7a971d Add saving and loading of complete flows for later replay and analysis. 2011-01-26 16:50:17 +13:00
Aldo Cortesi
29d8007678 Add serialization hooks to flows and flow component objects. 2011-01-26 14:52:03 +13:00
Aldo Cortesi
7983dbb26a Abstract flow management out of the interactive code. 2011-01-25 15:02:48 +13:00
Aldo Cortesi
b8d8030abd Documentation updates. 2011-01-25 10:59:50 +13:00
Aldo Cortesi
324c175d02 Fix indentation in previous patch. 2010-09-06 11:21:51 +12:00
Henrik Nordström
2e7e9f9759 Wrap read request, concatenating partial reads until whole request have been read 2010-09-06 07:18:11 +08:00
Aldo Cortesi
f457342aab Add mitmproxy command line flags for a number of filter expressions. 
For now, these are supported: intercept, limit, sticky cookies, beep
 2010-03-01 15:42:38 +13:00
Aldo Cortesi
fbe6e0a820 Housekeeping: todo updates, move put handler_ methods together in console.py. 2010-03-01 15:08:39 +13:00
Aldo Cortesi
ba92d22e1d Put DumpMaster in its own file. 
It's going to become a more important part of the mitmproxy suite now.
 2010-03-01 14:58:04 +13:00
Aldo Cortesi
0188cf8a1a Just make using the user's terminal background the default. 
This just makes sense. Later on, we'll have a config file in which users can
set the palette, so that they can over-ride the default background if they
really want.
 2010-03-01 14:50:35 +13:00
Aldo Cortesi
6300a57212 Don't intercept replays - it makes no sense to do so. 2010-03-01 14:25:37 +13:00
Aldo Cortesi
639b1557a6 Don't show error if user tries to save, but specifies no path. 2010-03-01 14:20:12 +13:00
Aldo Cortesi
f030718e32 Add "A" key, to accept all intercepted connections. 2010-03-01 14:08:10 +13:00
Aldo Cortesi
a7cea1749b Make beeping more general. 
- The "B" key now sets a filter pattern. Whenever the filter matches, the
  terminal beeps.
- The beep specification is now stored in the State object, where I want to
  keep all mutable state.
- I've removed the startup flags for beeping for now. We probably want to let
  users specify a number of options on startup, including interception patterns
  and limits. When we do this, we should also separate out the dumping and
  curses programs, because many of these options will only make sense in one of
  the two.
- This isn't an exact replacement for the beep functionality I ripped out,
  because specifying beeping on intercept is slightly cumbersome (you need to
  re-specify the intercept pattern using B). Sorry - we'll add a better way
  soon.
 2010-03-01 13:56:28 +13:00
Aldo Cortesi
a40775b846 Maintain compatibility with 0.9.8.x of urwid. 
This is the most common deployed version, so it's worth doing a bit of extra
work to make sure mitmproxy runs smoothly under it.
 2010-03-01 13:37:50 +13:00
Thomas Roth
72f8111f2b Show error if compiling of an expression fails. 2010-02-26 17:48:55 +01:00
Thomas Roth
a5f87657e1 Added beeping. --bireq and --bires enables beeping on intercepted requests/responses. 2010-02-26 14:01:38 +01:00
Aldo Cortesi
c5eb3c592d Fix hang when a POST is made with a 0 content length. 2010-02-26 20:07:37 +13:00
Thomas Roth
0e76f0f341 Added a 'keep terminal background' option (-t), so that you can use it with your own background color. (Foreground color following.) 2010-02-26 00:20:49 +01:00
Thomas Roth
fc6f170731 Huge speed improvements if more than one request is in the queue. Speeds up every request by 0.1 seconds. 2010-02-25 17:49:21 +01:00
Thomas Roth
1909744631 Small improvement in regular expression compiler exception handling. 2010-02-25 17:45:37 +01:00
Thomas Roth
000196fe21 Added exception if starting editor fails and added vi as default editor. 2010-02-25 12:14:47 +01:00
Thomas Roth
6d41b27f53 Fixes crash on illegal regular expression for interception. 2010-02-24 23:59:54 +01:00
Thomas Roth
0fc864c419 Fixes a bug which caused a crash if you've tried to accept a connection which wasn't intercepted. 2010-02-24 23:32:56 +01:00
Thomas Roth
79d2abb23a Fixes blocking problem with a workaround which sets a timeout of 0.1 seconds. 2010-02-24 23:25:37 +01:00
Thomas Roth
70b0bddc85 Additional fix for compatibility with Urwid >0.9.9 2010-02-24 18:52:26 +01:00
Thomas Roth
22b77e32cd Fixes compatibility with Urwid >0.9.9 2010-02-24 18:25:24 +01:00
Aldo Cortesi
8eb8814387 Add some error handling for connection saving. 2010-02-24 16:47:34 +13:00
Aldo Cortesi
146a8c6d7c Save a request or response to file. 
Ideally, we'd have a nice selection widget with tab completion and the like.
Maybe later...
 2010-02-24 16:30:00 +13:00
Aldo Cortesi
cb0e328709 Initial checkin. 2010-02-16 17:09:07 +13:00