Grasscutters/mitmproxy
3
0
Fork 0
mirror of https://github.com/Grasscutters/mitmproxy.git synced 2024-11-24 09:15:55 +00:00
Code Issues Projects Releases Wiki Activity
9,157 Commits 1 Branch 67 Tags 50 MiB
9409bf0368
Commit Graph

2291 Commits

Author SHA1 Message Date
Aldo Cortesi
7c32d4ea2a flow.state -> addons.state 2016-10-19 14:48:42 +13:00
Aldo Cortesi
22eebfd574 addons.Addons -> addonmanager, builtins -> addons 2016-10-19 14:39:39 +13:00
Aldo Cortesi
966418725b controller.Log -> log.Log 2016-10-19 14:14:59 +13:00
Aldo Cortesi
bce387a5a0 Kill flow.master - create master.Master 
Also extract events into .events
 2016-10-19 13:22:50 +13:00
Aldo Cortesi
264a09e778 Let's dispense with this fiction that we support multiple servers 
It's simple enough to re-add if needed, and until then is just cruft.
 2016-10-19 12:37:32 +13:00
Aldo Cortesi
87629586ae web app cleanups: tests and examples 2016-10-19 11:48:51 +13:00
Aldo Cortesi
ceb8caee98 Web apps to addons 
This commit removes the app registry, adds a wsgiapp addon base, and ports the
onboarding app to it.
 2016-10-19 11:40:51 +13:00
Aldo Cortesi
03cb5bb325 No need for all builtins to be loaded for server tests 2016-10-18 22:30:21 +13:00
Aldo Cortesi
466f5e56a1 Convert flow.state to an addon 
A first step, and already many things are much nicer.
 2016-10-18 21:51:36 +13:00
Aldo Cortesi
d5056c5627 accept_addon -> resume, intercept/resume events 2016-10-18 12:02:35 +13:00
Aldo Cortesi
825b02d495 master.duplicate_flow -> state.duplicate_flow 
This is a pure operation on state, and doesn't belong on master.

Part of prep to move state to an addon.
 2016-10-18 11:29:51 +13:00
Aldo Cortesi
b25d94a6ac Response body streaming to addon 2016-10-18 10:42:05 +13:00
Maximilian Hils
beed406058 fix tests 2016-10-17 03:37:42 -07:00
Thomas Kriechbaumer
94c9dd66e6 remove empty lines at beginning of file 2016-10-17 10:32:12 +02:00
Maximilian Hils
758860531a fix socks5 mode with tls 2016-10-16 23:05:51 -07:00
Aldo Cortesi
c774a9fec9 python3: clean up super and __future__ 2016-10-17 18:03:07 +13:00
Aldo Cortesi
a647b30365 python3: clean up class brackets 2016-10-17 17:37:08 +13:00
Aldo Cortesi
fb22f2ff4f Zap object base class 2016-10-17 17:37:08 +13:00
Aldo Cortesi
ce98a9219e test & examples: zap six 2016-10-17 16:45:45 +13:00
Aldo Cortesi
9a0195bf64 scripts: keep scripts just after the ScriptLoader in addon chain 
We need scripts to run _before_ filestreamer, so we can't just add them to the
end of the chain. This patch also fixes an issue that could cause scripts to be
initialised un-necessarily if only the order of scripts in options changed.
 2016-10-16 20:26:06 +13:00
Aldo Cortesi
fb69c9c345 docs: overview, classes, arguments 2016-10-16 20:26:06 +13:00
Maximilian Hils
e9284ab7ee add regression test for #1605 2016-10-14 01:22:30 -07:00
Aldo Cortesi
89d36713e3 certutils: cap the cert store size at 100 by default 
This should be enough to give us reuse without growing infinitely. This is part
of fixing the memory situation in mitmdump.

TODO: There's an opportunity here for a better algorithm, that expires certs
based on least-recently-accessed time, rather than oldest generated time.
 2016-10-05 10:44:31 +11:00
Thomas Kriechbaumer
90898f9084 use flt instead of filt for flowfilter expressions 2016-10-04 14:51:04 +02:00
Aldo Cortesi
882c363ec5 debug: Hard exit after debug signal 2016-10-04 16:48:13 +11:00
Aldo Cortesi
7fe09f4d26 Consolidate logging 
- Make log event handler work
- Move terminal event log into an addon
- Clean various log related errors and duplications
 2016-10-04 13:00:15 +11:00
Aldo Cortesi
2e48b60ff1 Clarify channel interaction pattern 
Before this patch, it wasn't clear when the arguments to an event could be
modified, and when a new object should be returned. After this patch, the right
thing to do is to modify the arguments in 99% of cases. The only exception to
this is root_layer, for which we can find a better structure down the track.
 2016-10-04 08:13:56 +11:00
Aldo Cortesi
1ae73b9157 Add requestheaders event 
HTTP1 only for now, HTTP2 coming next.
 2016-10-04 08:13:56 +11:00
Thomas Kriechbaumer
36c04f1631 fix flowfilter.match args 2016-10-03 12:04:17 +02:00
Thomas Kriechbaumer
bb60b76af4 use flowfilter.match 2016-10-03 11:45:54 +02:00
Thomas Kriechbaumer
ba84248910 rename mitmproxy.filt -> mitmproxy.flowfilter 2016-10-03 11:45:54 +02:00
Maximilian Hils
611dad12a7 Revert "Add API to programmatically create new requests (#1534)" 
This reverts commit ccbdcd684b,
which yielded a test timeout.
 2016-10-02 23:08:51 -07:00
Tyler St. Onge
ccbdcd684b Add API to programmatically create new requests (#1534) 2016-10-01 13:06:59 +02:00
Aldo Cortesi
070aa27cf5 parse_set_cookie header returns an empty list if no cookies are found 
This matches parse_cookie, and is more idiomatic.
 2016-09-28 15:10:31 +10:00
Aldo Cortesi
b21f076cc8 Merge branch 'multi-cookie' of https://github.com/dufferzafar/mitmproxy 2016-09-28 12:44:40 +10:00
Shadab Zafar
5383c15ff2 Add tests for comma separated Set-Cookies 2016-09-27 21:28:54 +05:30
Ryan Laughlin
ad6d675b20 In reverse proxy mode, always send SNI 2016-09-27 10:49:04 -04:00
Shadab Zafar
e8ffc5167b Can't have , in the sticky cookie test 
Since we support comma separated cookies now
 2016-09-27 16:44:09 +05:30
Shadab Zafar
9ec0790271 Make needed changes in tests 2016-09-27 16:44:09 +05:30
Shadab Zafar
85bf5e18f6 Update cookie tests 
This is needed since _read_pairs now returns a list of cookies each of which
is a list of [name, value] tuples.
 2016-09-27 16:44:08 +05:30
Maximilian Hils
eeec17902f update dependencies, minor fixes (#1577) 2016-09-24 21:01:48 -07:00
Maximilian Hils
0a643ad20f remove option argument for addons.add (#1576) 2016-09-24 18:21:12 -07:00
Maximilian Hils
9142da1a7d Merge pull request #1566 from mhils/issue-1546 
add websocket on/off switch, improve logging (fix #1546, fix #1547)
 2016-09-22 03:06:39 -07:00
Maximilian Hils
77868434e7 fix tests 2016-09-22 02:10:02 -07:00
Maximilian Hils
a3329c5a80 scripts: cut traceback properly, fix #1467 2016-09-22 02:03:12 -07:00
Maximilian Hils
9e0b935fa2 Merge pull request #1564 from mhils/issue-1554 
Fix Response.make content-length header
 2016-09-21 21:49:14 -07:00
Maximilian Hils
0b532789a0 fix Response.make content-length header 2016-09-21 20:34:01 -07:00
Maximilian Hils
770936f1f9 raise TypeError on invalid header assignment, fix #1562 2016-09-21 19:21:32 -07:00
Aldo Cortesi
cfb943bfdd Redesign client replay 
- Move to an addon
- Use a much simpler synchronisation mechanism
 2016-09-11 20:00:34 +12:00
Aldo Cortesi
6b9221ffeb Add "run_once" support to the script addon, use it in mitmproxy 
Fixes #1418
 2016-09-11 13:22:01 +12:00
Aldo Cortesi
9b814ca067 options.replay_ignore* -> options.server_replay_ignore* 2016-09-10 10:21:52 +12:00
Aldo Cortesi
8677e91074 options.nopop -> options.server_replay_nopop 2016-09-10 10:09:07 +12:00
Aldo Cortesi
5ebe8c1bc8 options.rheaders -> options.server_replay_use_headers 2016-09-10 10:09:07 +12:00
Aldo Cortesi
b0213a2c0e options.kill -> options.replay_kill_extra 2016-09-10 10:09:07 +12:00
Aldo Cortesi
4ff8a72521 Roll out synchronisation for mitmproxy tests 
This extends some of the work I did for pathod and netlib to the mitmproxy test
suite. It also fixes what may be a leak in replays.

Failing on connection leak is disabled on Windows for the moment.

Fixes #1535
 2016-09-10 09:18:11 +12:00
Aldo Cortesi
ea49b8a2e2 Merge pull request #1532 from cortesi/playback 
Playback and fix construct breakage
 2016-09-07 12:59:11 +12:00
Schamper
5728a1c900 Make 'none' synonymous to 'identity' 2016-09-06 17:00:08 +02:00
Aldo Cortesi
9abdd3a8d9 Move server playback in to an addon 
- Move server playback into an addon
- Implement a better sync strategy to decide when to exit if keepserving is
off. We now wait for the final flow played back to no longer be live.
- Leave interactive server playback in mitmproxy console broken for now - there
are broader addon-related changes that need to be made for that, and this patch
is already big.

Fixes #1229
 2016-09-06 23:48:02 +12:00
Schamper
85e1539d0a Add test 2016-09-06 11:07:50 +02:00
Thomas Kriechbaumer
c0b12da401 change test value 2016-09-04 09:31:45 +02:00
Thomas Kriechbaumer
4c7f99d2be improve netlib.encoding 2016-09-04 09:31:44 +02:00
Thomas Kriechbaumer
69b770469e http2: assemble connect-request from object 2016-09-03 12:31:35 +02:00
Thomas Kriechbaumer
ea2a51d3be http2: fixes 2016-09-03 11:50:37 +02:00
Thomas Kriechbaumer
5dda9505b6 http2: improve framereader 2016-09-03 10:00:31 +02:00
Thomas Kriechbaumer
0badb3ec91 cleanup 2016-09-03 10:00:31 +02:00
Thomas Kriechbaumer
0d0c2c788d cleanup tests 2016-09-01 10:12:13 +02:00
Thomas Kriechbaumer
e5b0dae7e9 add websockets support to mitmproxy 2016-09-01 09:57:18 +02:00
Thomas Kriechbaumer
d12515f84b websockets: refactor implementation and add tests 2016-09-01 09:56:14 +02:00
arjun23496
a8deed1f4e Lint fix - minor 2016-08-31 16:39:04 +05:30
arjun23496
2476809ff8 Lint fix - replace tests 2016-08-31 16:27:22 +05:30
arjun23496
d2cfa5ca41 Added tests for replace in request,message,header 2016-08-31 10:23:33 +05:30
Manish Kumar
2cf4efab07 fixing lint error 2016-08-26 14:22:02 +05:30
Manish Kumar
16401d5be8 fixed : Divide by zero error came when string was empty, also a test 2016-08-26 12:52:51 +05:30
Thomas Kriechbaumer
51771c01c6 allow test classes to dynamically specify ssl context 2016-08-24 09:19:42 +02:00
Maximilian Hils
f27028f58e introduce Response.make for simple response creation 2016-08-23 00:17:06 -07:00
Thomas Kriechbaumer
eb83107e1c reenable options test 2016-08-16 11:05:57 +02:00
Thomas Kriechbaumer
cd35a6ff47 add regression test for #1476 2016-08-16 11:05:57 +02:00
Thomas Kriechbaumer
bfe22e739c Merge pull request #1481 from Kriechi/http2-simplify-zombies 
http2: simplify zombie detection
 2016-08-16 10:10:39 +02:00
Thomas Kriechbaumer
17c65e46cb Merge pull request #1480 from mhils/reverse-proxy-auth 
Add basic auth support for reverse proxy mode
 2016-08-16 10:07:59 +02:00
Shadab Zafar
239eaf0078 Remove timezone specific value 2016-08-15 15:05:32 +05:30
Shadab Zafar
bc175b58f2 Use test times less than 2038 
So that tests pass on 32bit systems too
 2016-08-15 12:00:23 +05:30
Shadab Zafar
ec0bae35c5 Assert cookie groups explicitly rather than just the length 2016-08-15 12:00:23 +05:30
Shadab Zafar
b9426fcec1 Add a test for base64 encoding 2016-08-15 12:00:23 +05:30
Shadab Zafar
988174246b Add some cookie formatting related tests 2016-08-15 12:00:23 +05:30
Shadab Zafar
456f80d862 Open JSON file in text mode 2016-08-15 12:00:23 +05:30
Shadab Zafar
03e6117042 Add a function to get cookie expiration time 2016-08-15 12:00:23 +05:30
Shadab Zafar
a479c51465 Add method to group pairs by cookies 2016-08-15 12:00:23 +05:30
Shadab Zafar
99b32094d8 Add simple har_dump test 2016-08-15 12:00:23 +05:30
Thomas Kriechbaumer
8f4baf42ea http2: cleanup 2016-08-14 13:06:33 +02:00
Maximilian Hils
a9401472cd add basic auth support for reverse proxy mode 2016-08-13 11:04:53 -07:00
Maximilian Hils
5a22496ee8 clean up code, improve DummyReply 2016-08-09 22:29:07 -07:00
Maximilian Hils
818840f553 finalize Reply semantics, fix tests 2016-08-09 20:26:24 -07:00
Maximilian Hils
dcfa7027ae fix tests 2016-08-04 15:39:48 -07:00
Maximilian Hils
951885a5dd simplify contentview logic 2016-08-04 15:23:44 -07:00
Maximilian Hils
7ea9726e5d Merge pull request #1453 from dufferzafar/limit-to-filter 
Rename "Limit" feature to "Filter View"
 2016-07-31 23:13:31 -07:00
Aldo Cortesi
ca0eab0f05 Merge pull request #1447 from mitmproxy/secure-by-default 
Verify upstream certificates by default
 2016-08-01 14:44:16 +12:00
Shadab Zafar
10ad56c853 Rename "Limit" feature to "Filter View" 2016-07-31 16:54:09 +05:30
Maximilian Hils
ca2e338876 Merge pull request #1449 from mhils/fix-1448 
Add escape_single_quotes=False arg to bytes_to_escaped_str
 2016-07-30 15:56:55 -07:00
Angelo Agatino Nicolosi
6792ec4058 Integrated encode/decoder for brotli 2016-07-30 14:43:53 +02:00
Maximilian Hils
4534363671 add escape_single_quotes=False arg to bytes_to_escaped_str 2016-07-29 19:54:44 -07:00
dufferzafar
1cffa5f46b Use replace while decoding 2016-07-28 07:48:10 -07:00
Maximilian Hils
f54b302a59 fix cert verification tests, remove leftovers 2016-07-27 22:44:06 -07:00
Maximilian Hils
8b325fd65a improve invalid certificate ux 2016-07-27 21:01:28 -07:00
Sachin Kelkar
17fdb841f0 verify upstream certificates by default (#1111) 
squashed and merged by @mhils
 2016-07-27 17:57:38 -07:00
Maximilian Hils
013288f7ca add quote/unquote tests 2016-07-24 19:38:44 -07:00
Maximilian Hils
e07f515a20 remove content caching in netlib.http.Message 2016-07-23 19:05:10 -07:00
Maximilian Hils
d6209fa03b add tests for single-element cache 2016-07-23 19:04:57 -07:00
Maximilian Hils
a87d089561 Merge pull request #1380 from dufferzafar/marked-filter 
Marked property and filter
 2016-07-23 17:55:12 -07:00
Maximilian Hils
fa45722ed8 Merge pull request #1402 from mhils/console-fixes 
minor fixes
 2016-07-23 12:49:57 -07:00
Maximilian Hils
61de6fa1d6 fix test_view_urlencoded 2016-07-23 11:55:27 -07:00
Thomas Kriechbaumer
514ea1b701 http2: cleanup tests 2016-07-23 11:08:32 +02:00
Shadab Zafar
08000c940c Test marked property while serialization roundtrip 2016-07-23 10:41:57 +05:30
Aldo Cortesi
afda175e1c Addon iface: .configure(options) -> .configure(options, updated) 2016-07-23 15:43:55 +12:00
Aldo Cortesi
b5416895f5 script: convert test recorder to an addon class 2016-07-23 11:38:45 +12:00
Aldo Cortesi
9294d19f90 scripts: .start can now return an Addon instance 2016-07-23 11:29:13 +12:00
Aldo Cortesi
51a8ba57f1 script: add a namespace adaptor to match Addons 2016-07-23 11:03:50 +12:00
Maximilian Hils
65c2f30218 fix mitmproxy 0.11 dumpfile - was 0.18 before 2016-07-22 10:36:26 -07:00
Maximilian Hils
798759d2b3 fix content view cache invalidation 2016-07-21 19:49:32 -07:00
Aldo Cortesi
33ffbc28ed console: share one ActionBar instance 
This fixes an issues where popping a view off the stack caused stale ActionBar
contents to be displayed.
 2016-07-21 15:41:43 +12:00
Aldo Cortesi
6750ab899a Merge branch 'flow-export' of https://github.com/dufferzafar/mitmproxy into dufferzafar-flow-export 2016-07-21 13:53:41 +12:00
Aldo Cortesi
02acfb1242 Fix netlib.utils.is_valid_host 
- Don't crash when passed an empty string. This translated into an actual core
crash, discovered while fuzzing with afl.
- Taking a slice of length one out of bytes returns an integer, so the check
for trailing period in this function never worked on Python3.
- Add unit tests.
 2016-07-21 10:38:37 +12:00
Shadab Zafar
c84d0bd10b Add a url exporter 2016-07-20 10:03:00 +05:30
Shadab Zafar
8f4d49e22a Remove raw_request exporter 
This functionality will be handled by headers+content
 2016-07-20 10:03:00 +05:30
Aldo Cortesi
8317772377 mitproxy.options -> mitmproxy.optmanager, mitmproxy.flow.options -> mitmproxy.options 
It might be slightly  more felicitous to move optmanager into netlib at some
point, especially if we can also use it in pathod.

This also consolidates our constants in mitmproxy.options, removing some duplicates.
 2016-07-20 11:28:53 +12:00
Aldo Cortesi
09fbebf42a Test suite: get_proxy_config -> get_options 2016-07-19 19:36:40 +12:00
Aldo Cortesi
9c9d28d068 Remove proxy.config.process_proxy_options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
ef380917a8 ProxyConf: adapt mitmproxy console, ditch tcp|host filter attrs on flow.master 2016-07-19 16:25:09 +12:00
Aldo Cortesi
be081a24bc ProxyConf: auth_* into Options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
5cfe783b6c ProxyConfig: http2, rawtcp, no_upstream_cert -> Options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
26fa88a338 ProxyConfig: Refactor to move verification mode checks into configure 2016-07-19 16:25:09 +12:00
Aldo Cortesi
98bf544664 Regularize naming of upstream_trusted_ca* options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
0a3839375d ProxyConfig: various SSL options to Options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
78e8159007 config.parse_server_spec shouldn't pass byts to tcp.Address 2016-07-19 16:25:09 +12:00
Aldo Cortesi
f9622074cc ProxyConfig: mode, upstream_server and upstream_auth to Options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
d2268ddb1e ProxyConfig: --body-size-limit to Options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
856e1c2ba9 Move clientcers & cadir to Options 
Also adds a .configure mechanism to ProxyConfig, and hooks it up to the change
event.
 2016-07-19 16:25:09 +12:00
Aldo Cortesi
b1b1a1b9cf ProxyConfig: cadir to options 2016-07-19 16:25:09 +12:00
Aldo Cortesi
6908dc4d90 ProxyConfig refactor 
Step two of frog boiling: move listening address and port into options. This is
the hard bit, because it touches the test suite so deeply.
 2016-07-19 16:25:09 +12:00
Aldo Cortesi
9bc1514aef Unify proxy config and options 
This is step 1 in a gradual "boiling frog" strategy. Decorate ProxyConfig
with an Options object.
 2016-07-19 16:25:09 +12:00
Aldo Cortesi
e9865800a8 Console options cleanup 
Use setters, use togglers, move more things out of master
 2016-07-18 10:31:34 +12:00
Aldo Cortesi
f9e4249499 Add a toggller helper to Options, start using it in mitmproxy console 2016-07-18 10:03:10 +12:00
Thomas Kriechbaumer
7b57c2948e cleanup 2016-07-17 18:17:35 +02:00
Thomas Kriechbaumer
d9021f8505 http2: coverage++ 2016-07-17 18:07:02 +02:00
Aldo Cortesi
216e38d437 setheaders -> addon 2016-07-17 18:41:14 +12:00
Aldo Cortesi
a26cc75aab Merge pull request #1370 from cortesi/filestreamer 
Some work on stream-to-file
 2016-07-17 16:20:41 +12:00
Aldo Cortesi
9bf1d300d8 Some work on stream-to-file 
- stream -> filestreamer throughout
- move active_flows set into FileStreamer addon
 2016-07-17 16:07:29 +12:00
Aldo Cortesi
fb45d59c02 Merge pull request #1367 from Kriechi/fix-1366 
fix-1366
 2016-07-17 15:06:58 +12:00
Aldo Cortesi
08b3af98c2 Merge branch 'master' into dumper 2016-07-17 12:53:14 +12:00
Aldo Cortesi
b0b3b19ad6 Extract console dump functionality into an addon 
This removes all the code that deals with printing flows to screen from dump.py
into a self-contained addon.

- This fixes a bug - by moving dumping into an
addon, we now dump flows AFTER addon transformation, so we can see the changes
made.
- We get dumping "for free" in other places by simply adding the addon. It's
now easy to add dumping to console to mitmweb for debugging and development.
The same goes for external projects that derive from master.
- We also get major benefits in clarity for a frankly hairy part of our
project. Mitmdump is much clearer, and all the hairyness is now isolated for
further refactoring.
 2016-07-17 10:17:02 +12:00
Thomas Kriechbaumer
a32312cecc fix-1366 2016-07-16 18:31:37 +02:00
Maximilian Hils
afc3f6d854 Merge remote-tracking branch 'duffer/tcpflow-filter' 2016-07-16 00:19:01 -07:00
Maximilian Hils
b27d59095d Merge pull request #1306 from mitmproxy/message-body-encoding 
Improve Message Body Encoding
 2016-07-16 00:13:58 -07:00
Maximilian Hils
e6e39ce80f preserve content-type parameter order 2016-07-15 23:46:12 -07:00
Aldo Cortesi
1a1f7e6fd9 replacehooks -> addon 
Also fixes a bug in header replacements in netlib that resulted in a mutable
multidict.
 2016-07-16 18:37:30 +12:00
Maximilian Hils
3602fd7a36 Merge remote-tracking branch 'origin/master' into message-body-encoding 2016-07-15 23:17:57 -07:00
Maximilian Hils
a3c7c84d49 improve message content semantics 2016-07-15 22:50:33 -07:00
Shadab Zafar
fea4435dac Allow DummyFlow to match ~e, ~src and ~dst 2016-07-16 10:15:38 +05:30
Shadab Zafar
1d2ccb9170 Add DummyFlow and its Tests 2016-07-16 10:15:38 +05:30
Shadab Zafar
8c49f0e784 Add tests for TCPFlow.match 2016-07-16 10:15:38 +05:30
Shadab Zafar
6a2668d865 Add ~http and ~tcp filters 2016-07-16 10:15:38 +05:30
Shadab Zafar
262a420553 Use ~bq / ~bs for messages sent from client / server 2016-07-16 10:15:38 +05:30
Shadab Zafar
767457fda9 Add tests for HTTPFlow only filters 2016-07-16 10:15:38 +05:30
Shadab Zafar
317355a9af Add test for operators (& | !) 2016-07-16 10:15:38 +05:30
Shadab Zafar
4e29993c1f Add test for addresses (~src, ~dst) 2016-07-16 10:15:38 +05:30
Shadab Zafar
124b87b851 Add test for body (~b) 2016-07-16 10:15:38 +05:30
Shadab Zafar
88671c3aac Add test for error (~e) 2016-07-16 10:15:38 +05:30
Aldo Cortesi
5f23d4f5ca add_event -> add_log throughout project 
"Event" is terribly over-loaded in the project, and "log" is straight-forward
and self-explanatory.
 2016-07-16 10:19:06 +12:00
Aldo Cortesi
c4fb489a96 Zap unused test scripts 2016-07-15 22:37:52 +12:00
Aldo Cortesi
c7d0850d8f Script cleanups 
- Preserve script order on config change
- Prohibit script duplicates (i.e. identical script + args)
- Various cleanups and tweaks
 2016-07-15 16:35:24 +12:00
Aldo Cortesi
917d51bd22 Fix HAR extractor 
Thanks to @mhils
 2016-07-15 15:01:35 +12:00
Aldo Cortesi
92a5076bd0 Adjust concurrent tests for start method 2016-07-15 15:00:37 +12:00
Aldo Cortesi
4ba7ce50c6 Add .start for addons and scripts 
Also improve error messages, fix various unit tests
 2016-07-15 14:52:34 +12:00
Aldo Cortesi
fcc1416ffd Fix windows parse_args test 2016-07-15 10:33:51 +12:00
Aldo Cortesi
698af945a4 Adjust for new options scheme 2016-07-14 19:58:12 +12:00
Aldo Cortesi
a4127fb6d5 Add a tick event for addons and scripts, and use it for race-free reload 2016-07-14 19:54:15 +12:00
Aldo Cortesi
deffed2196 Script cleanup: editing in console, Python3 compatibility fixes 2016-07-14 19:54:15 +12:00
Aldo Cortesi
b94f5fd361 Convert examples and example tests for new-style scripts 
Remove the test that just loads all the example scripts for now - it's a very
low-value test, and we need to think of something better.
 2016-07-14 19:54:15 +12:00
Aldo Cortesi
a6821aad8e Zap old scripts infrastructure, fix concurrency tests 2016-07-14 19:54:15 +12:00
Aldo Cortesi
a3a22fba33 First-order integration of scripts addon 2016-07-14 19:54:15 +12:00
Shadab Zafar
1266255842 Re-add deletion of stickycookies (#1355) 
This was originally added in #1324 (fixing issue #1096)
but got lost during "the big refactor" (#1352)
 2016-07-14 00:37:27 -07:00
Maximilian Hils
f84a23f11d minor fixes 2016-07-13 22:01:16 -07:00
Maximilian Hils
ec6fbe9eb6 make options explicit 2016-07-13 21:45:25 -07:00
Aldo Cortesi
b2c4f301cb Stream to file -> addon 
This commit also clarifies a confusion about the "outfile" attribute and its
use in testing in the mitmdump master.
 2016-07-14 13:52:09 +12:00
Aldo Cortesi
703c05066e Fix indeterminacy in sticky cookie tests 
How has this not bitten us before?
 2016-07-14 12:59:00 +12:00
Aldo Cortesi
00509d86a8 StickyCookies to addon 2016-07-14 12:13:14 +12:00
Aldo Cortesi
143bf0dfa9 AntiCache to addon 2016-07-14 11:24:06 +12:00
Aldo Cortesi
97b2e6d958 Add a .setter helper to Options 
This returns a function that sets a named attribute, and is handy for
event-driven code like mitmproxy console.
 2016-07-14 11:11:41 +12:00
Aldo Cortesi
a8a083a10d Move StickyAuth into addon 
This is the first addon that handles an OptionsError, so this commit also demos
how this works in console. Handling of command-line erorrs is on its way.
 2016-07-14 10:08:41 +12:00
Aldo Cortesi
255e1eb00b Add the addons mechanism 
Demonstrate how it works and interacts with Options by adding our first new
builtin: anticomp.
 2016-07-13 23:45:49 +12:00
Aldo Cortesi
7f8fd3cdff Basic outline of addons 
Add addons.py, integrate with our event mechanism, and change the Master API so
options is the first init argument.
 2016-07-13 23:26:04 +12:00
Aldo Cortesi
a20f8e9620 More powerful Options scheme 
This prepares us for the addon configuration mechanism and gives us a more
flexible way to handle options changes. This changeset should spell the end of
the current anti-pattern in our codebase where we duplicate data out of options
onto the master when mutability is needed. From now on, Options can be the one
source of thruth.

- Change notifications
- Rollback on error
 2016-07-13 18:45:50 +12:00
Thomas Kriechbaumer
3579c6dd43 http2: fix race-condition in tests 2016-07-12 19:55:28 +02:00
Thomas Kriechbaumer
7c09255298 http2: more coverage 2016-07-10 20:07:43 +02:00
Thomas Kriechbaumer
b611997619 http2: simplify test class 2016-07-10 18:53:00 +02:00
Shadab Zafar
7eade1ef7c Remove redundant tests 2016-07-10 01:08:02 +05:30
Shadab Zafar
39f5108400 Test cookies.is_expired separately 2016-07-10 01:06:50 +05:30
Shadab Zafar
6a746deff5 Add tests for deletion of stickycookies 2016-07-09 22:20:25 +05:30
Shadab Zafar
83a1cc5a9a Make escape_control_characters handle strings on Py2 2016-07-09 13:28:20 +05:30
Shadab Zafar
c7cbbf5f0d Set verbosity & flow_detail to 0 2016-07-09 12:32:41 +05:30
Shadab Zafar
147f61fa62 py3++ 2016-07-09 11:14:12 +05:30
Maximilian Hils
15c10d49f6 Merge pull request #1327 from mitmproxy/remove-script-context 
Remove script context
 2016-07-08 20:11:12 -07:00
Maximilian Hils
5d2b7c52f9 move script context to mitmproxy.ctx 2016-07-08 19:57:57 -07:00
Maximilian Hils
f7639e077a py3++ 2016-07-08 15:58:12 -07:00
jpkrause
05ab9c1b37 Updated coding style and modified flow format compat test to use v0.11 and v0.10 dump files 2016-07-08 15:03:36 -07:00
Maximilian Hils
7c67faa8da remove script contexts 2016-07-07 23:50:55 -07:00
Maximilian Hils
c048ae1d5b remove context from all scripts 2016-07-07 20:14:52 -07:00
Maximilian Hils
00dce24015 tests++ 2016-07-07 18:02:59 -07:00
Maximilian Hils
76473d44e0 py3++ 2016-07-07 17:50:26 -07:00
Maximilian Hils
f62e976e1e py3++ 2016-07-07 17:29:22 -07:00
Thomas Kriechbaumer
d962bd4e83 py3++ 2016-07-07 20:25:48 +02:00
Shadab Zafar
c870d26bc7 py3++: test_protocol_http2 2016-07-07 16:39:25 +02:00
Maximilian Hils
f259b9478f disable harparser on py3 2016-07-07 01:21:15 -07:00
Thomas Kriechbaumer
64ac2c73fe Merge pull request #1318 from mhils/tnetstring3-unicode-type 
[tests actually pass] Add unicode to tnetstring3
 2016-07-07 09:48:22 +02:00
Linmiao Xu
275cf7a6c2 Detect JSON API media type as a json content view 
IANA assignment here:
https://www.iana.org/assignments/media-types/application/vnd.api+json

More about JSON API:
https://github.com/json-api/json-api
 2016-07-06 21:10:35 -07:00
Maximilian Hils
64a867973d sni is now str, not bytes 2016-07-06 21:03:17 -07:00
Maximilian Hils
8287ce7e6d fix tests 2016-07-06 19:55:48 -07:00
Maximilian Hils
d406bee988 tnetstring3: adapt to unicode support 2016-07-06 19:55:48 -07:00
Maximilian Hils
684b4b5130 tnetstring: keys are str on py3. migrate flow.io_compat 2016-07-06 19:55:48 -07:00
Maximilian Hils
9c873d63f4 py3++, multidict fixes 
This commit improves Python 3 compatibility and fixes two multidict
issues:

 1. Headers.items(multi=True) now decodes fields
 2. MultiDict.clear(item) has been removed, as Python's
    MutableMapping already defines .clear() with different
    semantics. This is confusing for everyone who expects a
    dict-like object. `.pop("attr", None)` is not fantastic,
    but it's the Python way to do it.
 2016-07-06 19:53:29 -07:00
Maximilian Hils
444f0a4c39 py3++ 2016-07-06 17:31:08 -07:00
Maximilian Hils
e84f5fcc72 Merge pull request #1315 from mitmproxy/better-unicode-handling 
remove clean_bin, clarify unicode handling
 2016-07-05 16:02:37 -07:00
Maximilian Hils
00a35aa895 Merge pull request #1297 from dufferzafar/py3-flow-export 
Python 3 - test_flow_export
 2016-07-05 15:17:18 -07:00
Maximilian Hils
d51cf543bb remove clean_bin, clarify unicode handling 2016-07-05 15:11:32 -07:00
Thomas Kriechbaumer
5fecb8c843 http2: test fatal connection termination 2016-07-05 22:15:57 +02:00
Thomas Kriechbaumer
8d2042ea8a http2: test throttling at MAX_CONCURRENT_STREAMS 2016-07-05 22:15:57 +02:00
Maximilian Hils
a6b3551934 raise ValueError if content-encoding is invalid 2016-07-04 13:58:09 -07:00
Shadab Zafar
45aa2174e2 http2: Remove TestReadRequestConnect test 2016-07-03 21:14:01 +05:30
Shadab Zafar
f623b3d99b py3++: test_flow_export 2016-07-02 21:15:20 +05:30
Shadab Zafar
393c1d74af fix lint issues 2016-07-02 18:56:22 +05:30
Maximilian Hils
2f8a1fd2cb tests++ 2016-07-02 03:03:42 -07:00
Maximilian Hils
d9f797e7e6 make the linter happy 2016-07-02 02:11:00 -07:00
Maximilian Hils
6032c4f235 message.content -> .raw_content, implement .text 
This PR improves our handling of HTTP message body encodings:

- The unaltered message body is now accessible as `.raw_content`
- The "content-encoding"-decoded content (i.e. gzip removed) content
  is not `.content`, as this is what we want in 99% of the cases.
- `.text` now provides the "content-encoding"-decoded and then
  "content-type charset"-decoded message body.
- The decoded values for `.content` and `.text` are cached,
  so that repeated access and `x.text = x.text` is cheap.
- The `decoded()` decorator is now deprecated, as we can now just use
  `.content`. Similarly `HTTPMessage.get_decoded_content()` is
  deprecated.
 2016-07-02 01:51:47 -07:00
Maximilian Hils
2c09e0416b minor fixes 2016-07-01 17:49:18 -07:00
Maximilian Hils
fa72462798 fix tcp message handling 2016-07-01 17:17:16 -07:00
Maximilian Hils
536c7acd13 py3++ 2016-07-01 14:10:48 -07:00
Shadab Zafar
268e42e55e py3++: test/mitmproxy/test_server 2016-07-01 10:23:04 -07:00
Maximilian Hils
f7d4319c5a minor fixes 2016-06-28 23:48:19 -07:00
Maximilian Hils
3b53d3e393 fix mysterious tox issue 2016-06-28 19:26:08 -07:00
Thomas Kriechbaumer
5d0de16f57 http2: improve error handling in tests 2016-06-22 22:09:57 +02:00
Shadab Zafar
c0ba47094d py3++: mitmproxy.protocol.http1 2016-06-22 13:37:41 +05:30
Shadab Zafar
0a53550955 mitmproxy.tnetstring -> mitmproxy.contrib.tnetstring 2016-06-21 23:53:59 +05:30
Shadab Zafar
10f4a36a5a Imported tests for tnetstring 
Copied from
a808f6b5ba/tests/test_format.py

Then ran pep8 etc.
 2016-06-21 23:47:36 +05:30
Maximilian Hils
44abb4caea Merge pull request #1276 from dufferzafar/console-pathedit 
mitmproxy.console tests - PathEdit
 2016-06-20 20:36:31 -07:00
Shadab Zafar
5b95803be0 py3++ 2016-06-20 21:57:18 +02:00
Shadab Zafar
efeade134a py3++ 2016-06-20 21:39:39 +02:00
Shadab Zafar
6aeba9777a Add dummy directory to completion 2016-06-20 01:24:27 +05:30
Shadab Zafar
ba13fda10d Add tests for PathEdit 2016-06-20 00:50:41 +05:30
Thomas Kriechbaumer
7bde38dc4a py3++ 2016-06-19 12:00:37 +02:00
Thomas Kriechbaumer
eb3ed87100 move custom HTTP/2 stack from netlib to pathod 2016-06-17 15:06:01 +02:00
Shadab Zafar
2a0e850615 pathod tests: do not string_escape specs 2016-06-16 17:08:21 -07:00
Shadab Zafar
6782b4ec00 Py3: Store certificate with a byte key 
Had to debug this for hours!
 2016-06-16 17:08:06 -07:00
Shadab Zafar
ea5576238f Py3: Use BytesIO in a pathoc test 2016-06-16 17:08:06 -07:00
Shadab Zafar
f399a5cd29 Py3: Fix tests by using byte literals 2016-06-16 17:08:06 -07:00
Shadab Zafar
52fa195204 Unexpected spaces around = 2016-06-15 09:49:39 +02:00
Shadab Zafar
f83433e674 Py3: pathoc 2016-06-15 09:49:38 +02:00
Shadab Zafar
fa40531a80 Py3: pathod 2016-06-15 09:49:38 +02:00
Tai Dickerson
c2d0910e59 Make har_extractor.py output HAR 1.2 spec-compliant 
HAR files were failing to load in harviewer (http://www.softwareishard.com/har/viewer/) due to:
- ISO 8601 dates for startedDateTime missing timezone (http://www.softwareishard.com/blog/har-12-spec/#pages)
-- Used UTC but could add detection of default system timezone if desired
- pages object missing pageTimings (http://www.softwareishard.com/blog/har-12-spec/#pageTimings)
-- Used {} because all child fields are optional
 2016-06-14 12:47:10 -04:00
Aldo Cortesi
a63817f2b8 Merge pull request #1253 from mhils/fix-debug 
Fix Debug Mode on Windows
 2016-06-14 17:12:02 +12:00
Aldo Cortesi
93276d45be Merge branch 'master' into netlibrace 2016-06-14 16:29:15 +12:00
Aldo Cortesi
4c292b0197 Fix Python3 tests 2016-06-14 16:13:50 +12:00
Aldo Cortesi
50a20fefee Satisfy lint 2016-06-14 14:34:30 +12:00
Aldo Cortesi
9e63350a96 Use handlers in http2 test suite 2016-06-14 14:32:18 +12:00
Aldo Cortesi
e6fd98bb72 Start rolling out leak detection in netlib 2016-06-14 14:32:18 +12:00
Maximilian Hils
63c6660895 update examples, tests, docs 2016-06-13 18:17:09 -07:00
Maximilian Hils
3c9a70b761 fix debug mode on Windows 2016-06-13 18:06:44 -07:00
Aldo Cortesi
9bea616441 Be stricter about handling connetcts in the pathoc test suite 
Attempt to clear a niggling Appveyor exception buggering up our CI.
 2016-06-12 10:39:19 +12:00
Aldo Cortesi
7b86560ded debug: On SIGUSR2, we dump tracebacks for all threads to screen 2016-06-11 23:07:42 +12:00
Aldo Cortesi
09edbd9492 Improve debugging of thread and other leaks 
- Add basethread.BaseThread that all threads outside of test suites should use
- Add a signal handler to mitmproxy, mitmdump and mitmweb that dumps resource
information to screen when SIGUSR1 is received.
- Improve thread naming throughout to make thread dumps understandable
 2016-06-11 19:52:24 +12:00
Aldo Cortesi
5b9f07c81c debug.sysinfo: tests and coverage 2016-06-11 17:56:17 +12:00
Aldo Cortesi
2cf79b7912 Remove timestamps from pathoc output 
Pathoc is an interactive tool, no need for a long leading timestamp. More
generally, make timestamps optional in the logging mechanism so we can
configure this with command-line flags or something down the track.
 2016-06-10 10:47:28 +12:00
Aldo Cortesi
c421c41307 Remove odict 
- Adds default implementations for _kconv and _reduce_values to MultiDict.
Without these, operations fail in really, really non-obvious ways.
- Replace the remaining few instances of ODict

Fixes #1159
 2016-06-09 13:28:43 +12:00
Shadab Zafar
b3b4156c2f Py3: Fix test_language_http2 tests by using byte literals 2016-06-08 16:15:54 +05:30
Shadab Zafar
6b03df2633 Py3: Use global next() instead of iterator method 2016-06-08 16:15:54 +05:30
Shadab Zafar
0280af9522 Py3: Use BytesIO instead of StringIO 2016-06-08 16:15:54 +05:30
Aldo Cortesi
a5cb241c7c If a message has been acked, all other processors are skipped 
This applies the constraint, but does to clumsily. When we've unified modules
and processors it will be much nicer. We also make some exceptions for the
master processors that we may want to re-evaluate down the track.
 2016-06-08 12:58:58 +12:00
Aldo Cortesi
b3bf754e53 Simplify script concurrency helpers 
We now have take() to prevent double-replies.
 2016-06-08 11:21:38 +12:00
Aldo Cortesi
a388ddfd78 A new interface for reply 
Reply is now explicit - it's no longer a callable itself. Instead, we have:

    reply.kill()            - kill the flow
    reply.ack()             - ack, but don't send anything
    reply.send(message)     - send a response

This is part of an incremental move to detach reply from our flow objects,
and unify the script and handler interfaces.
 2016-06-08 10:44:20 +12:00
Aldo Cortesi
982077ec31 Add reply.ack and reply.kill 2016-06-08 10:14:34 +12:00
Aldo Cortesi
db11fe0087 Merge branch 'master' into beautiful-json 2016-06-08 09:19:25 +12:00
Maximilian Hils
c98c83190b tests++ 2016-06-07 01:03:08 -07:00
Shadab Zafar
d254fe791a Py3: Fix a test_language_writer test by using a byte literal 2016-06-07 00:01:41 -07:00
Maximilian Hils
ba466dbb35 simplify pretty_json, improve test coverage 2016-06-06 23:08:58 -07:00
Aldo Cortesi
1ffc273c94 Utils cleanups 
- Move more stuff that belongs in netlib.human
- Move some stuff to near the only use
- Zap mitmproxy.utils.timestamp(). I see the rationale, but we used it
interchangeably with time.time() throughout the project. Since time.time()
dominates in the codebase and timestamp() is such low utility, away it goes.
 2016-06-07 17:12:52 +12:00
Maximilian Hils
80292c0ac3 fix all pathod tests on Windows 🎉 2016-06-06 16:03:45 -07:00
Maximilian Hils
2ee5e8ef0e fix NestedMessage encoding 2016-06-06 13:39:38 -07:00
Maximilian Hils
6447c8ae22 fix .freeze(), improve tests 2016-06-06 13:31:44 -07:00
Shadab Zafar
f48073af56 Py3: Fix status_code and other tests by using byte literals 2016-06-06 11:52:45 -07:00
Shadab Zafar
5a2932adc1 Py3: Fix corresponding base.Integer test in test_language_base 2016-06-06 11:52:45 -07:00
Shadab Zafar
40056e1cd7 Py3: Fix useragent tests 2016-06-06 11:52:45 -07:00
Shadab Zafar
f5a6b295f7 Py3: Fix websocket tests 2016-06-06 11:52:45 -07:00
Shadab Zafar
b86ef9fd04 Py3: Use BytesIO in tutils.render 2016-06-06 11:52:45 -07:00
Shadab Zafar
bafce6cfe9 Py3: Use byte string literals 2016-06-06 11:51:36 -07:00
Shadab Zafar
9a19540f8b Py3: Use BytesIO instead of StringIO for tests 2016-06-06 11:51:36 -07:00
Shadab Zafar
12dd6f6707 Py3: Use global next() instead of iterator method 2016-06-06 11:51:36 -07:00
Shadab Zafar
ced6fb4cf9 Py3: Fix tests by using byte literals 2016-06-06 11:48:58 -07:00
Shadab Zafar
efb0f7133c Py3: Use global next() instead of iterator method 2016-06-06 11:48:54 -07:00
Aldo Cortesi
210c376901 Fix sequencing in Pathoc websocket termination test 2016-06-06 18:17:22 +12:00
Aldo Cortesi
9458eaece7 Make more of the pathod test suite use synchronisation 2016-06-06 16:05:44 +12:00
Aldo Cortesi
08344ee38b Merge pull request #1215 from redfast00/master 
Now keeps header capitalization
 2016-06-06 08:45:35 +12:00
Aldo Cortesi
c31b9c461d Merge pull request #1211 from cortesi/pathod 
WIP: Radical webectomy of pathod
 2016-06-06 08:40:20 +12:00
redfast00
25505733bc
fix multidict unittests 2016-06-05 21:20:37 +02:00
Shadab Zafar
435bfeca0b Use BytesIO instead of StringIO 2016-06-06 00:36:15 +05:30
Shadab Zafar
7b6da98719 Py3: Use byte literals for comparison 2016-06-06 00:07:44 +05:30
Shadab Zafar
be306c8439 Py3: Use global next() function instead of iterator method 2016-06-06 00:07:44 +05:30
Aldo Cortesi
375d7c9741 Remove last vestiges of noapi and noweb from pathod 
Also hide HTTP2 freeze bug by making explain configurable in the tests
 2016-06-05 13:24:46 +12:00
Aldo Cortesi
48da24ae7e First-order removal of pathod API and app 2016-06-05 13:04:13 +12:00
Aldo Cortesi
c0c45c051a Fix test that may fail due to binary header data 
This is just inherently not a determinisitc test. We don't use the log HTTP
interface any more, so it can just go. A more radical "solution" is inbound
shortly.

Fixes #1207
 2016-06-05 11:47:52 +12:00
Shadab Zafar
6ffe8b376e Py3: Write bytes to file 2016-06-04 18:28:50 +05:30
Shadab Zafar
0d4437a404 Fix function name typo 2016-06-04 18:28:03 +05:30
Shadab Zafar
bd7b275d44 Py3: Convert all data to bytes in pathod.language.writer test 2016-06-04 15:23:09 +05:30
Shadab Zafar
614a3d7a54 Py3: Use global next() instead of iterator method 2016-06-04 15:23:09 +05:30
Aldo Cortesi
d5532319c1 Basic ConsoleMaster tests, based on mastertester 2016-06-04 19:07:27 +12:00
Aldo Cortesi
af63a3e064 console/__init__.py -> console/master.py 2016-06-04 16:06:44 +12:00
Aldo Cortesi
12a4f9ecf9 Very basic test truss for web.app 2016-06-04 15:30:28 +12:00
Aldo Cortesi
f7f9cab5dc Rebase on master 2016-06-04 14:37:07 +12:00
Aldo Cortesi
e60860e65d Make tcp.Client.connect return a context manager that closes the connection 2016-06-03 12:06:36 +12:00
Aldo Cortesi
c5076f5e01 Implement a service connection handler counter, use it in Pathod test suite 
Lots of failures, but that's a good thing.
 2016-06-03 11:47:07 +12:00
Aldo Cortesi
c6d15904f3 Merge remote-tracking branch 'upstream/master' into testsuite 2016-06-03 10:19:22 +12:00
Maximilian Hils
29c52d98dc Merge remote-tracking branch 'duffer/pathod-lang-generators' 2016-06-02 09:53:22 -07:00
Aldo Cortesi
5fd52970fd Satisfy linting gods 2016-06-02 19:55:52 +12:00
Aldo Cortesi
c9dd574f48 Silence requests warnings 
Once we convert the rest of the suite to pathoc, this can go.
 2016-06-02 19:40:39 +12:00
Aldo Cortesi
d8c52964c7 Reimplement test retrievals on pathoc and ditch requests 
Requests uses urllib3, which has a connection pool that's not easy to disable
(https://github.com/shazow/urllib3/issues/383).
 2016-06-02 19:36:02 +12:00
Shadab Zafar
b510616c69 Py3: Return bytes from FileGenerator 
and use __getitem__ instead of __getslice__
 2016-06-02 13:00:44 +05:30
Shadab Zafar
a04d7fd166 Py3: Return bytes from RandomGenerator 
and use __getitem__ rather than __getslice__
 2016-06-02 13:00:02 +05:30
Aldo Cortesi
a7522d9308 pathod.test shouldn't use the API at all 2016-06-02 18:27:51 +12:00
Aldo Cortesi
254614e9f7 Since we have locks over the logs, use direct access rather than API requests to get to them 2016-06-02 18:10:36 +12:00
Aldo Cortesi
ffca395e48 Honesty in testing 
- Don't skip tests that fail
- Don't omit console from coverage

I'm working on the pathod races, and can't seem to reproduce the failures
locally.

Also, let's not kid ourselves about the console coverage - it should be
tracked. To improve it, we should measure it. Regressions in coverage of the
module should be flagged by CI.
 2016-06-02 14:44:06 +12:00
Aldo Cortesi
09da1febbd Shift a bunch more string-related functions to strutils 2016-06-02 13:04:19 +12:00
Aldo Cortesi
31012d782f Consolidate some functions to strutils.bytes_to_escaped_str 2016-06-02 12:53:12 +12:00
Aldo Cortesi
cccdc98426 Utils reorganisation: add netlib.strutils 
Extract a number of string and format-related functions to netlib.strutils.
 2016-06-02 12:31:41 +12:00
Aldo Cortesi
eaa3b308f7 Fix non-deterministic test failures in export 
We had various places in the code where we relied on incidental order of dict
keys. Add a helper to multidict, and fix.
 2016-06-02 11:37:18 +12:00
Maximilian Hils
898602ad21 Merge pull request #1180 from dufferzafar/pathod-base 
Python 3 - pathod.language.base
 2016-05-31 14:46:28 -07:00
Maximilian Hils
30fff1fb32 be more strict about module namespaces 2016-05-31 14:05:57 -07:00
Aldo Cortesi
42e91fcfe1 http2.frame -> http2.framereader 2016-05-31 20:32:28 +12:00
Aldo Cortesi
5a75ea3fc6 Fix test failures 2016-05-31 20:19:54 +12:00
Aldo Cortesi
40a030f215 Satisfy flake8 2016-05-31 19:58:28 +12:00
Aldo Cortesi
f62efed304 Unify and make symmetric pretty_size and parse_size 2016-05-31 19:45:48 +12:00
Aldo Cortesi
b2f63458fc Move human-friendly format functions to netlib.human, remove redundant implementations 2016-05-31 19:32:08 +12:00
Aldo Cortesi
ec34cae618 utils.multipartdecode -> http.multipart.decode 
also

utils.parse_content_type -> http.headers.parse_content_type
 2016-05-31 19:07:55 +12:00
Aldo Cortesi
15b2374ef9 netlib.utils.get_header_tokens -> netlib.http1.read.get_header_tokens 
Placing this next to its only use.
 2016-05-31 18:54:42 +12:00
Aldo Cortesi
6dda2b2ee5 Module is part of the name - url.decode, not url.urldecode 
A pattern we need to use far more often in the codebase
 2016-05-31 18:46:19 +12:00
Aldo Cortesi
4e6c9c4e93 Extract url functions from netlib.utils and move to netlib.http.url 2016-05-31 18:42:56 +12:00
Aldo Cortesi
08fbe6f111 Start cleaning up netlib.utils 
- Remove http2 functions, move to http2.frame
- Remove Serializable, move to netlib.basetypes
 2016-05-31 17:16:31 +12:00
Aldo Cortesi
9ea68ebd28 Improve handling of pseudo-headers 
- The canonical source for :method, :scheme and :path are the .method, .scheme
and .path attributes on the request object.
- These pseudo-headers are stripped after reading the request, and re-inserted
just before sending.
- The :authority header remains, and should be handled analagously to the Host
header in HTTP1 with respect to display and user interaction.
 2016-05-31 14:36:17 +12:00
Aldo Cortesi
bc6509ab6d Fix test import error 2016-05-31 10:25:02 +12:00
Shadab Zafar
25e30f42be Py3: Use byte literals for comparison 2016-05-31 01:13:50 +05:30
Maximilian Hils
89f07603ca split mitmproxy.flow into mitmproxy.flow.* 2016-05-30 01:40:09 -07:00
Thomas Kriechbaumer
be9438b01c fix 2016-05-29 22:58:56 +02:00
Thomas Kriechbaumer
8cd0c9c330 fix windows paths 2016-05-29 22:47:51 +02:00
Thomas Kriechbaumer
fb0b591242 fix har test 2016-05-29 22:30:15 +02:00
Thomas Kriechbaumer
03167d5e71 fix headers order 2016-05-29 22:07:22 +02:00
Thomas Kriechbaumer
01fcaa64c3 fix #1162 2016-05-29 13:58:14 +02:00
Thomas Kriechbaumer
e2447406cc more style cleanup 2016-05-29 13:43:20 +02:00
Thomas Kriechbaumer
8eeab37a07 more style cleanup 
Use this to check:
flake8 --count mitmproxy netlib pathod examples test
 2016-05-29 11:55:12 +02:00
Aldo Cortesi
f3bee6f245 Merge pull request #1172 from cortesi/solidcore 
First steps to solidifying the core
 2016-05-29 20:03:40 +12:00
Maximilian Hils
85aa5da6e3 fix tests 2016-05-28 19:31:43 -07:00
Aldo Cortesi
0a875421c5 handle_* -> * 
Now that we have the controller.handler decorator, the _handler prefix
stutters.
 2016-05-29 12:49:01 +12:00
Aldo Cortesi
8bb9285678 Explicitly list all events 2016-05-29 12:15:52 +12:00
Aldo Cortesi
e4f043f69c Flatten ServerMaster into Master 2016-05-29 12:01:46 +12:00
Aldo Cortesi
ed415877d4 Merge branch 'master' into solidcore 2016-05-29 11:14:46 +12:00
Thomas Kriechbaumer
caee1adeb6 cleanup imports with flake8 2016-05-28 14:43:32 +02:00
Maximilian Hils
fa7964988f fix Python 2 tests 2016-05-28 00:17:57 -07:00
Shadab Zafar
b71ef054b5 Py3: Use cStringIO from six.moves 2016-05-28 00:17:57 -07:00
Aldo Cortesi
f4813d5649 Make @controller.handler inheritance-friendly 
Use this to adapt mitmweb and mitproxy console
 2016-05-28 12:38:07 +12:00
Aldo Cortesi
f29f6210bb Mandate that all handlers must be wrapped, make tests pass 
mitmproxy, mitmdump and mitmweb masters still to be done
 2016-05-28 11:00:00 +12:00
Aldo Cortesi
7c4843857b Sketch out a more solid core 
- Decorator for handler methods
- Stricter checking for double-acks and non-acks
 2016-05-28 11:00:00 +12:00
Shadab Zafar
bc6cd13356 Go Python 3 by default with the bytes conversion 2016-05-26 23:07:37 +05:30
Shadab Zafar
1bbb178b6a Remove inner_repr, fixup escape_unprintables 2016-05-26 23:07:37 +05:30
Shadab Zafar
22e4bc1938 Py3: Handle bytes case in inner_repr 2016-05-26 23:07:37 +05:30
Maximilian Hils
d3477e27fa bytes_to_escaped_str: always escape single quotes 2016-05-25 20:11:34 -07:00
Maximilian Hils
7a8da48a30 escaped_str_to_bytes: support unicode on python 2 2016-05-25 19:16:02 -07:00
Thomas Kriechbaumer
ebaad91484 Merge pull request #1127 from mitmproxy/tcp-flows 
mitmdump: Add Basic Support for TCP Flows
 2016-05-23 20:54:14 +02:00
Aldo Cortesi
97f3077082 Merge branch 'mhils-multidict' 2016-05-21 15:01:19 +12:00
Aldo Cortesi
43d7965532 Clean un-needed imports 2016-05-21 15:00:52 +12:00
Aldo Cortesi
123e3b130c We don't need 3 slightly different implementations of Data 2016-05-21 11:53:14 +12:00
Aldo Cortesi
a5c4cd0340 A clearer implementation of MultiDictView 
This makes MultiDictView work with a simple getter/setter pair, rather than
using attributes with implicit leading underscores. Also move MultiDictView
into multidict.py and adds some simple unit tests.
 2016-05-21 11:37:36 +12:00
Aldo Cortesi
96d8ec1ee3 Merge branch 'multidict' of https://github.com/mhils/mitmproxy into mhils-multidict 2016-05-21 10:15:37 +12:00
Maximilian Hils
84144ca0c6 tests: silence flaky pathod test 2016-05-20 14:05:04 -07:00
Maximilian Hils
a3946d2a2d tests++ 2016-05-20 13:27:26 -07:00
Maximilian Hils
b538138ead tests++ 2016-05-20 11:04:27 -07:00
Maximilian Hils
f96697646c add FlowMaster.active_flows 2016-05-19 22:09:00 -07:00
Maximilian Hils
d38989fe7e tests: finish connections properly 2016-05-19 22:07:56 -07:00
Maximilian Hils
d3c30d9005 fix tests, don't double-add error'd flows 2016-05-19 22:07:56 -07:00
Maximilian Hils
acd51befbb minor fixes, adjust tests 2016-05-19 22:07:56 -07:00
Thomas Kriechbaumer
79cb17ef03 Merge pull request #1152 from mhils/cleanup-odict 
clean up odict
 2016-05-19 16:47:51 +02:00
Maximilian Hils
a7b9e3033d tests: properly skip pathod tests 2016-05-18 22:51:27 -07:00
Maximilian Hils
6f8db2d7eb improve MultiDict, add ImmutableMultiDict, adjust response.cookies 2016-05-18 22:50:19 -07:00
Maximilian Hils
8e39b7bf38 test flow export with duplicate query string 2016-05-18 19:28:23 -07:00
Maximilian Hils
d1fc694952 clean up odict 2016-05-18 19:01:49 -07:00
Maximilian Hils
44ac64aa72 add MultiDict 
This commit introduces MultiDict, a multi-dictionary similar to
ODict, but with improved semantics (as in the Headers class).
MultiDict fixes a few issues that were present in the Request/Response
API. In particular, `request.cookies["foo"] = "bar"` has previously been a
no-op, as the cookies property returned a mutable _copy_ of the cookies.
 2016-05-18 18:46:42 -07:00
Maximilian Hils
4c3fb8f509 tests: xfail -> skip 2016-05-18 18:42:35 -07:00
Thomas Kriechbaumer
09b90fce9e remove debug print 2016-05-17 22:45:09 -07:00
Thomas Kriechbaumer
e61014d203 http2: add connection-lost test 2016-05-17 21:25:10 -07:00
Aldo Cortesi
d27fd55657 Fix broken test_flow_export tests 
- Make the tests safe for concurrency
- Fix a test that only passed due to prior modification of global state

This also fixes the project generally for parallel test execution.
 2016-05-18 12:34:45 +12:00
Aldo Cortesi
00cc836470 Clean up flow export tests 
- Externalise test data
- Define a less finicky comparator for Python code
 2016-05-18 12:16:50 +12:00
Thomas Kriechbaumer
2887480bcb fix naming which caused overwriting of functions 2016-05-15 10:39:39 -07:00
Thomas Kriechbaumer
f4a13d6f9f http2: improve header tests (#1143) (#1144) 2016-05-14 10:14:42 -07:00
Maximilian Hils
f1c922c652 Sanitize Print (#1135) 
* sanitize strings with shell control characters

* netlib: add utilities to safe-print bytes

* escaped str: add TODO for multi-byte chars
 2016-05-12 11:03:57 -06:00
Thomas Kriechbaumer
7a813936c6 disable http2 header encoding, use bytes everywhere 2016-05-11 16:47:58 -05:00
Thomas Kriechbaumer
bef72c1b02 Merge pull request #1121 from Kriechi/fix-cookies 
improve cookie parsing
 2016-05-11 12:30:44 -05:00
Thomas Kriechbaumer
e2ee48a0ee replace SimpleCookie with our own parser lib 2016-05-11 12:20:21 -05:00
Maximilian Hils
67a37e6d1f improve script loading 2016-05-11 11:15:36 -06:00
Maximilian Hils
4c8e334642 actually fix tests 2016-05-10 12:19:26 -06:00
Maximilian Hils
bb4f9611f5 Revert "fix tests" 
This reverts commit f315dc1eb9.
 2016-05-10 12:18:05 -06:00
Maximilian Hils
f315dc1eb9 fix tests 2016-05-10 12:17:25 -06:00
Thomas Kriechbaumer
08002282e8 improve cookie parsing 
allows '/' to be within a cookie name
removes deprecated cookie getter/setter
fixes #1118
 2016-05-10 11:13:12 -05:00
Maximilian Hils
9d47d3b1ec fix #1120 2016-05-09 12:15:20 -06:00
Maximilian Hils
74cfd7a4e2 stateobject: support lists 2016-04-29 20:59:26 -07:00
Maximilian Hils
317b4becca mitmdump: don't fail for other flowtypes 2016-04-29 20:18:32 -07:00
Maximilian Hils
54c4859a95 remove netlib version check 2016-04-29 12:00:38 -07:00
Maximilian Hils
909d5ec87e FlowReadError -> FlowReadException 2016-04-29 11:17:49 -07:00
Steven Van Acker
66267ad276 support for setting/sending multiple cookies (#1091) 
* support for setting/sending multiple cookies

* py.test for multiple cookie support
 2016-04-22 11:16:05 -07:00
Maximilian Hils
5b12720962 silence flaky pathod test 2016-04-21 18:29:16 -07:00
Maximilian Hils
028f5c4929 downgrade pyparsing to fix #1087 and #1090 2016-04-21 17:11:28 -07:00
Maximilian Hils
423c076c61 cleanup mitmproxy.controller, raise Kill in Channel (#1085) 2016-04-14 12:03:29 -07:00
Maximilian Hils
bc60c26c7b py3++ 2016-04-13 22:34:28 -07:00
Maximilian Hils
70e35b7017 peer_address -> ip_address 2016-04-11 14:14:18 -07:00
Maximilian Hils
dcdee7dc46 server replay: keep --kill active 2016-04-09 16:47:05 -07:00
Maximilian Hils
0259f47997 Merge branch 'better-replace' 2016-04-03 08:17:30 -07:00
Thomas Kriechbaumer
f5fe93bf13 Merge pull request #1074 from mitmproxy/move-response-refresh 
move HTTPResponse.refresh into netlib
 2016-04-03 10:16:39 +02:00
Thomas Kriechbaumer
16fdbb4f2f Merge pull request #1073 from mitmproxy/first-line-format 
form_(in|out) -> first_line_format
 2016-04-03 10:16:06 +02:00
Maximilian Hils
35a6d6e7f6 response.msg -> response.reason 2016-04-02 22:54:46 +02:00
Maximilian Hils
4e2acc4d9d move HTTPResponse.refresh into netlib 2016-04-02 22:49:05 +02:00
Maximilian Hils
e6fa2688c9 form_(in|out) -> first_line_format 2016-04-02 21:57:35 +02:00
Maximilian Hils
610842cda5 fix FlowMaster.create_request, add test 2016-04-02 21:15:55 +02:00
Maximilian Hils
806aa0f41c improve .replace() and move it into netlib 2016-04-02 14:38:33 +02:00
Maximilian Hils
4ee8808b44 add Serializeable.copy 2016-04-02 13:50:53 +02:00
Zohar Lorberbaum
cd2ef2fe13 merge 2016-03-28 10:17:58 -07:00
Zohar Lorberbaum
37483e228f Merge remote-tracking branch 'mitmproxy/master' 
Merge with master
 2016-03-28 10:06:02 -07:00
Zohar Lorberbaum
6d16f44ab7 Merge with master 2016-03-28 10:03:26 -07:00
Zohar Lorberbaum
e56198ae7c cleaner target url 2016-03-28 09:51:06 -07:00
Zohar Lorberbaum
ef3d24e8c8 locust_task re-use locust_code. 2016-03-27 21:42:52 -07:00
Zohar Lorberbaum
9f77c80a32 pep8 2016-03-25 17:29:42 -07:00
Zohar Lorberbaum
5b07e8b3af Add UI shortcuts. 2016-03-24 20:29:53 -07:00
Zohar Lorberbaum
a44062effb Flow export to locust.io load test tool. 2016-03-23 01:49:18 -07:00
Maximilian Hils
403ac82a7d netlib: request.path can be None 2016-03-20 23:22:50 +01:00
Maximilian Hils
726536689b py3++ 2016-03-20 22:58:35 +01:00
Maximilian Hils
2ce023a991 py3++ 2016-03-20 22:50:03 +01:00
Maximilian Hils
88d365cfe6 py3++: cStringIO 2016-03-20 19:40:03 +01:00
Maximilian Hils
c52c59f858 port mitmproxy.scripts to py3 2016-03-19 19:53:27 +01:00
Maximilian Hils
4be9074b49 fix tests on OSX 2016-03-19 19:19:36 +01:00
Maximilian Hils
fb0b17ee93 simplify tests for @concurrent 2016-03-19 03:04:55 +01:00
Maximilian Hils
7e49b8c186 add tests for mitmproxy.script.reloader 2016-03-19 03:04:55 +01:00
Maximilian Hils
898f5d10b9 improve mitmproxy.scripts semantics, clean up tests 2016-03-19 03:04:55 +01:00
Maximilian Hils
983b0dd4f6 Merge pull request #1014 from ikoz/master 
New option: Add server certs to client chain
 2016-03-17 02:28:00 +01:00
ikoz
9cc55f211f Rename 'server' to 'upstream' in identifiers related to the AddServerCertsToClientChain feature 2016-03-16 19:20:18 +00:00
ikoz
d26c7f4ca5 Move more init things inside AddServerCertsToClientChainMixin 2016-03-16 18:43:51 +00:00
ikoz
efc3e942d5 Restructuring of the AddServerCertsToClientChain test so that it uses a Mixin - also removed some extra printf statements 2016-03-16 17:43:48 +00:00
Maximilian Hils
2a20fc59b2 Merge pull request #1030 from xhy940801/socks 
Add 'UsernamePasswordAuth' 'UsernamePasswordAuthResponse' to SOCKS
 2016-03-16 12:16:52 +01:00
Thomas Kriechbaumer
0ab9e60168 Merge pull request #1019 from lilydjwg/master 
add resolved IP address in "Details" tab
 2016-03-15 23:28:08 +01:00
ikoz
776e625413 Add tests for add-server-certs-to-client-chain feature 2016-03-15 14:58:38 +00:00
lilydjwg
eecadadce3 use "peer_address" instead of "sock_address" 2016-03-15 21:32:06 +08:00
yonder
ba933dff2c Add 'UsernamePasswordAuth' 'UsernamePasswordAuthResponse' to SOCKS 2016-03-15 14:33:20 +08:00
lilydjwg
20ec502032 fix tests for "sock_address" 2016-03-14 22:17:12 +08:00
Shadab Zafar
5a1c3c4ad8 Update contentview test 2016-03-11 18:40:51 +05:30
Shadab Zafar
35204ecb1c Move har_extractor tests to the examples file 2016-03-11 17:57:57 +05:30
Shadab Zafar
7c5791e51a Use the contextmanager in all tests 2016-03-11 17:57:57 +05:30
Shadab Zafar
af9442a9ba Add contextmanager to load an example 2016-03-11 17:57:57 +05:30
Shadab Zafar
b95cc63b87 Add tests for redirect_requests example 2016-03-11 17:57:57 +05:30
Shadab Zafar
2cd5392657 Add test for iframe_injector example 2016-03-11 17:57:57 +05:30
Shadab Zafar
ba1cf18f42 Add tests for custom_contentviews example 2016-03-11 17:57:57 +05:30
Shadab Zafar
5eca2223d5 Add tests for modify_response_body example 2016-03-11 17:57:57 +05:30
Shadab Zafar
c378fe2199 Remove missing import 2016-03-11 17:57:57 +05:30
Shadab Zafar
9ef1522b5f Update modify_querystring example & add test for it 2016-03-11 17:57:57 +05:30
Shadab Zafar
96df077d54 Add tests for add_header example 2016-03-11 17:57:57 +05:30
Shadab Zafar
a2274f52e4 Add tests for modify_form example 2016-03-11 17:57:57 +05:30
Thomas Kriechbaumer
f59770e949 Merge pull request #969 from dufferzafar/pretty-export 
Indent JSON body while exporting it as code
 2016-03-04 19:57:22 +01:00
Thomas Kriechbaumer
428da2c4b1 Merge pull request #989 from dufferzafar/har-extractor 
Improve HAR Extractor Script
 2016-03-04 19:55:52 +01:00
yonder
0e27dfd9c1 Improve test case 2016-03-04 11:12:03 -05:00
Shadab Zafar
d17cd5c6ec Use custom Context object 2016-03-04 03:13:40 +05:30
Shadab Zafar
6a4f1ae7c7 Test har_extractor.response 2016-03-04 03:10:29 +05:30
Shadab Zafar
c982f579dd Add basic test for har_extractor 2016-03-04 02:32:01 +05:30
Shadab Zafar
48f1c9afc2 Add a test for json request 2016-03-04 01:06:09 +05:30
yonder
b8ae666c78 Add test for upstream-auth 2016-03-02 12:19:33 -05:00
Maximilian Hils
ade5078ebd suppress pathoc test timeouts on windows 2016-02-27 14:36:17 +01:00
Maximilian Hils
2bec6a35e2 fix #956 2016-02-19 00:30:37 +01:00
Maximilian Hils
ecb26c3c82 Merge remote-tracking branch 'duffer/pretty-host' 2016-02-18 23:17:02 +01:00
Shadab Zafar
d6ab9901d1 Fixup more tests 2016-02-18 21:58:32 +05:30
Maximilian Hils
20352f3453 fix test paths, move coverage config into setup.py 2016-02-18 13:24:17 +01:00
Shadab Zafar
175109e44e Use host header values only when the ports match 2016-02-18 07:01:52 +05:30
Aldo Cortesi
b6e5e50c24 Make fuzzing run again 2016-02-18 10:40:16 +13:00
Aldo Cortesi
92597f82ea Docs and examples to top level 2016-02-18 09:27:08 +13:00
Shadab Zafar
6f96da08c9 Handle port numbers in host header 
from: https://github.com/mitmproxy/netlib/pull/121
 2016-02-17 08:48:59 +05:30
Will Coster
595d2a2fa0 Add a content view for query parameters 
The query content view uses format_dict to display a table of query
parameters and is made the default content view for requests with
query parameters and no request body.

To facilitate this the query parameter dictionary of HTTPRequests is
added to the metadata content view parameter under the "query" key.

Additionally, the logic for handling "no content" messages is moved
from contentviews.get_content_view to ViewAuto. This is necessary as
it allows the query content view to be displayed when there is no
request body.
 2016-02-16 18:50:10 -08:00
Thomas Kriechbaumer
6b585023fd move tservers helper 2016-02-16 21:53:16 +01:00
Thomas Kriechbaumer
8242f2cc9d rename ProxTest -> ProxyTest 2016-02-16 21:15:50 +01:00
Thomas Kriechbaumer
1785f70a19 fix all libpathod->pathod references 2016-02-16 21:11:19 +01:00
Thomas Kriechbaumer
d95a1f9819 fix all libmproxy->mitmproxy references 2016-02-16 20:56:33 +01:00
Will Coster
0fc348d211 Fix failing unit tests caused by a69d223b 2016-02-15 20:36:21 -08:00
Maximilian Hils
2d569527f2 Merge branch 'master' of https://github.com/mitmproxy/mitmproxy 2016-02-16 02:02:40 +01:00
Maximilian Hils
bc7db1be2d temporarily silence flaky tests 2016-02-16 02:02:32 +01:00
Will Coster
a69d223b33 Preserve host header when modifying request path 
Currently the path_components and query setters of the Request object
use the url setter under the hood. The url setter updates all parts of
the URL including the host. If the host header and the host in the
request URL are different (as is common when making HTTPS requests)
then the host header will be updated to the value in the URL as a
result of modifying the path.

This change fixes this problem by modifying the query and
path_components setters to not use the url setter and instead directly
update the path field.
 2016-02-15 16:21:20 -08:00
Maximilian Hils
a78c6f1cb0 update README.rst, minor fixes 2016-02-15 19:43:55 +01:00
Maximilian Hils
fc9fa87827 fix ALPN checks 2016-02-15 18:43:06 +01:00
Maximilian Hils
d7158f975e move tests into shared folder 2016-02-15 16:34:22 +01:00
Maximilian Hils
33fa49277a move mitmproxy 2016-02-15 14:58:46 +01:00
Thomas Kriechbaumer
4767474d4b remove noop code 2016-02-14 14:58:36 +01:00
Thomas Kriechbaumer
6746cf5f54 autoflake 2016-02-14 14:45:27 +01:00
Maximilian Hils
324184a25a Merge pull request #916 from dufferzafar/export-code 
Export any flow request as Python code
 2016-02-10 16:05:03 +01:00
Shadab Zafar
5f044d03b7 Use classes to test a command, move code to separate methods under 
classes
 2016-02-09 21:58:29 +05:30
Shadab Zafar
0576f579ed Use textwrap.dedent to improve readability of tests 2016-02-09 21:52:22 +05:30
Shadab Zafar
1c2f8a608e Rename test_export_flow.py to test_flow_export.py 2016-02-09 21:26:05 +05:30
Shadab Zafar
ae8fb0a3d3 Fix content length header issues 
Caused by:
d253ebc142
 2016-02-08 23:31:56 +05:30
Shadab Zafar
56892f885b Tests for flow_export.raw_request 2016-02-08 23:31:30 +05:30
Shadab Zafar
ddeb3731a8 Tests for flow_export.python_code 2016-02-08 23:31:01 +05:30
Shadab Zafar
61a4743b62 Test exact return value of flow_export.curl_command 2016-02-08 21:56:20 +05:30
Shadab Zafar
abc1552800 Add some basic tests for flow_exports 2016-02-08 21:28:10 +05:30
Maximilian Hils
bdb763d9cf make stateobject simpler and stricter 2016-02-08 04:19:25 +01:00
Maximilian Hils
2654cd87af fix #876 2016-02-08 01:06:18 +01:00
Thomas Kriechbaumer
cf8c063773 fix http2 race condition 2016-02-04 09:52:29 +01:00
Thomas Kriechbaumer
ca5cc34d0b cleanup 2016-02-04 09:52:29 +01:00
Thomas Kriechbaumer
738094e167 improve test reliability 2016-02-04 09:52:28 +01:00
Thomas Kriechbaumer
d8ae7c3e29 fix tests and use netlib utils 2016-02-04 09:52:28 +01:00
Thomas Kriechbaumer
6d3b3994e2 code formatting 2016-02-04 09:52:28 +01:00
Thomas Kriechbaumer
44f83b5947 add more tests, improve coverage 2016-02-04 09:52:27 +01:00
Thomas Kriechbaumer
bd1d9e28e4 test stream resets in push promise 2016-02-04 09:52:27 +01:00
Thomas Kriechbaumer
276817e40e refactor http2 tests 2016-02-04 09:52:27 +01:00
Thomas Kriechbaumer
187691e65b remove print 2016-02-04 09:52:27 +01:00
Thomas Kriechbaumer
41f4197a0d test PushPromise support 2016-02-04 09:52:27 +01:00
Thomas Kriechbaumer
735c79a2ed increase coverage 2016-02-04 09:52:04 +01:00
Thomas Kriechbaumer
47cf27c011 silence 3rd party module loggers 2016-02-04 09:52:03 +01:00
Thomas Kriechbaumer
bfc7d3967c exclude tests if no alpn support present 2016-02-04 09:52:03 +01:00
Thomas Kriechbaumer
4501c8a0a1 add http2 full-stack test 2016-02-04 09:52:03 +01:00
Thomas Kriechbaumer
4de9cbb61e rename test file 2016-02-04 09:52:03 +01:00
Thomas Kriechbaumer
2964a607ad fix import in tests 2016-02-04 09:52:03 +01:00
Maximilian Hils
aaf82cae7d tests: use relative imports 2016-02-02 13:25:31 +01:00
Maximilian Hils
f39e50c4db silence third-party modules 2016-02-02 12:26:28 +01:00
Thomas Kriechbaumer
8c37538314 code formatting: fix whitespace issues 2016-01-27 10:36:28 +01:00
Thomas Kriechbaumer
c2bb29f669 cleanup imports 2016-01-27 10:36:28 +01:00
Maximilian Hils
0f54f2990c Merge pull request #872 from bazzinotti/inline-tcp 
Add "tcp_message" Inline Script Hook
 2016-01-17 19:54:59 +01:00
Michael J. Bazzinotti
e766838a1a Add Inline Script Hooks to TCP mode 2016-01-10 13:05:01 -05:00
Maximilian Hils
11215e46ec fix #659 2016-01-02 15:12:36 +01:00
Michael J. Bazzinotti
f8d8a80bd0 clientcert tests: fix whitespace 2015-12-29 18:14:11 +01:00
Michael J. Bazzinotti
09168e1274 client-cert: Simplify and cleanup 
simplify the error message when cert path does not exist.
During tests, reset the clientcerts proxy config setting to None
Remove now-unused clientcerts code from class ProxTestBase
 2015-12-29 11:48:02 -05:00
Michael J. Bazzinotti
80690b7906 [docs/libmproxy/test] Support single client-side cert file as argument to 
--client-certs
 2015-12-28 15:20:45 -05:00
Maximilian Hils
b362e9a013 test flow conversion 2015-12-03 14:43:56 +01:00
Maximilian Hils
85b553724c add tnetstring inspection tool 2015-12-02 19:59:02 +01:00
Maximilian Hils
f02da6448d disable concurrency tests on appveyor 2015-11-29 03:13:08 +01:00
Maximilian Hils
bb62f5f3f5 lessen test race conditions 2015-11-29 02:59:25 +01:00
Maximilian Hils
6b92497d4f remove unnecessary race conditions in tests 2015-11-27 01:11:06 +01:00
Maximilian Hils
abd8881491 add memoryleak debug script 2015-11-26 23:26:01 +01:00
Maximilian Hils
354f84c4c8 skip timing tests on appveyor 2015-11-24 19:07:04 +01:00
Maximilian Hils
57cd9022d9 fix appveyor race conditions in tests 2015-11-14 22:14:18 +01:00
Maximilian Hils
4499ab61c0 structure libmproxy.script 2015-11-14 04:21:38 +01:00
Maximilian Hils
dce469d4c1 Merge pull request #833 from zbuc/contentview_scripts 
Contentview scripts
 2015-11-13 18:41:05 -08:00
Chris Czub
e72a9a62a1 Feedback from PR #832 2015-11-13 16:55:27 -05:00
Chris Czub
d3feaa3bc6 Add custom content view plugin support for mitmproxy/mitmdump 2015-11-13 15:09:05 -05:00
Chris Czub
d7239d665e Add new contentviews' content type mappings 2015-11-13 15:08:52 -05:00
Chris Czub
82a90511bc Add "add" ContentView method & test 2015-11-13 15:07:56 -05:00
Chris Czub
b0648ee8a6 Fix failing protobuf contentview test 2015-11-06 11:18:48 -05:00
Maximilian Hils
07b60acb7f try to fix appveyor race conditions in tests 2015-11-01 18:44:50 +01:00
Maximilian Hils
c6811bd0e8 fix #773 2015-09-28 14:55:13 +02:00
Maximilian Hils
c11ab3676d Merge branch 'http-models' 2015-09-28 13:54:53 +02:00
Maximilian Hils
6661770d4e handle Expect: 100-continue header, fix #770 2015-09-28 10:59:10 +02:00
Maximilian Hils
6075957a97 move tests to netlib 2015-09-27 00:50:14 +02:00
Maximilian Hils
1b6ea5caf3 adjust to netlib response changes + docs 2015-09-26 17:41:14 +02:00
Maximilian Hils
b13acd7956 adjust to netlib request changes + docs 2015-09-26 01:23:59 +02:00
Maximilian Hils
bba5ec7144 experimental: run tests using py.tests 2015-09-21 18:31:30 +02:00
Maximilian Hils
88375ad64a fix minor bugs, add py.test compatibility 2015-09-21 02:26:47 +02:00
Maximilian Hils
2d4a5fb116 experimental: don't interfere with headers, fix #93 2015-09-18 18:07:38 +02:00
Maximilian Hils
906360f63f code -> status_code, httpversion -> http_version 2015-09-17 15:16:33 +02:00
Maximilian Hils
14a5f405fd use new netlib exceptions 2015-09-17 02:13:28 +02:00
Maximilian Hils
0af0608978 adjust to netlib changes 2015-09-16 18:45:22 +02:00
Maximilian Hils
32b487109f Merge pull request #765 from mitmproxy/such-colors-very-wow 
Improve Content Views
 2015-09-12 17:57:21 +02:00
Maximilian Hils
4106e19612 relax test suite time tests 2015-09-12 17:50:24 +02:00
Maximilian Hils
5fe12a467f fix tests 2015-09-12 17:40:30 +02:00
Maximilian Hils
049d253a83 simplify contentview api 2015-09-12 13:49:16 +02:00
Maximilian Hils
9c31669211 mitmdump: colorize output, add content views 2015-09-11 19:03:50 +02:00
Maximilian Hils
47602dc1a5 clean up tests 2015-09-11 13:41:16 +02:00
Maximilian Hils
960f62f363 fix bugs 2015-09-11 13:37:52 +02:00
Maximilian Hils
7c186a4edb Merge branch 'master' into contentviews 2015-09-11 12:13:39 +02:00
Maximilian Hils
c159c8ca13 fix chunked encoding 2015-09-11 01:18:17 +02:00
Maximilian Hils
a8d931089c fix tests 2015-09-05 20:53:44 +02:00
Maximilian Hils
5125c669cc adjust to new netlib Headers class 2015-09-05 20:45:58 +02:00
Maximilian Hils
f4272de5ec remove ServerConnectionMixin.reconnect 2015-09-03 17:01:25 +02:00
Chris Czub
b4e0be9052 PR #700 
Start moving the contentview up a level
Started shifting the contentview class up to libmproxy
 2015-09-02 20:30:58 +02:00
Maximilian Hils
481cc6ea84 we don't support socks auth, refs #738 2015-08-31 17:29:14 +02:00
Maximilian Hils
b04e6e56ab update inline script hooks 2015-08-31 17:05:52 +02:00
Maximilian Hils
a86ec56012 move files around 2015-08-30 15:27:29 +02:00
Maximilian Hils
421b241ff0 remove http2http references 2015-08-30 13:40:23 +02:00
Maximilian Hils
1dd09a5509 always insert tls layer for inline script upgrades 2015-08-30 02:27:38 +02:00
Maximilian Hils
dd7f50d64b restructure code, remove cruft 2015-08-30 01:21:58 +02:00
Maximilian Hils
a7058e2a3c fix bugs, fix tests 2015-08-29 20:53:25 +02:00
Maximilian Hils
63844df343 fix streaming 2015-08-29 14:28:11 +02:00
Maximilian Hils
1cc48345e1 clean up config/cmdline, fix bugs, remove cruft 2015-08-28 01:51:13 +02:00
Maximilian Hils
83decd6771 fix inline script redirects 2015-08-27 17:35:53 +02:00
Maximilian Hils
56a4bc381e request -> request_method 2015-08-24 18:17:04 +02:00
Maximilian Hils
f1f34e7713 fix bugs, fix tests 2015-08-24 16:52:03 +02:00
Thomas Kriechbaumer
05d26545e4 adapt netlib changes 2015-08-21 10:26:28 +02:00
Maximilian Hils
96de7ad562 various fixes 2015-08-18 14:15:08 +02:00
Thomas Kriechbaumer
891fa50e55 move code to netlib 2015-08-10 21:04:17 +02:00
Thomas Kriechbaumer
6cce2f14d7 fix tests 2015-08-02 11:26:33 +02:00
Thomas Kriechbaumer
ff1e7e6409 fix imports for encoding 2015-08-01 14:53:34 +02:00
Thomas Kriechbaumer
fa568c2d3b move encoding tests from mitmproxy to netlib 2015-08-01 14:53:20 +02:00
Thomas Kriechbaumer
db02553e2a move code from mitmproxy to netlib 2015-08-01 14:37:32 +02:00
Thomas Kriechbaumer
a5d9e1f44d move code to netlib and implement protocols 2015-07-30 13:53:17 +02:00
Thomas Kriechbaumer
89f22f7359 refactor connection & protocol handling 2015-07-30 13:53:17 +02:00
Maximilian Hils
d2ae6b630e Merge pull request #548 from macmantrl/no_serverconn 
Prevent unnecessary upstream server connects
 2015-07-26 11:58:21 +02:00
Thomas Kriechbaumer
07c9d46fab remove debug output in tests 2015-07-24 18:01:55 +02:00
Maximilian Hils
0892649d31 Merge pull request #675 from Kriechi/protocol-refactor 
HTTP protocol refactoring
 2015-07-24 17:52:44 +02:00
Maximilian Hils
e0d1e8caf6 fix tests 2015-07-24 03:09:08 +02:00
Maximilian Hils
7bf8088d80 improve inline scripts, fix #683, fix #684 2015-07-24 02:57:56 +02:00
Thomas Kriechbaumer
cedc0407de refactor to use netlib.http protocols 2015-07-22 15:30:12 +02:00
Thomas Kriechbaumer
bb265d0c40 use new netlib module names 2015-07-22 15:30:11 +02:00
Thomas Kriechbaumer
df1b0df39f use netlib.http_semantics for generic data 2015-07-22 15:30:11 +02:00
isra17
471e196e08 Add ~src ~dst REGEX filters 
This filter allow to match on the request source and destination address
in the form of `<IP>:<Port>`.

Also fixed the parsing grammar to add a `WordEnd` after each filter
name. That way, `~src` doesn't match `~s` instead and keep the behavior
consistent with `~hq` != `~h`.
 2015-07-13 18:11:14 -04:00
Terry Long
4c50c36345 Prevent unecessary upstream server connects 
Selectively connect to upstream server based on no_upstream_cert
option. When no_upstream_cert is used during server replay, prevent
connecting to the upstream server unless absolutely necessary.
 2015-07-03 07:55:37 -07:00
Maximilian Hils
4c831992aa fix #593, fix #656, coverage++ 2015-07-03 02:47:12 +02:00
Maximilian Hils
9bffd9cf03 fix windows compatibility 2015-07-03 02:46:06 +02:00
Kyle Morton
f0ad1f334c Enabling upstream server verification. Added flags --verify_upstream_cert, 
--upstream-trusted-cadir, and --upstream-trusted-ca.
 2015-06-29 11:00:20 -07:00
Maximilian Hils
b369962cbe remove certforward feature 
The certforward feature was implemented to support #gotofail,
which only works on unpatched iOS devices. Given that many apps don't
support iOS 7 anymore, jailbreak+ssl killswitch is usually the better option.
By removing certforward, we can make netlib a pure python module again,
which significantly simplifies distribution.
 2015-06-26 13:27:40 +02:00
iroiro123
fbb23b5c9f changed error handling (ssl spoof mode) 2015-06-23 01:49:22 +09:00
iroiro123
5c7fa7a594 add unit tests for spoof mode 2015-06-23 00:57:33 +09:00
Aldo Cortesi
7890450b0c Handle invalid IDNA encoding in hostnames 
Fixes #622
 2015-06-12 16:00:16 +12:00
Aldo Cortesi
57a61ae8fd console: convert add_event to a signal. 2015-06-02 13:45:19 +12:00
Maximilian Hils
ec92fca8c0 Merge branch 'master' of github.com:mitmproxy/mitmproxy 2015-05-31 01:51:31 +02:00
Maximilian Hils
0f8e8ee1ad add bench tool for windows 2015-05-31 01:51:13 +02:00
Aldo Cortesi
a05a70d816 Add coding style check, reformat. 2015-05-30 12:03:28 +12:00
Aldo Cortesi
fcbfe4b605 Adapt to new literal escaping style in pathod 2015-05-03 10:18:18 +12:00
Aldo Cortesi
50b7bfaadc Adapt to new Pathoc API - silence output to stdout. 2015-05-02 16:46:15 +12:00
Aldo Cortesi
1c26516b18 pretty_size now lives in netlib.utils 2015-04-30 12:18:01 +12:00
Aldo Cortesi
e9ae0b92ba Fudge a rare race condition in the test suite 2015-04-19 18:03:50 +12:00
Aldo Cortesi
0f269f7423 Whitespace, formatting 2015-04-15 09:14:20 +12:00
Aldo Cortesi
c335c2b533 Add set_cookies method to HTTPResponse 2015-04-14 16:23:51 +12:00
Aldo Cortesi
ab7e2857cc New get_cookies for HttpResponse 2015-04-14 15:14:36 +12:00
Aldo Cortesi
e17eacd8d7 New get_cookie and set_cookie implementations for HTTPRequest 2015-04-14 13:45:38 +12:00
Aldo Cortesi
937a358aa9 Merge branch 'master' of ssh.github.com:mitmproxy/mitmproxy 2015-04-14 12:01:37 +12:00
Aldo Cortesi
bea0bd236a Housekeeping and cleanups 
- No output to stdout on load in examples - they muck up the test suite.
- Use the odict module directly, rather than aliasing it. The small convenience
this gives to scripters is not worth it.
- Move the cookie tests from the flow test module to the protocol_http test
module.
 2015-04-14 11:58:10 +12:00
Maximilian Hils
c0a318566a add flowwriter example 2015-04-12 03:47:58 +02:00
Maximilian Hils
923503260e add tests 2015-04-10 19:42:32 +02:00
Maximilian Hils
6852eb9d0a fix #553 2015-04-10 14:59:38 +02:00
Maximilian Hils
f37efecd0a add yappi benchmark tool 2015-04-08 20:43:59 +02:00
Maximilian Hils
1a666bb5db Merge pull request #547 from macmantrl/blank_values 
Keep blank query values for flow hashing
 2015-04-08 01:22:23 +02:00
Maximilian Hils
f2ad047966 fix tests 2015-04-08 00:33:31 +02:00
Aldo Cortesi
f6a3bd15b3 console: All palettes now explicitly set a background colour 
There's a new option --palette-transparent to turn this off if you want
to use your own terminal background.
 2015-04-07 10:01:18 +12:00
Terry Long
6a24a4e3c5 Keep blank query values for flow hashing 2015-04-02 22:10:01 -07:00
Aldo Cortesi
82997cb311 Fix timestamps in detail view 
- Fix a crash when connection timestamps don't exist yet
- Fix display of response timestamps
- Get rid of those colossal ternaries. I want a device that pokes people in the
eye every time they try to use a ternary operator.
 2015-03-31 09:49:07 +13:00
Aldo Cortesi
bf012e0a89 Zap old search tests 2015-03-29 14:49:57 +13:00
Maximilian Hils
423530fc25 Merge pull request #515 from krzysiekbielicki/master 
[#514] Add support for ignoring payload params in multipart/form-data
 2015-03-26 18:16:31 +01:00
Aldo Cortesi
842e23d3e3 Replace far-too-clever decorator LRU cache with something simpler 2015-03-22 21:00:41 +13:00
Aldo Cortesi
a1c21d9774 Fix unit tests 2015-03-22 17:23:27 +13:00
Aldo Cortesi
89383e9c13 Refactor status bar prompting to use signal system 2015-03-22 13:32:24 +13:00
Aldo Cortesi
560e44c637 Pull PathEdit out into its own file. 2015-03-20 10:54:57 +13:00
Aldo Cortesi
b47019293e Nicer Solarized palettes, using the correct terminal values 2015-03-19 20:49:43 +13:00
Aldo Cortesi
05d1b99043 No output to stdout in the test suite, please. 2015-03-19 18:10:21 +13:00
Aldo Cortesi
1d5fcc6e0e Revamp palette specification 
- Split low-color and high-color specifications in palettes.
- Split off light, dark, lowlight and lowdark palettes. Lowlight and lowdark
will be the low-color base for most subsequent palettes.
- Add a small script that makes test pattern requests to pathod.
 2015-03-19 18:05:30 +13:00
Marcelo Glezer
7c75f4d0db test/test_console_help.py fixed 2015-03-18 13:49:33 -03:00
Krzysztof Bielicki
953f9aa641
Added tests 2015-03-16 10:23:50 +01:00
Maximilian Hils
3650d73662 Merge remote-tracking branch 'onlywade/master' 
Conflicts:
	libmproxy/console/__init__.py
	libmproxy/console/flowlist.py
 2015-03-15 14:36:06 +01:00
Aldo Cortesi
de3f84934d Test suite should pass even if example dependencies are not present 2015-03-12 16:57:10 +11:00
Wade Catron
803d631f04 Adding a server replay option to ignore host when searching for matching requests 2015-03-08 10:21:57 -07:00
Marcelo Glezer
58dba3f490 fixed formatting and added a 'test' (sort of) 2015-03-02 23:22:44 -03:00
Maximilian Hils
e65a8659f0 Merge pull request #489 from tekii/feature-487 
added time elapsed between request sent and response received
 2015-03-02 14:40:22 +01:00
Maximilian Hils
e1b6cf9401 fix #319 2015-02-27 15:24:27 +01:00
Maximilian Hils
81a274eb51 fix #479 2015-02-27 09:17:41 +01:00
Marcelo Glezer
3a78c95d0a added to flowlist / flowdetail time elapsed between request sent and response received 2015-02-26 18:14:20 -03:00
Maximilian Hils
5d56870818 disable replay from ui for live flows 2015-02-07 19:33:36 +01:00
Maximilian Hils
53fadd05f4 fix #451 2015-02-07 16:26:19 +01:00
Maximilian Hils
2ed09beae4 fix tests, re-enable pypy, do housekeeping 2015-02-07 01:17:24 +01:00
Maximilian Hils
6472942756 fix tests 2015-02-05 17:52:59 +01:00
Maximilian Hils
0ac3227b7b clean up flow reading 2015-02-05 14:44:45 +01:00
Marcelo Glezer
27950f1972 Merge remote-tracking branch 'base/master' 2015-01-12 10:55:47 -03:00
Maximilian Hils
8af26bd0b7 fix #443 2015-01-09 16:40:26 +01:00
Marcelo Glezer
9695377b28 first steps in scenario support 2015-01-05 18:12:38 -03:00
Aldo Cortesi
f1b040e808 Fix unit tests 2015-01-02 16:12:25 +13:00
Aldo Cortesi
1b5f5021dc Consolidate reading flows from file, use in mitmweb. 2015-01-02 13:26:22 +13:00
Aldo Cortesi
8a8437470e Move onboarding app to Tornado 
Two reasons for this. First, this removes flask and its dependencies, which are
quite sizeable. Second, pyinstaller now barfs on simplejson, which is a Flask
dependency. I just don't have time to fix this upstream, so doing what we
should be doing anyway is a no-brainer.
 2014-12-27 23:06:51 +13:00
Maximilian Hils
f265cff3a9 fix tests, experimental travis speedup 2014-12-24 01:39:38 +01:00
Maximilian Hils
2861d99de4 web: intercept feature 2014-12-23 20:33:42 +01:00
Marcelo Glezer
7e1b35bfc7 --replay_ignore_payload_params added, to filter params in form posts 2014-12-18 17:56:27 -03:00
Aldo Cortesi
01fa5d3f07 Merge pull request #414 from mitmproxy/flowviews2 
Flowviews2
 2014-12-12 22:08:15 +13:00
Marcelo Glezer
14a3e0073f append file mode for saving flows merged to master 2014-12-11 15:21:33 -03:00
Maximilian Hils
a7a9ef826c fix tests 2014-12-01 03:36:04 +01:00
Maximilian Hils
ed8249023f introduce revised views, port over changes from multiple_views branch 2014-11-26 04:18:21 +01:00
Maximilian Hils
667fe0c20b fix tests 2014-11-15 23:10:25 +01:00
Aldo Cortesi
09c503563a Enable config file parsing 
We support 4 different config files:

~/.mitmproxy/common.conf:       Options that are common to all tools
~/.mitmproxy/mitmproxy.conf:    Options for mitmproxy
~/.mitmproxy/mitmdump.conf:     Options for mitmdump
~/.mitmproxy/mitmweb.conf:      Options for mitmweb

Options in the tool-specific config files over-ride options in common.conf. If
a non-common option is put in common.conf, an error will be raised if a
non-supporting tool is used.
 2014-11-15 17:25:05 +13:00
Aldo Cortesi
9c88622e25 Adjust tests 2014-11-15 16:17:05 +13:00
Maximilian Hils
0c52b4e3b9 handle script hooks in replay, fix tests, fix #402 2014-11-14 00:26:22 +01:00
Maximilian Hils
a325ae638b fix tests 2014-11-11 13:09:05 +01:00
Maximilian Hils
4d090e09c7 fix tests 2014-11-07 09:59:11 +01:00
Aldo Cortesi
0fe83ce87b Fix bug in flow dumping, add unit test that should have caught this in the first place 2014-11-06 10:35:00 +13:00
Aldo Cortesi
34d419ead8 Merge branch 'master' of ssh.github.com:mitmproxy/mitmproxy 2014-11-04 10:22:13 +13:00
Aldo Cortesi
d0de490ef1 Release prep: binaries build script, release checklist, fuzzing 2014-11-02 18:04:57 +13:00
Wade 524
c4c42fa040 Updating OPTIONS test with related issue number. 2014-10-31 12:45:31 -07:00
Wade 524
2c64b90a3d Adding some test coverage for handling HTTP OPTIONS requests. 2014-10-31 11:49:45 -07:00
Aldo Cortesi
340d0570bf Legibility 2014-10-26 18:32:45 +13:00
Aldo Cortesi
16654ad6a4 Fix crash while streaming 
Found using fuzzing. Reproduction with pathoc, given "mitmproxy -s" and
pathod running on 9999:

get:'http://localhost:9999/p/':s'200:b\'foo\':h\'Content-Length\'=\'3\'':i58,'\x1a':r

 return flow.FlowMaster.run(self)
   File "/Users/aldo/mitmproxy/mitmproxy/libmproxy/controller.py", line 111, in run
     self.tick(self.masterq, 0.01)
   File "/Users/aldo/mitmproxy/mitmproxy/libmproxy/flow.py", line 613, in tick
     return controller.Master.tick(self, q, timeout)
   File "/Users/aldo/mitmproxy/mitmproxy/libmproxy/controller.py", line 101, in tick
     self.handle(*msg)
   File "/Users/aldo/mitmproxy/mitmproxy/libmproxy/controller.py", line 118, in handle
     m(obj)
   File "/Users/aldo/mitmproxy/mitmproxy/libmproxy/flow.py", line 738, in handle_responseheaders
     self.stream_large_bodies.run(f, False)
   File "/Users/aldo/mitmproxy/mitmproxy/libmproxy/flow.py", line 155, in run
     r.headers, is_request, flow.request.method, code
   File "/Users/aldo/mitmproxy/mitmproxy/netlib/http.py", line 401, in expected_http_body_size
     raise HttpError(400 if is_request else 502, "Invalid content-length header: %s" % headers["content-length"])
 netlib.http.HttpError: Invalid content-length header: ['\x1a3']
 2014-10-26 17:58:36 +13:00
Aldo Cortesi
efd6fdb0e2 Start a fuzzing architecture for mitmproxy 2014-10-26 17:13:25 +13:00
Maximilian Hils
c79bdeb4e5 fix #346 2014-10-22 14:03:19 +02:00
Maximilian Hils
37cc6ae0bb fix race conditions in tests 2014-10-19 01:26:08 +02:00
Maximilian Hils
e114858438 add generic tcp proxying, fix #374 2014-10-18 18:29:35 +02:00
Maximilian Hils
d7341e7798 add test case for #375 2014-10-17 17:08:41 +02:00
Marcelo Glezer
81f5636389 --replay-ignore-content & --replay-ignore-param ported from branch 0.10 2014-10-03 07:29:44 -03:00
Aldo Cortesi
d998790c2f Clean up and clarify StateObject 
- Flatten the class hierarchy
- get_state, load_state, from_state are public
- Simplify code
- Remove __eq__ and __neq__. This fundamentally changes the semantics of
inherited objects in a way that's not part of the core function of the
class
 2014-09-17 11:41:42 +12:00
Aldo Cortesi
17868f61a9 Remove external app option, make tick timeout explicit 2014-09-14 11:46:01 +12:00
Maximilian Hils
2842f25b82 fix tests 2014-09-12 03:16:52 +02:00
Maximilian Hils
5f5af9535a update distribution: pip install now works on Windows 2014-09-09 00:06:10 +02:00
Maximilian Hils
fbc4bb38df mitmproxy -> libmproxy.main. do some leanup on the way 2014-09-08 23:34:43 +02:00
Maximilian Hils
ece15b3c8a reverse proxy: adjust dst when reading flows, fix #346 2014-09-08 14:43:32 +02:00
Maximilian Hils
d06b4bfa4e remove proxy mode abstraction: always be clear which mode we are in 2014-09-08 14:32:42 +02:00
Maximilian Hils
2b2d21aff0 add --ignore ui indication 2014-09-08 12:20:40 +02:00
Maximilian Hils
ac27d1236f improve change_server api, add example how to change the upstream server 2014-09-07 18:01:37 +02:00
Aldo Cortesi
bf5fef1e0b Merge pull request #347 from mitmproxy/issue_341 
Remove BackReferenceMixin
 2014-09-07 13:04:18 +12:00
Maximilian Hils
df129736c3 test streaming 2014-09-06 13:30:00 +02:00
Maximilian Hils
3c65510ef5 coverage++ 2014-09-06 13:09:57 +02:00
Maximilian Hils
fdd7b2f108 Merge branch 'issue_341' into server_change_api 2014-09-06 12:43:32 +02:00
Maximilian Hils
0e0cff638c Merge branch 'master' into issue_341 2014-09-06 12:42:52 +02:00
Maximilian Hils
b6c8a22b67 coverage++ 2014-09-06 12:23:05 +02:00
Maximilian Hils
2a6337343a update docs, mostly revert 2f44b26b4c 2014-09-05 15:16:20 +02:00
Maximilian Hils
f2570c773a iframe injector example: use inline script 2014-09-05 15:05:44 +02:00
Maximilian Hils
a7a3b5703a change replay_request behaviour, refs #346; test upstream proxy mode 2014-09-05 00:18:17 +02:00
Maximilian Hils
b23a1aa4a4 much tests. so tcp. very wow. 2014-09-04 19:08:54 +02:00
Maximilian Hils
795e19f6b7 coverage++ 2014-09-04 16:37:50 +02:00
Maximilian Hils
f4d4332472 coverage++ 2014-09-04 14:46:25 +02:00
Maximilian Hils
1d45c54a04 Merge branch 'issue_341' into server_change_api 2014-09-04 14:18:21 +02:00
Maximilian Hils
1135666ee6 Merge branch 'master' into issue_341 2014-09-04 14:18:07 +02:00
Maximilian Hils
39fa579dd5 remove outdated IOError test case (f5fdfd8a9f) 2014-09-04 14:15:49 +02:00
Maximilian Hils
2f44b26b4c improve HTTPRequest syntax 2014-09-03 23:44:54 +02:00
Maximilian Hils
0c3b6ee667 fix IOError handling 2014-09-03 17:33:08 +02:00
Maximilian Hils
b0cfeff06d fix #341 - work on flows instead of request/response internally. 2014-09-03 16:57:56 +02:00
Maximilian Hils
1e4e332ef9 improve error handling 2014-09-02 18:13:18 +02:00
Maximilian Hils
1f47f7b6b2 refactor protocol handling, fix #332 2014-08-30 20:15:19 +02:00
Maximilian Hils
b97b1f17cf fix #328 2014-08-24 14:22:11 +02:00
Maximilian Hils
94fbf066f7 fix script tests 2014-08-18 01:47:39 +02:00
Maximilian Hils
bb2ce689a9 adjust to netlib changes 2014-08-17 23:06:25 +02:00
Maximilian Hils
a193c267f3 adjust tests 2014-08-10 16:20:04 +02:00
Maximilian Hils
4d2109ef92 fix display of urls in upstream mode, refs #308 2014-08-08 17:49:02 +02:00
Maximilian Hils
de05484d9d fix #318 2014-08-08 04:43:44 +02:00
Maximilian Hils
a3c3e4e504 remove platform.setup 2014-08-08 03:38:59 +02:00
Maximilian Hils
13f030ccb5 suppress SPDY/HTTP2 announcement headers, fix #277 2014-08-03 16:25:38 +02:00
Maximilian Hils
3133136da7 fix #307 2014-08-03 02:34:29 +02:00
Maximilian Hils
e9401a2123 Merge branch 'stream' 
Conflicts:
	libmproxy/protocol/http.py
 2014-07-31 22:29:13 +02:00
Maximilian Hils
3e3dbee936 fix tests 2014-07-31 22:18:57 +02:00
Maximilian Hils
c9be57b682 fix #313 2014-07-27 03:28:23 +02:00
Maximilian Hils
e9c834a30d fix #259 2014-07-26 12:02:18 +02:00
Maximilian Hils
a73ad1d945 fix tests 2014-07-26 11:51:27 +02:00
Maximilian Hils
562ac9e721 unify stream handling 2014-07-21 14:09:24 +02:00
Maximilian Hils
7398db80db simplify responseheader scripthook 2014-07-20 11:17:53 +02:00
Brad Peabody
560e23af09 fixed handling of Transfer-Encoding header during streaming; wrote tests for streaming support 2014-07-19 19:10:14 -07:00
Maximilian Hils
a48cccadb5 fix tests on windows 2014-07-18 03:13:52 +02:00
Maximilian Hils
b0c366aa45 Merge pull request #300 from zbrdge/freebsd-platform 
Freebsd 10 platform
 2014-07-03 02:24:00 +02:00
Zack B
64c265b710 added unit test update. coverage still at 97% 2014-07-01 16:33:48 -07:00
Vyacheslav Bakhmutov
b7c1d05782 Remove global should_exit and fix tests 2014-06-13 14:14:55 +07:00
Maximilian Hils
00fd243810 fix up error messages 2014-05-15 18:16:42 +02:00
Maximilian Hils
a17a53269d fix #253 2014-05-15 14:37:05 +02:00
Maximilian Hils
d883d93566 fix linux tests... 2014-03-13 01:30:26 +01:00
Maximilian Hils
1bfc164692 adjust tests, fix option handling 2014-03-13 01:22:12 +01:00
Maximilian Hils
aefb30ea60 restrict -e to mitmproxy only (not mitmdump) 2014-03-12 22:13:00 +01:00
Maximilian Hils
14db30080f Merge branch 'master' of github.com:mitmproxy/mitmproxy 2014-03-11 02:16:30 +01:00
Maximilian Hils
15c82f743f fix temporary server change, coverage++ 2014-03-11 02:16:22 +01:00
Aldo Cortesi
e89e035d4a Certificate forwarding. 2014-03-11 13:02:10 +13:00
Maximilian Hils
fe58c1c6eb add advanced proxying options, add SSL-terminating capability to mitmproxy 2014-03-10 05:11:51 +01:00
Maximilian Hils
78750a8b4d lay the foundations for --(in|out)(abs|rel) command line switches, as proposed in https://groups.google.com/forum/#!topic/mitmproxy/nApno2TXS0c 2014-03-10 02:32:27 +01:00
Maximilian Hils
5598a8de82 finish proxy.py split up 2014-03-09 21:51:24 +01:00
Maximilian Hils
fc4fe83eaf split up proxy.py 2014-03-09 21:13:08 +01:00
Maximilian Hils
2e50b10735 fix windows tests, include improvements from 0.10.1 2014-03-08 17:19:47 +01:00
Maximilian Hils
2b01c4eee7 Merge branch 'master' of github.com:mitmproxy/mitmproxy 2014-03-08 15:47:27 +01:00
Maximilian Hils
3032672f10 combine asterisk-form and origin-form into relative form 2014-03-08 15:47:09 +01:00
Aldo Cortesi
221973aff6 Revert to old cert names, use a subdir for test conf dir. 2014-03-07 16:38:24 +13:00
Aldo Cortesi
d65f2215cb Much more sophisticated cert handling 
- Specify per-domain certificates and keys
- Certs are no longer regenerated for SANs
- And more. :)
 2014-03-05 17:28:04 +13:00
Aldo Cortesi
32af668814 Minor cleanups. 2014-03-02 22:15:53 +13:00
Aldo Cortesi
f373ac5b6c Improve explicit certificate specification 
- Support cert/key in the same PEM file
- Rationalize arguments, expand tests, clean up a bit
 2014-03-02 17:27:24 +13:00
Aldo Cortesi
863b1e1455 Update for pathod.SSLOptions changes. 2014-03-02 15:58:53 +13:00
Aldo Cortesi
a34a483184 Adapt for new pathod and netlib APIs. 2014-03-02 15:14:22 +13:00
Pedro Worcel
cffae49e34 add helpful messages on wrap 2014-02-22 18:04:56 +13:00
Pedro Worcel
3e50034428 fix the wrapping on backward searches 2014-02-22 17:15:37 +13:00
Pedro Worcel
4284fd3614 add multi-line support to backwards search 2014-02-22 16:32:22 +13:00
Pedro Worcel
3c02865e8b add multi-line support 2014-02-22 15:32:35 +13:00
Pedro Worcel
7a154e1ae1 add basic backwards search 2014-02-22 15:00:10 +13:00
Aldo Cortesi
a85974eaa8 Fix flow initialization order error 
Resolves #210
 2014-02-08 17:17:35 +13:00
Aldo Cortesi
fe4bab2e6f Cleanup: fix unit tests, simpler app test, remove print statements 2014-02-08 12:33:59 +13:00
Maximilian Hils
33b10da57c fix tests 2014-02-07 18:26:42 +01:00
Maximilian Hils
735e4400c4 add tests for reconnect to upstream proxy, ensure that server_reconnect is always hooked 2014-02-07 18:14:15 +01:00
Maximilian Hils
545fc2506b Achievement Unlocked: Proxy Chain 2014-02-07 07:08:59 +01:00
Maximilian Hils
9526c5d565 fix race conditions in test suite 2014-02-07 04:15:24 +01:00
Maximilian Hils
d07029d575 coverage++ 2014-02-07 03:56:57 +01:00
Maximilian Hils
9f5f2b7071 test libmproxy.app, increase coverage 2014-02-07 02:36:39 +01:00
Maximilian Hils
cb397ec788 fix bugs, expose timestamp information to console ui 2014-02-06 00:53:39 +01:00
Maximilian Hils
80683e77bc mock urwid to run console tests on windows, push failing testcases down to 0. :-) 2014-02-05 23:17:26 +01:00
Maximilian Hils
f26d91cb81 add skeleton to change destinatin server during intercept, fix all testcases on windows 2014-02-05 20:26:47 +01:00
Maximilian Hils
d864a326d2 fix all tests except those using set_url and get_url 2014-02-05 14:33:17 +01:00
Maximilian Hils
6a53ae5fd3 push failing tests down to 43 2014-02-04 05:02:17 +01:00
Maximilian Hils
2db5f9de26 bring failing tests back into the double-digit range 2014-01-31 04:45:39 +01:00
Maximilian Hils
7d96ff00ef remove _flow_map from state, improve logging 2014-01-31 03:01:51 +01:00
Maximilian Hils
607f777811 fix most testsuite errors, improve proxy error handling 2014-01-29 02:49:11 +01:00
Maximilian Hils
1e3e0dd127 merge master 2014-01-28 17:29:28 +01:00
Maximilian Hils
17f09aa0af unify ipv4/ipv6 address handling 2014-01-28 17:28:20 +01:00
Maximilian Hils
125166020b move test requirements out of main requirements.txt 2014-01-28 03:09:21 +01:00
Aldo Cortesi
069aca1df4 Deprecate Flow.unload_script in favor of Flow.unload_scripts 
In the presence of arguments, unloading a single script is clumsy and
less useful.
 2014-01-13 14:20:34 +13:00
Aldo Cortesi
4f69eef8f3 Extract command parsing and use in script grid editor 2014-01-13 14:15:17 +13:00
Aldo Cortesi
42d4a2fae9 Script refactoring: move script command parsing into script module. 2014-01-12 23:02:29 +13:00
Aldo Cortesi
e5776b8be3 Clean up and clarify script API 2014-01-12 13:59:32 +13:00
Aldo Cortesi
58e1b3a47f Start refactoring scripts 
- Move ScriptContext into script module
- Use mock module instead of hand-rolled mock objects in tests
 2014-01-12 12:49:19 +13:00
Maximilian Hils
2f63da99c9 ignore console test utilities on windows 2014-01-10 01:39:16 +01:00
Aldo Cortesi
d5f9b02615 Merge pull request #192 from mitmproxy/refactor_read_http_body 
move CONTINUE checks into mitmproxy
 2014-01-04 14:37:47 -08:00
Aldo Cortesi
7d37e0ce10 Merge pull request #193 from droope/search-functionality 
Search functionality
 2014-01-04 14:34:20 -08:00
Aldo Cortesi
ac31039ad3 Merge pull request #198 from Kami/add_css_view 
Add CSS view which beautifies minified CSS files
 2014-01-04 14:04:02 -08:00
Tomaz Muraus
e9f6302ec7 Add CSS view which beautifies CSS files if cssutils library is available, 
otherwise it acts as a no-op.
 2014-01-04 04:49:50 +01:00
Aldo Cortesi
45eab17e0c Decouple message type from message class name. 2014-01-04 14:42:32 +13:00
Aldo Cortesi
1e07d9e6e7 Move app mechanism to flow.py 
Disable apps while message passing is improved.
 2014-01-04 14:35:11 +13:00
Pedro Worcel
799c877676 now really fix it + test 2013-12-26 22:18:34 +13:00
Pedro Worcel
21efe2f2c8 add looping around 2013-12-26 17:04:18 +13:00
Pedro Worcel
9cf8a1a89d fix failing test 2013-12-25 21:08:20 +13:00
root
95406bd119 Add focusing, and fixes non-clearance of prev searches. Add 
documentation.
 2013-12-25 16:50:29 +13:00
root
932464d0a0 test passing, UI still not working 2013-12-24 14:28:20 +13:00
Maximilian Hils
e643759ef6 merge smurfix/fix2, add serverconnect hook docs, adjust tests 2013-12-16 22:10:06 +01:00
Maximilian Hils
f4b58ba495 move CONTINUE checks into mitmproxy 2013-12-15 06:33:18 +01:00
Maximilian Hils
6ec2e6f24f fix script path error in tests 2013-12-15 02:51:35 +01:00
Maximilian Hils
605950bfdf add @concurrent decorator for inline scripts, fixes #176 2013-12-15 02:43:16 +01:00
Maximilian Hils
39ffe10334 add content-length 0 if we remove header for chunked encoding, fixes #186 2013-12-12 04:42:29 +01:00
Maximilian Hils
7db1430ee7 ignore missing CN in certificates. fixes #169 2013-12-12 03:24:17 +01:00
Maximilian Hils
28a234e28b store resolved ip addresses, fixes #187 2013-12-12 02:11:22 +01:00
Maximilian Hils
3bd238de37 backport changes to fix tests 2013-12-08 15:46:11 +01:00
Maximilian Hils
a509a9037b Merge branch 'master' into 0.10 2013-12-08 14:14:57 +01:00
Aldo Cortesi
3a1d85ab18 Merge pull request #134 from mhils/scripts_improvements 
Support multiple scripts and script arguments. refs #76
 2013-12-08 01:14:12 -08:00
Aldo Cortesi
7aeaf9d448 Merge pull request #161 from mitmproxy/external_webapp 
External webapp
 2013-12-08 01:09:46 -08:00
Aldo Cortesi
cb3a4a0f3c Remove obsolete test. 2013-12-08 21:58:55 +13:00
Maximilian Hils
2956c144d3 Merge branch 'master' into 0.10 2013-09-14 23:47:04 +02:00
Aldo Cortesi
aaba940dea Fix pretty_size unit tests. 2013-08-23 10:28:16 +12:00
Maximilian Hils
bb4748fb8f add option to expose webapp externally, remove distinct ip setting 2013-08-18 20:03:53 +02:00
Maximilian Hils
729677cd85 Merge branch 'master' into 0.10 2013-08-17 13:30:36 +02:00
Aldo Cortesi
f850bdd848 Revamp dummy cert store 
We no longer keep these on disk. This is for a number of reasons, including
some race conditions and the fact that some valid IDNA-encoded domain names are
not valid file names on Windows.
 2013-08-12 16:04:02 +12:00
Aldo Cortesi
439d9a294c Make use of a change to netlib.tcp that clarifies error conditions for flush and close. 
Should fix #144.
 2013-07-30 09:42:29 +12:00
Aldo Cortesi
10a9e3365f Some refactoring of the console inteface. 
Reduce some state duplication, by removing currentflow variable.

Fixes #141
 2013-07-28 18:00:49 +12:00
Aldo Cortesi
5c1157ddaf Move app instantiation out of proxy.py. 2013-07-24 10:32:56 +12:00
Maximilian Hils
6fe175913e fix request_timestamps test case 2013-06-18 00:54:07 +02:00
Maximilian Hils
ca9740df20 skip tests on windows that don't make sense 2013-06-18 00:53:38 +02:00
Maximilian Hils
d462b444b7 update tests to reflect changes to options.scripts 2013-06-17 16:48:06 +02:00
Maximilian Hils
fd36142018 Merge remote-tracking branch 'origin/scripts_improvements' into 0.10 2013-06-16 00:41:43 +02:00
Maximilian Hils
ba47690a03 always read files in binary mode 2013-06-16 00:23:44 +02:00
Maximilian Hils
b4f6f09c83 update test script 2013-06-16 00:10:53 +02:00
Maximilian Hils
2b4af8d475 add support for multiple scripts and script arguments. refs #76 2013-06-13 16:09:38 +02:00
Aldo Cortesi
11cc48ca25 Adapt for new request_client_cert option in netlib. 2013-05-13 09:08:24 +12:00
Aldo Cortesi
5cd7563d12 Minor coverage. 2013-04-30 09:13:33 +12:00
Aldo Cortesi
61c794e08f Merge pull request #107 from rouli/master 
Adding remote TCP and SSL setup timestamps
 2013-04-19 17:19:26 -07:00
Nicolas Esteves
d4cfbbb822 Adds a new view for protocol buffers 
The view uses protoc from the Google protocol buffer
tools. If the tool isn't installed, the view isn't
shown.

Google protobuf repo:
https://code.google.com/p/protobuf/
 2013-04-06 19:21:13 +01:00
Aldo Cortesi
51b775cfd4 Merge pull request #101 from eentzel/keep-blank-params 
Keep blank URL parameters
 2013-04-04 15:55:51 -07:00
Aldo Cortesi
e3fd0e838d Add a basic built-in web app. 2013-03-25 09:20:26 +13:00
Rouli
c6bf28f3f7 adding tcp and ssl setup timestamps to get better resolution on flows performance 2013-03-19 18:21:52 +02:00
Aldo Cortesi
d2d3eb6490 Un-break unit tests. Tsk tsk. 2013-03-17 17:53:48 +13:00
Aldo Cortesi
0e993bec6f Add the --host option, which uses the value in the Host header for dispaly URLs. 
- Can be toggled with "o" then "h" in mitmproxy
- Useful for transparent mode
 2013-03-17 17:37:54 +13:00
Aldo Cortesi
cfb5ba89ce Introduce a filtered flow writer, and use it in dump.py 
Fixes #104
 2013-03-14 09:19:43 +13:00
Aldo Cortesi
8a850dc8e6 Few more fuzzing commands, exclude console app from coverage. 2013-03-03 22:39:26 +13:00
Aldo Cortesi
cde66cd584 Fuzzing, and fixes for errors found with fuzzing. 2013-03-03 22:03:27 +13:00
Aldo Cortesi
8216801728 Three more errors turned up with fuzzing. 
These are fixed in netlib, regression tests added here.
 2013-03-03 15:14:49 +13:00
Aldo Cortesi
7835e0c2c7 Begin some simple fuzzing with pathod. 
Finally doing what I started writing pathod for in the first place...
 2013-03-03 14:56:56 +13:00
Aldo Cortesi
e608d10f45 Remove __slots__ to make it possible to inherit from Options classes. 2013-03-03 12:26:20 +13:00
Aldo Cortesi
2465b8a376 100% unit test coverage on proxy.py. Hallelujah! 2013-03-03 12:13:33 +13:00
Aldo Cortesi
d5876a12ed Unit test proxy option parsing. 2013-03-03 11:58:57 +13:00
Aldo Cortesi
33cdd5d083 Unit test proxy auth. 2013-03-03 11:04:33 +13:00
Aldo Cortesi
5c6587d4a8 Move HTTP auth module to netlib. 2013-03-03 10:37:06 +13:00
Aldo Cortesi
bbdb59b9f9 Test controller message default reply. 2013-03-03 09:33:22 +13:00
Aldo Cortesi
c20d1d7d32 Extend unit tests for proxy.py to some tricky cases. 2013-03-02 22:42:36 +13:00
Aldo Cortesi
415844511c Test cert generation errors. 2013-03-02 16:59:16 +13:00
Aldo Cortesi
09c73019c5 Remove human_curl requirement for the test suite - it's pathoc all the way, baby. 2013-03-02 15:09:22 +13:00
Aldo Cortesi
a95d78438c Test SNI for transparent mode. 2013-03-02 15:06:49 +13:00
Aldo Cortesi
10db82e9a0 Test SNI for ordinary proxy connections. 2013-03-02 14:52:05 +13:00
Aldo Cortesi
ba674ad551 New SNI handling mechanism. 2013-03-01 09:05:39 +13:00
Aldo Cortesi
b077189dd5 Test cert file specification, spruce up server testing truss a bit. 2013-02-24 22:52:59 +13:00
Aldo Cortesi
0257815141 Significantly simplify server connection handling, and test. 2013-02-24 22:24:21 +13:00
Aldo Cortesi
705559d65e Refactor to prepare for SNI fixes. 2013-02-24 17:35:24 +13:00
Aldo Cortesi
d0639e8925 Handle server disconnects better. 
Server connections can be closed for legitimate reasons, like timeouts. If
we've already pumped data over a server connection, we reconnect on error. If
not, we treat it as a legitimate error and pass it on to the client.

Fixes #85
 2013-02-24 14:04:56 +13:00
Aldo Cortesi
64285140f9 Test a difficult-to-trigger IOError, fix cert generation in test suite. 2013-02-24 11:34:01 +13:00
Aldo Cortesi
51de9f9fdf Test client connection close conditions. 2013-02-24 10:51:14 +13:00
Aldo Cortesi
05e4d4468e Test request and response kill functionality. 2013-02-23 21:59:25 +13:00
Aldo Cortesi
269780c577 Unit test dummy response functions. 2013-02-23 16:34:59 +13:00
Aldo Cortesi
aaf892e3af Significantly refactor the master/slave message passing interface. 2013-02-17 12:42:48 +13:00
Aldo Cortesi
1ccb2c5dea Test WSGI app calling. 
- Factor out test servers into a separate file
- Adjust docs to note new Flask dependency
 2013-02-16 16:46:16 +13:00
Eric Entzel
6bcf29c0ed Keep blank URL parameters 
TODO: This should probably be configurable
 2013-02-11 13:22:25 +11:00
Aldo Cortesi
782bbee8c0 Unit tests for ServerConnectionPool 2013-01-29 11:35:57 +13:00
Aldo Cortesi
53792a5a28 Beef up unit tests for HAR utility functions - flow.py coverage now 100%. 2013-01-29 10:41:45 +13:00
Aldo Cortesi
6f157d936f Merge pull request #99 from rouli/master 
Adding helper functions to make HAR export easier
 2013-01-28 13:28:49 -08:00
Rouli
330fbfe8cc adding helper functions to make HAR export easier 2013-01-28 17:37:25 +02:00
Aldo Cortesi
57f01ffb07 Test suite, remove extraneous code. 2013-01-28 21:59:03 +13:00
Aldo Cortesi
25cb9471f0 Add tests for client certificate support. 2013-01-20 22:39:28 +13:00
Aldo Cortesi
6600c589ab Rudimentary testing for client certs. 2013-01-18 17:08:30 +13:00
Aldo Cortesi
0f406e9daa Speed up the test suite by reducing time sleeping in tests. 2013-01-18 14:52:19 +13:00
Aldo Cortesi
7a79eeb143 Merge branch 'master' of ssh.github.com:cortesi/mitmproxy 
Conflicts:
	test/test_server.py
 2013-01-18 14:50:31 +13:00
Rouli
6212b69fb4 fixes due to merge with master 2013-01-17 17:36:18 +02:00
Rouli
446f9f0a0f Merge remote-tracking branch 'upstream/master' 2013-01-17 17:33:29 +02:00
Rouli
20fa6a3083 changing requests and responses to have two timestamps, one marking their initiation, and the other their complete 2013-01-17 17:32:56 +02:00
Aldo Cortesi
d0ee4d60d0 Unit tests and minor code refactoring for ServerConnection. 2013-01-05 19:44:12 -08:00
Aldo Cortesi
891c441a6d Use new netlib certificate store implementation. 2013-01-06 01:16:08 +13:00
Aldo Cortesi
9cfc785cd3 Unit test love - 100% for flow.py, dump.py 2013-01-05 21:56:33 +13:00
Aldo Cortesi
46ab6ed491 Minor cleanups of proxy request handling. 2013-01-04 14:19:32 +13:00
Aldo Cortesi
f5e49ef598 First draft of "How mitmproxy works", a complete guide to the mechanics of the proxy process 2013-01-03 17:26:59 +13:00
Aldo Cortesi
e2dc7ba09d First draft of OSX transparent proxy mode. 2013-01-01 11:13:56 +13:00
Aldo Cortesi
5347cb9c26 More work on proxy auth 
- Strip auth header if auth succeeds, so it's not passed upstream
- Actually use realm specification to BasicProxyAuth, and make it mandatory
- Cleanups and unit tests
 2012-12-31 10:56:44 +13:00
Aldo Cortesi
3b84111493 Test and robustify BasicProxyAuth.parse_auth_value 
- This is partly in preparation for moving the implementation to netlib
- Also add an unparse_auth_value for testing (and use in pathod once the move is done)
 2012-12-31 10:34:25 +13:00
Aldo Cortesi
018c229ae4 Start solidifying proxy authentication 
- Add a unit test file
- Remove some extraneous methods
- Change the auth API to make the authenticate method take a header object.
 2012-12-31 09:15:56 +13:00
Aldo Cortesi
8c976ac7f0 Substantially rewrite AMF decoding. 
This is tricky, but we should now handle a lot more corner-cases.
 2012-11-26 13:25:07 +13:00
Aldo Cortesi
0d59fd7e01 Move cleanBin and hexdump into netutils. 2012-09-24 11:21:12 +12:00
Aldo Cortesi
d115b5ae70 Expand Flow.match to accept either a string or a compiled filter expression. 2012-09-14 09:41:01 +12:00
Aldo Cortesi
a77ccc406d Getter and setter for path component on Requests. 2012-08-19 13:03:21 +12:00
Aldo Cortesi
60659a89c3 Little bit of love for the unit tests. 2012-08-19 00:22:42 +12:00
Aldo Cortesi
1b7990897e Command-line options for header setting. 2012-08-19 00:14:16 +12:00
Aldo Cortesi
3e96015e61 Add SetHeaders, analogous to ReplaceHooks, with a graphical editor in mitmproxy (H shortcut). 
SetHeaders defines headers that are set on flows, based on a matching pattern.
Existing headers are over-ridden.
 2012-08-18 23:41:04 +12:00
Aldo Cortesi
b70e91bbd4 Send tracebacks from content viewers to event log. 
Also, 100% test coverage for content viewers.
 2012-08-18 17:42:40 +12:00
Aldo Cortesi
e8553f966f Further simplifcation and testing of contentviews. 2012-08-18 17:29:29 +12:00
Aldo Cortesi
11c63dcb9f Huge cleanup of content viewers. 2012-08-18 17:08:17 +12:00
Sahn Lam
3189d144a5 Optional AMF decoding support 
If PyAMF is installed, enable AMF decoding.
 2012-08-17 18:45:26 -07:00
Aldo Cortesi
a66d018363 Fix unit tests after argparse conversion. 2012-08-18 10:27:31 +12:00
Aldo Cortesi
32ad26f8bf Add a size() method to flow.Request and flow.Response. 2012-08-04 13:18:05 +12:00
Aldo Cortesi
84bffad3fc Fix flow read unit test to accomodate more tolerant dumpfile parsing. 2012-07-24 15:30:52 +12:00
András Veres-Szentkirályi
1a26f8215d removed assigned but unread variables 2012-07-15 22:42:59 +02:00
András Veres-Szentkirályi
da496669c2 removed unused imports 2012-07-15 22:42:45 +02:00
Aldo Cortesi
e4079aa746 Add an ~a filter expression, matching an asset content type in responses. 
Asset content types are Javascript, images, Flash and CSS. This is useful
because doing a quick "!~a" while auditing an app will filter out the majority
of the static asset cruft, letting you focus on what matters.
 2012-07-14 16:55:21 +12:00
Aldo Cortesi
01b8b0d876 Refine semantics of replay_request method. 2012-07-10 23:29:33 +12:00
Aldo Cortesi
04d9ec8c3c Make WSGI apps work in transparent mode. 2012-07-10 15:53:53 +12:00
Aldo Cortesi
79af9e89c4 Test replay corner cases. Fix discovered bugs. 2012-07-09 11:18:03 +12:00
Aldo Cortesi
aab45078ad Unit test request replay thread. 
This is a small patch, but is the culmination of lots of work: we can now unit
test the deep innards of mitmproxy, with coverage. There's a lot more to come
in this vein.
 2012-07-09 11:03:55 +12:00
Aldo Cortesi
e49c920d16 Refator server tests to use flow.FlowMaster and flow.State 2012-07-09 10:58:28 +12:00
Aldo Cortesi
572e8a4962 Add streaming to FlowMaster 2012-07-09 10:18:37 +12:00
Aldo Cortesi
4b6fdc92dc Remove ODict tests. ODict is now in netlib. 2012-07-09 09:54:15 +12:00
Aldo Cortesi
d02bcade3a Add a domain match filter (~d regex) 2012-07-06 22:21:44 +12:00
Aldo Cortesi
aa708a2d28 Fix error when serializing reverted SSL flows. 2012-07-05 11:52:56 +12:00
Aldo Cortesi
90365e270e Catch and handle SSL connection errors. 2012-07-01 12:10:32 +12:00
Aldo Cortesi
d74a341e5d Beef up logging substantially. 2012-07-01 00:15:03 +12:00
Aldo Cortesi
f070e4523a Handle invalid data more gracefully. 
Fixes #47
 2012-06-30 15:59:42 +12:00
Aldo Cortesi
47651b1ff2 Serialization and de-serialization of new cert format. 2012-06-28 14:29:15 +12:00
Aldo Cortesi
35ee0c098f Remove certutils from mitmproxy. 2012-06-27 16:43:33 +12:00
Aldo Cortesi
8ccfb376f3 Remove -T and -U command-line options. 
They're redundant convenience options, and we need more space.
 2012-06-26 19:57:59 +12:00
Aldo Cortesi
874649f134 Adapt for API changes in netlib. 2012-06-23 14:06:34 +12:00
Aldo Cortesi
7cb242c168 Move wsgi to netlib. 2012-06-19 10:42:55 +12:00
Aldo Cortesi
1b1ccab8b7 Extract protocol and tcp server implementations into netlib. 2012-06-19 09:58:50 +12:00
Aldo Cortesi
7b9756f48e Refactor protocol.py to remove dependence on flow and utils. 2012-06-17 10:52:39 +12:00
Aldo Cortesi
aae8a9959c Pull out protocol components into protocol.py 2012-06-16 21:23:32 +12:00
Aldo Cortesi
d5a0099f49 Test suite and refactoring for netlib. 2012-06-16 16:22:51 +12:00
Aldo Cortesi
4e53f1ee90 Rename our tcpserver to netlib, expand to include client network functions. 2012-06-16 13:38:10 +12:00
Aldo Cortesi
c7952371b7 Fix a problem in ODictCaseless that could cause duplicate headers. 2012-06-15 17:40:08 +12:00
Aldo Cortesi
8ae3270807 Basic transparent mode. 2012-06-15 09:47:04 +12:00
Aldo Cortesi
a9495dc02f Refactor test suite to make room for transparent mode tests. 2012-06-15 09:20:10 +12:00
Aldo Cortesi
8a9352b3f7 First draft conversion of server to PyOpenSSL. 2012-06-13 18:16:47 +12:00
Aldo Cortesi
d032504b17 Fix an exception when replaying a flow with no response. 2012-06-10 17:10:43 +12:00
Aldo Cortesi
d60fa9918b Localise client connection object manipulation. 
This simplifies the call signature for a bunch of functions.
 2012-06-10 16:49:59 +12:00
Aldo Cortesi
6ba5f0f35b Add HTTP version to response objects. 
Another change in the serialization format.
 2012-06-10 13:27:43 +12:00
Aldo Cortesi
52779d9db9 Refactoring of proxy.py 
- Correctly pass HTTP request version on to upstream servers
- Adjust tests not to hang due to a pathod response with no content-length
 2012-06-10 13:17:18 +12:00
Aldo Cortesi
55ddf853cd Add HTTP version to flow.Request 
This is a serialization format change, that makes us incompatible with previous
versions.
 2012-06-10 10:46:22 +12:00
Aldo Cortesi
a3b47e0cb5 Consolidate HTTP major and minor versions into a single variable. 2012-06-10 10:31:04 +12:00
Aldo Cortesi
8254187bf3 Add proxy.should_connection_close, and strip out unused code. 2012-06-10 10:10:46 +12:00
Aldo Cortesi
18c1b44475 Reverse proxy testing. 2012-06-09 20:41:28 +12:00
Aldo Cortesi
05492baf8d Move from requests to human_curl. 
It turns out that _none_ of the Python stdlib or anything that relies on it
supports CONNECT through a proxy. Beggars belief, but there you go.
 2012-06-09 16:17:51 +12:00
Aldo Cortesi
22192d1a46 Nose mopup: docs, no cover pragmas, a few missing path specs. 2012-06-09 13:55:55 +12:00
Aldo Cortesi
b7b357528c Port mitmproxy test suite entirely to nose. 2012-06-09 13:42:43 +12:00
Aldo Cortesi
a63240a848 Move pathod service testing truss to nose. 2012-06-09 12:13:01 +12:00
Aldo Cortesi
e78b48ab20 Start conversion to nose. 
RIP pry.
 2012-06-09 10:57:00 +12:00
Aldo Cortesi
7a312546f3 Shift mitmproxy test suite over to pathod. 
This opens a whole brave new world of testing for mitmproxy.
 2012-06-08 10:00:16 +12:00
Aldo Cortesi
e9109812e1 Split parsing of intial line into separate protocols. 2012-06-03 06:04:57 -07:00
Aldo Cortesi
491f9bdcee Add unit tests for console/help.py 2012-06-03 01:11:07 -07:00
Aldo Cortesi
ee2950cd19 Fix a crashing bug when replacing text in a flow with unicode bodies. 2012-05-25 18:10:31 -07:00
Aldo Cortesi
0a90a3eaba Refuse to replay a request with missing content. 2012-05-16 18:24:32 +12:00
Aldo Cortesi
0c2d894cea Add the ability to flag content as missing in a request or a response. 
We'll use this in a number of situations. First, we'll soon have response
streaming that directly pipes responses to clients. These will be content-less
from mitmproxy's perspective. Second, we'll be growing new events that fire
after headers are received, but before content is read.
 2012-05-16 15:42:58 +12:00
Aldo Cortesi
116fcfcf7a Internal error page for WSGI. 
Also, 100% test coverage.
 2012-04-27 15:56:42 +12:00
Aldo Cortesi
c8d2b2594b Add a WSGI adapter that lets us serve a WSGI app out of mitmproxy. 
This commit adds:
    - A WSGI App adapter for mitmproxy
    - An app registry in the proxy instance that lets us link WSGI apps with
    (hostname, port) combinations.
    - Fixes for a number of bugs discovered while creating this feature.
 2012-04-24 14:52:29 +12:00
Aldo Cortesi
8c96264304 Serialized data version check. 2012-04-11 10:10:53 +12:00
Aldo Cortesi
4e2d19714c Add an "f" shortcut key to load full body contents. 2012-04-08 19:44:01 +12:00
Aldo Cortesi
a4f7728fad XML/HTML pretty view tweaks. 2012-04-07 22:15:31 +12:00
Aldo Cortesi
f1dc3f2ab2 Integrate lxml for pretty-printing HTML and XML. 
Tackling the pretty-printing performance problem head-on, at the cost of a
major dependency.
 2012-04-07 13:47:03 +12:00
Aldo Cortesi
ab0e10e60f Serialize requestcount for ClientConnect objects. 2012-04-03 22:37:24 +12:00
Aldo Cortesi
61fab03b24 Add a details page, available from a flow view with the 'X' shortcut 
At the moment, this shows the upstream SSL certificate details. More
fine-grained detail that doesn't fit in the flow view itself will be added.
 2012-04-03 11:10:25 +12:00
Aldo Cortesi
f526e5fa12 Minor unit test bump. 2012-04-03 09:52:26 +12:00
Aldo Cortesi
4979a22d3e Add accessor method for SSLCert object on Response. 2012-04-02 17:02:23 +12:00
Aldo Cortesi
ab1d8fa350 Expand SSL cert support 
- Capture the remote SSL certificate
- Expose the remote cert as an attribute on Response
- Expand the certutils.SSLCert interface to expose more cert info
 2012-04-02 16:21:23 +12:00
Aldo Cortesi
c02fdb2463 Refactor proxy.Server to fix a crash when replaying with -n 2012-04-02 13:24:51 +12:00
Aldo Cortesi
15cc09f1b8 Start rationalizing content views. 
We now no longer have distinction between "pretty" view and hex/raw. Instead,
we simply a default AUTO view with a global override (M) and a local override
(m).
 2012-04-02 10:30:35 +12:00
Aldo Cortesi
7fef0ecdf5 Make "T" pretty view over-ride persistent when switching between flows. 
We do this by adding a flow settings mechanism to ConsoleState. This is pretty
rough at the moment and should become more sophisticated as needed.
 2012-04-02 09:30:38 +12:00
Aldo Cortesi
e9ac4bef20 Add a variant of cleanBin that escapes newlines and tabs. 
Use this to fix the hex display option.
 2012-03-27 11:25:50 +13:00
Aldo Cortesi
a050eeef05 Add a pretty-viewer for images. 
This shows basic image information like dimensions, plus extracted EXIF tags
and other metadata.
 2012-03-26 11:26:02 +13:00
Aldo Cortesi
2240d2a6a5 Pretty view now indents Javascript. 
Thanks to the JSBeautifier project, which is now included in the contrib directory.
 2012-03-25 10:56:45 +13:00
Aldo Cortesi
74c51df580 Re-enable simple multipart form parsing and preview. 2012-03-25 10:10:48 +13:00
Aldo Cortesi
62e51018d0 Refactor pretty view mechanism. 
Also start adding unit tests for this subsystem.
 2012-03-24 14:20:24 +13:00
Aldo Cortesi
ed74ed24a0 Add error indications to GridEditor. 2012-03-23 13:28:33 +13:00
Aldo Cortesi
2739cb4861 Add a simple parser for content type specifications. 2012-03-20 10:31:07 +13:00
Aldo Cortesi
5690e7c399 Generalize GridEditor to N columns. 
Start adding a replacement rule editor.
 2012-03-18 14:39:21 +13:00
Aldo Cortesi
76175672ad Add specification of replacement patterns on the command line. 2012-03-17 17:20:34 +13:00
Aldo Cortesi
c8ae1e85b3 Hooks -> ReplaceHooks 
It makes more sense to specialize this, which will let me build a nicer
interface for replacement hooks in mitmproxy.
 2012-03-17 11:31:05 +13:00
Aldo Cortesi
08f410cacc Add a hooks mechanism, based on filter expressions. 2012-03-16 17:13:11 +13:00
Aldo Cortesi
d138af7217 replace() methods now decode and re-encode contents before substitution. 2012-03-16 11:24:18 +13:00
Aldo Cortesi
d51b8cab0c Add a decoded context manager. 
This simplifies a common chore when modifying traffic - decoding the object,
modifying it, then re-encoding it with the same encoding afterwards. You can
now simply say:

with flow.decoded(request):
    request.content = "bar"
 2012-03-16 11:12:56 +13:00
Aldo Cortesi
fa6305ee98 Cleanliness fixes. 
- Remove unused code during previous commit.
- Code coverage fixes.
 2012-03-12 11:25:50 +13:00
Aldo Cortesi
d3aad7a185 Merge remote-tracking branch 'taiste/server-replay-pop' 2012-03-10 13:36:50 +13:00
Valtteri Virtanen
041eafba73 Added tests for ServerPlaybackState with nopop 2012-03-05 13:57:57 +02:00
Valtteri Virtanen
5b5b79f5c4 Fixed old tests 2012-03-05 13:40:18 +02:00
Aldo Cortesi
e1356dd2b6 Create an SSL certificate class. 2012-03-05 10:22:47 +13:00
András Veres-Szentkirályi
15ad7704d2 Removed imports left unused after Py{OpenSSL,ASN1} 
Commits 533f61f67a and
8b841bc9e3 left some imports unused while
swithing to PyOpenSSL and PyASN1 -- this commit removes these imports.
 2012-03-01 16:20:34 +01:00
Aldo Cortesi
533f61f67a Use PyOpenSSL and PyASN1 for certificate parsing. 
Yes, these are two more major dependencies for mitmproxy, but if we're going to
do all the cool things I want to do with SSL certs, there is no other way.
 2012-03-01 21:08:44 +13:00
Aldo Cortesi
8b841bc9e3 Factor out cert operations in to certutils.py. 2012-02-29 13:20:53 +13:00
Aldo Cortesi
688faa9baa Repair unit tests. 2012-02-27 20:34:47 +13:00
Aldo Cortesi
764724748b Fix cert generation harder. 2012-02-27 15:59:29 +13:00
Aldo Cortesi
2ba8296843 Better certificate parsing. 2012-02-27 15:21:05 +13:00
Aldo Cortesi
00942c1431 Add upstream certificate lookup. 
This initiates a connection to the server to obtain certificate information to
generate interception certificates. At the moment, the information used is the
Common Name, and the list of Subject Alternative Names.
 2012-02-27 15:05:45 +13:00
Aldo Cortesi
986a41d180 Unit test++. 2012-02-25 12:19:54 +13:00
Aldo Cortesi
25fa596cd6 Fix detection of URL-encoded forms. 
Thanks to Paul Capestany <capestany@gmail.com> for reporting this.
 2012-02-24 13:03:24 +13:00
Aldo Cortesi
ddc9155c24 Make "~q" filter work more intuitively. 
It now matches any flow that has no response.
 2012-02-23 17:06:09 +13:00
Aldo Cortesi
2df9c52c09 Refactor filter matching. 2012-02-23 17:03:58 +13:00
Aldo Cortesi
dbd75e02f7 Create ODictCaseless for headers, use vanilla ODict for everything else. 2012-02-20 11:29:36 +13:00
Aldo Cortesi
18029df99c Use ODict for request.get_form_urlencoded and set_form_urlencoded 2012-02-20 11:13:35 +13:00
Aldo Cortesi
b0f77dfefd Unit test import cleanups. 2012-02-20 11:04:07 +13:00
Aldo Cortesi
fa11b7c9be Use ODict for Request.get_query and Request.set_query 2012-02-20 10:44:47 +13:00
Aldo Cortesi
2616f490fe Rename Headers class to ODict 
ODict is an ordered dictionary class that will be useful in many other parts of
our API.
 2012-02-20 10:39:00 +13:00
Aldo Cortesi
d32d6bc5e3 Add "p" key binding to connection list view to copy a flow. 2012-02-19 00:17:47 +13:00
Aldo Cortesi
8ddc3b4ef2 Add API for duplicating flows. 2012-02-18 23:56:40 +13:00
Aldo Cortesi
7aa79b89e8 Firm up what we consider to be a valid proxy spec. 2012-02-18 16:29:02 +13:00
Aldo Cortesi
6ad8b1a15d Firm up reverse proxy specification. 
- Extract proxy spec parsing and unparsing functions.
- Add a status indicator in mitmproxy.
- Add the "R" keybinding for changing the reverse proxy from within mitmproxy.
 2012-02-18 16:27:09 +13:00
Aldo Cortesi
a7df6e1503 Refactor reverse proxying 
- Retain the specification from the Host header as a Request's description.
- Expand upstream proxy specifications to include the scheme. We now say https://hostname:port
- Move the "R" revert keybinding to "v" to make room for a reverse proxy
binding that matches the command-line flag.
 2012-02-18 14:45:22 +13:00
Aldo Cortesi
acdc2d00b4 Repair unit tests. 2012-02-18 12:27:59 +13:00
Aldo Cortesi
da1ccfddeb 100% test coverage for flow.py 2012-02-10 15:55:58 +13:00
Aldo Cortesi
1ad7e91527 Make filter matching act more sensibly. 2012-02-10 15:31:45 +13:00
Aldo Cortesi
5f785e26b9 Add filter for detecting flows with errors. 
Also, remove dependency on weird _is_response method.
 2012-02-10 15:22:26 +13:00
Aldo Cortesi
b14c29b25c Expand test coverage. 2012-02-10 15:04:20 +13:00
Aldo Cortesi
9c985f2d20 Methods for getting and setting form urlencoded data on Request. 2012-02-10 14:27:39 +13:00
Aldo Cortesi
2709441d5b Add get_query and set_query methods to Request. 2012-02-09 16:40:31 +13:00
Aldo Cortesi
cdd5a53767 Refactor console. 
Split the console implementation out into logical components.
 2012-02-07 16:39:37 +13:00
Aldo Cortesi
35a914a549 Fix unit tests broken during previous commit. 2012-01-21 14:39:36 +13:00
Aldo Cortesi
d5e3722c97 Fix an issue caused by some editors when editing a request/response body. 
Many editors make it hard save a file without a terminating newline on the last
line. When editing message bodies, this can cause problems. For now, I just
strip the newlines off the end of the body when we return from an editor.
 2012-01-21 12:43:00 +13:00
Mark E. Haase
05111f093d Add support for filtering by HTTP method (get, post, etc.) using ~m operator. 2011-12-28 17:32:29 -05:00
Aldo Cortesi
67f2610032 Add HTTP body size limit specification to command-line tools. 2011-09-09 15:27:31 +12:00
Aldo Cortesi
28daa93268 Basic infrastructure for request and response body size limits. 2011-09-09 14:49:34 +12:00
Aldo Cortesi
e5bded7dee Improve robustness against invalid data. 2011-09-05 07:47:47 +12:00
Aldo Cortesi
45f4768a5c Add attribution and license for tnetstring.py 2011-08-19 21:53:52 +12:00
Aldo Cortesi
a566684e32 Move to typed netstrings for serialization. 
This change is backwards incompatible with the old serialization format!
 2011-08-19 21:30:24 +12:00
Aldo Cortesi
b51aac8a86 Code cleanliness - appease pychecker. 2011-08-04 10:34:34 +12:00
Aldo Cortesi
730c78ac53 Move script.Context to flow.ScriptContext 2011-08-04 10:14:44 +12:00
Aldo Cortesi
7a3b871b33 Request class now has a clean pydoc profile. 2011-08-04 09:26:26 +12:00
Aldo Cortesi
0760607a7d Further interface cleaning. 2011-08-03 23:02:33 +12:00
Aldo Cortesi
9042d3f3b9 Clean up interfaces by making some methods pseudo-private. 2011-08-03 22:48:57 +12:00
Aldo Cortesi
57c653be5f Move all HTTP objects to flow.py 
That's Request, Response, ClientConnect, ClientDisconnect, Error, and Headers.
 2011-08-03 22:41:38 +12:00
Aldo Cortesi
e337682d8e Enable "|" command to run a oneshot script on a single flow. 2011-08-03 17:35:18 +12:00
Aldo Cortesi
179cf75862 Add script hooks, enable new engine for mitmdump. 2011-08-03 16:36:20 +12:00
Aldo Cortesi
12d2b1f926 Rip out old script interface, start replacing with new stubs. 
Scripts are broken for now.
 2011-08-03 13:20:36 +12:00
Aldo Cortesi
62088a6661 Start stubbing out a much more powerful script architecture. 2011-08-03 11:06:29 +12:00
Aldo Cortesi
a817db5bd6 Refresh current connection when toggling autodecode. 
Also fix the unit tests I forgot to commit...
 2011-08-02 20:47:53 +12:00
Aldo Cortesi
1ff6a767d0 Unit test++ 2011-08-02 16:52:47 +12:00
Aldo Cortesi
357502fe03 General cleanup. 
Cut out unused variables and code, generally shut up pychecker as much as is
reasonable.
 2011-08-02 16:14:33 +12:00
Aldo Cortesi
f3742f29da We no longer need to track clientconnections. 2011-08-02 14:56:09 +12:00
Aldo Cortesi
675b3133b4 Improve performance of loading flows from a file hugely. 
Fell into the "expensive __eq__ method" trap. Oh, Python, you little scamp.
 2011-08-01 11:26:09 +12:00
Aldo Cortesi
43f1c72511 Refactor the way we calculate views of the flow list. 
The naive approach we used before recalculated the view on every access, and
consequently had serious performance problems.
 2011-08-01 11:17:01 +12:00
Aldo Cortesi
ddb5748a76 Add decoding/encoding for requests. 2011-08-01 10:43:01 +12:00
Aldo Cortesi
c89c4361c3 Merge remote-tracking branch 'alts/encoding' 2011-07-28 11:19:07 +12:00
Stephen Altamirano
78049abac1 Changes replace logic to function in both Python 2.6.x and 2.7.x 
Tests now only assume Python 2.6.x rather than requiring 2.7.x. This does not preclude the use of flags as a kwarg in replace
 2011-07-26 22:47:08 -07:00
Stephen Altamirano
acd511f676 Adds test method for Response encoding and decoding 2011-07-26 22:38:42 -07:00
Aldo Cortesi
4043829cf2 Add an eventlog option to mitmdump 
This shows client connections, disconnections and requests (before a complete
flow is assembled). We need to add an analogous display to mitmproxy.
 2011-07-23 12:57:54 +12:00
Aldo Cortesi
47e1695512 Also replace strings path for requests. 2011-07-22 20:52:13 +12:00
Aldo Cortesi
1b961fc4ad Add utility functions to search and replace strings in flows 
This is a common task in pentesting scenarios. This commit adds the following
functions:

utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
 2011-07-22 17:48:42 +12:00
Stephen Altamirano
ecd4645988 Adds encode counterparts to decode functions 2011-07-17 20:16:47 -07:00
Aldo Cortesi
ce41046786 Refine encoding support 
- Push decoding down into the LRU cache
- Cope gracefully with corrupted data and incorrect encoding headers
 2011-07-17 10:25:25 +12:00
alts
6dc0f105cc Adds support for content encoding, namely gip and deflate 2011-07-16 02:47:06 -07:00
Aldo Cortesi
94ae720a22 Add a pretty-printing mode for urlencoded form data. 2011-07-15 16:46:54 +12:00
Aldo Cortesi
76b4c6ba82 Introduce an anti-compression command-line argument. 
This is on by default, which means we avoid compressed content unless the -z
flag is specified.
 2011-07-15 15:24:56 +12:00
Aldo Cortesi
7e21ac0eb8 Refine path completion somewhat. 
Make it match the behaviour of vim and mutt more closely
 2011-07-14 16:54:04 +12:00
Aldo Cortesi
1c9e7b982a Rewrite Headers object to preserve order and case. 2011-07-14 16:01:54 +12:00
Aldo Cortesi
8e176c2086 Cast some data read from dump files to str, to prevent unicode promotion. 
This fixes a bug that caused a traceback when de-serialized requests were
replayed. Also adds unit tests for the problem.
 2011-07-01 14:20:42 +12:00
Aldo Cortesi
18d4c3a9e9 JSON pretty-printing. 
Also rename the display modes ("pretty" instead of "indent"), and expand the
built-in documentation.
 2011-06-30 13:27:27 +12:00
Aldo Cortesi
f004326855 Try not to hang when user views large request & response bodies 
Two different strategies here:

    - Use a simple heuristic to detect if we're looking at XML data when indent
    mode is used. On non-XML data we can hang even on small documents.

    - Only view partial data for large bodies. At the moment the cutoff is
    100k. I might finetune this later.
 2011-06-27 15:59:17 +12:00
Aldo Cortesi
7d7803a4d9 Add a hideous kludge to fix not-yet-valid certificates. 
- The OpenSSL x509 has no way to explicitly set the notBefore value on
certificates.

- If two systems have the same configured time, it's possible to return a
certificate before the validity start time has arrived.

- We "solve" this by waiting for one second when a certificate is first
generated before returning the cert. The alternative is to rewrite pretty much
all of our certificate generation, a thought too horrible to contemplate.
 2011-06-11 15:16:16 +12:00
Aldo Cortesi
613e9a298e Add a new flow loading mechanism. 
We now simulate the normal connection flow when we load flows. That means
that we can run scripts, hooks, sticky cookies, etc.
 2011-05-15 11:22:35 +12:00
Aldo Cortesi
c0bd1a39e4 unit test coverage ++ 2011-03-20 18:52:16 +13:00
Aldo Cortesi
c726519e73 Add a stickyauth option. 
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
 2011-03-20 17:31:54 +13:00
Aldo Cortesi
4f877cde6a Reverse order of flows in mitmproxy. 
It matches user expectations much better to have new flows appended to the bottom.
 2011-03-20 09:31:39 +13:00
Aldo Cortesi
e22fd74d06 Revamp key generation. 
We now create three different files in the .mitmproxy directory when a dummy CA
is made:

mitmproxy-ca.pem - the CA, including private key

mitmproxy-ca-cert.p12 - A pkcs12 version of the certificate, for distribution to Windows.

mitmproxy-ca-cert.pem - A PEM version of the certificate, for distribution to everyone else.
 2011-03-18 16:45:31 +13:00
Aldo Cortesi
394bd1d6b0 Interrupt interception when deleting an intercepting flow. 
Prompting the user for this is annoying.
 2011-03-15 17:53:29 +13:00
Aldo Cortesi
6d5c32ad4b Fix hang on shutdown. 2011-03-15 17:21:35 +13:00
Aldo Cortesi
fb28e71f0b Do pre-processing of requests before replay. 
This enables scripts, anticache, server playback and sticky cookies for
request replays.
 2011-03-15 15:11:03 +13:00
Aldo Cortesi
fe1e2f16ff Improve responsiveness of request and response viewing. 
- Computing the view of a large body is expensive, so we introduce an LRU cache
to hold the latest 20 results.

- Use ListView more correctly, passing it individual urwid.Text snippets,
rather than a single large one. This hugely improves render time.
 2011-03-15 13:05:33 +13:00
Aldo Cortesi
2e70a0e44b Fix unit tests on OSX when a system proxy is configured. 2011-03-14 14:05:14 +13:00
Aldo Cortesi
e35fd7126b Fix flow formatting unit tests. 2011-03-14 13:49:30 +13:00
Aldo Cortesi
1e47167ac6 Fix a traceback sometimes triggered viewing a connection with no response. 2011-03-13 20:03:13 +13:00
Aldo Cortesi
324d7c3955 Add client plaback to mitmproxy. 2011-03-13 16:50:11 +13:00
Aldo Cortesi
3ef9d2cffc flow.py 100% test coverage 2011-03-13 14:51:25 +13:00
Aldo Cortesi
c901cbbada mitmproxy prompted input now display previously set value. 
E.g. if you set a limit, then re-enter the limit prompt, you start with the
currently set value.
 2011-03-13 11:24:49 +13:00
Aldo Cortesi
9a9a2d9125 Unit tests for proxy.read_chunked 2011-03-12 16:00:01 +13:00
Aldo Cortesi
4ce393cc41 Unit tests for cmdline.py 2011-03-12 15:14:25 +13:00
Aldo Cortesi
96b10ca486 console.py 100% test coverage 2011-03-12 15:08:06 +13:00
Aldo Cortesi
fbbc43a5b2 netstring.py 100% unit test coverage. 2011-03-12 14:39:41 +13:00
Aldo Cortesi
9f16a84a9e Make mitmdump handle invalid serialized data gracefully. 2011-03-11 15:16:31 +13:00
Aldo Cortesi
7d85db0da3 Display Errors and killed connections in mitmdump. 2011-03-11 13:06:51 +13:00
Aldo Cortesi
daa9653ebe Add --norefresh to stop refreshing server playback to mitmdump. 
Also, make cookie parsing for refreshing more error-tolerant.
 2011-03-11 11:56:10 +13:00
Aldo Cortesi
e99b1d1949 Stub out refresh for server-side replay. 2011-03-09 20:05:30 +13:00
Aldo Cortesi
03f1345385 Add an --anticache option to mitmdump. 
This removes all headers that might cause a server to return 304-not-modified.

For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
 2011-03-09 13:15:31 +13:00
Aldo Cortesi
765871bd11 Store timestamps on flow components as a UTC time tuple. 
Format is:

(tm_year,tm_mon,tm_mday,tm_hour,tm_min, tm_sec,tm_wday,tm_yday,tm_isdst)
 2011-03-07 13:46:02 +13:00
Aldo Cortesi
7330f5fccf Make mitmdump server playback also exit by default. 
Like client playback, the --keepserving option makes mitmdump keep serving.
 2011-03-06 17:08:56 +13:00
Aldo Cortesi
ec00b5a66e Make mitmdump exit after client replay is complete by default. 
Add an option --keepserving to make it keep serving after replay.
 2011-03-06 16:54:49 +13:00
Aldo Cortesi
e794cbc0d8 Add flow_count and active_flow_count methods to flow.State. 
Use these in unit tests where sensible.
 2011-03-06 16:11:45 +13:00
Aldo Cortesi
42ffded626 Bump unit tests, rearrange mitmdump command-line options slightly. 2011-03-06 16:02:28 +13:00
Aldo Cortesi
bad77f3470 Add client replay options to mitmdump. 2011-03-06 11:21:31 +13:00
Aldo Cortesi
d9cb083809 Reorganize test suite to remove confusion between test utils and libmproxy utils. 2011-03-05 15:58:48 +13:00
Aldo Cortesi
5da4099ddf Massage namespace to make room for client replay. 
Mostly replay -> server_replay
 2011-03-05 13:03:26 +13:00
Aldo Cortesi
96471fde1d Basic client playback state structure. 2011-03-04 13:08:43 +13:00
Aldo Cortesi
6921b9ff2a Add an indicator that sticky cookies have been applied in mitmdump. 2011-02-25 21:23:44 +13:00
Aldo Cortesi
8cade9fbbf Move stringification funcs from proxy to dump. 2011-02-25 21:11:44 +13:00
Aldo Cortesi
57f96c5fe0 Minor housekeeping - unused code, nocover directives. 
95% test coverage.
 2011-02-24 15:44:08 +13:00
Aldo Cortesi
79039eb5d2 More mature sticky cookie primitive. Use it in console.py. 2011-02-24 15:26:34 +13:00
Aldo Cortesi
57947b328e Start abstracting out sticky cookie state. 2011-02-24 10:33:39 +13:00
Aldo Cortesi
39207ffdd2 Add a way for users to specify header significance in server replay. 
Also add the --rheader command-line option to mitmdump to let the user specify
an arbitrary number of significant headers. The default is to treat no headers
as significant.
 2011-02-23 10:54:51 +13:00
Aldo Cortesi
7bc913c40d Move script hooks into the flow primitives. 
This lets handle scripts in corner cases like playback situations more easily.
 2011-02-21 11:40:49 +13:00
Aldo Cortesi
fe99871df8 Add --kill option to mitmdump 
If this option is passed all requests that are not part of a replayed
conversation are killed. If the option is not passed, such requests are passed
through to the server as usual.
 2011-02-21 11:08:35 +13:00
Aldo Cortesi
fd4dd8cb6b First pass of playback function for mitmdump. 2011-02-21 09:54:39 +13:00
Aldo Cortesi
deb79a9c5a Add a simple server playback state object. 
We use a loose hash to match incoming requests with recorded flows. At the
moment, this hash is over the host, port, scheme, method, path and content of
the request. Note that headers are not included here - if we do want to include
headers, we would have to do some work to normalize them to remove variations
between user agents, header order, etc. etc.
 2011-02-21 08:47:19 +13:00
Aldo Cortesi
aa16194518 Clean up and strip down netstrings module. 2011-02-20 14:03:32 +13:00
Aldo Cortesi
7ddba22f51 Certificates are now generated in a temporary per-session directory. 
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
 2011-02-20 13:29:41 +13:00
Aldo Cortesi
c2ae8285f4 Revamp SSL configuration. 
- Move option parsing utiliities to proxy.py

- Don't have a global config object. Pass it as an argument to ProxyServer.

- Simplify certificate generation logic.
 2011-02-20 12:53:42 +13:00
Aldo Cortesi
4fc807cedd Clean up certificate generation. 
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.

- Split CA and cert generation into separate functions.

- Generation methods provide an error return when generation fails.

- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
 2011-02-20 12:17:10 +13:00
Aldo Cortesi
d7ace1ce9e Bump unit tests for flow.py and dump.py 2011-02-20 09:55:42 +13:00
Aldo Cortesi
9c5c3c2b1a Implement state loading that doesn't change object identity. 
We need this to let us load state from copied Flows returned from scripts.
 2011-02-20 09:36:13 +13:00
Aldo Cortesi
1549ec8079 Bump test coverage to 100% for flow.py 2011-02-19 17:21:08 +13:00
Aldo Cortesi
d9c6be3d13 Fix serialization when a Request has no associated client connection. 2011-02-19 17:03:44 +13:00
Aldo Cortesi
5da27a9905 Refactor Flow primitives to remove HTTP1.0 assumption. 
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
 2011-02-19 17:00:24 +13:00
Aldo Cortesi
cd4eea3934 First pass of script hooks for mitmdump. 
Also stub out docs, improve mitmdump error handling.
 2011-02-18 12:40:45 +13:00
Aldo Cortesi
f028dc7e1b Add filtering to mitmdump. 2011-02-17 10:44:08 +13:00
Aldo Cortesi
f0f1fb4b55 Add file writing to mitmdump. 2011-02-17 10:18:38 +13:00
Aldo Cortesi
f5511350eb Tweaks and cleanups for mitmdump. 2011-02-16 23:03:46 +13:00
Aldo Cortesi
0dd1aa7cc7 Initial port of mitmdump to Flows. 2011-02-16 22:10:24 +13:00
Aldo Cortesi
66349c9783 FlowMaster bugfixes and unit tests. 2011-02-16 16:43:35 +13:00
Aldo Cortesi
33dac700fa Switch over to new serialization format. 
Remove BSON from contrib.
 2011-02-16 15:10:00 +13:00
Aldo Cortesi
0c6f846861 First draft of the new serialization mechanism. 2011-02-16 14:33:04 +13:00
Aldo Cortesi
5692c7359c Import Will McGugan's netstring module. 
Module is in the Public Domain.

I expect to modify and extend this module, so I've imported into main library
rather than contrib. Code has been reformatted to suite our code standard,
tests have been extrated into /tests directory.
 2011-02-16 11:21:06 +13:00
Aldo Cortesi
6339f521fc Repair unit test to match removal of arg from Response class. 2011-02-16 11:20:00 +13:00
Henrik Nordstrom
89627a702a drop unused protocol parameter from Response 2011-02-15 22:24:38 +01:00
Aldo Cortesi
759496e24e Repair unit test suite. 2011-02-11 11:06:30 +13:00
Aldo Cortesi
a4eaafab5a Further fine-tuning for pretty_xmlish. 2011-02-06 16:56:13 +13:00
Aldo Cortesi
7156d1a73a Rip out BeautifulSoup, and use a custom XML-ish prettyprinter. 2011-02-06 14:17:30 +13:00
Aldo Cortesi
d0c2d9480c Also serialize ClientConnection and flow backups. 2011-02-06 10:28:43 +13:00
Aldo Cortesi
2ad4c5adf3 Get rid of ReplayConnection - we now have only one ClientConnection class. 2011-02-04 10:58:08 +13:00
Aldo Cortesi
2cb7429d38 Change "connection" to the less confusing "client_conn" throughout. 2011-02-03 14:51:32 +13:00
Aldo Cortesi
7924f07971 Fine-tune completion a bit. 2011-02-03 13:50:57 +13:00
Aldo Cortesi
6c89749f0a Add timestamps to flows. 
For now, these are only displayed on the connection view screen, with second
granularity.
 2011-02-03 13:30:47 +13:00
Aldo Cortesi
d8cb826361 Test suite rejiggering and cleanup. 2011-02-03 12:16:03 +13:00
Aldo Cortesi
db99da6af5 Improve script handling. 
- Display output in external viewer when script exits with error.

- Add a "changed" indicator to show if a request can be reverted.
 2011-02-02 11:44:28 +13:00
Aldo Cortesi
405e60215c View script debug output (stderr) in pager. 2011-02-02 10:09:16 +13:00
Aldo Cortesi
b886f808be Add an external script API. 
External scripts can read a flow, modify it, and then return it to mitmproxy
using a simple API.

The "|" keyboard shortcut within mitmproxy prompts the user for a script.
 2011-01-31 13:26:56 +13:00
Aldo Cortesi
edb8228dd2 Factor out flow unit tests into speparate file. 2011-01-31 11:48:53 +13:00
Aldo Cortesi
152b97fa0b Restructure to make subclassing Flow unnecessary. 2011-01-31 11:44:52 +13:00
Aldo Cortesi
2a39387d27 UI tweaks 
- Make flow view state persistent (request/response, body view mode).
- Don't exit flow view mode when viewing help.
 2011-01-30 14:52:51 +13:00
Aldo Cortesi
0742bc4fd0 Nicer statusbar messages, with timed expiry. 2011-01-28 12:18:01 +13:00
Aldo Cortesi
dfefe3cdda Make pretty-printing more robust. 
Also, since BeautifulSoup is so damn slow, print a statusbar message saying
that we're calculating a pretty version of the response. Maybe I should add
hangman or something, becuase on a 200k document this can take ages.
 2011-01-28 12:08:25 +13:00
Aldo Cortesi
93ef691bad Basix XML/HTML pretty-printing in flow viewer. 2011-01-27 17:26:01 +13:00
Aldo Cortesi
460107589c Add tab completion for save and load path specs. 2011-01-27 10:29:37 +13:00
Aldo Cortesi
635f7a971d Add saving and loading of complete flows for later replay and analysis. 2011-01-26 16:50:17 +13:00
Aldo Cortesi
29d8007678 Add serialization hooks to flows and flow component objects. 2011-01-26 14:52:03 +13:00
Aldo Cortesi
7983dbb26a Abstract flow management out of the interactive code. 2011-01-25 15:02:48 +13:00
Aldo Cortesi
cb0e328709 Initial checkin. 2010-02-16 17:09:07 +13:00