Grasscutters/mitmproxy
3
0
Fork 0
mirror of https://github.com/Grasscutters/mitmproxy.git synced 2024-11-27 10:26:23 +00:00
Code Issues Projects Releases Wiki Activity
4,925 Commits 1 Branch 67 Tags 50 MiB
5f7d61f864
Commit Graph

976 Commits

Author SHA1 Message Date
Aldo Cortesi
9bea616441 Be stricter about handling connetcts in the pathoc test suite 
Attempt to clear a niggling Appveyor exception buggering up our CI.
 2016-06-12 10:39:19 +12:00
Aldo Cortesi
7b86560ded debug: On SIGUSR2, we dump tracebacks for all threads to screen 2016-06-11 23:07:42 +12:00
Aldo Cortesi
09edbd9492 Improve debugging of thread and other leaks 
- Add basethread.BaseThread that all threads outside of test suites should use
- Add a signal handler to mitmproxy, mitmdump and mitmweb that dumps resource
information to screen when SIGUSR1 is received.
- Improve thread naming throughout to make thread dumps understandable
 2016-06-11 19:52:24 +12:00
Aldo Cortesi
5b9f07c81c debug.sysinfo: tests and coverage 2016-06-11 17:56:17 +12:00
Aldo Cortesi
2cf79b7912 Remove timestamps from pathoc output 
Pathoc is an interactive tool, no need for a long leading timestamp. More
generally, make timestamps optional in the logging mechanism so we can
configure this with command-line flags or something down the track.
 2016-06-10 10:47:28 +12:00
Aldo Cortesi
c421c41307 Remove odict 
- Adds default implementations for _kconv and _reduce_values to MultiDict.
Without these, operations fail in really, really non-obvious ways.
- Replace the remaining few instances of ODict

Fixes #1159
 2016-06-09 13:28:43 +12:00
Shadab Zafar
b3b4156c2f Py3: Fix test_language_http2 tests by using byte literals 2016-06-08 16:15:54 +05:30
Shadab Zafar
6b03df2633 Py3: Use global next() instead of iterator method 2016-06-08 16:15:54 +05:30
Shadab Zafar
0280af9522 Py3: Use BytesIO instead of StringIO 2016-06-08 16:15:54 +05:30
Aldo Cortesi
a5cb241c7c If a message has been acked, all other processors are skipped 
This applies the constraint, but does to clumsily. When we've unified modules
and processors it will be much nicer. We also make some exceptions for the
master processors that we may want to re-evaluate down the track.
 2016-06-08 12:58:58 +12:00
Aldo Cortesi
b3bf754e53 Simplify script concurrency helpers 
We now have take() to prevent double-replies.
 2016-06-08 11:21:38 +12:00
Aldo Cortesi
a388ddfd78 A new interface for reply 
Reply is now explicit - it's no longer a callable itself. Instead, we have:

    reply.kill()            - kill the flow
    reply.ack()             - ack, but don't send anything
    reply.send(message)     - send a response

This is part of an incremental move to detach reply from our flow objects,
and unify the script and handler interfaces.
 2016-06-08 10:44:20 +12:00
Aldo Cortesi
982077ec31 Add reply.ack and reply.kill 2016-06-08 10:14:34 +12:00
Aldo Cortesi
db11fe0087 Merge branch 'master' into beautiful-json 2016-06-08 09:19:25 +12:00
Maximilian Hils
c98c83190b tests++ 2016-06-07 01:03:08 -07:00
Shadab Zafar
d254fe791a Py3: Fix a test_language_writer test by using a byte literal 2016-06-07 00:01:41 -07:00
Maximilian Hils
ba466dbb35 simplify pretty_json, improve test coverage 2016-06-06 23:08:58 -07:00
Aldo Cortesi
1ffc273c94 Utils cleanups 
- Move more stuff that belongs in netlib.human
- Move some stuff to near the only use
- Zap mitmproxy.utils.timestamp(). I see the rationale, but we used it
interchangeably with time.time() throughout the project. Since time.time()
dominates in the codebase and timestamp() is such low utility, away it goes.
 2016-06-07 17:12:52 +12:00
Maximilian Hils
80292c0ac3 fix all pathod tests on Windows 🎉 2016-06-06 16:03:45 -07:00
Maximilian Hils
2ee5e8ef0e fix NestedMessage encoding 2016-06-06 13:39:38 -07:00
Maximilian Hils
6447c8ae22 fix .freeze(), improve tests 2016-06-06 13:31:44 -07:00
Shadab Zafar
f48073af56 Py3: Fix status_code and other tests by using byte literals 2016-06-06 11:52:45 -07:00
Shadab Zafar
5a2932adc1 Py3: Fix corresponding base.Integer test in test_language_base 2016-06-06 11:52:45 -07:00
Shadab Zafar
40056e1cd7 Py3: Fix useragent tests 2016-06-06 11:52:45 -07:00
Shadab Zafar
f5a6b295f7 Py3: Fix websocket tests 2016-06-06 11:52:45 -07:00
Shadab Zafar
b86ef9fd04 Py3: Use BytesIO in tutils.render 2016-06-06 11:52:45 -07:00
Shadab Zafar
bafce6cfe9 Py3: Use byte string literals 2016-06-06 11:51:36 -07:00
Shadab Zafar
9a19540f8b Py3: Use BytesIO instead of StringIO for tests 2016-06-06 11:51:36 -07:00
Shadab Zafar
12dd6f6707 Py3: Use global next() instead of iterator method 2016-06-06 11:51:36 -07:00
Shadab Zafar
ced6fb4cf9 Py3: Fix tests by using byte literals 2016-06-06 11:48:58 -07:00
Shadab Zafar
efb0f7133c Py3: Use global next() instead of iterator method 2016-06-06 11:48:54 -07:00
Aldo Cortesi
210c376901 Fix sequencing in Pathoc websocket termination test 2016-06-06 18:17:22 +12:00
Aldo Cortesi
9458eaece7 Make more of the pathod test suite use synchronisation 2016-06-06 16:05:44 +12:00
Aldo Cortesi
08344ee38b Merge pull request #1215 from redfast00/master 
Now keeps header capitalization
 2016-06-06 08:45:35 +12:00
Aldo Cortesi
c31b9c461d Merge pull request #1211 from cortesi/pathod 
WIP: Radical webectomy of pathod
 2016-06-06 08:40:20 +12:00
redfast00
25505733bc
fix multidict unittests 2016-06-05 21:20:37 +02:00
Shadab Zafar
435bfeca0b Use BytesIO instead of StringIO 2016-06-06 00:36:15 +05:30
Shadab Zafar
7b6da98719 Py3: Use byte literals for comparison 2016-06-06 00:07:44 +05:30
Shadab Zafar
be306c8439 Py3: Use global next() function instead of iterator method 2016-06-06 00:07:44 +05:30
Aldo Cortesi
375d7c9741 Remove last vestiges of noapi and noweb from pathod 
Also hide HTTP2 freeze bug by making explain configurable in the tests
 2016-06-05 13:24:46 +12:00
Aldo Cortesi
48da24ae7e First-order removal of pathod API and app 2016-06-05 13:04:13 +12:00
Aldo Cortesi
c0c45c051a Fix test that may fail due to binary header data 
This is just inherently not a determinisitc test. We don't use the log HTTP
interface any more, so it can just go. A more radical "solution" is inbound
shortly.

Fixes #1207
 2016-06-05 11:47:52 +12:00
Shadab Zafar
6ffe8b376e Py3: Write bytes to file 2016-06-04 18:28:50 +05:30
Shadab Zafar
0d4437a404 Fix function name typo 2016-06-04 18:28:03 +05:30
Shadab Zafar
bd7b275d44 Py3: Convert all data to bytes in pathod.language.writer test 2016-06-04 15:23:09 +05:30
Shadab Zafar
614a3d7a54 Py3: Use global next() instead of iterator method 2016-06-04 15:23:09 +05:30
Aldo Cortesi
d5532319c1 Basic ConsoleMaster tests, based on mastertester 2016-06-04 19:07:27 +12:00
Aldo Cortesi
af63a3e064 console/__init__.py -> console/master.py 2016-06-04 16:06:44 +12:00
Aldo Cortesi
12a4f9ecf9 Very basic test truss for web.app 2016-06-04 15:30:28 +12:00
Aldo Cortesi
f7f9cab5dc Rebase on master 2016-06-04 14:37:07 +12:00
Aldo Cortesi
e60860e65d Make tcp.Client.connect return a context manager that closes the connection 2016-06-03 12:06:36 +12:00
Aldo Cortesi
c5076f5e01 Implement a service connection handler counter, use it in Pathod test suite 
Lots of failures, but that's a good thing.
 2016-06-03 11:47:07 +12:00
Aldo Cortesi
c6d15904f3 Merge remote-tracking branch 'upstream/master' into testsuite 2016-06-03 10:19:22 +12:00
Maximilian Hils
29c52d98dc Merge remote-tracking branch 'duffer/pathod-lang-generators' 2016-06-02 09:53:22 -07:00
Aldo Cortesi
5fd52970fd Satisfy linting gods 2016-06-02 19:55:52 +12:00
Aldo Cortesi
c9dd574f48 Silence requests warnings 
Once we convert the rest of the suite to pathoc, this can go.
 2016-06-02 19:40:39 +12:00
Aldo Cortesi
d8c52964c7 Reimplement test retrievals on pathoc and ditch requests 
Requests uses urllib3, which has a connection pool that's not easy to disable
(https://github.com/shazow/urllib3/issues/383).
 2016-06-02 19:36:02 +12:00
Shadab Zafar
b510616c69 Py3: Return bytes from FileGenerator 
and use __getitem__ instead of __getslice__
 2016-06-02 13:00:44 +05:30
Shadab Zafar
a04d7fd166 Py3: Return bytes from RandomGenerator 
and use __getitem__ rather than __getslice__
 2016-06-02 13:00:02 +05:30
Aldo Cortesi
a7522d9308 pathod.test shouldn't use the API at all 2016-06-02 18:27:51 +12:00
Aldo Cortesi
254614e9f7 Since we have locks over the logs, use direct access rather than API requests to get to them 2016-06-02 18:10:36 +12:00
Aldo Cortesi
ffca395e48 Honesty in testing 
- Don't skip tests that fail
- Don't omit console from coverage

I'm working on the pathod races, and can't seem to reproduce the failures
locally.

Also, let's not kid ourselves about the console coverage - it should be
tracked. To improve it, we should measure it. Regressions in coverage of the
module should be flagged by CI.
 2016-06-02 14:44:06 +12:00
Aldo Cortesi
09da1febbd Shift a bunch more string-related functions to strutils 2016-06-02 13:04:19 +12:00
Aldo Cortesi
31012d782f Consolidate some functions to strutils.bytes_to_escaped_str 2016-06-02 12:53:12 +12:00
Aldo Cortesi
cccdc98426 Utils reorganisation: add netlib.strutils 
Extract a number of string and format-related functions to netlib.strutils.
 2016-06-02 12:31:41 +12:00
Aldo Cortesi
eaa3b308f7 Fix non-deterministic test failures in export 
We had various places in the code where we relied on incidental order of dict
keys. Add a helper to multidict, and fix.
 2016-06-02 11:37:18 +12:00
Maximilian Hils
898602ad21 Merge pull request #1180 from dufferzafar/pathod-base 
Python 3 - pathod.language.base
 2016-05-31 14:46:28 -07:00
Maximilian Hils
30fff1fb32 be more strict about module namespaces 2016-05-31 14:05:57 -07:00
Aldo Cortesi
42e91fcfe1 http2.frame -> http2.framereader 2016-05-31 20:32:28 +12:00
Aldo Cortesi
5a75ea3fc6 Fix test failures 2016-05-31 20:19:54 +12:00
Aldo Cortesi
40a030f215 Satisfy flake8 2016-05-31 19:58:28 +12:00
Aldo Cortesi
f62efed304 Unify and make symmetric pretty_size and parse_size 2016-05-31 19:45:48 +12:00
Aldo Cortesi
b2f63458fc Move human-friendly format functions to netlib.human, remove redundant implementations 2016-05-31 19:32:08 +12:00
Aldo Cortesi
ec34cae618 utils.multipartdecode -> http.multipart.decode 
also

utils.parse_content_type -> http.headers.parse_content_type
 2016-05-31 19:07:55 +12:00
Aldo Cortesi
15b2374ef9 netlib.utils.get_header_tokens -> netlib.http1.read.get_header_tokens 
Placing this next to its only use.
 2016-05-31 18:54:42 +12:00
Aldo Cortesi
6dda2b2ee5 Module is part of the name - url.decode, not url.urldecode 
A pattern we need to use far more often in the codebase
 2016-05-31 18:46:19 +12:00
Aldo Cortesi
4e6c9c4e93 Extract url functions from netlib.utils and move to netlib.http.url 2016-05-31 18:42:56 +12:00
Aldo Cortesi
08fbe6f111 Start cleaning up netlib.utils 
- Remove http2 functions, move to http2.frame
- Remove Serializable, move to netlib.basetypes
 2016-05-31 17:16:31 +12:00
Aldo Cortesi
9ea68ebd28 Improve handling of pseudo-headers 
- The canonical source for :method, :scheme and :path are the .method, .scheme
and .path attributes on the request object.
- These pseudo-headers are stripped after reading the request, and re-inserted
just before sending.
- The :authority header remains, and should be handled analagously to the Host
header in HTTP1 with respect to display and user interaction.
 2016-05-31 14:36:17 +12:00
Aldo Cortesi
bc6509ab6d Fix test import error 2016-05-31 10:25:02 +12:00
Shadab Zafar
25e30f42be Py3: Use byte literals for comparison 2016-05-31 01:13:50 +05:30
Maximilian Hils
89f07603ca split mitmproxy.flow into mitmproxy.flow.* 2016-05-30 01:40:09 -07:00
Thomas Kriechbaumer
be9438b01c fix 2016-05-29 22:58:56 +02:00
Thomas Kriechbaumer
8cd0c9c330 fix windows paths 2016-05-29 22:47:51 +02:00
Thomas Kriechbaumer
fb0b591242 fix har test 2016-05-29 22:30:15 +02:00
Thomas Kriechbaumer
03167d5e71 fix headers order 2016-05-29 22:07:22 +02:00
Thomas Kriechbaumer
01fcaa64c3 fix #1162 2016-05-29 13:58:14 +02:00
Thomas Kriechbaumer
e2447406cc more style cleanup 2016-05-29 13:43:20 +02:00
Thomas Kriechbaumer
8eeab37a07 more style cleanup 
Use this to check:
flake8 --count mitmproxy netlib pathod examples test
 2016-05-29 11:55:12 +02:00
Aldo Cortesi
f3bee6f245 Merge pull request #1172 from cortesi/solidcore 
First steps to solidifying the core
 2016-05-29 20:03:40 +12:00
Maximilian Hils
85aa5da6e3 fix tests 2016-05-28 19:31:43 -07:00
Aldo Cortesi
0a875421c5 handle_* -> * 
Now that we have the controller.handler decorator, the _handler prefix
stutters.
 2016-05-29 12:49:01 +12:00
Aldo Cortesi
8bb9285678 Explicitly list all events 2016-05-29 12:15:52 +12:00
Aldo Cortesi
e4f043f69c Flatten ServerMaster into Master 2016-05-29 12:01:46 +12:00
Aldo Cortesi
ed415877d4 Merge branch 'master' into solidcore 2016-05-29 11:14:46 +12:00
Thomas Kriechbaumer
caee1adeb6 cleanup imports with flake8 2016-05-28 14:43:32 +02:00
Maximilian Hils
fa7964988f fix Python 2 tests 2016-05-28 00:17:57 -07:00
Shadab Zafar
b71ef054b5 Py3: Use cStringIO from six.moves 2016-05-28 00:17:57 -07:00
Aldo Cortesi
f4813d5649 Make @controller.handler inheritance-friendly 
Use this to adapt mitmweb and mitproxy console
 2016-05-28 12:38:07 +12:00
Aldo Cortesi
f29f6210bb Mandate that all handlers must be wrapped, make tests pass 
mitmproxy, mitmdump and mitmweb masters still to be done
 2016-05-28 11:00:00 +12:00
Aldo Cortesi
7c4843857b Sketch out a more solid core 
- Decorator for handler methods
- Stricter checking for double-acks and non-acks
 2016-05-28 11:00:00 +12:00
Shadab Zafar
bc6cd13356 Go Python 3 by default with the bytes conversion 2016-05-26 23:07:37 +05:30
Shadab Zafar
1bbb178b6a Remove inner_repr, fixup escape_unprintables 2016-05-26 23:07:37 +05:30
Shadab Zafar
22e4bc1938 Py3: Handle bytes case in inner_repr 2016-05-26 23:07:37 +05:30
Maximilian Hils
d3477e27fa bytes_to_escaped_str: always escape single quotes 2016-05-25 20:11:34 -07:00
Maximilian Hils
7a8da48a30 escaped_str_to_bytes: support unicode on python 2 2016-05-25 19:16:02 -07:00
Thomas Kriechbaumer
ebaad91484 Merge pull request #1127 from mitmproxy/tcp-flows 
mitmdump: Add Basic Support for TCP Flows
 2016-05-23 20:54:14 +02:00
Aldo Cortesi
97f3077082 Merge branch 'mhils-multidict' 2016-05-21 15:01:19 +12:00
Aldo Cortesi
43d7965532 Clean un-needed imports 2016-05-21 15:00:52 +12:00
Aldo Cortesi
123e3b130c We don't need 3 slightly different implementations of Data 2016-05-21 11:53:14 +12:00
Aldo Cortesi
a5c4cd0340 A clearer implementation of MultiDictView 
This makes MultiDictView work with a simple getter/setter pair, rather than
using attributes with implicit leading underscores. Also move MultiDictView
into multidict.py and adds some simple unit tests.
 2016-05-21 11:37:36 +12:00
Aldo Cortesi
96d8ec1ee3 Merge branch 'multidict' of https://github.com/mhils/mitmproxy into mhils-multidict 2016-05-21 10:15:37 +12:00
Maximilian Hils
84144ca0c6 tests: silence flaky pathod test 2016-05-20 14:05:04 -07:00
Maximilian Hils
a3946d2a2d tests++ 2016-05-20 13:27:26 -07:00
Maximilian Hils
b538138ead tests++ 2016-05-20 11:04:27 -07:00
Maximilian Hils
f96697646c add FlowMaster.active_flows 2016-05-19 22:09:00 -07:00
Maximilian Hils
d38989fe7e tests: finish connections properly 2016-05-19 22:07:56 -07:00
Maximilian Hils
d3c30d9005 fix tests, don't double-add error'd flows 2016-05-19 22:07:56 -07:00
Maximilian Hils
acd51befbb minor fixes, adjust tests 2016-05-19 22:07:56 -07:00
Thomas Kriechbaumer
79cb17ef03 Merge pull request #1152 from mhils/cleanup-odict 
clean up odict
 2016-05-19 16:47:51 +02:00
Maximilian Hils
a7b9e3033d tests: properly skip pathod tests 2016-05-18 22:51:27 -07:00
Maximilian Hils
6f8db2d7eb improve MultiDict, add ImmutableMultiDict, adjust response.cookies 2016-05-18 22:50:19 -07:00
Maximilian Hils
8e39b7bf38 test flow export with duplicate query string 2016-05-18 19:28:23 -07:00
Maximilian Hils
d1fc694952 clean up odict 2016-05-18 19:01:49 -07:00
Maximilian Hils
44ac64aa72 add MultiDict 
This commit introduces MultiDict, a multi-dictionary similar to
ODict, but with improved semantics (as in the Headers class).
MultiDict fixes a few issues that were present in the Request/Response
API. In particular, `request.cookies["foo"] = "bar"` has previously been a
no-op, as the cookies property returned a mutable _copy_ of the cookies.
 2016-05-18 18:46:42 -07:00
Maximilian Hils
4c3fb8f509 tests: xfail -> skip 2016-05-18 18:42:35 -07:00
Thomas Kriechbaumer
09b90fce9e remove debug print 2016-05-17 22:45:09 -07:00
Thomas Kriechbaumer
e61014d203 http2: add connection-lost test 2016-05-17 21:25:10 -07:00
Aldo Cortesi
d27fd55657 Fix broken test_flow_export tests 
- Make the tests safe for concurrency
- Fix a test that only passed due to prior modification of global state

This also fixes the project generally for parallel test execution.
 2016-05-18 12:34:45 +12:00
Aldo Cortesi
00cc836470 Clean up flow export tests 
- Externalise test data
- Define a less finicky comparator for Python code
 2016-05-18 12:16:50 +12:00
Thomas Kriechbaumer
2887480bcb fix naming which caused overwriting of functions 2016-05-15 10:39:39 -07:00
Thomas Kriechbaumer
f4a13d6f9f http2: improve header tests (#1143) (#1144) 2016-05-14 10:14:42 -07:00
Maximilian Hils
f1c922c652 Sanitize Print (#1135) 
* sanitize strings with shell control characters

* netlib: add utilities to safe-print bytes

* escaped str: add TODO for multi-byte chars
 2016-05-12 11:03:57 -06:00
Thomas Kriechbaumer
7a813936c6 disable http2 header encoding, use bytes everywhere 2016-05-11 16:47:58 -05:00
Thomas Kriechbaumer
bef72c1b02 Merge pull request #1121 from Kriechi/fix-cookies 
improve cookie parsing
 2016-05-11 12:30:44 -05:00
Thomas Kriechbaumer
e2ee48a0ee replace SimpleCookie with our own parser lib 2016-05-11 12:20:21 -05:00
Maximilian Hils
67a37e6d1f improve script loading 2016-05-11 11:15:36 -06:00
Maximilian Hils
4c8e334642 actually fix tests 2016-05-10 12:19:26 -06:00
Maximilian Hils
bb4f9611f5 Revert "fix tests" 
This reverts commit f315dc1eb9.
 2016-05-10 12:18:05 -06:00
Maximilian Hils
f315dc1eb9 fix tests 2016-05-10 12:17:25 -06:00
Thomas Kriechbaumer
08002282e8 improve cookie parsing 
allows '/' to be within a cookie name
removes deprecated cookie getter/setter
fixes #1118
 2016-05-10 11:13:12 -05:00
Maximilian Hils
9d47d3b1ec fix #1120 2016-05-09 12:15:20 -06:00
Maximilian Hils
74cfd7a4e2 stateobject: support lists 2016-04-29 20:59:26 -07:00
Maximilian Hils
317b4becca mitmdump: don't fail for other flowtypes 2016-04-29 20:18:32 -07:00
Maximilian Hils
54c4859a95 remove netlib version check 2016-04-29 12:00:38 -07:00
Maximilian Hils
909d5ec87e FlowReadError -> FlowReadException 2016-04-29 11:17:49 -07:00
Steven Van Acker
66267ad276 support for setting/sending multiple cookies (#1091) 
* support for setting/sending multiple cookies

* py.test for multiple cookie support
 2016-04-22 11:16:05 -07:00
Maximilian Hils
5b12720962 silence flaky pathod test 2016-04-21 18:29:16 -07:00
Maximilian Hils
028f5c4929 downgrade pyparsing to fix #1087 and #1090 2016-04-21 17:11:28 -07:00
Maximilian Hils
423c076c61 cleanup mitmproxy.controller, raise Kill in Channel (#1085) 2016-04-14 12:03:29 -07:00