Grasscutters/mitmproxy
3
0
Fork 0
mirror of https://github.com/Grasscutters/mitmproxy.git synced 2024-11-26 18:18:25 +00:00
Code Issues Projects Releases Wiki Activity
8,890 Commits 1 Branch 67 Tags 50 MiB
3f7b850268
Commit Graph

2156 Commits

Author SHA1 Message Date
Matías Lang
e6da62a50a Merge branch 'master' of https://github.com/mitmproxy/mitmproxy 2019-01-13 23:55:27 -03:00
Matías Lang
eab4174b87 Fix command injection when exporting to httpie 
The command generated by `export.clip httpie @focus` or `export.file
httpie @focus /path/to/file` wasn't being properly escaped so it could
contain a malicious command instead of just a simple httpie call.
 2019-01-13 23:45:28 -03:00
Matías Lang
d027891cec Fix command injection when exporting to curl 
The command generated by `export.clip curl @focus` or `export.file curl
@focus /path/to/file` wasn't being properly escaped so it could contain
a malicious command instead of just a simple curl.
 2019-01-13 23:39:50 -03:00
Maximilian Hils
82bc8c7ca2
Merge pull request #3444 from BoboTiG/fix-resource-leaks 
Fix ResourceWarning: unclosed file, prevent resource leaks
 2019-01-06 17:43:47 +01:00
Mickaël Schoentgen
c03b07930c Fix ResourceWarning: unclosed file, prevent resource leaks 
Signed-off-by: Mickaël Schoentgen <contact@tiger-222.fr>
 2019-01-06 15:06:30 +01:00
Mickaël Schoentgen
3a2d7bb119 Fix several DeprecationWarning: invalid escape sequence 
Signed-off-by: Mickaël Schoentgen <contact@tiger-222.fr>
 2019-01-05 23:37:48 +01:00
Chih-Hsuan Yen
cc33f40f29
Fix a failing test on macOS Mojave 2018-12-29 21:24:46 +08:00
Rajat Gupta
e2bcca47b1 charset in meta tags (#3411) 
original contribution from @0xHJK in https://github.com/mitmproxy/mitmproxy/pull/3150
 2018-12-13 15:34:12 +01:00
Rajat Gupta
db658b12ed fix query array 
fixes #3072 
closes #3254
 2018-12-03 22:05:59 +01:00
Thomas Kriechbaumer
2fb2b48a06 bump dependencies 2018-12-02 15:48:20 +01:00
Abhigyan Khaund
cda4248610 Change variable o to organization for generated certficates. 2018-11-11 10:58:14 +05:30
Abcdefghijklmnopqrstuvwxyzxyz
d4f4cfe225 Add Organization field for the generated certificate (#3376) 
add organization field for the generated certificate
 2018-11-09 09:06:04 +01:00
Jessica Favin
0cbbcffd89 test_dumper.py - Add sio_err everywhere + adjust test_simple 2018-11-04 17:24:34 +01:00
Jessica Favin
312f922316 Fix test_dumper.py - Dumper constructor 2018-11-04 16:26:51 +01:00
Jessica Favin
ade136dc4d Update test_dumper.py 2018-11-04 16:06:17 +01:00
Maximilian Hils
28551e9655
use skip_windows decorator consistently 2018-10-23 15:24:59 +02:00
Fred Miller
bf3570b3b9
Skip file permission test on Windows 2018-10-23 08:53:51 +08:00
Fred Miller
f16621a38b
Make private keys readable only by the owner 2018-10-22 22:53:58 +08:00
Miroslav
ed9e3d5137 keys.yaml priority over defaultkeys.py. Test for this. 2018-09-29 15:54:17 +03:00
Maximilian Hils
e15619f34e
Merge pull request #3319 from mhils/issue-3316 
Fix #3316
 2018-09-07 10:34:55 +02:00
Maximilian Hils
1b6a8d6acd fix #3316 2018-09-07 10:24:28 +02:00
David Kremer
dcd8ba34ab [test/xss_scanner] add fixtures in tests 
The previous version of that script made a repetitive use
of the  pytest builtin fixture, with always
the same arguments.

This is a small refactoring and cleanup, mainly adding the 'function'
scope to ensure proper cleaning and using the @pytest.fixture
decorator where I could.
 2018-08-15 19:47:55 +02:00
madt1m
a52451900c session: implemented changes requested after PR review. 2018-08-05 21:57:55 +02:00
madt1m
e9c2b12dab tests: Full coverage. Everything working, ready for review 2018-08-03 16:32:50 +02:00
madt1m
4e0c10b88b tests: 97% coverage reached. Session opportunely patched after emerged defects. 2018-08-02 05:55:35 +02:00
Pietro Francesco Tirenna
9c949bd2f8
Merge pull request #3252 from madt1m/session-db 
Session - Hybrid DB
 2018-07-24 16:26:10 +02:00
madt1m
8c7793b91a session: temporary DB is now stored in temporary dir 2018-07-24 15:57:11 +02:00
madt1m
68eb07b668 session: modified schema. Now SessionDB uses tempfile module for temp session 2018-07-24 11:58:33 +02:00
madt1m
e727446f14 benchmark: some improvements - limit to queue size 2018-07-23 21:18:24 +02:00
madt1m
8ab82ad9a3 benchmark: new protobuf serialization performance profiler 2018-07-22 12:47:54 +02:00
Aldo Cortesi
ec092fdc12
Merge pull request #3245 from madt1m/protobuf-serialization-clean 
Shifting to Protobuf Serialization - Cleaned
 2018-07-21 16:13:25 +12:00
madt1m
3b5cdf7f67 test_linting: removed some unused local variables 2018-07-20 18:15:27 +02:00
madt1m
fad8e7c99b tests: SessionDB fully tested 2018-07-20 16:58:11 +02:00
Miroslav
dcb3de40b1 Some refactoring. New test case. 2018-07-19 16:56:34 +03:00
Pietro Francesco Tirenna
d5da74645b protobuf: tests implemented, full coverage 2018-07-17 19:23:01 +02:00
Miroslav
ffbd7c20e5 Command history implementation 2018-07-17 18:37:45 +03:00
Thomas Kriechbaumer
3cd3765270
Merge pull request #3211 from cortesi/tcheck 
Add typechecking of Any values for state object
 2018-06-17 08:45:03 +02:00
Aldo Cortesi
77b49aa8de Add typechecking of Any values for state object 
An ugly solution for an ugly little problem. This patch uses JSON's type
checker to validate Any values in stateobject, in order to avoid a circular
import.

Fixes #3180
 2018-06-17 10:22:17 +12:00
Aldo Cortesi
9ff4f55614
Merge pull request #3202 from madt1m/view-cleanup 
View Cleanup - Initial steps
 2018-06-17 09:20:34 +12:00
Aldo Cortesi
9463fee764 cibuild: permit non-dev versions on maintenance branches 
Cater for the corner case where commits are incorporated on a maintenance
branch. We should be able to test these without adding a dev suffix to the tool
versions.
 2018-06-16 15:09:34 +12:00
Aldo Cortesi
85526e5370 cibuild: Match Travis env variable behaviour 
Whenever TRAVIS_TAG is non-empty, TRAVIS_BRANCH is over-ridden to match the
TRAVIS_TAG value. Adjust our tests to reflect this, and add a sanity check that
fails hard if this constraint is ever not met.

See https://github.com/travis-ci/travis-ci/issues/4745
 2018-06-16 10:23:33 +12:00
Aldo Cortesi
bf791ba1d5 cibuild: refactor to handle v prefix for release tag names 2018-06-16 09:48:21 +12:00
madt1m
af54c26014 fixed exception type; full coverage on view 2018-06-13 17:39:46 +02:00
madt1m
40faf2c662 Added tests for new primitives 2018-06-13 15:57:21 +02:00
madt1m
773c953514 View API slightly extended; codebase cleaned in some points 2018-06-13 11:56:14 +02:00
Thomas Kriechbaumer
be26958ea8 improve websocket frame masking api 2018-06-11 13:28:28 +02:00
Maximilian Hils
09ff5df2fb Such CI, Very Wow. (#3182) 2018-06-02 21:37:44 +02:00
Maximilian Hils
aa0cb2ba2f add missing await 2018-05-28 22:27:27 +02:00
Aldo Cortesi
a38d2d7b0e addons/block: teach block about IPv6 scope suffixes 
Fixes #3160
 2018-05-27 10:48:23 +12:00
Aldo Cortesi
ec2ae19e22 optmanager: tweaks and cleanups 2018-05-27 10:43:14 +12:00
Aldo Cortesi
e59ba13417 Use deferral mechanism for cfg file options 
Fixes #3162
 2018-05-27 10:12:24 +12:00
Aldo Cortesi
b6e1c4bb9d cibuild: tweak docker tag, only upload prod builds to pypi 2018-05-25 12:46:56 +12:00
Aldo Cortesi
ff92962c51 cibuild: start building version awareness 2018-05-25 10:50:48 +12:00
Aldo Cortesi
fdf0d9974e cibuild: more accurate PR test, better dump output 2018-05-24 22:58:33 +12:00
Aldo Cortesi
e9b19dba8a cibuild: fix docker upload condition, expand tests 2018-05-24 22:32:31 +12:00
Aldo Cortesi
c91b511bc7 cibuild: extract upload conditions 2018-05-24 22:18:01 +12:00
Aldo Cortesi
0afff3a952 cibuild: add docker tag calculation to environ 2018-05-24 21:59:45 +12:00
Aldo Cortesi
bae4a3393f release: include release directory in coverage 
With lots of exclusions for now
 2018-05-24 19:03:15 +12:00
Aldo Cortesi
8fceaca6b8 cibuild: Consolidate build environment and add tests 2018-05-24 11:06:43 +12:00
Aldo Cortesi
064929b559 clientreplay: fix racy tests harder 2018-05-18 09:51:56 +12:00
Thomas Kriechbaumer
976ab0c466 websocket: inject messages via flow 2018-05-16 21:50:15 +02:00
Thomas Kriechbaumer
9979be6487 fix py36/37 escaping issues 
closes #3122

https://bugs.python.org/issue16285
https://bugs.python.org/issue12910
21024f0662
 2018-05-14 10:41:46 +02:00
Aldo Cortesi
58ff51da10 command: handle string without terminal escaped char 
Fixes #2810
 2018-05-12 14:02:20 +12:00
Aldo Cortesi
88fe26997c script: revamp ephemeral script running 
Fixes 2838
 2018-05-12 12:35:24 +12:00
Aldo Cortesi
482043cdcf commands: handle type errors on startup 
fixes #3088
fixes #2787
 2018-05-12 11:10:14 +12:00
Aldo Cortesi
6dff8c58ad commands: if no explicit return type is specified, assume None 
This is going to be a super common error for addon authors, so we might as well
handle it.
 2018-05-12 10:15:08 +12:00
Aldo Cortesi
8c63a8818d keymap: read keys from CONFDIR/keys.yaml by default 2018-05-10 16:06:52 +12:00
Aldo Cortesi
9830e5b597 cadir -> confdir 
We store a lot more than just the CAs in our configuration directory. Clarify
the option name.
 2018-05-10 11:40:33 +12:00
Aldo Cortesi
3438912236 console keybindings: define YAML-based format for console key binding persistence 2018-05-10 11:30:51 +12:00
Aldo Cortesi
7d9b626d2e clientplayback: try to fix flaky replay test 2018-05-08 16:33:41 +12:00
Aldo Cortesi
af1a4ffdcd Ditch the addon stdout wrapper 
This results in a 30% improvement in our core request throughput.

Fixes #3102
 2018-05-08 14:26:41 +12:00
Aldo Cortesi
717fbaa990 optmanager: refactor for coverage and clarity 2018-05-08 14:24:41 +12:00
Aldo Cortesi
1b0eed19d1 benchmarks: quick-run scripts for mitmproxy and mitmdump 2018-05-08 11:08:37 +12:00
Aldo Cortesi
9b51393b6d benchmark: keep track of requests/responses seen by proxy 2018-05-08 11:00:20 +12:00
Aldo Cortesi
f7d7e31f06 options: add the concept of deferred settings 
We've had a perpetual sequencing problem with addon startup. Users need to be
able to specify options to addons on the command-line, before addons are
actually loaded. This is only exacerbated with the new async core, where load
order can't be relied on.

This patch introduces deferred options. Options passed with "--set" on the
command line are deferred if they are unknown, and are automatically applied by
the addon manager once matching addons are registered and their options are defined.
 2018-05-08 10:56:00 +12:00
Maximilian Hils
752a367436 remove leftover processing_complete 2018-05-06 15:47:12 +02:00
Aldo Cortesi
60acbd79b9 Remove allowremote addon, add an improved take called block 
We now have two options: block_global blocks global networks, block_private
blocks private networks. The block_global option is true by default, and
block_private is false by default. The addon name is "block" so the options are
correctly prefixed.

Also make option documentation precise, reduce verbosity of logs.
 2018-05-06 12:43:25 +12:00
Aldo Cortesi
c53bc39c95
Merge pull request #3099 from Kriechi/fix-3024 
fix #3024
 2018-05-06 11:27:14 +12:00
Thomas Kriechbaumer
54e2daa21e fix #3024 2018-05-05 14:38:02 +02:00
Aldo Cortesi
6bd2d5059b Remove custom events 
These were always weird, and were added to support exit when processing
completed to mitmdump. We now have better ways to do this.
 2018-05-03 09:42:51 +12:00
Aldo Cortesi
f380a77dee Remove the tick event 
Mitmproxy: the tickless wonder.
 2018-05-02 14:29:15 +12:00
Aldo Cortesi
38ff8109fb taddons: remove has_event 
We no longer use this anywhere, so ditch it.
 2018-05-02 11:31:28 +12:00
Aldo Cortesi
22a4b1d5d4 Redesign keepserving 
- Instead of listening for a pseudo-event, we periodically check whether client
replay, server replay or file reading is active.
- Adjust server replay not to
use tick.
- Adjust readfile to expose a command to check whether reading is in progress.
 2018-05-02 11:26:40 +12:00
Aldo Cortesi
00d790fe84 commands: clarify command call interface, fix web app replay 2018-05-02 08:36:15 +12:00
Aldo Cortesi
6d27b28b85 client replay: expad and consolidate tests 2018-05-01 08:47:26 +12:00
Aldo Cortesi
236a2fb6fd client replay: re-design 
Re-design the way client replay works. Before, we would fire up a thread,
replay, wait for the thread to complete, get the next flow, and repeat the
procedure. Now, we have one replay thread that starts when the addon starts,
which pops flows off a thread-safe queue. This is much cleaner, removes the
need for busy tick, and sets the scene for optimisations like server connection
reuse down the track.
 2018-04-30 17:17:03 +12:00
Aldo Cortesi
28d53d5a24 client replay: move all client replay-related code into addon 2018-04-27 16:34:56 +12:00
Aldo Cortesi
a4a48a96d6
Merge pull request #3078 from kajojify/expanduser 
Fix #3002. Auto-expanding for tilda.
 2018-04-26 21:02:10 +12:00
Aldo Cortesi
b18e0fa2bc
Merge pull request #3071 from cortesi/tickless 
asyncio: shift script reloading out of the tick event
 2018-04-26 11:47:07 +12:00
Miroslav
ea6fab09b5 Fix #3002. Auto-expanding 2018-04-26 01:35:44 +03:00
Maximilian Hils
19ceb6f49a fix pytest invocation errors 2018-04-25 14:00:50 +02:00
Aldo Cortesi
95e690ba31 test: shift test_data out of our public API 2018-04-23 19:16:33 +12:00
Aldo Cortesi
0ba10b6109 addons/script: improve relability of reload test 
The granularity of mtime is surprisingly bad. Make the tests more robust
against this, and promote has_log back to a public method, now that we have a
few legitimate examples.
 2018-04-23 13:19:20 +12:00
Aldo Cortesi
44016a0de5 asyncio: shift script reloading out of the tick event 
The tick event is a nasty compromise, left over from when we didn't have an
event loop. This is the first patch in a series that explores moving our
built-in addons to managing coroutines on the eventloop directly for periodic
tasks.
 2018-04-23 10:28:18 +12:00
Aldo Cortesi
ef4db52887 Ignore signal errors on windows + various fixes 
- Ignore the NotImplementedError raised by add_signal_handler on Windows.
- Entrypoints return an integer exit code, or None. Adjust our type annotations
and code to suit.

Fixes #3061
 2018-04-17 08:20:52 +12:00
Aldo Cortesi
565146311a asyncio: clarify shutdown semantics 
This patch clarifies proxy shutdown, and specifies that the master.shutdown()
method is thread-save.
 2018-04-16 10:16:51 +12:00
Aldo Cortesi
850c855495 Clean up test/helper_tools directory 2018-04-16 09:20:22 +12:00
Aldo Cortesi
b663a224a3 Improve benchmarking 
- The benchmark addon now manages setting up and tearing down the backend and
traffic processes itself.
- Use wrk instead of hey. I get more consistent results with this tool, and hey
shows a strange tail-latency bump that seems artificial.
- Make termination behaviour simpler. The bencmark revealed a bug where .done
events were not called if the proxy was shut down by an addon.
 2018-04-16 09:19:14 +12:00
Aldo Cortesi
5f74adc2df
Merge pull request #3059 from obscure76/enh/issue-3053 
fix Python 3.6 variable type annotations #3053
 2018-04-16 07:35:20 +12:00
oscure76
0e984e1442 fix Python 3.6 variable type annotations #3053 2018-04-14 16:24:41 -07:00
Aldo Cortesi
865a373bd3
Merge pull request #3050 from kira0204/error-script 
Handling user script's exceptions, fix #2839
 2018-04-15 10:12:44 +12:00
Aldo Cortesi
8609de6f31 readfile: add a readfile_filter option 
* Add a readfile_filter option that filters flows on read.
* Adjust test suite for asyncio.
* Add asynctest as a dev dependency.
 2018-04-14 11:52:02 +12:00
kira0204
6780e5025a Handling user script exceptions, fix #2839 2018-04-10 08:33:40 +05:30
Aldo Cortesi
9dcd15d350 asyncio: ditch the handler context 
There are a few reasons for this. First, logs are now async, and can be called
at any time. Second, the event loop is thread local, so there can only ever be
one master per thread. These two things together completely obviate the need
for a handler context.
 2018-04-07 09:04:14 +12:00
Aldo Cortesi
6a08ef465f asyncio: remove master.add_log, in favor of a persistent log.Log instance 2018-04-07 09:02:10 +12:00
Aldo Cortesi
ea213782d8 asyncio: remove test master.has_log 
Now that logs are async, using this call is almost always a mistake. Signal
this by making it semi-private. The method may go away entirely down the track.
 2018-04-07 09:02:10 +12:00
Aldo Cortesi
80f2bac356 asyncio: move log mechanism onto the event loop 
Logs are now asynchronous, with a log entry pushed onto the event loop for
handling. To support this, the test mechanism grows an await_log method that
waits for a log entry to appear.
 2018-04-07 09:02:10 +12:00
Aldo Cortesi
0fa1280daa asyncio simplify: we don't need a queue for proxy->main loop comms 
Instead, we just schedule coroutines directly onto the core loop.
 2018-04-07 08:59:08 +12:00
Aldo Cortesi
cdbe6f97af asyncio: remove last vestiage of channel input from master 2018-04-07 08:59:08 +12:00
Aldo Cortesi
506ccc5693
Merge pull request #3031 from itzikBraun/add-httpie-export 
added option to export request as httpie command
 2018-04-06 10:02:02 +12:00
Aldo Cortesi
4177f50a0f
Merge pull request #2971 from kira0204/user-script-exception 
A unified handler for user's scripts exception
 2018-04-06 10:01:35 +12:00
Maximilian Hils
50ea3aa420 fix http retry timeout 
this fixes #3038
 2018-04-05 15:30:04 +02:00
Aldo Cortesi
659fceb697 minor fixes 2018-04-03 10:26:09 +12:00
Aldo Cortesi
7e73e5fa51 asyncio: factor out test server startup wait 2018-04-03 09:46:11 +12:00
itzikBraun
29f4164344 added option to export request as httpie command 2018-04-02 12:57:33 +02:00
Aldo Cortesi
14f461c5d5 asyncio: cleanup and lint 
Also fix a racy websocket test.
 2018-04-02 16:41:49 +12:00
Aldo Cortesi
a3da43d3e5 asyncio: test cleanup 
Also silence asyncio logs. We sometimes end up with messages on the queue that
need to be ignored when the proxy shuts down, and asyncio complains loudly
about this.
 2018-04-02 14:51:14 +12:00
Aldo Cortesi
2b040ff093 asyncio: make http2 tests pass 2018-04-02 11:03:05 +12:00
Aldo Cortesi
28a8ddc0e8 asyncio: fix a few remaining issues in proxy/test_server.py 2018-04-02 09:46:02 +12:00
Aldo Cortesi
3cc5d81a4a asyncio: fix channel interface and tests 
We now need to synthesize a tick event when changing addons in tests, because
tick is asynchronously called on the event loop.
 2018-04-01 11:37:35 +12:00
Aldo Cortesi
54170ee657 asyncio: start a test loop for unit tests 
Also move signal handling out of master. These only work in the main thread,
and properly belong in main.py.
 2018-04-01 09:46:32 +12:00
Aldo Cortesi
976b2018a3 asyncio: clean up event loop acquisition 
We now acquire the event loop through asyncio.get_event_loop, avoiding having
to pass the loop explicity in a bunch of places. This function does not return
the currently running loop from within coroutines in versions of Python prior
to 3.6.
 2018-04-01 09:46:32 +12:00
Aldo Cortesi
a2d4519354 asyncio: brutally rip out our old queue mechanism 2018-04-01 09:46:32 +12:00
Thomas Kriechbaumer
46f8901b8c Python 3.5 is dead -- long live Python 3.6! 
fixes #2266
 2018-03-31 18:46:32 +02:00
kira0204
22b5149777 use error handler function 2018-03-26 03:28:28 +05:30
Thomas Kriechbaumer
eb297d82aa fix Tornado 5.0 event loop 
https://github.com/tornadoweb/tornado/issues/2183#issuecomment-371001254
 2018-03-24 11:19:04 +01:00
Maximilian Hils
fed54fa3d0 don't crash if server address is unknown, fix #2969 2018-03-23 04:26:46 +01:00
Aldo Cortesi
c5ad026cbe bench: Add some very simple manual benchmarking helpers 
This includes a profiler addon that we might consider for promotion to a
builtin down the track.
 2018-03-17 10:06:46 +13:00
Maximilian Hils
2001184b6c
Merge pull request #2956 from kira0204/test-for-2850 
Test for #2850
 2018-03-15 00:29:52 +01:00
kira0204
8aad2d63cf adding command 2018-03-12 09:13:29 +05:30
kira0204
5dcc3b4ff8 Testing using addon 2018-03-12 00:28:43 +05:30
kira0204
167e01acdf fixing lint 2018-03-07 16:12:12 +05:30
kira0204
b8fbe71c93 User script exception handler[squash] 
fixing 2837, added test
unified-function
deleting the wrong commit
 2018-03-07 13:18:19 +05:30
kira0204
57197c3e6c using asserts 2018-03-07 06:34:16 +05:30
kira0204
be50f3f481 wrong additions 
test-for-2850

few fixes

mock testing

Typo error
 2018-03-06 05:44:05 +05:30
Aldo Cortesi
bc6550abc5
Merge pull request #2945 from kajojify/shortening 
Polite shortening of statusbar messages. Fix #1433
 2018-03-05 09:07:18 +13:00
Maximilian Hils
a2740ee4ae
Merge pull request #2868 from kira0204/data-crash 
Fix crashing when editing form with random data, fix #2794
 2018-03-04 19:53:15 +01:00
Miroslav
237320a539 Minor fixes. 
Renaming prep_message method to shorten_message.
Docstring.
\u2026 instead of ....
max_width parameter.
 2018-03-03 22:12:56 +02:00
Miroslav
0dcdb6c3cc Finalization of tests. Conditions rewritten again. 2018-03-02 23:40:58 +02:00
Miroslav
d151c6c322 Initial tests. New conditions. 2018-03-01 01:18:05 +02:00
Maximilian Hils
944e81dcfc clean up ProxyConfig 
some of these options weren't even used anymore,
others only in one place where it makes sense to use options directly.
 2018-02-27 19:05:59 +01:00
Aldo Cortesi
7bd63ee713 Start consolidating core options 
This is a preparatory patch that paves the way to consolidating our core
options in the core addon. It amalgamates the core_option_validation and core
addons, prepares the test suite for a world where options live in core, and
moves over two trivial options as a trial balloon.

From here, things will get harder, but at the end of the process we'll have a
core that's responsive to options.
 2018-02-26 11:26:32 +13:00
Aldo Cortesi
e5cdc20eb8 Split dumper_filter off from view_filter 
These don't even do the same thing.
 2018-02-26 09:43:15 +13:00
Aldo Cortesi
2fb85553cc Revamp verbosity options 
Split verbosity into termlog_verbosity and console_eventlog_verbosity.

This patch also removes printing to console if there are unknown options in the
command-line. Options now live in separate addons, so having uknown options
remaining is common and expected. We definitely shoould have some other way for
users to see what was ignored so they can catch typos and the like, but that's
a different patch.
 2018-02-26 09:02:24 +13:00
Aldo Cortesi
154309b811
Merge pull request #2909 from fristonio/Fix-2867 
Fix test fails in test_xss_scanner when running tox without internet connection.
 2018-02-25 23:00:23 +13:00
luz.paz
afbb7f117b Misc. typos 
Found via `codespell -q 3 -I ../mitmproxy-word-whitelist.txt`
Where whitelist contains:
```
cas
doubleclick
nd
ot
seeked
statics
te
thru
```
 2018-02-24 21:45:11 -05:00
Aldo Cortesi
6b5b71aefa
Merge pull request #2907 from cortesi/optionscomp 
Start moving more complicated options over to /addons
 2018-02-25 10:15:21 +13:00
Deepesh Pathak
9fed4fa40a
Fix test fails in test_xss_scanner when running tox without internet connections. 
- Add mock implementation for gethostbyname in test_xss_scanner.
- Fix failed tests when running tox without internet connection.
- Fixes #2867
 2018-02-24 16:50:40 +05:30
Aldo Cortesi
6ea81a96d4 addon options: proxyauth to /addons 2018-02-24 19:16:13 +13:00
Aldo Cortesi
59db25bd0f Split default_contentview into dumper_* and console_* varieties 
Then move the new options into their respective addons. This was a shared
option, and things are much much simpler if it isn't.
 2018-02-24 19:09:32 +13:00
Aldo Cortesi
f49aae312f Move intercept options to addon 
As part of this, we zap an un-needed console command and use a boolean toggle
setter instead.

Also remove an ancient regression test that breaks encapsulation.
 2018-02-24 18:33:15 +13:00
Aldo Cortesi
3a766a420c
Merge branch 'master' into opts 2018-02-24 16:47:58 +13:00
Aldo Cortesi
93c49c47ae flow_detail option to /addons/dumper 2018-02-24 16:42:32 +13:00
Aldo Cortesi
4fe83be63c Resolve some left-over cross-addon dependencies 2018-02-24 16:06:11 +13:00
Aldo Cortesi
f361ea491c addon options: upstream_auth 
Also the last "easy" option to shift.
 2018-02-24 15:58:37 +13:00
Aldo Cortesi
2aa7ac584b addon options: stickycookie, streambodies 2018-02-24 15:49:38 +13:00
Aldo Cortesi
704c1db1b7 addon options: setheaders, stickyauth 2018-02-24 15:45:12 +13:00
Aldo Cortesi
12633adeb9 addon options: readfile, save, script 2018-02-24 15:28:43 +13:00
Aldo Cortesi
144b559b46 addon options: migrate replace, simplify taddons.context 2018-02-24 15:22:28 +13:00
Aldo Cortesi
52c8d7e0f8 addon options: shift server_replay options 2018-02-24 15:00:40 +13:00
Aldo Cortesi
74a0230788 addon options: shift keepserving and onboarding options 2018-02-24 14:53:54 +13:00
Aldo Cortesi
a12d149230 client options: migrate allowremote and clientplayback 
Also triage options, and categorize them into core options (won't be migrated),
options that are hard to migrate for various reasons, and easy migrations.
 2018-02-24 14:39:03 +13:00
Aldo Cortesi
42094b29ff Clean up un-necessary arguments to taddons.context 
Also test coverage ++
 2018-02-24 13:22:15 +13:00
Aldo Cortesi
4522a6f7b7 Start moving addon options into /addons 
This takes the first few steps:

- Extends taddons to make loading addon options easier
- Removes dependencies in the test suite on options in addons
- Tweaks command-line parser autocreation to ignore nonexistent options. This
lets us load common options without over-depending on loaded addons.
 2018-02-24 12:13:52 +13:00
Aldo Cortesi
16dd7f3ddf
Merge pull request #2886 from tran-tien-dat/set-cookie 
Parse Set-Cookie header more permissively. Fix #2829
 2018-02-24 10:49:33 +13:00
Aldo Cortesi
c471c42c7c Fix an issue with quoting and commands 
Previously, quotes would disappear during editing, making it
impossible to enter command arguments with spaces.
 2018-02-23 13:04:31 +13:00
Tran Tien Dat
97e534e76b Fix #2829 2018-02-21 19:58:29 +08:00
Miroslav
34dbd9adbb Flake, be happy 2018-02-20 21:45:14 +02:00
Miroslav
6c525317cc one more cleanup 2018-02-20 20:14:36 +02:00
Miroslav
aea6fd28c8 Hotkeys cleanup 2018-02-20 16:25:39 +02:00
Maximilian Hils
da4c1ee625
Merge pull request #2881 from cortesi/refreshopt 
option refresh_server_playback -> server_replay_refresh
 2018-02-19 22:52:19 +01:00
Aldo Cortesi
e9bdada1f5 option replay_kill_extra -> server_replay_kill_extra 
Continuing effort to collect all sever replay options under a commmon prefix.
 2018-02-20 09:46:47 +13:00
Aldo Cortesi
8129b93bb0 option refresh_server_playback -> server_replay_refresh 
All other server playback options have this form.
 2018-02-20 09:30:29 +13:00
kira0204
fb54bb3777 Testing 2018-02-19 22:45:30 +05:30
kira0204
60cd8cb0b1 Adding test 2018-02-16 22:30:29 +05:30
Maximilian Hils
26cb6b7936
Merge pull request #2847 from kira0204/key-cookie-issue 
Handling empty key with value for cookies, fix #2827
 2018-02-12 11:22:09 +01:00
kira0204
06f689aa34 keeping consistency, better testing 2018-02-11 17:25:44 +05:30
Maximilian Hils
cda7c8d754
Merge pull request #2851 from mhils/always-use-tls-settings 
Use TLS options also for request replay
 2018-02-10 12:40:39 +01:00
Maximilian Hils
e4f70b7957 use TLS options also for request replay, refs #2840 2018-02-08 13:17:33 +01:00
Miroslav
bd375c2669 Excess return was deleted 2018-02-08 00:31:57 +02:00
kira0204
4e9d4b37b3 fixing logic 2018-02-07 04:41:12 +05:30
Arushit Mudgal
91834f98cc Extend mypy checking, fix #2194 (#2819) 2018-02-03 21:37:33 +01:00
Samoilenko Roman
7733252627 don't raise when pyperclip doesn't find a clipboard, fix #2816 2018-02-03 13:53:09 +01:00
Maximilian Hils
4decb5c2e3 remove quotes around filename 2018-02-01 09:57:25 +01:00
Samoilenko Roman
4b93e16e78 Fix for #2750 and #2783 (#2809) 2018-01-25 16:07:09 +01:00
Arushit Mudgal
f41d521ce5 fix #2800 2018-01-25 16:06:19 +01:00
Samoilenko Roman
2d07678fe5 Fix #2765 (#2792) 2018-01-14 21:54:43 +01:00
Samoilenko Roman
6b22da6a34 Fix #2768 (#2778) 2018-01-14 21:54:09 +01:00
Maximilian Hils
9a43ff5da1
Merge pull request #2793 from kajojify/issue-2399 
Issue 2399
 2018-01-14 21:52:32 +01:00
Miroslav
267b1af835 Fix #2399 2018-01-14 13:41:15 +02:00
Maximilian Hils
d9e3fcf5ef store ClientHello extensions with client connection 2018-01-13 01:49:20 +01:00
Maximilian Hils
96a5ed9dff
Merge pull request #2790 from mhils/stateobject-improvements 
stateobject: use typing, enable tuples and more complex datatypes
 2018-01-13 00:35:49 +01:00
Maximilian Hils
69726f180a stateobject: use typing, enable tuples and more complex datatypes 2018-01-13 00:33:37 +01:00
Maximilian Hils
93d9640ea5 improve net.tls.ClientHello API, fix bugs, add tests 2018-01-13 00:10:47 +01:00
Maximilian Hils
fc80aa562e move ClientHello and related functions to mitmproxy.net.tls 2018-01-12 21:40:35 +01:00
Matthew Shao
137a78f22a Add test for fix#2766. 2018-01-11 09:23:50 +08:00
Matthew Shao
a6f6f8cd32 Raise a CommandError for the exception 2018-01-08 21:51:03 +08:00
Matthew Shao
633bc350d2 Fix #2760 2018-01-08 21:14:35 +08:00
Samoilenko Roman
821d76df02 Fix #2767 (#2770) 2018-01-07 23:05:39 +01:00
Maximilian Hils
15f525e86b only use annotated tags for version calculation 2018-01-07 21:18:42 +01:00
Thomas Kriechbaumer
4fb894cad4 avoid TLS/SSL ambiguity for Cert class 2018-01-07 19:55:04 +01:00
Thomas Kriechbaumer
d15e96dee1 rename TLS/SSL-related functions 
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for function names.
 2018-01-06 10:43:54 +01:00
Thomas Kriechbaumer
9aae3213b9 rename TLS/SSL-related attributes 
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for attributes.
 2018-01-06 10:43:47 +01:00
Maximilian Hils
2e2daeed89 refactor common.format_keyvals 
the semantics here were really quite unclear,
now it is hopefully a bit more obvious what's happening.
Once we are Python 3.6+ exclusively, we may consider changing
the signature to accept a (order-preserving) dict instead of a list.
 2018-01-05 16:45:50 +01:00
Maximilian Hils
c2adcb58f4 fix test flow timestamps to values that don't overflow in certain timezones 2018-01-05 16:45:43 +01:00
Maximilian Hils
dd7902cf7f fix #2470 2018-01-04 16:31:45 +01:00
Maximilian Hils
d38b5e8552 do not add leading zeroes to .dev version part. 
wheel normalizes to no leading zeroes, and we don't want to special-case
that.
 2018-01-04 15:42:09 +01:00
Maximilian Hils
d9d4d15ee0
Merge pull request #2733 from mitmproxy/pyinstaller 
Update PyInstaller, add dev version info to builds.
 2017-12-30 22:29:49 +01:00
Maximilian Hils
9dc3d16bf2 single-source version processing 2017-12-30 18:48:47 +01:00
Maximilian Hils
afe6e587f9
Merge pull request #2703 from mhils/test-defaultkeys 
Test that default key bindings are valid commands
 2017-12-30 17:37:33 +01:00
Maximilian Hils
465044c373 test that defaultkeys are valid commands 2017-12-29 23:20:03 +01:00
Maximilian Hils
6232622774 fix Flow.kill behaviour 
This now just sets a kill reply instead of committing directly.
First, this seems like the more sane thing to do.
Second, we have an iffy race condition where we call Reply.commit()
before the addonmanager finishes its invocation, the proxy thread then progresses
and sets a new flow.reply attribute, and the addonmanager then gets confused
when finishing. This commit doesn't fix that, but mitigates it for Flow.kill
which is now committed by the addonmanager.
 2017-12-29 22:56:29 +01:00
Thomas Kriechbaumer
59c277effd websocket: add flow kill test 2017-12-29 22:51:13 +01:00
Thomas Kriechbaumer
a35716a3a7 websocket: test killable 2017-12-28 19:50:31 +01:00
Thomas Kriechbaumer
f123fb600d websocket: docs++ and kill messages 
fixes #2724
 2017-12-27 22:01:28 +01:00
Maximilian Hils
ddb8f43b87 Display "*" when listening on all interfaces 
The set of people who understand "0.0.0.0" (and/or "::") is a subset of
the set of people who understand "*", so we can make things easier to understand
by using the less technical notation.

Empirical Evidence:
 - https://discourse.mitmproxy.org/t/unable-to-use-mitmproxy-on-windows/735
 - https://discourse.mitmproxy.org/t/how-to-view-the-proxy-servers-url/23
 2017-12-26 20:01:37 +01:00
fate0
5661b40942 @concurrent decorator works for class methods. (#2708) 
* @concurrent decorator should works for class methods
* modify test for @concurrent decorator in class.
* add docs
 2017-12-26 18:44:43 +01:00
Aldo Cortesi
7d45d7f15e cuts: don't crash when retrieving header of a non-existent response 2017-12-20 11:53:44 +13:00
Aldo Cortesi
79ca2c8437 commander: command argument underlay 
Display context-sensitive argument types as an "underlay" in commander.
 2017-12-20 10:07:35 +13:00
Aldo Cortesi
d7ee5d8f85 commander: palette entries, highlight errors 
- Add palette entries specific to commander
- Highlight errors
- Introduce an Unknown type to keep track of extra unknown arguments to
commands
 2017-12-20 09:09:40 +13:00
Aldo Cortesi
3341edc9fa commander: highlight commands 
Step 1 of the fancification of commander.
 2017-12-20 08:21:48 +13:00
Aldo Cortesi
d98fc4f03a commands: partial parser now returns help for remaining arguments 2017-12-19 11:50:12 +13:00
Aldo Cortesi
843bad187c types: add validation to partial parser 2017-12-19 11:22:03 +13:00
Aldo Cortesi
6563feaf05 types: use new type validation mechanism in commands 2017-12-19 10:19:08 +13:00
Aldo Cortesi
cda14830d3 types: add validation functions 2017-12-19 10:19:08 +13:00
Aldo Cortesi
38b37ba7f5 types: cleanups and minor refactorings 2017-12-19 10:19:08 +13:00
Thomas Kriechbaumer
9faad6bc9c
Merge pull request #2702 from Kriechi/fix-2640 
improve websocket dumps
 2017-12-18 21:31:00 +01:00
Thomas Kriechbaumer
417a860f32 add tests 2017-12-18 11:20:18 +01:00
Thomas Kriechbaumer
8d836d251e fix #2640 2017-12-18 09:19:21 +01:00
Thomas Kriechbaumer
1a7ce384da websocket: support domain and url filters 2017-12-18 09:19:20 +01:00
Aldo Cortesi
b0b67fe2a7 commands: refactor types 
The type system was scattered over a number of places, making it hard to
follow. This collects all command types in types.py, and completion, validation
and parsing for each type is centralised. We should use the same mechanism for
options.
 2017-12-18 17:02:48 +13:00
Aldo Cortesi
7dbe4209d9
Merge pull request #2690 from cortesi/types 
mitmproxy.types -> mitmproxy.coretypes
 2017-12-18 08:14:07 +13:00
Aldo Cortesi
978b8d095c mitmproxy.types -> mitmproxy.coretypes 
The types name is valuable, and we have a better use for it in collecting and
exposing types for options and commands.

The coretypes module should probably be split up anyway - it contains a
threading base class, a few container objects, and the defintion of our
serialization protocol. I was tempted to rename it to "uncagegorized" for the
sake of honesty.
 2017-12-17 13:31:36 +13:00
Aldo Cortesi
a289db8d75 commands: minor refactoring and command renaming 2017-12-17 11:23:15 +13:00
Aldo Cortesi
ac335057a7 Disable implicit JS test asset generation 
The JS test assets depend in a brittle way on the details of the tflow()
utility functions. We shouldn't have to fix JS tests when adjusting these.

Options:
    - Manually generate the test assets in a script.
    - Define the JS assets without using tflow, so they don't unexpextedly
    vary.
 2017-12-17 10:17:05 +13:00
Aldo Cortesi
2649e57691 minor: fix key bindings, adjust flow command presentation 
Fixes #2681
 2017-12-17 10:11:02 +13:00
Aldo Cortesi
cd913d598d command cuts: add completion 
- Remove shortcuts for request, response, etc. - we don't need them if we have completion
- Restrict cuts specification to a set of prefixes
- Extend cuts to add a few more items
 2017-12-17 10:11:02 +13:00
Aldo Cortesi
50a94db2cc commands: Reassess the cuts API 
Make the cuts API more transparent. Cut specifications are no longer a
centrally resolved core type, and flows are now passed explicitly.
 2017-12-17 10:11:02 +13:00
Aldo Cortesi
198c7b19a3 commander: test++ 2017-12-15 17:51:02 +13:00
Aldo Cortesi
a8ae006f2e command: path completion 2017-12-15 16:02:34 +13:00
Aldo Cortesi
582e6a9fa6 command: recursive command parsing 
This lets us complete commands passed to commands correctly.
 2017-12-15 16:02:34 +13:00
Aldo Cortesi
a436af537a
Merge pull request #2678 from mhils/console-startup-events 
Fix console logging during startup
 2017-12-15 16:02:19 +13:00
Maximilian Hils
3ff380054c improve eventstore truncation 2017-12-15 01:36:41 +01:00
Aldo Cortesi
1c097813c1 commands: emit types from partial parser, implement choice completion 2017-12-15 11:38:37 +13:00
Aldo Cortesi
8c0ba71fd8 commander: tab completion for command names 2017-12-15 10:07:47 +13:00
Aldo Cortesi
4d358c49fb WIP: autocompletion 2017-12-15 10:07:47 +13:00
Aldo Cortesi
0cd4a77268 commands: add a parser for partial commands 
We only return Cmd and str types for the moment.
 2017-12-15 10:07:47 +13:00
Aldo Cortesi
e64d5c6bb9 commands: add a Cmd argument type 
This represents a command passed as an argument. Also split arguments from
command values themselves, making the command help for meta-commands much
clearer.
 2017-12-15 10:07:47 +13:00
Aldo Cortesi
04e19f9171 Introduce a custom widget for command editing 
The builtin urwid.Edit widget is not sufficiently flexible for what we want to
do.
 2017-12-15 10:07:47 +13:00
Aldo Cortesi
dfcf62ff2b
Merge pull request #2545 from mitmproxy/wsproto 
Replace our WebSocket stack with wsproto
 2017-12-15 09:49:24 +13:00