Commit Graph

1914 Commits

Author SHA1 Message Date
Aldo Cortesi
e5cdc20eb8 Split dumper_filter off from view_filter
These don't even do the same thing.
2018-02-26 09:43:15 +13:00
Aldo Cortesi
2fb85553cc Revamp verbosity options
Split verbosity into termlog_verbosity and console_eventlog_verbosity.

This patch also removes printing to console if there are unknown options in the
command-line. Options now live in separate addons, so having uknown options
remaining is common and expected. We definitely shoould have some other way for
users to see what was ignored so they can catch typos and the like, but that's
a different patch.
2018-02-26 09:02:24 +13:00
Aldo Cortesi
154309b811
Merge pull request #2909 from fristonio/Fix-2867
Fix test fails in test_xss_scanner when running tox without internet connection.
2018-02-25 23:00:23 +13:00
luz.paz
afbb7f117b Misc. typos
Found via `codespell -q 3 -I ../mitmproxy-word-whitelist.txt`
Where whitelist contains:
```
cas
doubleclick
nd
ot
seeked
statics
te
thru
```
2018-02-24 21:45:11 -05:00
Aldo Cortesi
6b5b71aefa
Merge pull request #2907 from cortesi/optionscomp
Start moving more complicated options over to /addons
2018-02-25 10:15:21 +13:00
Deepesh Pathak
9fed4fa40a
Fix test fails in test_xss_scanner when running tox without internet connections.
- Add mock implementation for gethostbyname in test_xss_scanner.
- Fix failed tests when running tox without internet connection.
- Fixes #2867
2018-02-24 16:50:40 +05:30
Aldo Cortesi
6ea81a96d4 addon options: proxyauth to /addons 2018-02-24 19:16:13 +13:00
Aldo Cortesi
59db25bd0f Split default_contentview into dumper_* and console_* varieties
Then move the new options into their respective addons. This was a shared
option, and things are much much simpler if it isn't.
2018-02-24 19:09:32 +13:00
Aldo Cortesi
f49aae312f Move intercept options to addon
As part of this, we zap an un-needed console command and use a boolean toggle
setter instead.

Also remove an ancient regression test that breaks encapsulation.
2018-02-24 18:33:15 +13:00
Aldo Cortesi
3a766a420c
Merge branch 'master' into opts 2018-02-24 16:47:58 +13:00
Aldo Cortesi
93c49c47ae flow_detail option to /addons/dumper 2018-02-24 16:42:32 +13:00
Aldo Cortesi
4fe83be63c Resolve some left-over cross-addon dependencies 2018-02-24 16:06:11 +13:00
Aldo Cortesi
f361ea491c addon options: upstream_auth
Also the last "easy" option to shift.
2018-02-24 15:58:37 +13:00
Aldo Cortesi
2aa7ac584b addon options: stickycookie, streambodies 2018-02-24 15:49:38 +13:00
Aldo Cortesi
704c1db1b7 addon options: setheaders, stickyauth 2018-02-24 15:45:12 +13:00
Aldo Cortesi
12633adeb9 addon options: readfile, save, script 2018-02-24 15:28:43 +13:00
Aldo Cortesi
144b559b46 addon options: migrate replace, simplify taddons.context 2018-02-24 15:22:28 +13:00
Aldo Cortesi
52c8d7e0f8 addon options: shift server_replay options 2018-02-24 15:00:40 +13:00
Aldo Cortesi
74a0230788 addon options: shift keepserving and onboarding options 2018-02-24 14:53:54 +13:00
Aldo Cortesi
a12d149230 client options: migrate allowremote and clientplayback
Also triage options, and categorize them into core options (won't be migrated),
options that are hard to migrate for various reasons, and easy migrations.
2018-02-24 14:39:03 +13:00
Aldo Cortesi
42094b29ff Clean up un-necessary arguments to taddons.context
Also test coverage ++
2018-02-24 13:22:15 +13:00
Aldo Cortesi
4522a6f7b7 Start moving addon options into /addons
This takes the first few steps:

- Extends taddons to make loading addon options easier
- Removes dependencies in the test suite on options in addons
- Tweaks command-line parser autocreation to ignore nonexistent options. This
lets us load common options without over-depending on loaded addons.
2018-02-24 12:13:52 +13:00
Aldo Cortesi
16dd7f3ddf
Merge pull request #2886 from tran-tien-dat/set-cookie
Parse Set-Cookie header more permissively. Fix #2829
2018-02-24 10:49:33 +13:00
Aldo Cortesi
c471c42c7c Fix an issue with quoting and commands
Previously, quotes would disappear during editing, making it
impossible to enter command arguments with spaces.
2018-02-23 13:04:31 +13:00
Tran Tien Dat
97e534e76b Fix #2829 2018-02-21 19:58:29 +08:00
Miroslav
34dbd9adbb Flake, be happy 2018-02-20 21:45:14 +02:00
Miroslav
6c525317cc one more cleanup 2018-02-20 20:14:36 +02:00
Miroslav
aea6fd28c8 Hotkeys cleanup 2018-02-20 16:25:39 +02:00
Maximilian Hils
da4c1ee625
Merge pull request #2881 from cortesi/refreshopt
option refresh_server_playback -> server_replay_refresh
2018-02-19 22:52:19 +01:00
Aldo Cortesi
e9bdada1f5 option replay_kill_extra -> server_replay_kill_extra
Continuing effort to collect all sever replay options under a commmon prefix.
2018-02-20 09:46:47 +13:00
Aldo Cortesi
8129b93bb0 option refresh_server_playback -> server_replay_refresh
All other server playback options have this form.
2018-02-20 09:30:29 +13:00
kira0204
fb54bb3777 Testing 2018-02-19 22:45:30 +05:30
kira0204
60cd8cb0b1 Adding test 2018-02-16 22:30:29 +05:30
Maximilian Hils
26cb6b7936
Merge pull request #2847 from kira0204/key-cookie-issue
Handling empty key with value for cookies, fix #2827
2018-02-12 11:22:09 +01:00
kira0204
06f689aa34 keeping consistency, better testing 2018-02-11 17:25:44 +05:30
Maximilian Hils
cda7c8d754
Merge pull request #2851 from mhils/always-use-tls-settings
Use TLS options also for request replay
2018-02-10 12:40:39 +01:00
Maximilian Hils
e4f70b7957 use TLS options also for request replay, refs #2840 2018-02-08 13:17:33 +01:00
Miroslav
bd375c2669 Excess return was deleted 2018-02-08 00:31:57 +02:00
kira0204
4e9d4b37b3 fixing logic 2018-02-07 04:41:12 +05:30
Arushit Mudgal
91834f98cc Extend mypy checking, fix #2194 (#2819) 2018-02-03 21:37:33 +01:00
Samoilenko Roman
7733252627 don't raise when pyperclip doesn't find a clipboard, fix #2816 2018-02-03 13:53:09 +01:00
Maximilian Hils
4decb5c2e3 remove quotes around filename 2018-02-01 09:57:25 +01:00
Samoilenko Roman
4b93e16e78 Fix for #2750 and #2783 (#2809) 2018-01-25 16:07:09 +01:00
Arushit Mudgal
f41d521ce5 fix #2800 2018-01-25 16:06:19 +01:00
Samoilenko Roman
2d07678fe5 Fix #2765 (#2792) 2018-01-14 21:54:43 +01:00
Samoilenko Roman
6b22da6a34 Fix #2768 (#2778) 2018-01-14 21:54:09 +01:00
Maximilian Hils
9a43ff5da1
Merge pull request #2793 from kajojify/issue-2399
Issue 2399
2018-01-14 21:52:32 +01:00
Miroslav
267b1af835 Fix #2399 2018-01-14 13:41:15 +02:00
Maximilian Hils
d9e3fcf5ef store ClientHello extensions with client connection 2018-01-13 01:49:20 +01:00
Maximilian Hils
96a5ed9dff
Merge pull request #2790 from mhils/stateobject-improvements
stateobject: use typing, enable tuples and more complex datatypes
2018-01-13 00:35:49 +01:00
Maximilian Hils
69726f180a stateobject: use typing, enable tuples and more complex datatypes 2018-01-13 00:33:37 +01:00
Maximilian Hils
93d9640ea5 improve net.tls.ClientHello API, fix bugs, add tests 2018-01-13 00:10:47 +01:00
Maximilian Hils
fc80aa562e move ClientHello and related functions to mitmproxy.net.tls 2018-01-12 21:40:35 +01:00
Matthew Shao
137a78f22a Add test for fix#2766. 2018-01-11 09:23:50 +08:00
Matthew Shao
a6f6f8cd32 Raise a CommandError for the exception 2018-01-08 21:51:03 +08:00
Matthew Shao
633bc350d2 Fix #2760 2018-01-08 21:14:35 +08:00
Samoilenko Roman
821d76df02 Fix #2767 (#2770) 2018-01-07 23:05:39 +01:00
Maximilian Hils
15f525e86b only use annotated tags for version calculation 2018-01-07 21:18:42 +01:00
Thomas Kriechbaumer
4fb894cad4 avoid TLS/SSL ambiguity for Cert class 2018-01-07 19:55:04 +01:00
Thomas Kriechbaumer
d15e96dee1 rename TLS/SSL-related functions
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for function names.
2018-01-06 10:43:54 +01:00
Thomas Kriechbaumer
9aae3213b9 rename TLS/SSL-related attributes
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for attributes.
2018-01-06 10:43:47 +01:00
Maximilian Hils
2e2daeed89 refactor common.format_keyvals
the semantics here were really quite unclear,
now it is hopefully a bit more obvious what's happening.
Once we are Python 3.6+ exclusively, we may consider changing
the signature to accept a (order-preserving) dict instead of a list.
2018-01-05 16:45:50 +01:00
Maximilian Hils
c2adcb58f4 fix test flow timestamps to values that don't overflow in certain timezones 2018-01-05 16:45:43 +01:00
Maximilian Hils
dd7902cf7f fix #2470 2018-01-04 16:31:45 +01:00
Maximilian Hils
d38b5e8552 do not add leading zeroes to .dev version part.
wheel normalizes to no leading zeroes, and we don't want to special-case
that.
2018-01-04 15:42:09 +01:00
Maximilian Hils
d9d4d15ee0
Merge pull request #2733 from mitmproxy/pyinstaller
Update PyInstaller, add dev version info to builds.
2017-12-30 22:29:49 +01:00
Maximilian Hils
9dc3d16bf2 single-source version processing 2017-12-30 18:48:47 +01:00
Maximilian Hils
afe6e587f9
Merge pull request #2703 from mhils/test-defaultkeys
Test that default key bindings are valid commands
2017-12-30 17:37:33 +01:00
Maximilian Hils
465044c373 test that defaultkeys are valid commands 2017-12-29 23:20:03 +01:00
Maximilian Hils
6232622774 fix Flow.kill behaviour
This now just sets a kill reply instead of committing directly.
First, this seems like the more sane thing to do.
Second, we have an iffy race condition where we call Reply.commit()
before the addonmanager finishes its invocation, the proxy thread then progresses
and sets a new flow.reply attribute, and the addonmanager then gets confused
when finishing. This commit doesn't fix that, but mitigates it for Flow.kill
which is now committed by the addonmanager.
2017-12-29 22:56:29 +01:00
Thomas Kriechbaumer
59c277effd websocket: add flow kill test 2017-12-29 22:51:13 +01:00
Thomas Kriechbaumer
a35716a3a7 websocket: test killable 2017-12-28 19:50:31 +01:00
Thomas Kriechbaumer
f123fb600d websocket: docs++ and kill messages
fixes #2724
2017-12-27 22:01:28 +01:00
Maximilian Hils
ddb8f43b87 Display "*" when listening on all interfaces
The set of people who understand "0.0.0.0" (and/or "::") is a subset of
the set of people who understand "*", so we can make things easier to understand
by using the less technical notation.

Empirical Evidence:
 - https://discourse.mitmproxy.org/t/unable-to-use-mitmproxy-on-windows/735
 - https://discourse.mitmproxy.org/t/how-to-view-the-proxy-servers-url/23
2017-12-26 20:01:37 +01:00
fate0
5661b40942 @concurrent decorator works for class methods. (#2708)
* @concurrent decorator should works for class methods
* modify test for @concurrent decorator in class.
* add docs
2017-12-26 18:44:43 +01:00
Aldo Cortesi
7d45d7f15e cuts: don't crash when retrieving header of a non-existent response 2017-12-20 11:53:44 +13:00
Aldo Cortesi
79ca2c8437 commander: command argument underlay
Display context-sensitive argument types as an "underlay" in commander.
2017-12-20 10:07:35 +13:00
Aldo Cortesi
d7ee5d8f85 commander: palette entries, highlight errors
- Add palette entries specific to commander
- Highlight errors
- Introduce an Unknown type to keep track of extra unknown arguments to
commands
2017-12-20 09:09:40 +13:00
Aldo Cortesi
3341edc9fa commander: highlight commands
Step 1 of the fancification of commander.
2017-12-20 08:21:48 +13:00
Aldo Cortesi
d98fc4f03a commands: partial parser now returns help for remaining arguments 2017-12-19 11:50:12 +13:00
Aldo Cortesi
843bad187c types: add validation to partial parser 2017-12-19 11:22:03 +13:00
Aldo Cortesi
6563feaf05 types: use new type validation mechanism in commands 2017-12-19 10:19:08 +13:00
Aldo Cortesi
cda14830d3 types: add validation functions 2017-12-19 10:19:08 +13:00
Aldo Cortesi
38b37ba7f5 types: cleanups and minor refactorings 2017-12-19 10:19:08 +13:00
Thomas Kriechbaumer
9faad6bc9c
Merge pull request #2702 from Kriechi/fix-2640
improve websocket dumps
2017-12-18 21:31:00 +01:00
Thomas Kriechbaumer
417a860f32 add tests 2017-12-18 11:20:18 +01:00
Thomas Kriechbaumer
8d836d251e fix #2640 2017-12-18 09:19:21 +01:00
Thomas Kriechbaumer
1a7ce384da websocket: support domain and url filters 2017-12-18 09:19:20 +01:00
Aldo Cortesi
b0b67fe2a7 commands: refactor types
The type system was scattered over a number of places, making it hard to
follow. This collects all command types in types.py, and completion, validation
and parsing for each type is centralised. We should use the same mechanism for
options.
2017-12-18 17:02:48 +13:00
Aldo Cortesi
7dbe4209d9
Merge pull request #2690 from cortesi/types
mitmproxy.types -> mitmproxy.coretypes
2017-12-18 08:14:07 +13:00
Aldo Cortesi
978b8d095c mitmproxy.types -> mitmproxy.coretypes
The types name is valuable, and we have a better use for it in collecting and
exposing types for options and commands.

The coretypes module should probably be split up anyway - it contains a
threading base class, a few container objects, and the defintion of our
serialization protocol. I was tempted to rename it to "uncagegorized" for the
sake of honesty.
2017-12-17 13:31:36 +13:00
Aldo Cortesi
a289db8d75 commands: minor refactoring and command renaming 2017-12-17 11:23:15 +13:00
Aldo Cortesi
ac335057a7 Disable implicit JS test asset generation
The JS test assets depend in a brittle way on the details of the tflow()
utility functions. We shouldn't have to fix JS tests when adjusting these.

Options:
    - Manually generate the test assets in a script.
    - Define the JS assets without using tflow, so they don't unexpextedly
    vary.
2017-12-17 10:17:05 +13:00
Aldo Cortesi
2649e57691 minor: fix key bindings, adjust flow command presentation
Fixes #2681
2017-12-17 10:11:02 +13:00
Aldo Cortesi
cd913d598d command cuts: add completion
- Remove shortcuts for request, response, etc. - we don't need them if we have completion
- Restrict cuts specification to a set of prefixes
- Extend cuts to add a few more items
2017-12-17 10:11:02 +13:00
Aldo Cortesi
50a94db2cc commands: Reassess the cuts API
Make the cuts API more transparent. Cut specifications are no longer a
centrally resolved core type, and flows are now passed explicitly.
2017-12-17 10:11:02 +13:00
Aldo Cortesi
198c7b19a3 commander: test++ 2017-12-15 17:51:02 +13:00
Aldo Cortesi
a8ae006f2e command: path completion 2017-12-15 16:02:34 +13:00
Aldo Cortesi
582e6a9fa6 command: recursive command parsing
This lets us complete commands passed to commands correctly.
2017-12-15 16:02:34 +13:00
Aldo Cortesi
a436af537a
Merge pull request #2678 from mhils/console-startup-events
Fix console logging during startup
2017-12-15 16:02:19 +13:00
Maximilian Hils
3ff380054c improve eventstore truncation 2017-12-15 01:36:41 +01:00
Aldo Cortesi
1c097813c1 commands: emit types from partial parser, implement choice completion 2017-12-15 11:38:37 +13:00
Aldo Cortesi
8c0ba71fd8 commander: tab completion for command names 2017-12-15 10:07:47 +13:00
Aldo Cortesi
4d358c49fb WIP: autocompletion 2017-12-15 10:07:47 +13:00
Aldo Cortesi
0cd4a77268 commands: add a parser for partial commands
We only return Cmd and str types for the moment.
2017-12-15 10:07:47 +13:00
Aldo Cortesi
e64d5c6bb9 commands: add a Cmd argument type
This represents a command passed as an argument. Also split arguments from
command values themselves, making the command help for meta-commands much
clearer.
2017-12-15 10:07:47 +13:00
Aldo Cortesi
04e19f9171 Introduce a custom widget for command editing
The builtin urwid.Edit widget is not sufficiently flexible for what we want to
do.
2017-12-15 10:07:47 +13:00
Aldo Cortesi
dfcf62ff2b
Merge pull request #2545 from mitmproxy/wsproto
Replace our WebSocket stack with wsproto
2017-12-15 09:49:24 +13:00
Maximilian Hils
0af6e2e97f adjust tests 2017-12-14 15:44:47 +01:00
Aldo Cortesi
62561ed428
Merge pull request #2669 from cortesi/path
commands: add a Path argument type
2017-12-14 07:42:12 +13:00
Maximilian Hils
5e0e08a4d6
Merge pull request #2670 from Kriechi/fix-2664
fix #2664
2017-12-13 13:50:43 +01:00
Thomas Kriechbaumer
488ec6f3f1 fix #2664 2017-12-13 09:45:40 +01:00
Aldo Cortesi
79cf6d2a5d
Merge pull request #2668 from cortesi/commandopts
Command improvements
2017-12-13 14:42:56 +13:00
Aldo Cortesi
e63bb8cde5 commands: add a Path argument type
This is just an alias for str, and in this patch is used mostly to give an
appropriate type in help strings. More to come.

Fixes #2198
2017-12-13 14:41:46 +13:00
Aldo Cortesi
4cee1a4f96 commands: formalise a Choice type
This resolves as a string during MyPy checks, but at runtime has an additional
attribute that is a command that returns valid options.

This is very ugly and clumsy, basically because MyPy is super restrictive about
what it accepts as a type. Almost any attempt to construct these types in a
more sophisticated way fails in one way or another. I'm open to suggestions.
2017-12-13 11:08:14 +13:00
Maximilian Hils
1a45cf17b3
Merge pull request #2643 from Ga-ryo/master
Fix #2594
2017-12-12 22:40:47 +01:00
Ujjwal Verma
5214f544e7 Use wsproto for websockets 2017-12-12 22:09:46 +01:00
Thomas Kriechbaumer
130021b76d prepare WebSocket stack to move to wsproto 2017-12-12 22:09:46 +01:00
Maximilian Hils
8e9194c2b4 fix #2529 2017-12-12 16:46:35 +01:00
Maximilian Hils
0fb48bc6a7 minor improvements, add tests for ipv4-mapped addresses 2017-12-12 14:00:12 +01:00
Aldo Cortesi
4912920573
Merge branch 'master' into browser-win 2017-12-12 08:03:09 +13:00
Maximilian Hils
7654ad2d31 fix browser addon on Windows 2017-12-11 10:46:55 +01:00
Aldo Cortesi
b8cbb4dce8 commands: verify command function signatures before call (#2659)
Fixes #2652, and many other possible crashes on user input.
2017-12-11 10:03:07 +01:00
Aldo Cortesi
aeefcd08a4 browser addon: start an isolated browser attached to the proxy
A simple addon that starts an instance of Chrome attached to the current
proxy. The instance is isolated in its own user data directory, and addons are
turned off.

Future work:

- I wasn't able to test the Windows executable path - a Windows dev should
confirm this for us.
- In future it would be nice to support other browsers like Firefox.
2017-12-10 18:26:24 +13:00
Suraj Tripathi
5067438ec1 Bug in expected_http_body_size fix (#2642)
fixes #2618
2017-11-27 19:01:32 +01:00
Maximilian Hils
4a2a718bc2 improve clientreplay addon
- always refresh UI after flow is finished (refs #2616)
 - count currently active replay
 - make replay thread daemonic so that users can exit mitmproxy
   if replay hangs. This is not perfect yet, but vastly better
   than how it has been.
2017-11-07 11:49:12 +01:00
Maximilian Hils
e6e28c2ac3 fix revert of duplicated flows 2017-11-06 15:24:54 +01:00
Maximilian Hils
7314081b82 make safecall threadsafe. 2017-10-26 21:43:59 +02:00
Maximilian Hils
4a6d838ecc fix #2563 2017-10-24 23:27:16 +02:00
Maximilian Hils
a5d74356dd simplify version output 2017-10-24 14:33:56 +02:00
Maximilian Hils
f31d5dc3eb Merge pull request #2591 from ddworken/master
Added scanning for CSS injection and iframe injection to XSS scanner
2017-10-22 18:02:28 +02:00
Maximilian Hils
4522f15c89 skip ipv6 tests on sudo-enabled travis builds 2017-10-20 16:00:25 +02:00
David Dworken
04a06eb6b5 Added scanning for CSS injection and iframe injection 2017-10-17 23:39:33 -04:00
Maximilian Hils
289ddfe988 remove html outline contentview, fix #2572 2017-09-14 15:11:24 -07:00
Maximilian Hils
930d78b554 Merge pull request #2537 from ujjwal96/content-view
JS beautifier
2017-09-04 19:49:42 +02:00
Maximilian Hils
68fab8bd92 Merge pull request #2560 from mhils/mitmproxy-net-tls
Split TLS parts from net.tcp into net.tls
2017-09-04 17:32:49 +02:00
Maximilian Hils
de006ea8ad move hostname validation into mitmproxy.net.tls 2017-09-04 14:02:43 +02:00
Maximilian Hils
0b79304e98 suppress tornado logging in tests 2017-09-03 21:22:21 +02:00
Maximilian Hils
e1877577bc make split_special_areas more flexible, refs #2537
(cherry picked from commit 31ef7f149e4553eb9403634c0eec6de4d0123386)
2017-09-03 21:02:29 +02:00
Ujjwal Verma
8030f5003d Added test 2017-09-03 21:02:29 +02:00
Ujjwal Verma
b05bf588e6 JS beautifier 2017-09-03 21:02:29 +02:00
Maximilian Hils
2b4f58eb44 split TLS parts from net.tcp into net.tls 2017-09-03 03:06:15 +02:00
Matthew Shao
a98dd1642c [web] Update test for static viewer. 2017-08-30 21:15:04 +08:00
Maximilian Hils
e1af76e71f fix strutils.split_special_areas 2017-08-26 17:04:56 +02:00
Maximilian Hils
8eb6558570 add new, faster css prettifier 🎉 2017-08-24 20:39:21 +02:00
Maximilian Hils
72ac89f666 add strutils.escape_special_areas 2017-08-24 18:57:32 +02:00
Maximilian Hils
721f7cc92e Merge pull request #2540 from mhils/relatively-static
Make it possible to run static viewer in subdirectories
2017-08-22 00:59:51 +02:00
Maximilian Hils
bf243244ce [web] make it possible to run static viewer in subdirectories 2017-08-21 23:46:48 +02:00
Maximilian Hils
ca3e98640c fix mitmweb master shutdown, tests++ 2017-08-21 21:39:40 +02:00
Maximilian Hils
fa8bc78ca8 fix tests 2017-08-21 17:35:32 +02:00