Commit Graph

1869 Commits

Author SHA1 Message Date
Maximilian Hils
3262b6e705 Squashed commit of the following:
commit edfbd41200a854f0bb7bb99f8bb70af9dbb9b8e0
Author: Matt Weidner <matt.weidner@gmail.com>
Date:   Tue Jul 25 01:19:53 2017 -0500

    Extended view.load test to check for unhandled IOError exception.

commit a523b534bc59ea97ed1fd5a3e6f78112fee19b6f
Author: requires.io <support@requires.io>
Date:   Mon Jul 24 21:25:04 2017 +0200

    [requires.io] dependency update

commit c725540c6eb92c003616b649ba43bee1f14e56ac
Author: Thomas Kriechbaumer <thomas@kriechbaumer.name>
Date:   Mon Jul 24 21:01:25 2017 +0200

    update travis

commit eeb6cfb4c76e60ac1813b839f589cd489c041c6c
Author: Thomas Kriechbaumer <Kriechi@users.noreply.github.com>
Date:   Mon Jul 24 21:03:14 2017 +0200

    [requires.io] dependency update on master branch (#2435)

commit 51a2672c782ee8ba8c7d5c7116073feccb4d8430
Author: Maximilian Hils <git@maximilianhils.com>
Date:   Mon Jul 24 19:03:01 2017 +0200

    require latest mypy version (refs #2452)

commit 5685a4850af6edda7100cae900487955c8b7a3ab
Author: Maximilian Hils <git@maximilianhils.com>
Date:   Fri Jul 21 11:24:42 2017 +0200

    fix addon tracebacks

    `.tb_next` discards the first interesting frame,
    this shouldn't happen.

commit a2da9b6c02030293f3a412d16df819868c581a29
Author: Matt Weidner <matt.weidner@gmail.com>
Date:   Sat Jul 22 12:30:15 2017 -0500

    Added os.path.expanduser() before open() calls with user supplied paths

commit 05db6e32c7957f267e97c34aa8f5a3cd6cb7dbb2
Author: Matt Weidner <matt.weidner@gmail.com>
Date:   Fri Jul 21 16:25:16 2017 -0500

    Added support for the ~ path shortcut when loading flows from disk.
    for consistency. Saving flows supports using paths with the ~ shortcut.

commit b7f864b6bbd221093aeb1c384dd16038f490441a
Author: Matt Weidner <matt.weidner@gmail.com>
Date:   Fri Jul 21 16:06:38 2017 -0500

    Fixed crash when loading flows from a file that does not exist
2017-07-25 21:55:26 +02:00
Ujjwal Verma
ed0f219167 fixed addon error 2017-07-26 01:02:30 +05:30
Thomas Kriechbaumer
4855659eeb nuke old openssl 2017-07-25 10:56:24 +02:00
Thomas Kriechbaumer
a2c2b1a9cb update cert with too small key
$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
  (keep everything default, do not enter a Common Name or FQDN)
$ cat key.pem cert.pem > test/mitmproxy/data/no_common_name.pem
2017-07-25 10:56:24 +02:00
Matthew Shao
5c45a90ce9 [web] Update tests. 2017-07-24 20:36:21 +08:00
Maximilian Hils
779677bcc6 fix loading scripts with same filename 2017-07-20 15:21:09 +02:00
Thomas Kriechbaumer
74708d546b Update test_allowremote.py 2017-07-15 10:01:14 +02:00
Ujjwal Verma
fdc64be04a Allow remote addon 2017-07-15 10:01:14 +02:00
Thomas Kriechbaumer
3814f171dd Merge pull request #2427 from ujjwal96/protobuf
Kaitai parser for protobuf
2017-07-10 22:58:38 +02:00
Ujjwal Verma
3f269d2b68 Kaitai parser for protobuf 2017-07-10 01:24:43 +05:30
Maximilian Hils
062a58f848 Merge pull request #2423 from MatthewShao/mitmweb-options
[web] [WIP] Mitmweb options editor content
2017-07-05 16:25:38 +02:00
Matthew Shao
37fea267c1 Fix the test for websocket connection. 2017-07-05 18:57:19 +08:00
Matthew Shao
aa01a62df7 [web] Add keys argument for dump_dict in optmanager. 2017-07-05 08:40:21 +08:00
Ujjwal Verma
7e04ac0578 add websocket streaming option 2017-07-04 10:53:23 +02:00
Ujjwal Verma
9e1902be62 fix HTTP retry if sending a request fails once 2017-07-04 10:52:50 +02:00
Ujjwal Verma
389d37ebff websocket streaming 2017-07-04 10:52:50 +02:00
Ujjwal Verma
d4f35d7a4a request streaming for HTTP/2 2017-07-04 10:52:50 +02:00
Ujjwal Verma
47c9604aed request streaming for HTTP/1 2017-07-04 10:52:50 +02:00
Ujjwal Verma
7dfa557441 Fixes #2315 2017-06-28 19:18:57 +05:30
Maximilian Hils
d58abc9200 Merge pull request #2402 from ujjwal96/request-replay
Fixes #2197
2017-06-27 11:18:12 +02:00
Ujjwal Verma
08735ab0ae request replay test 2017-06-26 00:53:14 +05:30
Maximilian Hils
13dac9f212 Merge pull request #2411 from ujjwal96/ico-parser
ICO content view
2017-06-25 02:02:01 +02:00
Ujjwal Verma
8ca29679df Added test 2017-06-25 01:05:20 +05:30
Maximilian Hils
bde64746a9 Merge pull request #2395 from MatthewShao/mitmweb-options
[WIP] Add RESTful API for mitmweb option
2017-06-24 19:54:04 +02:00
Matthew Shao
1863326220 Minor Update for /options API of mitmweb. 2017-06-24 10:18:00 +08:00
Matthew Shao
c7ce7f84e6 Add test for POST /options API of mitmweb. 2017-06-23 10:58:59 +08:00
Matthew Shao
9687d67601 Add test for typespec_to_str. 2017-06-14 07:44:00 +08:00
Aldo Cortesi
788f0f5784 console: console.key.bind console.key.unbind commands 2017-06-14 08:34:34 +12:00
Matthew Shao
2c0f6c2023 Minor Update and add test. 2017-06-13 23:03:07 +08:00
Aldo Cortesi
4a7cafee9e console: teach keymap to understand "space"
Urwid uses " ", which is not a great user experience.
2017-06-13 15:36:00 +12:00
Aldo Cortesi
ba49b55684 console: key binding viewer
Read-only for now.
2017-06-13 11:48:08 +12:00
Aldo Cortesi
88832f92a3 view: show a message if we remove more than one flow from view 2017-06-13 11:02:32 +12:00
Aldo Cortesi
56eb0441da commands: teach parser correct annotations for variable args
We should annotate with the base type, not the resulting sequence.
2017-06-13 10:26:03 +12:00
Aldo Cortesi
d95f28e6bf config: don't crash with a config file containing only comments 2017-06-13 09:43:12 +12:00
Ujjwal Verma
6ca45856b4 Test for streaming 2017-06-12 23:46:43 +05:30
Matthew Shao
03bb0a09b9 Add tests for dump_dicts in optmanager.py. 2017-06-12 23:23:09 +08:00
Ujjwal Verma
4587b54aed tests for Proxy config 2017-06-12 01:32:58 +05:30
Aldo Cortesi
7caa7e7538 console: auto-generate context key binding help 2017-06-11 17:09:52 +12:00
Maximilian Hils
9b59937e12 always pretend to have a tty in tests 2017-06-05 00:19:52 +02:00
Thomas Kriechbaumer
020e5a5c95 Merge pull request #2367 from mhils/issue-2329
Fix #2329
2017-06-02 14:18:56 +02:00
Ujjwal Verma
619566fc9f Fixes #2230 (#2361) 2017-06-02 13:31:12 +02:00
Maximilian Hils
5e17b5f62e minor improvements 2017-06-02 00:01:37 +02:00
Maximilian Hils
2ba0791810 fix #2329 2017-06-01 23:39:49 +02:00
Ujjwal Verma
33f735ef50 Increase test coverage 2017-05-31 04:05:10 +05:30
Thomas Kriechbaumer
ef1c36194e coverage++ 2017-05-26 22:58:01 +02:00
Thomas Kriechbaumer
f5eaddb12c Merge pull request #2346 from ujjwal96/test-coverage
Tests for wbxml
2017-05-26 18:38:24 +02:00
Thomas Kriechbaumer
47b6d1823e remove old test file 2017-05-26 17:04:04 +02:00
Thomas Kriechbaumer
ee6ea31147 Merge pull request #2353 from ujjwal96/tls-parser
Using kaitai struct to parse tls client hello
2017-05-26 13:58:29 +02:00
Ujjwal Verma
6637630e3e Using kaitai struct to parse tls client hello 2017-05-26 00:41:18 +05:30
Thomas Kriechbaumer
cfed4432a0 pathod: fix leaking fds 2017-05-24 19:32:20 +02:00
Ujjwal Verma
bd8a869118 Tests for wbxml 2017-05-24 21:37:49 +05:30
Thomas Kriechbaumer
673ed5b45e pathod: properly verify certificate in tests 2017-05-24 17:40:44 +02:00
Thomas Kriechbaumer
6214c6e185 fix openssl cipher_list bytes/str deprecation 2017-05-24 16:17:04 +02:00
Thomas Kriechbaumer
e77f07a36b cleanup 2017-05-24 14:23:36 +02:00
Thomas Kriechbaumer
ae7e9efb5c fix various fd/socket leaks 2017-05-24 14:09:41 +02:00
Thomas Kriechbaumer
2faaa0b2a2 connections tests: fix leaking sockets 2017-05-24 13:21:58 +02:00
Thomas Kriechbaumer
928085c597 proxy tests: fix leaking sockets 2017-05-24 12:52:13 +02:00
Thomas Kriechbaumer
c9529ffe45 http1 tests: fix leaking sockets 2017-05-24 11:43:57 +02:00
Thomas Kriechbaumer
618044c637 http2 tests: fix leaking sockets 2017-05-24 11:43:50 +02:00
Thomas Kriechbaumer
c4f028d701 websocket tests: fix leaking sockets 2017-05-24 11:35:53 +02:00
Thomas Kriechbaumer
0bd3d0ff4c upgrade pytest and remove our pytest.raises wrapper 2017-05-24 11:35:53 +02:00
Thomas Kriechbaumer
4961d782ed fix typo 2017-05-24 11:35:53 +02:00
Thomas Kriechbaumer
b4f65049f2 auto-close file after reading 2017-05-24 11:35:53 +02:00
Maximilian Hils
5626c384ab Merge pull request #2342 from MatthewShao/jest-dev
[web] Add  _flow.js for component tests.
2017-05-22 10:00:35 +02:00
Matthew Shao
2935ff5f3a [web] Set some value as constant, so _tflow.js would not change. 2017-05-22 08:53:59 +08:00
Matthew Shao
2c38fddea5 [web] TFlow minor improved. 2017-05-21 20:59:19 +08:00
Matthew Shao
369e1f1eee [web] Fix minior lint problem. 2017-05-19 09:56:52 +08:00
Matthew Shao
50c0703424 [web] Add _tflow.js generator in test_app.py 2017-05-19 09:45:30 +08:00
Ujjwal Verma
5833b218b2 Increase net.tcp.py coverage (#2336) 2017-05-17 23:27:57 +08:00
Ujjwal Verma
37053f0efc Increased mitmproxy.controller coverage 2017-05-17 11:06:40 +05:30
Ujjwal Verma
204d06f93f Increased xml_html contentview coverage 2017-05-17 11:05:49 +05:30
Ujjwal Verma
8eea05fcaf Increase contentviews api coverage 2017-05-17 11:04:43 +05:30
Ujjwal Verma
d47de94303 Increase io.compat coverage 2017-05-12 21:09:50 +05:30
Ujjwal Verma
766c5caea8 Increased net.http.cookies.py and net.http.message.py coverage 2017-05-12 21:09:50 +05:30
charlesdhdt
9f8e83259e Fixed LDAP Auth (#2333) 2017-05-12 15:37:00 +02:00
Ujjwal Verma
3e85dfd242 Handle head request on onboarding app: Fix #2324 (#2325) 2017-05-09 20:08:42 +02:00
Maximilian Hils
d734f6bbd6 Merge pull request #2286 from charlesdhdt/master
Added LDAP Auth
2017-05-04 16:24:57 +02:00
Charles d'Hondt
154e8ac0fc fixed lint 2017-05-04 13:39:48 +02:00
Charles d'Hondt
f67d9adc31 Added ldap mock test 2017-05-04 13:25:15 +02:00
Maximilian Hils
315daa042a fix #2310 2017-05-03 17:38:17 +02:00
Aldo Cortesi
2659b52209 console: add a two-pane layout
- Replace options.console_eventlog with options.console_layout
- This can be "single", "vertical" and "horizontal"
- At the base of the primary pane is the flowlist. At the base of the secondary
pane is the event log.
- Any of the other primary windows can be opened in each of the panes.

For now, I've bound "-" to the flow layout switch, "shift tab" to the layout
pane switch, and "P" to open the currently focused flow in whichever pane
you're in. These are just temporary  - we'll reassess the default bindings
carefully once the keybindings work is complete.
2017-05-03 14:55:02 +12:00
Aldo Cortesi
dcae79e017 console: various bugs
- Action bar now correctly gets the palette background
- Viewing connection details for a killed flow no longer crashes
- Editing options with selections no longer crashes
2017-05-02 12:56:27 +12:00
Aldo Cortesi
4e39d387d3 commands: options control
Use the new commands to bind the Options view in console.
2017-05-02 12:18:24 +12:00
Aldo Cortesi
9e58c36639 console: globally respected nav commands
Use this to bind nav keys with commands throughout.
2017-05-02 12:18:24 +12:00
Aldo Cortesi
2f52d8b9ad commands: view.setval, view.getval, view.setval.toggle
Use these to bind the last of the key for flowview. Flow views are now 100%
keybound.
2017-05-01 19:52:24 +12:00
Aldo Cortesi
7d56ec9d49 commands: flow.encode, flow.decode, flow.encode.toggle
Use this to bind "z" in flowview.
2017-05-01 18:31:22 +12:00
Aldo Cortesi
1ea4a5a48e keymap: keys can now bind to multiple contexts
Use this to map the majority of the keys in flowview.
2017-05-01 16:52:58 +12:00
Aldo Cortesi
670d1e408b command: flow.set
Use this to replace the flow edit components of flowview entirely.
2017-05-01 14:56:57 +12:00
Aldo Cortesi
685487d33c commands: view.order.options, view.marked.toggle, view.create
And use these commands to remove the last hard-coded keybindings from flow
list. This means the flow list is now 100% command-driven, which is very
exciting.
2017-04-30 22:45:31 +12:00
Aldo Cortesi
a570caccbd commands: view.load
Plus replace the flow list keybinding.
2017-04-30 22:02:29 +12:00
Aldo Cortesi
3cd93567f5 commands: support *args for commands
Use this to simplify meta-commands in console, and to create a console_choose
command that prompts the user for a choice, and then executes a command with
variable substitution.
2017-04-30 21:24:00 +12:00
Aldo Cortesi
bcbe87bb09 Move export to addon, kill Python and Locust export
Also add a "raw" export format.

The Python and Locust exports are hard to maintain, their tests are extremely
brittle, they didn't have full test coverage, and are by my guess very rarely
used. I feel the Locust export should certainly be an externally maintained
addon. The Python/requests export can come back if someone cares enough, and it
can be structured in a way we can maintain.
2017-04-30 17:53:30 +12:00
Aldo Cortesi
befbe88d96 commands: cut.clip copies cuts to system clipboard 2017-04-30 14:42:37 +12:00
Aldo Cortesi
ed62bbad1d Add help text to --commands output 2017-04-30 14:20:21 +12:00
Aldo Cortesi
075d452a6d cut: more flexible cut specification based on attribute paths
Also support certificate types, which are converted to ASCII-encoded PEM format.
2017-04-30 14:05:45 +12:00
Aldo Cortesi
7ffb2c7981 cut: use csv module to encode multi-values for saving
Also add q.text, q.raw_content, s.text, s.raw_content selectors
2017-04-30 11:48:32 +12:00
Aldo Cortesi
4b568f99d6 Introduce cuts: a flow dissector
This PR introduces the cuts addon, a flow dissector that  allows you to select
and operate on specific components of flows. It also adds the first consumer
for cuts - the cuts.save command.

Save the content of the focus to /tmp/foo:

    cuts.save s.content|@focus /tmp/foo

Save the URL and response content-type headers for all flows currently shown to
file, comma-separated, one flow per line:

    cuts.save s.url,q.header[content-type]|@focus /tmp/foo

We also use this to replace the body save shortcut in the console flowlist.
2017-04-29 23:32:31 +12:00
Aldo Cortesi
d439b34511 command: script.run
Plus the flowlist binding adjustments.
2017-04-29 13:24:19 +12:00
Aldo Cortesi
1d5eedcc9c command: flow.revert
Plus matching binding in flowlist.
2017-04-29 11:48:25 +12:00
Aldo Cortesi
f908ea220f commands: "replay.client.file" and "replay.server.file" 2017-04-29 11:34:50 +12:00
Aldo Cortesi
a92017a6c1 Rework client and server replay
- Add client.replay [flows], client.replay.stop
- Add server.replay [flows], server.replay.stop
- The corresponding options for file loading are only read on startup, further
changes are ignored. In interactive contexts, replay is started with the
commands, not through option changes.
- Deprecate flow.replay, use replay.client instead
2017-04-29 11:02:36 +12:00
Aldo Cortesi
7317ea134e command: flow.kill, flow.replay
Plus the matching bindings in the flow list.
2017-04-29 09:58:32 +12:00
Aldo Cortesi
f21a970f29 commands: marking
Add "view.mark [flows] bool" and "view.mark.toggle [flows]". Use this to rebind
marking keys in flowlist.
2017-04-29 09:14:44 +12:00
Aldo Cortesi
217addbf31 commands: view.go
bind G to "view.go -1"
bind g to "view.go 0"
2017-04-29 08:40:38 +12:00
Aldo Cortesi
0b090f7ae1 Commands, core update event
This patch:

- Introduces a core update() event that should be invoked whenever flows are
changed outside of the normal lifecycle.
- Extend view.resolve to know about @all, which matches all flows in the view.
- Add a core flow.resume comand, which resumes flows and broadcasts an update event.
- Define flow list bindings for:

A -> flow.resume @all
a -> flow.resume @focus
d -> view.remove @focus
z -> view.remove @all
2017-04-29 08:01:00 +12:00
Charles d'Hondt
65202f5f1c Added ldapsss vef 2017-04-28 17:00:21 +02:00
Charles d'Hondt
6d3837fd54 fix 2017-04-28 16:44:50 +02:00
Charles d'Hondt
29c1f303d6 Fixed typo 2017-04-28 16:23:32 +02:00
Charles d'Hondt
5fb18ad275 Added LDAP Auth 2017-04-28 16:19:27 +02:00
Aldo Cortesi
6af1a49464 commands: add a command.command decorator
Use this for our built-ins and the console commands.
2017-04-28 15:07:52 +12:00
Aldo Cortesi
7ff84673fd Support unary flow command arguments
Use this to add a console.view.flow command and bind "enter" in flowlist.
2017-04-28 14:05:35 +12:00
Aldo Cortesi
cfae95f5c3 console: start migrating hotkeys to keymap
This shifts a set of flow list keys to keymaps. Much more to come.
2017-04-28 12:10:48 +12:00
Aldo Cortesi
ce01cb9c09 command: partials
Support partial commands in console.command, use it to keybind "i" for
intercept, which prompts the user with ":set intercept=".
2017-04-28 11:24:28 +12:00
Aldo Cortesi
8a07059cf4 commands: add the core command addon, and the command "set"
The set command sets an option using the same syntax as commandline --set.
2017-04-28 10:41:44 +12:00
Aldo Cortesi
be1b76b975 console: add a keymap
This sketches out a keymap system for consone, and adds the first few top-level
commands and mappings.
2017-04-28 10:21:15 +12:00
Aldo Cortesi
18edc11145 console: interactive command browser
Simple browser that lets users view and select commands interactively.
Key binding for this is still to be sorted out.
2017-04-28 07:43:58 +12:00
Aldo Cortesi
fde1159ae3 Test coverage ++ 2017-04-27 18:43:44 +12:00
Aldo Cortesi
97000aa85c command: save.file flowspec path -> None
Our first user-facing command. The following commands do the obvious things:

save.file @marked /tmp/flows
save.file @focus /tmp/flows
save.file @hidden /tmp/flows
save.file "~m get" /tmp/flows
2017-04-27 17:05:00 +12:00
Aldo Cortesi
b7afcb5dc2 addons.streamfile -> addons.save
Options:

    streamfile -> save_stream_file
    streamfile_filter -> save_stream_filter
2017-04-27 15:58:54 +12:00
Aldo Cortesi
8c4810f606 console: flow resolution command
This is our first built-in command, which will be used by very many other
commands.

Also add a --commands option to dump all commands, analogous to --options.
2017-04-27 15:27:51 +12:00
Aldo Cortesi
169baabcab Basic outline of the command subsystem
- Add a command manager
- Sketch out the type system with a few simple supported types
2017-04-27 11:09:40 +12:00
Aldo Cortesi
2a46f3851a Merge pull request #2265 from cortesi/addons
Addons and addon testing
2017-04-27 07:40:14 +12:00
Maximilian Hils
0f9081b18d Merge pull request #2269 from mhils/issue-2250
fix #2250, add type info to cookie module
2017-04-26 14:42:48 +02:00
Maximilian Hils
87610cc8b2 fix #2250, add type info to cookie module 2017-04-26 14:17:14 +02:00
Maximilian Hils
d5ea08db62 Merge pull request #2258 from mhils/readfile
Integrate readstdin into readfile
2017-04-26 13:40:51 +02:00
Maximilian Hils
0a8e54edea Merge pull request #2268 from mhils/issue-2257
fix #2257
2017-04-26 13:39:19 +02:00
Maximilian Hils
1aa6d9d324 fix #2257 2017-04-26 12:26:49 +02:00
Maximilian Hils
ca2827886a separate reading from stdin into its own addon 2017-04-26 12:11:37 +02:00
Maximilian Hils
df7701bb6d fix #2228 2017-04-26 11:32:24 +02:00
Maximilian Hils
b3a1143338 integrate readstdin into readfile addon 2017-04-26 10:56:32 +02:00
Maximilian Hils
2b500f234f typecheck: add support for typing.Any 2017-04-26 10:54:36 +02:00
Aldo Cortesi
5327756377 Addons and addon testing
- Fix some loading sequence bugs affecting command-line script invocation
- Allow addons to over-ride existing options (with a warning). We need this for
reloading.
- Convert har_dump to new-style arguments, fix and re-instate its test suite.
- Covnert miscelaneous other exmples to new-style args.
2017-04-26 19:56:33 +12:00
Aldo Cortesi
e32efcae49 Merge pull request #2262 from Kriechi/reduced-coverage++
improve tests (extracted from #2011)
2017-04-26 19:52:33 +12:00
Aldo Cortesi
b72f139093 configure(options, updated) -> configure(updated)
Options are now available globally on ctx, so the first argument of configure
is redundant.
2017-04-26 11:01:27 +12:00
Aldo Cortesi
f90b4c2ff0 Move options into ctx
Many addons currently save options on configure(), either as individual options
or sometimes by saving the entire options object. The current options should
simply be available on the ctx object, simplifying state management for addons
considerably.
2017-04-26 10:25:56 +12:00
Aldo Cortesi
7aa2081894 Remove watchdog, solidify script testing
- Remove the watchdog dependency. We now just stat the script file every 2
seconds to check for an updated mtime.
- Further solidify our script testing, and in particular make the example tests
nicer. These should exemplify how we want users to test their own addon
scripts. More work on addon testing to follow.
2017-04-26 09:01:40 +12:00
Thomas Kriechbaumer
a35a377cbb improve tests 2017-04-25 20:49:49 +02:00
Aldo Cortesi
e6eeab6094 Revamp how addons work
- Addons now nest, which means that addons can manage addons. This has a number
of salutary effects - the scripts addon no longer has to poke into the global
addons list, we no longer have to replace/remove/boot-outof parent addons when
we load scripts, and this paves the way for making our top-level tools into
addons themselves.
- All addon calls are now wrapped in a safe execution environment where
exceptions are caught, and output to stdout/stderr are intercepted and turned
into logs.
- We no longer support script arguments in sys.argv - creating an option
properly is the only way to pass arguments. This means that all scripts are
always directly controllable from interctive tooling, and that arguments are
type-checked.

For now, I've disabled testing of the har dump example - it needs to be moved
to the new argument handling, and become a class addon. I'll address that in a
separate patch.
2017-04-25 22:13:44 +12:00
Ujjwal Verma
742127ef7b Fixes #2109 2017-04-09 18:55:20 +05:30
Robert C Jensen
7365f18542 fixes ipv6 authority form parsing in CONNECT 2017-04-07 15:46:21 -04:00
Nikhil Soni
442999c75c Add tests for set_focus 2017-03-29 18:34:17 +05:30
Aldo Cortesi
65f0885bd6 addon loader: add boot_into, which replaces returning from start()
While we're here, expand test coverage for addonmanager to 100%, and promote to
individual coverage.
2017-03-25 10:48:12 +13:00
Aldo Cortesi
541c1e8b9f addons: start -> load throughout 2017-03-25 10:48:12 +13:00
Thomas Kriechbaumer
ef9f0e22ea bump h2 2017-03-24 21:27:14 +01:00
Aldo Cortesi
335861f490 Merge pull request #2207 from nikofil/scripts-redirect-stdout
scripts: redirect stdout to ctx.log.warn
2017-03-24 14:58:33 +13:00
Nikos Filippakis
abf291b0f9 scripts: redirect stdout to ctx.log.warn
Redirect messages written to stdout in scripts to ctx.log.warn.
(closes #1530)

Signed-off-by: Nikos Filippakis <nikolaos.filippakis@cern.ch>
2017-03-24 02:35:02 +01:00
Aldo Cortesi
0d3e2c6adf optmanager: make serialization and loading more robust
Also handle errors in console options manager.
2017-03-24 09:37:17 +13:00
Thomas Kriechbaumer
907825714b move examples tests (#2199) 2017-03-22 12:02:18 +01:00
Aldo Cortesi
4ca78604af Factor out an io module
Include tnetstring - we've made enough changes that this no longer belongs in
contrib.
2017-03-20 12:50:09 +13:00
Aldo Cortesi
04f1a4443d options: reraise when there's an error with a partial update
Fixes #2174
2017-03-19 14:36:23 +13:00
Aldo Cortesi
43e7b9d68c options: choices for content views and tighten console order spec
Also clean up some leftovers from filter option name change.
2017-03-19 13:39:28 +13:00
Aldo Cortesi
22d7c31ea8 dumper, mitmdump: kill filtstr
Dumper now also uses view_filter. Passing arguments to mitmdump is now
precisely equivalent to setting both filestream_filter and view_filter.
2017-03-19 13:21:46 +13:00
Aldo Cortesi
475a4e3eb0 streamfile: add streamfile_filter and use it instead of filtstr 2017-03-19 13:08:26 +13:00
Aldo Cortesi
21794c7bbe optmanager: test coverage 2017-03-19 11:29:41 +13:00
Aldo Cortesi
49b0a67eb9 console options: add an overlay grid editor for sequence options 2017-03-19 10:32:22 +13:00
Aldo Cortesi
4e24c95a61 optmanager: cope with bound methods in .subscribe
Fixes #2122
2017-03-19 10:32:22 +13:00
Aldo Cortesi
8130b9880a options cleanups
- Move default client ciphers - no need to have a 10-line string default.
- Add some style guidance on options help
- Be consistent about reflowing options help
2017-03-19 10:32:22 +13:00
Aldo Cortesi
eac210829e Rip out master handler mechanism
All events are now handled by addons, and we no longer support any events on
master.
2017-03-17 07:56:32 +13:00
Maximilian Hils
6d66184eba Merge pull request #2168 from cortesi/handlers
Rip out old handlers mechanism - all events are now handled in addons
2017-03-16 14:57:26 +01:00
Felix Yan
b352557092 Fix test_format_xml with dot in path
When the path contains dot ".", replacing all dots will generate a non-exist result and raises a FileNotFoundError. Replacing only the last dot fixes this.
2017-03-16 16:48:21 +08:00
Aldo Cortesi
c5e0dc64b9 Rip out master handler mechanism
All events are now handled by addons, and we no longer support any events on
master.
2017-03-16 18:33:24 +13:00
Aldo Cortesi
3de9829003 Remove last handlers in test suite
While we're here, make checking that there are NO logs on the proxy more
efficient by setting a short timeout. This reduces total test suite running
time by 15% on my system.
2017-03-16 18:05:57 +13:00
Aldo Cortesi
1410cbb4b6 Remove test handlers by using taddons.RecordingMaster
This also means expanding and tweaking the recording master API, which we
reflect through the current test suite
2017-03-16 18:05:57 +13:00
Aldo Cortesi
b6abe96202 Move dump handlers to addon 2017-03-16 18:05:57 +13:00
Aldo Cortesi
1e6c51c698 Merge pull request #2152 from cortesi/readfile
Extract flow reading into addons
2017-03-16 15:51:21 +13:00
Aldo Cortesi
228a22b3c0 Add a light-weight custom event system, use it for keepserving
This patch implements the lightweight event system I propose in #2144, adds a
custom event "processing_complete" that is triggered after file read, client
replay and server replay, and introduces a KeepServing addon to handle this for
mitmdump.
2017-03-16 11:07:00 +13:00
Aldo Cortesi
169068c7ec Clean up addonmanager interface
Clarify the plethora of invocation methods we've sprouted, correct some usages
in the codebase.
2017-03-16 08:50:29 +13:00
Maximilian Hils
65c4a3cf1d add Request.make 2017-03-15 14:06:00 +01:00
Aldo Cortesi
ef582333ff Extract flow reading into addons
This patch moves the final pieces of master functionality into addons.

- Add a ReadFile addon to read from file
- Add a separate ReadStdin addon to read from stdin, only used by mitmdump
- Remove all methods that know about io and serialization from master.Master
2017-03-15 13:44:48 +13:00
Maximilian Hils
30797755fb stateobject: automatically change id when copying 2017-03-14 17:08:40 +01:00
Maximilian Hils
375680a3be add connection ids 2017-03-14 17:08:40 +01:00
Maximilian Hils
2d016775fe Merge pull request #2149 from mhils/multidictview-copy
Add MultiDictView.copy()
2017-03-14 14:38:38 +01:00
Maximilian Hils
0de8f46c6e Merge pull request #2145 from mhils/ne
py3: __ne__ delegates to __eq__ by default
2017-03-14 14:38:00 +01:00
Maximilian Hils
961747c4bc Merge pull request #2120 from ujjwal96/replace-from-file
Replace from file through replacement editor #1458
2017-03-14 02:23:31 +01:00
Maximilian Hils
de1de3f626 add MultiDictView.copy 2017-03-14 02:18:02 +01:00
Maximilian Hils
875ce8c9c1 py3: __ne__ delegates to __eq__ by default 2017-03-14 00:40:15 +01:00
Aldo Cortesi
b745428b5c Enable custom options in config files
We also now ignore unknown options in config files by default, and print a
warning if verbosity is incremented.
2017-03-14 09:40:33 +13:00
Aldo Cortesi
2832e790fd Move serialization methods out of Options
Attributes on options share a namespace with options themselves. It's getting
too crowded on our Options object, so let's shift some obvious stuff into the
module.
2017-03-14 08:34:25 +13:00
Aldo Cortesi
c24f7d8e12 Optmanager: handle unknown options in value sets 2017-03-14 08:32:19 +13:00
Aldo Cortesi
75f83ba380 Show custom addon options in --options output 2017-03-14 08:32:19 +13:00
Aldo Cortesi
0c6663d0d5 Enable custom options for addons
- Add an options parameter to the start() event. This is to be used by addons
on startup to add custom options.
- Add a running() event that is called once the proxy is up and running.
- With the new paradigm we can't log during master __init__, so add a tiny
termstatus addon to print proxy status to terminal once we're running.
2017-03-14 08:32:19 +13:00
Maximilian Hils
05e11547f5 Merge remote-tracking branch 'origin/master' into pr-2120
Conflicts:
	test/mitmproxy/addons/test_replace.py
2017-03-12 23:33:49 +01:00
Maximilian Hils
0f4b523868 replacements: error right away when file does not exist 2017-03-12 23:28:49 +01:00
Thomas Kriechbaumer
1b045d24bc nuke tutils.tmpdir, use pytest tmpdir 2017-03-12 22:55:22 +01:00
Maximilian Hils
4684617d2c minor fixes 2017-03-12 20:24:50 +01:00
Maximilian Hils
39154e628e Merge pull request #2118 from lymanZerga11/patch-3
Update proxyauth.py to add proxyauth metadata
2017-03-11 03:04:38 +01:00
Thomas Kriechbaumer
e9746c5182 Merge pull request #2114 from mitmproxy/fix-websocket-serialization
make websocket flows serializable
2017-03-10 21:15:46 +01:00
Thomas Kriechbaumer
45bf1ff64d disable h2c prior knowledge connections 2017-03-10 11:37:25 +01:00
Thomas Kriechbaumer
b1dd86d7ae make websocket flows serializable
fixes #2113
2017-03-09 19:08:59 +01:00
Ujjwal Verma
25703f8c53 Fixed Replacement test 2017-03-09 23:33:11 +05:30
lymanZerga11
8e67c6f512 add tests 2017-03-09 09:01:37 +08:00
lymanZerga11
6d53d08b6c Add tests 2017-03-09 09:01:04 +08:00
Maximilian Hils
63179d9751 core -> core_option_validation
longer, but much clearer for devs who are unfamiliar with the codebase.
2017-03-08 15:17:07 +01:00
Maximilian Hils
f276c7a80d change add_option parameter order
name, type, value matches natural language ($x is a bool with a default value of $y)
and also matches the python type annotation order of name: type = value
2017-03-08 00:52:37 +01:00
Maximilian Hils
8707928b16 unify server spec parsing 2017-03-08 00:18:34 +01:00
Aldo Cortesi
98ec3b77fe Carefully re-add a small number of short flags
The mechanism for booleans attaches the short flag to whatever the opposite of
the default is.
2017-03-07 21:01:21 +13:00
Aldo Cortesi
aed780bf48 Change the way proxy authetication is specified
We now have one option "proxyauth". If this is "any", we accept any
credentials, if it starts with an @ it's treated as a path to an htpasswd file,
if it is of the form username:password it's a single-user credential.
2017-03-07 20:18:30 +13:00
Aldo Cortesi
320d8848ab Options tweaks
- Regularise presentation and wording
- Help is mandatory for all options
- Auto-generate wording to say that sequence options can be passed multiple
times on the command-line
2017-03-07 15:23:46 +13:00
Aldo Cortesi
10db254791 Remove less commonly used command-line options
We now have --set, so only options that really deserve it get a dedicated flag.
I'm inclined to strip this back even more. Feel free to argue that YOUR
favourite option deserves special treatment here.
2017-03-07 14:46:28 +13:00
Aldo Cortesi
ac3b0d69cc Add the --set option to set options directly
The --set option is a universal flag for setting options. Some examples:

Turn on a boolean:

mitmdump --set onboarding=false

Add a value to a sequence:

mitumdupm --set setheaders=/foo/bar/voing

Zero a sequence:

mitumdupm --set setheaders
2017-03-07 14:27:50 +13:00
Aldo Cortesi
79f5883c2f Option spacing and coverage 2017-03-07 13:47:39 +13:00
Aldo Cortesi
99a6b0dbc1 Add --options that dumps annotated option defaults 2017-03-07 13:16:28 +13:00
Aldo Cortesi
b51df9a0b1 Cleanups and test coverage 2017-03-07 10:28:15 +13:00
Aldo Cortesi
2312cf6fb0 Automate slurping up command-line options
Now that options are completely regular, we can automate reading them out of
arguments, rather than listing them out by hand like savages.
2017-03-07 08:49:19 +13:00
Aldo Cortesi
edfd62e42a Replacements and setheaders are always strings
Instead of having two representations we have one canonical specification.
Fixing the editor in console is left ot a further patch.
2017-03-06 20:58:51 +13:00
Aldo Cortesi
82163a1e68 Unify mode specification
We now have:

    --mode regular (the default)
    --mode transparent
    --mode socks5
    --mode reverse:SPEC
    --mode upstream:SPEC

Where SPEC is a host specification.
2017-03-06 20:30:49 +13:00
Aldo Cortesi
9b1f40da37 Options unification: streamfile
We now have one option to control this. If the path is prefixed with a "+" we
append, otherwise we overwrite.
2017-03-06 13:48:37 +13:00
Aldo Cortesi
71a830c836 Options unification: string choices 2017-03-06 13:20:17 +13:00
Aldo Cortesi
18a6b66ba4 Options unification: simple strings
Move all simple string options to the new scheme. Also regularise some names.
2017-03-06 11:39:19 +13:00
Aldo Cortesi
201c65960e Options unification: sizes
Start dealing with corner cases:

- Sizes are always stored in options as strings
- Add a new core addon that's responsible for verifying settings that don't
belong to an addon
- Add a _processed scratch space on the Options object for processed core
values to be stored in. This is pretty dirty, but less dirty than re-parsing
values every time. We'll come up with something better down the track.
2017-03-06 09:07:16 +13:00
Aldo Cortesi
f15a628561 Start unifying options and the command-line: booleans
This commit:

- Adds a help field to options
- Adds a function to generate parser definitions from options
- Uses this to migrate all boolean flags over to the new system
- Makes all booleans consistently follow the --foo/--not-foo convention

There are a number of things left to be done here:

- Argparse doesn't give us a nice way to format --foo --not-foo help. Click
does, and moving to click is a goal down the track.
- For now, we remove all short aliases. I want to re-evaluate these
systematically once we have the new structure in place.
2017-03-05 20:16:33 +13:00
Aldo Cortesi
67381ae550 Revamp options
- Options are now explicitly initialized with an add_option method
- We have one canonical Options class - ditch dump.Options
2017-03-05 15:10:03 +13:00
Nikhil Soni
0081d9b828 Merge branch 'master' into on-issues 2017-03-03 12:58:44 +05:30
Nikhil Soni
3da8532bed Adds test for --keep-host-header 2017-03-03 12:34:36 +05:30
Nikhil Soni
317d183ba4 Changes dns_spoofing example to use --keep-host-header 2017-03-03 12:34:36 +05:30
Thomas Kriechbaumer
a505221e1a http2: fix h2 header normalization test (#2080) 2017-03-01 12:46:42 +01:00
David Dworken
99b584ad7d added XSS scanner example (#1907) 2017-02-27 18:22:39 +01:00
Thomas Kriechbaumer
e0bc1109c0 do not return IP addresses in SSLCert.altnames 2017-02-26 20:54:06 +01:00
Thomas Kriechbaumer
6d228dfb9f certs: coverage++ 2017-02-26 20:54:04 +01:00
Thomas Kriechbaumer
9b6986ea87 connections: coverage++ (#2064) 2017-02-26 20:50:52 +01:00
Thomas Kriechbaumer
2aecffd39a optmanager: coverage++ (#2062) 2017-02-25 12:12:43 +01:00
Thomas Kriechbaumer
ccd8e1e617 http: coverage++ (#2060) 2017-02-24 22:39:16 +01:00
Thomas Kriechbaumer
3c66eb5b4b move tests around (#2059) 2017-02-24 22:00:41 +01:00
Thomas Kriechbaumer
e89c785719 move tests around (#2058) 2017-02-24 21:39:13 +01:00
Thomas Kriechbaumer
d17309eda8 flowfilter: coverage++ 2017-02-24 18:47:56 +01:00
Thomas Kriechbaumer
3e9125a3c1 nuke tcp.Address and add proper IPv6 support 2017-02-24 13:39:02 +01:00
Maximilian Hils
ba76dbc672 Merge pull request #2048 from ujjwal96/ipv6-addresses
Absolute IPv6 addresses supported
2017-02-22 13:17:53 +01:00
Matthew Shao
5fc4fc28b6 Add test for @concurrent decorator in class. 2017-02-21 16:56:48 +08:00
Ujjwal Verma
11134b669e Absolute IPv6 addresses supported 2017-02-20 21:02:43 +05:30
Thomas Kriechbaumer
0de277c18b improve test 2017-02-20 11:48:57 +01:00
Thomas Kriechbaumer
050245e842 Merge pull request #2017 from lymanZerga11/patch-1
Catch ValueErrors from url.parse()
2017-02-20 11:48:40 +01:00
Thomas Kriechbaumer
ac6915c002 improve tests 2017-02-20 11:11:10 +01:00
Thomas Kriechbaumer
51e08a5b85 Merge pull request #2041 from krsoninikhil/debug
Adds PyInstaller build indicator to --version argument
2017-02-20 10:29:15 +01:00
Nikhil Soni
ed084f5413 Adds test for pyinstaller indicator 2017-02-19 19:15:36 +05:30
Thomas Kriechbaumer
36352c9539 protobuf: coverage++ 2017-02-19 14:29:09 +01:00
Thomas Kriechbaumer
83c2de8849 http2: disable priority forwarding 2017-02-18 14:30:08 +01:00
Maximilian Hils
4158a1ae55 Merge pull request #2040 from mhils/request-host-header
Add "Request.host_header"
2017-02-18 12:08:54 +01:00
Thomas Kriechbaumer
6ef4f094b3 Merge pull request #2038 from Kriechi/improve-http2-cov
test forbidden HTTP/2 headers in reponses
2017-02-18 11:50:57 +01:00
Thomas Kriechbaumer
47e6f977de test forbidden HTTP/2 headers in reponses 2017-02-18 11:26:54 +01:00
Maximilian Hils
49c8e19f80 add request.host_header, fix #2036 2017-02-17 23:31:53 +01:00
Maximilian Hils
6a1e03ac6f tests: make full cov plugin cwd-independent 2017-02-17 23:29:10 +01:00
Maximilian Hils
33acb48c71 Merge pull request #2032 from Kriechi/individual-coverage
add individual-coverage check
2017-02-17 22:56:13 +01:00
Thomas Kriechbaumer
4cec88fc7f remove unneeded test module 2017-02-17 20:41:10 +01:00
mame82
ffdbccd571 Use existing urlencoding conventions when re-encoding edited form #1946 (#2022) 2017-02-16 15:03:27 +01:00
Thomas Kriechbaumer
6b22ca7a32 add individual-coverage check 2017-02-16 11:43:45 +01:00
Thomas Kriechbaumer
337b1c9399 fix wrong filename 2017-02-15 18:45:49 +01:00
Maximilian Hils
d1186eea18 Merge pull request #2029 from mhils/proxyauth
Improve ProxyAuth
2017-02-15 16:52:20 +01:00
Thomas Kriechbaumer
94a7e99fda Merge pull request #2028 from Kriechi/nuke-pillow
nuke Pillow
2017-02-15 15:29:57 +01:00
Maximilian Hils
4bac850bb1 fix #1722, fix #1734, refs #2019 2017-02-15 14:20:46 +01:00
Maximilian Hils
0d9c7ce50c image view: add fallback, catch all images but svgs 2017-02-15 14:04:24 +01:00
Thomas Kriechbaumer
8a6f8bd461 nuke Pillow 2017-02-15 00:54:14 +01:00
Thomas Kriechbaumer
24a51df9cb extract full-coverage pytest plugin 2017-02-15 00:29:43 +01:00
Thomas Kriechbaumer
4d973e8295 fix imports 2017-02-14 23:47:33 +01:00
Thomas Kriechbaumer
a12c3d3f8e restructure and move test files
add empty test files to satisfy linter
2017-02-14 23:31:37 +01:00
Thomas Kriechbaumer
04748e6f3f add filename-matching linter 2017-02-14 22:48:24 +01:00
lymanZerga11
d30ef7ee3e Update test_flowlist.py 2017-02-14 21:17:18 +08:00
lymanZerga11
a52d8c1dab Update test_flowlist.py 2017-02-14 21:01:01 +08:00
lymanZerga11
26a17a3d82 Update test_flowlist.py 2017-02-13 15:05:29 +08:00
lymanZerga11
a912d67c06 Update test_flowlist.py 2017-02-13 10:47:50 +08:00
lymanZerga11
577fb818b9 Update test_flowlist.py 2017-02-13 10:39:48 +08:00
lymanZerga11
f5b30b8872 Update test_flowlist.py 2017-02-12 23:36:26 +08:00
lymanZerga11
df8a5aa9be Update test_flowlist.py 2017-02-12 23:23:23 +08:00
lymanZerga11
c622e4a649 Create test_flowlist.py 2017-02-12 23:10:49 +08:00
Maximilian Hils
3067a971f9 Merge pull request #2000 from s4chin/add-jpeg-parser
Add jpeg parser
2017-02-11 23:00:14 +01:00
Sachin Kelkar
245e24dcf3 Add sources of images, remove copyrighted image 2017-02-12 01:19:36 +05:30
Maximilian Hils
71742654e3 Merge pull request #2005 from ujjwal96/script-reload
Closes #1828 script reloads on py file changes
2017-02-11 01:45:14 +01:00
Thomas Kriechbaumer
c8c79cc291 coverage++ 2017-02-10 22:55:15 +01:00
Sachin Kelkar
8eb1d34644 Add app1 marker and tests 2017-02-10 22:40:19 +05:30
Ujjwal Verma
809207195d closes #1828 script reloads on py file changes 2017-02-10 22:21:12 +05:30
Sachin Kelkar
c4e141a000 Add jpeg app0 sof0 com parsing 2017-02-10 11:16:23 +05:30
Ujjwal Verma
d6465b907f Closes #1580 Gives warning when explicit proxy configured in transparent mode (#1996) 2017-02-09 20:56:06 +01:00
Maximilian Hils
380ff50e57 Merge pull request #2003 from Kriechi/coverage++
test refactoring and coverage++
2017-02-09 17:45:50 +01:00
Maximilian Hils
da8444b11f flow_count() -> len(flows) 2017-02-09 16:44:27 +01:00
Maximilian Hils
1084588103 Merge pull request #1998 from mhils/fix-change-upstream
Fix change_upstream_proxy_server
2017-02-09 16:39:20 +01:00
Thomas Kriechbaumer
d10560d54c improve coverage report 2017-02-09 16:08:36 +01:00
Thomas Kriechbaumer
2ff5d72236 minor changes 2017-02-09 16:08:36 +01:00
Thomas Kriechbaumer
0299bb5b2e eventsequence: coverage++ 2017-02-09 16:08:36 +01:00
Thomas Kriechbaumer
5a3976c43e coverage whitelist -> blacklist 2017-02-09 16:08:36 +01:00
Thomas Kriechbaumer
f6cea09d5a stateobject: coverage++ 2017-02-09 16:08:36 +01:00
Thomas Kriechbaumer
1847cf175c websockets, tcp, version: coverage++ 2017-02-09 16:08:35 +01:00
Thomas Kriechbaumer
ae94ca6fa9 remove deprecated flow_count function 2017-02-09 16:08:35 +01:00
Maximilian Hils
9b97b63891 fix change_upstream_proxy_server 2017-02-09 15:24:31 +01:00
Maximilian Hils
7a205e80aa improve stateobject tests 2017-02-09 14:21:25 +01:00
Thomas Kriechbaumer
7a9d40817c pytest.raises: shim new API 2017-02-08 20:04:07 +01:00
Maximilian Hils
28c0596742 Merge pull request #1997 from s4chin/add-gif-parser
Add gif parser and tests
2017-02-08 12:09:48 +01:00
Sachin Kelkar
66ad95c330 Fix gif parsing error on some gifs and add test 2017-02-08 00:03:07 +05:30
Sachin Kelkar
4ef5de2cce Add gif parser and tests 2017-02-06 17:57:59 +05:30
Ujjwal Verma
c622622c59 Encoding fixes and tests 2017-02-05 18:59:01 +05:30
Thomas Kriechbaumer
288f9a3857 fix flaky test 2017-02-05 10:42:47 +01:00
Thomas Kriechbaumer
966ffaa3d6 coverage++ 2017-02-04 13:48:55 +01:00
Maximilian Hils
53f298ac41 Merge pull request #1967 from s4chin/remove-pillow
Add png parser
2017-02-03 18:58:34 +01:00
Maximilian Hils
bbdb7300fd Merge pull request #1983 from Kriechi/coverage++
coverage++
2017-02-03 18:56:56 +01:00
Sachin Kelkar
15548ff433 Feedback 2017-02-03 18:40:50 +05:30
Thomas Kriechbaumer
bcaaa2f40b coverage++ 2017-02-03 11:55:33 +01:00
Sachin Kelkar
0674485e76 Fix and cleanup 2017-02-03 15:02:35 +05:30
Sachin Kelkar
88dbfd5257 Add iTXt and zTXt tests 2017-02-03 14:12:54 +05:30
Sachin Kelkar
abef020e07 Fix as per feedback and add more tests 2017-02-03 14:12:54 +05:30
Sachin Kelkar
6202958048 Add png parser 2017-02-03 14:12:54 +05:30
Ujjwal Verma
e10c36fe11 Added test 2017-02-03 02:58:02 +05:30
Thomas Kriechbaumer
79aa994275 fix merge 2017-02-02 17:26:31 +01:00