Commit Graph

1990 Commits

Author SHA1 Message Date
Fred Miller
bf3570b3b9
Skip file permission test on Windows 2018-10-23 08:53:51 +08:00
Fred Miller
f16621a38b
Make private keys readable only by the owner 2018-10-22 22:53:58 +08:00
Miroslav
ed9e3d5137 keys.yaml priority over defaultkeys.py. Test for this. 2018-09-29 15:54:17 +03:00
Maximilian Hils
e15619f34e
Merge pull request #3319 from mhils/issue-3316
Fix #3316
2018-09-07 10:34:55 +02:00
Maximilian Hils
1b6a8d6acd fix #3316 2018-09-07 10:24:28 +02:00
David Kremer
dcd8ba34ab [test/xss_scanner] add fixtures in tests
The previous version of that script made a repetitive use
of the  pytest builtin fixture, with always
the same arguments.

This is a small refactoring and cleanup, mainly adding the 'function'
scope to ensure proper cleaning and using the @pytest.fixture
decorator where I could.
2018-08-15 19:47:55 +02:00
madt1m
a52451900c session: implemented changes requested after PR review. 2018-08-05 21:57:55 +02:00
madt1m
e9c2b12dab tests: Full coverage. Everything working, ready for review 2018-08-03 16:32:50 +02:00
madt1m
4e0c10b88b tests: 97% coverage reached. Session opportunely patched after emerged defects. 2018-08-02 05:55:35 +02:00
Pietro Francesco Tirenna
9c949bd2f8
Merge pull request #3252 from madt1m/session-db
Session - Hybrid DB
2018-07-24 16:26:10 +02:00
madt1m
8c7793b91a session: temporary DB is now stored in temporary dir 2018-07-24 15:57:11 +02:00
madt1m
68eb07b668 session: modified schema. Now SessionDB uses tempfile module for temp session 2018-07-24 11:58:33 +02:00
madt1m
e727446f14 benchmark: some improvements - limit to queue size 2018-07-23 21:18:24 +02:00
madt1m
8ab82ad9a3 benchmark: new protobuf serialization performance profiler 2018-07-22 12:47:54 +02:00
Aldo Cortesi
ec092fdc12
Merge pull request #3245 from madt1m/protobuf-serialization-clean
Shifting to Protobuf Serialization - Cleaned
2018-07-21 16:13:25 +12:00
madt1m
3b5cdf7f67 test_linting: removed some unused local variables 2018-07-20 18:15:27 +02:00
madt1m
fad8e7c99b tests: SessionDB fully tested 2018-07-20 16:58:11 +02:00
Miroslav
dcb3de40b1 Some refactoring. New test case. 2018-07-19 16:56:34 +03:00
Pietro Francesco Tirenna
d5da74645b protobuf: tests implemented, full coverage 2018-07-17 19:23:01 +02:00
Miroslav
ffbd7c20e5 Command history implementation 2018-07-17 18:37:45 +03:00
Thomas Kriechbaumer
3cd3765270
Merge pull request #3211 from cortesi/tcheck
Add typechecking of Any values for state object
2018-06-17 08:45:03 +02:00
Aldo Cortesi
77b49aa8de Add typechecking of Any values for state object
An ugly solution for an ugly little problem. This patch uses JSON's type
checker to validate Any values in stateobject, in order to avoid a circular
import.

Fixes #3180
2018-06-17 10:22:17 +12:00
Aldo Cortesi
9ff4f55614
Merge pull request #3202 from madt1m/view-cleanup
View Cleanup - Initial steps
2018-06-17 09:20:34 +12:00
Aldo Cortesi
9463fee764 cibuild: permit non-dev versions on maintenance branches
Cater for the corner case where commits are incorporated on a maintenance
branch. We should be able to test these without adding a dev suffix to the tool
versions.
2018-06-16 15:09:34 +12:00
Aldo Cortesi
85526e5370 cibuild: Match Travis env variable behaviour
Whenever TRAVIS_TAG is non-empty, TRAVIS_BRANCH is over-ridden to match the
TRAVIS_TAG value. Adjust our tests to reflect this, and add a sanity check that
fails hard if this constraint is ever not met.

See https://github.com/travis-ci/travis-ci/issues/4745
2018-06-16 10:23:33 +12:00
Aldo Cortesi
bf791ba1d5 cibuild: refactor to handle v prefix for release tag names 2018-06-16 09:48:21 +12:00
madt1m
af54c26014 fixed exception type; full coverage on view 2018-06-13 17:39:46 +02:00
madt1m
40faf2c662 Added tests for new primitives 2018-06-13 15:57:21 +02:00
madt1m
773c953514 View API slightly extended; codebase cleaned in some points 2018-06-13 11:56:14 +02:00
Thomas Kriechbaumer
be26958ea8 improve websocket frame masking api 2018-06-11 13:28:28 +02:00
Maximilian Hils
09ff5df2fb Such CI, Very Wow. (#3182) 2018-06-02 21:37:44 +02:00
Maximilian Hils
aa0cb2ba2f add missing await 2018-05-28 22:27:27 +02:00
Aldo Cortesi
a38d2d7b0e addons/block: teach block about IPv6 scope suffixes
Fixes #3160
2018-05-27 10:48:23 +12:00
Aldo Cortesi
ec2ae19e22 optmanager: tweaks and cleanups 2018-05-27 10:43:14 +12:00
Aldo Cortesi
e59ba13417 Use deferral mechanism for cfg file options
Fixes #3162
2018-05-27 10:12:24 +12:00
Aldo Cortesi
b6e1c4bb9d cibuild: tweak docker tag, only upload prod builds to pypi 2018-05-25 12:46:56 +12:00
Aldo Cortesi
ff92962c51 cibuild: start building version awareness 2018-05-25 10:50:48 +12:00
Aldo Cortesi
fdf0d9974e cibuild: more accurate PR test, better dump output 2018-05-24 22:58:33 +12:00
Aldo Cortesi
e9b19dba8a cibuild: fix docker upload condition, expand tests 2018-05-24 22:32:31 +12:00
Aldo Cortesi
c91b511bc7 cibuild: extract upload conditions 2018-05-24 22:18:01 +12:00
Aldo Cortesi
0afff3a952 cibuild: add docker tag calculation to environ 2018-05-24 21:59:45 +12:00
Aldo Cortesi
bae4a3393f release: include release directory in coverage
With lots of exclusions for now
2018-05-24 19:03:15 +12:00
Aldo Cortesi
8fceaca6b8 cibuild: Consolidate build environment and add tests 2018-05-24 11:06:43 +12:00
Aldo Cortesi
064929b559 clientreplay: fix racy tests harder 2018-05-18 09:51:56 +12:00
Thomas Kriechbaumer
976ab0c466 websocket: inject messages via flow 2018-05-16 21:50:15 +02:00
Thomas Kriechbaumer
9979be6487 fix py36/37 escaping issues
closes #3122

https://bugs.python.org/issue16285
https://bugs.python.org/issue12910
21024f0662
2018-05-14 10:41:46 +02:00
Aldo Cortesi
58ff51da10 command: handle string without terminal escaped char
Fixes #2810
2018-05-12 14:02:20 +12:00
Aldo Cortesi
88fe26997c script: revamp ephemeral script running
Fixes 2838
2018-05-12 12:35:24 +12:00
Aldo Cortesi
482043cdcf commands: handle type errors on startup
fixes #3088
fixes #2787
2018-05-12 11:10:14 +12:00
Aldo Cortesi
6dff8c58ad commands: if no explicit return type is specified, assume None
This is going to be a super common error for addon authors, so we might as well
handle it.
2018-05-12 10:15:08 +12:00
Aldo Cortesi
8c63a8818d keymap: read keys from CONFDIR/keys.yaml by default 2018-05-10 16:06:52 +12:00
Aldo Cortesi
9830e5b597 cadir -> confdir
We store a lot more than just the CAs in our configuration directory. Clarify
the option name.
2018-05-10 11:40:33 +12:00
Aldo Cortesi
3438912236 console keybindings: define YAML-based format for console key binding persistence 2018-05-10 11:30:51 +12:00
Aldo Cortesi
7d9b626d2e clientplayback: try to fix flaky replay test 2018-05-08 16:33:41 +12:00
Aldo Cortesi
af1a4ffdcd Ditch the addon stdout wrapper
This results in a 30% improvement in our core request throughput.

Fixes #3102
2018-05-08 14:26:41 +12:00
Aldo Cortesi
717fbaa990 optmanager: refactor for coverage and clarity 2018-05-08 14:24:41 +12:00
Aldo Cortesi
1b0eed19d1 benchmarks: quick-run scripts for mitmproxy and mitmdump 2018-05-08 11:08:37 +12:00
Aldo Cortesi
9b51393b6d benchmark: keep track of requests/responses seen by proxy 2018-05-08 11:00:20 +12:00
Aldo Cortesi
f7d7e31f06 options: add the concept of deferred settings
We've had a perpetual sequencing problem with addon startup. Users need to be
able to specify options to addons on the command-line, before addons are
actually loaded. This is only exacerbated with the new async core, where load
order can't be relied on.

This patch introduces deferred options. Options passed with "--set" on the
command line are deferred if they are unknown, and are automatically applied by
the addon manager once matching addons are registered and their options are defined.
2018-05-08 10:56:00 +12:00
Maximilian Hils
752a367436 remove leftover processing_complete 2018-05-06 15:47:12 +02:00
Aldo Cortesi
60acbd79b9 Remove allowremote addon, add an improved take called block
We now have two options: block_global blocks global networks, block_private
blocks private networks. The block_global option is true by default, and
block_private is false by default. The addon name is "block" so the options are
correctly prefixed.

Also make option documentation precise, reduce verbosity of logs.
2018-05-06 12:43:25 +12:00
Aldo Cortesi
c53bc39c95
Merge pull request #3099 from Kriechi/fix-3024
fix #3024
2018-05-06 11:27:14 +12:00
Thomas Kriechbaumer
54e2daa21e fix #3024 2018-05-05 14:38:02 +02:00
Aldo Cortesi
6bd2d5059b Remove custom events
These were always weird, and were added to support exit when processing
completed to mitmdump. We now have better ways to do this.
2018-05-03 09:42:51 +12:00
Aldo Cortesi
f380a77dee Remove the tick event
Mitmproxy: the tickless wonder.
2018-05-02 14:29:15 +12:00
Aldo Cortesi
38ff8109fb taddons: remove has_event
We no longer use this anywhere, so ditch it.
2018-05-02 11:31:28 +12:00
Aldo Cortesi
22a4b1d5d4 Redesign keepserving
- Instead of listening for a pseudo-event, we periodically check whether client
replay, server replay or file reading is active.
- Adjust server replay not to
use tick.
- Adjust readfile to expose a command to check whether reading is in progress.
2018-05-02 11:26:40 +12:00
Aldo Cortesi
00d790fe84 commands: clarify command call interface, fix web app replay 2018-05-02 08:36:15 +12:00
Aldo Cortesi
6d27b28b85 client replay: expad and consolidate tests 2018-05-01 08:47:26 +12:00
Aldo Cortesi
236a2fb6fd client replay: re-design
Re-design the way client replay works. Before, we would fire up a thread,
replay, wait for the thread to complete, get the next flow, and repeat the
procedure. Now, we have one replay thread that starts when the addon starts,
which pops flows off a thread-safe queue. This is much cleaner, removes the
need for busy tick, and sets the scene for optimisations like server connection
reuse down the track.
2018-04-30 17:17:03 +12:00
Aldo Cortesi
28d53d5a24 client replay: move all client replay-related code into addon 2018-04-27 16:34:56 +12:00
Aldo Cortesi
a4a48a96d6
Merge pull request #3078 from kajojify/expanduser
Fix #3002. Auto-expanding for tilda.
2018-04-26 21:02:10 +12:00
Aldo Cortesi
b18e0fa2bc
Merge pull request #3071 from cortesi/tickless
asyncio: shift script reloading out of the tick event
2018-04-26 11:47:07 +12:00
Miroslav
ea6fab09b5 Fix #3002. Auto-expanding 2018-04-26 01:35:44 +03:00
Maximilian Hils
19ceb6f49a fix pytest invocation errors 2018-04-25 14:00:50 +02:00
Aldo Cortesi
95e690ba31 test: shift test_data out of our public API 2018-04-23 19:16:33 +12:00
Aldo Cortesi
0ba10b6109 addons/script: improve relability of reload test
The granularity of mtime is surprisingly bad. Make the tests more robust
against this, and promote has_log back to a public method, now that we have a
few legitimate examples.
2018-04-23 13:19:20 +12:00
Aldo Cortesi
44016a0de5 asyncio: shift script reloading out of the tick event
The tick event is a nasty compromise, left over from when we didn't have an
event loop. This is the first patch in a series that explores moving our
built-in addons to managing coroutines on the eventloop directly for periodic
tasks.
2018-04-23 10:28:18 +12:00
Aldo Cortesi
ef4db52887 Ignore signal errors on windows + various fixes
- Ignore the NotImplementedError raised by add_signal_handler on Windows.
- Entrypoints return an integer exit code, or None. Adjust our type annotations
and code to suit.

Fixes #3061
2018-04-17 08:20:52 +12:00
Aldo Cortesi
565146311a asyncio: clarify shutdown semantics
This patch clarifies proxy shutdown, and specifies that the master.shutdown()
method is thread-save.
2018-04-16 10:16:51 +12:00
Aldo Cortesi
850c855495 Clean up test/helper_tools directory 2018-04-16 09:20:22 +12:00
Aldo Cortesi
b663a224a3 Improve benchmarking
- The benchmark addon now manages setting up and tearing down the backend and
traffic processes itself.
- Use wrk instead of hey. I get more consistent results with this tool, and hey
shows a strange tail-latency bump that seems artificial.
- Make termination behaviour simpler. The bencmark revealed a bug where .done
events were not called if the proxy was shut down by an addon.
2018-04-16 09:19:14 +12:00
Aldo Cortesi
5f74adc2df
Merge pull request #3059 from obscure76/enh/issue-3053
fix Python 3.6 variable type annotations #3053
2018-04-16 07:35:20 +12:00
oscure76
0e984e1442 fix Python 3.6 variable type annotations #3053 2018-04-14 16:24:41 -07:00
Aldo Cortesi
865a373bd3
Merge pull request #3050 from kira0204/error-script
Handling user script's exceptions, fix #2839
2018-04-15 10:12:44 +12:00
Aldo Cortesi
8609de6f31 readfile: add a readfile_filter option
* Add a readfile_filter option that filters flows on read.
* Adjust test suite for asyncio.
* Add asynctest as a dev dependency.
2018-04-14 11:52:02 +12:00
kira0204
6780e5025a Handling user script exceptions, fix #2839 2018-04-10 08:33:40 +05:30
Aldo Cortesi
9dcd15d350 asyncio: ditch the handler context
There are a few reasons for this. First, logs are now async, and can be called
at any time. Second, the event loop is thread local, so there can only ever be
one master per thread. These two things together completely obviate the need
for a handler context.
2018-04-07 09:04:14 +12:00
Aldo Cortesi
6a08ef465f asyncio: remove master.add_log, in favor of a persistent log.Log instance 2018-04-07 09:02:10 +12:00
Aldo Cortesi
ea213782d8 asyncio: remove test master.has_log
Now that logs are async, using this call is almost always a mistake. Signal
this by making it semi-private. The method may go away entirely down the track.
2018-04-07 09:02:10 +12:00
Aldo Cortesi
80f2bac356 asyncio: move log mechanism onto the event loop
Logs are now asynchronous, with a log entry pushed onto the event loop for
handling. To support this, the test mechanism grows an await_log method that
waits for a log entry to appear.
2018-04-07 09:02:10 +12:00
Aldo Cortesi
0fa1280daa asyncio simplify: we don't need a queue for proxy->main loop comms
Instead, we just schedule coroutines directly onto the core loop.
2018-04-07 08:59:08 +12:00
Aldo Cortesi
cdbe6f97af asyncio: remove last vestiage of channel input from master 2018-04-07 08:59:08 +12:00
Aldo Cortesi
506ccc5693
Merge pull request #3031 from itzikBraun/add-httpie-export
added option to export request as httpie command
2018-04-06 10:02:02 +12:00
Aldo Cortesi
4177f50a0f
Merge pull request #2971 from kira0204/user-script-exception
A unified handler for user's scripts exception
2018-04-06 10:01:35 +12:00
Maximilian Hils
50ea3aa420 fix http retry timeout
this fixes #3038
2018-04-05 15:30:04 +02:00
Aldo Cortesi
659fceb697 minor fixes 2018-04-03 10:26:09 +12:00
Aldo Cortesi
7e73e5fa51 asyncio: factor out test server startup wait 2018-04-03 09:46:11 +12:00
itzikBraun
29f4164344 added option to export request as httpie command 2018-04-02 12:57:33 +02:00
Aldo Cortesi
14f461c5d5 asyncio: cleanup and lint
Also fix a racy websocket test.
2018-04-02 16:41:49 +12:00