Commit Graph

2502 Commits

Author SHA1 Message Date
Maximilian Hils
1bc265ace7
Merge pull request #5226 from mhils/cleanup
`pyupgrade --py38-plus`
2022-03-29 15:36:32 +02:00
Maximilian Hils
9d1e3107e8 pyupgrade --keep-runtime-typing --py38-plus 2022-03-29 15:29:37 +02:00
Maximilian Hils
b8bed1d770
errorcheck: print message to stderr for console UI (#5225) 2022-03-29 15:26:46 +02:00
Maximilian Hils
f0da667516
tlsconfig: make sure to use the correct connection, fix #5109 (#5224) 2022-03-29 08:27:53 +00:00
Paulo Romeira
e534086053
Add flatpak support to the browser addon (#5200) 2022-03-23 13:56:20 +01:00
Maximilian Hils
b06fb6d157 security: reject whitespace in HTTP/1 header names
This commit fixes GHSA-gcx2-gvj7-pxv3 by making mitmproxy
reject header names that contain whitespace characters by default.
A new `validate_inbound_headers` option is provided to turn this behavior
off at the expense of allowing HTTP smuggling vulnerabilities.
2022-03-19 17:20:32 +01:00
Maximilian Hils
9243ba4e25 fix vt code detection on Windows 2022-03-19 17:20:32 +01:00
Alexander Prinzhorn
e8ae38c8b6
Fixed encoding guessing: only search for meta tags in HTML bodies (#4566) 2022-03-17 14:25:26 +00:00
Maximilian Hils
a9283befad
improve self-connect error message (#5192) 2022-03-17 14:24:42 +00:00
EndUser509
3a5550a09c
Rotate stream files (#5097)
* Example addon for saving streamed data including a small bug fix to make it work.

* Revert "Example addon for saving streamed data including a small bug fix to make it work."

This reverts commit 02ab78def9a52eaca1a89d0757cd9475ce250eaa.

* Add support for rotating stream files every hour or day

* Added tests

* Modified to change the stream file every time the formating string changes as time moves on.

* Update to more compact version

* simplify save addon logic

* make mypy happy

* fix compatibility with Python 3.8

Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2022-03-17 10:41:05 +00:00
Maximilian Hils
ecd4790cbb
vendor click.style instead of depending on click (#5188)
click introduces subdependencies like colorama, this approach is much simpler.
2022-03-16 17:02:53 +00:00
Maximilian Hils
e2f42ddb30
exit for all tools on startup error, fix #4544 (#5187) 2022-03-16 13:23:32 +00:00
Maximilian Hils
148429c0b3
lowercase user-added HTTP/2 headers, fix #4746 (#5186) 2022-03-16 10:59:30 +01:00
Maximilian Hils
6f0587734e
Refactor how we process --set options (#5067)
* refactor how we process `--set` options

* minor improvements based on @marwinxxii's review

* fix nits

* update changelog
2022-03-16 07:33:30 +00:00
Alexander Prinzhorn
8e1adbc5df
Add WebSocketMessage.injected flag (#5105)
* Add WebSocketMessage.injected flag

* add flow format migration

Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2022-03-16 07:10:23 +00:00
Vinayak Khandelwal
3d5f6da048
ISSUE_5068 (#5161)
* changes for custom port number

* indent correction

* test coverage

* coverage correction

* simplify LDAP auth

* make mypy hapy

Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2022-03-15 20:39:38 +00:00
Maximilian Hils
e22f8db526 add asyncio exception handler test 2022-03-15 19:59:51 +01:00
Maximilian Hils
290ec68aca remove pytest.mark.asyncio 2022-03-15 19:50:06 +01:00
Maximilian Hils
a7d473c1c1 fix compatibility with Python <3.10 2022-03-15 19:50:06 +01:00
Maximilian Hils
5fc20e3e8c tests++ 2022-03-15 18:43:56 +01:00
Maximilian Hils
b491528981 exclude time-sensitive test from CI 2022-03-04 11:38:28 +01:00
Maximilian Hils
372a632161 reintroduce Flow.live
We previously relied on the state of `Flow.reply` to check if a flow can be killed,
but this doesn't work anymore with `Flow.reply` being removed. Instead, we now
reintroduce the `Flow.live` attribute, which signals if we are on a live connection.
Killing still is not ideal (see comment in `Flow.killable`), but this paves the way.
2022-02-04 17:30:58 +01:00
Maximilian Hils
4f5930ec5d simplify concurrency tests 2022-02-04 17:30:20 +01:00
Robert Xiao
794c18bec0 Entirely remove the reply system.
The major, breaking change is that it is no longer possible to "take" a reply in
order to block the effect of a later addon hook.

This is patch 4/4 of the reply-ectomy.
2022-02-04 17:30:20 +01:00
Robert Xiao
ede269fce4 Flow.intercept: use an Event instead of the reply system
This is patch 3/4 of the reply-ectomy.
2022-02-04 17:30:20 +01:00
Robert Xiao
fd43ca19c4 Flow.kill: don't depend on reply status.
In principle, a flow is killable as long as the connection handler is still
checking the error status of the flow.

This is patch 2/4 of the reply-ectomy.
2022-02-04 17:30:20 +01:00
Robert Xiao
4448550746 Rewrite @concurrent to use async instead of the reply mechanism.
This should improve behaviour, since calls to @concurrent will now be serialized
relative to other hooks on the same flow (but will still run in parallel with
hooks on different flows). Unlike a plain async hook, @concurrent allows blocking
sync APIs to run concurrently (e.g. requests).

This is patch 1/4 of the reply-ectomy.
2022-02-04 17:30:20 +01:00
Robert Xiao
caf49300c2 Use async for tctx.cycle/tctx.invoke. 2022-02-04 17:30:20 +01:00
Robert Xiao
e186ccb3ba fix test.helper_tools.dumperview script.
Previously, it was throwing the following error:

  File mitmproxy/addons/dumper.py, line 133, in _echo_request_line
    elif flow.client_conn.peername:
AttributeError: 'bool' object has no attribute 'peername'
2022-02-04 17:30:20 +01:00
Robert Xiao
ee4999e8e4 Rename new async helper functions.
async_trigger -> trigger_event
invoke_addon -> invoke_addon_sync (API breakage)
async_invoke_addon -> invoke_addon
2022-02-04 17:30:20 +01:00
Robert Xiao
687ab4b589 Tests for async hook support 2022-02-04 17:30:20 +01:00
Maximilian Hils
8c86fd06db add changelog linkifier util 2022-02-04 16:00:44 +01:00
Maximilian Hils
53f60c88b1
fix a crash when refreshing headers with a negative unix timestamp, fix #5054 (#5078) 2022-01-18 22:32:38 +00:00
Maximilian Hils
d47fd3e9c3
ClientHello: add .raw_bytes(), refs #4877 (#5075) 2022-01-18 11:34:09 +00:00
Maximilian Hils
ef8c88da1f
setup websocket connection before finishing HTTP response, fix #5017 (#5020) 2021-12-27 15:58:24 +00:00
Maximilian Hils
3fbf3cf8ee tlsconfig: don't overwrite existing TLS context, refs #5019 2021-12-27 12:22:08 +01:00
Maximilian Hils
c74806feac
fix tests 2021-12-20 21:06:14 +01:00
Maximilian Hils
6997129bc0
make sure that running() is only invoked once on startup. (#4964)
fix #3584
2021-11-27 13:11:23 +00:00
Maximilian Hils
2dd845ed95 catch malformed cert warning in tests
we may need to catch this properly in `get_cert` at some point,
let's see if this ever turns out to be an issue.
2021-11-22 12:55:17 +01:00
Maximilian Hils
3cb87f5a2f split tls_handshake hook into client/server and success/fail variants 2021-11-22 10:23:21 +01:00
Maximilian Hils
a72f61ef57 Merge remote-tracking branch 'origin/main' into ignore-after-clienthello 2021-11-22 09:54:08 +01:00
shindexro
9a469806eb
quote argument of view.flows.resolve (#4910)
* Fix #4902

* Update type signature

* Switch to None check

* Fix spacing

* Quote view.flows.resolve argument

* Switch to call_strings
2021-11-19 13:04:20 +01:00
shindexro
888ce66f90
Correct flow-detail documentation (#4909)
* Fix #4902

* Update type signature

* Switch to None check

* Fix spacing

* Skip URL shortening when flow-detail is 0

* Sync docs with implementation

* Update URL shortening test
2021-11-18 08:26:05 +01:00
shindexro
39fa242e25
fix crash when invoking replay.server.count from console (#4905)
fix #4902
2021-11-17 16:53:43 +00:00
Peter Hoffmann
260fc68211
Fix #4876 Don't do CONNECT on plaintext HTTP replays via upstream (#4882)
* Replays via upstream also need to comply with upstream handling

* Adjusted test for HTTP upstream replay which should NOT do a CONNECT

* Added Changelog

* Test for replay https pver upstream with CONNECT

* Proxy requests use full URL with host & port

* Finally remove some prints

* lint!

Co-authored-by: Maximilian Hils <github@maximilianhils.com>
2021-10-31 20:23:04 +00:00
mame82
3a884ceb8f
gRPC packed repeated encoding (#4872)
* Replaced Kaitai protobuf parser with custom implementation, to prepare packed data handling

* Clamp varint size for bool conversion to 64bit, to satify tests

* moved WireParser into ProtoParser

* preserve work state

* Full packed support

* noc changes to example addon

* Adjusted test regex for ValueError

* Do not try to unpack fields twice
2021-10-21 13:20:13 +02:00
Maximilian Hils
79f464bc78
ignore superfluous newlines after HTTP CONNECT, fix #4870 (#4871) 2021-10-19 11:25:22 +00:00
mame82
8b6345e4a7
gRPC view: Adjusted parsing strategy for deprecated protobuf fields (#4858)
* Adjusted parsing strategy for deprecated protobuf fields

* Added test comment

* adjusted comment to reflect correct field IDs
2021-10-13 13:16:49 +02:00
mame82
480052f58b
Grpc contentview (#4851)
* Partial gRPC contentview prototype, not linted, no tests, not as add-on

* Linted (flake8)

* Save dev state

* Rewrote of protobuf parser, use decoding strategy, reduced rendered data. Parser uses  generators

* minor cleanup

* fix: preferred encoding was provided as function instead of value

* flake8: line length

* Backlinked message tree objects, temporary debug out

* Partial implementation of gRPC definitions. Save state to fix a cras (data invalidate in edit mode)

* hack: deal with missing exception handling for generator based content views

* gRPC/Protoparser descriptions (with test code)

* replaced manual gzip decoding with mitmproxy.net.encoding.decode

* Refactored typing imports

* Reafctoring

* distinguish request vs response definitions, separate view config from parser config

* Code cleaning, moved customized protobuf definitions to example addon

* final cleanup

* changelog

* Stubs for tests

* Fixed render_riority of addon example

* Started adding tests

* Work on tests

* mypy

* Added pseudo encoder to tests, to cover special decodings

* Example addon test added

* finalized tests, no 100 percent coverage possible, see comments un uncovered code

* minor adjustments

* fixup tests

* Typos

Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2021-10-12 13:32:56 +02:00
Maximilian Hils
aad92c9d5a
Merge pull request #4847 from mhils/flowfilter
Flowfilter Improvements
2021-10-09 18:38:56 +02:00