Commit Graph

2464 Commits

Author SHA1 Message Date
Maximilian Hils
2dd845ed95 catch malformed cert warning in tests
we may need to catch this properly in `get_cert` at some point,
let's see if this ever turns out to be an issue.
2021-11-22 12:55:17 +01:00
Maximilian Hils
3cb87f5a2f split tls_handshake hook into client/server and success/fail variants 2021-11-22 10:23:21 +01:00
Maximilian Hils
a72f61ef57 Merge remote-tracking branch 'origin/main' into ignore-after-clienthello 2021-11-22 09:54:08 +01:00
shindexro
9a469806eb
quote argument of view.flows.resolve (#4910)
* Fix #4902

* Update type signature

* Switch to None check

* Fix spacing

* Quote view.flows.resolve argument

* Switch to call_strings
2021-11-19 13:04:20 +01:00
shindexro
888ce66f90
Correct flow-detail documentation (#4909)
* Fix #4902

* Update type signature

* Switch to None check

* Fix spacing

* Skip URL shortening when flow-detail is 0

* Sync docs with implementation

* Update URL shortening test
2021-11-18 08:26:05 +01:00
shindexro
39fa242e25
fix crash when invoking replay.server.count from console (#4905)
fix #4902
2021-11-17 16:53:43 +00:00
Peter Hoffmann
260fc68211
Fix #4876 Don't do CONNECT on plaintext HTTP replays via upstream (#4882)
* Replays via upstream also need to comply with upstream handling

* Adjusted test for HTTP upstream replay which should NOT do a CONNECT

* Added Changelog

* Test for replay https pver upstream with CONNECT

* Proxy requests use full URL with host & port

* Finally remove some prints

* lint!

Co-authored-by: Maximilian Hils <github@maximilianhils.com>
2021-10-31 20:23:04 +00:00
mame82
3a884ceb8f
gRPC packed repeated encoding (#4872)
* Replaced Kaitai protobuf parser with custom implementation, to prepare packed data handling

* Clamp varint size for bool conversion to 64bit, to satify tests

* moved WireParser into ProtoParser

* preserve work state

* Full packed support

* noc changes to example addon

* Adjusted test regex for ValueError

* Do not try to unpack fields twice
2021-10-21 13:20:13 +02:00
Maximilian Hils
79f464bc78
ignore superfluous newlines after HTTP CONNECT, fix #4870 (#4871) 2021-10-19 11:25:22 +00:00
mame82
8b6345e4a7
gRPC view: Adjusted parsing strategy for deprecated protobuf fields (#4858)
* Adjusted parsing strategy for deprecated protobuf fields

* Added test comment

* adjusted comment to reflect correct field IDs
2021-10-13 13:16:49 +02:00
mame82
480052f58b
Grpc contentview (#4851)
* Partial gRPC contentview prototype, not linted, no tests, not as add-on

* Linted (flake8)

* Save dev state

* Rewrote of protobuf parser, use decoding strategy, reduced rendered data. Parser uses  generators

* minor cleanup

* fix: preferred encoding was provided as function instead of value

* flake8: line length

* Backlinked message tree objects, temporary debug out

* Partial implementation of gRPC definitions. Save state to fix a cras (data invalidate in edit mode)

* hack: deal with missing exception handling for generator based content views

* gRPC/Protoparser descriptions (with test code)

* replaced manual gzip decoding with mitmproxy.net.encoding.decode

* Refactored typing imports

* Reafctoring

* distinguish request vs response definitions, separate view config from parser config

* Code cleaning, moved customized protobuf definitions to example addon

* final cleanup

* changelog

* Stubs for tests

* Fixed render_riority of addon example

* Started adding tests

* Work on tests

* mypy

* Added pseudo encoder to tests, to cover special decodings

* Example addon test added

* finalized tests, no 100 percent coverage possible, see comments un uncovered code

* minor adjustments

* fixup tests

* Typos

Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2021-10-12 13:32:56 +02:00
Maximilian Hils
aad92c9d5a
Merge pull request #4847 from mhils/flowfilter
Flowfilter Improvements
2021-10-09 18:38:56 +02:00
Maximilian Hils
e3640801a5 lint! 2021-10-09 18:34:29 +02:00
Maximilian Hils
25bdf2f9d8 web: update filters 2021-10-08 18:39:22 +02:00
Maximilian Hils
c43a2ef8dc improve flowfilter api: raise on invalid input, add ~all 2021-10-08 17:45:47 +02:00
Maximilian Hils
adfccb90a5 minor fixes 2021-10-08 17:36:16 +02:00
Brad Dixon
9346002e0f
Add client_playback_concurrency option (#4842)
* nowait

* docs, tests, flake8

* we ideally support other values in the future

Co-authored-by: Maximilian Hils <github@maximilianhils.com>
2021-10-08 13:08:36 +02:00
Maximilian Hils
aa2f935dbb
Don't allow server.via change for live connections (#4841)
* don't allow `server.via` change for live connections

* return early if no tls context was set
2021-10-05 21:19:51 +02:00
Thomas Kriechbaumer
6be24f452e
Merge pull request #4810 from mhils/h2-fuzzing
Improve h2 Fuzzing Setup
2021-10-05 21:15:57 +02:00
Maximilian Hils
0f5b8c88af don't allow server.via change for live connections 2021-10-05 19:45:52 +02:00
Maximilian Hils
3cb89069b9 fix compat with upcoming cryptography release 2021-10-05 08:19:37 +02:00
Maximilian Hils
4a3fefdf25 don't crash when sending content-length+transfer-encoding 2021-09-28 18:31:08 +02:00
Matthew Hughes
60a056a2d8 Don't set 'content-length' with 'transfer-encoding'
When updating the response content for a response, avoid adding the
'content-length' header if the response contains a 'transfer-encoding'
header, from the spec [1]:

> When a message does not have a Transfer-Encoding header field, a
Content-Length header field can provide the anticipated size, as a
decimal number of octets, for a potential payload body

Note the 'transfer-encoding' header is not used with HTTP/2

https://httpwg.org/specs/rfc7230.html#header.content-length
2021-09-28 18:31:08 +02:00
Maximilian Hils
7e24e77ac4
improve handling of flows with invalid content-lengths (#4819) 2021-09-23 18:03:52 +00:00
Matthew Hughes
7b4e219c4e
mitmweb: handle {en,de}coding on server-side (#4811)
* mitmweb: handle {en,de}coding on server-side

Handle this server-side rather than passing the message content encoding
details back when fetching flow content. If {en,de}coding fails, return
the raw request contents.

This addresses https://github.com/mitmproxy/mitmproxy/issues/4809

* fix typo

Co-authored-by: Maximilian Hils <github@maximilianhils.com>
2021-09-23 15:55:43 +02:00
Maximilian Hils
a124b1eceb improve h2 fuzzing setup 2021-09-16 11:55:37 +02:00
Maximilian Hils
b41416b729
Merge pull request from GHSA-22gh-3r9q-xf38
This commit makes mitmproxy hard-fail when it encounters any attempts
at request/response smuggling.

For details, see https://github.com/mitmproxy/mitmproxy/security/advisories/GHSA-22gh-3r9q-xf38
2021-09-16 11:12:59 +02:00
Matthew Hughes
d9cac6fbcd
Make Cert.not{before,after} timezone aware (#4805)
I noticed when running tests the output of
`web/src/js/__tests__/ducks/_tflow.ts` would change depending on how I
set my timezone, e.g.

    $ TZ=America/Los_Angeles pytest --quiet \
        test/mitmproxy/tools/web/test_app.py >/dev/null \
        && grep --extended-regexp 'not(after|before)' web/src/js/__tests__/ducks/_tflow.ts
                "notafter": 2235132207,
                "notbefore": 1604415807,
    $ TZ=Asia/Tokyo pytest --quiet \
        test/mitmproxy/tools/web/test_app.py >/dev/null \
        && grep --extended-regexp 'not(after|before)'  web/src/js/__tests__/ducks/_tflow.ts
                "notafter": 2235074607,
                "notbefore": 1604354607

It looks like this is because the `cert_to_json` function simply calls
`timestamp` the `datetime` object from
`x509.Certificate.not_valid_before`, however, this `datetime` object is
not timestamp aware, from the docs [1]:

> A naïve datetime representing the beginning of the validity period for
the certificate in UTC

So when serializing to JSON, first convert the `datetime` to UTC then
call `timestamp`.

A test was added by inspecting one of the test certs with:

    $ openssl x509 -in test/mitmproxy/net/data/text_cert_2 -text

Extracting the date and asserting on that.

The corresponding test has also been re-run so that `_tflow.ts` was
regenerated with it's correct value. Snapshots were also updated via:

    $(npm bin)/jest --updateSnapshot

[1] https://cryptography.io/en/latest/x509/reference/#cryptography.x509.Certificate.not_valid_after
2021-09-16 10:53:36 +02:00
Maximilian Hils
7fd887a553 move tls hook data to mitmproxy.tls 2021-09-04 16:37:39 +02:00
Maximilian Hils
bdf4e31c58 move ClientHello to mitmproxy.tls 2021-09-04 16:10:39 +02:00
Maximilian Hils
9f39e2f387 tests++ 2021-09-04 16:03:06 +02:00
Maximilian Hils
98a3e33477 tls: add tls_handshake, ignore-after-clienthello
this fixes #4702
2021-09-04 00:24:41 +02:00
Maximilian Hils
0437d2935e make flake8 happy 2021-09-04 00:23:45 +02:00
Maximilian Hils
75ccd44c42 add more benchmark/memoryleak tooling 2021-09-03 21:34:53 +02:00
Maximilian Hils
d5bba9878b
Merge pull request #4780 from mhils/socks5-auth
Support SOCKS5 Authentication
2021-08-27 10:30:54 +02:00
Maximilian Hils
f9ffe8279d socks5: use correct version for auth negotiation 2021-08-27 10:25:31 +02:00
Maximilian Hils
f8826b29a2 cache socks5 auth for entire connection 2021-08-25 17:23:49 +02:00
Maximilian Hils
73d809a4c7 refactor proxyauth addon
the previous version was difficult to read,
this is hopefully better now.
2021-08-25 17:23:49 +02:00
Maximilian Hils
a3eca0b859 socks5 upstream auth: use proxyauth option 2021-08-25 17:23:49 +02:00
Maximilian Hils
c0fd6cfc09
Merge pull request #4779 from mhils/mitmweb-tcp
web: add support for viewing tcp flows
2021-08-25 13:35:37 +02:00
Maximilian Hils
550e1a4ab3
fix #3506 (#4777) 2021-08-23 19:35:31 +00:00
Maximilian Hils
fca21cbf72
GSoC Mitmweb Improvements (#4763)
* Add hover menu to FlowRow

* fix linting issue

* fix linting issue

* apply suggested changes

* add dropdown submenu

* apply requested change

* replace dropdown with popper.js-based implementation

* implement actions to  the menu

* clean FlowRow arguments

* web: major upgrades

This commit brings a bunch of under-the-hood mitmweb improvements:

 - migrate large parts of the codebase to typescript
 - introduce modern react testing conventions
 - vendor react-codemirror to silence warnings
 - use esbuild for both bundles and tests
 - move from yarn to npm
 - various fixes across the board

* web: fix options ducks

* add command bar to mitmweb

* fix app.py

* fix web test failure

* include command.js to commit

* fix optionsSpec.js

* transfer the argument suggestion into popup, show available commands, highlight currently typing argument

* fix lintings issue and test

* include package-lock.json

* add response to fetchMock

* update test

* update obsolete snapshot

* display help, description and add tab autocompletion

* change command history into mitmproxy's one

* move available commands popover to command help

* fix bug related to tab completion

* change CommandBar component into typescript, autoscroll overflowed result to bottom

* convert components in common and ContentView folder into typescript, and modified test

* fix bug of not dispatching when type in the field in FilterInput component

* make formatTimeStamp function return UTC time when updating snapshot in jest

* bind "this" to functions in CodeMirror

* convert components in FlowView, Header, Modal, ValueEditor into typescript

* applied requested changes

* [web] update fontawesome

* view: add id-based flow selection

* add raw `export` command

* contentviews: ignore empty data

* console: fix cert validity order

* major mitmweb upgrades

* fix tests

* fix web tests

* minor fixes

* please the coverage gods

* web: test coverage++, adjust commandbar

* web: update dependencies

* fix tests

* web: add backend tests

* web: convert everything to TypeScript

* web: fix HTTP trailer update

* tests++

Co-authored-by: zokutyou2@gmail.com <zokutyou2@gmail.com>
2021-08-23 11:54:54 +02:00
Maximilian Hils
d55aa7ea42 tests++ 2021-08-23 11:48:27 +02:00
Maximilian Hils
7219284d44 web: fix HTTP trailer update 2021-08-23 11:22:11 +02:00
Jinjie Zhang
0c366f6436 feat: add test script for socks5auth 2021-08-23 16:55:57 +08:00
Maximilian Hils
81c911345b
improve TLS version mismatch error, fix #4758 (#4772) 2021-08-23 07:15:56 +00:00
Maximilian Hils
03606fb0c0 fix tests 2021-08-20 18:49:44 +02:00
Maximilian Hils
2945ba925b please the coverage gods 2021-08-20 15:10:14 +02:00
Maximilian Hils
8561a620c8 minor fixes 2021-08-19 19:15:51 +02:00
Maximilian Hils
3ad4949c0e fix web tests 2021-08-19 19:15:51 +02:00