Commit Graph

2451 Commits

Author SHA1 Message Date
mame82
8b6345e4a7
gRPC view: Adjusted parsing strategy for deprecated protobuf fields (#4858)
* Adjusted parsing strategy for deprecated protobuf fields

* Added test comment

* adjusted comment to reflect correct field IDs
2021-10-13 13:16:49 +02:00
mame82
480052f58b
Grpc contentview (#4851)
* Partial gRPC contentview prototype, not linted, no tests, not as add-on

* Linted (flake8)

* Save dev state

* Rewrote of protobuf parser, use decoding strategy, reduced rendered data. Parser uses  generators

* minor cleanup

* fix: preferred encoding was provided as function instead of value

* flake8: line length

* Backlinked message tree objects, temporary debug out

* Partial implementation of gRPC definitions. Save state to fix a cras (data invalidate in edit mode)

* hack: deal with missing exception handling for generator based content views

* gRPC/Protoparser descriptions (with test code)

* replaced manual gzip decoding with mitmproxy.net.encoding.decode

* Refactored typing imports

* Reafctoring

* distinguish request vs response definitions, separate view config from parser config

* Code cleaning, moved customized protobuf definitions to example addon

* final cleanup

* changelog

* Stubs for tests

* Fixed render_riority of addon example

* Started adding tests

* Work on tests

* mypy

* Added pseudo encoder to tests, to cover special decodings

* Example addon test added

* finalized tests, no 100 percent coverage possible, see comments un uncovered code

* minor adjustments

* fixup tests

* Typos

Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2021-10-12 13:32:56 +02:00
Maximilian Hils
aad92c9d5a
Merge pull request #4847 from mhils/flowfilter
Flowfilter Improvements
2021-10-09 18:38:56 +02:00
Maximilian Hils
e3640801a5 lint! 2021-10-09 18:34:29 +02:00
Maximilian Hils
25bdf2f9d8 web: update filters 2021-10-08 18:39:22 +02:00
Maximilian Hils
c43a2ef8dc improve flowfilter api: raise on invalid input, add ~all 2021-10-08 17:45:47 +02:00
Maximilian Hils
adfccb90a5 minor fixes 2021-10-08 17:36:16 +02:00
Brad Dixon
9346002e0f
Add client_playback_concurrency option (#4842)
* nowait

* docs, tests, flake8

* we ideally support other values in the future

Co-authored-by: Maximilian Hils <github@maximilianhils.com>
2021-10-08 13:08:36 +02:00
Maximilian Hils
aa2f935dbb
Don't allow server.via change for live connections (#4841)
* don't allow `server.via` change for live connections

* return early if no tls context was set
2021-10-05 21:19:51 +02:00
Thomas Kriechbaumer
6be24f452e
Merge pull request #4810 from mhils/h2-fuzzing
Improve h2 Fuzzing Setup
2021-10-05 21:15:57 +02:00
Maximilian Hils
0f5b8c88af don't allow server.via change for live connections 2021-10-05 19:45:52 +02:00
Maximilian Hils
3cb89069b9 fix compat with upcoming cryptography release 2021-10-05 08:19:37 +02:00
Maximilian Hils
4a3fefdf25 don't crash when sending content-length+transfer-encoding 2021-09-28 18:31:08 +02:00
Matthew Hughes
60a056a2d8 Don't set 'content-length' with 'transfer-encoding'
When updating the response content for a response, avoid adding the
'content-length' header if the response contains a 'transfer-encoding'
header, from the spec [1]:

> When a message does not have a Transfer-Encoding header field, a
Content-Length header field can provide the anticipated size, as a
decimal number of octets, for a potential payload body

Note the 'transfer-encoding' header is not used with HTTP/2

https://httpwg.org/specs/rfc7230.html#header.content-length
2021-09-28 18:31:08 +02:00
Maximilian Hils
7e24e77ac4
improve handling of flows with invalid content-lengths (#4819) 2021-09-23 18:03:52 +00:00
Matthew Hughes
7b4e219c4e
mitmweb: handle {en,de}coding on server-side (#4811)
* mitmweb: handle {en,de}coding on server-side

Handle this server-side rather than passing the message content encoding
details back when fetching flow content. If {en,de}coding fails, return
the raw request contents.

This addresses https://github.com/mitmproxy/mitmproxy/issues/4809

* fix typo

Co-authored-by: Maximilian Hils <github@maximilianhils.com>
2021-09-23 15:55:43 +02:00
Maximilian Hils
a124b1eceb improve h2 fuzzing setup 2021-09-16 11:55:37 +02:00
Maximilian Hils
b41416b729
Merge pull request from GHSA-22gh-3r9q-xf38
This commit makes mitmproxy hard-fail when it encounters any attempts
at request/response smuggling.

For details, see https://github.com/mitmproxy/mitmproxy/security/advisories/GHSA-22gh-3r9q-xf38
2021-09-16 11:12:59 +02:00
Matthew Hughes
d9cac6fbcd
Make Cert.not{before,after} timezone aware (#4805)
I noticed when running tests the output of
`web/src/js/__tests__/ducks/_tflow.ts` would change depending on how I
set my timezone, e.g.

    $ TZ=America/Los_Angeles pytest --quiet \
        test/mitmproxy/tools/web/test_app.py >/dev/null \
        && grep --extended-regexp 'not(after|before)' web/src/js/__tests__/ducks/_tflow.ts
                "notafter": 2235132207,
                "notbefore": 1604415807,
    $ TZ=Asia/Tokyo pytest --quiet \
        test/mitmproxy/tools/web/test_app.py >/dev/null \
        && grep --extended-regexp 'not(after|before)'  web/src/js/__tests__/ducks/_tflow.ts
                "notafter": 2235074607,
                "notbefore": 1604354607

It looks like this is because the `cert_to_json` function simply calls
`timestamp` the `datetime` object from
`x509.Certificate.not_valid_before`, however, this `datetime` object is
not timestamp aware, from the docs [1]:

> A naïve datetime representing the beginning of the validity period for
the certificate in UTC

So when serializing to JSON, first convert the `datetime` to UTC then
call `timestamp`.

A test was added by inspecting one of the test certs with:

    $ openssl x509 -in test/mitmproxy/net/data/text_cert_2 -text

Extracting the date and asserting on that.

The corresponding test has also been re-run so that `_tflow.ts` was
regenerated with it's correct value. Snapshots were also updated via:

    $(npm bin)/jest --updateSnapshot

[1] https://cryptography.io/en/latest/x509/reference/#cryptography.x509.Certificate.not_valid_after
2021-09-16 10:53:36 +02:00
Maximilian Hils
0437d2935e make flake8 happy 2021-09-04 00:23:45 +02:00
Maximilian Hils
75ccd44c42 add more benchmark/memoryleak tooling 2021-09-03 21:34:53 +02:00
Maximilian Hils
d5bba9878b
Merge pull request #4780 from mhils/socks5-auth
Support SOCKS5 Authentication
2021-08-27 10:30:54 +02:00
Maximilian Hils
f9ffe8279d socks5: use correct version for auth negotiation 2021-08-27 10:25:31 +02:00
Maximilian Hils
f8826b29a2 cache socks5 auth for entire connection 2021-08-25 17:23:49 +02:00
Maximilian Hils
73d809a4c7 refactor proxyauth addon
the previous version was difficult to read,
this is hopefully better now.
2021-08-25 17:23:49 +02:00
Maximilian Hils
a3eca0b859 socks5 upstream auth: use proxyauth option 2021-08-25 17:23:49 +02:00
Maximilian Hils
c0fd6cfc09
Merge pull request #4779 from mhils/mitmweb-tcp
web: add support for viewing tcp flows
2021-08-25 13:35:37 +02:00
Maximilian Hils
550e1a4ab3
fix #3506 (#4777) 2021-08-23 19:35:31 +00:00
Maximilian Hils
fca21cbf72
GSoC Mitmweb Improvements (#4763)
* Add hover menu to FlowRow

* fix linting issue

* fix linting issue

* apply suggested changes

* add dropdown submenu

* apply requested change

* replace dropdown with popper.js-based implementation

* implement actions to  the menu

* clean FlowRow arguments

* web: major upgrades

This commit brings a bunch of under-the-hood mitmweb improvements:

 - migrate large parts of the codebase to typescript
 - introduce modern react testing conventions
 - vendor react-codemirror to silence warnings
 - use esbuild for both bundles and tests
 - move from yarn to npm
 - various fixes across the board

* web: fix options ducks

* add command bar to mitmweb

* fix app.py

* fix web test failure

* include command.js to commit

* fix optionsSpec.js

* transfer the argument suggestion into popup, show available commands, highlight currently typing argument

* fix lintings issue and test

* include package-lock.json

* add response to fetchMock

* update test

* update obsolete snapshot

* display help, description and add tab autocompletion

* change command history into mitmproxy's one

* move available commands popover to command help

* fix bug related to tab completion

* change CommandBar component into typescript, autoscroll overflowed result to bottom

* convert components in common and ContentView folder into typescript, and modified test

* fix bug of not dispatching when type in the field in FilterInput component

* make formatTimeStamp function return UTC time when updating snapshot in jest

* bind "this" to functions in CodeMirror

* convert components in FlowView, Header, Modal, ValueEditor into typescript

* applied requested changes

* [web] update fontawesome

* view: add id-based flow selection

* add raw `export` command

* contentviews: ignore empty data

* console: fix cert validity order

* major mitmweb upgrades

* fix tests

* fix web tests

* minor fixes

* please the coverage gods

* web: test coverage++, adjust commandbar

* web: update dependencies

* fix tests

* web: add backend tests

* web: convert everything to TypeScript

* web: fix HTTP trailer update

* tests++

Co-authored-by: zokutyou2@gmail.com <zokutyou2@gmail.com>
2021-08-23 11:54:54 +02:00
Maximilian Hils
d55aa7ea42 tests++ 2021-08-23 11:48:27 +02:00
Maximilian Hils
7219284d44 web: fix HTTP trailer update 2021-08-23 11:22:11 +02:00
Jinjie Zhang
0c366f6436 feat: add test script for socks5auth 2021-08-23 16:55:57 +08:00
Maximilian Hils
81c911345b
improve TLS version mismatch error, fix #4758 (#4772) 2021-08-23 07:15:56 +00:00
Maximilian Hils
03606fb0c0 fix tests 2021-08-20 18:49:44 +02:00
Maximilian Hils
2945ba925b please the coverage gods 2021-08-20 15:10:14 +02:00
Maximilian Hils
8561a620c8 minor fixes 2021-08-19 19:15:51 +02:00
Maximilian Hils
3ad4949c0e fix web tests 2021-08-19 19:15:51 +02:00
Maximilian Hils
0585e690c4 fix tests 2021-08-19 10:53:46 +02:00
Maximilian Hils
b320c6aa14 Merge remote-tracking branch 'origin/main' into gsoc 2021-08-18 17:39:01 +02:00
Maximilian Hils
65aa0a6ef5 major mitmweb upgrades 2021-08-18 17:38:22 +02:00
Maximilian Hils
f9b63e973e
Remove asyncio event loop workaround for tornado (#4762)
* remove asyncio event loop workaround for tornado

* Update CHANGELOG.md
2021-08-18 14:12:39 +00:00
Maximilian Hils
4fb3e4c321 treat encoding names case-insensitively, fix #4735
Co-authored-by: Mattwmaster58 <mattwmaster58@gmail.com>
2021-08-10 08:11:34 +02:00
Maximilian Hils
fb53df0cc1
expose full ClientHello object in tls_clienthello hook (#4744) 2021-08-09 15:41:05 +00:00
Maximilian Hils
fcec63eebc fix tnetstread crash found via fuzzing 2021-08-08 16:12:37 +02:00
Maximilian Hils
226e31290c Merge branch 'main' into gsoc 2021-08-04 19:16:31 +02:00
Maximilian Hils
4abd00afab
Merge pull request #4731 from aaron-tan/improve-render-size
Improve rendering of size column
2021-08-04 17:26:20 +02:00
Maximilian Hils
6aa9d8658f size column: add a bit more precision 2021-08-04 17:14:42 +02:00
Maximilian Hils
53a916e39b
fix websocket regression introduced in 7.0.1 (#4733) 2021-08-04 12:59:48 +00:00
Aaron Tan
4266c4852c Improve rendering of size column 2021-08-04 21:02:43 +10:00
Maximilian Hils
416e5046bc add proxy-auth header for plain http requests, fix #4728 2021-08-03 17:02:49 +02:00