The .configure and .tick events can be called before the server is running. If
an addon calls ctx.master.shutdown() at this point, the server hangs forever
waiting on __is_shutdown_set. This patch fixes this by starting off with it
already set.
- Replace options.console_eventlog with options.console_layout
- This can be "single", "vertical" and "horizontal"
- At the base of the primary pane is the flowlist. At the base of the secondary
pane is the event log.
- Any of the other primary windows can be opened in each of the panes.
For now, I've bound "-" to the flow layout switch, "shift tab" to the layout
pane switch, and "P" to open the currently focused flow in whichever pane
you're in. These are just temporary - we'll reassess the default bindings
carefully once the keybindings work is complete.
- Action bar now correctly gets the palette background
- Viewing connection details for a killed flow no longer crashes
- Editing options with selections no longer crashes
- All top-level window objects are now persistent. It turns out that urwid
keeps a reference to these even after they're no longer used, so they were
leaking before.
- Revamp editors to work with console commands, and start reworking bindings
for flowview.
And use these commands to remove the last hard-coded keybindings from flow
list. This means the flow list is now 100% command-driven, which is very
exciting.
Use this to simplify meta-commands in console, and to create a console_choose
command that prompts the user for a choice, and then executes a command with
variable substitution.
Also add a "raw" export format.
The Python and Locust exports are hard to maintain, their tests are extremely
brittle, they didn't have full test coverage, and are by my guess very rarely
used. I feel the Locust export should certainly be an externally maintained
addon. The Python/requests export can come back if someone cares enough, and it
can be structured in a way we can maintain.
This PR introduces the cuts addon, a flow dissector that allows you to select
and operate on specific components of flows. It also adds the first consumer
for cuts - the cuts.save command.
Save the content of the focus to /tmp/foo:
cuts.save s.content|@focus /tmp/foo
Save the URL and response content-type headers for all flows currently shown to
file, comma-separated, one flow per line:
cuts.save s.url,q.header[content-type]|@focus /tmp/foo
We also use this to replace the body save shortcut in the console flowlist.
- Add client.replay [flows], client.replay.stop
- Add server.replay [flows], server.replay.stop
- The corresponding options for file loading are only read on startup, further
changes are ignored. In interactive contexts, replay is started with the
commands, not through option changes.
- Deprecate flow.replay, use replay.client instead
This patch:
- Introduces a core update() event that should be invoked whenever flows are
changed outside of the normal lifecycle.
- Extend view.resolve to know about @all, which matches all flows in the view.
- Add a core flow.resume comand, which resumes flows and broadcasts an update event.
- Define flow list bindings for:
A -> flow.resume @all
a -> flow.resume @focus
d -> view.remove @focus
z -> view.remove @all
This has the same urgency as "info", but also signals to interactive tools that
the user's attention should be drawn to the output, even if they're not looking
at the event log. In the console app, this means the message appears in the
status bar with a timeout.
Our first user-facing command. The following commands do the obvious things:
save.file @marked /tmp/flows
save.file @focus /tmp/flows
save.file @hidden /tmp/flows
save.file "~m get" /tmp/flows
This is our first built-in command, which will be used by very many other
commands.
Also add a --commands option to dump all commands, analogous to --options.
- Fix some loading sequence bugs affecting command-line script invocation
- Allow addons to over-ride existing options (with a warning). We need this for
reloading.
- Convert har_dump to new-style arguments, fix and re-instate its test suite.
- Covnert miscelaneous other exmples to new-style args.
Many addons currently save options on configure(), either as individual options
or sometimes by saving the entire options object. The current options should
simply be available on the ctx object, simplifying state management for addons
considerably.
- Remove the watchdog dependency. We now just stat the script file every 2
seconds to check for an updated mtime.
- Further solidify our script testing, and in particular make the example tests
nicer. These should exemplify how we want users to test their own addon
scripts. More work on addon testing to follow.
- Addons now nest, which means that addons can manage addons. This has a number
of salutary effects - the scripts addon no longer has to poke into the global
addons list, we no longer have to replace/remove/boot-outof parent addons when
we load scripts, and this paves the way for making our top-level tools into
addons themselves.
- All addon calls are now wrapped in a safe execution environment where
exceptions are caught, and output to stdout/stderr are intercepted and turned
into logs.
- We no longer support script arguments in sys.argv - creating an option
properly is the only way to pass arguments. This means that all scripts are
always directly controllable from interctive tooling, and that arguments are
type-checked.
For now, I've disabled testing of the har dump example - it needs to be moved
to the new argument handling, and become a class addon. I'll address that in a
separate patch.
Mypy doesn't support recursive types yet, so we can't properly express
TSerializable nested structures. For now, we just disable type checking in the
appropriate locations.
https://github.com/python/mypy/issues/731
In some places, this involved removing type declarations where our types were
terminally confused. The grideditor specifically needs a cleanup and
restructure.
- Move default client ciphers - no need to have a 10-line string default.
- Add some style guidance on options help
- Be consistent about reflowing options help
We get little benefit from our mypy QA checks at the moment, because we skip
imports. This patch is what's needed to make mypy succeed with imports on a
single file: master.py
It also updates mypy to the current version, and enables a QA check.
Mypy bugs I encountered:
dict.update with kwargs not supported:
https://github.com/python/mypy/issues/1031
property setters and getters must be adjacent:
https://github.com/python/mypy/issues/1465
We get little benefit from our mypy QA checks at the moment, because we skip
imports. This patch is what's needed to make mypy succeed with imports on a
single file: master.py
It also updates mypy to the current version, and enables a QA check.
Mypy bugs I encountered:
dict.update with kwargs not supported:
https://github.com/python/mypy/issues/1031
property setters and getters must be adjacent:
https://github.com/python/mypy/issues/1465
This patch implements the lightweight event system I propose in #2144, adds a
custom event "processing_complete" that is triggered after file read, client
replay and server replay, and introduces a KeepServing addon to handle this for
mitmdump.
This patch moves the final pieces of master functionality into addons.
- Add a ReadFile addon to read from file
- Add a separate ReadStdin addon to read from stdin, only used by mitmdump
- Remove all methods that know about io and serialization from master.Master
Attributes on options share a namespace with options themselves. It's getting
too crowded on our Options object, so let's shift some obvious stuff into the
module.
- Add an options parameter to the start() event. This is to be used by addons
on startup to add custom options.
- Add a running() event that is called once the proxy is up and running.
- With the new paradigm we can't log during master __init__, so add a tiny
termstatus addon to print proxy status to terminal once we're running.
name, type, value matches natural language ($x is a bool with a default value of $y)
and also matches the python type annotation order of name: type = value
We now have one option "proxyauth". If this is "any", we accept any
credentials, if it starts with an @ it's treated as a path to an htpasswd file,
if it is of the form username:password it's a single-user credential.
- Regularise presentation and wording
- Help is mandatory for all options
- Auto-generate wording to say that sequence options can be passed multiple
times on the command-line
We now have --set, so only options that really deserve it get a dedicated flag.
I'm inclined to strip this back even more. Feel free to argue that YOUR
favourite option deserves special treatment here.
The --set option is a universal flag for setting options. Some examples:
Turn on a boolean:
mitmdump --set onboarding=false
Add a value to a sequence:
mitumdupm --set setheaders=/foo/bar/voing
Zero a sequence:
mitumdupm --set setheaders
We now have:
--mode regular (the default)
--mode transparent
--mode socks5
--mode reverse:SPEC
--mode upstream:SPEC
Where SPEC is a host specification.
Start dealing with corner cases:
- Sizes are always stored in options as strings
- Add a new core addon that's responsible for verifying settings that don't
belong to an addon
- Add a _processed scratch space on the Options object for processed core
values to be stored in. This is pretty dirty, but less dirty than re-parsing
values every time. We'll come up with something better down the track.
Moves all integer options apart from a few tricky ones like verbosity over to
auto generation. Also add a metavar argument to parser generation to support
this.
This commit:
- Adds a help field to options
- Adds a function to generate parser definitions from options
- Uses this to migrate all boolean flags over to the new system
- Makes all booleans consistently follow the --foo/--not-foo convention
There are a number of things left to be done here:
- Argparse doesn't give us a nice way to format --foo --not-foo help. Click
does, and moving to click is a goal down the track.
- For now, we remove all short aliases. I want to re-evaluate these
systematically once we have the new structure in place.
