We've had a perpetual sequencing problem with addon startup. Users need to be
able to specify options to addons on the command-line, before addons are
actually loaded. This is only exacerbated with the new async core, where load
order can't be relied on.
This patch introduces deferred options. Options passed with "--set" on the
command line are deferred if they are unknown, and are automatically applied by
the addon manager once matching addons are registered and their options are defined.
We now have two options: block_global blocks global networks, block_private
blocks private networks. The block_global option is true by default, and
block_private is false by default. The addon name is "block" so the options are
correctly prefixed.
Also make option documentation precise, reduce verbosity of logs.
- Instead of listening for a pseudo-event, we periodically check whether client
replay, server replay or file reading is active.
- Adjust server replay not to
use tick.
- Adjust readfile to expose a command to check whether reading is in progress.
Re-design the way client replay works. Before, we would fire up a thread,
replay, wait for the thread to complete, get the next flow, and repeat the
procedure. Now, we have one replay thread that starts when the addon starts,
which pops flows off a thread-safe queue. This is much cleaner, removes the
need for busy tick, and sets the scene for optimisations like server connection
reuse down the track.
The granularity of mtime is surprisingly bad. Make the tests more robust
against this, and promote has_log back to a public method, now that we have a
few legitimate examples.
The tick event is a nasty compromise, left over from when we didn't have an
event loop. This is the first patch in a series that explores moving our
built-in addons to managing coroutines on the eventloop directly for periodic
tasks.
- Ignore the NotImplementedError raised by add_signal_handler on Windows.
- Entrypoints return an integer exit code, or None. Adjust our type annotations
and code to suit.
Fixes#3061
- The benchmark addon now manages setting up and tearing down the backend and
traffic processes itself.
- Use wrk instead of hey. I get more consistent results with this tool, and hey
shows a strange tail-latency bump that seems artificial.
- Make termination behaviour simpler. The bencmark revealed a bug where .done
events were not called if the proxy was shut down by an addon.
There are a few reasons for this. First, logs are now async, and can be called
at any time. Second, the event loop is thread local, so there can only ever be
one master per thread. These two things together completely obviate the need
for a handler context.
Now that logs are async, using this call is almost always a mistake. Signal
this by making it semi-private. The method may go away entirely down the track.
Logs are now asynchronous, with a log entry pushed onto the event loop for
handling. To support this, the test mechanism grows an await_log method that
waits for a log entry to appear.
Also silence asyncio logs. We sometimes end up with messages on the queue that
need to be ignored when the proxy shuts down, and asyncio complains loudly
about this.
We now acquire the event loop through asyncio.get_event_loop, avoiding having
to pass the loop explicity in a bunch of places. This function does not return
the currently running loop from within coroutines in versions of Python prior
to 3.6.
This is a preparatory patch that paves the way to consolidating our core
options in the core addon. It amalgamates the core_option_validation and core
addons, prepares the test suite for a world where options live in core, and
moves over two trivial options as a trial balloon.
From here, things will get harder, but at the end of the process we'll have a
core that's responsive to options.
Split verbosity into termlog_verbosity and console_eventlog_verbosity.
This patch also removes printing to console if there are unknown options in the
command-line. Options now live in separate addons, so having uknown options
remaining is common and expected. We definitely shoould have some other way for
users to see what was ignored so they can catch typos and the like, but that's
a different patch.
As part of this, we zap an un-needed console command and use a boolean toggle
setter instead.
Also remove an ancient regression test that breaks encapsulation.
Also triage options, and categorize them into core options (won't be migrated),
options that are hard to migrate for various reasons, and easy migrations.
This takes the first few steps:
- Extends taddons to make loading addon options easier
- Removes dependencies in the test suite on options in addons
- Tweaks command-line parser autocreation to ignore nonexistent options. This
lets us load common options without over-depending on loaded addons.
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for function names.
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for attributes.
the semantics here were really quite unclear,
now it is hopefully a bit more obvious what's happening.
Once we are Python 3.6+ exclusively, we may consider changing
the signature to accept a (order-preserving) dict instead of a list.
This now just sets a kill reply instead of committing directly.
First, this seems like the more sane thing to do.
Second, we have an iffy race condition where we call Reply.commit()
before the addonmanager finishes its invocation, the proxy thread then progresses
and sets a new flow.reply attribute, and the addonmanager then gets confused
when finishing. This commit doesn't fix that, but mitigates it for Flow.kill
which is now committed by the addonmanager.
The type system was scattered over a number of places, making it hard to
follow. This collects all command types in types.py, and completion, validation
and parsing for each type is centralised. We should use the same mechanism for
options.
The types name is valuable, and we have a better use for it in collecting and
exposing types for options and commands.
The coretypes module should probably be split up anyway - it contains a
threading base class, a few container objects, and the defintion of our
serialization protocol. I was tempted to rename it to "uncagegorized" for the
sake of honesty.
The JS test assets depend in a brittle way on the details of the tflow()
utility functions. We shouldn't have to fix JS tests when adjusting these.
Options:
- Manually generate the test assets in a script.
- Define the JS assets without using tflow, so they don't unexpextedly
vary.
- Remove shortcuts for request, response, etc. - we don't need them if we have completion
- Restrict cuts specification to a set of prefixes
- Extend cuts to add a few more items
This represents a command passed as an argument. Also split arguments from
command values themselves, making the command help for meta-commands much
clearer.
This resolves as a string during MyPy checks, but at runtime has an additional
attribute that is a command that returns valid options.
This is very ugly and clumsy, basically because MyPy is super restrictive about
what it accepts as a type. Almost any attempt to construct these types in a
more sophisticated way fails in one way or another. I'm open to suggestions.
A simple addon that starts an instance of Chrome attached to the current
proxy. The instance is isolated in its own user data directory, and addons are
turned off.
Future work:
- I wasn't able to test the Windows executable path - a Windows dev should
confirm this for us.
- In future it would be nice to support other browsers like Firefox.
- always refresh UI after flow is finished (refs #2616)
- count currently active replay
- make replay thread daemonic so that users can exit mitmproxy
if replay hangs. This is not perfect yet, but vastly better
than how it has been.
