This is a common task in pentesting scenarios. This commit adds the following
functions:
utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
Two different strategies here:
- Use a simple heuristic to detect if we're looking at XML data when indent
mode is used. On non-XML data we can hang even on small documents.
- Only view partial data for large bodies. At the moment the cutoff is
100k. I might finetune this later.
- The OpenSSL x509 has no way to explicitly set the notBefore value on
certificates.
- If two systems have the same configured time, it's possible to return a
certificate before the validity start time has arrived.
- We "solve" this by waiting for one second when a certificate is first
generated before returning the cert. The alternative is to rewrite pretty much
all of our certificate generation, a thought too horrible to contemplate.
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
We now create three different files in the .mitmproxy directory when a dummy CA
is made:
mitmproxy-ca.pem - the CA, including private key
mitmproxy-ca-cert.p12 - A pkcs12 version of the certificate, for distribution to Windows.
mitmproxy-ca-cert.pem - A PEM version of the certificate, for distribution to everyone else.
- Computing the view of a large body is expensive, so we introduce an LRU cache
to hold the latest 20 results.
- Use ListView more correctly, passing it individual urwid.Text snippets,
rather than a single large one. This hugely improves render time.
This removes all headers that might cause a server to return 304-not-modified.
For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
Also add the --rheader command-line option to mitmdump to let the user specify
an arbitrary number of significant headers. The default is to treat no headers
as significant.
If this option is passed all requests that are not part of a replayed
conversation are killed. If the option is not passed, such requests are passed
through to the server as usual.
We use a loose hash to match incoming requests with recorded flows. At the
moment, this hash is over the host, port, scheme, method, path and content of
the request. Note that headers are not included here - if we do want to include
headers, we would have to do some work to normalize them to remove variations
between user agents, header order, etc. etc.
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
- Move option parsing utiliities to proxy.py
- Don't have a global config object. Pass it as an argument to ProxyServer.
- Simplify certificate generation logic.
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.
- Split CA and cert generation into separate functions.
- Generation methods provide an error return when generation fails.
- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
Module is in the Public Domain.
I expect to modify and extend this module, so I've imported into main library
rather than contrib. Code has been reformatted to suite our code standard,
tests have been extrated into /tests directory.
External scripts can read a flow, modify it, and then return it to mitmproxy
using a simple API.
The "|" keyboard shortcut within mitmproxy prompts the user for a script.
Also, since BeautifulSoup is so damn slow, print a statusbar message saying
that we're calculating a pretty version of the response. Maybe I should add
hangman or something, becuase on a 200k document this can take ages.
