Dispense with the loose parsing of client requests that we had before. We now
have service modes ("proxy" and "reverse proxy" for now), and we only accept
requests that are appropriate for the mode we're in.
We'll use this in a number of situations. First, we'll soon have response
streaming that directly pipes responses to clients. These will be content-less
from mitmproxy's perspective. Second, we'll be growing new events that fire
after headers are received, but before content is read.
This commit adds:
- A WSGI App adapter for mitmproxy
- An app registry in the proxy instance that lets us link WSGI apps with
(hostname, port) combinations.
- Fixes for a number of bugs discovered while creating this feature.
- Capture the remote SSL certificate
- Expose the remote cert as an attribute on Response
- Expand the certutils.SSLCert interface to expose more cert info
This initiates a connection to the server to obtain certificate information to
generate interception certificates. At the moment, the information used is the
Common Name, and the list of Subject Alternative Names.
- Retain the specification from the Host header as a Request's description.
- Expand upstream proxy specifications to include the scheme. We now say https://hostname:port
- Move the "R" revert keybinding to "v" to make room for a reverse proxy
binding that matches the command-line flag.
Many editors make it hard save a file without a terminating newline on the last
line. When editing message bodies, this can cause problems. For now, I just
strip the newlines off the end of the body when we return from an editor.
- Don't fail to identity encoding when an unknown encoding is specified.
- Don't constrain encodings. I want to try to modify traffic as little as
possible by default.
- When decoding, delete content-encoding header rather than set it to "identity"
- Refuse to decode/encode when there is an existing but unknown
content-encoding header.
This is a common task in pentesting scenarios. This commit adds the following
functions:
utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
Since OpenSSL doesn't let us set certificate start times in the past, the
client and proxy machine time must be synchronized, or the client might reject
the certificate. We can bodgy over small discrepancies by waiting a few seconds
after a new certificate is generated (i.e. the first time an SSL domain is contacted).
Make this a configurable option, and turn it off by default.
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
We now create three different files in the .mitmproxy directory when a dummy CA
is made:
mitmproxy-ca.pem - the CA, including private key
mitmproxy-ca-cert.p12 - A pkcs12 version of the certificate, for distribution to Windows.
mitmproxy-ca-cert.pem - A PEM version of the certificate, for distribution to everyone else.
- Computing the view of a large body is expensive, so we introduce an LRU cache
to hold the latest 20 results.
- Use ListView more correctly, passing it individual urwid.Text snippets,
rather than a single large one. This hugely improves render time.
This removes all headers that might cause a server to return 304-not-modified.
For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
If msg is Unicode, the proto string is automatically promoted to Unicode. If
the proto string is promoted to Unicode, then the FMT interpolation is also
done in Unicode. If this happens, then binary data in content will cause an
exception.
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
- Move option parsing utiliities to proxy.py
- Don't have a global config object. Pass it as an argument to ProxyServer.
- Simplify certificate generation logic.
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.
- Split CA and cert generation into separate functions.
- Generation methods provide an error return when generation fails.
- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
We use the ClientConnection object to tie requests, responses and errors
together. This is an HTTP 1.0 assumption, but we can fix it by just making
copies of the connection object when we handle multiple requests.
Adds support for chunked transfer encoding, and a couple other minor
protocol corrections.
Improve HTTP support
- Support intercepted requests with Host header
- Support HEAD requests proper
- Support any HTTP method including extensions, not just a couple known ones
Support expect: 100-continue and 100 Continue messages
Persistent client connections
Generalize ServerConnection a bit in preparation for keep-alive support
Correct HTTP status codes on errors forwarding the request
