Grasscutters/mitmproxy
3
0
Fork 0
mirror of https://github.com/Grasscutters/mitmproxy.git synced 2024-11-22 07:08:10 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #751 from mitmproxy/readthedocs

Browse Source 
Move Docs to readthedocs
This commit is contained in:
Maximilian Hils 2015-09-16 02:53:01 +02:00
commit 457025df9a
144 changed files with 2555 additions and 9064 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.travis.yml
View File

@ -15,6 +15,8 @@ matrix:
- debian-sid
packages:
- libssl-dev
- python: 2.7
env: DOCS=1
- python: pypy
- python: pypy
env: OPENSSL=1.0.2
@ -38,7 +40,8 @@ before_script:
- "openssl version -a"
script:
- "nosetests --with-cov --cov-report term-missing"
- "if [ -z \"$DOCS\" ]; then nosetests --with-cov --cov-report term-missing; fi"
- "if [ -n \"$DOCS\" ]; then cd docs && make html; fi"
after_success:
- coveralls
@ -51,7 +54,7 @@ notifications:
on_failure: always
slack:
rooms:
- mitmproxy:YaDGC9Gt9TEM7o8zkC2OLNsu
- mitmproxy:YaDGC9Gt9TEM7o8zkC2OLNsu
on_success: change
on_failure: always

6706
doc-src/01-vendor.css
View File

File diff suppressed because it is too large Load Diff

43
doc-src/02-app.css
View File

@ -1,43 +0,0 @@
.masthead {
text-align: center;
border-bottom: 0;
}
.frontpage .talks div {
margin-bottom: 10px;
}
.nav-sidebar {
background-color: #f0f0f0;
margin-bottom: 20px;
}
.nav-sidebar li {
line-height: 1.1;
}
.nav-sidebar li > a,
.nav-sidebar .nav-header {
padding-left: 20px;
}
.nav-sidebar .nav-header {
margin-top: 1em;
font-size: 1.2em;
font-weight: bold;
}
.nav-sidebar .active > a,
.nav-sidebar .active > a:hover,
.nav-sidebar .active > a:focus {
color: #fff;
background-color: #428bca;
}
.tablenum {
font-weight: bold;
}
.nowrap {
white-space: nowrap;
}
.page-header {
margin: 0px 0 22px;
}
.page-header h1 {
margin-top: 0px;
}
/*# sourceMappingURL=02-app.css.map */

44
doc-src/_layout.html
View File

@ -1,44 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<title>@!pageTitle!@</title>
$!header!$
</head>
<body>
<div class="navbar navbar-default navbar-static-top">
<div class="container">
<div class="navbar-header">
<a class="navbar-brand" href="@!urlTo("/index.html")!@">
<img height="20px" src="@!urlTo("mitmproxy-long.png")!@"/>
</a>
</div>
<div class="navbar-header navbar-right">
<a class="navbar-brand" hre="#">$!VERSION!$ docs</a>
</div>
</div>
</div>
<div class="container">
<div class="row">
<div class="col-md-3">
$!navbar!$
</div>
<div class="col-md-9">
<div class="page-header">
<h1>@!this.title!@</h1>
</div>
$!body!$
</div>
</div>
</div>
<div class="container">
<hr>
<footer>
<p>@!copyright!@</p>
</footer>
</div>
</body>
</html>

50
doc-src/_nav.html
View File

@ -1,50 +0,0 @@
<ul class="nav nav-sidebar">
$!nav(idxpath, this, state)!$
$!nav("install.html", this, state)!$
$!nav("certinstall.html", this, state)!$
$!nav("howmitmproxy.html", this, state)!$
$!nav("modes.html", this, state)!$
<li class="nav-header">Tools</li>
$!nav("mitmproxy.html", this, state)!$
$!nav("mitmdump.html", this, state)!$
$!nav("config.html", this, state)!$
<li class="nav-header">Features</li>
$!nav("anticache.html", this, state)!$
$!nav("filters.html", this, state)!$
$!nav("replacements.html", this, state)!$
$!nav("clientreplay.html", this, state)!$
$!nav("serverreplay.html", this, state)!$
$!nav("setheaders.html", this, state)!$
$!nav("passthrough.html", this, state)!$
$!nav("proxyauth.html", this, state)!$
$!nav("reverseproxy.html", this, state)!$
$!nav("responsestreaming.html", this, state)!$
$!nav("socksproxy.html", this, state)!$
$!nav("sticky.html", this, state)!$
$!nav("tcpproxy.html", this, state)!$
$!nav("upstreamproxy.html", this, state)!$
$!nav("upstreamcerts.html", this, state)!$
<li class="nav-header">Transparent Proxying</li>
$!nav("transparent.html", this, state)!$
$!nav("transparent/linux.html", this, state)!$
$!nav("transparent/osx.html", this, state)!$
<li class="nav-header">Scripting</li>
$!nav("scripting/inlinescripts.html", this, state)!$
$!nav("scripting/libmproxy.html", this, state)!$
<li class="nav-header">Tutorials</li>
$!nav("tutorials/30second.html", this, state)!$
$!nav("tutorials/gamecenter.html", this, state)!$
$!nav("tutorials/transparent-dhcp.html", this, state)!$
<li class="nav-header">Hacking</li>
$!nav("dev/architecture.html", this, state)!$
$!nav("dev/testing.html", this, state)!$
$!nav("dev/sslkeylogfile.html", this, state)!$
</ul>

151
doc-src/certinstall.html
View File

@ -1,151 +0,0 @@
## On This Page
* [Introduction](#docIntro)
* [Quick Setup](#docQuick)
* [Installing the mitmproxy CA certificate manually](#docManual)
* [More on mitmproxy certificates](#docMore)
* [CA and cert files](#docCertfiles)
* [Using a custom certificate](#docCustom)
* [Using a client side certificate](#docClient)
* [Using a custom certificate authority](#docCA)
## <a id="docIntro"></a>Introduction
Mitmproxy can decrypt encrypted traffic on the fly, as long as the client
trusts its built-in certificate authority. Usually this means that the
mitmproxy CA certificates have to be installed on the client device.
## <a id="docQuick"></a>Quick Setup
By far the easiest way to install the mitmproxy certificates is to use the
built-in certificate installation app. To do this, just start mitmproxy and
configure your target device with the correct proxy settings. Now start a
browser on the device, and visit the magic domain **mitm.it**. You should see
something like this:
<img class="img-responsive" src="@!urlTo("certinstall-webapp.png")!@" ></img>
Click on the relevant icon, and follow the setup instructions for the platform
you're on, and you are good to go.
## <a id="docManual"></a>Installing the mitmproxy CA certificate manually
Sometimes using the quick install app is not an option - Java or the iOS
Simulator spring to mind - or you just need to do it manually for some other
reason. Below is a list of pointers to manual certificate installation
documentation for some common platforms:
<table class="table">
<tr>
<td><a href="https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore"</a>iOS Simulator</td>
<td><a href="http://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html">Java</a></td>
</tr>
<tr>
<td><a href="http://kb.mit.edu/confluence/pages/viewpage.action?pageId=152600377">iOS</a></td>
<td><a href="http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets">Android/Android Simulator</a></td>
</tr>
<tr>
<td><a href="http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7">Windows</a></td>
<td><a href="https://support.apple.com/kb/PH7297?locale=en_US">Mac OS X</a></td>
</tr>
<tr>
<td><a href="http://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861">Ubuntu/Debian</a></td>
<td><a href="https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox">Firefox</a></td>
</tr>
<tr>
<td><a href="https://code.google.com/p/chromium/wiki/LinuxCertManagement">Chrome on Linux</a></td>
</tr>
</table>
## <a id="docMore"></a>More on mitmproxy certificates
The first time __mitmproxy__ or __mitmdump__ is run, the mitmproxy Certificate
Authority(CA) is created in the config directory (~/.mitmproxy by default).
This CA is used for on-the-fly generation of dummy certificates for each of the
SSL sites that your client visits. Since your browser won't trust the
__mitmproxy__ CA out of the box , you will see an SSL certificate warning every
time you visit a new SSL domain through __mitmproxy__. When you are testing a
single site through a browser, just accepting the bogus SSL cert manually is
not too much trouble, but there are a many circumstances where you will want to
configure your testing system or browser to trust the __mitmproxy__ CA as a
signing root authority.
## <a id="docCertfiles"></a>CA and cert files
The files created by mitmproxy in the .mitmproxy directory are as follows:
<table class="table">
<tr>
<td class="nowrap">mitmproxy-ca.pem</td>
<td>The private key and certificate in PEM format.</td>
</tr>
<tr>
<td class="nowrap">mitmproxy-ca-cert.pem</td>
<td>The certificate in PEM format. Use this to distribute to most
non-Windows platforms.</td>
</tr>
<tr>
<td class="nowrap">mitmproxy-ca-cert.p12</td>
<td>The certificate in PKCS12 format. For use on Windows.</td>
</tr>
<tr>
<td class="nowrap">mitmproxy-ca-cert.cer</td>
<td>Same file as .pem, but with an extension expected by some Android
devices.</td>
</tr>
</table>
## <a id="docCustom"></a>Using a custom certificate
You can use your own certificate by passing the <kbd>--cert</kbd> option to
mitmproxy. mitmproxy then uses the provided certificate for interception of the
specified domains instead of generating a certificate signed by its own CA.
The certificate file is expected to be in the PEM format. You can include
intermediary certificates right below your leaf certificate, so that you PEM
file roughly looks like this:
<pre>
-----BEGIN PRIVATE KEY-----
&lt;private key&gt;
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
&lt;cert&gt;
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
&lt;intermediary cert (optional)&gt;
-----END CERTIFICATE-----
</pre>
For example, you can generate a certificate in this format using these instructions:
<pre class="terminal">
$ openssl genrsa -out cert.key 2048
$ openssl req -new -x509 -key cert.key -out cert.crt
(Specify the mitm domain as Common Name, e.g. *.google.com)
$ cat cert.key cert.crt > cert.pem
$ mitmproxy --cert=cert.pem
</pre>
## <a id="docClient"></a>Using a client side certificate
You can use a client certificate by passing the <kbd>--client-certs
DIRECTORY</kbd> option to mitmproxy. If you visit example.org, mitmproxy looks
for a file named example.org.pem in the specified directory and uses this as
the client cert. The certificate file needs to be in the PEM format and should
contain both the unencrypted private key and the certificate.
## <a id="docCA"></a>Using a custom certificate authority
By default, mitmproxy will use <samp>~/.mitmproxy/mitmproxy-ca.pem</samp> as
the certificate authority to generate certificates for all domains for which no
custom certificate is provided (see above). You can use your own certificate
authority by passing the <kbd>--confdir</kbd> option to mitmproxy. Mitmproxy
will then look for <samp>mitmproxy-ca.pem</samp> in the specified directory. If
no such file exists, it will be generated automatically.

86
doc-src/config.html
View File

@ -1,86 +0,0 @@
Mitmproxy is configured through a set of files in the users ~/.mitmproxy
directory.
<table class="table">
<tbody>
<tr>
<th>mitmproxy.conf</th>
<td>Settings for the <b>mitmproxy</b>. This file can contain any options supported by mitmproxy.</td>
</tr>
<tr>
<th>mitmdump.conf</th>
<td>Settings for the <b>mitmdump</b>. This file can contain any options supported by mitmdump.</td>
</tr>
<tr>
<th>common.conf</th>
<td>Settings shared between all command-line tools. Settings in
this file are over-ridden by those in the tool-specific
files. Only options shared by mitmproxy and mitmdump should be used in this file. </td>
</tr>
</tbody>
</table>
# Syntax
## Comments
<pre>
# this is a comment
; this is also a comment (.ini style)
--- and this is a comment too (yaml style)
</pre>
## Key/Value pairs
- Keys and values are case-sensitive
- Whitespace is ignored
- Lists are comma-delimited, and enclosed in square brackets
<pre>
name = value # (.ini style)
name: value # (yaml style)
--name value # (command-line option style)
fruit = [apple, orange, lemon]
indexes = [1, 12, 35 , 40]
</pre>
## Flags
These are boolean options that take no value but true/false.
<pre>
name = true # (.ini style)
name
--name # (command-line option style)
</pre>
# Options
The options available in the config files are precisely those available as
command-line flags, with the key being the option's long name. To get a
complete list of these, use the __--help__ option on each of the tools. Be
careful to only specify common options in the __common.conf__ file -
unsupported options in this file will be detected as an error on startup.
# Examples
## common.conf
Note that __port__ is an option supported by all tools.
<pre class="code">
port = 8080
</pre>
## mitmproxy.conf
<pre class="code">
palette = light
</pre>

8
doc-src/dev/architecture.html
View File

@ -1,8 +0,0 @@
To give you a better understanding of how mitmproxy works, mitmproxy's
high-level architecture is detailed in the following graphic:
<img class="img-responsive" src="@!urlTo('schematics/architecture.png')!@">
<a href="@!urlTo('schematics/architecture.pdf')!@">(architecture.pdf)</a>
<p>Please don't refrain from asking any further
questions on the mailing list, the IRC channel or the GitHub issue tracker.</p>

8
doc-src/dev/index.py
View File

@ -1,8 +0,0 @@
from countershape import Page
pages = [
Page("testing.html", "Testing"),
Page("architecture.html", "Architecture"),
Page("sslkeylogfile.html", "TLS Master Secrets"),
# Page("addingviews.html", "Writing Content Views"),
]

8
doc-src/dev/sslkeylogfile.html
View File

@ -1,8 +0,0 @@
The SSL master keys can be logged by mitmproxy so that external programs can decrypt TLS connections both from and to the proxy.
Key logging is enabled by setting the environment variable <samp>SSLKEYLOGFILE</samp> so that it points to a writable
text file. Recent versions of WireShark can use these log files to decrypt packets.
You can specify the key file path in WireShark via<br>
<samp>Edit → Preferences → Protocols → SSL → (Pre)-Master-Secret log filename</samp>.
Note that <samp>SSLKEYLOGFILE</samp> is respected by other programs as well, e.g. Firefox and Chrome.
If this creates any issues, you can set <samp>MITMPROXY_SSLKEYLOGFILE</samp> alternatively.

43
doc-src/dev/testing.html
View File

@ -1,43 +0,0 @@
All the mitmproxy projects strive to maintain 100% code coverage. In general,
patches and pull requests will be declined unless they're accompanied by a
suitable extension to the test suite.
Our tests are written for the [nose](https://nose.readthedocs.org/en/latest/).
At the point where you send your pull request, a command like this:
<pre class="terminal">
> nosetests --with-cov --cov-report term-missing ./test
</pre>
Should give output something like this:
<pre class="terminal">
> ---------- coverage: platform darwin, python 2.7.2-final-0 --
> Name Stmts Miss Cover Missing
> ----------------------------------------------------
> libmproxy/__init__ 0 0 100%
> libmproxy/app 4 0 100%
> libmproxy/cmdline 100 0 100%
> libmproxy/controller 69 0 100%
> libmproxy/dump 150 0 100%
> libmproxy/encoding 39 0 100%
> libmproxy/filt 201 0 100%
> libmproxy/flow 891 0 100%
> libmproxy/proxy 427 0 100%
> libmproxy/script 27 0 100%
> libmproxy/utils 133 0 100%
> libmproxy/version 4 0 100%
> ----------------------------------------------------
> TOTAL 2045 0 100%
> ----------------------------------------------------
> Ran 251 tests in 11.864s
</pre>
There are exceptions to the coverage requirement - for instance, much of the
console interface code can't sensibly be unit tested. These portions are
excluded from coverage analysis either in the **.coveragerc** file, or using
**#pragma no-cover** directives. To keep our coverage analysis relevant, we use
these measures as sparingly as possible.

18
doc-src/features/anticache.html
View File

@ -1,18 +0,0 @@
When the __anticache__ option is passed to mitmproxy, it removes headers
(__if-none-match__ and __if-modified-since__) that might elicit a
304-not-modified response from the server. This is useful when you want to make
sure you capture an HTTP exchange in its totality. It's also often used during
[client replay](@!urlTo("clientreplay.html")!@), when you want to make sure the
server responds with complete data.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>--anticache</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>a</b></td>
</tr>
</tbody>
</table>

19
doc-src/features/index.py
View File

@ -1,19 +0,0 @@
from countershape import Page
pages = [
Page("anticache.html", "Anticache"),
Page("clientreplay.html", "Client-side replay"),
Page("filters.html", "Filter expressions"),
Page("passthrough.html", "Ignore Domains"),
Page("proxyauth.html", "Proxy Authentication"),
Page("replacements.html", "Replacements"),
Page("responsestreaming.html", "Response Streaming"),
Page("reverseproxy.html", "Reverse proxy mode"),
Page("socksproxy.html", "SOCKS Mode"),
Page("setheaders.html", "Set Headers"),
Page("serverreplay.html", "Server-side replay"),
Page("sticky.html", "Sticky cookies and auth"),
Page("tcpproxy.html", "TCP Proxy"),
Page("upstreamcerts.html", "Upstream Certs"),
Page("upstreamproxy.html", "Upstream proxy mode"),
]

84
doc-src/features/passthrough.html
View File

@ -1,84 +0,0 @@
There are two main reasons why you may want to exempt some traffic from mitmproxy's interception mechanism:
- **Certificate pinning:** Some traffic is is protected using
[certificate pinning](https://security.stackexchange.com/questions/29988/what-is-certificate-pinning) and mitmproxy's
interception leads to errors. For example, Windows Update or the Apple App Store fail to work if mitmproxy is active.
- **Convenience:** You really don't care about some parts of the traffic and just want them to go away.
If you want to peek into (SSL-protected) non-HTTP connections, check out the [tcp proxy](@!urlTo("tcpproxy.html")!@) feature.
If you want to ignore traffic from mitmproxy's processing because of large response bodies, take a look at the
[response streaming](@!urlTo("responsestreaming.html")!@) feature.
## How it works
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>--ignore regex</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>I</b></td>
</tr>
</tbody>
</table>
mitmproxy allows you to specify a regex which is matched against a <code>host:port</code> string (e.g. "example.com:443")
to determine hosts that should be excluded.
There are two important quirks to consider:
- **In transparent mode, the ignore pattern is matched against the IP.** While we usually infer the hostname from the
Host header if the --host argument is passed to mitmproxy, we do not have access to this information before the SSL
handshake.
- In regular mode, explicit HTTP requests are never ignored.[^explicithttp] The ignore pattern is applied on CONNECT
requests, which initiate HTTPS or clear-text WebSocket connections.
### Tutorial
If you just want to ignore one specific domain, there's usually a bulletproof method to do so:
1. Run mitmproxy or mitmdump in verbose mode (-v) and observe the host:port information in the serverconnect
messages. mitmproxy will filter on these.
2. Take the host:port string, surround it with ^ and $, escape all dots (. becomes \\.)
and use this as your ignore pattern:
<pre class="terminal">
$ mitmdump -v
127.0.0.1:50588: clientconnect
127.0.0.1:50588: request
-> CONNECT example.com:443 HTTP/1.1
127.0.0.1:50588: Set new server address: example.com:443
127.0.0.1:50588: serverconnect
-> example.com:443
^C
$ mitmproxy --ignore ^example\.com:443$
</pre>
Here are some other examples for ignore patterns:
<pre>
# Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
--ignore apple.com:443
# "Correct" version without false-positives:
--ignore '^(.+\.)?apple\.com:443$'
# Ignore example.com, but not its subdomains:
--ignore '^example.com:'
# Ignore everything but example.com and mitmproxy.org:
--ignore '^(?!example\.com)(?!mitmproxy\.org)'
# Transparent mode:
--ignore 17\.178\.96\.59:443
# IP address range:
--ignore 17\.178\.\d+\.\d+:443
</pre>
### See Also
- [TCP Proxy](@!urlTo("tcpproxy.html")!@)
- [Response Streaming](@!urlTo("responsestreaming.html")!@)
[^explicithttp]: This stems from an limitation of explicit HTTP proxying: A single connection can be re-used for multiple target domains - a <code>GET http://example.com/</code> request may be followed by a <code>GET http://evil.com/</code> request on the same connection. If we start to ignore the connection after the first request, we would miss the relevant second one.

26
doc-src/features/proxyauth.html
View File

@ -1,26 +0,0 @@
Asks the user for authentication before they are permitted to use the proxy.
Authentication headers are stripped from the flows, so they are not passed to
upstream servers. For now, only HTTP Basic authentication is supported. The
proxy auth options are ignored if the proxy is in transparent or reverse proxy
mode.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th>
<td>
<ul>
<li>--nonanonymous</li>
<li>--singleuser USER</li>
<li>--htpasswd PATH</li>
</ul>
</td>
</tr>
</tbody>
</table>

58
doc-src/features/responsestreaming.html
View File

@ -1,58 +0,0 @@
By using mitmproxy's streaming feature, response contents can be passed to the client incrementally before they have been fully received by the proxy.
This is especially useful for large binary files such as videos, where buffering the whole file slows down the client's browser.
By default, mitmproxy will read the entire response, perform any indicated
manipulations on it and then send the (possibly modified) response to
the client. In some cases this is undesirable and you may wish to "stream"
the reponse back to the client. When streaming is enabled, the response is
not buffered on the proxy but directly sent back to the client instead.
<h2>On the command-line</h2>
Streaming can be enabled on the command line for all response bodies exceeding a certain size. The SIZE argument understands
k/m/g suffixes, e.g. 3m for 3 megabytes.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th>
<td>
--stream SIZE
</td>
</tr>
</tbody>
</table>
<h2>Caveats</h2>
When response streaming is enabled, <strong>streamed response contents will not be
recorded or preserved in any way.</strong>
When response streaming is enabled, the response body cannot be modified.
<h2>Customizing Response Streaming</h2>
You can also use an <a href="@!urlTo("scripting/inlinescripts.html")!@">inline script</a> to customize exactly
which responses are streamed.
Responses that should be tagged for streaming by setting their respective .stream attribute to True:
$!example("examples/stream.py")!$
<h2>Implementation Details</h2>
When response streaming is enabled, portions of the code which would have otherwise performed changes
on the response body will see an empty response body instead (<code>libmproxy.protocol.http.CONTENT_MISSING</code>). Any modifications will be ignored.
Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a <code>Transfer-Encoding:
chunked</code> header, the response will be streamed one chunk at a time.
<h2>Modifying streamed data</h2>
If the <code>.stream</code> attribute is callable, .stream will work as a hook in chunk data processing.
$!example("examples/stream_modify.py")!$
### See Also
- [Ignore Domains](@!urlTo("passthrough.html")!@)

47
doc-src/features/reverseproxy.html
View File

@ -1,47 +0,0 @@
In reverse proxy mode, mitmproxy accepts standard HTTP requests and forwards
them to the specified upstream server. This is in contrast to
<a href="@!urlTo("upstreamproxy.html")!@">upstream proxy mode</a>, in which
mitmproxy forwards HTTP proxy requests to an upstream proxy server.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>-R <i>scheme</i>://hostname[:port]</td>
</tr>
</tbody>
</table>
Here, **scheme** signifies if the proxy should use TLS to connect to the server.
mitmproxy accepts both encrypted and unencrypted requests and transforms them to what the server
expects.
mitmdump -R https://httpbin.org -p 80
mitmdump -R https://httpbin.org -p 443
### Host Header
In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this
may lead to issues with public web servers. For example, consider the following scenario:
$ python mitmdump -d -R http://example.com/ &
$ curl http://localhost:8080/
>> GET https://example.com/
Host: localhost:8080
User-Agent: curl/7.35.0
[...]
<< 404 Not Found 345B
Since the Host header doesn't match <samp>example.com</samp>, an error is returned.<br>
There are two ways to solve this:
<ol>
<li>Modify the hosts file of your OS so that example.com resolves to 127.0.0.1.</li>
<li>
Instruct mitmproxy to rewrite the host header by passing <kbd>&#8209;&#8209;setheader&nbsp;:~q:Host:example.com</kbd>.
However, keep in mind that absolute URLs within the returned document or HTTP redirects will cause the client application
to bypass the proxy.
</li>
</ol>

18
doc-src/features/setheaders.html
View File

@ -1,18 +0,0 @@
This feature lets you specify a set of headers to be added to requests or
responses, based on a filter pattern. You can specify these either on the
command-line, or through an interactive editor in mitmproxy.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th>
<td>
--setheader PATTERN
</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>H</b></td>
</tr>
</tbody>
</table>

10
doc-src/features/socksproxy.html
View File

@ -1,10 +0,0 @@
In this mode, mitmproxy acts as a SOCKS5 proxy server.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>--socks</td>
</tr>
</tbody>
</table>

60
doc-src/features/sticky.html
View File

@ -1,60 +0,0 @@
## Sticky cookies
When the sticky cookie option is set, __mitmproxy__ will add the cookie most
recently set by the server to any cookie-less request. Consider a service that
sets a cookie to track the session after authentication. Using sticky cookies,
you can fire up mitmproxy, and authenticate to a service as you usually would
using a browser. After authentication, you can request authenticated resources
through mitmproxy as if they were unauthenticated, because mitmproxy will
automatically add the session tracking cookie to requests. Among other things,
this lets you script interactions with authenticated resources (using tools
like wget or curl) without having to worry about authentication.
Sticky cookies are especially powerful when used in conjunction with [client
replay](@!urlTo("clientreplay.html")!@) - you can record the authentication
process once, and simply replay it on startup every time you need to interact
with the secured resources.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th>
<td>
<ul>
<li>-t FILTER</li>
</ul>
</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>t</b></td>
</tr>
</tbody>
</table>
## Sticky auth
The sticky auth option is analogous to the sticky cookie option, in that HTTP
__Authorization__ headers are simply replayed to the server once they have been
seen. This is enough to allow you to access a server resource using HTTP Basic
authentication through the proxy. Note that __mitmproxy__ doesn't (yet) support
replay of HTTP Digest authentication.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th>
<td>
<ul>
<li>-u FILTER</li>
</ul>
</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>A</b></td>
</tr>
</tbody>
</table>

30
doc-src/features/tcpproxy.html
View File

@ -1,30 +0,0 @@
WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt hostnames from
processing, so that mitmproxy acts as a generic TCP forwarder. This feature is closely related to the
[ignore domains](@!urlTo("passthrough.html")!@) functionality, but differs in two important aspects:
- The raw TCP messages are printed to the event log.
- SSL connections will be intercepted.
Please note that message interception or modification are not possible yet.
If you are not interested in the raw TCP messages, you should use the ignore domains feature.
## How it works
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>--tcp HOST</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>T</b></td>
</tr>
</tbody>
</table>
For a detailed description on the structure of the hostname pattern, please refer to the [Ignore Domains](@!urlTo("passthrough.html")!@) feature.
### See Also
- [Ignore Domains](@!urlTo("passthrough.html")!@)
- [Response Streaming](@!urlTo("responsestreaming.html")!@)

13
doc-src/features/upstreamproxy.html
View File

@ -1,13 +0,0 @@
In this mode, mitmproxy accepts proxy requests and unconditionally forwards all
requests to a specified upstream proxy server. This is in contrast to <a
href="@!urlTo("reverseproxy.html")!@">reverse proxy mode</a>, in which
mitmproxy forwards ordinary HTTP requests to an upstream server.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>-U http://hostname[:port]</td>
</tr>
</tbody>
</table>

76
doc-src/index.py
View File

@ -1,76 +0,0 @@
import os
import sys
import datetime
import countershape
from countershape import Page, Directory, markup
import countershape.template
MITMPROXY_SRC = os.path.abspath(
os.path.expanduser(os.environ.get("MITMPROXY_SRC", ".."))
)
sys.path.insert(0, MITMPROXY_SRC)
from libmproxy import filt, version
ns.VERSION = version.VERSION
if ns.options.website:
ns.idxpath = "doc/index.html"
else:
ns.idxpath = "index.html"
this.layout = countershape.layout.FileLayout("_layout.html")
ns.title = countershape.template.Template(None, "<h1>@!this.title!@</h1>")
this.titlePrefix = "%s - " % version.NAMEVERSION
this.markup = markup.Markdown(extras=["footnotes"])
ns.docMaintainer = "Aldo Cortesi"
ns.docMaintainerEmail = "aldo@corte.si"
ns.copyright = u"\u00a9 mitmproxy project, %s" % datetime.date.today().year
def mpath(p):
p = os.path.join(MITMPROXY_SRC, p)
return os.path.expanduser(p)
def example(s):
d = file(mpath(s)).read().rstrip()
extemp = """<div class="example">%s<div class="example_legend">(%s)</div></div>"""
return extemp % (countershape.template.Syntax("py")(d), s)
ns.example = example
ns.filt_help = filt.help
def nav(page, current, state):
if current.match(page, False):
pre = '<li class="active">'
else:
pre = "<li>"
p = state.application.getPage(page)
return pre + \
'<a href="%s">%s</a></li>' % (countershape.widgets.UrlTo(page), p.title)
ns.nav = nav
ns.navbar = countershape.template.File(None, "_nav.html")
pages = [
Page("index.html", "Introduction"),
Page("install.html", "Installation"),
Page("certinstall.html", "About Certificates"),
Page("howmitmproxy.html", "How mitmproxy works"),
Page("modes.html", "Modes of Operation"),
Page("mitmproxy.html", "mitmproxy"),
Page("mitmdump.html", "mitmdump"),
Page("config.html", "configuration"),
Directory("scripting"),
Directory("tutorials"),
Page("transparent.html", "Overview"),
Directory("transparent"),
]

79
doc-src/install.html
View File

@ -1,79 +0,0 @@
## On This Page
* [Installation On Ubuntu](#docUbuntu)
* [Installation On Mac OS X](#docOSX)
* [Installation On Windows](#docWindows)
## <a id=docUbuntu></a>Installation On Ubuntu
Ubuntu comes with Python but we need to install pip, python-dev and several libraries. This was tested on a fully patched installation of Ubuntu 14.04.
<pre class="terminal">
$ sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev
$ sudo pip install mitmproxy
</pre>
Once installation is complete you can run <a href="mitmproxy.html">mitmproxy</a> or <a href="mitmdump.html">mitmdump</a> from a terminal.
### Installation From Source
If you would like to install mitmproxy directly from the master branch on GitHub or would like to get set up to contribute to the project,
install the dependencies as you would for a regular mitmproxy installation (see previous section).
Then see the <a href="https://github.com/mitmproxy/mitmproxy/blob/master/README.mkd#hacking">Hacking</a> section of the README on GitHub.
## <a id=docOSX></a>Installation On Mac OS X
The easiest way to get up and running on OSX is to download the pre-built binary packages from [mitmproxy.org](http://mitmproxy.org).
There are a few bits of customization you might want to do to make mitmproxy comfortable to use on OSX. The default color scheme is optimized for a dark background terminal, but you can select a palette for a light terminal background with the --palette option. You can use the OSX <b>open</b> program to create a simple and effective <b>~/.mailcap</b> file to view request and response bodies:
<pre class="terminal">
application/*; /usr/bin/open -Wn %s
audio/*; /usr/bin/open -Wn %s
image/*; /usr/bin/open -Wn %s
video/*; /usr/bin/open -Wn %s
</pre>
Once installation is complete you can run <a href="mitmproxy.html">mitmproxy</a> or <a href="mitmdump.html">mitmdump</a> from a terminal.
### Installation From Source
If you would like to install mitmproxy directly from the master branch on GitHub or would like to get set up to contribute to the project, ithere are a few OS X specific things to keep in mind.
- Make sure that XCode is installed from the App Store, and that the command-line tools have been downloaded (XCode/Preferences/Downloads).
- If you're running a Python interpreter installed with homebrew (or similar), you may have to install some dependencies by hand.
Then see the <a href="https://github.com/mitmproxy/mitmproxy/blob/master/README.mkd#hacking">Hacking</a> section of the README on GitHub.
## <a id=docWindows></a>Installation On Windows
Please note that mitmdump is the only component of mitmproxy that is supported on Windows at the moment.
There is no interactive user interface on Windows.
First, install the latest version of Python 2.7 from the <a href="https://www.python.org/downloads/windows/">Python website</a>.
If you already have an older version of Python 2.7 installed, make sure to install <a href="https://pip.pypa.io/en/latest/installing.html">pip</a>
(pip is included in Python 2.7.9+ by default).
Next, add Python and the Python Scripts directory to your <strong>PATH</strong> variable. You can do this easily by running the following in powershell:
<pre class="terminal">
[Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27\;C:\Python27\Scripts\", "User")
</pre>
Now, you can install mitmproxy by running
<pre class="terminal">
pip install mitmproxy
</pre>
Once the installation is complete, you can run <a href="mitmdump.html">mitmdump</a> from a command prompt.
### Installation From Source
If you would like to install mitmproxy directly from the master branch on GitHub or would like to get set up to contribute to the project, install Python as outlined above, then see the <a href="https://github.com/mitmproxy/mitmproxy/blob/master/README.mkd#hacking">Hacking</a> section of the README on GitHub.

68
doc-src/mitmdump.html
View File

@ -1,68 +0,0 @@
__mitmdump__ is the command-line companion to mitmproxy. It provides
tcpdump-like functionality to let you view, record, and programmatically
transform HTTP traffic. See the _--help_ flag output for complete
documentation.
# Examples
## Saving traffic
<pre class="terminal">
> mitmdump -w outfile
</pre>
Start up mitmdump in proxy mode, and write all traffic to __outfile__.
## Filtering saved traffic
<pre class="terminal">
> mitmdump -nr infile -w outfile "~m post"
</pre>
Start mitmdump without binding to the proxy port (_-n_), read all flows from
infile, apply the specified filter expression (only match POSTs), and write to
outfile.
## Client replay
<pre class="terminal">
> mitmdump -nc outfile
</pre>
Start mitmdump without binding to the proxy port (_-n_), then replay all
requests from outfile (_-c filename_). Flags combine in the obvious way, so
you can replay requests from one file, and write the resulting flows to
another:
<pre class="terminal">
> mitmdump -nc srcfile -w dstfile
</pre>
See the [Client-side Replay](@!urlTo("clientreplay.html")!@) section for more information.
## Running a script
<pre class="terminal">
> mitmdump -s examples/add_header.py
</pre>
This runs the __add_header.py__ example script, which simply adds a new header
to all responses.
## Scripted data transformation
<pre class="terminal">
> mitmdump -ns examples/add_header.py -r srcfile -w dstfile
</pre>
This command loads flows from __srcfile__, transforms it according to the
specified script, then writes it back to __dstfile__.

115
doc-src/mitmproxy.html
View File

@ -1,115 +0,0 @@
__mitmproxy__ is a console tool that allows interactive examination and
modification of HTTP traffic. It differs from mitmdump in that all flows are
kept in memory, which means that it's intended for taking and manipulating
small-ish samples. Use the _?_ shortcut key to view, context-sensitive
documentation from any __mitmproxy__ screen.
## Flow list
The flow list shows an index of captured flows in chronological order.
<img class="img-responsive" src="@!urlTo('screenshots/mitmproxy.png')!@"/>
- __1__: A GET request, returning a 302 Redirect response.
- __2__: A GET request, returning 16.75kb of text/html data.
- __3__: A replayed request.
- __4__: Intercepted flows are indicated with orange text. The user may edit
these flows, and then accept them (using the _a_ key) to continue. In this
case, the request has been intercepted on the way to the server.
- __5__: A response intercepted from the server on the way to the client.
- __6__: The event log can be toggled on and off using the _e_ shortcut key. This
pane shows events and errors that may not result in a flow that shows up in the
flow pane.
- __7__: Flow count.
- __8__: Various information on mitmproxy's state. In this case, we have an
interception pattern set to ".*".
- __9__: Bind address indicator - mitmproxy is listening on port 8080 of all
interfaces.
## Flow view
The __Flow View__ lets you inspect and manipulate a single flow:
<img class="img-responsive" src="@!urlTo('screenshots/mitmproxy-flowview.png')!@"/>
- __1__: Flow summary.
- __2__: The Request/Response tabs, showing you which part of the flow you are
currently viewing. In the example above, we're viewing the Response. Hit _tab_
to switch between the Response and the Request.
- __3__: Headers.
- __4__: Body.
- __5__: View Mode indicator. In this case, we're viewing the body in __hex__
mode. The other available modes are __pretty__, which uses a number of
heuristics to show you a friendly view of various content types, and __raw__,
which shows you exactly what's there without any changes. You can change modes
using the _m_ key.
## Grid Editor
Much of the data that we'd like to interact with in mitmproxy is structured.
For instance, headers, queries and form data can all be thought of as a list of
key/value pairs. Mitmproxy has a built-in editor that lays this type of data
out in a grid for easy manipulation.
At the moment, the Grid Editor is used in four parts of mitmproxy:
- Editing request or response headers (_e_ for edit, then _h_ for headers in flow view)
- Editing a query string (_e_ for edit, then _q_ for query in flow view)
- Editing a URL-encoded form (_e_ for edit, then _f_ for form in flow view)
- Editing replacement patterns (_o_ for options, then _R_ for Replacement Patterns)
If there is is no data, an empty editor will be started to let you add some.
Here is the editor showing the headers from a request:
<img class="img-responsive" src="@!urlTo('screenshots/mitmproxy-kveditor.png')!@"/>
To edit, navigate to the key or value you want to modify using the arrow or vi
navigation keys, and press enter. The background color will change to show that
you are in edit mode for the specified field:
<img class="img-responsive" src="@!urlTo('screenshots/mitmproxy-kveditor-editmode.png')!@"/>
Modify the field as desired, then press escape to exit edit mode when you're
done. You can also add a row (_a_ key), delete a row (_d_ key), spawn an
external editor on a field (_e_ key). Be sure to consult the context-sensitive
help (_?_ key) for more.
# Example: Interception
__mitmproxy__'s interception functionality lets you pause an HTTP request or
response, inspect and modify it, and then accept it to send it on to the server
or client.
### 1: Set an interception pattern
<img class="img-responsive" src="@!urlTo('mitmproxy-intercept-filt.png')!@"/>
We press _i_ to set an interception pattern. In this case, the __~q__ filter
pattern tells __mitmproxy__ to intercept all requests. For complete filter
syntax, see the [Filter expressions](@!urlTo("filters.html")!@) section of this
document, or the built-in help function in __mitmproxy__.
### 2: Intercepted connections are indicated with orange text:
<img class="img-responsive" src="@!urlTo('mitmproxy-intercept-mid.png')!@"/>
### 3: You can now view and modify the request:
<img class="img-responsive" src="@!urlTo('mitmproxy-intercept-options.png')!@"/>
In this case, we viewed the request by selecting it, pressed _e_ for "edit"
and _m_ for "method" to change the HTTP request method.
### 4: Accept the intercept to continue:
<img class="img-responsive" src="@!urlTo('mitmproxy-intercept-result.png')!@"/>
Finally, we press _a_ to accept the modified request, which is then sent on to
the server. In this case, we changed the request from an HTTP GET to
OPTIONS, and Google's server has responded with a 405 "Method not allowed".

222
doc-src/modes.html
View File

@ -1,222 +0,0 @@
Mitmproxy has four modes of operation that allow you to use mitmproxy in a
variety of scenarios:
- **Regular** (the default)
- **Transparent**
- **Reverse Proxy**
- **Upstream Proxy**
Now, which one should you pick? Use this flow chart:
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-flowchart.png')!@"/>
<div class="page-header">
<h1>Regular Proxy</h1>
</div>
Mitmproxy's regular mode is the simplest and the easiest to set up.
1. Start mitmproxy.
2. Configure your client to use mitmproxy. For instance on IOS, the settings might look like <a href="@!urlTo('screenshots/ios-manual.png')!@">this</a>.
3. Quick Check: You should already be able to visit an unencrypted HTTP site
through the proxy.
4. Open the magic domain <strong>mitm.it</strong> and install the certificate for your device.
<div class="well">
<strong>Heads Up:</strong> Unfortunately, some applications bypass the
system HTTP proxy settings - Android applications are a common example. In
these cases, you need to use mitmproxy's transparent mode.
</div>
If you are proxying an external device, your network will probably look like this:
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-regular.png')!@">
The square brackets signify the source and destination IP addresses. Your
client explicitly connects to mitmproxy and mitmproxy explicitly connects
to the target server.
<div class="page-header">
<h1>Transparent Proxy</h1>
</div>
In transparent mode, traffic is directed into a proxy at the network layer,
without any client configuration required. This makes transparent proxying
ideal for situations where you can't change client behaviour. In the graphic
below, a machine running mitmproxy has been inserted between the router and
the internet:
<a href="@!urlTo('schematics/proxy-modes-transparent-1.png')!@">
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-transparent-1.png')!@">
</a>
The square brackets signify the source and destination IP addresses. Round
brackets mark the next hop on the *Ethernet/data link* layer. This distinction
is important: when the packet arrives at the mitmproxy machine, it must still
be addressed to the target server. This means that Network Address Translation
should not be applied before the traffic reaches mitmproxy, since this would
remove the target information, leaving mitmproxy unable to determine the real
destination.
<a href="@!urlTo('schematics/proxy-modes-transparent-wrong.png')!@">
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-transparent-wrong.png')!@"></a>
<h2>Common Configurations</h2>
There are many ways to configure your network for transparent proxying. We'll
look at three common scenarios:
1. Configuring the client to use a custom gateway/router/"next hop"
2. Implementing custom routing on the router
In most cases, the first option is recommended due to its ease of use.
<h3>(a) Custom Gateway</h3>
One simple way to get traffic to the mitmproxy machine with the destination IP
intact, is to simply configure the client with the mitmproxy box as the
default gateway.
<a href="@!urlTo('schematics/proxy-modes-transparent-2.png')!@">
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-transparent-2.png')!@"></a>
In this scenario, we would:
- Configure the proxy machine for transparent mode. You can find instructions
in the <em>Transparent Proxying</em> section of the mitmproxy docs.
- Configure the client to use the proxy machine's IP as the default gateway.
<a href="@!urlTo('screenshots/ios-gateway.png')!@">Here</a> is what this would
look like on IOS.
- Quick Check: At this point, you should already be able to visit an
unencrypted HTTP site over the proxy.
- Open the magic domain <strong>mitm.it</strong> and install the certificate
for your device.
Setting the custom gateway on clients can be automated by serving the settings
out to clients over DHCP. This lets set up an interception network where all
clients are proxied automatically, which can save time and effort.
<div class="well">
<strong style="text-align: center; display: block">Troubleshooting Transparent Mode</strong>
<p>Incorrect transparent mode configurations are a frequent source of
error. If it doesn't work for you, try the following things:</p>
<ul>
<li>
Open mitmproxy's event log (press `e`) - do you see clientconnect
messages? If not, the packets are not arriving at the proxy. One common
cause is the occurrence of ICMP redirects, which means that your
machine is telling the client that there's a faster way to the
internet by contacting your router directly (see the
<em>Transparent Proxying</em> section on how to disable them). If in
doubt, <a href="https://wireshark.org/">Wireshark</a> may help you
to see whether something arrives at your machine or not.
</li>
<li>
Make sure you have not explicitly configured an HTTP proxy on the
client. This is not needed in transparent mode.
</li>
<li>
Re-check the instructions in the <em>Transparent Proxying</em> section. Anything you missed?
</li>
</ul>
If you encounter any other pitfalls that should be listed here, please let us know!
</div>
<h3>(b) Custom Routing</h3>
In some cases, you may need more fine-grained control of which traffic reaches
the mitmproxy instance, and which doesn't. You may, for instance, choose only
to divert traffic to some hosts into the transparent proxy. There are a huge
number of ways to accomplish this, and much will depend on the router or
packet filter you're using. In most cases, the configuration will look like
this:
<a href="@!urlTo('schematics/proxy-modes-transparent-3.png')!@">
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-transparent-3.png')!@">
</a>
<div class="page-header">
<h1>Reverse Proxy</h1>
</div>
mitmproxy is usually used with a client that uses the proxy to access the
Internet. Using reverse proxy mode, you can use mitmproxy to act like a normal
HTTP server:
<a href="@!urlTo('schematics/proxy-modes-reverse.png')!@">
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-reverse.png')!@">
</a>
There are various use-cases:
- Say you have an internal API running at http://example.local/. You could now
set up mitmproxy in reverse proxy mode at http://debug.example.local/ and
dynamically point clients to this new API endpoint, which provides clients
with the same data and you with debug information. Similarly, you could move
your real server to a different IP/port and set up mitmproxy at the original
place to debug all sessions.
- Say you're a web developer working on example.com (with a development
version running on localhost:8000). You can modify your hosts file so that
example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy mode
on port 80. You can test your app on the example.com domain and get all
requests recorded in mitmproxy.
- Say you have some toy project that should get SSL support. Simply set up
mitmproxy as a reverse proxy on port 443 and you're done (<code>mitmdump -p 443 -R
http://localhost:80/</code>). mitmproxy auto-detects TLS traffic and intercepts it dynamically.
There are better tools for this specific task, but mitmproxy is very quick and simple way to
set up an SSL-speaking server.
- Want to add a non-SSL-capable compression proxy in front of your server? You
could even spawn a mitmproxy instance that terminates SSL (-R http://...),
point it to the compression proxy and let the compression proxy point to a
SSL-initiating mitmproxy (-R https://...), which then points to the real
server. As you see, it's a fairly flexible thing.
Note that mitmproxy supports either an HTTP or an HTTPS upstream server, not
both at the same time. You can work around this by spawning a second mitmproxy
instance.
<div class="well">
<strong style="text-align: center; display: block">Caveat: Interactive Use</strong>
One caveat is that reverse proxy mode is often not sufficient for interactive
browsing. Consider trying to clone Google by using:
<code>mitmproxy -R http://google.com/</code>
This works for the initial request, but the HTML served to the client remains
unchanged. As soon as the user clicks on an non-relative URL (or downloads a
non-relative image resource), traffic no longer passes through mitmproxy, and
the client connects to Google directly again.
</div>
<div class="page-header">
<h1>Upstream Proxy</h1>
</div>
If you want to chain proxies by adding mitmproxy in front of a different proxy
appliance, you can use mitmproxy's upstream mode. In upstream mode, all
requests are unconditionally transferred to an upstream proxy of your choice.
<a href="@!urlTo('schematics/proxy-modes-upstream.png')!@">
<img class="img-responsive" src="@!urlTo('schematics/proxy-modes-upstream.png')!@"></a>
mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy
mode. You could in theory chain multiple mitmproxy instances in a row, but
that doesn't make any sense in practice (i.e. outside of our tests).

6
doc-src/scripting/index.py
View File

@ -1,6 +0,0 @@
from countershape import Page
pages = [
Page("inlinescripts.html", "Inline Scripts"),
Page("libmproxy.html", "libmproxy"),
]

177
doc-src/scripting/inlinescripts.html
View File

@ -1,177 +0,0 @@
__mitmproxy__ has a powerful scripting API that allows you to modify flows
on-the-fly or rewrite previously saved flows locally.
The mitmproxy scripting API is event driven - a script is simply a Python
module that exposes a set of event methods. Here's a complete mitmproxy script
that adds a new header to every HTTP response before it is returned to the
client:
$!example("examples/add_header.py")!$
The first argument to each event method is an instance of ScriptContext that
lets the script interact with the global mitmproxy state. The __response__
event also gets an instance of Flow, which we can use to manipulate the
response itself.
We can now run this script using mitmdump or mitmproxy as follows:
<pre class="terminal">
> mitmdump -s add_header.py
</pre>
The new header will be added to all responses passing through the proxy.
## Example Scripts
mitmproxy comes with a variety of example inline scripts, which demonstrate
many basic tasks. We encourage you to either browse them locally or in our
[GitHub repo](https://github.com/mitmproxy/mitmproxy/tree/master/examples).
## Events
### start(ScriptContext, argv)
Called once on startup, before any other events.
### clientconnect(ScriptContext, ConnectionHandler)
Called when a client initiates a connection to the proxy. Note that
a connection can correspond to multiple HTTP requests.
### serverconnect(ScriptContext, ConnectionHandler)
Called when the proxy initiates a connection to the target server. Note that
a connection can correspond to multiple HTTP requests.
### request(ScriptContext, HTTPFlow)
Called when a client request has been received. The __HTTPFlow__ object is
guaranteed to have a non-None __request__ attribute.
### responseheaders(ScriptContext, HTTPFlow)
Called when the headers of a server response have been received.
This will always be called before the response hook.
The __HTTPFlow__ object is guaranteed to have non-None __request__ and
__response__ attributes. __response.content__ will be None,
as the response body has not been read yet.
### response(ScriptContext, HTTPFlow)
Called when a server response has been received. The __HTTPFlow__ object is
guaranteed to have non-None __request__ and __response__ attributes.
Note that if response streaming is enabled for this response,
__response.content__ will not contain the response body.
### error(ScriptContext, HTTPFlow)
Called when a flow error has occurred, e.g. invalid server responses, or
interrupted connections. This is distinct from a valid server HTTP error
response, which is simply a response with an HTTP error code. The __HTTPFlow__
object is guaranteed to have non-None __request__ and __error__ attributes.
### clientdisconnect(ScriptContext, ConnectionHandler)
Called when a client disconnects from the proxy.
### done(ScriptContext)
Called once on script shutdown, after any other events.
## API
The main classes you will deal with in writing mitmproxy scripts are:
<table class="table">
<tr>
<th>libmproxy.proxy.server.ConnectionHandler</th>
<td>Describes a proxy client connection session. Always has a client_conn attribute, might have a server_conn
attribute.
</td>
</tr>
<tr>
<th>libmproxy.proxy.connection.ClientConnection</th>
<td>Describes a client connection.</td>
</tr>
<tr>
<th>libmproxy.proxy.connection.ServerConnection</th>
<td>Describes a server connection.</td>
</tr>
<tr>
<th>libmproxy.protocol.http.HTTPFlow</th>
<td>A collection of objects representing a single HTTP transaction.</td>
</tr>
<tr>
<th>libmproxy.protocol.http.HTTPResponse</th>
<td>An HTTP response.</td>
</tr>
<tr>
<th>libmproxy.protocol.http.HTTPRequest</th>
<td>An HTTP request.</td>
</tr>
<tr>
<th>libmproxy.protocol.primitives.Error</th>
<td>A communications error.</td>
</tr>
<tr>
<th>libmproxy.script.ScriptContext</th>
<td> A handle for interacting with mitmproxy's from within scripts.</td>
</tr>
<tr>
<th>netlib.odict.ODict</th>
<td>A dictionary-like object for managing sets of key/value data. There
is also a variant called ODictCaseless that ignores key case for some
calls (used mainly for headers).</td>
</tr>
<tr>
<th>netlib.certutils.SSLCert</th>
<td>Exposes information SSL certificates.</td>
</tr>
</table>
The canonical API documentation is the code, which you can browse locally or in our
[GitHub repo](https://github.com/mitmproxy/mitmproxy).
You can view the API documentation using pydoc (which is installed with Python by default), like this:
<pre class="terminal">
> pydoc libmproxy.protocol.http.HTTPRequest
</pre>
## Running scripts in parallel
We have a single flow primitive, so when a script is blocking, other requests are not processed.
While that's usually a very desirable behaviour, blocking scripts can be run threaded by using the <code>libmproxy.script.concurrent</code> decorator.
If your script does not block, you should avoid the overhead of the decorator.
$!example("examples/nonblocking.py")!$
## Make scripts configurable with arguments
Sometimes, you want to pass runtime arguments to the inline script. This can be simply done by surrounding the script call with quotes, e.g.
<code>mitmdump -s "script.py --foo 42"</code>. The arguments are then exposed in the start event:
$!example("examples/modify_response_body.py")!$
## Running scripts on saved flows
Sometimes, we want to run a script on __Flow__ objects that are already
complete. This happens when you start a script, and then load a saved set of
flows from a file (see the "scripted data transformation" example on the
[mitmdump](@!urlTo("mitmdump.html")!@) page). It also happens when you run a
one-shot script on a single flow through the _|_ (pipe) shortcut in mitmproxy.
In this case, there are no client connections, and the events are run in the
following order: __start__, __request__, __responseheaders__, __response__, __error__, __done__. If
the flow doesn't have a __response__ or __error__ associated with it, the
matching events will be skipped.
## Spaces in the script path
By default, spaces are interpreted as separator between the inline script and its arguments (e.g. <code>-s "foo.py
42"</code>). Consequently, the script path needs to be wrapped in a separate pair of quotes if it contains spaces:
<code>-s "'./foo bar/baz.py' 42"</code>.

22
doc-src/scripting/libmproxy.html
View File

@ -1,22 +0,0 @@
<div class="well">
<strong>Heads up!</strong> We strongly encourage you to use <a href="@!urlTo("scripting/inlinescripts.html")!@">inline scripts</a> rather than libmproxy
directly.<br><br>
<ul>
<li>Inline Scripts are equally powerful and provide an easier syntax.</li>
<li>Most examples are written as inline scripts.</li>
<li>Multiple inline scripts can be combined and used together.</li>
<li>Inline Scripts can either be executed headless with mitmdump or within the mitmproxy UI.</li>
</ul>
</div>
All of mitmproxy's basic functionality is exposed through the __libmproxy__
library. The example below shows a simple implementation of the "sticky cookie"
functionality included in the interactive mitmproxy program. Traffic is
monitored for __cookie__ and __set-cookie__ headers, and requests are rewritten
to include a previously seen cookie if they don't already have one. In effect,
this lets you log in to a site using your browser, and then make subsequent
requests using a tool like __curl__, which will then seem to be part of the
authenticated session.
$!example("examples/stickycookies")!$

120
doc-src/syntax.css
View File

@ -1,120 +0,0 @@
.highlight { background: #f8f8f8; }
.highlight .c { color: #408080; font-style: italic } /* Comment */
.highlight .err { border: 1px solid #FF0000 } /* Error */
.highlight .k { color: #008000; font-weight: bold } /* Keyword */
.highlight .o { color: #666666 } /* Operator */
.highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */
.highlight .cp { color: #BC7A00 } /* Comment.Preproc */
.highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */
.highlight .cs { color: #408080; font-style: italic } /* Comment.Special */
.highlight .gd { color: #A00000 } /* Generic.Deleted */
.highlight .ge { font-style: italic } /* Generic.Emph */
.highlight .gr { color: #FF0000 } /* Generic.Error */
.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.highlight .gi { color: #00A000 } /* Generic.Inserted */
.highlight .go { color: #808080 } /* Generic.Output */
.highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */
.highlight .gs { font-weight: bold } /* Generic.Strong */
.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.highlight .gt { color: #0040D0 } /* Generic.Traceback */
.highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */
.highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */
.highlight .kp { color: #008000 } /* Keyword.Pseudo */
.highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */
.highlight .kt { color: #B00040 } /* Keyword.Type */
.highlight .m { color: #666666 } /* Literal.Number */
.highlight .s { color: #BA2121 } /* Literal.String */
.highlight .na { color: #7D9029 } /* Name.Attribute */
.highlight .nb { color: #008000 } /* Name.Builtin */
.highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */
.highlight .no { color: #880000 } /* Name.Constant */
.highlight .nd { color: #AA22FF } /* Name.Decorator */
.highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */
.highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */
.highlight .nf { color: #0000FF } /* Name.Function */
.highlight .nl { color: #A0A000 } /* Name.Label */
.highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
.highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */
.highlight .nv { color: #19177C } /* Name.Variable */
.highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */
.highlight .w { color: #bbbbbb } /* Text.Whitespace */
.highlight .mf { color: #666666 } /* Literal.Number.Float */
.highlight .mh { color: #666666 } /* Literal.Number.Hex */
.highlight .mi { color: #666666 } /* Literal.Number.Integer */
.highlight .mo { color: #666666 } /* Literal.Number.Oct */
.highlight .sb { color: #BA2121 } /* Literal.String.Backtick */
.highlight .sc { color: #BA2121 } /* Literal.String.Char */
.highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */
.highlight .s2 { color: #BA2121 } /* Literal.String.Double */
.highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */
.highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */
.highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */
.highlight .sx { color: #008000 } /* Literal.String.Other */
.highlight .sr { color: #BB6688 } /* Literal.String.Regex */
.highlight .s1 { color: #BA2121 } /* Literal.String.Single */
.highlight .ss { color: #19177C } /* Literal.String.Symbol */
.highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */
.highlight .vc { color: #19177C } /* Name.Variable.Class */
.highlight .vg { color: #19177C } /* Name.Variable.Global */
.highlight .vi { color: #19177C } /* Name.Variable.Instance */
.highlight .il { color: #666666 } /* Literal.Number.Integer.Long */
.grokdoc { background: #f8f8f8; }
.grokdoc .c { color: #408080; font-style: italic } /* Comment */
.grokdoc .err { border: 1px solid #FF0000 } /* Error */
.grokdoc .k { color: #008000; font-weight: bold } /* Keyword */
.grokdoc .o { color: #666666 } /* Operator */
.grokdoc .cm { color: #408080; font-style: italic } /* Comment.Multiline */
.grokdoc .cp { color: #BC7A00 } /* Comment.Preproc */
.grokdoc .c1 { color: #408080; font-style: italic } /* Comment.Single */
.grokdoc .cs { color: #408080; font-style: italic } /* Comment.Special */
.grokdoc .gd { color: #A00000 } /* Generic.Deleted */
.grokdoc .ge { font-style: italic } /* Generic.Emph */
.grokdoc .gr { color: #FF0000 } /* Generic.Error */
.grokdoc .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.grokdoc .gi { color: #00A000 } /* Generic.Inserted */
.grokdoc .go { color: #808080 } /* Generic.Output */
.grokdoc .gp { color: #000080; font-weight: bold } /* Generic.Prompt */
.grokdoc .gs { font-weight: bold } /* Generic.Strong */
.grokdoc .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.grokdoc .gt { color: #0040D0 } /* Generic.Traceback */
.grokdoc .kc { color: #008000; font-weight: bold } /* Keyword.Constant */
.grokdoc .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */
.grokdoc .kp { color: #008000 } /* Keyword.Pseudo */
.grokdoc .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */
.grokdoc .kt { color: #B00040 } /* Keyword.Type */
.grokdoc .m { color: #666666 } /* Literal.Number */
.grokdoc .s { color: #BA2121 } /* Literal.String */
.grokdoc .na { color: #7D9029 } /* Name.Attribute */
.grokdoc .nb { color: #008000 } /* Name.Builtin */
.grokdoc .nc { color: #0000FF; font-weight: bold } /* Name.Class */
.grokdoc .no { color: #880000 } /* Name.Constant */
.grokdoc .nd { color: #AA22FF } /* Name.Decorator */
.grokdoc .ni { color: #999999; font-weight: bold } /* Name.Entity */
.grokdoc .ne { color: #D2413A; font-weight: bold } /* Name.Exception */
.grokdoc .nf { color: #0000FF } /* Name.Function */
.grokdoc .nl { color: #A0A000 } /* Name.Label */
.grokdoc .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
.grokdoc .nt { color: #008000; font-weight: bold } /* Name.Tag */
.grokdoc .nv { color: #19177C } /* Name.Variable */
.grokdoc .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */
.grokdoc .w { color: #bbbbbb } /* Text.Whitespace */
.grokdoc .mf { color: #666666 } /* Literal.Number.Float */
.grokdoc .mh { color: #666666 } /* Literal.Number.Hex */
.grokdoc .mi { color: #666666 } /* Literal.Number.Integer */
.grokdoc .mo { color: #666666 } /* Literal.Number.Oct */
.grokdoc .sb { color: #BA2121 } /* Literal.String.Backtick */
.grokdoc .sc { color: #BA2121 } /* Literal.String.Char */
.grokdoc .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */
.grokdoc .s2 { color: #BA2121 } /* Literal.String.Double */
.grokdoc .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */
.grokdoc .sh { color: #BA2121 } /* Literal.String.Heredoc */
.grokdoc .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */
.grokdoc .sx { color: #008000 } /* Literal.String.Other */
.grokdoc .sr { color: #BB6688 } /* Literal.String.Regex */
.grokdoc .s1 { color: #BA2121 } /* Literal.String.Single */
.grokdoc .ss { color: #19177C } /* Literal.String.Symbol */
.grokdoc .bp { color: #008000 } /* Name.Builtin.Pseudo */
.grokdoc .vc { color: #19177C } /* Name.Variable.Class */
.grokdoc .vg { color: #19177C } /* Name.Variable.Global */
.grokdoc .vi { color: #19177C } /* Name.Variable.Instance */
.grokdoc .il { color: #666666 } /* Literal.Number.Integer.Long */

6
doc-src/transparent/index.py
View File

@ -1,6 +0,0 @@
from countershape import Page
pages = [
Page("osx.html", "OSX"),
Page("linux.html", "Linux"),
]

53
doc-src/transparent/linux.html
View File

@ -1,53 +0,0 @@
On Linux, mitmproxy integrates with the iptables redirection mechanism to
achieve transparent mode.
<ol class="tlist">
<li> <a href="@!urlTo('certinstall.html')!@">Install the mitmproxy
certificates on the test device</a>. </li>
<li> Enable IP forwarding:
<pre class="terminal">sysctl -w net.ipv4.ip_forward=1</pre>
You may also want to consider enabling this permanently in
<b>/etc/sysctl.conf</b>.
</li>
<li> If your target machine is on the same physical network and you configured it to use a custom gateway,
disable ICMP redirects:
<pre class="terminal">echo 0 | sudo tee /proc/sys/net/ipv4/conf/*/send_redirects</pre>
You may also want to consider enabling this permanently in
<b>/etc/sysctl.conf</b> as demonstrated <a href="http://unix.stackexchange.com/a/58081">here</a>.
</li>
<li> Create an iptables ruleset that redirects the desired traffic to the
mitmproxy port. Details will differ according to your setup, but the
ruleset should look something like this:
<pre class="terminal">iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080</pre>
</li>
<li> Fire up mitmproxy. You probably want a command like this:
<pre class="terminal">mitmproxy -T --host</pre>
The <b>-T</b> flag turns on transparent mode, and the <b>--host</b>
argument tells mitmproxy to use the value of the Host header for URL
display.
</li>
<li> Finally, configure your test device to use the host on which mitmproxy is
running as the default gateway.</li>
</ol>
For a detailed walkthrough, have a look at the <a href="@!urlTo('tutorials/transparent-dhcp.html')!@"><i>Transparently proxify virtual machines</i></a> tutorial.

78
doc-src/transparent/osx.html
View File

@ -1,78 +0,0 @@
OSX Lion integrated the [pf](http://www.openbsd.org/faq/pf/) packet filter from
the OpenBSD project, which mitmproxy uses to implement transparent mode on OSX.
Note that this means we don't support transparent mode for earlier versions of
OSX.
<ol class="tlist">
<li> <a href="@!urlTo('certinstall.html')!@">Install the mitmproxy
certificates on the test device</a>. </li>
<li> Enable IP forwarding:
<pre class="terminal">sudo sysctl -w net.inet.ip.forwarding=1</pre>
</li>
<li> Place the following two lines in a file called, say, <b>pf.conf</b>:
<pre class="terminal">rdr on en2 inet proto tcp to any port 80 -&gt; 127.0.0.1 port 8080
rdr on en2 inet proto tcp to any port 443 -&gt; 127.0.0.1 port 8080
</pre>
These rules tell pf to redirect all traffic destined for port 80 or 443
to the local mitmproxy instance running on port 8080. You should
replace <b>en2</b> with the interface on which your test device will
appear.
</li>
<li> Configure pf with the rules:
<pre class="terminal">sudo pfctl -f pf.conf</pre>
</li>
<li> And now enable it:
<pre class="terminal">sudo pfctl -e</pre>
</li>
<li> Configure sudoers to allow mitmproxy to access pfctl. Edit the file
<b>/etc/sudoers</b> on your system as root. Add the following line to the end
of the file:
<pre>ALL ALL=NOPASSWD: /sbin/pfctl -s state</pre>
Note that this allows any user on the system to run the command
"/sbin/pfctl -s state" as root without a password. This only allows
inspection of the state table, so should not be an undue security risk. If
you're special feel free to tighten the restriction up to the user running
mitmproxy.</li>
<li> Fire up mitmproxy. You probably want a command like this:
<pre class="terminal">mitmproxy -T --host</pre>
The <b>-T</b> flag turns on transparent mode, and the <b>--host</b>
argument tells mitmproxy to use the value of the Host header for URL
display.
</li>
<li> Finally, configure your test device to use the host on which mitmproxy is
running as the default gateway.</li>
</ol>
Note that the **rdr** rules in the pf.conf given above only apply to inbound
traffic. This means that they will NOT redirect traffic coming from the box
running pf itself. We can't distinguish between an outbound connection from a
non-mitmproxy app, and an outbound connection from mitmproxy itself - if you
want to intercept your OSX traffic, you should use an external host to run
mitmproxy. None the less, pf is flexible to cater for a range of creative
possibilities, like intercepting traffic emanating from VMs. See the
**pf.conf** man page for more.

119
doc-src/tutorials/gamecenter.html
View File

@ -1,119 +0,0 @@
## The setup
In this tutorial, I'm going to show you how simple it is to creatively
interfere with Apple Game Center traffic using mitmproxy. To set things up,
install the [mitmproxy root certificate](@!urlTo("certinstall.html")!@). Then
start mitmproxy on your desktop, and confige the iPhone to use it as a proxy.
## Taking a look at the Game Center traffic
Lets take a first look at the Game Center traffic. The game I'll use in this
tutorial is [Super Mega
Worm](http://itunes.apple.com/us/app/super-mega-worm/id388541990?mt=8) - a
great little retro-apocalyptic sidescroller for the iPhone:
<center>
<img class="img-responsive" src="@!urlTo('tutorials/supermega.png')!@"/>
</center>
After finishing a game (take your time), watch the traffic flowing through
mitmproxy:
<center>
<img class="img-responsive" src="@!urlTo('tutorials/one.png')!@"/>
</center>
We see a bunch of things we might expect - initialisation, the retrieval of
leaderboards and so forth. Then, right at the end, there's a POST to this
tantalising URL:
<pre>
https://service.gc.apple.com/WebObjects/GKGameStatsService.woa/wa/submitScore
</pre>
The contents of the submission are particularly interesting:
<!--(block|syntax("xml"))-->
<plist version="1.0">
<dict>
<key>scores</key>
<array>
<dict>
<key>category</key>
<string>SMW_Adv_USA1</string>
<key>context</key>
<integer>0</integer>
<key>score-value</key>
<integer>0</integer>
<key>timestamp</key>
<integer>1363515361321</integer>
</dict>
</array>
</dict>
</plist>
<!--(end)-->
This is a [property list](http://en.wikipedia.org/wiki/Property_list),
containing an identifier for the game, a score (55, in this case), and a
timestamp. Looks pretty simple to mess with.
## Modifying and replaying the score submission
Lets edit the score submission. First, select it in mitmproxy, then press
__enter__ to view it. Make sure you're viewing the request, not the response -
you can use __tab__ to flick between the two. Now press __e__ for edit. You'll
be prompted for the part of the request you want to change - press __r__ for
raw body. Your preferred editor (taken from the EDITOR environment variable) will
now fire up. Lets bump the score up to something a bit more ambitious:
<!--(block|syntax("xml"))-->
<plist version="1.0">
<dict>
<key>scores</key>
<array>
<dict>
<key>category</key>
<string>SMW_Adv_USA1</string>
<key>context</key>
<integer>0</integer>
<key>score-value</key>
<integer>2200272667</integer>
<key>timestamp</key>
<integer>1363515361321</integer>
</dict>
</array>
</dict>
</plist>
<!--(end)-->
Save the file and exit your editor.
The final step is to replay this modified request. Simply press __r__ for
replay.
## The glorious result and some intrigue
<center>
<img class="img-responsive" src="@!urlTo('tutorials/leaderboard.png')!@"/>
</center>
And that's it - according to the records, I am the greatest Super Mega Worm
player of all time.
There's a curious addendum to this tale. When I first wrote this tutorial, all
the top competitors' scores were the same: 2,147,483,647 (this is no longer the
case, beacause there are now so many fellow cheaters using this tutorial). If
you think that number seems familiar, you're right: it's 2^31-1, the maximum
value you can fit into a signed 32-bit int. Now let me tell you another
peculiar thing about Super Mega Worm - at the end of every game, it submits
your highest previous score to the Game Center, not your current score. This
means that it stores your highscore somewhere, and I'm guessing that it reads
that stored score back into a signed integer. So, if you _were_ to cheat by the
relatively pedestrian means of modifying the saved score on your jailbroken
phone, then 2^31-1 might well be the maximum score you could get. Then again,
if the game itself stores its score in a signed 32-bit int, you could get the
same score through perfect play, effectively beating the game. So, which is it
in this case? I'll leave that for you to decide.

7
doc-src/tutorials/index.py
View File

@ -1,7 +0,0 @@
from countershape import Page
pages = [
Page("30second.html", "Client playback: a 30 second example"),
Page("gamecenter.html", "Setting highscores on Apple's GameCenter"),
Page("transparent-dhcp.html", "Transparently proxify virtual machines")
]

53
doc-src/tutorials/transparent-dhcp.html
View File

@ -1,53 +0,0 @@
This walkthrough illustrates how to set up transparent proxying with mitmproxy. We use VirtualBox VMs with an Ubuntu proxy machine in this example, but the general principle can be applied to other setups.
1. **Configure VirtualBox Network Adapters for the proxy machine**
The network setup is simple: `internet <--> proxy vm <--> (virtual) internal network`.
For the proxy machine, *eth0* represents the outgoing network. *eth1* is connected to the internal network that will be proxified, using a static ip (192.168.3.1).
<hr>VirtualBox configuration:
<img class="img-responsive" src="@!urlTo('tutorials/transparent-dhcp/step1_vbox_eth0.png')!@"/><br><br>
<img class="img-responsive" src="@!urlTo('tutorials/transparent-dhcp/step1_vbox_eth1.png')!@"/>
<br>Proxy VM:
<img class="img-responsive" src="@!urlTo('tutorials/transparent-dhcp/step1_proxy.png')!@"/>
<hr>
2. **Configure DHCP and DNS**
We use dnsmasq to provide DHCP and DNS in our internal network.
Dnsmasq is a lightweight server designed to provide DNS (and optionally DHCP and TFTP) services to a small-scale
network.
- Before we get to that, we need to fix some Ubuntu quirks:
**Ubuntu >12.04** runs an internal dnsmasq instance (listening on loopback only) by default
<a href="https://www.stgraber.org/2012/02/24/dns-in-ubuntu-12-04/">[1]</a>. For our use case, this needs to be
disabled by changing <br>`dns=dnsmasq` to `#dns=dnsmasq` in */etc/NetworkManager/NetworkManager.conf*
and running `sudo restart network-manager` afterwards.
- Now, dnsmasq can be be installed and configured:
`sudo apt-get install dnsmasq`
Replace */etc/dnsmasq.conf* with the following configuration:
<pre>\# Listen for DNS requests on the internal network
interface=eth1
\# Act as a DHCP server, assign IP addresses to clients
dhcp-range=192.168.3.10,192.168.3.100,96h
\# Broadcast gateway and dns server information
dhcp-option=option:router,192.168.3.1
dhcp-option=option:dns-server,192.168.3.1
</pre>
Apply changes:
`sudo service dnsmasq restart`
<hr>
Your proxied machine's network settings should now look similar to this:
<img class="img-responsive" src="@!urlTo('tutorials/transparent-dhcp/step2_proxied_vm.png')!@"/>
<hr>
3. **Set up traffic redirection to mitmproxy**
To redirect traffic to mitmproxy, we need to add two iptables rules:
<pre class="terminal">
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 \
-j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 \
-j REDIRECT --to-port 8080
</pre>
4. If required, <a href="@!urlTo('certinstall.html')!@">install the mitmproxy
certificates on the test device</a>.
5. Finally, we can run <code>mitmproxy -T</code>.
The proxied machine cannot to leak any data outside of HTTP or DNS requests.

1
docs/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
_build/

195
docs/Makefile Normal file
View File

@ -0,0 +1,195 @@
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " applehelp to make an Apple Help Book"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
@echo " coverage to run coverage check of the documentation (if enabled)"
clean:
rm -rf $(BUILDDIR)/*
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/mitmproxy.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/mitmproxy.qhc"
applehelp:
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
@echo
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
@echo "N.B. You won't be able to view it unless you put it in" \
"~/Library/Documentation/Help or install it in your application" \
"bundle."
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/mitmproxy"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/mitmproxy"
@echo "# devhelp"
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
coverage:
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
@echo "Testing of coverage in the sources finished, look at the " \
"results in $(BUILDDIR)/coverage/python.txt."
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
livehtml:
sphinx-autobuild -b html -z '../libmproxy' -r '___jb_(old|bak)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html

0
doc-src/certinstall-webapp.png → docs/certinstall-webapp.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 60 KiB

After

Width:  |  Height:  |  Size: 60 KiB

174
docs/certinstall.rst Normal file
View File

@ -0,0 +1,174 @@
.. _certinstall:
About Certificates
==================
Introduction
------------
Mitmproxy can decrypt encrypted traffic on the fly, as long as the client
trusts its built-in certificate authority. Usually this means that the
mitmproxy CA certificates have to be installed on the client device.
Quick Setup
-----------
By far the easiest way to install the mitmproxy certificates is to use the
built-in certificate installation app. To do this, just start mitmproxy and
configure your target device with the correct proxy settings. Now start a
browser on the device, and visit the magic domain **mitm.it**. You should see
something like this:
.. image:: certinstall-webapp.png
Click on the relevant icon, follow the setup instructions for the platform
you're on and you are good to go.
Installing the mitmproxy CA certificate manually
------------------------------------------------
Sometimes using the quick install app is not an option - Java or the iOS
Simulator spring to mind - or you just need to do it manually for some other
reason. Below is a list of pointers to manual certificate installation
documentation for some common platforms.
The mitmproxy CA cert is located in ``~/.mitmproxy`` after it has been generated at the first
start of mitmproxy.
iOS
^^^
http://kb.mit.edu/confluence/pages/viewpage.action?pageId=152600377
iOS Simulator
^^^^^^^^^^^^^
See https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore
Java
^^^^
See http://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html
Android/Android Simulator
^^^^^^^^^^^^^^^^^^^^^^^^^
See http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets
Windows
^^^^^^^
See http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7
Windows (automated)
^^^^^^^^^^^^^^^^^^^
>>> certutil.exe -importpfx mitmproxy-ca-cert.p12
See also: https://technet.microsoft.com/en-us/library/cc732443.aspx
Mac OS X
^^^^^^^^
See https://support.apple.com/kb/PH7297?locale=en_US
Ubuntu/Debian
^^^^^^^^^^^^^
See http://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861
Mozilla Firefox
^^^^^^^^^^^^^^^
See https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox
Chrome on Linux
^^^^^^^^^^^^^^^
See https://code.google.com/p/chromium/wiki/LinuxCertManagement
More on mitmproxy certificates
------------------------------
The first time **mitmproxy** or **mitmdump** is run, the mitmproxy Certificate
Authority (CA) is created in the config directory (``~/.mitmproxy`` by default).
This CA is used for on-the-fly generation of dummy certificates for each of the
SSL sites that your client visits. Since your browser won't trust the
mitmproxy CA out of the box , you will see an SSL certificate warning every
time you visit a new SSL domain through mitmproxy. When you are testing a
single site through a browser, just accepting the bogus SSL cert manually is
not too much trouble, but there are a many circumstances where you will want to
configure your testing system or browser to trust the mitmproxy CA as a
signing root authority. For security reasons, the mitmproxy CA is generated uniquely on the first
start and is not shared between mitmproxy installations on different devices.
CA and cert files
-----------------
The files created by mitmproxy in the .mitmproxy directory are as follows:
===================== ==========================================================================
mitmproxy-ca.pem The certificate **and the private key** in PEM format.
mitmproxy-ca-cert.pem The certificate in PEM format.
Use this to distribute on most non-Windows platforms.
mitmproxy-ca-cert.p12 The certificate in PKCS12 format. For use on Windows.
mitmproxy-ca-cert.cer Same file as .pem, but with an extension expected by some Android devices.
===================== ==========================================================================
Using a custom certificate
--------------------------
You can use your own certificate by passing the ``--cert`` option to
mitmproxy. Mitmproxy then uses the provided certificate for interception of the
specified domains instead of generating a certificate signed by its own CA.
The certificate file is expected to be in the PEM format. You can include
intermediary certificates right below your leaf certificate, so that you PEM
file roughly looks like this:
.. code-block:: none
-----BEGIN PRIVATE KEY-----
<private key>
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
<cert>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<intermediary cert (optional)>
-----END CERTIFICATE-----
For example, you can generate a certificate in this format using these instructions:
>>> openssl genrsa -out cert.key 2048
>>> openssl req -new -x509 -key cert.key -out cert.crt
(Specify the mitm domain as Common Name, e.g. *.google.com)
>>> cat cert.key cert.crt > cert.pem
>>> mitmproxy --cert=cert.pem
Using a custom certificate authority
------------------------------------
By default, mitmproxy will use ``~/.mitmproxy/mitmproxy-ca.pem`` as
the certificate authority to generate certificates for all domains for which no
custom certificate is provided (see above). You can use your own certificate
authority by passing the ``--confdir`` option to mitmproxy. Mitmproxy
will then look for ``mitmproxy-ca.pem`` in the specified directory. If
no such file exists, it will be generated automatically.
Using a client side certificate
-------------------------------
You can use a client certificate by passing the ``--client-certs DIRECTORY`` option to mitmproxy.
If you visit example.org, mitmproxy looks for a file named ``example.org.pem`` in the specified
directory and uses this as the client cert. The certificate file needs to be in the PEM format and
should contain both the unencrypted private key and the certificate.

216
docs/conf.py Normal file
View File

@ -0,0 +1,216 @@
# -*- coding: utf-8 -*-
#
# mitmproxy documentation build configuration file, created by
# sphinx-quickstart on Thu Sep 03 14:04:13 2015.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
import shlex
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('..'))
import libmproxy.version
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.viewcode',
'sphinx.ext.napoleon',
'sphinxcontrib.documentedlist'
]
autodoc_member_order = "bysource"
# Add any paths that contain templates here, relative to this directory.
#templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'mitmproxy docs'
copyright = u'2015, the mitmproxy project'
author = u'The mitmproxy project'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = libmproxy.version.VERSION
# The full version, including alpha/beta/rc tags.
release = libmproxy.version.VERSION
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['libmproxy.']
# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {
# 'logo_only': True,
}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
html_title = "mitmproxy %s documentation" % version
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
html_logo = "mitmproxy-long.png"
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
html_favicon = "favicon.ico"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages:
# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
#html_search_language = 'en'
# A dictionary with options for the search language support, empty by default.
# Now only 'ja' uses this config value
#html_search_options = {'type': 'default'}
# The name of a javascript file (relative to the configuration directory) that
# implements a search results scorer. If empty, the default will be used.
#html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'mitmproxydoc'

86
docs/config.rst Normal file
View File

@ -0,0 +1,86 @@
.. _config:
Configuration
=============
Mitmproxy is configured through a set of files in the users ~/.mitmproxy
directory.
mitmproxy.conf
Settings for the :program:`mitmproxy`. This file can contain any options supported by
mitmproxy.
mitmdump.conf
Settings for the :program:`mitmdump`. This file can contain any options supported by mitmdump.
common.conf
Settings shared between all command-line tools. Settings in this file are over-ridden by those
in the tool-specific files. Only options shared by mitmproxy and mitmdump should be used in
this file.
Syntax
------
Comments
^^^^^^^^
.. code-block:: none
# this is a comment
; this is also a comment (.ini style)
--- and this is a comment too (yaml style)
Key/Value pairs
^^^^^^^^^^^^^^^
- Keys and values are case-sensitive
- Whitespace is ignored
- Lists are comma-delimited, and enclosed in square brackets
.. code-block:: none
name = value # (.ini style)
name: value # (yaml style)
--name value # (command-line option style)
fruit = [apple, orange, lemon]
indexes = [1, 12, 35 , 40]
Flags
^^^^^
These are boolean options that take no value but true/false.
.. code-block:: none
name = true # (.ini style)
name
--name # (command-line option style)
Options
-------
The options available in the config files are precisely those available as
command-line flags, with the key being the option's long name. To get a
complete list of these, use the :option:`--help` option on each of the tools. Be
careful to only specify common options in the **common.conf** file -
unsupported options in this file will be detected as an error on startup.
Examples
--------
common.conf
^^^^^^^^^^^
Note that :option:`--port` is an option supported by all tools.
.. code-block:: none
port = 8080
mitmproxy.conf
^^^^^^^^^^^^^^
.. code-block:: none
palette = light

0
doc-src/custom-routing.txt → docs/custom-routing.txt
View File

0
doc-src/dev/addingviews.html → docs/dev/addingviews.html
View File

14
docs/dev/architecture.rst Normal file
View File

@ -0,0 +1,14 @@
.. _architecture:
Architecture
============
To give you a better understanding of how mitmproxy works, mitmproxy's
high-level architecture is detailed in the following graphic:
.. image:: ../schematics/architecture.png
:download:`architecture.pdf <../schematics/architecture.pdf>`
Please don't refrain from asking any further
questions on the mailing list, the Slack channel or the GitHub issue tracker.

9
docs/dev/exceptions.rst Normal file
View File

@ -0,0 +1,9 @@
.. _exceptions:
Exceptions
==========
.. automodule:: libmproxy.exceptions
:show-inheritance:
:members:
:undoc-members:

25
docs/dev/models.rst Normal file
View File

@ -0,0 +1,25 @@
.. _models:
Models
======
.. warning::
The documentation for models has not been converted to rst yet and **many attributes/features
are missing**.
Please read the source code instead.
.. automodule:: libmproxy.models
:show-inheritance:
:members:
:undoc-members:
.. automodule:: netlib.http.semantics
:members: Request, Response
:undoc-members:
.. autoclass:: Headers
:show-inheritance:
:members:
:special-members:
:no-undoc-members:

15
docs/dev/protocols.rst Normal file
View File

@ -0,0 +1,15 @@
.. _protocols:
Protocols
=========
.. automodule:: libmproxy.protocol
.. autoclass:: Layer
:members:
:special-members:
.. autoclass:: ServerConnectionMixin
:members:
.. autoexception:: Kill

12
docs/dev/proxy.rst Normal file
View File

@ -0,0 +1,12 @@
.. _proxy:
Proxy Server
============
.. automodule:: libmproxy.proxy
.. autoclass:: ProxyServer
.. autoclass:: DummyServer
.. autoclass:: ProxyConfig
.. autoclass:: RootContext
:members:

14
docs/dev/sslkeylogfile.rst Normal file
View File

@ -0,0 +1,14 @@
.. _sslkeylogfile:
TLS Master Secrets
==================
The SSL master keys can be logged by mitmproxy so that external programs can decrypt TLS
connections both from and to the proxy. Key logging is enabled by setting the environment variable
:envvar:`SSLKEYLOGFILE` so that it points to a writable text file.
Recent versions of WireShark can use these log files to decrypt packets.
You can specify the key file path in WireShark via
:samp:`Edit -> Preferences -> Protocols -> SSL -> (Pre)-Master-Secret log filename`.
Note that :envvar:`SSLKEYLOGFILE` is respected by other programs as well, e.g. Firefox and Chrome.
If this creates any issues, you can set :envvar:`MITMPROXY_SSLKEYLOGFILE` alternatively.

46
docs/dev/testing.rst Normal file
View File

@ -0,0 +1,46 @@
.. _testing:
Testing
=======
All the mitmproxy projects strive to maintain 100% code coverage. In general,
patches and pull requests will be declined unless they're accompanied by a
suitable extension to the test suite.
Our tests are written for the nose_ test framework.
At the point where you send your pull request, a command like this:
>>> nosetests --with-cov --cov-report term-missing ./test
Should give output something like this:
.. code-block:: none
> ---------- coverage: platform darwin, python 2.7.2-final-0 --
> Name Stmts Miss Cover Missing
> ----------------------------------------------------
> libmproxy/__init__ 0 0 100%
> libmproxy/app 4 0 100%
> libmproxy/cmdline 100 0 100%
> libmproxy/controller 69 0 100%
> libmproxy/dump 150 0 100%
> libmproxy/encoding 39 0 100%
> libmproxy/filt 201 0 100%
> libmproxy/flow 891 0 100%
> libmproxy/proxy 427 0 100%
> libmproxy/script 27 0 100%
> libmproxy/utils 133 0 100%
> libmproxy/version 4 0 100%
> ----------------------------------------------------
> TOTAL 2045 0 100%
> ----------------------------------------------------
> Ran 251 tests in 11.864s
There are exceptions to the coverage requirement - for instance, much of the
console interface code can't sensibly be unit tested. These portions are
excluded from coverage analysis either in the **.coveragerc** file, or using
**#pragma no-cover** directives. To keep our coverage analysis relevant, we use
these measures as sparingly as possible.
.. _nose: https://nose.readthedocs.org/en/latest/

BIN
docs/favicon.ico Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.3 KiB

15
docs/features/anticache.rst Normal file
View File

@ -0,0 +1,15 @@
.. _anticache:
Anticache
=========
When the :option:`--anticache` option is passed to mitmproxy, it removes headers
(``if-none-match`` and ``if-modified-since``) that might elicit a
``304 not modified`` response from the server. This is useful when you want to make
sure you capture an HTTP exchange in its totality. It's also often used during
:ref:`clientreplay`, when you want to make sure the server responds with complete data.
================== ======================
command-line :option:`--anticache`
mitmproxy shortcut :kbd:`o` then :kbd:`a`
================== ======================

22
doc-src/features/clientreplay.html → docs/features/clientreplay.rst
View File

@ -1,3 +1,7 @@
.. _clientreplay:
Client-side replay
==================
Client-side replay does what it says on the tin: you provide a previously saved
HTTP conversation, and mitmproxy replays the client requests one by one. Note
@ -6,17 +10,9 @@ before starting the next request. This might differ from the recorded
conversation, where requests may have been made concurrently.
You may want to use client-side replay in conjunction with the
[anticache](@!urlTo("anticache.html")!@) option, to make sure the server
responds with complete data.
:ref:`anticache` option, to make sure the server responds with complete data.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>-c path</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>c</b></td>
</tr>
</tbody>
</table>
================== =================
command-line :option:`-c path`
mitmproxy shortcut :kbd:`c`
================== =================

31
doc-src/features/filters.html → docs/features/filters.rst
View File

@ -1,36 +1,39 @@
.. _filters:
Many commands in __mitmproxy__ and __mitmdump__ take a filter expression.
Filter expressions
==================
Many commands in :program:`mitmproxy` and :program:`mitmdump` take a filter expression.
Filter expressions consist of the following operators:
<table class="table">
<tbody>
<!--(for i in filt_help)-->
<tr>
<td class="filt_cmd">@!i[0]!@</td>
<td class="filt_help">@!i[1]!@</td>
</tr>
<!--(end)-->
</tbody>
</table>
.. documentedlist::
:header: "Expression" "Description"
:listobject: libmproxy.filt.help
- Regexes are Python-style
- Regexes can be specified as quoted strings
- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value".
- Strings with no operators are matched against the request URL.
- The default binary operator is &amp;.
- The default binary operator is &.
Examples
========
--------
URL containing "google.com":
.. code-block:: none
google\.com
Requests whose body contains the string "test":
.. code-block:: none
~q ~b test
Anything but requests with a text/html content type:
!(~q & ~t \"text/html\")
.. code-block:: none
!(~q & ~t "text/html")

97
docs/features/passthrough.rst Normal file
View File

@ -0,0 +1,97 @@
.. _passthrough:
Ignore Domains
==============
There are two main reasons why you may want to exempt some traffic from mitmproxy's interception
mechanism:
- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and
mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or
the Apple App Store fail to work if mitmproxy is active.
- **Convenience:** You really don't care about some parts of the traffic and just want them to go
away.
If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy`
feature.
If you want to ignore traffic from mitmproxy's processing because of large response bodies,
take a look at the :ref:`responsestreaming` feature.
How it works
------------
================== =============================
command-line :option:`--ignore regex`
mitmproxy shortcut :kbd:`o` then :kbd:`I`
================== =============================
mitmproxy allows you to specify a regex which is matched against a ``host:port`` string
(e.g. "example.com:443") to determine hosts that should be excluded.
There are two important quirks to consider:
- **In transparent mode, the ignore pattern is matched against the IP.** While we usually infer the
hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not
have access to this information before the SSL handshake.
- In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is
applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections.
Tutorial
--------
If you just want to ignore one specific domain, there's usually a bulletproof method to do so:
1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port``
information in the serverconnect messages. mitmproxy will filter on these.
2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.)
and use this as your ignore pattern:
.. code-block:: none
:emphasize-lines: 6,7,9
>>> mitmdump -v
127.0.0.1:50588: clientconnect
127.0.0.1:50588: request
-> CONNECT example.com:443 HTTP/1.1
127.0.0.1:50588: Set new server address: example.com:443
127.0.0.1:50588: serverconnect
-> example.com:443
^C
>>> mitmproxy --ignore ^example\.com:443$
Here are some other examples for ignore patterns:
.. code-block:: none
# Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
--ignore apple.com:443
# "Correct" version without false-positives:
--ignore '^(.+\.)?apple\.com:443$'
# Ignore example.com, but not its subdomains:
--ignore '^example.com:'
# Ignore everything but example.com and mitmproxy.org:
--ignore '^(?!example\.com)(?!mitmproxy\.org)'
# Transparent mode:
--ignore 17\.178\.96\.59:443
# IP address range:
--ignore 17\.178\.\d+\.\d+:443
.. seealso::
- :ref:`tcpproxy`
- :ref:`responsestreaming`
.. rubric:: Footnotes
.. [#explicithttp] This stems from an limitation of explicit HTTP proxying:
A single connection can be re-used for multiple target domains - a
``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the
same connection. If we start to ignore the connection after the first request,
we would miss the relevant second one.
.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning

17
docs/features/proxyauth.rst Normal file
View File

@ -0,0 +1,17 @@
.. _proxyauth:
Proxy Authentication
====================
Asks the user for authentication before they are permitted to use the proxy.
Authentication headers are stripped from the flows, so they are not passed to
upstream servers. For now, only HTTP Basic authentication is supported. The
proxy auth options are not compatible with the transparent, socks or reverse proxy
mode.
================== =============================
command-line :option:`--nonanonymous`,
:option:`--singleuser USER`,
:option:`--htpasswd PATH`
================== =============================

56
doc-src/features/replacements.html → docs/features/replacements.rst
View File

@ -1,3 +1,8 @@
.. _replacements:
Replacements
============
Mitmproxy lets you specify an arbitrary number of patterns that define text
replacements within flows. Each pattern has 3 components: a filter that defines
which flows a replacement applies to, a regular expression that defines what
@ -19,56 +24,49 @@ mitmproxy's own interception abilities, replacement hooks can be an amazingly
flexible and powerful feature.
## On the command-line
On the command-line
-------------------
The replacement hook command-line options use a compact syntax to make it easy
to specify all three components at once. The general form is as follows:
.. code-block:: none
/patt/regex/replacement
Here, __patt__ is a mitmproxy filter expression, __regex__ is a valid Python
regular expression, and __replacement__ is a string literal. The first
character in the expression (__/__ in this case) defines what the separation
Here, **patt** is a mitmproxy filter expression, **regex** is a valid Python
regular expression, and **replacement** is a string literal. The first
character in the expression (``/`` in this case) defines what the separation
character is. Here's an example of a valid expression that replaces "foo" with
"bar" in all requests:
.. code-block:: none
:~q:foo:bar
In practice, it's pretty common for the replacement literal to be long and
complex. For instance, it might be an XSS exploit that weighs in at hundreds or
thousands of characters. To cope with this, there's a variation of the
replacement hook specifier that lets you load the replacement text from a file.
So, you might start __mitmdump__ as follows:
So, you might start **mitmdump** as follows:
<pre class="terminal">
mitmdump --replace-from-file :~q:foo:~/xss-exploit
</pre>
>>> mitmdump --replace-from-file :~q:foo:~/xss-exploit
This will load the replacement text from the file __~/xss-exploit__.
This will load the replacement text from the file ``~/xss-exploit``.
Both the _--replace_ and _--replace-from-file_ flags can be passed multiple
Both the :option:`--replace` and :option:`--replace-from-file` flags can be passed multiple
times.
## Interactively
Interactively
-------------
The _R_ shortcut key in the mitmproxy options menu (_o_) lets you add and edit
replacement hooks using a built-in editor. The context-sensitive help (_?_) has
The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`o`) lets you add and edit
replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has
complete usage information.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th>
<td>
<ul>
<li>--replace</li>
<li>--replace-from-file</li>
</ul>
</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>R</b></td>
</tr>
</tbody>
</table>
================== =============================
command-line :option:`--replace`,
:option:`--replace-from-file`
mitmproxy shortcut :kbd:`o` then :kbd:`R`
================== =============================

69
docs/features/responsestreaming.rst Normal file
View File

@ -0,0 +1,69 @@
.. _responsestreaming:
Response Streaming
==================
By using mitmproxy's streaming feature, response contents can be passed to the client incrementally
before they have been fully received by the proxy. This is especially useful for large binary files
such as videos, where buffering the whole file slows down the client's browser.
By default, mitmproxy will read the entire response, perform any indicated
manipulations on it and then send the (possibly modified) response to
the client. In some cases this is undesirable and you may wish to "stream"
the reponse back to the client. When streaming is enabled, the response is
not buffered on the proxy but directly sent back to the client instead.
On the command-line
-------------------
Streaming can be enabled on the command line for all response bodies exceeding a certain size.
The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes.
================== =============================
command-line :option:`--stream SIZE`
================== =============================
.. warning::
When response streaming is enabled, **streamed response contents will not be
recorded or preserved in any way.**
.. note::
When response streaming is enabled, the response body cannot be modified by the usual means.
Customizing Response Streaming
------------------------------
You can also use an :ref:`inlinescripts` to customize exactly
which responses are streamed.
Responses that should be tagged for streaming by setting their ``.stream`` attribute to ``True``:
.. literalinclude:: ../../examples/stream.py
:caption: examples/stream.py
:language: python
Implementation Details
----------------------
When response streaming is enabled, portions of the code which would have otherwise performed
changes on the response body will see an empty response body instead
(:py:data:`netlib.http.CONTENT_MISSING`). Any modifications will be ignored.
Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a
``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time.
Modifying streamed data
-----------------------
If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all
chunks.
.. literalinclude:: ../../examples/stream_modify.py
:caption: examples/stream_modify.py
:language: python
.. seealso::
- :ref:`passthrough`

57
docs/features/reverseproxy.rst Normal file
View File

@ -0,0 +1,57 @@
.. _reverseproxy:
Reverse Proxy
=============
In reverse proxy mode, mitmproxy accepts standard HTTP requests and forwards
them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which
mitmproxy forwards HTTP proxy requests to an upstream proxy server.
================== =====================================
command-line :option:`-R http[s]://hostname[:port]`
================== =====================================
Here, **scheme** signifies if the proxy should use TLS to connect to the server.
mitmproxy always accepts both encrypted and unencrypted requests and transforms
them to what the server expects.
.. code-block:: none
>>> mitmdump -R https://httpbin.org -p 80
>>> curl http://localhost/
# requests will be transparently upgraded to TLS by mitmproxy
>>> mitmdump -R https://httpbin.org -p 443
>>> curl https://localhost/
# mitmproxy will use TLS on both ends.
Host Header
-----------
In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this
may lead to issues with public web servers. For example, consider the following scenario:
.. code-block:: none
:emphasize-lines: 5
>>> mitmdump -d -R http://example.com/
>>> curl http://localhost:8080/
>> GET https://example.com/
Host: localhost:8080
User-Agent: curl/7.35.0
[...]
<< 404 Not Found 345B
Since the Host header doesn't match "example.com", an error is returned.
There are two ways to solve this:
1. Modify the hosts file of your OS so that "example.com" resolves to your proxy's IP.
Then, access example.com directly. Make sure that your proxy can still resolve the original IP
or specify an IP in mitmproxy.
2. Use mitmproxy's :ref:`setheaders` feature to rewrite the host header:
``--setheader :~q:Host:example.com``.
However, keep in mind that absolute URLs within the returned document or HTTP redirects will
cause the client application to bypass the proxy.

26
doc-src/features/serverreplay.html → docs/features/serverreplay.rst
View File

@ -1,6 +1,7 @@
.. _serverreplay:
- command-line: _-S path_
- mitmproxy shortcut: _S_
Server-side replay
==================
Server-side replay lets us replay server responses from a saved HTTP
conversation.
@ -8,12 +9,12 @@ conversation.
Matching requests with responses
--------------------------------
By default, __mitmproxy__ excludes request headers when matching incoming
By default, :program:`mitmproxy` excludes request headers when matching incoming
requests with responses from the replay file. This works in most circumstances,
and makes it possible to replay server responses in situations where request
headers would naturally vary, e.g. using a different user agent. The _--rheader
headername_ command-line option allows you to override this behaviour by
specifying individual headers that should be included in matching.
headers would naturally vary, e.g. using a different user agent.
The :option:`--rheader headername` command-line option allows you to override
this behaviour by specifying individual headers that should be included in matching.
Response refreshing
@ -22,14 +23,17 @@ Response refreshing
Simply replaying server responses without modification will often result in
unexpected behaviour. For example cookie timeouts that were in the future at
the time a conversation was recorded might be in the past at the time it is
replayed. By default, __mitmproxy__ refreshes server responses before sending
them to the client. The __date__, __expires__ and __last-modified__ headers are
replayed. By default, :program:`mitmproxy` refreshes server responses before sending
them to the client. The **date**, **expires** and **last-modified** headers are
all updated to have the same relative time offset as they had at the time of
recording. So, if they were in the past at the time of recording, they will be
in the past at the time of replay, and vice versa. Cookie expiry times are
updated in a similar way.
You can turn off response refreshing using the _--norefresh_ argument, or using
the _o_ options shortcut within __mitmproxy__.
You can turn off response refreshing using the :option:`--norefresh` argument, or using
the :kbd:`o` options shortcut within :program:`mitmproxy`.
================== =================
command-line :option:`-S path`
mitmproxy shortcut :kbd:`S`
================== =================

19
docs/features/setheaders.rst Normal file
View File

@ -0,0 +1,19 @@
.. _setheaders:
Set Headers
===========
This feature lets you specify a set of headers to be added to requests or
responses, based on a filter pattern. You can specify these either on the
command-line, or through an interactive editor in mitmproxy.
Example: Set the **Host** header to "example.com" for all requests.
.. code-block:: none
mitmdump -R http://example.com --setheader :~q:Host:example.com
================== =============================
command-line :option:`--setheader PATTERN`
mitmproxy shortcut :kbd:`o` then :kbd:`H`
================== =============================

10
docs/features/socksproxy.rst Normal file
View File

@ -0,0 +1,10 @@
.. _socksproxy:
SOCKS Mode
==========
In this mode, mitmproxy acts as a SOCKS5 proxy server.
================== =================
command-line :option:`--socks`
================== =================

41
docs/features/sticky.rst Normal file
View File

@ -0,0 +1,41 @@
.. _sticky:
Sticky cookies and auth
=======================
Sticky cookies
--------------
When the sticky cookie option is set, __mitmproxy__ will add the cookie most
recently set by the server to any cookie-less request. Consider a service that
sets a cookie to track the session after authentication. Using sticky cookies,
you can fire up mitmproxy, and authenticate to a service as you usually would
using a browser. After authentication, you can request authenticated resources
through mitmproxy as if they were unauthenticated, because mitmproxy will
automatically add the session tracking cookie to requests. Among other things,
this lets you script interactions with authenticated resources (using tools
like wget or curl) without having to worry about authentication.
Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can
record the authentication process once, and simply replay it on startup every time you need
to interact with the secured resources.
================== ======================
command-line :option:`-t FILTER`
mitmproxy shortcut :kbd:`o` then :kbd:`t`
================== ======================
Sticky auth
-----------
The sticky auth option is analogous to the sticky cookie option, in that HTTP
**Authorization** headers are simply replayed to the server once they have been
seen. This is enough to allow you to access a server resource using HTTP Basic
authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support
replay of HTTP Digest authentication.
================== ======================
command-line :option:`-u FILTER`
mitmproxy shortcut :kbd:`o` then :kbd:`A`
================== ======================

31
docs/features/tcpproxy.rst Normal file
View File

@ -0,0 +1,31 @@
.. _tcpproxy:
TCP Proxy
=========
WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt
hostnames from processing, so that mitmproxy acts as a generic TCP forwarder.
This feature is closely related to the :ref:`passthrough` functionality,
but differs in two important aspects:
- The raw TCP messages are printed to the event log.
- SSL connections will be intercepted.
Please note that message interception or modification are not possible yet.
If you are not interested in the raw TCP messages, you should use the ignore domains feature.
How it works
------------
================== ======================
command-line :option:`--tcp HOST`
mitmproxy shortcut :kbd:`o` then :kbd:`T`
================== ======================
For a detailed description how the hostname pattern works, please look at the :ref:`passthrough`
feature.
.. seealso::
- :ref:`passthrough`
- :ref:`responsestreaming`

21
doc-src/features/upstreamcerts.html → docs/features/upstreamcerts.rst
View File

@ -1,7 +1,12 @@
.. _upstreamcerts:
Upstream Certificates
=====================
When mitmproxy receives a connection destined for an SSL-protected service, it
freezes the connection before reading its request data, and makes a connection
to the upstream server to "sniff" the contents of its SSL certificate. The
information gained - the __Common Name__ and __Subject Alternative Names__ - is
information gained - the **Common Name** and **Subject Alternative Names** - is
then used to generate the interception certificate, which is sent to the client
so the connection can continue.
@ -12,13 +17,7 @@ certs in transparent mode.
Upstream cert sniffing is on by default, and can optionally be turned off.
<table class="table">
<tbody>
<tr>
<th width="20%">command-line</th> <td>--no-upstream-cert</td>
</tr>
<tr>
<th>mitmproxy shortcut</th> <td><b>o</b> then <b>U</b></td>
</tr>
</tbody>
</table>
================== =============================
command-line :option:`--no-upstream-cert`
mitmproxy shortcut :kbd:`o` then :kbd:`U`
================== =============================

12
docs/features/upstreamproxy.rst Normal file
View File

@ -0,0 +1,12 @@
.. _upstreamproxy:
Upstream proxy mode
===================
In this mode, mitmproxy accepts proxy requests and unconditionally forwards all
requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`,
in which mitmproxy forwards ordinary HTTP requests to an upstream server.
================== ===================================
command-line :option:`-U http://hostname[:port]`
================== ===================================

310
doc-src/howmitmproxy.html → docs/howmitmproxy.rst
View File

@ -1,63 +1,47 @@
How mitmproxy works
===================
Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying
process works will help you deploy it creatively, and take into account its
fundamental assumptions and how to work around them. This document explains
mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted
explicit proxying, and working up to the most complicated interaction -
transparent proxying of SSL-protected traffic[^ssl] in the presence of
[SNI](http://en.wikipedia.org/wiki/Server_Name_Indication).
transparent proxying of SSL-protected traffic [#ssl]_ in the presence of `Server Name Indication`_.
<div class="page-header">
<h1>Explicit HTTP</h1>
</div>
Explicit HTTP
-------------
Configuring the client to use mitmproxy as an explicit proxy is the simplest
and most reliable way to intercept traffic. The proxy protocol is codified in
the [HTTP RFC](http://www.ietf.org/rfc/rfc2068.txt), so the behaviour of both
and most reliable way to intercept traffic. The proxy protocol is codified in the
`HTTP RFC`_, so the behaviour of both
the client and the server is well defined, and usually reliable. In the
simplest possible interaction with mitmproxy, a client connects directly to the
proxy, and makes a request that looks like this:
<pre>GET http://example.com/index.html HTTP/1.1</pre>
.. code-block:: http
GET http://example.com/index.html HTTP/1.1
This is a proxy GET request - an extended form of the vanilla HTTP GET request
that includes a schema and host specification, and it includes all the
information mitmproxy needs to proceed.
<img class="img-responsive" src="explicit.png"/>
.. image:: schematics/how-mitmproxy-works-explicit.png
:align: center
<table class="table">
<tbody>
<tr>
<td><b>1</b></td>
<td>The client connects to the proxy and makes a request.</td>
</tr>
<tr>
<td><b>2</b></td>
<td>Mitmproxy connects to the upstream server and simply forwards
the request on.</td>
</tr>
</tbody>
</table>
1. The client connects to the proxy and makes a request.
2. Mitmproxy connects to the upstream server and simply forwards the request on.
<div class="page-header">
<h1>Explicit HTTPS</h1>
</div>
Explicit HTTPS
--------------
The process for an explicitly proxied HTTPS connection is quite different. The
client connects to the proxy and makes a request that looks like this:
<pre>CONNECT example.com:443 HTTP/1.1</pre>
.. code-block:: http
CONNECT example.com:443 HTTP/1.1
A conventional proxy can neither view nor manipulate an SSL-encrypted data
stream, so a CONNECT request simply asks the proxy to open a pipe between the
@ -66,15 +50,15 @@ data in both directions without knowing anything about the contents. The
negotiation of the SSL connection happens over this pipe, and the subsequent
flow of requests and responses are completely opaque to the proxy.
## The MITM in mitmproxy
The MITM in mitmproxy
^^^^^^^^^^^^^^^^^^^^^
This is where mitmproxy's fundamental trick comes into play. The MITM in its
name stands for Man-In-The-Middle - a reference to the process we use to
intercept and interfere with these theoretically opaque data streams. The basic
idea is to pretend to be the server to the client, and pretend to be the client
to the server, while we sit in the middle decoding traffic from both sides. The
tricky part is that the [Certificate
Authority](http://en.wikipedia.org/wiki/Certificate_authority) system is
tricky part is that the `Certificate Authority`_ system is
designed to prevent exactly this attack, by allowing a trusted third-party to
cryptographically sign a server's SSL certificates to verify that they are
legit. If this signature doesn't match or is from a non-trusted party, a secure
@ -83,10 +67,11 @@ shortcomings of the CA system as it exists today, this is usually fatal to
attempts to MITM an SSL connection for analysis. Our answer to this conundrum
is to become a trusted Certificate Authority ourselves. Mitmproxy includes a
full CA implementation that generates interception certificates on the fly. To
get the client to trust these certificates, we [register mitmproxy as a trusted
CA with the device manually](@!urlTo("certinstall.html")!@).
get the client to trust these certificates, we :ref:`register mitmproxy as a trusted
CA with the device manually <certinstall>`.
## Complication 1: What's the remote hostname?
Complication 1: What's the remote hostname?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To proceed with this plan, we need to know the domain name to use in the
interception certificate - the client will verify that the certificate is for
@ -95,14 +80,16 @@ blush, it seems that the CONNECT request above gives us all we need - in this
example, both of these values are "example.com". But what if the client had
initiated the connection as follows:
<pre>CONNECT 10.1.1.1:443 HTTP/1.1</pre>
.. code-block:: http
CONNECT 10.1.1.1:443 HTTP/1.1
Using the IP address is perfectly legitimate because it gives us enough
information to initiate the pipe, even though it doesn't reveal the remote
hostname.
Mitmproxy has a cunning mechanism that smooths this over - [upstream
certificate sniffing](@!urlTo("features/upstreamcerts.html")!@). As soon as we
Mitmproxy has a cunning mechanism that smooths this over - :ref:`upstream
certificate sniffing <upstreamcerts>`. As soon as we
see the CONNECT request, we pause the client part of the conversation, and
initiate a simultaneous connection to the server. We complete the SSL handshake
with the server, and inspect the certificates it used. Now, we use the Common
@ -111,12 +98,12 @@ client. Voila, we have the correct hostname to present to the client, even if
it was never specified.
## Complication 2: Subject Alternative Name
Complication 2: Subject Alternative Name
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Enter the next complication. Sometimes, the certificate Common Name is not, in
fact, the hostname that the client is connecting to. This is because of the
optional [Subject Alternative
Name](http://en.wikipedia.org/wiki/SubjectAltName) field in the SSL certificate
optional `Subject Alternative Name`_ field in the SSL certificate
that allows an arbitrary number of alternative domains to be specified. If the
expected domain matches any of these, the client will proceed, even though the
domain doesn't match the certificate Common Name. The answer here is simple:
@ -124,14 +111,14 @@ when we extract the CN from the upstream cert, we also extract the SANs, and
add them to the generated dummy certificate.
## Complication 3: Server Name Indication
Complication 3: Server Name Indication
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
One of the big limitations of vanilla SSL is that each certificate requires its
own IP address. This means that you couldn't do virtual hosting where multiple
domains with independent certificates share the same IP address. In a world
with a rapidly shrinking IPv4 address pool this is a problem, and we have a
solution in the form of the [Server Name
Indication](http://en.wikipedia.org/wiki/Server_Name_Indication) extension to
solution in the form of the `Server Name Indication`_ extension to
the SSL and TLS protocols. This lets the client specify the remote server name
at the start of the SSL handshake, which then lets the server select the right
certificate to complete the process.
@ -145,72 +132,29 @@ passed to us. Now we can pause the conversation, and initiate an upstream
connection using the correct SNI value, which then serves us the correct
upstream certificate, from which we can extract the expected CN and SANs.
## Putting it all together
Putting it all together
^^^^^^^^^^^^^^^^^^^^^^^
Lets put all of this together into the complete explicitly proxied HTTPS flow.
<img class="img-responsive" src="explicit_https.png"/>
.. image:: schematics/how-mitmproxy-works-explicit-https.png
:align: center
<table class="table">
<tbody>
<tr>
<td><b>1</b></td>
<td>The client makes a connection to mitmproxy, and issues an HTTP
CONNECT request.</td>
</tr>
<tr>
<td><b>2</b></td>
1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT request.
2. Mitmproxy responds with a ``200 Connection Established``, as if it has set up the CONNECT pipe.
3. The client believes it's talking to the remote server, and initiates the SSL connection.
It uses SNI to indicate the hostname it is connecting to.
4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname
indicated by the client.
5. The server responds with the matching SSL certificate, which contains the CN and SAN values
needed to generate the interception certificate.
6. Mitmproxy generates the interception cert, and continues the
client SSL handshake paused in step 3.
7. The client sends the request over the established SSL connection.
8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4.
<td>Mitmproxy responds with a 200 Connection Established, as if it
has set up the CONNECT pipe.</td>
</tr>
<tr>
<td><b>3</b></td>
<td>The client believes it's talking to the remote server, and
initiates the SSL connection. It uses SNI to indicate the hostname
it is connecting to.</td>
</tr>
<tr>
<td><b>4</b></td>
<td>Mitmproxy connects to the server, and establishes an SSL
connection using the SNI hostname indicated by the client.</td>
</tr>
<tr>
<td><b>5</b></td>
<td>The server responds with the matching SSL certificate, which
contains the CN and SAN values needed to generate the interception
certificate.</td>
</tr>
<tr>
<td><b>6</b></td>
<td>Mitmproxy generates the interception cert, and continues the
client SSL handshake paused in step 3.</td>
</tr>
<tr>
<td><b>7</b></td>
<td>The client sends the request over the established SSL
connection.</td>
</tr>
<tr>
<td><b>7</b></td>
<td>Mitmproxy passes the request on to the server over the SSL
connection initiated in step 4.</td>
</tr>
</tbody>
</table>
<div class="page-header">
<h1>Transparent HTTP</h1>
</div>
Transparent HTTP
----------------
When a transparent proxy is used, the HTTP/S connection is redirected into a
proxy at the network layer, without any client configuration being required.
@ -221,13 +165,13 @@ example.
To achieve this, we need to introduce two extra components. The first is a
redirection mechanism that transparently reroutes a TCP connection destined for
a server on the Internet to a listening proxy server. This usually takes the
form of a firewall on the same host as the proxy server -
[iptables](http://www.netfilter.org/) on Linux or
[pf](http://en.wikipedia.org/wiki/PF_\(firewall\)) on OSX. Once the client has
initiated the connection, it makes a vanilla HTTP request, which might look
something like this:
form of a firewall on the same host as the proxy server - `iptables`_ on Linux or
pf_ on OSX. Once the client has initiated the connection, it makes a vanilla HTTP request,
which might look something like this:
<pre>GET /index.html HTTP/1.1</pre>
.. code-block:: http
GET /index.html HTTP/1.1
Note that this request differs from the explicit proxy variation, in that it
omits the scheme and hostname. How, then, do we know which upstream host to
@ -237,45 +181,21 @@ mechanism has a different way of exposing this data, so this introduces the
second component required for working transparent proxying: a host module that
knows how to retrieve the original destination address from the router. In
mitmproxy, this takes the form of a built-in set of
[modules](https://github.com/mitmproxy/mitmproxy/tree/master/libmproxy/platform)
that know how to talk to each platform's redirection mechanism. Once we have
this information, the process is fairly straight-forward.
modules_ that know how to talk to each platform's redirection mechanism.
Once we have this information, the process is fairly straight-forward.
<img class="img-responsive" src="transparent.png"/>
.. image:: schematics/how-mitmproxy-works-transparent.png
:align: center
1. The client makes a connection to the server.
2. The router redirects the connection to mitmproxy, which is typically listening on a local port
of the same host. Mitmproxy then consults the routing mechanism to establish what the original
destination was.
3. Now, we simply read the client's request...
4. ... and forward it upstream.
<table class="table">
<tbody>
<tr>
<td><b>1</b></td>
<td>The client makes a connection to the server.</td>
</tr>
<tr>
<td><b>2</b></td>
<td>The router redirects the connection to mitmproxy, which is
typically listening on a local port of the same host. Mitmproxy
then consults the routing mechanism to establish what the original
destination was.</td>
</tr>
<tr>
<td><b>3</b></td>
<td>Now, we simply read the client's request...</td>
</tr>
<tr>
<td><b>4</b></td>
<td>... and forward it upstream.</td>
</tr>
</tbody>
</table>
<div class="page-header">
<h1>Transparent HTTPS</h1>
</div>
Transparent HTTPS
-----------------
The first step is to determine whether we should treat an incoming connection
as HTTPS. The mechanism for doing this is simple - we use the routing mechanism
@ -287,65 +207,33 @@ transparently proxying HTTP, and explicitly proxying HTTPS. We use the routing
mechanism to establish the upstream server address, and then proceed as for
explicit HTTPS connections to establish the CN and SANs, and cope with SNI.
<img class="img-responsive" src="transparent_https.png"/>
.. image:: schematics/how-mitmproxy-works-transparent-https.png
:align: center
1. The client makes a connection to the server.
2. The router redirects the connection to mitmproxy, which is typically listening on a local port
of the same host. Mitmproxy then consults the routing mechanism to establish what the original
destination was.
3. The client believes it's talking to the remote server, and initiates the SSL connection.
It uses SNI to indicate the hostname it is connecting to.
4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname
indicated by the client.
5. The server responds with the matching SSL certificate, which contains the CN and SAN values
needed to generate the interception certificate.
6. Mitmproxy generates the interception cert, and continues the client SSL handshake paused in
step 3.
7. The client sends the request over the established SSL connection.
8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4.
<table class="table">
<tbody>
<tr>
<td><b>1</b></td>
<td>The client makes a connection to the server.</td>
</tr>
<tr>
<td><b>2</b></td>
.. rubric:: Footnotes
<td>The router redirects the connection to mitmproxy, which is
typically listening on a local port of the same host. Mitmproxy
then consults the routing mechanism to establish what the original
destination was.</td>
</tr>
<tr>
<td><b>3</b></td>
.. [#ssl] I use "SSL" to refer to both SSL and TLS in the generic sense, unless otherwise
specified.
<td>The client believes it's talking to the remote server, and
initiates the SSL connection. It uses SNI to indicate the hostname
it is connecting to.</td>
</tr>
<tr>
<td><b>4</b></td>
<td>Mitmproxy connects to the server, and establishes an SSL
connection using the SNI hostname indicated by the client.</td>
</tr>
<tr>
<td><b>5</b></td>
<td>The server responds with the matching SSL certificate, which
contains the CN and SAN values needed to generate the interception
certificate.</td>
</tr>
<tr>
<td><b>6</b></td>
<td>Mitmproxy generates the interception cert, and continues the
client SSL handshake paused in step 3.</td>
</tr>
<tr>
<td><b>7</b></td>
<td>The client sends the request over the established SSL
connection.</td>
</tr>
<tr>
<td><b>7</b></td>
<td>Mitmproxy passes the request on to the server over the SSL
connection initiated in step 4.</td>
</tr>
</tbody>
</table>
[^ssl]: I use "SSL" to refer to both SSL and TLS in the generic sense, unless otherwise specified.
.. _Server Name Indication: https://en.wikipedia.org/wiki/Server_Name_Indication
.. _HTTP RFC: https://tools.ietf.org/html/rfc7230
.. _Certificate Authority: https://en.wikipedia.org/wiki/Certificate_authority
.. _Subject Alternative Name: https://en.wikipedia.org/wiki/SubjectAltName
.. _iptables: http://www.netfilter.org/
.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\)
.. _modules: https://github.com/mitmproxy/mitmproxy/tree/master/libmproxy/platform

83
docs/index.rst Normal file
View File

@ -0,0 +1,83 @@
.. include:: introduction.rst
.. toctree::
:hidden:
:maxdepth: 1
introduction
install
certinstall
howmitmproxy
modes
.. toctree::
:hidden:
:caption: Tools
mitmproxy
mitmdump
config
.. toctree::
:hidden:
:caption: Features
features/anticache
features/filters
features/replacements
features/clientreplay
features/serverreplay
features/setheaders
features/passthrough
features/proxyauth
features/reverseproxy
features/responsestreaming
features/socksproxy
features/sticky
features/tcpproxy
features/upstreamproxy
features/upstreamcerts
.. toctree::
:hidden:
:caption: Transparent Proxying
transparent
transparent/linux
transparent/osx
.. toctree::
:hidden:
:caption: Scripting
scripting/inlinescripts
scripting/libmproxy
.. toctree::
:hidden:
:caption: Tutorials
tutorials/30second
tutorials/gamecenter
tutorials/transparent-dhcp
.. toctree::
:hidden:
:caption: Hacking
dev/architecture
dev/testing
dev/sslkeylogfile
dev/protocols
dev/proxy
dev/exceptions
dev/models
.. Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`

100
docs/install.rst Normal file
View File

@ -0,0 +1,100 @@
.. _install:
Installation
============
.. _install-ubuntu:
Installation On Ubuntu
----------------------
Ubuntu comes with Python but we need to install pip, python-dev and several libraries.
This was tested on a fully patched installation of Ubuntu 14.04.
>>> sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev
>>> sudo pip install mitmproxy
Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal.
Installation From Source (Ubuntu)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you would like to install mitmproxy directly from the master branch on GitHub or would like to
get set up to contribute to the project, install the dependencies as you would for a regular
mitmproxy installation (see :ref:`install-ubuntu`).
Then see the Hacking_ section of the README on GitHub.
Installation On Mac OS X
------------------------
The easiest way to get up and running on OSX is to download the pre-built binary packages from
`mitmproxy.org`_.
There are a few bits of customization you might want to do to make mitmproxy comfortable to use on
OSX. The default color scheme is optimized for a dark background terminal, but you can select a
palette for a light terminal background with the ``--palette`` option.
You can use the OSX **open** program to create a simple and effective ``~/.mailcap`` file to view
request and response bodies:
.. code-block:: none
application/*; /usr/bin/open -Wn %s
audio/*; /usr/bin/open -Wn %s
image/*; /usr/bin/open -Wn %s
video/*; /usr/bin/open -Wn %s
Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal.
Installation From Source (Mac OS X)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you would like to install mitmproxy directly from the master branch on GitHub or would like to
get set up to contribute to the project, there are a few OS X specific things to keep in mind.
- Make sure that XCode is installed from the App Store, and that the command-line tools have been
downloaded (XCode/Preferences/Downloads).
- If you're running a Python interpreter installed with homebrew (or similar), you may have to
install some dependencies by hand.
Then see the Hacking_ section of the README on GitHub.
Installation On Windows
-----------------------
.. note::
Please note that mitmdump is the only component of mitmproxy that is supported on Windows at
the moment.
**There is no interactive user interface on Windows.**
First, install the latest version of Python 2.7 from the `Python website`_.
If you already have an older version of Python 2.7 installed, make sure to install pip_
(pip is included in Python 2.7.9+ by default).
Next, add Python and the Python Scripts directory to your **PATH** variable.
You can do this easily by running the following in powershell:
>>> [Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27;C:\Python27\Scripts", "User")
Now, you can install mitmproxy by running
>>> pip install mitmproxy
Once the installation is complete, you can run :ref:`mitmdump` from a command prompt.
Installation From Source (Windows)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you would like to install mitmproxy directly from the master branch on GitHub or would like to
get set up to contribute to the project, install Python as outlined above, then see the
Hacking_ section of the README on GitHub.
.. _Hacking: https://github.com/mitmproxy/mitmproxy/blob/master/README.mkd#hacking
.. _mitmproxy.org: https://mitmproxy.org/
.. _`Python website`: https://www.python.org/downloads/windows/
.. _pip: https://pip.pypa.io/en/latest/installing.html

17
doc-src/index.html → docs/introduction.rst
View File

@ -1,19 +1,19 @@
Introduction
============
__mitmproxy__ is an interactive, SSL-capable man-in-the-middle proxy for HTTP
**mitmproxy** is an interactive, SSL-capable man-in-the-middle proxy for HTTP
with a console interface.
__mitmdump__ is the command-line version of mitmproxy. Think tcpdump for HTTP.
**mitmdump** is the command-line version of mitmproxy. Think tcpdump for HTTP.
__libmproxy__ is the library that mitmproxy and mitmdump are built on.
**libmproxy** is the library that mitmproxy and mitmdump are built on.
Documentation, tutorials and distribution packages can be found on the
mitmproxy.org website:
[mitmproxy.org](http://mitmproxy.org).
mitmproxy website: `mitmproxy.org <https://mitmproxy.org/>`_
Features
--------
.. rubric:: Features
- Intercept HTTP requests and responses and modify them on the fly.
- Save complete HTTP conversations for later replay and analysis.
@ -24,4 +24,3 @@ Features
- Make scripted changes to HTTP traffic using Python.
- SSL certificates for interception are generated on the fly.
- And much, much more.

66
docs/mitmdump.rst Normal file
View File

@ -0,0 +1,66 @@
.. _mitmdump:
.. program:: mitmdump
mitmdump
========
**mitmdump** is the command-line companion to mitmproxy. It provides
tcpdump-like functionality to let you view, record, and programmatically
transform HTTP traffic. See the :option:`--help` flag output for complete
documentation.
Examples
--------
Saving traffic
^^^^^^^^^^^^^^
>>> mitmdump -w outfile
Start up mitmdump in proxy mode, and write all traffic to **outfile**.
Filtering saved traffic
^^^^^^^^^^^^^^^^^^^^^^^
>>> mitmdump -nr infile -w outfile "~m post"
Start mitmdump without binding to the proxy port (:option:`-n`), read all flows from
infile, apply the specified filter expression (only match POSTs), and write to
outfile.
Client replay
^^^^^^^^^^^^^
>>> mitmdump -nc outfile
Start mitmdump without binding to the proxy port (:option:`-n`), then replay all
requests from outfile (:option:`-c filename`). Flags combine in the obvious way, so
you can replay requests from one file, and write the resulting flows to
another:
>>> mitmdump -nc srcfile -w dstfile
See the :ref:`clientreplay` section for more information.
Running a script
^^^^^^^^^^^^^^^^
>>> mitmdump -s examples/add_header.py
This runs the **add_header.py** example script, which simply adds a new header
to all responses.
Scripted data transformation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> mitmdump -ns examples/add_header.py -r srcfile -w dstfile
This command loads flows from **srcfile**, transforms it according to the
specified script, then writes it back to **dstfile**.

0
doc-src/mitmproxy-long.png → docs/mitmproxy-long.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 121 KiB

After

Width:  |  Height:  |  Size: 121 KiB

126
docs/mitmproxy.rst Normal file
View File

@ -0,0 +1,126 @@
.. _mitmproxy:
.. program:: mitmproxy
mitmproxy
=========
**mitmproxy** is a console tool that allows interactive examination and
modification of HTTP traffic. It differs from mitmdump in that all flows are
kept in memory, which means that it's intended for taking and manipulating
small-ish samples. Use the :kbd:`?` shortcut key to view, context-sensitive
documentation from any **mitmproxy** screen.
Flow list
---------
The flow list shows an index of captured flows in chronological order.
.. image:: screenshots/mitmproxy.png
- **1**: A GET request, returning a 302 Redirect response.
- **2**: A GET request, returning 16.75kb of text/html data.
- **3**: A replayed request.
- **4**: Intercepted flows are indicated with orange text. The user may edit
these flows, and then accept them (using the :kbd:`a` key) to continue. In this
case, the request has been intercepted on the way to the server.
- **5**: A response intercepted from the server on the way to the client.
- **6**: The event log can be toggled on and off using the :kbd:`e` shortcut key. This
pane shows events and errors that may not result in a flow that shows up in the
flow pane.
- **7**: Flow count.
- **8**: Various information on mitmproxy's state. In this case, we have an
interception pattern set to ``.*``.
- **9**: Bind address indicator - mitmproxy is listening on port 8080 of all
interfaces.
Flow view
---------
The **Flow View** lets you inspect and manipulate a single flow:
.. image:: screenshots/mitmproxy-flowview.png
- **1**: Flow summary.
- **2**: The Request/Response tabs, showing you which part of the flow you are
currently viewing. In the example above, we're viewing the Response. Hit :kbd:`tab`
to switch between the Response and the Request.
- **3**: Headers.
- **4**: Body.
- **5**: View Mode indicator. In this case, we're viewing the body in **hex** mode. The other
available modes are **pretty**, which uses a number of heuristics to show you a friendly
view of various content types, and **raw**, which shows you exactly what's there without any
changes. You can change modes using the :kbd:`m` key.
Grid Editor
-----------
Much of the data that we'd like to interact with in mitmproxy is structured.
For instance, headers, queries and form data can all be thought of as a list of
key/value pairs. Mitmproxy has a built-in editor that lays this type of data
out in a grid for easy manipulation.
At the moment, the Grid Editor is used in four parts of mitmproxy:
- Editing request or response headers (:kbd:`e` for edit, then :kbd:`h` for headers in flow view)
- Editing a query string (:kbd:`e` for edit, then :kbd:`q` for query in flow view)
- Editing a URL-encoded form (:kbd:`e` for edit, then :kbd:`f` for form in flow view)
- Editing replacement patterns (:kbd:`o` for options, then :kbd:`R` for Replacement Patterns)
If there is is no data, an empty editor will be started to let you add some.
Here is the editor showing the headers from a request:
.. image:: screenshots/mitmproxy-kveditor.png
To edit, navigate to the key or value you want to modify using the arrow or vi
navigation keys, and press enter. The background color will change to show that
you are in edit mode for the specified field:
.. image:: screenshots/mitmproxy-kveditor-editmode.png
Modify the field as desired, then press escape to exit edit mode when you're
done. You can also add a row (:kbd:`a` key), delete a row (:kbd:`d` key), spawn an
external editor on a field (:kbd:`e` key). Be sure to consult the context-sensitive
help (:kbd:`?` key) for more.
Example: Interception
---------------------
**mitmproxy**'s interception functionality lets you pause an HTTP request or
response, inspect and modify it, and then accept it to send it on to the server
or client.
1: Set an interception pattern
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-filt.png
We press :kbd:`i` to set an interception pattern. In this case, the ``~q`` filter
pattern tells **mitmproxy** to intercept all requests. For complete filter
syntax, see the :ref:`filters` section of the documentation,
or the built-in help function in **mitmproxy**.
2: Intercepted connections are indicated with orange text:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-mid.png
3: You can now view and modify the request:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-options.png
In this case, we viewed the request by selecting it, pressed :kbd:`e` for "edit"
and :kbd:`m` for "method" to change the HTTP request method.
4: Accept the intercept to continue:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-result.png
Finally, we press :kbd:`a` to accept the modified request, which is then sent on to
the server. In this case, we changed the request from an HTTP GET to
OPTIONS, and Google's server has responded with a 405 "Method not allowed".

193
docs/modes.rst Normal file
View File

@ -0,0 +1,193 @@
.. _modes:
Modes of Operation
==================
Mitmproxy has four modes of operation that allow you to use mitmproxy in a
variety of scenarios:
- **Regular** (the default)
- **Transparent**
- **Reverse Proxy**
- **Upstream Proxy**
Now, which one should you pick? Use this flow chart:
.. image:: schematics/proxy-modes-flowchart.png
:align: center
Regular Proxy
-------------
Mitmproxy's regular mode is the simplest and the easiest to set up.
1. Start mitmproxy.
2. Configure your client to use mitmproxy by explicitly setting an HTTP proxy.
3. Quick Check: You should already be able to visit an unencrypted HTTP site through the proxy.
4. Open the magic domain **mitm.it** and install the certificate for your device.
.. note::
Unfortunately, some applications bypass the system HTTP proxy settings - Android applications
are a common example. In these cases, you need to use mitmproxy's transparent mode.
If you are proxying an external device, your network will probably look like this:
.. image:: schematics/proxy-modes-regular.png
:align: center
The square brackets signify the source and destination IP addresses. Your
client explicitly connects to mitmproxy and mitmproxy explicitly connects
to the target server.
Transparent Proxy
-----------------
In transparent mode, traffic is directed into a proxy at the network layer,
without any client configuration required. This makes transparent proxying
ideal for situations where you can't change client behaviour. In the graphic
below, a machine running mitmproxy has been inserted between the router and
the internet:
.. image:: schematics/proxy-modes-transparent-1.png
:align: center
The square brackets signify the source and destination IP addresses. Round
brackets mark the next hop on the *Ethernet/data link* layer. This distinction
is important: when the packet arrives at the mitmproxy machine, it must still
be addressed to the target server. This means that Network Address Translation
should not be applied before the traffic reaches mitmproxy, since this would
remove the target information, leaving mitmproxy unable to determine the real
destination.
.. image:: schematics/proxy-modes-transparent-wrong.png
:align: center
Common Configurations
^^^^^^^^^^^^^^^^^^^^^
There are many ways to configure your network for transparent proxying. We'll
look at two common scenarios:
1. Configuring the client to use a custom gateway/router/"next hop"
2. Implementing custom routing on the router
In most cases, the first option is recommended due to its ease of use.
(a) Custom Gateway
~~~~~~~~~~~~~~~~~~
One simple way to get traffic to the mitmproxy machine with the destination IP
intact, is to simply configure the client with the mitmproxy box as the
default gateway.
.. image:: schematics/proxy-modes-transparent-2.png
:align: center
In this scenario, we would:
1. Configure the proxy machine for transparent mode. You can find instructions
in the :ref:`transparent` section.
2. Configure the client to use the proxy machine's IP as the default gateway.
3. Quick Check: At this point, you should already be able to visit an
unencrypted HTTP site over the proxy.
4. Open the magic domain **mitm.it** and install the certificate
for your device.
Setting the custom gateway on clients can be automated by serving the settings
out to clients over DHCP. This lets set up an interception network where all
clients are proxied automatically, which can save time and effort.
.. admonition:: Troubleshooting Transparent Mode
:class: note
Incorrect transparent mode configurations are a frequent source of
error. If it doesn't work for you, try the following things:
- Open mitmproxy's event log (press :kbd:`e`) - do you see clientconnect messages?
If not, the packets are not arriving at the proxy. One common cause is the occurrence of ICMP
redirects, which means that your machine is telling the client that there's a faster way to
the internet by contacting your router directly (see the :ref:`transparent` section on how to
disable them). If in doubt, Wireshark_ may help you to see whether something arrives at your
machine or not.
- Make sure you have not explicitly configured an HTTP proxy on the client.
This is not needed in transparent mode.
- Re-check the instructions in the :ref:`transparent` section. Anything you missed?
If you encounter any other pitfalls that should be listed here, please let us know!
(b) Custom Routing
~~~~~~~~~~~~~~~~~~
In some cases, you may need more fine-grained control of which traffic reaches
the mitmproxy instance, and which doesn't. You may, for instance, choose only
to divert traffic to some hosts into the transparent proxy. There are a huge
number of ways to accomplish this, and much will depend on the router or
packet filter you're using. In most cases, the configuration will look like
this:
.. image:: schematics/proxy-modes-transparent-3.png
:align: center
Reverse Proxy
-------------
mitmproxy is usually used with a client that uses the proxy to access the
Internet. Using reverse proxy mode, you can use mitmproxy to act like a normal
HTTP server:
.. image:: schematics/proxy-modes-reverse.png
:align: center
There are various use-cases:
- Say you have an internal API running at http://example.local/. You could now
set up mitmproxy in reverse proxy mode at http://debug.example.local/ and
dynamically point clients to this new API endpoint, which provides them
with the same data and you with debug information. Similarly, you could move
your real server to a different IP/port and set up mitmproxy in the original
place to debug and or redirect all sessions.
- Say you're a web developer working on http://example.com/ (with a development
version running on http://localhost:8000/). You can modify your hosts file so that
example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy mode
on port 80. You can test your app on the example.com domain and get all
requests recorded in mitmproxy.
- Say you have some toy project that should get SSL support. Simply set up
mitmproxy as a reverse proxy on port 443 and you're done (``mitmdump -p 443 -R
http://localhost:80/``). Mitmproxy auto-detects TLS traffic and intercepts it dynamically.
There are better tools for this specific task, but mitmproxy is very quick and simple way to
set up an SSL-speaking server.
- Want to add a non-SSL-capable compression proxy in front of your server? You
could even spawn a mitmproxy instance that terminates SSL (``-R http://...``),
point it to the compression proxy and let the compression proxy point to a
SSL-initiating mitmproxy (``-R https://...``), which then points to the real
server. As you see, it's a fairly flexible thing.
.. admonition:: Caveat: Interactive Use
:class: warning
Reverse Proxy mode is usually not sufficient to create a copy of an interactive website at
different URL. The HTML served to the client remains unchanged - as soon as the user clicks on
an non-relative URL (or downloads a non-relative image resource), traffic no longer passes
through mitmproxy.
Upstream Proxy
--------------
If you want to chain proxies by adding mitmproxy in front of a different proxy
appliance, you can use mitmproxy's upstream mode. In upstream mode, all
requests are unconditionally transferred to an upstream proxy of your choice.
.. image:: schematics/proxy-modes-upstream.png
:align: center
mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy
mode. You could in theory chain multiple mitmproxy instances in a row, but
that doesn't make any sense in practice (i.e. outside of our tests).
.. _Wireshark: https://wireshark.org/

0
doc-src/_explicit.graffle/data.plist → docs/schematics/_explicit.graffle/data.plist
View File

0
doc-src/_explicit.graffle/image3.icns → docs/schematics/_explicit.graffle/image3.icns
View File

0
doc-src/_explicit.graffle/image6.tiff → docs/schematics/_explicit.graffle/image6.tiff
View File

0
doc-src/_explicit_https.graffle/data.plist → docs/schematics/_explicit_https.graffle/data.plist
View File

0
doc-src/_explicit_https.graffle/image3.icns → docs/schematics/_explicit_https.graffle/image3.icns
View File

0
doc-src/_explicit_https.graffle/image6.tiff → docs/schematics/_explicit_https.graffle/image6.tiff
View File

0
doc-src/_transparent.graffle/data.plist → docs/schematics/_transparent.graffle/data.plist
View File

0
doc-src/_transparent.graffle/image3.icns → docs/schematics/_transparent.graffle/image3.icns
View File

0
doc-src/_transparent.graffle/image6.tiff → docs/schematics/_transparent.graffle/image6.tiff
View File

0
doc-src/_transparent_https.graffle/data.plist → docs/schematics/_transparent_https.graffle/data.plist
View File

0
doc-src/_transparent_https.graffle/image3.icns → docs/schematics/_transparent_https.graffle/image3.icns
View File

0
doc-src/_transparent_https.graffle/image6.tiff → docs/schematics/_transparent_https.graffle/image6.tiff
View File

0
doc-src/schematics/architecture.pdf → docs/schematics/architecture.pdf
View File

0
doc-src/schematics/architecture.png → docs/schematics/architecture.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 85 KiB

0
doc-src/schematics/architecture.vsdx → docs/schematics/architecture.vsdx
View File

0
doc-src/explicit_https.png → docs/schematics/how-mitmproxy-works-explicit-https.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 77 KiB

0
doc-src/explicit.png → docs/schematics/how-mitmproxy-works-explicit.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 64 KiB

After

Width:  |  Height:  |  Size: 64 KiB

0
doc-src/transparent_https.png → docs/schematics/how-mitmproxy-works-transparent-https.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 78 KiB

0
doc-src/transparent.png → docs/schematics/how-mitmproxy-works-transparent.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 68 KiB

After

Width:  |  Height:  |  Size: 68 KiB

0
doc-src/schematics/proxy-modes-flowchart.png → docs/schematics/proxy-modes-flowchart.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 34 KiB

0
doc-src/schematics/proxy-modes-regular.png → docs/schematics/proxy-modes-regular.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 18 KiB

0
doc-src/schematics/proxy-modes-reverse.png → docs/schematics/proxy-modes-reverse.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

0
doc-src/schematics/proxy-modes-transparent-1.png → docs/schematics/proxy-modes-transparent-1.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 14 KiB

0
doc-src/schematics/proxy-modes-transparent-2.png → docs/schematics/proxy-modes-transparent-2.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 23 KiB

Some files were not shown because too many files have changed in this diff Show More