rewrite basic docs for readthedocs

doc-src/dev-docs/index.rst

@@ -1,24 +0,0 @@
-Welcome to mitmproxy's dev documentation!
-=========================================
-
-
-Contents:
-
-.. toctree::
-    :maxdepth: 2
-
-    End User Documentation <https://mitmproxy.org/doc/>
-
-    inlinescripts
-    protocols
-    proxy
-    exceptions
-    models
-
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-

docs/Makefile

@@ -192,4 +192,4 @@ pseudoxml:
 	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
 
 livehtml:
-	sphinx-autobuild -b html -z '../../libmproxy' -r '___jb_(old|bak)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	sphinx-autobuild -b html -z '../libmproxy' -r '___jb_(old|bak)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html

docs/certinstall.rst

@@ -0,0 +1,173 @@
+.. _certinstall:
+
+About Certificates
+==================
+
+Introduction
+------------
+
+Mitmproxy can decrypt encrypted traffic on the fly, as long as the client
+trusts its built-in certificate authority. Usually this means that the
+mitmproxy CA certificates have to be installed on the client device.
+
+Quick Setup
+-----------
+
+By far the easiest way to install the mitmproxy certificates is to use the
+built-in certificate installation app. To do this, just start mitmproxy and
+configure your target device with the correct proxy settings. Now start a
+browser on the device, and visit the magic domain **mitm.it**. You should see
+something like this:
+
+.. image:: certinstall-webapp.png
+
+Click on the relevant icon, follow the setup instructions for the platform
+you're on and you are good to go.
+
+
+Installing the mitmproxy CA certificate manually
+------------------------------------------------
+
+Sometimes using the quick install app is not an option - Java or the iOS
+Simulator spring to mind - or you just need to do it manually for some other
+reason. Below is a list of pointers to manual certificate installation
+documentation for some common platforms.
+
+The mitmproxy CA cert is located in ``~/.mitmproxy`` after it has been generated at the first
+start of mitmproxy.
+
+
+iOS
+^^^
+
+http://kb.mit.edu/confluence/pages/viewpage.action?pageId=152600377
+
+iOS Simulator
+^^^^^^^^^^^^^
+
+See https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore
+
+Java
+^^^^
+
+See http://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html
+
+Android/Android Simulator
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+See http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets
+
+Windows
+^^^^^^^
+
+See http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7
+
+Windows (automated)
+^^^^^^^^^^^^^^^^^^^
+
+>>> certutil.exe -importpfx mitmproxy-ca-cert.p12
+
+See also: https://technet.microsoft.com/en-us/library/cc732443.aspx
+
+Mac OS X
+^^^^^^^^
+
+See https://support.apple.com/kb/PH7297?locale=en_US
+
+Ubuntu/Debian
+^^^^^^^^^^^^^
+
+See http://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861
+
+Mozilla Firefox
+^^^^^^^^^^^^^^^
+
+See https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox
+
+Chrome on Linux
+^^^^^^^^^^^^^^^
+
+See https://code.google.com/p/chromium/wiki/LinuxCertManagement
+
+
+More on mitmproxy certificates
+------------------------------
+
+The first time **mitmproxy** or **mitmdump** is run, the mitmproxy Certificate
+Authority (CA) is created in the config directory (``~/.mitmproxy`` by default).
+This CA is used for on-the-fly generation of dummy certificates for each of the
+SSL sites that your client visits. Since your browser won't trust the
+mitmproxy CA out of the box , you will see an SSL certificate warning every
+time you visit a new SSL domain through mitmproxy. When you are testing a
+single site through a browser, just accepting the bogus SSL cert manually is
+not too much trouble, but there are a many circumstances where you will want to
+configure your testing system or browser to trust the mitmproxy CA as a
+signing root authority. For security reasons, the mitmproxy CA is generated uniquely on the first
+start and is not shared between mitmproxy installations on different devices.
+
+
+CA and cert files
+-----------------
+
+The files created by mitmproxy in the .mitmproxy directory are as follows:
+
+===================== ====================================================================================
+mitmproxy-ca.pem      The certificate **and the private key** in PEM format.
+mitmproxy-ca-cert.pem The certificate in PEM format. Use this to distribute on most non-Windows platforms.
+mitmproxy-ca-cert.p12 The certificate in PKCS12 format. For use on Windows.
+mitmproxy-ca-cert.cer Same file as .pem, but with an extension expected by some Android devices.
+===================== ====================================================================================
+
+Using a custom certificate
+--------------------------
+
+You can use your own certificate by passing the ``--cert`` option to
+mitmproxy. Mitmproxy then uses the provided certificate for interception of the
+specified domains instead of generating a certificate signed by its own CA.
+
+The certificate file is expected to be in the PEM format.  You can include
+intermediary certificates right below your leaf certificate, so that you PEM
+file roughly looks like this:
+
+.. code-block:: none
+
+    -----BEGIN PRIVATE KEY-----
+    <private key>
+    -----END PRIVATE KEY-----
+    -----BEGIN CERTIFICATE-----
+    <cert>
+    -----END CERTIFICATE-----
+    -----BEGIN CERTIFICATE-----
+    <intermediary cert (optional)>
+    -----END CERTIFICATE-----
+
+
+For example, you can generate a certificate in this format using these instructions:
+
+
+>>> openssl genrsa -out cert.key 2048
+>>> openssl req -new -x509 -key cert.key -out cert.crt
+    (Specify the mitm domain as Common Name, e.g. *.google.com)
+>>> cat cert.key cert.crt > cert.pem
+>>> mitmproxy --cert=cert.pem
+
+
+Using a custom certificate authority
+------------------------------------
+
+By default, mitmproxy will use ``~/.mitmproxy/mitmproxy-ca.pem`` as
+the certificate authority to generate certificates for all domains for which no
+custom certificate is provided (see above). You can use your own certificate
+authority by passing the ``--confdir`` option to mitmproxy. Mitmproxy
+will then look for ``mitmproxy-ca.pem`` in the specified directory. If
+no such file exists, it will be generated automatically.
+
+
+Using a client side certificate
+-------------------------------
+
+You can use a client certificate by passing the ``--client-certs DIRECTORY``
+option to mitmproxy.  If you visit example.org, mitmproxy looks for a file named ``example.org.pem``
+in the specified directory and uses this as the client cert. The certificate file needs to be in
+the PEM format and should contain both the unencrypted private key and the certificate.
+

docs/conf.py

@@ -149,7 +149,7 @@ html_favicon = "favicon.ico"
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+# html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied

docs/dev/proxy.rst

@@ -1,7 +1,7 @@
-.. _exceptions:
+.. _proxy:
 
-Proxy
-=====
+Proxy Server
+============
 
 .. automodule:: libmproxy.proxy
 

docs/features/upstreamcerts.rst

@@ -0,0 +1,4 @@
+.. _upstreamcerts:
+
+Upstream Certificates
+=====================

docs/howmitmproxy.rst

@@ -0,0 +1,238 @@
+How mitmproxy works
+===================
+
+Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying
+process works will help you deploy it creatively, and take into account its
+fundamental assumptions and how to work around them. This document explains
+mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted
+explicit proxying, and working up to the most complicated interaction -
+transparent proxying of SSL-protected traffic [#ssl]_ in the presence of `Server Name Indication`_.
+
+Explicit HTTP
+-------------
+
+Configuring the client to use mitmproxy as an explicit proxy is the simplest
+and most reliable way to intercept traffic. The proxy protocol is codified in the
+`HTTP RFC`_, so the behaviour of both
+the client and the server is well defined, and usually reliable. In the
+simplest possible interaction with mitmproxy, a client connects directly to the
+proxy, and makes a request that looks like this:
+
+.. code-block:: http
+
+    GET http://example.com/index.html HTTP/1.1
+
+This is a proxy GET request - an extended form of the vanilla HTTP GET request
+that includes a schema and host specification, and it includes all the
+information mitmproxy needs to proceed.
+
+.. image:: schematics/how-mitmproxy-works-explicit.png
+    :align: center
+
+1. The client connects to the proxy and makes a request.
+2. Mitmproxy connects to the upstream server and simply forwards the request on.
+
+
+Explicit HTTPS
+--------------
+
+The process for an explicitly proxied HTTPS connection is quite different. The
+client connects to the proxy and makes a request that looks like this:
+
+.. code-block:: http
+
+    CONNECT example.com:443 HTTP/1.1
+
+A conventional proxy can neither view nor manipulate an SSL-encrypted data
+stream, so a CONNECT request simply asks the proxy to open a pipe between the
+client and server. The proxy here is just a facilitator - it blindly forwards
+data in both directions without knowing anything about the contents. The
+negotiation of the SSL connection happens over this pipe, and the subsequent
+flow of requests and responses are completely opaque to the proxy.
+
+The MITM in mitmproxy
+^^^^^^^^^^^^^^^^^^^^^
+
+This is where mitmproxy's fundamental trick comes into play. The MITM in its
+name stands for Man-In-The-Middle - a reference to the process we use to
+intercept and interfere with these theoretically opaque data streams. The basic
+idea is to pretend to be the server to the client, and pretend to be the client
+to the server, while we sit in the middle decoding traffic from both sides. The
+tricky part is that the `Certificate Authority`_ system is
+designed to prevent exactly this attack, by allowing a trusted third-party to
+cryptographically sign a server's SSL certificates to verify that they are
+legit. If this signature doesn't match or is from a non-trusted party, a secure
+client will simply drop the connection and refuse to proceed. Despite the many
+shortcomings of the CA system as it exists today, this is usually fatal to
+attempts to MITM an SSL connection for analysis. Our answer to this conundrum
+is to become a trusted Certificate Authority ourselves. Mitmproxy includes a
+full CA implementation that generates interception certificates on the fly. To
+get the client to trust these certificates, we :ref:`register mitmproxy as a trusted
+CA with the device manually <certinstall>`.
+
+Complication 1: What's the remote hostname?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To proceed with this plan, we need to know the domain name to use in the
+interception certificate - the client will verify that the certificate is for
+the domain it's connecting to, and abort if this is not the case. At first
+blush, it seems that the CONNECT request above gives us all we need - in this
+example, both of these values are "example.com".  But what if the client had
+initiated the connection as follows:
+
+.. code-block:: http
+
+    CONNECT 10.1.1.1:443 HTTP/1.1
+
+Using the IP address is perfectly legitimate because it gives us enough
+information to initiate the pipe, even though it doesn't reveal the remote
+hostname.
+
+Mitmproxy has a cunning mechanism that smooths this over - :ref:`upstream
+certificate sniffing <upstreamcerts>`. As soon as we
+see the CONNECT request, we pause the client part of the conversation, and
+initiate a simultaneous connection to the server. We complete the SSL handshake
+with the server, and inspect the certificates it used. Now, we use the Common
+Name in the upstream SSL certificates to generate the dummy certificate for the
+client. Voila, we have the correct hostname to present to the client, even if
+it was never specified.
+
+
+Complication 2: Subject Alternative Name
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Enter the next complication. Sometimes, the certificate Common Name is not, in
+fact, the hostname that the client is connecting to. This is because of the
+optional `Subject Alternative Name`_ field in the SSL certificate
+that allows an arbitrary number of alternative domains to be specified. If the
+expected domain matches any of these, the client will proceed, even though the
+domain doesn't match the certificate Common Name. The answer here is simple:
+when we extract the CN from the upstream cert, we also extract the SANs, and
+add them to the generated dummy certificate.
+
+
+Complication 3: Server Name Indication
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+One of the big limitations of vanilla SSL is that each certificate requires its
+own IP address. This means that you couldn't do virtual hosting where multiple
+domains with independent certificates share the same IP address. In a world
+with a rapidly shrinking IPv4 address pool this is a problem, and we have a
+solution in the form of the `Server Name Indication`_ extension to
+the SSL and TLS protocols. This lets the client specify the remote server name
+at the start of the SSL handshake, which then lets the server select the right
+certificate to complete the process.
+
+SNI breaks our upstream certificate sniffing process, because when we connect
+without using SNI, we get served a default certificate that may have nothing to
+do with the certificate expected by the client. The solution is another tricky
+complication to the client connection process. After the client connects, we
+allow the SSL handshake to continue until just _after_ the SNI value has been
+passed to us. Now we can pause the conversation, and initiate an upstream
+connection using the correct SNI value, which then serves us the correct
+upstream certificate, from which we can extract the expected CN and SANs.
+
+Putting it all together
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Lets put all of this together into the complete explicitly proxied HTTPS flow.
+
+.. image:: schematics/how-mitmproxy-works-explicit-https.png
+    :align: center
+
+1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT request.
+2. Mitmproxy responds with a ``200 Connection Established``, as if it has set up the CONNECT pipe.
+3. The client believes it's talking to the remote server, and initiates the SSL connection.
+   It uses SNI to indicate the hostname it is connecting to.
+4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname
+   indicated by the client.
+5. The server responds with the matching SSL certificate, which contains the CN and SAN values
+   needed to generate the interception certificate.
+6. Mitmproxy generates the interception cert, and continues the
+   client SSL handshake paused in step 3.
+7. The client sends the request over the established SSL connection.
+8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4.
+
+Transparent HTTP
+----------------
+
+When a transparent proxy is used, the HTTP/S connection is redirected into a
+proxy at the network layer, without any client configuration being required.
+This makes transparent proxying ideal for those situations where you can't
+change client behaviour - proxy-oblivious Android applications being a common
+example.
+
+To achieve this, we need to introduce two extra components. The first is a
+redirection mechanism that transparently reroutes a TCP connection destined for
+a server on the Internet to a listening proxy server. This usually takes the
+form of a firewall on the same host as the proxy server - `iptables`_ on Linux or
+pf_ on OSX. Once the client has initiated the connection, it makes a vanilla HTTP request,
+which might look something like this:
+
+.. code-block:: http
+
+    GET /index.html HTTP/1.1
+
+Note that this request differs from the explicit proxy variation, in that it
+omits the scheme and hostname. How, then, do we know which upstream host to
+forward the request to? The routing mechanism that has performed the
+redirection keeps track of the original destination for us.  Each routing
+mechanism has a different way of exposing this data, so this introduces the
+second component required for working transparent proxying: a host module that
+knows how to retrieve the original destination address from the router. In
+mitmproxy, this takes the form of a built-in set of
+modules_ that know how to talk to each platform's redirection mechanism.
+Once we have this information, the process is fairly straight-forward.
+
+.. image:: schematics/how-mitmproxy-works-transparent.png
+    :align: center
+
+1. The client makes a connection to the server.
+2. The router redirects the connection to mitmproxy, which is typically listening on a local port
+   of the same host. Mitmproxy then consults the routing mechanism to establish what the original
+   destination was.
+3. Now, we simply read the client's request...
+4. ... and forward it upstream.
+
+Transparent HTTPS
+-----------------
+
+The first step is to determine whether we should treat an incoming connection
+as HTTPS. The mechanism for doing this is simple - we use the routing mechanism
+to find out what the original destination port is. By default, we treat all
+traffic destined for ports 443 and 8443 as SSL.
+
+From here, the process is a merger of the methods we've described for
+transparently proxying HTTP, and explicitly proxying HTTPS. We use the routing
+mechanism to establish the upstream server address, and then proceed as for
+explicit HTTPS connections to establish the CN and SANs, and cope with SNI.
+
+.. image:: schematics/how-mitmproxy-works-transparent-https.png
+    :align: center
+
+1. The client makes a connection to the server.
+2. The router redirects the connection to mitmproxy, which is typically listening on a local port of
+   the same host. Mitmproxy then consults the routing mechanism to establish what the original
+   destination was.
+3. The client believes it's talking to the remote server, and initiates the SSL connection. It uses
+   SNI to indicate the hostname it is connecting to.
+4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname
+   indicated by the client.
+5. The server responds with the matching SSL certificate, which contains the CN and SAN values
+   needed to generate the interception certificate.
+6. Mitmproxy generates the interception cert, and continues the client SSL handshake paused in
+   step 3.
+7. The client sends the request over the established SSL connection.
+8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4.
+
+.. rubric:: Footnotes
+
+.. [#ssl] I use "SSL" to refer to both SSL and TLS in the generic sense, unless otherwise specified.
+
+.. _Server Name Indication: https://en.wikipedia.org/wiki/Server_Name_Indication
+.. _HTTP RFC: https://tools.ietf.org/html/rfc7230
+.. _Certificate Authority: https://en.wikipedia.org/wiki/Certificate_authority
+.. _Subject Alternative Name: https://en.wikipedia.org/wiki/SubjectAltName
+.. _iptables: http://www.netfilter.org/
+.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\)
+.. _modules: https://github.com/mitmproxy/mitmproxy/tree/master/libmproxy/platform

docs/index.rst

@@ -0,0 +1,43 @@
+.. include:: introduction.rst
+
+
+..  toctree::
+    :hidden:
+    :maxdepth: 1
+
+    introduction
+    install
+    certinstall
+    howmitmproxy
+    modes
+
+.. toctree::
+    :hidden:
+    :caption: Tools
+
+    mitmproxy
+    mitmdump
+
+.. toctree::
+    :hidden:
+    :caption: Scripting
+
+    scripting/inlinescripts
+    scripting/libmproxy
+
+
+.. toctree::
+    :hidden:
+    :caption: Development
+
+    dev/protocols
+    dev/proxy
+    dev/exceptions
+    dev/models
+
+..  Indices and tables
+    ==================
+
+    * :ref:`genindex`
+    * :ref:`modindex`
+

docs/install.rst

@@ -0,0 +1,100 @@
+.. _install:
+
+Installation
+============
+
+.. _install-ubuntu:
+
+Installation On Ubuntu
+----------------------
+
+Ubuntu comes with Python but we need to install pip, python-dev and several libraries.
+This was tested on a fully patched installation of Ubuntu 14.04.
+
+>>> sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev
+>>> sudo pip install mitmproxy
+
+Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal.
+
+Installation From Source (Ubuntu)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you would like to install mitmproxy directly from the master branch on GitHub or would like to
+get set up to contribute to the project, install the dependencies as you would for a regular
+mitmproxy installation (see :ref:`install-ubuntu`).
+Then see the Hacking_ section of the README on GitHub.
+
+
+
+Installation On Mac OS X
+------------------------
+
+The easiest way to get up and running on OSX is to download the pre-built binary packages from
+`mitmproxy.org`_.
+
+There are a few bits of customization you might want to do to make mitmproxy comfortable to use on
+OSX. The default color scheme is optimized for a dark background terminal, but you can select a
+palette for a light terminal background with the ``--palette`` option.
+You can use the OSX **open** program to create a simple and effective ``~/.mailcap`` file to view
+request and response bodies:
+
+.. code-block:: none
+
+    application/*; /usr/bin/open -Wn %s
+    audio/*; /usr/bin/open -Wn %s
+    image/*; /usr/bin/open -Wn %s
+    video/*; /usr/bin/open -Wn %s
+
+Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal.
+
+
+Installation From Source (Mac OS X)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you would like to install mitmproxy directly from the master branch on GitHub or would like to
+get set up to contribute to the project, there are a few OS X specific things to keep in mind.
+
+- Make sure that XCode is installed from the App Store, and that the command-line tools have been
+  downloaded (XCode/Preferences/Downloads).
+- If you're running a Python interpreter installed with homebrew (or similar), you may have to
+  install some dependencies by hand.
+
+Then see the Hacking_ section of the README on GitHub.
+
+Installation On Windows
+-----------------------
+
+.. note::
+    Please note that mitmdump is the only component of mitmproxy that is supported on Windows at
+    the moment.
+
+    **There is no interactive user interface on Windows.**
+
+
+First, install the latest version of Python 2.7 from the `Python website`_.
+If you already have an older version of Python 2.7 installed, make sure to install pip_
+(pip is included in Python 2.7.9+ by default).
+
+Next, add Python and the Python Scripts directory to your **PATH** variable.
+You can do this easily by running the following in powershell:
+
+>>> [Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27;C:\Python27\Scripts", "User")
+
+Now, you can install mitmproxy by running
+
+>>> pip install mitmproxy
+
+Once the installation is complete, you can run :ref:`mitmdump` from a command prompt.
+
+Installation From Source (Windows)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you would like to install mitmproxy directly from the master branch on GitHub or would like to
+get set up to contribute to the project, install Python as outlined above, then see the
+Hacking_ section of the README on GitHub.
+
+
+.. _Hacking: https://github.com/mitmproxy/mitmproxy/blob/master/README.mkd#hacking
+.. _mitmproxy.org: https://mitmproxy.org/
+.. _`Python website`: https://www.python.org/downloads/windows/
+.. _pip: https://pip.pypa.io/en/latest/installing.html

docs/introduction.rst

@@ -0,0 +1,26 @@
+Introduction
+============
+
+**mitmproxy** is an interactive, SSL-capable man-in-the-middle proxy for HTTP
+with a console interface.
+
+**mitmdump** is the command-line version of mitmproxy. Think tcpdump for HTTP.
+
+**libmproxy** is the library that mitmproxy and mitmdump are built on.
+
+Documentation, tutorials and distribution packages can be found on the
+mitmproxy website: `mitmproxy.org <https://mitmproxy.org/>`_
+
+
+.. rubric:: Features
+
+
+- Intercept HTTP requests and responses and modify them on the fly.
+- Save complete HTTP conversations for later replay and analysis.
+- Replay the client-side of an HTTP conversations.
+- Replay HTTP responses of a previously recorded server.
+- Reverse proxy mode to forward traffic to a specified server.
+- Transparent proxy mode on OSX and Linux.
+- Make scripted changes to HTTP traffic using Python.
+- SSL certificates for interception are generated on the fly.
+- And much, much more.

docs/mitmdump.rst

@@ -0,0 +1,4 @@
+.. _mitmdump:
+
+mitmdump
+========

docs/mitmproxy.rst

@@ -0,0 +1,4 @@
+.. _mitmproxy:
+
+mitmproxy
+=========

docs/modes.rst

@@ -0,0 +1,193 @@
+.. _modes:
+
+Modes of Operation
+==================
+
+Mitmproxy has four modes of operation that allow you to use mitmproxy in a
+variety of scenarios:
+
+- **Regular** (the default)
+- **Transparent**
+- **Reverse Proxy**
+- **Upstream Proxy**
+
+
+Now, which one should you pick? Use this flow chart:
+
+.. image:: schematics/proxy-modes-flowchart.png
+    :align: center
+
+Regular Proxy
+-------------
+
+Mitmproxy's regular mode is the simplest and the easiest to set up.
+
+1. Start mitmproxy.
+2. Configure your client to use mitmproxy by explicitly setting an HTTP proxy.
+3. Quick Check: You should already be able to visit an unencrypted HTTP site through the proxy.
+4. Open the magic domain <strong>mitm.it</strong> and install the certificate for your device.
+
+.. note::
+    Unfortunately, some applications bypass the system HTTP proxy settings - Android applications
+    are a common example. In these cases, you need to use mitmproxy's transparent mode.
+
+If you are proxying an external device, your network will probably look like this:
+
+.. image:: schematics/proxy-modes-regular.png
+    :align: center
+
+The square brackets signify the source and destination IP addresses. Your
+client explicitly connects to mitmproxy and mitmproxy explicitly connects
+to the target server.
+
+Transparent Proxy
+-----------------
+
+In transparent mode, traffic is directed into a proxy at the network layer,
+without any client configuration required. This makes transparent proxying
+ideal for situations where you can't change client behaviour. In the graphic
+below, a machine running mitmproxy has been inserted between the router and
+the internet:
+
+.. image:: schematics/proxy-modes-transparent-1.png
+    :align: center
+
+The square brackets signify the source and destination IP addresses. Round
+brackets mark the next hop on the *Ethernet/data link* layer. This distinction
+is important: when the packet arrives at the mitmproxy machine, it must still
+be addressed to the target server. This means that Network Address Translation
+should not be applied before the traffic reaches mitmproxy, since this would
+remove the target information, leaving mitmproxy unable to determine the real
+destination.
+
+.. image:: schematics/proxy-modes-transparent-wrong.png
+    :align: center
+
+Common Configurations
+^^^^^^^^^^^^^^^^^^^^^
+
+There are many ways to configure your network for transparent proxying. We'll
+look at two common scenarios:
+
+1. Configuring the client to use a custom gateway/router/"next hop"
+2. Implementing custom routing on the router
+
+In most cases, the first option is recommended due to its ease of use.
+
+(a) Custom Gateway
+~~~~~~~~~~~~~~~~~~
+
+One simple way to get traffic to the mitmproxy machine with the destination IP
+intact, is to simply configure the client with the mitmproxy box as the
+default gateway.
+
+.. image:: schematics/proxy-modes-transparent-2.png
+    :align: center
+
+In this scenario, we would:
+
+1. Configure the proxy machine for transparent mode. You can find instructions
+   in the :ref:`transparent` section.
+2. Configure the client to use the proxy machine's IP as the default gateway.
+3. Quick Check: At this point, you should already be able to visit an
+   unencrypted HTTP site over the proxy.
+4. Open the magic domain <strong>mitm.it</strong> and install the certificate
+   for your device.
+
+Setting the custom gateway on clients can be automated by serving the settings
+out to clients over DHCP. This lets set up an interception network where all
+clients are proxied automatically, which can save time and effort.
+
+.. admonition:: Troubleshooting Transparent Mode
+    :class: note
+
+    Incorrect transparent mode configurations are a frequent source of
+    error. If it doesn't work for you, try the following things:
+
+    - Open mitmproxy's event log (press :kbd:`e`) - do you see clientconnect messages?
+      If not, the packets are not arriving at the proxy. One common cause is the occurrence of ICMP
+      redirects, which means that your machine is telling the client that there's a faster way to
+      the internet by contacting your router directly (see the :ref:`transparent` section on how to
+      disable them). If in doubt, Wireshark_ may help you to see whether something arrives at your
+      machine or not.
+    - Make sure you have not explicitly configured an HTTP proxy on the client.
+      This is not needed in transparent mode.
+    - Re-check the instructions in the :ref:`transparent` section. Anything you missed?
+
+    If you encounter any other pitfalls that should be listed here, please let us know!
+
+(b) Custom Routing
+~~~~~~~~~~~~~~~~~~
+
+In some cases, you may need more fine-grained control of which traffic reaches
+the mitmproxy instance, and which doesn't. You may, for instance, choose only
+to divert traffic to some hosts into the transparent proxy. There are a huge
+number of ways to accomplish this, and much will depend on the router or
+packet filter you're using. In most cases, the configuration will look like
+this:
+
+.. image:: schematics/proxy-modes-transparent-3.png
+    :align: center
+
+
+Reverse Proxy
+-------------
+
+mitmproxy is usually used with a client that uses the proxy to access the
+Internet. Using reverse proxy mode, you can use mitmproxy to act like a normal
+HTTP server:
+
+.. image:: schematics/proxy-modes-reverse.png
+    :align: center
+
+There are various use-cases:
+
+- Say you have an internal API running at http://example.local/. You could now
+  set up mitmproxy in reverse proxy mode at http://debug.example.local/ and
+  dynamically point clients to this new API endpoint, which provides them
+  with the same data and you with debug information. Similarly, you could move
+  your real server to a different IP/port and set up mitmproxy in the original
+  place to debug and or redirect all sessions.
+
+- Say you're a web developer working on http://example.com/ (with a development
+  version running on http://localhost:8000/). You can modify your hosts file so that
+  example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy mode
+  on port 80. You can test your app on the example.com domain and get all
+  requests recorded in mitmproxy.
+
+- Say you have some toy project that should get SSL support. Simply set up
+  mitmproxy as a reverse proxy on port 443 and you're done (``mitmdump -p 443 -R
+  http://localhost:80/``). Mitmproxy auto-detects TLS traffic and intercepts it dynamically.
+  There are better tools for this specific task, but mitmproxy is very quick and simple way to
+  set up an SSL-speaking server.
+
+- Want to add a non-SSL-capable compression proxy in front of your server? You
+  could even spawn a mitmproxy instance that terminates SSL (``-R http://...``),
+  point it to the compression proxy and let the compression proxy point to a
+  SSL-initiating mitmproxy (``-R https://...``), which then points to the real
+  server. As you see, it's a fairly flexible thing.
+
+.. admonition:: Caveat: Interactive Use
+    :class: warning
+
+    Reverse Proxy mode is usually not sufficient to create a copy of an interactive website at
+    different URL. The HTML served to the client remains unchanged - as soon as the user clicks on
+    an non-relative URL (or downloads a non-relative image resource), traffic no longer passes
+    through mitmproxy.
+
+Upstream Proxy
+--------------
+
+If you want to chain proxies by adding mitmproxy in front of a different proxy
+appliance, you can use mitmproxy's upstream mode. In upstream mode, all
+requests are unconditionally transferred to an upstream proxy of your choice.
+
+.. image:: schematics/proxy-modes-upstream.png
+    :align: center
+
+mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy
+mode. You could in theory chain multiple mitmproxy instances in a row, but
+that doesn't make any sense in practice (i.e. outside of our tests).
+
+
+.. _Wireshark: https://wireshark.org/

docs/scripting/inlinescripts.rst

@@ -1,3 +1,5 @@
+.. _inline-scripts:
+
 Inline Scripts
 ==============
 
@@ -56,6 +58,7 @@ Events are listed in the order they usually occur.
     a connection can correspond to multiple HTTP requests.
 
     .. versionchanged:: 0.14
+
     :param Layer root_layer: The root layer (see :ref:`protocols` for an explanation what the root
         layer is), which provides transparent access to all attributes of the
         :py:class:`~libmproxy.proxy.RootContext`. For example, ``root_layer.client_conn.address``
@@ -111,6 +114,7 @@ Events are listed in the order they usually occur.
     Called when the proxy has closed the server connection.
 
     .. versionadded:: 0.14
+
     :param ServerConnection server_conn: see :py:func:`serverconnect`
 
 .. py:function:: clientdisconnect(context, root_layer)
@@ -118,6 +122,7 @@ Events are listed in the order they usually occur.
     Called when a client disconnects from the proxy.
 
     .. versionchanged:: 0.14
+
     :param Layer root_layer: see :py:func:`clientconnect`
 
 .. py:function:: done(context)

docs/scripting/libmproxy.rst

@@ -0,0 +1,27 @@
+.. _libmproxy:
+
+libmproxy
+=========
+
+.. note::
+
+    We strongly encourage you to use :ref:`inline-scripts` rather than libmproxy.
+        - Inline Scripts are equally powerful and provide an easier syntax.
+        - Most examples are written as inline scripts.
+        - Multiple inline scripts can be used together.
+        - Inline Scripts can either be executed headless with mitmdump or within the mitmproxy UI.
+
+
+All of mitmproxy's basic functionality is exposed through the **libmproxy**
+library. The example below shows a simple implementation of the "sticky cookie"
+functionality included in the interactive mitmproxy program. Traffic is
+monitored for ``Cookie`` and ``Set-Cookie`` headers, and requests are rewritten
+to include a previously seen cookie if they don't already have one. In effect,
+this lets you log in to a site using your browser, and then make subsequent
+requests using a tool like curl, which will then seem to be part of the
+authenticated session.
+
+
+.. literalinclude:: ../../examples/stickycookies
+   :caption: examples/stickycookies
+   :language: python

docs/transparent.rst

@@ -0,0 +1,6 @@
+.. _transparent:
+
+Transparent Proxying
+====================
+
+TODO