Grasscutters/mitmproxy
3
0
Fork 0
mirror of https://github.com/Grasscutters/mitmproxy.git synced 2024-11-25 09:37:37 +00:00
Code Issues Projects Releases Wiki Activity

All new documentation

Browse Source 
This patch does a lot.

- Ditch sphinx in favor of hugo. This gives us complete control of the layout
and presentation of our docs. Henceforth, docs will be hosted on our website
rather than ReadTheDocs.
- Create a simple, clean doc layout and theme.
- Remove large parts of the documentaion. I've ditched anything that was a)
woefully out of date, b) too detailed, or c) too hard to maintain in the long
term.
- Huge updates to the docs themselves: completely rewrite addons documentation,
add docs for core concepts like commands and options, and revise and tweak a
lot of the existing docs.

With this patch, we're also changing the way we publish and maintain the docs.
From now on, we don't publish docs for every release. Instead, the website will
contain ONE set of docs for each major release. The online docs will be updated
if needed as minor releases are made. Docs are free to improve during minor
releases, but anything that changes behaviour sufficiently to require a doc
change warrants a new major release. This also leaves us free to progressively
update and improve docs out of step with our release cadence.

With this new scheme, I feel CI over the docs is less important. I've removed
it for now, but won't object if someone wants to add it back in.
This commit is contained in:
Aldo Cortesi 2018-02-22 17:21:34 +13:00
parent 1cacefa104
commit 982508d30f
181 changed files with 12809 additions and 3982 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.travis.yml
View File

@ -29,8 +29,6 @@ matrix:
env: TOXENV=py36
- python: 3.6
env: TOXENV=individual_coverage
- python: 3.6
env: TOXENV=docs
- language: node_js
node_js: "node"
before_install:

20
README.rst
View File

@ -110,19 +110,18 @@ suite. The project tries to maintain 100% test coverage and enforces this strict
Documentation
-------------
The mitmproxy documentation is build using Sphinx_, which is installed
automatically if you set up a development environment as described above. After
installation, you can render the documentation like this:
The following tools are required to build the mitmproxy docs:
- Hugo_
- modd_
- yarn_
.. code-block:: bash
cd docs
make clean
make html
make livehtml
yarn
modd
The last command invokes `sphinx-autobuild`_, which watches the Sphinx directory and rebuilds
the documentation when a change is detected.
Code Style
----------
@ -181,8 +180,9 @@ with the following command:
.. _virtualenv: https://virtualenv.pypa.io/
.. _`pytest`: http://pytest.org/
.. _tox: https://tox.readthedocs.io/
.. _Sphinx: http://sphinx-doc.org/
.. _sphinx-autobuild: https://pypi.python.org/pypi/sphinx-autobuild
.. _Hugo: https://gohugo.io/
.. _modd: https://github.com/cortesi/modd
.. _yarn: https://yarnpkg.com/en/
.. _PEP8: https://www.python.org/dev/peps/pep-0008
.. _`Google Style Guide`: https://google.github.io/styleguide/pyguide.html
.. _forums: https://discourse.mitmproxy.org/

4
docs/.gitignore vendored
View File

@ -1 +1,3 @@
_build/
src/public
node_modules
public

195
docs/Makefile
View File

@ -1,195 +0,0 @@
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " applehelp to make an Apple Help Book"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
@echo " coverage to run coverage check of the documentation (if enabled)"
clean:
rm -rf $(BUILDDIR)/*
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/mitmproxy.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/mitmproxy.qhc"
applehelp:
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
@echo
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
@echo "N.B. You won't be able to view it unless you put it in" \
"~/Library/Documentation/Help or install it in your application" \
"bundle."
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/mitmproxy"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/mitmproxy"
@echo "# devhelp"
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
coverage:
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
@echo "Testing of coverage in the sources finished, look at the " \
"results in $(BUILDDIR)/coverage/python.txt."
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
livehtml:
sphinx-autobuild -b html -z '../mitmproxy' -r '___jb_(old|bak|tmp)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html

44
docs/_static/theme_overrides.css vendored
View File

@ -1,44 +0,0 @@
/* override table width restrictions */
.wy-table-responsive table td, .wy-table-responsive table th {
white-space: normal;
}
.wy-table-responsive > table > tbody > tr > td {
vertical-align: top !important;
}
.wy-table-responsive {
margin-bottom: 24px;
max-width: 100%;
overflow: visible;
}
.wy-menu-vertical header, .wy-menu-vertical p.caption {
color: #e0e0e0;
}
.code-block-caption {
height: 1.5em;
}
.code-block-caption .caption-text {
font-size: 0.8em;
float: right;
}
.code-block-caption .headerlink {
display: none !important;
}
.function .headerlink {
display: none !important;
}
dl .reference.internal {
display: none !important;
}
dl .headerlink {
display: none !important;
}

10
docs/_templates/page.html vendored
View File

@ -1,10 +0,0 @@
{% extends "!page.html" %}
{% block sidebartitle %}
<a href="https://mitmproxy.org/" style="margin-bottom: 7px; background: none !important;">
<button class="btn btn-info">
<i class="fa fa-arrow-left"></i>
Return to mitmproxy.org
</button>
</a>
{{ super() }}
{% endblock %}

211
docs/certinstall.rst
View File

@ -1,211 +0,0 @@
.. _certinstall:
About Certificates
==================
Introduction
------------
Mitmproxy can decrypt encrypted traffic on the fly, as long as the client
trusts its built-in certificate authority. Usually this means that the
mitmproxy CA certificates have to be installed on the client device.
Quick Setup
-----------
By far the easiest way to install the mitmproxy certificates is to use the
built-in certificate installation app. To do this, just start mitmproxy and
configure your target device with the correct proxy settings. Now start a
browser on the device, and visit the magic domain **mitm.it**. You should see
something like this:
.. image:: certinstall-webapp.png
Click on the relevant icon, follow the setup instructions for the platform
you're on and you are good to go.
Installing the mitmproxy CA certificate manually
------------------------------------------------
Sometimes using the quick install app is not an option - Java or the iOS
Simulator spring to mind - or you just need to do it manually for some other
reason. Below is a list of pointers to manual certificate installation
documentation for some common platforms.
The mitmproxy CA cert is located in ``~/.mitmproxy`` after it has been generated at the first
start of mitmproxy.
iOS
^^^
See http://jasdev.me/intercepting-ios-traffic
and https://web.archive.org/web/20150920082614/http://kb.mit.edu/confluence/pages/viewpage.action?pageId=152600377
On iOS 10.3 and onwards, you also need to enable full trust for the mitmproxy root certificate:
1. Go to Settings > General > About > Certificate Trust Settings.
2. Under "Enable full trust for root certificates", turn on trust for the mitmproxy certificate.
iOS Simulator
^^^^^^^^^^^^^
See https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore
Java
^^^^
See https://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html
Android/Android Simulator
^^^^^^^^^^^^^^^^^^^^^^^^^
See http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets
Windows
^^^^^^^
See https://web.archive.org/web/20160612045445/http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7
Windows (automated)
^^^^^^^^^^^^^^^^^^^
>>> certutil.exe -importpfx Root mitmproxy-ca-cert.p12
See also: https://technet.microsoft.com/en-us/library/cc732443.aspx
Mac OS X
^^^^^^^^
See https://support.apple.com/kb/PH7297?locale=en_US
Ubuntu/Debian
^^^^^^^^^^^^^
See https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861
Mozilla Firefox
^^^^^^^^^^^^^^^
See https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox
Chrome on Linux
^^^^^^^^^^^^^^^
See https://stackoverflow.com/a/15076602/198996
The mitmproxy certificate authority
-----------------------------------
The first time **mitmproxy** or **mitmdump** is run, the mitmproxy Certificate
Authority (CA) is created in the config directory (``~/.mitmproxy`` by default).
This CA is used for on-the-fly generation of dummy certificates for each of the
SSL sites that your client visits. Since your browser won't trust the
mitmproxy CA out of the box, you will see an SSL certificate warning every
time you visit a new SSL domain through mitmproxy. When you are testing a
single site through a browser, just accepting the bogus SSL cert manually is
not too much trouble, but there are a many circumstances where you will want to
configure your testing system or browser to trust the mitmproxy CA as a
signing root authority. For security reasons, the mitmproxy CA is generated uniquely on the first start and is not shared between mitmproxy installations on different devices.
Certificate Pinning
^^^^^^^^^^^^^^^^^^^
Some applications employ `Certificate Pinning`_ to prevent man-in-the-middle attacks.
This means that **mitmproxy** and **mitmdump's** certificates will not be
accepted by these applications without modifying them. It is recommended to use the
:ref:`passthrough` feature in order to prevent **mitmproxy** and **mitmdump** from intercepting
traffic to these specific domains. If you want to intercept the pinned connections, you need to patch the application manually. For Android and (jailbroken) iOS devices, various tools exist to accomplish this.
CA and cert files
-----------------
The files created by mitmproxy in the .mitmproxy directory are as follows:
===================== ==========================================================================
mitmproxy-ca.pem The certificate **and the private key** in PEM format.
mitmproxy-ca-cert.pem The certificate in PEM format.
Use this to distribute on most non-Windows platforms.
mitmproxy-ca-cert.p12 The certificate in PKCS12 format. For use on Windows.
mitmproxy-ca-cert.cer Same file as .pem, but with an extension expected by some Android devices.
===================== ==========================================================================
Using a custom certificate
--------------------------
You can use your own (leaf) certificate by passing the ``--cert [domain=]path_to_certificate`` option to
mitmproxy. Mitmproxy then uses the provided certificate for interception of the
specified domain instead of generating a certificate signed by its own CA.
The certificate file is expected to be in the PEM format. You can include
intermediary certificates right below your leaf certificate, so that your PEM
file roughly looks like this:
.. code-block:: none
-----BEGIN PRIVATE KEY-----
<private key>
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
<cert>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<intermediary cert (optional)>
-----END CERTIFICATE-----
For example, you can generate a certificate in this format using these instructions:
>>> openssl genrsa -out cert.key 2048
>>> openssl req -new -x509 -key cert.key -out cert.crt
(Specify the mitm domain as Common Name, e.g. *.google.com)
>>> cat cert.key cert.crt > cert.pem
Now, you can run mitmproxy with the generated certificate:
**For all domain names**
``>>>mitmproxy --cert *=cert.pem``
**For specific domain names**
``>>>mitmproxy --cert *.example.com=cert.pem``
**Note:** ``*.example.com`` is for all the subdomains. You can also use ``www.example.com`` for a particular subdomain.
Using a custom certificate authority
------------------------------------
By default, mitmproxy will use ``~/.mitmproxy/mitmproxy-ca.pem`` as
the certificate authority to generate certificates for all domains for which no
custom certificate is provided (see above). You can use your own certificate
authority by passing the ``--cadir DIRECTORY`` option to mitmproxy. Mitmproxy
will then look for ``mitmproxy-ca.pem`` in the specified directory. If
no such file exists, it will be generated automatically.
Using a client side certificate
-------------------------------
You can use a client certificate by passing the ``--client-certs DIRECTORY|FILE``
option to mitmproxy. Using a directory allows certs to be selected based on
hostname, while using a filename allows a single specific certificate to be used for
all SSL connections. Certificate files must be in the PEM format and should
contain both the unencrypted private key and the certificate.
Multiple certs by Hostname
^^^^^^^^^^^^^^^^^^^^^^^^^^
If you've specified a directory to ``--client-certs``, then the following
behavior will be taken:
If you visit example.org, mitmproxy looks for a file named ``example.org.pem`` in the specified
directory and uses this as the client cert.
.. _Certificate Pinning: https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning

243
docs/conf.py
View File

@ -1,243 +0,0 @@
import importlib
import inspect
import os
import subprocess
import sys
sys.path.insert(0, os.path.abspath('..'))
from mitmproxy import version as mversion
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.extlinks',
'sphinx.ext.linkcode',
'sphinx.ext.viewcode',
'sphinx.ext.napoleon',
'sphinxcontrib.documentedlist'
]
# https://github.com/sphinx-doc/sphinx/pull/2053
napoleon_include_special_with_doc = False
autodoc_member_order = "bysource"
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'mitmproxy docs'
copyright = u'2016, the mitmproxy project'
author = u'The mitmproxy project'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = mversion.VERSION
# The full version, including alpha/beta/rc tags.
release = mversion.VERSION
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['mitmproxy.']
# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {
'logo_only': True,
}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
html_title = "mitmproxy %s documentation" % version
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
html_logo = "mitmproxy-docs.png"
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
html_favicon = "favicon.ico"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
html_show_sourcelink = False
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages:
# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
#html_search_language = 'en'
# A dictionary with options for the search language support, empty by default.
# Now only 'ja' uses this config value
#html_search_options = {'type': 'default'}
# The name of a javascript file (relative to the configuration directory) that
# implements a search results scorer. If empty, the default will be used.
#html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'mitmproxydoc'
last_tag, tag_dist, commit = (
subprocess.check_output(["git", "describe", "--tags", "--long"])
.decode()
.strip()
.rsplit("-", 2)
)
tag_dist = int(tag_dist)
if tag_dist == 0:
tag = last_tag
else:
tag = "master"
SRCBASE = "https://github.com/mitmproxy/mitmproxy/blob/{}".format(tag)
extlinks = dict(
src = (SRCBASE + r"/%s", '')
)
def linkcode_resolve(domain, info):
if domain != 'py':
return None
module, fullname = info['module'], info['fullname']
if not module:
return None
obj = importlib.import_module(module)
for item in fullname.split('.'):
obj = getattr(obj, item, None)
if obj is None:
return None
try:
spath = inspect.getsourcefile(obj)
_, line = inspect.getsourcelines(obj)
except (TypeError, IOError):
return None
if spath.rfind("mitmproxy") > -1:
off = spath.rfind("mitmproxy")
mpath = spath[off:]
else:
return None
return SRCBASE + "/%s#L%s" % (mpath, line)
def setup(app):
app.add_stylesheet('theme_overrides.css')

13
docs/config.rst
View File

@ -1,13 +0,0 @@
.. _config:
Configuration
=============
Mitmproxy is configured with a YAML_ file, located at
``~/.mitmproxy/config.yaml``. We'll have complete documentation for all
supported options in the next release in the meantime, please consult the
source_ for a complete list of options and types.
.. _YAML: http://www.yaml.org/start.html
.. _source: https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/options.py

24
docs/custom-routing.txt
View File

@ -1,24 +0,0 @@
# Adapted from http://tldp.org/HOWTO/TransparentProxy-6.html (6.2 Second method)
# Note that the choice of firewall mark (3) and routing table (2) was fairly arbitrary.
# If you are already using policy routing or firewall marking for some other purpose,
# make sure you choose unique numbers here. Otherwise, don't worry about it.
# On the router, run
PROXY_IP=192.168.1.100
TARGET_IP=192.168.1.110
iptables -t mangle -A PREROUTING -j ACCEPT -p tcp -m multiport --dports 80,443 -s ! $TARGET_IP
# Alternative to MITM the whole network:
# iptables -t mangle -A PREROUTING -j ACCEPT -p tcp -m multiport --dports 80,443 -s $PROXY_IP
iptables -t mangle -A PREROUTING -j MARK --set-mark 3 -p tcp -m multiport --dports 80,443
ip rule add fwmark 3 table 2
ip route add default via $PROXY_IP dev br0 table 2
# On the proxy machine, run
iptables -A PREROUTING -t nat -i eth0 -p tcp -m multiport --dports 80,443 -j REDIRECT --to-port 8080

52
docs/dev/addingviews.html
View File

@ -1,52 +0,0 @@
As discussed in [the Flow View section of the mitmproxy
overview](@!urlTo("mitmproxy.html")!@), mitmproxy allows you to inspect and
manipulate flows. When inspecting a single flow, mitmproxy uses a number of
heuristics to show a friendly view of various content types; if mitmproxy
cannot show a friendly view, mitmproxy defaults to a __raw__ view.
Each content type invokes a different flow viewer to parse the data and display
the friendly view. Users can add custom content viewers by adding a view class
to contentview.py, discussed below.
## Adding a new View class to contentview.py
The content viewers used by mitmproxy to present a friendly view of various
content types are stored in contentview.py. Reviewing this file shows a number
of classes named ViewSomeDataType, each with the properties: __name__,
__prompt__, and __content\_types__ and a function named __\_\_call\_\___.
Adding a new content viewer to parse a data type is as simple as writing a new
View class. Your new content viewer View class should have the same properties
as the other View classes: __name__, __prompt__, and __content\_types__ and a
__\_\_call\_\___ function to parse the content of the request/response.
* The __name__ property should be a string describing the contents and new content viewer;
* The __prompt__ property should be a two item tuple:
- __1__: A string that will be used to display the new content viewer's type; and
- __2__: A one character string that will be the hotkey used to select the new content viewer from the Flow View screen;
* The __content\_types__ property should be a list of strings of HTTP Content\-Types that the new content viewer can parse.
* Note that mitmproxy will use the content\_types to try and heuristically show a friendly view of content and that you can override the built-in views by populating content\_types with values for content\_types that are already parsed -- e.g. "image/png".
After defining the __name__, __prompt__, and __content\_types__ properties of
the class, you should write the __\_\_call\_\___ function, which will parse the
request/response data and provide a friendly view of the data. The
__\_\_call\_\___ function should take the following arguments: __self__,
__hdrs__, __content__, __limit__; __hdrs__ is a MultiDict object containing
the headers of the request/response; __content__ is the content of the
request/response, and __limit__ is an integer representing the amount of data
to display in the view window.
The __\_\_call\_\___ function returns two values: (1) a string describing the
parsed data; and (2) the parsed data for friendly display. The parsed data to
be displayed should be a list of strings formatted for display. You can use
the __\_view\_text__ function in contentview.py to format text for display.
Alternatively, you can display content as a series of key-value pairs; to do
so, prepare a list of lists, where each list item is a two item list -- a key
that describes the data, and then the data itself; after preparing the list of
lists, use the __common.format\_keyvals__ function on it to prepare it as text
for display.
If the new content viewer fails or throws an exception, mitmproxy will default
to a __raw__ view.

11
docs/dev/contributing.rst
View File

@ -1,11 +0,0 @@
.. _contributing:
Contributing
============
As an open source project, **mitmproxy** welcomes contributions of all forms.
Please head over to the README_ to get started! 😃
.. _README: https://github.com/mitmproxy/mitmproxy/blob/master/README.rst

15
docs/dev/sslkeylogfile.rst
View File

@ -1,15 +0,0 @@
.. _sslkeylogfile:
TLS Master Secrets
==================
The SSL master keys can be logged by mitmproxy so that external programs can decrypt TLS
connections both from and to the proxy. Key logging is enabled by setting the environment variable
:envvar:`SSLKEYLOGFILE` so that it points to a writable text file.
Recent versions of WireShark can use these log files to decrypt packets.
You can specify the key file path in WireShark via
:samp:`Edit -> Preferences -> Protocols -> SSL -> (Pre)-Master-Secret log filename`.
Note that :envvar:`SSLKEYLOGFILE` is respected by other programs as well, e.g. Firefox and Chrome.
If this creates any issues, you can set :envvar:`MITMPROXY_SSLKEYLOGFILE` alternatively.

BIN
docs/favicon.ico
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 5.3 KiB

15
docs/features/anticache.rst
View File

@ -1,15 +0,0 @@
.. _anticache:
Anticache
=========
When the ``--anticache`` option is passed to mitmproxy, it removes headers
(``if-none-match`` and ``if-modified-since``) that might elicit a
``304 not modified`` response from the server. This is useful when you want to make
sure you capture an HTTP exchange in its totality. It's also often used during
:ref:`clientreplay`, when you want to make sure the server responds with complete data.
================== ======================
command-line ``--anticache``
mitmproxy shortcut :kbd:`O` then :kbd:`a`
================== ======================

18
docs/features/clientreplay.rst
View File

@ -1,18 +0,0 @@
.. _clientreplay:
Client-side replay
==================
Client-side replay does what it says on the tin: you provide a previously saved
HTTP conversation, and mitmproxy replays the client requests one by one. Note
that mitmproxy serializes the requests, waiting for a response from the server
before starting the next request. This might differ from the recorded
conversation, where requests may have been made concurrently.
You may want to use client-side replay in conjunction with the
:ref:`anticache` option, to make sure the server responds with complete data.
================== ===========
command-line ``-c path``
mitmproxy shortcut :kbd:`R` then :kbd:`c`
================== ===========

38
docs/features/filters.rst
View File

@ -1,38 +0,0 @@
.. _filters:
Filter expressions
==================
Many commands in :program:`mitmproxy` and :program:`mitmdump` take a filter expression.
Filter expressions consist of the following operators:
.. documentedlist::
:header: "Expression" "Description"
:listobject: mitmproxy.flowfilter.help
- Regexes are Python-style
- Regexes can be specified as quoted strings
- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value".
- Strings with no operators are matched against the request URL.
- The default binary operator is &.
Examples
--------
URL containing "google.com":
.. code-block:: none
google\.com
Requests whose body contains the string "test":
.. code-block:: none
~q ~b test
Anything but requests with a text/html content type:
.. code-block:: none
!(~q & ~t "text/html")

102
docs/features/passthrough.rst
View File

@ -1,102 +0,0 @@
.. _passthrough:
Ignore Domains
==============
There are two main reasons why you may want to exempt some traffic from mitmproxy's interception
mechanism:
- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and
mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or
the Apple App Store fail to work if mitmproxy is active.
- **Convenience:** You really don't care about some parts of the traffic and just want them to go
away. Note that mitmproxy's "Limit" option is often the better alternative here, as it is
not affected by the limitations listed below.
If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcp_proxy`
feature.
If you want to ignore traffic from mitmproxy's processing because of large response bodies,
take a look at the :ref:`streaming` feature.
How it works
------------
================== ======================
command-line ``--ignore regex``
mitmproxy shortcut :kbd:`O` then :kbd:`I`
================== ======================
mitmproxy allows you to specify a regex which is matched against a ``host:port`` string
(e.g. "example.com:443") to determine hosts that should be excluded.
Limitations
-----------
There are two important quirks to consider:
- **In transparent mode, the ignore pattern is matched against the IP and ClientHello SNI host.** While we usually infer the
hostname from the Host header if the ``--host`` argument is passed to mitmproxy, we do not
have access to this information before the SSL handshake. If the client uses SNI however, then we treat the SNI host as an ignore target.
- **In regular and upstream proxy mode, explicit HTTP requests are never ignored.** [#explicithttp]_ The ignore pattern is
applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections.
Tutorial
--------
If you just want to ignore one specific domain, there's usually a bulletproof method to do so:
1. Run mitmproxy or mitmdump in verbose mode (``-v``) and observe the ``host:port``
information in the serverconnect messages. mitmproxy will filter on these.
2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.)
and use this as your ignore pattern:
.. code-block:: none
:emphasize-lines: 6,7,9
>>> mitmdump -v
127.0.0.1:50588: clientconnect
127.0.0.1:50588: request
-> CONNECT example.com:443 HTTP/1.1
127.0.0.1:50588: Set new server address: example.com:443
127.0.0.1:50588: serverconnect
-> example.com:443
^C
>>> mitmproxy --ignore ^example\.com:443$
Here are some other examples for ignore patterns:
.. code-block:: none
# Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
--ignore apple.com:443
# "Correct" version without false-positives:
--ignore '^(.+\.)?apple\.com:443$'
# Ignore example.com, but not its subdomains:
--ignore '^example.com:'
# Ignore everything but example.com and mitmproxy.org:
--ignore '^(?!example\.com)(?!mitmproxy\.org)'
# Transparent mode:
--ignore 17\.178\.96\.59:443
# IP address range:
--ignore 17\.178\.\d+\.\d+:443
.. seealso::
- :ref:`tcp_proxy`
- :ref:`streaming`
- mitmproxy's "Limit" feature
.. rubric:: Footnotes
.. [#explicithttp] This stems from an limitation of explicit HTTP proxying:
A single connection can be re-used for multiple target domains - a
``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the
same connection. If we start to ignore the connection after the first request,
we would miss the relevant second one.
.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning

17
docs/features/proxyauth.rst
View File

@ -1,17 +0,0 @@
.. _proxyauth:
Proxy Authentication
====================
Asks the user for authentication before they are permitted to use the proxy.
Authentication headers are stripped from the flows, so they are not passed to
upstream servers. For now, only HTTP Basic authentication is supported. The
proxy auth options are not compatible with the transparent, socks or reverse proxy
mode.
================== ======================
command-line ``--nonanonymous``,
``--singleuser USER``,
``--htpasswd PATH``
================== ======================

71
docs/features/replacements.rst
View File

@ -1,71 +0,0 @@
.. _replacements:
Replacements
============
Mitmproxy lets you specify an arbitrary number of patterns that define text
replacements within flows. Each pattern has 3 components: a filter that defines
which flows a replacement applies to, a regular expression that defines what
gets replaced, and a target value that defines what is substituted in.
Replace hooks fire when either a client request or a server response is
received. Only the matching flow component is affected: so, for example, if a
replace hook is triggered on server response, the replacement is only run on
the Response object leaving the Request intact. You control whether the hook
triggers on the request, response or both using the filter pattern. If you need
finer-grained control than this, it's simple to create a script using the
replacement API on Flow components.
Replacement hooks are extremely handy in interactive testing of applications.
For instance you can use a replace hook to replace the text "XSS" with a
complicated XSS exploit, and then "inject" the exploit simply by interacting
with the application through the browser. When used with tools like Firebug and
mitmproxy's own interception abilities, replacement hooks can be an amazingly
flexible and powerful feature.
On the command-line
-------------------
The replacement hook command-line options use a compact syntax to make it easy
to specify all three components at once. The general form is as follows:
.. code-block:: none
/patt/regex/replacement
Here, **patt** is a mitmproxy filter expression, **regex** is a valid Python
regular expression, and **replacement** is a string literal. The first
character in the expression (``/`` in this case) defines what the separation
character is. Here's an example of a valid expression that replaces "foo" with
"bar" in all requests:
.. code-block:: none
:~q:foo:bar
In practice, it's pretty common for the replacement literal to be long and
complex. For instance, it might be an XSS exploit that weighs in at hundreds or
thousands of characters. To cope with this, there's a variation of the
replacement hook specifier that lets you load the replacement text from a file.
To specify a file as replacement, prefix the file path with ``@``.
You might start **mitmdump** as follows:
>>> mitmdump --replacements :~q:foo:@~/xss-exploit
This will load the replacement text from the file ``~/xss-exploit``.
The ``--replacements`` flag can be passed multiple times.
Interactively
-------------
The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`O`) lets you add and edit
replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has
complete usage information.
================== =======================
command-line ``--replacements``
mitmproxy shortcut :kbd:`O` then :kbd:`R`
================== =======================

43
docs/features/reverseproxy.rst
View File

@ -1,43 +0,0 @@
.. _reverseproxy:
Reverse Proxy
=============
In reverse proxy mode, mitmproxy accepts standard HTTP(S) requests and forwards
them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which
mitmproxy forwards HTTP(S) proxy requests to an upstream proxy server.
================== ================================
command-line ``-R http[s]://hostname[:port]``
================== ================================
Here, **http[s]** signifies if the proxy should use TLS to connect to the server.
mitmproxy always accepts both encrypted and unencrypted requests and transforms
them to what the server expects.
.. code-block:: none
>>> mitmdump -R https://httpbin.org -p 80
>>> curl http://localhost/
# requests will be transparently upgraded to TLS by mitmproxy
>>> mitmdump -R https://httpbin.org -p 443
>>> curl https://localhost/
# mitmproxy will use TLS on both ends.
Host Header
-----------
In reverse proxy mode, mitmproxy automatically rewrites the Host header to match the
upstream server. This allows mitmproxy to easily connect to existing endpoints on the
open web (e.g. ``mitmproxy -R https://example.com``). You can disable this behaviour
by passing ``--keep-host-header`` on the console.
However, keep in mind that absolute URLs within the returned document or HTTP redirects will
NOT be rewritten by mitmproxy. This means that if you click on a link for "http://example.com"
in the returned web page, you will be taken directly to that URL, bypassing mitmproxy.
One possible way to address this is to modify the hosts file of your OS so that "example.com"
resolves to your proxy's IP, and then access the proxy by going directly to example.com.
Make sure that your proxy can still resolve the original IP, or specify an IP in mitmproxy.

52
docs/features/serverreplay.rst
View File

@ -1,52 +0,0 @@
.. _serverreplay:
Server-side replay
==================
Server-side replay lets us replay server responses from a saved HTTP
conversation.
Matching requests with responses
--------------------------------
By default, :program:`mitmproxy` excludes request headers when matching incoming
requests with responses from the replay file. This works in most circumstances,
and makes it possible to replay server responses in situations where request
headers would naturally vary, e.g. using a different user agent.
The ``--rheader headername`` command-line option allows you to override
this behaviour by specifying individual headers that should be included in matching.
Response refreshing
-------------------
Simply replaying server responses without modification will often result in
unexpected behaviour. For example cookie timeouts that were in the future at
the time a conversation was recorded might be in the past at the time it is
replayed. By default, :program:`mitmproxy` refreshes server responses before sending
them to the client. The **date**, **expires** and **last-modified** headers are
all updated to have the same relative time offset as they had at the time of
recording. So, if they were in the past at the time of recording, they will be
in the past at the time of replay, and vice versa. Cookie expiry times are
updated in a similar way.
You can turn off response refreshing using the ``--norefresh`` argument, or using
the :kbd:`O` options shortcut within :program:`mitmproxy`.
Replaying a session recorded in Reverse-proxy Mode
--------------------------------------------------
If you have captured the session in reverse proxy mode, in order to replay it you
still have to specify the server URL, otherwise you may get the error:
'HTTP protocol error in client request: Invalid HTTP request form (expected authority or absolute...)'.
During replay, when the client's requests match previously recorded requests, then the
respective recorded responses are simply replayed by mitmproxy.
Otherwise, the unmatched requests is forwarded to the upstream server.
If forwarding is not desired, you can use the --kill (-k) switch to prevent that.
================== ===========
command-line ``-S path``
mitmproxy shortcut :kbd:`R` then :kbd:`s`
================== ===========

19
docs/features/setheaders.rst
View File

@ -1,19 +0,0 @@
.. _setheaders:
Set Headers
===========
This feature lets you specify a set of headers to be added to requests or
responses, based on a filter pattern. You can specify these either on the
command-line, or through an interactive editor in mitmproxy.
Example: Set the **Host** header to "example.com" for all requests.
.. code-block:: none
mitmdump -R http://example.com --setheader :~q:Host:example.com
================== =======================
command-line ``--setheader PATTERN``
mitmproxy shortcut :kbd:`O` then :kbd:`H`
================== =======================

10
docs/features/socksproxy.rst
View File

@ -1,10 +0,0 @@
.. _socksproxy:
SOCKS Mode
==========
In this mode, mitmproxy acts as a SOCKS5 proxy server.
================== ===========
command-line ``--socks``
================== ===========

41
docs/features/sticky.rst
View File

@ -1,41 +0,0 @@
.. _sticky:
Sticky cookies and auth
=======================
Sticky cookies
--------------
When the sticky cookie option is set, __mitmproxy__ will add the cookie most
recently set by the server to any cookie-less request. Consider a service that
sets a cookie to track the session after authentication. Using sticky cookies,
you can fire up mitmproxy, and authenticate to a service as you usually would
using a browser. After authentication, you can request authenticated resources
through mitmproxy as if they were unauthenticated, because mitmproxy will
automatically add the session tracking cookie to requests. Among other things,
this lets you script interactions with authenticated resources (using tools
like wget or curl) without having to worry about authentication.
Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can
record the authentication process once, and simply replay it on startup every time you need
to interact with the secured resources.
================== ======================
command-line ``-t FILTER``
mitmproxy shortcut :kbd:`O` then :kbd:`t`
================== ======================
Sticky auth
-----------
The sticky auth option is analogous to the sticky cookie option, in that HTTP
**Authorization** headers are simply replayed to the server once they have been
seen. This is enough to allow you to access a server resource using HTTP Basic
authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support
replay of HTTP Digest authentication.
================== ======================
command-line ``-u FILTER``
mitmproxy shortcut :kbd:`O` then :kbd:`A`
================== ======================

102
docs/features/streaming.rst
View File

@ -1,102 +0,0 @@
.. _streaming:
HTTP Streaming
==============
By default, mitmproxy will read the entire request/response, perform any indicated
manipulations on it and then send the (possibly modified) message to
the other party. In some cases this is undesirable and you may wish to "stream"
the request/response. When streaming is enabled, the request/response is
not buffered on the proxy but directly sent to the server/client instead.
HTTP headers are still fully buffered before being sent.
Request Streaming
-----------------
Request streaming can be used to incrementally stream a request body to the server
before it has been fully received by the proxy. This is useful for large file uploads.
Response Streaming
------------------
By using mitmproxy's streaming feature, response contents can be passed to the client incrementally
before they have been fully received by the proxy. This is especially useful for large binary files
such as videos, where buffering the whole file slows down the client's browser.
On the command-line
-------------------
Streaming can be enabled on the command line for all request and response bodies exceeding a certain size.
The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes.
================== =================
command-line ``--set stream_large_bodies=SIZE``
================== =================
.. warning::
When streaming is enabled, **streamed request/response contents will not be
recorded or preserved in any way.**
.. note::
When streaming is enabled, the request/response body cannot be modified by the usual means.
Customizing Streaming
---------------------
You can also use a script to customize exactly which requests or responses are streamed.
Requests/Responses that should be tagged for streaming by setting their ``.stream``
attribute to ``True``:
.. literalinclude:: ../../examples/complex/stream.py
:caption: examples/complex/stream.py
:language: python
Implementation Details
----------------------
When response streaming is enabled, portions of the code which would have otherwise performed
changes on the request/response body will see an empty body. Any modifications will be ignored.
Streamed bodies are usually sent in chunks of 4096 bytes. If the response is sent with a
``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time.
Modifying streamed data
-----------------------
If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all
chunks.
.. literalinclude:: ../../examples/complex/stream_modify.py
:caption: examples/complex/stream_modify.py
:language: python
WebSocket Streaming
===================
The WebSocket streaming feature can be used to send the frames as soon as they arrive. This can be useful for large binary file transfers.
On the command-line
-------------------
Streaming can be enabled on the command line for all WebSocket frames
================== =================
command-line ``--set stream_websockets=true``
================== =================
.. note::
When Web Socket streaming is enabled, the message payload cannot be modified.
Implementation Details
----------------------
When WebSocket streaming is enabled, portions of the code which may perform changes to the WebSocket message payloads will not have
any effect on the actual payload sent to the server as the frames are immediately forwarded to the server.
In contrast to HTTP streaming, where the body is not stored, the message payload will still be stored in the WebSocket Flow.
.. seealso::
- :ref:`passthrough`

23
docs/features/upstreamcerts.rst
View File

@ -1,23 +0,0 @@
.. _upstreamcerts:
Upstream Certificates
=====================
When mitmproxy receives a connection destined for an SSL-protected service, it
freezes the connection before reading its request data, and makes a connection
to the upstream server to "sniff" the contents of its SSL certificate. The
information gained - the **Common Name** and **Subject Alternative Names** - is
then used to generate the interception certificate, which is sent to the client
so the connection can continue.
This rather intricate little dance lets us seamlessly generate correct
certificates even if the client has specified only an IP address rather than the
hostname. It also means that we don't need to sniff additional data to generate
certs in transparent mode.
Upstream cert sniffing is on by default, and can optionally be turned off.
================== ======================
command-line ``--no-upstream-cert``
mitmproxy shortcut :kbd:`O` then :kbd:`U`
================== ======================

12
docs/features/upstreamproxy.rst
View File

@ -1,12 +0,0 @@
.. _upstreamproxy:
Upstream proxy mode
===================
In this mode, mitmproxy accepts proxy requests and unconditionally forwards all
requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`,
in which mitmproxy forwards ordinary HTTP requests to an upstream server.
================== =============================
command-line ``-U http://hostname[:port]``
================== =============================

240
docs/howmitmproxy.rst
View File

@ -1,240 +0,0 @@
How mitmproxy works
===================
Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying
process works will help you deploy it creatively, and take into account its
fundamental assumptions and how to work around them. This document explains
mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted
explicit proxying, and working up to the most complicated interaction -
transparent proxying of TLS-protected traffic [#tls]_ in the presence of `Server
Name Indication`_.
Explicit HTTP
-------------
Configuring the client to use mitmproxy as an explicit proxy is the simplest and
most reliable way to intercept traffic. The proxy protocol is codified in the
`HTTP RFC`_, so the behaviour of both the client and the server is well defined,
and usually reliable. In the simplest possible interaction with mitmproxy, a
client connects directly to the proxy, and makes a request that looks like this:
.. code-block:: none
GET http://example.com/index.html HTTP/1.1
This is a proxy GET request - an extended form of the vanilla HTTP GET request
that includes a schema and host specification, and it includes all the
information mitmproxy needs to proceed.
.. image:: schematics/how-mitmproxy-works-explicit.png
:align: center
1. The client connects to the proxy and makes a request.
2. Mitmproxy connects to the upstream server and simply forwards the request on.
Explicit HTTPS
--------------
The process for an explicitly proxied HTTPS connection is quite different. The
client connects to the proxy and makes a request that looks like this:
.. code-block:: none
CONNECT example.com:443 HTTP/1.1
A conventional proxy can neither view nor manipulate a TLS-encrypted data
stream, so a CONNECT request simply asks the proxy to open a pipe between the
client and server. The proxy here is just a facilitator - it blindly forwards
data in both directions without knowing anything about the contents. The
negotiation of the TLS connection happens over this pipe, and the subsequent
flow of requests and responses are completely opaque to the proxy.
The MITM in mitmproxy
^^^^^^^^^^^^^^^^^^^^^
This is where mitmproxy's fundamental trick comes into play. The MITM in its
name stands for Man-In-The-Middle - a reference to the process we use to
intercept and interfere with these theoretically opaque data streams. The basic
idea is to pretend to be the server to the client, and pretend to be the client
to the server, while we sit in the middle decoding traffic from both sides. The
tricky part is that the `Certificate Authority`_ system is designed to prevent
exactly this attack, by allowing a trusted third-party to cryptographically sign
a server's certificates to verify that they are legit. If this signature doesn't
match or is from a non-trusted party, a secure client will simply drop the
connection and refuse to proceed. Despite the many shortcomings of the CA system
as it exists today, this is usually fatal to attempts to MITM a TLS connection
for analysis. Our answer to this conundrum is to become a trusted Certificate
Authority ourselves. Mitmproxy includes a full CA implementation that generates
interception certificates on the fly. To get the client to trust these
certificates, we :ref:`register mitmproxy as a trusted CA with the device
manually <certinstall>`.
Complication 1: What's the remote hostname?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To proceed with this plan, we need to know the domain name to use in the
interception certificate - the client will verify that the certificate is for
the domain it's connecting to, and abort if this is not the case. At first
blush, it seems that the CONNECT request above gives us all we need - in this
example, both of these values are "example.com". But what if the client had
initiated the connection as follows:
.. code-block:: none
CONNECT 10.1.1.1:443 HTTP/1.1
Using the IP address is perfectly legitimate because it gives us enough
information to initiate the pipe, even though it doesn't reveal the remote
hostname.
Mitmproxy has a cunning mechanism that smooths this over - :ref:`upstream
certificate sniffing <upstreamcerts>`. As soon as we see the CONNECT request, we
pause the client part of the conversation, and initiate a simultaneous
connection to the server. We complete the TLS handshake with the server, and
inspect the certificates it used. Now, we use the Common Name in the upstream
certificates to generate the dummy certificate for the client. Voila, we have
the correct hostname to present to the client, even if it was never specified.
Complication 2: Subject Alternative Name
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Enter the next complication. Sometimes, the certificate Common Name is not, in
fact, the hostname that the client is connecting to. This is because of the
optional `Subject Alternative Name`_ field in the certificate that allows an
arbitrary number of alternative domains to be specified. If the expected domain
matches any of these, the client will proceed, even though the domain doesn't
match the certificate CN. The answer here is simple: when we extract the CN from
the upstream cert, we also extract the SANs, and add them to the generated dummy
certificate.
Complication 3: Server Name Indication
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
One of the big limitations of vanilla TLS is that each certificate requires its
own IP address. This means that you couldn't do virtual hosting where multiple
domains with independent certificates share the same IP address. In a world with
a rapidly shrinking IPv4 address pool this is a problem, and we have a solution
in the form of the `Server Name Indication`_ extension to the TLS protocols.
This lets the client specify the remote server name at the start of the TLS
handshake, which then lets the server select the right certificate to complete
the process.
SNI breaks our upstream certificate sniffing process, because when we connect
without using SNI, we get served a default certificate that may have nothing to
do with the certificate expected by the client. The solution is another tricky
complication to the client connection process. After the client connects, we
allow the TLS handshake to continue until just **after** the SNI value has been
passed to us. Now we can pause the conversation, and initiate an upstream
connection using the correct SNI value, which then serves us the correct
upstream certificate, from which we can extract the expected CN and SANs.
Putting it all together
^^^^^^^^^^^^^^^^^^^^^^^
Lets put all of this together into the complete explicitly proxied HTTPS flow.
.. image:: schematics/how-mitmproxy-works-explicit-https.png
:align: center
1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT request.
2. Mitmproxy responds with a ``200 Connection Established``, as if it has set up the CONNECT pipe.
3. The client believes it's talking to the remote server, and initiates the TLS connection.
It uses SNI to indicate the hostname it is connecting to.
4. Mitmproxy connects to the server, and establishes a TLS connection using the SNI hostname
indicated by the client.
5. The server responds with the matching certificate, which contains the CN and SAN values
needed to generate the interception certificate.
6. Mitmproxy generates the interception cert, and continues the
client TLS handshake paused in step 3.
7. The client sends the request over the established TLS connection.
8. Mitmproxy passes the request on to the server over the TLS connection initiated in step 4.
Transparent HTTP
----------------
When a transparent proxy is used, the connection is redirected into a proxy at
the network layer, without any client configuration being required. This makes
transparent proxying ideal for those situations where you can't change client
behaviour - proxy-oblivious Android applications being a common example.
To achieve this, we need to introduce two extra components. The first is a
redirection mechanism that transparently reroutes a TCP connection destined for
a server on the Internet to a listening proxy server. This usually takes the
form of a firewall on the same host as the proxy server - `iptables`_ on Linux
or pf_ on OSX. Once the client has initiated the connection, it makes a vanilla
HTTP request, which might look something like this:
.. code-block:: none
GET /index.html HTTP/1.1
Note that this request differs from the explicit proxy variation, in that it
omits the scheme and hostname. How, then, do we know which upstream host to
forward the request to? The routing mechanism that has performed the redirection
keeps track of the original destination for us. Each routing mechanism has a
different way of exposing this data, so this introduces the second component
required for working transparent proxying: a host module that knows how to
retrieve the original destination address from the router. In mitmproxy, this
takes the form of a built-in set of modules_ that know how to talk to each
platform's redirection mechanism. Once we have this information, the process is
fairly straight-forward.
.. image:: schematics/how-mitmproxy-works-transparent.png
:align: center
1. The client makes a connection to the server.
2. The router redirects the connection to mitmproxy, which is typically
listening on a local port of the same host. Mitmproxy then consults the
routing mechanism to establish what the original destination was.
3. Now, we simply read the client's request...
4. ... and forward it upstream.
Transparent HTTPS
-----------------
The first step is to determine whether we should treat an incoming connection as
HTTPS. The mechanism for doing this is simple - we use the routing mechanism to
find out what the original destination port is. All incoming connections pass
through different layers which can determin the actual protocol to use.
Automatic TLS detection works for SSLv3, TLS 1.0, TLS 1.1, and TLS 1.2 by
looking for a *ClientHello* message at the beginning of each connection. This
works independently of the used TCP port.
From here, the process is a merger of the methods we've described for
transparently proxying HTTP, and explicitly proxying HTTPS. We use the routing
mechanism to establish the upstream server address, and then proceed as for
explicit HTTPS connections to establish the CN and SANs, and cope with SNI.
.. image:: schematics/how-mitmproxy-works-transparent-https.png
:align: center
1. The client makes a connection to the server.
2. The router redirects the connection to mitmproxy, which is typically listening on a local port
of the same host. Mitmproxy then consults the routing mechanism to establish what the original
destination was.
3. The client believes it's talking to the remote server, and initiates the TLS connection.
It uses SNI to indicate the hostname it is connecting to.
4. Mitmproxy connects to the server, and establishes a TLS connection using the SNI hostname
indicated by the client.
5. The server responds with the matching certificate, which contains the CN and SAN values
needed to generate the interception certificate.
6. Mitmproxy generates the interception cert, and continues the client TLS handshake paused in
step 3.
7. The client sends the request over the established TLS connection.
8. Mitmproxy passes the request on to the server over the TLS connection initiated in step 4.
.. rubric:: Footnotes
.. [#tls] The use of "TLS" refers to both SSL (outdated and insecure) and TLS
(1.0 and up) in the generic sense, unless otherwise specified.
.. _Server Name Indication: https://en.wikipedia.org/wiki/Server_Name_Indication
.. _HTTP RFC: https://tools.ietf.org/html/rfc7230
.. _Certificate Authority: https://en.wikipedia.org/wiki/Certificate_authority
.. _Subject Alternative Name: https://en.wikipedia.org/wiki/SubjectAltName
.. _iptables: http://www.netfilter.org/
.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\)
.. _modules: https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/platform

99
docs/index.rst
View File

@ -1,99 +0,0 @@
.. include:: introduction.rst
.. toctree::
:hidden:
:maxdepth: 1
introduction
install
certinstall
howmitmproxy
modes
.. toctree::
:hidden:
:caption: Tools
mitmproxy
mitmdump
mitmweb
config
.. toctree::
:hidden:
:caption: Protocols
protocols/http1
protocols/http2
protocols/websocket
protocols/tcpproxy
.. toctree::
:hidden:
:caption: Features
features/anticache
features/filters
features/replacements
features/clientreplay
features/serverreplay
features/setheaders
features/passthrough
features/proxyauth
features/reverseproxy
features/streaming
features/socksproxy
features/sticky
features/upstreamproxy
features/upstreamcerts
.. toctree::
:hidden:
:caption: Transparent Proxying
transparent
transparent/linux
transparent/osx
transparent/openbsd
.. toctree::
:hidden:
:caption: Scripting
scripting/overview
scripting/events
scripting/api
.. toctree::
:hidden:
:caption: Tutorials
tutorials/30second
tutorials/gamecenter
tutorials/transparent-dhcp
.. toctree::
:hidden:
:caption: Pathod & Pathoc
pathod/intro
pathod/language
pathod/library
pathod/test
.. toctree::
:hidden:
:caption: Development
dev/contributing
dev/sslkeylogfile
.. Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`

159
docs/install.rst
View File

@ -1,159 +0,0 @@
.. _install:
Installation
============
Please follow the steps for your operating system.
Once installation is complete, you can run :ref:`mitmproxy`, :ref:`mitmdump` or
:ref:`mitmweb` from a terminal.
.. _install-macos:
Installation on macOS
---------------------
The recommended way to install mitmproxy on macOS is to use `Homebrew`_:
.. code:: bash
brew install mitmproxy
Alternatively you can download our :ref:`binary-packages` from our `releases`_
page.
.. _install-linux:
Installation on Linux
---------------------
The recommended way to install mitmproxy on Linux is to download our
:ref:`binary-packages` from our `releases`_ page.
Some Linux distributions and their community provide mitmproxy packages via
their native package repositories (e.g., Arch Linux, Debian, Ubuntu, Kali Linux,
OpenSUSE, etc.). While we do encourage seeing mitmproxy in a great variety of
repositories and distributions, we are not maintaining or involved with their
downstream packaging efforts. If you are looking for the latest version or have
other problems, please contact the repository maintainers directly.
.. _install-windows:
Installation on Windows
-----------------------
The recommended way to install mitmproxy on Windows is to download our
:ref:`binary-packages` from our `releases`_ page.
After installation, you'll find shortcuts for :ref:`mitmweb` (the web-based
interface) and :ref:`mitmdump` in the start menu. Both executables are added to
your PATH and can be invoked from the command line.
.. note::
The console interface is not supported on Windows, but you can
use `mitmweb` (the web-based interface) and `mitmdump`.
.. _install-advanced:
Advanced Installation
---------------------
.. _binary-packages:
Self-contained Pre-built Binary Packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For some platforms we provide pre-built binary packages containing ready-to-run
executables. This includes a self-contained Python 3 environment, a recent
OpenSSL that support ALPN and HTTP/2, and other dependencies that would
otherwise we cumbersome to compile and install.
Please be advised that we do not update these binaries after the initial
release. This means we do not include security-related updates of our
dependencies in already released mitmproxy versions. If there is a severe issue,
we might consider releasing a bugfix release of mitmproxy and corresponding
binary packages.
We only support the latest version of mitmproxy with bugfix and security updates
through regular minor releases.
.. _install-docker:
Docker Images
^^^^^^^^^^^^^
You can use the official mitmproxy images from `DockerHub`_. The same security
considerations apply as for our binary packages.
.. _install-linux-pip3:
Installation on Linux via pip3
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please make sure to install Python 3.5 (or higher) and pip3 for your
distribtion. If your distribution does not provide a suitable Python version,
you can use `pyenv`_ to get a recent Python environment.
.. code:: bash
sudo apt install python3-pip # Debian 8 or higher, Ubuntu 16.04 or higher
sudo dnf install python3-pip # Fedora 24 or higher
sudo pacman -S python-pip # Arch Linux
Please make sure to upgrade pip3 itself:
.. code:: bash
sudo pip3 install -U pip
Now you can install mitmproxy via pip3:
.. code:: bash
sudo pip3 install mitmproxy
.. _install-windows-pip3:
Installation on Windows via pip3
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. note::
The console interface is not supported on Windows, but you can
use `mitmweb` (the web-based interface) and `mitmdump`.
First, install the latest version of Python 3.5 or higher from the `Python
website`_. During installation, make sure to select `Add Python to PATH`. There
are no other dependencies on Windows.
Now you can install mitmproxy via pip3:
.. code:: powershell
pip3 install mitmproxy
.. _install-from-source:
Installation from Source Code
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you would like to install mitmproxy directly from source code or the GitHub
master branch, please see the our README_ on GitHub.
.. _README: https://github.com/mitmproxy/mitmproxy/blob/master/README.rst
.. _releases: https://github.com/mitmproxy/mitmproxy/releases/latest
.. _mitmproxy.org: https://mitmproxy.org/
.. _`Python website`: https://www.python.org/downloads/windows/
.. _pip: https://pip.pypa.io/en/latest/installing.html
.. _pyenv: https://github.com/yyuu/pyenv
.. _DockerHub: https://hub.docker.com/r/mitmproxy/mitmproxy/
.. _Homebrew: https://brew.sh/

66
docs/mitmdump.rst
View File

@ -1,66 +0,0 @@
.. _mitmdump:
.. program:: mitmdump
mitmdump
========
**mitmdump** is the command-line companion to mitmproxy. It provides
tcpdump-like functionality to let you view, record, and programmatically
transform HTTP traffic. See the ``--help`` flag output for complete
documentation.
Examples
--------
Saving traffic
^^^^^^^^^^^^^^
>>> mitmdump -w outfile
Start up mitmdump in proxy mode, and write all traffic to **outfile**.
Filtering saved traffic
^^^^^^^^^^^^^^^^^^^^^^^
>>> mitmdump -nr infile -w outfile "~m post"
Start mitmdump without binding to the proxy port (``-n``), read all flows from
infile, apply the specified filter expression (only match POSTs), and write to
outfile.
Client replay
^^^^^^^^^^^^^
>>> mitmdump -nc outfile
Start mitmdump without binding to the proxy port (``-n``), then replay all
requests from outfile (``-c filename``). Flags combine in the obvious way, so
you can replay requests from one file, and write the resulting flows to
another:
>>> mitmdump -nc srcfile -w dstfile
See the :ref:`clientreplay` section for more information.
Running a script
^^^^^^^^^^^^^^^^
>>> mitmdump -s examples/add_header.py
This runs the **add_header.py** example script, which simply adds a new header
to all responses.
Scripted data transformation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> mitmdump -ns examples/add_header.py -r srcfile -w dstfile
This command loads flows from **srcfile**, transforms it according to the
specified script, then writes it back to **dstfile**.

BIN
docs/mitmproxy-docs.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 15 KiB

125
docs/mitmproxy.rst
View File

@ -1,125 +0,0 @@
.. _mitmproxy:
.. program:: mitmproxy
mitmproxy
=========
**mitmproxy** is a console tool that allows interactive examination and
modification of HTTP traffic. It differs from mitmdump in that all flows are
kept in memory, which means that it's intended for taking and manipulating
small-ish samples. Use the :kbd:`?` shortcut key to view, context-sensitive
documentation from any **mitmproxy** screen.
Flow list
---------
The flow list shows an index of captured flows in chronological order.
.. image:: screenshots/mitmproxy.png
- **1**: A GET request, returning a 302 Redirect response.
- **2**: A GET request, returning 16.75kb of text/html data.
- **3**: A replayed request.
- **4**: Intercepted flows are indicated with orange text. The user may edit
these flows, and then accept them (using the :kbd:`a` key) to continue. In this
case, the request has been intercepted on the way to the server.
- **5**: A response intercepted from the server on the way to the client.
- **6**: The event log can be toggled on and off using the :kbd:`e` shortcut key. This
pane shows events and errors that may not result in a flow that shows up in the
flow pane.
- **7**: Flow count.
- **8**: Various information on mitmproxy's state. In this case, we have an
interception pattern set to ``.*``.
- **9**: Bind address indicator - mitmproxy is listening on port 8080 of all
interfaces.
Flow view
---------
The **Flow View** lets you inspect and manipulate a single flow:
.. image:: screenshots/mitmproxy-flowview.png
- **1**: Flow summary.
- **2**: The Request/Response tabs, showing you which part of the flow you are
currently viewing. In the example above, we're viewing the Response. Hit :kbd:`tab`
to switch between the Response and the Request.
- **3**: Headers.
- **4**: Body.
- **5**: View Mode indicator. In this case, we're viewing the body in **hex** mode. The other
available modes are **pretty**, which uses a number of heuristics to show you a friendly
view of various content types, and **raw**, which shows you exactly what's there without any
changes. You can change modes using the :kbd:`m` key.
Grid Editor
-----------
Much of the data that we'd like to interact with in mitmproxy is structured.
For instance, headers, queries and form data can all be thought of as a list of
key/value pairs. Mitmproxy has a built-in editor that lays this type of data
out in a grid for easy manipulation.
At the moment, the Grid Editor is used in four parts of mitmproxy:
- Editing request or response headers (:kbd:`e` for edit, then :kbd:`h` for headers in flow view)
- Editing a query string (:kbd:`e` for edit, then :kbd:`q` for query in flow view)
- Editing a URL-encoded form (:kbd:`e` for edit, then :kbd:`f` for form in flow view)
- Editing replacement patterns (:kbd:`O` for options, then :kbd:`R` for Replacement Patterns)
If there is is no data, an empty editor will be started to let you add some.
Here is the editor showing the headers from a request:
.. image:: screenshots/mitmproxy-kveditor.png
To edit, navigate to the key or value you want to modify using the arrow or vi
navigation keys, and press enter. The background color will change to show that
you are in edit mode for the specified field:
.. image:: screenshots/mitmproxy-kveditor-editmode.png
Modify the field as desired, then press escape to exit edit mode when you're
done. You can also add a row (:kbd:`a` key), delete a row (:kbd:`d` key), spawn an
external editor on a field (:kbd:`e` key). Be sure to consult the context-sensitive
help (:kbd:`?` key) for more.
Example: Interception
---------------------
**mitmproxy**'s interception functionality lets you pause an HTTP request or
response, inspect and modify it, and then accept it to send it on to the server
or client.
1: Set an interception pattern
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-filt.png
We press :kbd:`i` to set an interception pattern. In this case, the ``~q`` filter
pattern tells **mitmproxy** to intercept all requests. For complete filter
syntax, see the :ref:`filters` section of the documentation,
or the built-in help function in **mitmproxy**.
2: Intercepted connections are indicated with orange text:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-mid.png
3: You can now view and modify the request:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-options.png
In this case, we viewed the request by selecting it, pressed :kbd:`e` for "edit"
and :kbd:`m` for "method" to change the HTTP request method.
4: Accept the intercept to continue:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: screenshots/mitmproxy-intercept-result.png
Finally, we press :kbd:`a` to accept the modified request, which is then sent on to
the server. In this case, we changed the request from an HTTP GET to
OPTIONS, and Google's server has responded with a 405 "Method not allowed".

18
docs/mitmweb.rst
View File

@ -1,18 +0,0 @@
.. _mitmweb:
.. program:: mitmweb
mitmweb
=======
**mitmweb** is mitmproxy's web-based user interface that allows interactive
examination and modification of HTTP traffic. Like mitmproxy, it differs from
mitmdump in that all flows are kept in memory, which means that it's intended
for taking and manipulating small-ish samples.
.. warning::
Mitmweb is currently in beta. We consider it stable for all features currently
exposed in the UI, but it still misses a lot of mitmproxy's features.
.. image:: screenshots/mitmweb.png

12
docs/modd.conf
View File

@ -1,6 +1,8 @@
@build = ./_build
** !_build/** ../mitmproxy/**/*.py {
prep: sphinx-build -W -d @build/doctrees -b html . @build/html
daemon: devd -m @build/html
{
daemon: cd src; hugo server -D
}
style/** {
# This is quite fast, so it can stay a prep rather than a daemon
prep: ./node_modules/.bin/node-sass -o ./src/themes/mitmproxydocs/static/css ./style/style.scss --error-bell
}

193
docs/modes.rst
View File

@ -1,193 +0,0 @@
.. _modes:
Modes of Operation
==================
Mitmproxy has four modes of operation that allow you to use mitmproxy in a
variety of scenarios:
- **Regular** (the default)
- **Transparent**
- **Reverse Proxy**
- **Upstream Proxy**
Now, which one should you pick? Use this flow chart:
.. image:: schematics/proxy-modes-flowchart.png
:align: center
Regular Proxy
-------------
Mitmproxy's regular mode is the simplest and the easiest to set up.
1. Start mitmproxy.
2. Configure your client to use mitmproxy by explicitly setting an HTTP proxy.
3. Quick Check: You should already be able to visit an unencrypted HTTP site through the proxy.
4. Open the magic domain **mitm.it** and install the certificate for your device.
.. note::
Unfortunately, some applications bypass the system HTTP proxy settings - Android applications
are a common example. In these cases, you need to use mitmproxy's transparent mode.
If you are proxying an external device, your network will probably look like this:
.. image:: schematics/proxy-modes-regular.png
:align: center
The square brackets signify the source and destination IP addresses. Your
client explicitly connects to mitmproxy and mitmproxy explicitly connects
to the target server.
Transparent Proxy
-----------------
In transparent mode, traffic is directed into a proxy at the network layer,
without any client configuration required. This makes transparent proxying
ideal for situations where you can't change client behaviour. In the graphic
below, a machine running mitmproxy has been inserted between the router and
the internet:
.. image:: schematics/proxy-modes-transparent-1.png
:align: center
The square brackets signify the source and destination IP addresses. Round
brackets mark the next hop on the *Ethernet/data link* layer. This distinction
is important: when the packet arrives at the mitmproxy machine, it must still
be addressed to the target server. This means that Network Address Translation
should not be applied before the traffic reaches mitmproxy, since this would
remove the target information, leaving mitmproxy unable to determine the real
destination.
.. image:: schematics/proxy-modes-transparent-wrong.png
:align: center
Common Configurations
^^^^^^^^^^^^^^^^^^^^^
There are many ways to configure your network for transparent proxying. We'll
look at two common scenarios:
1. Configuring the client to use a custom gateway/router/"next hop"
2. Implementing custom routing on the router
In most cases, the first option is recommended due to its ease of use.
(a) Custom Gateway
~~~~~~~~~~~~~~~~~~
One simple way to get traffic to the mitmproxy machine with the destination IP
intact, is to simply configure the client with the mitmproxy box as the
default gateway.
.. image:: schematics/proxy-modes-transparent-2.png
:align: center
In this scenario, we would:
1. Configure the proxy machine for transparent mode. You can find instructions
in the :ref:`transparent` section.
2. Configure the client to use the proxy machine's IP as the default gateway.
3. Quick Check: At this point, you should already be able to visit an
unencrypted HTTP site over the proxy.
4. Open the magic domain **mitm.it** and install the certificate
for your device.
Setting the custom gateway on clients can be automated by serving the settings
out to clients over DHCP. This lets set up an interception network where all
clients are proxied automatically, which can save time and effort.
.. admonition:: Troubleshooting Transparent Mode
:class: note
Incorrect transparent mode configurations are a frequent source of
error. If it doesn't work for you, try the following things:
- Open mitmproxy's event log (press :kbd:`e`) - do you see clientconnect messages?
If not, the packets are not arriving at the proxy. One common cause is the occurrence of ICMP
redirects, which means that your machine is telling the client that there's a faster way to
the internet by contacting your router directly (see the :ref:`transparent` section on how to
disable them). If in doubt, Wireshark_ may help you to see whether something arrives at your
machine or not.
- Make sure you have not explicitly configured an HTTP proxy on the client.
This is not needed in transparent mode.
- Re-check the instructions in the :ref:`transparent` section. Anything you missed?
If you encounter any other pitfalls that should be listed here, please let us know!
(b) Custom Routing
~~~~~~~~~~~~~~~~~~
In some cases, you may need more fine-grained control of which traffic reaches
the mitmproxy instance, and which doesn't. You may, for instance, choose only
to divert traffic to some hosts into the transparent proxy. There are a huge
number of ways to accomplish this, and much will depend on the router or
packet filter you're using. In most cases, the configuration will look like
this:
.. image:: schematics/proxy-modes-transparent-3.png
:align: center
Reverse Proxy
-------------
mitmproxy is usually used with a client that uses the proxy to access the
Internet. Using reverse proxy mode, you can use mitmproxy to act like a normal
HTTP server:
.. image:: schematics/proxy-modes-reverse.png
:align: center
There are various use-cases:
- Say you have an internal API running at http://example.local/. You could now
set up mitmproxy in reverse proxy mode at http://debug.example.local/ and
dynamically point clients to this new API endpoint, which provides them
with the same data and you with debug information. Similarly, you could move
your real server to a different IP/port and set up mitmproxy in the original
place to debug and or redirect all sessions.
- Say you're a web developer working on http://example.com/ (with a development
version running on http://localhost:8000/). You can modify your hosts file so that
example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy mode
on port 80. You can test your app on the example.com domain and get all
requests recorded in mitmproxy.
- Say you have some toy project that should get SSL support. Simply set up
mitmproxy as a reverse proxy on port 443 and you're done (``mitmdump -p 443 -R
http://localhost:80/``). Mitmproxy auto-detects TLS traffic and intercepts it dynamically.
There are better tools for this specific task, but mitmproxy is very quick and simple way to
set up an SSL-speaking server.
- Want to add a non-SSL-capable compression proxy in front of your server? You
could even spawn a mitmproxy instance that terminates SSL (``-R http://...``),
point it to the compression proxy and let the compression proxy point to a
SSL-initiating mitmproxy (``-R https://...``), which then points to the real
server. As you see, it's a fairly flexible thing.
.. admonition:: Caveat: Interactive Use
:class: warning
Reverse Proxy mode is usually not sufficient to create a copy of an interactive website at
different URL. The HTML served to the client remains unchanged - as soon as the user clicks on
an non-relative URL (or downloads a non-relative image resource), traffic no longer passes
through mitmproxy.
Upstream Proxy
--------------
If you want to chain proxies by adding mitmproxy in front of a different proxy
appliance, you can use mitmproxy's upstream mode. In upstream mode, all
requests are unconditionally transferred to an upstream proxy of your choice.
.. image:: schematics/proxy-modes-upstream.png
:align: center
mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy
mode. You could in theory chain multiple mitmproxy instances in a row, but
that doesn't make any sense in practice (i.e. outside of our tests).
.. _Wireshark: https://wireshark.org/

6
docs/package.json Normal file
View File

@ -0,0 +1,6 @@
{
"dependencies": {
"bulma": "^0.6.1",
"node-sass": "^4.7.2"
}
}

307
docs/pathod/intro.rst
View File

@ -1,307 +0,0 @@
.. _intro:
Pathology 101
=============
.. _pathod:
pathod
------
Pathod is a pathological HTTP daemon designed to let you craft almost any
conceivable HTTP response, including ones that creatively violate the
standards. HTTP responses are specified using a :ref:`small, terse language
<language>` which pathod shares with its evil twin :ref:`pathoc`. To start
playing with pathod, fire up the daemon:
>>> pathod
By default, the service listens on port 9999 of localhost, and the default
crafting anchor point is the path **/p/**. Anything after this URL prefix is
treated as a response specifier. So, hitting the following URL will generate an
HTTP 200 response with 100 bytes of random data:
http://localhost:9999/p/200:b@100
See the :ref:`language documentation <language>` to get (much) fancier. The
pathod daemon also takes a range of configuration options. To view those, use
the command-line help:
>>> pathod --help
Mimicing a proxy
^^^^^^^^^^^^^^^^
Pathod automatically responds to both straight HTTP and proxy requests. For
proxy requests, the upstream host is ignored, and the path portion of the URL
is used to match anchors. This lets you test software that supports a proxy
configuration by spoofing responses from upstream servers.
By default, we treat all proxy CONNECT requests as HTTPS traffic, serving the
response using either pathod's built-in certificates, or the cert/key pair
specified by the user. You can over-ride this behaviour if you're testing a
client that makes a non-SSL CONNECT request using the **-C** command-line
option.
Anchors
^^^^^^^
Anchors provide an alternative to specifying the response in the URL. Instead,
you attach a response to a pre-configured anchor point, specified with a regex.
When a URL matching the regex is requested, the specified response is served.
>>> pathod -a "/foo=200"
Here, "/foo" is the regex specifying the anchor path, and the part after the "="
is a response specifier.
File Access
^^^^^^^^^^^
There are two operators in the :ref:`language <language>` that load contents
from file - the **+** operator to load an entire request specification from
file, and the **>** value specifier. In pathod, both of these operators are
restricted to a directory specified at startup, or disabled if no directory is
specified:
>>> pathod -d ~/staticdir"
Internal Error Responses
^^^^^^^^^^^^^^^^^^^^^^^^
Pathod uses the non-standard 800 response code to indicate internal errors, to
distinguish them from crafted responses. For example, a request to:
http://localhost:9999/p/foo
... will return an 800 response because "foo" is not a valid page specifier.
.. _pathoc:
pathoc
------
Pathoc is a perverse HTTP daemon designed to let you craft almost any
conceivable HTTP request, including ones that creatively violate the standards.
HTTP requests are specified using a :ref:`small, terse language <language>`,
which pathoc shares with its server-side twin :ref:`pathod`. To view pathoc's
complete range of options, use the command-line help:
>>> pathoc --help
Getting Started
^^^^^^^^^^^^^^^
The basic pattern for pathoc commands is as follows:
pathoc hostname request [request ...]
That is, we specify the hostname to connect to, followed by one or more
requests. Lets start with a simple example::
> pathoc google.com get:/
07-06-16 12:13:43: >> 'GET':/
<< 302 Found: 261 bytes
Here, we make a GET request to the path / on port 80 of google.com. Pathoc's
output tells us that the server responded with a 302 redirection. We can tell
pathoc to connect using SSL, in which case the default port is changed to 443
(you can over-ride the default port with the **-p** command-line option)::
> pathoc -s www.google.com get:/
07-06-16 12:14:56: >> 'GET':/
<< 302 Found: 262 bytes
Multiple Requests
^^^^^^^^^^^^^^^^^
There are two ways to tell pathoc to issue multiple requests. The first is to specify
them on the command-line, like so::
> pathoc google.com get:/ get:/
07-06-16 12:21:04: >> 'GET':/
<< 302 Found: 261 bytes
07-06-16 12:21:04: >> 'GET':/
<< 302 Found: 261 bytes
In this case, pathoc issues the specified requests over the same TCP connection -
so in the above example only one connection is made to google.com
The other way to issue multiple requests is to use the **-n** flag::
> pathoc -n 2 google.com get:/
07-06-16 12:21:04: >> 'GET':/
<< 302 Found: 261 bytes
07-06-16 12:21:04: >> 'GET':/
<< 302 Found: 261 bytes
The output is identical, but two separate TCP connections are made to the
upstream server. These two specification styles can be combined::
pathoc -n 2 google.com get:/ get:/
Here, two distinct TCP connections are made, with two requests issued over
each.
Basic Fuzzing
^^^^^^^^^^^^^
The combination of pathoc's powerful request specification language and a few
of its command-line options makes for quite a powerful basic fuzzer. Here's an
example::
pathoc -e -I 200 -t 2 -n 1000 localhost get:/:b@10:ir,@1
The request specified here is a valid GET with a body consisting of 10 random bytes,
but with 1 random byte inserted in a random place. This could be in the headers,
in the initial request line, or in the body itself. There are a few things
to note here:
- Corrupting the request in this way will often make the server enter a state where
it's awaiting more input from the client. This is where the
**-t** option comes in, which sets a timeout that causes pathoc to
disconnect after two seconds.
- The **-n** option tells pathoc to repeat the request 1000 times.
- The **-I** option tells pathoc to ignore HTTP 200 response codes.
You can use this to fine-tune what pathoc considers to be an exceptional
condition, and therefore log-worthy.
- The **-e** option tells pathoc to print an explanation of each logged
request, in the form of an expanded pathoc specification with all random
portions and automatic header additions resolved. This lets you precisely
replay a request that triggered an error.
Interacting with Proxies
^^^^^^^^^^^^^^^^^^^^^^^^
Pathoc has a reasonably sophisticated suite of features for interacting with
proxies. The proxy request syntax very closely mirrors that of straight HTTP,
which means that it is possible to make proxy-style requests using pathoc
without any additional syntax, by simply specifying a full URL instead of a
simple path:
>>> pathoc -p 8080 localhost "get:'http://google.com'"
Another common use case is to use an HTTP CONNECT request to probe remote
servers via a proxy. This is done with the **-c** command-line option, which
allows you to specify a remote host and port pair:
>>> pathoc -c google.com:80 -p 8080 localhost get:/
Note that pathoc does **not** negotiate SSL without being explictly instructed
to do so. If you're making a CONNECT request to an SSL-protected resource, you
must also pass the **-s** flag:
>>> pathoc -sc google.com:443 -p 8080 localhost get:/
Embedded response specification
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
One interesting feature of the Request specification language is that you can
embed a response specification in it, which is then added to the request path.
Here's an example:
>>> pathoc localhost:9999 "get:/p/:s'401:ir,@1'"
This crafts a request that connects to the pathod server, and which then crafts
a response that generates a 401, with one random byte embedded at a random
point. The response specification is parsed and expanded by pathoc, so you see
syntax errors immediately. This really becomes handy when combined with the
**-e** flag to show the expanded request::
07-06-16 12:32:01: >> 'GET':/p/:s'401:i35,\x27\\x1b\x27:h\x27Content-Length\x27=\x270\x27:h\x27Content-Length\x27=\x270\x27':h'Host'='localhost'
<< 401 Unauthorized: 0 bytes
Note that the embedded response has been resolved *before* being sent to
the server, so that "ir,@1" (embed a random byte at a random location) has
become "i15,\'o\'" (embed the character "o" at offset 15). You now have a
pathoc request specification that is precisely reproducible, even with random
components. This feature comes in terribly handy when testing a proxy, since
you can now drive the server response completely from the client, and have a
complete log of reproducible requests to analyze afterwards.
Request Examples
----------------
.. list-table::
:widths: 50 50
:header-rows: 0
* - get:/
- Get path /
* - get:/:b@100
- 100 random bytes as the body
* - get:/:h"Etag"="&;drop table browsers;"
- Add a header
* - get:/:u"&;drop table browsers;"
- Add a User-Agent header
* - get:/:b@100:dr
- Drop the connection randomly
* - get:/:b@100,ascii:ir,@1
- 100 ASCII bytes as the body, and randomly inject a random byte
* - ws:/
- Initiate a websocket handshake.
Response Examples
-----------------
.. list-table::
:widths: 50 50
:header-rows: 0
* - 200
- A basic HTTP 200 response.
* - 200:r
- A basic HTTP 200 response with no Content-Length header. This will hang.
* - 200:da
- Server-side disconnect after all content has been sent.
* - 200:b\@100
- 100 random bytes as the body. A Content-Length header is added, so the disconnect
is no longer needed.
* - 200:b\@100:h"Etag"="';drop table servers;"
- Add a Server header
* - 200:b\@100:dr
- Drop the connection randomly
* - 200:b\@100,ascii:ir,@1
- 100 ASCII bytes as the body, and randomly inject a random byte
* - 200:b\@1k:c"text/json"
- 1k of random bytes, with a text/json content type
* - 200:b\@1k:p50,120
- 1k of random bytes, pause for 120 seconds after 50 bytes
* - 200:b\@1k:pr,f
- 1k of random bytes, but hang forever at a random location
* - 200:b\@100:h\@1k,ascii_letters='foo'
- 100 ASCII bytes as the body, randomly generated 100k header name, with the value
'foo'.

257
docs/pathod/language.rst
View File

@ -1,257 +0,0 @@
.. _language:
language spec
=============
************
HTTP Request
************
**method:path:[colon-separated list of features]**
.. list-table::
:widths: 20 80
:header-rows: 0
* - method
- A :ref:`VALUE` specifying the HTTP method to
use. Standard methods do not need to be enclosed in quotes, while
non-standard methods can be specified as quoted strings.
The special method **ws** creates a valid websocket upgrade
GET request, and signals to pathoc to switch to websocket recieve
mode if the server responds correctly. Apart from that, websocket
requests are just like any other, and all aspects of the request
can be over-ridden.
* - h\ :ref:`VALUE`\ =\ :ref:`VALUE`\
- Set a header.
* - r
- Set the **raw** flag on this response. Pathod will not calculate a
*Content-Length* header if a body is set.
* - c\ :ref:`VALUE`
- A shortcut for setting the Content-Type header. Equivalent to
``h"Content-Type"=VALUE``
* - u\ :ref:`VALUE`
uSHORTCUT
- Set a User-Agent header on this request. You can specify either a
complete :ref:`VALUE`, or a User-Agent shortcut: **android**,
**blackberry**, **bingbot**, **chrome**, **firefox**, **googlebot**,
**ie9**, **ipad**, **iphone**, **safari**.
* - b\ :ref:`VALUE`
- Set the body. The appropriate Content-Length header is added
automatically unless the **r** flag is set.
* - s\ :ref:`VALUE`
- An embedded Response specification, appended to the path of the request.
* - x\ :ref:`INTEGER`
- Repeat this message N times.
* - d\ :ref:`OFFSET`
- Disconnect after OFFSET bytes (HTTP/1 only).
* - i\ :ref:`OFFSET`,\ :ref:`VALUE`
- Inject the specified value at the offset (HTTP/1 only)
* - p\ :ref:`OFFSET`,SECONDS
- Pause for SECONDS seconds after OFFSET bytes. SECONDS can be an integer
or "f" to pause forever (HTTP/1 only)
*************
HTTP Response
*************
**code:[colon-separated list of features]**
.. list-table::
:widths: 20 80
:header-rows: 0
* - code
- An integer specifying the HTTP response code.
The special method **ws** creates a valid websocket upgrade
response (code 101), and moves pathod to websocket mode. Apart
from that, websocket responses are just like any other, and all
aspects of the response can be over-ridden.
* - m\ :ref:`VALUE`
- HTTP Reason message. Automatically chosen according to the response
code if not specified. (HTTP/1 only)
* - h\ :ref:`VALUE`\ =\ :ref:`VALUE`\
- Set a header.
* - r
- Set the **raw** flag on this response. Pathod will not calculate a
*Content-Length* header if a body is set.
* - l\ :ref:`VALUE`
- A shortcut for setting the Location header. Equivalent to
``h"Location"=VALUE``
* - c\ :ref:`VALUE`
- A shortcut for setting the Content-Type header. Equivalent to
``h"Content-Type"=VALUE``
* - b\ :ref:`VALUE`
- Set the body. The appropriate Content-Length header is added
automatically unless the **r** flag is set.
* - d\ :ref:`OFFSET`
- Disconnect after OFFSET bytes (HTTP/1 only).
* - i\ :ref:`OFFSET`,\ :ref:`VALUE`
- Inject the specified value at the offset (HTTP/1 only)
* - p\ :ref:`OFFSET`,SECONDS
- Pause for SECONDS seconds after OFFSET bytes. SECONDS can be an integer
or "f" to pause forever (HTTP/1 only)
***************
Websocket Frame
***************
**wf:[colon-separated list of features]**
.. list-table::
:widths: 20 80
:header-rows: 0
* - b\ :ref:`VALUE`
- Set the frame payload. If a masking key is present, the value is
encoded automatically.
* - c\ :ref:`INTEGER`
- Set the op code. This can either be an integer from 0-15, or be one of
the following opcode names: **text** (the default), **continue**,
**binary**, **close**, **ping**, **pong**.
* - d\ :ref:`OFFSET`
- Disconnect after OFFSET bytes
* - i\ :ref:`OFFSET`,\ :ref:`VALUE`
- Inject the specified value at the offset
* - p\ :ref:`OFFSET`,SECONDS
- Pause for SECONDS seconds after OFFSET bytes. SECONDS can be an integer
or "f" to pause forever
* - x\ :ref:`INTEGER`
- Repeat this message N times.
* - [-]fin
- Set or un-set the **fin** bit.
* - k\ :ref:`VALUE`
- Set the masking key. The resulting value must be exactly 4 bytes long.
The special form **knone** specifies that no key should be set, even if
the mask bit is on.
* - l\ :ref:`INTEGER`
- Set the payload length in the frame header, regardless of the actual
body length.
* - [-]mask
- Set or un-set the <b>mask</b> bit.
* - r\ :ref:`VALUE`
- Set the raw frame payload. This disables masking, even if the key is present.
* - [-]rsv1
- Set or un-set the **rsv1** bit.
* - [-]rsv2
- Set or un-set the **rsv2** bit.
* - [-]rsv2
- Set or un-set the **rsv2** bit.
**********
Data types
**********
.. _INTEGER:
INTEGER
^^^^^^^
.. _OFFSET:
OFFSET
^^^^^^
Offsets are calculated relative to the base message, before any injections or
other transforms are applied. They have 3 flavors:
======= ==========================
integer An integer byte offset
**r** A random location
**a** The end of the message
======= ==========================
.. _VALUE:
VALUE
^^^^^
Literals
""""""""
Literal values are specified as a quoted strings::
"foo"
Either single or double quotes are accepted, and quotes can be escaped with
backslashes within the string::
'fo\'o'
Literal values can contain Python-style backslash escape sequences::
'foo\r\nbar'
Generated
"""""""""
An @-symbol lead-in specifies that generated data should be used. There are two
components to a generator specification - a size, and a data type. By default
pathod assumes a data type of "bytes".
Here's a value specifier for generating 100 bytes::
@100
You can use standard suffixes to indicate larger values. Here, for instance, is
a specifier for generating 100 megabytes:
@100m
Data is generated and served efficiently - if you really want to send a
terabyte of data to a client, pathod can do it. The supported suffixes are:
========== ====================
b 1024**0 (bytes)
k 1024**1 (kilobytes)
m 1024**2 (megabytes)
g 1024**3 (gigabytes)
t 1024**4 (terabytes)
========== ====================
Data types are separated from the size specification by a comma. This specification
generates 100mb of ASCII::
@100m,ascii
Supported data types are:
================= ==============================================
ascii All ASCII characters
ascii_letters A-Za-z
ascii_lowercase a-z
ascii_uppercase A-Z
bytes All 256 byte values
digits 0-9
hexdigits 0-f
octdigits 0-7
punctuation !"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ and space
whitespace \\t \\n \\x0b \\x0c \\r and space
================= ==============================================
Files
"""""
You can load a value from a specified file path. To do so, you have to specify a
_staticdir_ option to pathod on the command-line, like so:
>>> pathod -d ~/myassets
All paths are relative paths under this directory. File loads are indicated by
starting the value specifier with the left angle bracket::
<my/path
The path value can also be a quoted string, with the same syntax as literals::
<"my/path"

14
docs/pathod/library.rst
View File

@ -1,14 +0,0 @@
.. _library:
pathod library
==============
Behind the pathod and pathoc command-line tools lurks the **pathod** library, a
powerful way to manipulate and serve HTTP requests and responses from code. The
canonical documentation for the library is in the code, and can be accessed
using pydoc.
.. literalinclude:: ../../examples/pathod/libpathod_pathoc.py
:caption: examples/pathod/libpathod_pathoc.py
:language: python

38
docs/pathod/test.rst
View File

@ -1,38 +0,0 @@
.. _test:
pathod.test
===========
The **pathod.test** module is a light, flexible testing layer for HTTP clients.
It works by firing up a Pathod instance in a separate thread, letting you use
Pathod's full abilities to generate responses, and then query Pathod's internal
logs to establish what happened. All the mechanics of startup, shutdown, finding
free ports and so forth are taken care of for you.
The canonical docs can be accessed using pydoc:
>>> pydoc pathod.test
The remainder of this page demonstrates some common interaction patterns using
`Nose`_. These examples are
also applicable with only minor modification to most commonly used Python testing
engines.
Context Manager
---------------
.. literalinclude:: ../../examples/pathod/test_context.py
:caption: examples/pathod/test_context.py
:language: python
One instance per test
---------------------
.. literalinclude:: ../../examples/pathod/test_setup.py
:caption: examples/pathod/test_setup.py
:language: python
.. _Nose: https://nose.readthedocs.org/en/latest/

15
docs/protocols/http1.rst
View File

@ -1,15 +0,0 @@
.. _http1_protocol:
HTTP/1.0 and HTTP/1.1
===========================
.. seealso::
- `RFC7230: HTTP/1.1: Message Syntax and Routing <http://tools.ietf.org/html/rfc7230>`_
- `RFC7231: HTTP/1.1: Semantics and Content <http://tools.ietf.org/html/rfc7231>`_
HTTP/1.0 and HTTP/1.1 support in mitmproxy is based on our custom HTTP stack,
which takes care of all semantics and on-the-wire parsing/serialization tasks.
mitmproxy currently does not support HTTP trailers - but if you want to send
us a PR, we promise to take look!

16
docs/protocols/http2.rst
View File

@ -1,16 +0,0 @@
.. _http2_protocol:
HTTP/2
======
.. seealso::
- `RFC7540: Hypertext Transfer Protocol Version 2 (HTTP/2) <http://tools.ietf.org/html/rfc7540>`_
HTTP/2 support in mitmproxy is based on the amazing work by the python-hyper
community with the `hyper-h2 <https://github.com/python-hyper/hyper-h2>`_
project. It fully encapsulates the internal state of HTTP/2 connections and
provides an easy-to-use event-based API.
mitmproxy currently does not support HTTP/2 trailers - but if you want to send
us a PR, we promise to take look!

31
docs/protocols/tcpproxy.rst
View File

@ -1,31 +0,0 @@
.. _tcp_proxy:
TCP Proxy / Fallback
====================
In case mitmproxy does not handle a specific protocol, you can exempt
hostnames from processing, so that mitmproxy acts as a generic TCP forwarder.
This feature is closely related to the :ref:`passthrough` functionality,
but differs in two important aspects:
- The raw TCP messages are printed to the event log.
- SSL connections will be intercepted.
Please note that message interception or modification are not possible yet.
If you are not interested in the raw TCP messages, you should use the ignore domains feature.
How it works
------------
================== ======================
command-line ``--tcp HOST``
mitmproxy shortcut :kbd:`O` then :kbd:`T`
================== ======================
For a detailed description how the hostname pattern works, please look at the :ref:`passthrough`
feature.
.. seealso::
- :ref:`passthrough`
- :ref:`streaming`

22
docs/protocols/websocket.rst
View File

@ -1,22 +0,0 @@
.. _websocket_protocol:
WebSocket
=========
.. seealso::
- `RFC6455: The WebSocket Protocol <http://tools.ietf.org/html/rfc6455>`_
- `RFC7692: Compression Extensions for WebSocket <http://tools.ietf.org/html/rfc7692>`_
WebSocket support in mitmproxy is based on the amazing work by the python-hyper
community with the `wsproto <https://github.com/python-hyper/wsproto>`_
project. It fully encapsulates WebSocket frames/messages/connections and
provides an easy-to-use event-based API.
mitmproxy fully supports the compression extension for WebSocket messages,
provided by wsproto.
If an endpoint sends a PING to mitmproxy, a PONG will be sent back immediately
(with the same payload if present). To keep the other connection alive, a new
PING (without a payload) is sent to the other endpoint. Unsolicited PONG's are
not forwarded. All PING's and PONG's are logged (with payload if present).

0
docs/schematics/_explicit.graffle/data.plist → docs/raw/_explicit.graffle/data.plist
View File

0
docs/schematics/_explicit.graffle/image3.icns → docs/raw/_explicit.graffle/image3.icns
View File

0
docs/schematics/_explicit.graffle/image6.tiff → docs/raw/_explicit.graffle/image6.tiff
View File

0
docs/schematics/_explicit_https.graffle/data.plist → docs/raw/_explicit_https.graffle/data.plist
View File

0
docs/schematics/_explicit_https.graffle/image3.icns → docs/raw/_explicit_https.graffle/image3.icns
View File

0
docs/schematics/_explicit_https.graffle/image6.tiff → docs/raw/_explicit_https.graffle/image6.tiff
View File

0
docs/schematics/_transparent.graffle/data.plist → docs/raw/_transparent.graffle/data.plist
View File

0
docs/schematics/_transparent.graffle/image3.icns → docs/raw/_transparent.graffle/image3.icns
View File

0
docs/schematics/_transparent.graffle/image6.tiff → docs/raw/_transparent.graffle/image6.tiff
View File

0
docs/schematics/_transparent_https.graffle/data.plist → docs/raw/_transparent_https.graffle/data.plist
View File

0
docs/schematics/_transparent_https.graffle/image3.icns → docs/raw/_transparent_https.graffle/image3.icns
View File

0
docs/schematics/_transparent_https.graffle/image6.tiff → docs/raw/_transparent_https.graffle/image6.tiff
View File

0
docs/schematics/proxy-modes.vsdx → docs/raw/proxy-modes.vsdx
View File

BIN
docs/screenshots/firefox3-import.jpg
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 54 KiB

BIN
docs/screenshots/firefox3-trust.jpg
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 31 KiB

BIN
docs/screenshots/firefox3.jpg
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 56 KiB

BIN
docs/screenshots/ios-gateway.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 151 KiB

BIN
docs/screenshots/ios-installed.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 78 KiB

BIN
docs/screenshots/ios-manual.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 192 KiB

BIN
docs/screenshots/ios-profile.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 81 KiB

BIN
docs/screenshots/ios-reverse.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 65 KiB

BIN
docs/screenshots/ios-warning.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 74 KiB

BIN
docs/screenshots/mitmproxy-flowview.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 308 KiB

BIN
docs/screenshots/mitmproxy-intercept-filt.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 18 KiB

BIN
docs/screenshots/mitmproxy-intercept-mid.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 19 KiB

BIN
docs/screenshots/mitmproxy-intercept-options.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 40 KiB

BIN
docs/screenshots/mitmproxy-intercept-result.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 22 KiB

BIN
docs/screenshots/mitmproxy-kveditor-editmode.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 44 KiB

BIN
docs/screenshots/mitmproxy-kveditor.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 44 KiB

BIN
docs/screenshots/mitmproxy.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 149 KiB

BIN
docs/screenshots/osx-addcert-alwaystrust.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 46 KiB

BIN
docs/screenshots/win7-certstore-trustedroot.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 38 KiB

BIN
docs/screenshots/win7-certstore.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 37 KiB

BIN
docs/screenshots/win7-wizard.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 65 KiB

BIN
docs/screenshots/winpythoninstaller.jpg
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 46 KiB

54
docs/scripting/api.rst
View File

@ -1,54 +0,0 @@
.. _api:
API
===
- Errors
- `mitmproxy.flow.Error <#mitmproxy.flow.Error>`_
- HTTP
- `mitmproxy.http.HTTPRequest <#mitmproxy.http.HTTPRequest>`_
- `mitmproxy.http.HTTPResponse <#mitmproxy.http.HTTPResponse>`_
- `mitmproxy.http.HTTPFlow <#mitmproxy.http.HTTPFlow>`_
- WebSocket
- `mitmproxy.websocket.WebSocketFlow <#mitmproxy.websocket.WebSocketFlow>`_
- `mitmproxy.websocket.WebSocketMessage <#mitmproxy.websocket.WebSocketMessage>`_
- Logging
- `mitmproxy.log.Log <#mitmproxy.controller.Log>`_
- `mitmproxy.log.LogEntry <#mitmproxy.controller.LogEntry>`_
Errors
------
.. autoclass:: mitmproxy.flow.Error
:inherited-members:
HTTP
----
.. autoclass:: mitmproxy.http.HTTPRequest
:inherited-members:
.. autoclass:: mitmproxy.http.HTTPResponse
:inherited-members:
.. autoclass:: mitmproxy.http.HTTPFlow
:inherited-members:
WebSocket
---------
.. autoclass:: mitmproxy.websocket.WebSocketFlow
:inherited-members:
.. autoclass:: mitmproxy.websocket.WebSocketMessage
:inherited-members:
Logging
--------
.. autoclass:: mitmproxy.log.Log
:inherited-members:
.. autoclass:: mitmproxy.log.LogEntry
:inherited-members:

248
docs/scripting/events.rst
View File

@ -1,248 +0,0 @@
.. _events:
Events
=======
General
-------
.. list-table::
:widths: 40 60
:header-rows: 0
* - .. py:function:: configure(options, updated)
- Called once on startup, and whenever options change.
*options*
An ``options.Options`` object with the total current configuration
state of mitmproxy.
*updated*
A set of strings indicating which configuration options have been
updated. This contains all options when *configure* is called on
startup, and only changed options subsequently.
* - .. py:function:: done()
- Called once when the script shuts down, either because it's been
unloaded, or because the proxy itself is shutting down.
* - .. py:function:: log(entry)
- Called whenever an event log is added.
*entry*
An ``controller.LogEntry`` object - ``entry.msg`` is the log text,
and ``entry.level`` is the urgency level ("debug", "info", "warn",
"error").
* - .. py:function:: start()
- Called once on startup, before any other events. If you return a
value from this event, it will replace the current addon. This
allows you to, "boot into" an addon implemented as a class instance
from the module level.
* - .. py:function:: tick()
- Called at a regular sub-second interval as long as the addon is
executing.
Connection
----------
.. list-table::
:widths: 40 60
:header-rows: 0
* - .. py:function:: clientconnect(root_layer)
- Called when a client initiates a connection to the proxy. Note that a
connection can correspond to multiple HTTP requests.
*root_layer*
The root layer (see `mitmproxy.proxy.protocol` for an explanation what
the root layer is), provides transparent access to all attributes
of the :py:class:`~mitmproxy.proxy.RootContext`. For example,
``root_layer.client_conn.address`` gives the remote address of the
connecting client.
* - .. py:function:: clientdisconnect(root_layer)
- Called when a client disconnects from the proxy.
*root_layer*
The root layer object.
* - .. py:function:: next_layer(layer)
- Called whenever layers are switched. You may change which layer will
be used by returning a new layer object from this event.
*layer*
The next layer, as determined by mitmpmroxy.
* - .. py:function:: serverconnect(server_conn)
- Called before the proxy initiates a connection to the target server.
Note that a connection can correspond to multiple HTTP requests.
*server_conn*
A ``ServerConnection`` object. It is guaranteed to have a non-None
``address`` attribute.
* - .. py:function:: serverdisconnect(server_conn)
- Called when the proxy has closed the server connection.
*server_conn*
A ``ServerConnection`` object.
HTTP Events
-----------
.. list-table::
:widths: 40 60
:header-rows: 0
* - .. py:function:: http_connect(flow)
- Called when we receive an HTTP CONNECT request. Setting a non 2xx
response on the flow will return the response to the client and abort
the connection. CONNECT requests and responses do not generate the
usual HTTP handler events. CONNECT requests are only valid in regular
and upstream proxy modes.
*flow*
A ``models.HTTPFlow`` object. The flow is guaranteed to have
non-None ``request`` and ``requestheaders`` attributes.
* - .. py:function:: request(flow)
- Called when a client request has been received.
*flow*
A ``models.HTTPFlow`` object. At this point, the flow is
guaranteed to have a non-None ``request`` attribute.
* - .. py:function:: requestheaders(flow)
- Called when the headers of a client request have been received, but
before the request body is read.
*flow*
A ``models.HTTPFlow`` object. At this point, the flow is
guaranteed to have a non-None ``request`` attribute.
* - .. py:function:: responseheaders(flow)
- Called when the headers of a server response have been received, but
before the response body is read.
*flow*
A ``models.HTTPFlow`` object. At this point, the flow is
guaranteed to have a non-none ``request`` and ``response``
attributes, however the response will have no content.
* - .. py:function:: response(flow)
- Called when a server response has been received.
*flow*
A ``models.HTTPFlow`` object. At this point, the flow is
guaranteed to have a non-none ``request`` and ``response``
attributes. The raw response body will be in ``response.body``,
unless response streaming has been enabled.
* - .. py:function:: error(flow)
- Called when a flow error has occurred, e.g. invalid server responses,
or interrupted connections. This is distinct from a valid server HTTP
error response, which is simply a response with an HTTP error code.
*flow*
The flow containing the error. It is guaranteed to have
non-None ``error`` attribute.
WebSocket Events
-----------------
These events are called only after a connection made an HTTP upgrade with
"101 Switching Protocols". No further HTTP-related events after the handshake
are issued, only new WebSocket messages are called.
.. list-table::
:widths: 40 60
:header-rows: 0
* - .. py:function:: websocket_handshake(flow)
- Called when a client wants to establish a WebSocket connection. The
WebSocket-specific headers can be manipulated to alter the
handshake. The ``flow`` object is guaranteed to have a non-None
``request`` attribute.
*flow*
The flow containing the HTTP WebSocket handshake request. The
object is guaranteed to have a non-None ``request`` attribute.
* - .. py:function:: websocket_start(flow)
- Called when WebSocket connection is established after a successful
handshake.
*flow*
A ``models.WebSocketFlow`` object.
* - .. py:function:: websocket_message(flow)
- Called when a WebSocket message is received from the client or server. The
sender and receiver are identifiable. The most recent message will be
``flow.messages[-1]``. The message is user-modifiable and is killable.
A message is either of TEXT or BINARY type.
*flow*
A ``models.WebSocketFlow`` object.
* - .. py:function:: websocket_end(flow)
- Called when WebSocket connection ends.
*flow*
A ``models.WebSocketFlow`` object.
* - .. py:function:: websocket_error(flow)
- Called when a WebSocket error occurs - e.g. the connection closing
unexpectedly.
*flow*
A ``models.WebSocketFlow`` object.
TCP Events
----------
These events are called only if the connection is in :ref:`TCP mode
<tcp_proxy>`. So, for instance, TCP events are not called for ordinary HTTP/S
connections.
.. list-table::
:widths: 40 60
:header-rows: 0
* - .. py:function:: tcp_start(flow)
- Called when TCP streaming starts.
*flow*
A ``models.TCPFlow`` object.
* - .. py:function:: tcp_message(flow)
- Called when a TCP payload is received from the client or server. The
sender and receiver are identifiable. The most recent message will be
``flow.messages[-1]``. The message is user-modifiable.
*flow*
A ``models.TCPFlow`` object.
* - .. py:function:: tcp_end(flow)
- Called when TCP streaming ends.
*flow*
A ``models.TCPFlow`` object.
* - .. py:function:: tcp_error(flow)
- Called when a TCP error occurs - e.g. the connection closing
unexpectedly.
*flow*
A ``models.TCPFlow`` object.

148
docs/scripting/overview.rst
View File

@ -1,148 +0,0 @@
.. _overview:
Overview
========
Mitmproxy has a powerful scripting API that allows you to control almost any
aspect of traffic being proxied. In fact, much of mitmproxy's own core
functionality is implemented using the exact same API exposed to scripters (see
:src:`mitmproxy/addons`).
A simple example
----------------
Scripting is event driven, with named handlers on the script object called at
appropriate points of mitmproxy's operation. Here's a complete mitmproxy script
that adds a new header to every HTTP response before it is returned to the
client:
.. literalinclude:: ../../examples/simple/add_header.py
:caption: :src:`examples/simple/add_header.py`
:language: python
All events that deal with an HTTP request get an instance of `HTTPFlow
<api.html#mitmproxy.models.http.HTTPFlow>`_, which we can use to manipulate the
response itself. We can now run this script using mitmdump, and the new header
will be added to all responses passing through the proxy:
>>> mitmdump -s add_header.py
Examples
--------
A collection of addons that demonstrate popular features can be found at :src:`examples/simple`.
Using classes
-------------
In the example above, the script object is the ``add_header`` module itself.
That is, the handlers are declared at the global level of the script. This is
great for quick hacks, but soon becomes limiting as scripts become more
sophisticated.
When a script first starts up, the `start <events.html#start>`_, event is
called before anything else happens. You can replace the current script object
by returning it from this handler. Here's how this looks when applied to the
example above:
.. literalinclude:: ../../examples/simple/add_header_class.py
:caption: :src:`examples/simple/add_header_class.py`
:language: python
So here, we're using a module-level script to "boot up" into a class instance.
From this point on, the module-level script is removed from the handler chain,
and is replaced by the class instance.
Handling arguments
------------------
FIXME
Logging and the context
-----------------------
Scripts should not output straight to stderr or stdout. Instead, the `log
<api.html#mitmproxy.controller.Log>`_ object on the ``ctx`` context module
should be used, so that the mitmproxy host program can handle output
appropriately. So, mitmdump can print colorised script output to the terminal,
and mitmproxy console can place script output in the event buffer.
Here's how this looks:
.. literalinclude:: ../../examples/simple/log_events.py
:caption: :src:`examples/simple/log_events.py`
:language: python
The ``ctx`` module also exposes the mitmproxy master object at ``ctx.master``
for advanced usage.
Running scripts on saved flows
------------------------------
When a flow is loaded from disk, the sequence of events that the flow would
have gone through on the wire is partially replayed. So, for instance, an HTTP
flow loaded from disk will trigger `requestheaders
<events.html#requestheaders>`_, `request <events.html#request>`_,
`responseheaders <events.html#responseheaders>`_ and `response
<events.html#response>`_ in order. We can use this behaviour to transform saved
traffic using scripts. For example, we can invoke the replacer script from
above on saved traffic as follows:
>>> mitmdump -dd -s "./arguments.py html fakehtml" -r saved -w changed
This command starts the ``arguments`` script, reads all the flows from
``saved`` transforming them in the process, then writes them all to
``changed``.
The mitmproxy console tool provides interactive ways to run transforming
scripts on flows - for instance, you can run a one-shot script on a single flow
through the ``|`` (pipe) shortcut.
Concurrency
-----------
The mitmproxy script mechanism is single threaded, and the proxy blocks while
script handlers execute. This hugely simplifies the most common case, where
handlers are light-weight and the blocking doesn't have a performance impact.
It's possible to implement a concurrent mechanism on top of the blocking
framework, and mitmproxy includes a handy example of this that is fit for most
purposes. You can use it as follows:
.. literalinclude:: ../../examples/complex/nonblocking.py
:caption: :src:`examples/complex/nonblocking.py`
:language: python
Testing
-------
Mitmproxy includes a number of helpers for testing addons. The
``mitmproxy.test.taddons`` module contains a context helper that takes care of
setting up and tearing down the addon event context. The
``mitmproxy.test.tflow`` module contains helpers for quickly creating test
flows. Pydoc is the canonical reference for these modules, and mitmproxy's own
test suite is an excellent source of examples of usage. Here, for instance, is
the mitmproxy unit tests for the `anticache` option, demonstrating a good
cross-section of the test helpers:
.. literalinclude:: ../../test/mitmproxy/addons/test_anticache.py
:caption: :src:`test/mitmproxy/addons/test_anticache.py`
:language: python
Developing scripts
------------------
Mitmproxy monitors scripts for modifications, and reloads them on change. When
this happens, the script is shut down (the `done <events.html#done>`_ event is
called), and the new instance is started up as if the script had just been
loaded (the `start <events.html#start>`_ and `configure
<events.html#configure>`_ events are called).

9
docs/scripts/filters.py Executable file
View File

@ -0,0 +1,9 @@
#!/usr/bin/env python
from mitmproxy import flowfilter
print("<table class=\"table filtertable\"><tbody>")
for i in flowfilter.help:
print("<tr><th>%s</th><td>%s</td></tr>" % i)
print("</tbody></table>")

9
docs/src/config.toml Normal file
View File

@ -0,0 +1,9 @@
baseURL = ""
languageCode = "en-us"
title = "mitmproxy.org docs"
theme = "mitmproxydocs"
publishDir = "../public"
RelativeURLs = true
[indexes]
tag = "tags"

15
docs/introduction.rst → docs/src/content/_index.md
View File

@ -1,5 +1,11 @@
Introduction
============
---
title: "Introduction"
menu:
overview:
weight: 1
---
# Introduction
**mitmproxy** is an interactive man-in-the-middle proxy for HTTP and HTTPS
with a console interface.
@ -9,10 +15,10 @@ with a console interface.
**mitmweb** is a web-based interface for mitmproxy.
Documentation, tutorials and distribution packages can be found on the
mitmproxy website: `mitmproxy.org <https://mitmproxy.org/>`_
[mitmproxy website](https://mitmproxy.org).
.. rubric:: Features
## Features
- Intercept HTTP & HTTPS requests and responses and modify them on the fly
- Save complete HTTP conversations for later replay and analysis
@ -23,3 +29,4 @@ mitmproxy website: `mitmproxy.org <https://mitmproxy.org/>`_
- Make scripted changes to HTTP traffic using Python
- SSL/TLS certificates for interception are generated on the fly
- And much, much more...

134
docs/src/content/addons-commands.md Normal file
View File

@ -0,0 +1,134 @@
---
title: "Commands"
menu:
addons:
weight: 4
---
# Commands
Commands allow users to actively interact with addons - querying their state,
commanding them to perform actions, and having them transform data. Like
[options]({{< relref addons-options >}}), commands are typed, and both
invocations and data returned from commands are checked at runtime. Commands are
a very powerful construct - for instance, all user interaction in mitmproxy
console are built by binding commands to keys.
## Simple example
Let's begin with a simple example.
{{< example src="examples/addons/commands-simple.py" lang="py" >}}
To see this example in action, start mitmproxy console with the addon loaded:
{{< highlight bash >}}
> mitmproxy -s ./examples/addons/commands-simple.py
{{< /highlight >}}
Now, make sure the event log is showing, and then execute the command at the
prompt (started by typing ":"):
{{< highlight none>}}
:myaddon.inc
{{< /highlight >}}
Notice that tab completion works - our addon command has complete parity with
builtin commands. There are a few things to note about this example:
- Commands are declared through the `command.command` decorator. Each command
has a unique name - by convention, we use period-separated names, with the
name of the addon as a prefix.
- Annotating commands with types is mandatory, including the return type (in
this case `None`). This allows mitmproxy to support addon commands throughout
its toolset - runtime invocations are type checked, addon commands are
included in the built-in help, the command editor in mitmproxy console can
perform sophisticated completion and error checking, and so forth.
## Working with flows
Since command arguments are typed, we can provide special conveniences for
working with certain important data types. The most useful of these are the
`Flows` classes that represent mitmproxy traffic.
Consider the following addon:
{{< example src="examples/addons/commands-flows.py" lang="py" >}}
The `myaddon.addheader` command is quite simple: it takes a sequence of flows,
and adds a header to every request. The really interesting aspect of this
example is how users specify flows. Because mitmproxy can inspect the type
signature, it can expand a text flow selector into a sequence of flows for us
transparently. This means that the user has the full flexibility of [flow
filters]({{< relref addons-options >}}) available. Let's try it out.
Start by loading the addon into mitmproxy and sending some traffic through so we
have flows to work with:
{{< highlight bash >}}
> mitmproxy -s ./examples/addons/commands-flows.py
{{< /highlight >}}
We can now invoke our toy command in various ways. Let's begin by running it
just on the currently focused flow:
{{< highlight none >}}
:myaddon.addheader @focus
{{< /highlight >}}
We can also invoke it on all flows:
{{< highlight none >}}
:myaddon.addheader @all
{{< /highlight >}}
Or only flows from **google.com**:
{{< highlight none >}}
:myaddon.addheader ~d google.com
{{< /highlight >}}
What's more, we can trivially bind these commands to keyboard shortcuts within
mitmproxy if we plan to use them frequently. Flow selectors combined with
commands are amazingly powerful, and lets us build and expose re-usable functions
for operating on flows.
## Paths
Commands can take an arbitrary number of arguments. Let's build on the previous
example to illustrate this, and also demonstrate another special type: paths.
{{< example src="examples/addons/commands-paths.py" lang="py" >}}
Our command calculates a histogram of the domains in the specified set of flows,
and writes it to a path which is specified as the second argument to the
command. Try invoking it like this:
{{< highlight none >}}
:myaddon.histogram @all /tmp/xxx
{{< /highlight >}}
Notice that mitmproxy provides tab completion both for the flow specification
and the path.
## Supported Types
The following types are supported for options. If you need to use a type not
listed here, please send us a pull request.
- Primitive types: `str`, `int`, `bool`
- Sequences: `typing.Sequence[str]`
- Flows and flow sequences: `flow.Flow` and `typing.Sequence[flow.Flow]`
- Multiple choice strings: `types.Choice`
- Meta-types: `types.Command` and `types.Arg`. These are for constructing
commands that invoke other commands. This is most commonly useful in
keybinding - see the built-in mitmproxy console keybindings for a rich suite
of examples.
- Data types: `types.CutSpec` and `types.Data`. The cuts mechanism is in alpha
at the moment, and provides a convenient way to snip up flow data.
- Path: `types.Path`

24
docs/src/content/addons-events.md Normal file
View File

@ -0,0 +1,24 @@
---
title: "Events"
menu:
addons:
weight: 2
---
# Events
Addons hook into mitmproxy's internal mechanisms through events. These are
implemented on addons as methods with a set of well-known names. Many events
receive `Flow` objects as arguments - by modifying these objects, addons can
change traffic on the fly. For instance, here is an addon that adds a response
header with a count of the number of responses seen:
{{< example src="examples/addons/addheader.py" lang="py" >}}
## Supported Events
Below is an addon class that implements stubs for all events. We've added
annotations to illustrate the argument types for the various events.
{{< example src="examples/addons/events.py" lang="py" >}}

101
docs/src/content/addons-options.md Normal file
View File

@ -0,0 +1,101 @@
---
title: "Options"
menu:
addons:
weight: 3
---
# Options
At the heart of mitmproxy is a global options store, containing the settings
that determine the behaviour of both mitmproxy and its addons. Options can be
read from a configuration file, set on the command-line and changed
interactively by users on the fly.
All options are annotated with one of a set of supported types. Mitmproxy knows
how to serialise and deserialise these types, and has standard ways of
presenting typed values for editing in interactive programs. Attempting to set a
value with the wrong type will result in an error. This means that addon options
get full support throughout mitmproxy's toolchain simply by declaring a type.
## Simple example
{{< example src="examples/addons/options-simple.py" lang="py" >}}
The `load` event receives an instance of `mitmproxy.addonmanager.Loader`, which
allows addons declare options and commands. In this case, the addon adds a
single `addheader` option with type `bool`. Let's try this out by running the
script in mitmproxy console:
{{< highlight bash >}}
> mitmproxy -s ./examples/addons/options-simple.py
{{< /highlight >}}
You can now use CURL to make a request through the proxy like this:
{{< highlight bash >}}
> env http_proxy=http://localhost:8080 curl -I http://google.com
{{< /highlight >}}
If you run this request immediately, you'll notice that no count header is
added. This is because our default value for the option was `false`. Press `O`
to enter the options editor, and find the `addheader` option. You'll notice that
mitmproxy knows this is a boolean, and lets you toggle the value between true
and false. Set the value to `true`, and you should see a result something like
this:
{{< highlight bash >}}
> env http_proxy=http://localhost:8080 curl -I http://google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Length: 219
count: 1
{{< /highlight >}}
When this addon is loaded, the `addheader` setting is available in the
persistent [YAML configuration file]({{< relref "concepts-options" >}}). You can
also over-ride the value directly from the command-line for any of the tools
using the `--set` flag:
{{< highlight bash >}}
mitmproxy -s ./examples/addons/options-simple.py --set addheader=true
{{< /highlight >}}
## Handling configuration updates
Sometimes, simply testing the value of an option from an event is not
sufficient. Instead, we want to act immediately when an option is changed by the
user. This is what the `configure` event is for - when it is triggered, it
receives a set of changed options. An addon can check if an option is in this
set, and then read the value from the options object on the context.
One common use for this function is to check that an option is valid, and give
the user feedback if it's not. If an `exceptions.OptionsError` exception is
raised during configure, all the changes in the update are automatically rolled
back, and an error is displayed to the user. Let's see an example.
{{< example src="examples/addons/options-configure.py" lang="py" >}}
There are a few things to note here. First, the option we add uses
`typing.Optional`. This signals to mitmproxy that `None` is a valid value for
this option - that is, it can be unset. Second, the `configure` method is first
called with our default value (`None`), and then later with an updated value if
the option is changed. If we try to load the script with an incorrect value, we
now see an error:
{{< highlight none >}}
> mitmdump -s ./examples/addons/options-configure.py --set addheader=1000
Loading script: ./examples/addons/options-configure.py
/Users/cortesi/mitmproxy/mitmproxy/venv/bin/mitmdump: addheader must be <= 100
{{< /highlight >}}
## Supported Types
The following types are supported for options.
- Primitive types - `str`, `int`, `float`, `bool`.
- Optional values, annotated using `typing.Optional`.
- Sequences of values, annotated using `typing.Sequence`.

74
docs/src/content/addons-overview.md Normal file
View File

@ -0,0 +1,74 @@
---
title: "Addons"
menu:
addons:
weight: 1
---
# Addons
Mitmproxy's addon mechanism consists of a set of APIs that support components of
any complexity. Addons interact with mitmproxy by responding to **events**,
which allow them to hook into and change mitmproxy's behaviour. They are
configured through **[options]({{< relref concepts-options >}})**, which can be
set in mitmproxy's config file, changed interactively by users, or passed on the
command-line. Finally, they can expose **commands**, which allows users to
invoke their actions either directly or by binding them to keys in the
interactive tools.
Addons are an exceptionally powerful part of mitmproxy. In fact, much of
mitmproxy's own functionality is defined in [a suite of built-in
addons](https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/addons),
implementing everything from functionality like [anticaching]({{< relref
"overview-features#anticache" >}}) and [sticky cookies]({{< relref
"overview-features#sticky-cookies" >}}) to our onboarding webapp. The built-in
addons make for instructive reading, and you will quickly see that quite complex
functionality can often boil down to a very small, completely self-contained
modules. Mitmproxy provides the exact same set of facilities it uses for its own
functionality to third-party scripters and extenders.
This document will show you how to build addons using **events**, **options**
and **commands**. However, this is not an API manual, and the mitmproxy source
code remains the canonical reference. One easy way to explore the API from the
command-line is to use [pydoc](https://docs.python.org/3/library/pydoc.html).
Here, for example, is a command that shows the API documentation for the
mitmproxy's HTTP flow classes:
{{< highlight bash >}}
pydoc mimtproxy.http
{{< /highlight >}}
You will be referring to the mitmproxy API documentation frequently, so keep
**pydoc** or an equivalent handy.
# Anatomy of an addon
{{< example src="examples/addons/anatomy.py" lang="py" >}}
Above is a simple addon that keeps track of the number of flows (or more
specifically HTTP requests) we've seen. Every time it sees a new flow, it uses
mitmproxy's internal logging mechanism to announce its tally. The output can be
found in the event log in the interactive tools, or on the console in mitmdump.
Take it for a spin and make sure that it does what it's supposed to, by loading
it into your mitmproxy tool of choice. We'll use mitmpdump in these examples,
but the flag is identical for all tools:
{{< highlight bash >}}
> mitmdump -s ./anatomy.py
{{< /highlight >}}
Here are a few things to note about the code above:
- Mitmproxy picks up the contents of the `addons` global list and loads what it
finds into the addons mechanism.
- Addons are just objects - in this case our addon is an instance of `Counter`.
- The `request` method is an example of an **event**. Addons simply implement a
method for each event they wan to handle. Each event has a signature
consisting of arguments that are passed to the method. For `request`, this is
an instance of `mitmproxy.http.HTTPFlow`.
- Finally, the `ctx` module is a holdall module that exposes a set of standard
objects that are commonly used in addons. We could pass a `ctx` object as the
first parameter to every event, but we've found it neater to just expose it as
an importable global. In this case, we're using the `ctx.log` object to do our
logging.

17
docs/src/content/addons-scripting.md Normal file
View File

@ -0,0 +1,17 @@
---
title: "Scripting"
menu:
addons:
weight: 5
---
# Scripting
Sometimes, we would like to write a quick script without going through the
trouble of creating a class. The addons mechanism has a shorthand that allows a
module as a whole to be treated as an addon object. This lets us place event
handler functions in the module scope. For instance, here is a complete script
that adds a header to every request.
{{< example src="examples/addons/scripting.py" lang="py" >}}

164
docs/src/content/concepts-certificates.md Normal file
View File

@ -0,0 +1,164 @@
---
title: "Certificates"
menu:
concepts:
weight: 3
---
# About Certificates
Mitmproxy can decrypt encrypted traffic on the fly, as long as the client trusts
its built-in certificate authority. Usually this means that the mitmproxy CA
certificates have to be installed on the client device.
## Quick Setup
By far the easiest way to install the mitmproxy certificates is to use the
built-in certificate installation app. To do this, just start mitmproxy and
configure your target device with the correct proxy settings. Now start a
browser on the device, and visit the magic domain **mitm.it**. You should see
something like this:
{{< figure src="/certinstall-webapp.png" >}}
Click on the relevant icon, follow the setup instructions for the platform
you're on and you are good to go.
## Installing the mitmproxy CA certificate manually
Sometimes using the quick install app is not an option - Java or the iOS
Simulator spring to mind - or you just need to do it manually for some other
reason. Below is a list of pointers to manual certificate installation
documentation for some common platforms. The mitmproxy CA cert is located in
`~/.mitmproxy` after it has been generated at the first start of mitmproxy.
- [IOS](http://jasdev.me/intercepting-ios-traffic) On
iOS 10.3 and onwards, you also need to enable full trust for the mitmproxy
root certificate:
1. Go to Settings > General > About > Certificate Trust Settings.
2. Under "Enable full trust for root certificates", turn on trust for
the mitmproxy certificate.
- [IOS Simulator](https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore)
- [Java](https://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html)
- [Android/Android Simulator](http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets)
- [Windows](https://web.archive.org/web/20160612045445/http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7)
- [Windows (automated)](https://technet.microsoft.com/en-us/library/cc732443.aspx)
{{< highlight bash >}}
certutil.exe -importpfx Root mitmproxy-ca-cert.p12
{{< / highlight >}}
- [Mac OS X](https://support.apple.com/kb/PH7297?locale=en_US)
- [Ubuntu/Debian]( https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861)
- [Mozilla Firefox](https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox)
- [Chrome on Linux](https://stackoverflow.com/a/15076602/198996)
## The mitmproxy certificate authority
The first time **mitmproxy** or **mitmdump** is run, the mitmproxy Certificate
Authority (CA) is created in the config directory (`~/.mitmproxy` by default).
This CA is used for on-the-fly generation of dummy certificates for each of the
SSL sites that your client visits. Since your browser won't trust the mitmproxy
CA out of the box, you will see an SSL certificate warning every time you visit
a new SSL domain through mitmproxy. When you are testing a single site through a
browser, just accepting the bogus SSL cert manually is not too much trouble, but
there are a many circumstances where you will want to configure your testing
system or browser to trust the mitmproxy CA as a signing root authority. For
security reasons, the mitmproxy CA is generated uniquely on the first start and
is not shared between mitmproxy installations on different devices.
### Certificate Pinning
Some applications employ [Certificate
Pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning) to prevent
man-in-the-middle attacks. This means that **mitmproxy** and **mitmdump's**
certificates will not be accepted by these applications without modifying them.
It is recommended to use the passthrough feature in order to prevent
**mitmproxy** and **mitmdump** from intercepting traffic to these specific
domains. If you want to intercept the pinned connections, you need to patch the
application manually. For Android and (jailbroken) iOS devices, various tools
exist to accomplish this.
## CA and cert files
The files created by mitmproxy in the .mitmproxy directory are as follows:
| | |
| --------------------- | ------------------------------------------------------------------------------------ |
| mitmproxy-ca.pem | The certificate **and the private key** in PEM format. |
| mitmproxy-ca-cert.pem | The certificate in PEM format. Use this to distribute on most non-Windows platforms. |
| mitmproxy-ca-cert.p12 | The certificate in PKCS12 format. For use on Windows. |
| mitmproxy-ca-cert.cer | Same file as .pem, but with an extension expected by some Android devices. |
## Using a custom certificate
You can use your own (leaf) certificate by passing the `--cert
[domain=]path_to_certificate` option to mitmproxy. Mitmproxy then uses the
provided certificate for interception of the specified domain instead of
generating a certificate signed by its own CA.
The certificate file is expected to be in the PEM format. You can include
intermediary certificates right below your leaf certificate, so that your PEM
file roughly looks like this:
-----BEGIN PRIVATE KEY-----
<private key>
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
<cert>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<intermediary cert (optional)>
-----END CERTIFICATE-----
For example, you can generate a certificate in this format using these
instructions:
{{< highlight bash >}}
openssl genrsa -out cert.key 2048
# (Specify the mitm domain as Common Name, e.g. \*.google.com)
openssl req -new -x509 -key cert.key -out cert.crt
cat cert.key cert.crt \> cert.pem
{{< / highlight >}}
Now, you can run mitmproxy with the generated certificate:
**For all domain names**
{{< highlight bash >}}
mitmproxy --cert *=cert.pem
{{< / highlight >}}
**For specific domain names**
{{< highlight bash >}}
mitmproxy --cert *.example.com=cert.pem
{{< / highlight >}}
**Note:** `*.example.com` is for all the subdomains. You can also use
`www.example.com` for a particular subdomain.
## Using a custom certificate authority
By default, mitmproxy will use `~/.mitmproxy/mitmproxy-ca.pem` as the
certificate authority to generate certificates for all domains for which
no custom certificate is provided (see above). You can use your own
certificate authority by passing the `--cadir DIRECTORY` option to
mitmproxy. Mitmproxy will then look for `mitmproxy-ca.pem` in the
specified directory. If no such file exists, it will be generated
automatically.
## Using a client side certificate
You can use a client certificate by passing the `--client-certs DIRECTORY|FILE`
option to mitmproxy. Using a directory allows certs to be selected based on
hostname, while using a filename allows a single specific certificate to be used
for all SSL connections. Certificate files must be in the PEM format and should
contain both the unencrypted private key and the certificate.
### Multiple certs by Hostname
You can specify a directory to `--client-certs`, in which case the matching
certificate is looked up by filename. So, if you visit example.org, mitmproxy
looks for a file named `example.org.pem` in the specified directory and uses
this as the client cert.

72
docs/src/content/concepts-commands.md Normal file
View File

@ -0,0 +1,72 @@
---
title: "Commands"
menu:
concepts:
weight: 6
---
# Commands
Commands are the mechanism that allows users to actively interact with addons.
Perhaps the most prominent example of this is the mitmproxy console user
interface - every interaction in this tool consists of commands bound to keys.
Commands also form a flexible and very powerful way to interact with mitmproxy
from the command prompt. In mitmproxy console you can enter the command prompt
with the `:` key. The prompt has intelligent tab completion for command names
and many of the built-in argument types - give it a try.
The canonical reference for commands is the `--commands` flag, which is exposed
by each of the mitmproxy tools. Passing this flag will dump an annotated list of
all registered commands, their arguments and their return values to screen. In
mimtproxy console you can also view a palette of all commands in the command
browser (by default accessible with the `C` key binding).
# Working with flows
Many of mitmproxy's commands take flows as arguments. For instance, the
signature for the client replay commands looks like this:
{{< highlight none >}}
replay.client [flow]
{{< /highlight >}}
That means that it expects a sequence of one or more flows. This is where [flow
specifications]({{< relref concepts-filters >}}) come in - mitmproxy will
intelligently expand a flexible flow selection language to a list of flows when
invoking commands.
Fire up mitmproxy console, and intercept some traffic so we have flows to work
with. Now type the following command:
{{< highlight none >}}
:replay.client @focus
{{< /highlight >}}
Make sure you try using tab completion for the command name and the flow
specification. The `@focus` specifiers expands to the currently focused flow, so
you should see this flow replay. However, replay can take any number of flows.
Try the following command:
{{< highlight none >}}
:replay.client @all
{{< /highlight >}}
Now you should see all flows replay one by one. We have the full power of the
mitmproxy filter language at our disposal here, so we could also, for example,
just replay flows for a specific domain:
{{< highlight none >}}
:replay.client "~d google.com"
{{< /highlight >}}

81
docs/src/content/concepts-filters.md Normal file
View File

@ -0,0 +1,81 @@
---
title: "Filter expressions"
menu:
concepts:
weight: 4
---
# Filter expressions
Many commands in the mitmproxy tool make use of filter expressions. Filter
expressions consist of the following operators:
<table class="table filtertable"><tbody>
<tr><th>~a</th><td>Match asset in response: CSS, Javascript, Flash, images.</td></tr>
<tr><th>~b regex</th><td>Body</td></tr>
<tr><th>~bq regex</th><td>Request body</td></tr>
<tr><th>~bs regex</th><td>Response body</td></tr>
<tr><th>~c int</th><td>HTTP response code</td></tr>
<tr><th>~d regex</th><td>Domain</td></tr>
<tr><th>~dst regex</th><td>Match destination address</td></tr>
<tr><th>~e</th><td>Match error</td></tr>
<tr><th>~h regex</th><td>Header</td></tr>
<tr><th>~hq regex</th><td>Request header</td></tr>
<tr><th>~hs regex</th><td>Response header</td></tr>
<tr><th>~http</th><td>Match HTTP flows</td></tr>
<tr><th>~m regex</th><td>Method</td></tr>
<tr><th>~marked</th><td>Match marked flows</td></tr>
<tr><th>~q</th><td>Match request with no response</td></tr>
<tr><th>~s</th><td>Match response</td></tr>
<tr><th>~src regex</th><td>Match source address</td></tr>
<tr><th>~t regex</th><td>Content-type header</td></tr>
<tr><th>~tcp</th><td>Match TCP flows</td></tr>
<tr><th>~tq regex</th><td>Request Content-Type header</td></tr>
<tr><th>~ts regex</th><td>Response Content-Type header</td></tr>
<tr><th>~u regex</th><td>URL</td></tr>
<tr><th>~websocket</th><td>Match WebSocket flows</td></tr>
<tr><th>!</th><td>unary not</td></tr>
<tr><th>&</th><td>and</td></tr>
<tr><th>|</th><td>or</td></tr>
<tr><th>(...)</th><td>grouping</td></tr>
</tbody></table>
- Regexes are Python-style
- Regexes can be specified as quoted strings
- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value".
- Strings with no operators are matched against the request URL.
- The default binary operator is &.
## View flow selectors
In interactive contexts, mitmproxy has a set of convenient flow selectors that
operate on the current view:
<table class="table filtertable"><tbody>
<tr><th>@all</th><td>All flows</td></tr>
<tr><th>@focus</th><td>The currently focused flow</td></tr>
<tr><th>@shown</th><td>All flows currently shown</td></tr>
<tr><th>@hidden</th><td>All flows currently hidden</td></tr>
<tr><th>@marked</th><td>All marked flows</td></tr>
<tr><th>@unmarked</th><td>All unmarked flows</td></tr>
</tbody></table>
These are frequently used in commands and key bindings.
## Examples
URL containing "google.com":
google\.com
Requests whose body contains the string "test":
~q ~b test
Anything but requests with a text/html content type:
!(~q & ~t "text/html")

252
docs/src/content/concepts-howmitmproxyworks.md Normal file
View File

@ -0,0 +1,252 @@
---
title: "How mitmproxy works"
menu:
concepts:
weight: 1
---
# How mitmproxy works
Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying
process works will help you deploy it creatively, and take into account its
fundamental assumptions and how to work around them. This document explains
mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted
explicit proxying, and working up to the most complicated interaction
-transparent proxying of TLS-protected traffic[^1] in the presence of [Server
Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication).
## Explicit HTTP
Configuring the client to use mitmproxy as an explicit proxy is the
simplest and most reliable way to intercept traffic. The proxy protocol
is codified in the [HTTP RFC](https://tools.ietf.org/html/rfc7230), so
the behaviour of both the client and the server is well defined, and
usually reliable. In the simplest possible interaction with mitmproxy, a
client connects directly to the proxy, and makes a request that looks
like this:
{{< highlight http >}}
GET http://example.com/index.html HTTP/1.1
{{< / highlight >}}
This is a proxy GET request - an extended form of the vanilla HTTP GET
request that includes a schema and host specification, and it includes
all the information mitmproxy needs to proceed.
{{< figure src="/schematics/how-mitmproxy-works-explicit.png" title="Explicit" >}}
1. The client connects to the proxy and makes a request.
2. Mitmproxy connects to the upstream server and simply forwards the request on.
## Explicit HTTPS
The process for an explicitly proxied HTTPS connection is quite
different. The client connects to the proxy and makes a request that
looks like this:
{{< highlight http >}}
CONNECT example.com:443 HTTP/1.1
{{< / highlight >}}
A conventional proxy can neither view nor manipulate a TLS-encrypted
data stream, so a CONNECT request simply asks the proxy to open a pipe
between the client and server. The proxy here is just a facilitator - it
blindly forwards data in both directions without knowing anything about
the contents. The negotiation of the TLS connection happens over this
pipe, and the subsequent flow of requests and responses are completely
opaque to the proxy.
### The MITM in mitmproxy
This is where mitmproxy's fundamental trick comes into play. The MITM
in its name stands for Man-In-The-Middle - a reference to the process we
use to intercept and interfere with these theoretically opaque data
streams. The basic idea is to pretend to be the server to the client,
and pretend to be the client to the server, while we sit in the middle
decoding traffic from both sides. The tricky part is that the
[Certificate
Authority](https://en.wikipedia.org/wiki/Certificate_authority) system
is designed to prevent exactly this attack, by allowing a trusted
third-party to cryptographically sign a server's certificates to verify
that they are legit. If this signature doesn't match or is from a
non-trusted party, a secure client will simply drop the connection and
refuse to proceed. Despite the many shortcomings of the CA system as it
exists today, this is usually fatal to attempts to MITM a TLS connection
for analysis. Our answer to this conundrum is to become a trusted
Certificate Authority ourselves. Mitmproxy includes a full CA
implementation that generates interception certificates on the fly. To
get the client to trust these certificates, we
[register mitmproxy as a trusted CA with the device
manually]({{< relref concepts-certificates >}}).
### Complication 1: What's the remote hostname?
To proceed with this plan, we need to know the domain name to use in the
interception certificate - the client will verify that the certificate is for
the domain it's connecting to, and abort if this is not the case. At first
blush, it seems that the CONNECT request above gives us all we need - in this
example, both of these values are "example.com". But what if the client had
initiated the connection as follows:
{{< highlight http >}}
CONNECT 10.1.1.1:443 HTTP/1.1
{{< / highlight >}}
Using the IP address is perfectly legitimate because it gives us enough
information to initiate the pipe, even though it doesn't reveal the
remote hostname.
Mitmproxy has a cunning mechanism that smooths this over - [upstream certificate
sniffing]({{< relref "overview-features#upstream-certificates" >}}). As soon as
we see the CONNECT request, we pause the client part of the conversation, and
initiate a simultaneous connection to the server. We complete the TLS handshake
with the server, and inspect the certificates it used. Now, we use the Common
Name in the upstream certificates to generate the dummy certificate for the
client. Voila, we have the correct hostname to present to the client, even if it
was never specified.
### Complication 2: Subject Alternative Name
Enter the next complication. Sometimes, the certificate Common Name is
not, in fact, the hostname that the client is connecting to. This is
because of the optional [Subject Alternative
Name](https://en.wikipedia.org/wiki/SubjectAltName) field in the
certificate that allows an arbitrary number of alternative domains to be
specified. If the expected domain matches any of these, the client will
proceed, even though the domain doesn't match the certificate CN. The
answer here is simple: when we extract the CN from the upstream cert, we
also extract the SANs, and add them to the generated dummy certificate.
### Complication 3: Server Name Indication
One of the big limitations of vanilla TLS is that each certificate
requires its own IP address. This means that you couldn't do virtual
hosting where multiple domains with independent certificates share the
same IP address. In a world with a rapidly shrinking IPv4 address pool
this is a problem, and we have a solution in the form of the [Server
Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication)
extension to the TLS protocols. This lets the client specify the remote
server name at the start of the TLS handshake, which then lets the
server select the right certificate to complete the process.
SNI breaks our upstream certificate sniffing process, because when we
connect without using SNI, we get served a default certificate that may
have nothing to do with the certificate expected by the client. The
solution is another tricky complication to the client connection
process. After the client connects, we allow the TLS handshake to
continue until just **after** the SNI value has been passed to us. Now
we can pause the conversation, and initiate an upstream connection using
the correct SNI value, which then serves us the correct upstream
certificate, from which we can extract the expected CN and SANs.
### Putting it all together
Lets put all of this together into the complete explicitly proxied HTTPS
flow.
{{< figure src="/schematics/how-mitmproxy-works-explicit-https.png" title="Explicit HTTPS" >}}
1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT
request.
2. Mitmproxy responds with a `200 Connection Established`, as if it has set up
the CONNECT pipe.
3. The client believes it's talking to the remote server, and initiates the TLS
connection. It uses SNI to indicate the hostname it is connecting to.
4. Mitmproxy connects to the server, and establishes a TLS connection using the
SNI hostname indicated by the client.
5. The server responds with the matching certificate, which contains the CN and
SAN values needed to generate the interception certificate.
6. Mitmproxy generates the interception cert, and continues the client TLS
handshake paused in step 3.
7. The client sends the request over the established TLS connection.
8. Mitmproxy passes the request on to the server over the TLS connection
initiated in step 4.
## Transparent HTTP
When a transparent proxy is used, the connection is redirected into a
proxy at the network layer, without any client configuration being
required. This makes transparent proxying ideal for those situations
where you can't change client behaviour - proxy-oblivious Android
applications being a common example.
To achieve this, we need to introduce two extra components. The first is
a redirection mechanism that transparently reroutes a TCP connection
destined for a server on the Internet to a listening proxy server. This
usually takes the form of a firewall on the same host as the proxy
server - [iptables](http://www.netfilter.org/) on Linux or
[pf](https://en.wikipedia.org/wiki/PF_\(firewall\)) on OSX. Once the
client has initiated the connection, it makes a vanilla HTTP request,
which might look something like this:
{{< highlight http >}}
GET /index.html HTTP/1.1
{{< / highlight >}}
Note that this request differs from the explicit proxy variation, in
that it omits the scheme and hostname. How, then, do we know which
upstream host to forward the request to? The routing mechanism that has
performed the redirection keeps track of the original destination for
us. Each routing mechanism has a different way of exposing this data, so
this introduces the second component required for working transparent
proxying: a host module that knows how to retrieve the original
destination address from the router. In mitmproxy, this takes the form
of a built-in set of
[modules](https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/platform)
that know how to talk to each platform's redirection mechanism. Once we
have this information, the process is fairly straight-forward.
{{< figure src="/schematics/how-mitmproxy-works-transparent.png" title="Transparent" >}}
1. The client makes a connection to the server.
2. The router redirects the connection to mitmproxy, which is typically
listening on a local port of the same host. Mitmproxy then consults
the routing mechanism to establish what the original destination
was.
3. Now, we simply read the client's request...
4. ... and forward it upstream.
## Transparent HTTPS
The first step is to determine whether we should treat an incoming
connection as HTTPS. The mechanism for doing this is simple - we use the
routing mechanism to find out what the original destination port is. All
incoming connections pass through different layers which can determine
the actual protocol to use. Automatic TLS detection works for SSLv3, TLS
1.0, TLS 1.1, and TLS 1.2 by looking for a *ClientHello* message at the
beginning of each connection. This works independently of the used TCP
port.
From here, the process is a merger of the methods we've described for
transparently proxying HTTP, and explicitly proxying HTTPS. We use the
routing mechanism to establish the upstream server address, and then
proceed as for explicit HTTPS connections to establish the CN and SANs,
and cope with SNI.
{{< figure src="/schematics/how-mitmproxy-works-transparent-https.png" title="Transparent HTTPS" >}}
1. The client makes a connection to the server.
2. The router redirects the connection to mitmproxy, which is typically
listening on a local port of the same host. Mitmproxy then consults the
routing mechanism to establish what the original destination was.
3. The client believes it's talking to the remote server, and
initiates the TLS connection. It uses SNI to indicate the hostname
it is connecting to.
4. Mitmproxy connects to the server, and establishes a TLS connection
using the SNI hostname indicated by the client.
5. The server responds with the matching certificate, which contains
the CN and SAN values needed to generate the interception
certificate.
6. Mitmproxy generates the interception cert, and continues the client
TLS handshake paused in step 3.
7. The client sends the request over the established TLS connection.
8. Mitmproxy passes the request on to the server over the TLS
connection initiated in step 4.
### Footnotes
[^1]: The use of "TLS" refers to both SSL (outdated and insecure) and TLS (1.0
and up) in the generic sense, unless otherwise specified.

215
docs/src/content/concepts-modes.md Normal file
View File

@ -0,0 +1,215 @@
---
title: "Modes of operation"
menu:
concepts:
weight: 2
---
# Modes of Operation
- [Regular](#regular-proxy) (the default)
- [Transparent](#transparent-proxy)
- [Reverse Proxy](#reverse-proxy)
- [Upstream Proxy](#upstream-proxy)
- [SOCKS Proxy](#socks-proxy)
Now, which one should you pick? Use this flow chart:
{{< figure src="/schematics/proxy-modes-flowchart.png" >}}
## Regular Proxy
Mitmproxy's regular mode is the simplest and the easiest to set up.
1. Start mitmproxy.
2. Configure your client to use mitmproxy by explicitly setting an HTTP
proxy.
3. Quick Check: You should already be able to visit an unencrypted HTTP
site through the proxy.
4. Open the magic domain **mitm.it** and install the certificate for your
device.
{{< note >}}
Unfortunately, some applications bypass the system HTTP proxy settings -
Android applications are a common example. In these cases, you need to
use mitmproxy's transparent mode.
{{< /note >}}
If you are proxying an external device, your network will probably look
like this:
{{< figure src="/schematics/proxy-modes-regular.png" >}}
The square brackets signify the source and destination IP addresses.
Your client explicitly connects to mitmproxy and mitmproxy explicitly
connects to the target server.
## Transparent Proxy
In transparent mode, traffic is directed into a proxy at the network
layer, without any client configuration required. This makes transparent
proxying ideal for situations where you can't change client behaviour.
In the graphic below, a machine running mitmproxy has been inserted
between the router and the internet:
{{< figure src="/schematics/proxy-modes-transparent-1.png" >}}
The square brackets signify the source and destination IP addresses.
Round brackets mark the next hop on the *Ethernet/data link* layer. This
distinction is important: when the packet arrives at the mitmproxy
machine, it must still be addressed to the target server. This means
that Network Address Translation should not be applied before the
traffic reaches mitmproxy, since this would remove the target
information, leaving mitmproxy unable to determine the real destination.
{{< figure src="/schematics/proxy-modes-transparent-wrong.png" title="Modes Transparent Wrong" >}}
### Common Configurations
There are many ways to configure your network for transparent proxying.
We'll look at two common scenarios:
1. Configuring the client to use a custom gateway/router/"next hop"
2. Implementing custom routing on the router
In most cases, the first option is recommended due to its ease of use.
#### (a) Custom Gateway
One simple way to get traffic to the mitmproxy machine with the
destination IP intact, is to simply configure the client with the
mitmproxy box as the default gateway.
{{< figure src="/schematics/proxy-modes-transparent-2.png" >}}
In this scenario, we would:
1. Configure the proxy machine for transparent mode. You can find instructions
in the [transparent]({{< relref "howto-transparent"
>}}) section.
2. Configure the client to use the proxy machine's IP as the default gateway.
3. Quick Check: At this point, you should already be able to visit an
unencrypted HTTP site over the proxy.
4. Open the magic domain **mitm.it** and install the certificate for your
device.
Setting the custom gateway on clients can be automated by serving the
settings out to clients over DHCP. This lets set up an interception
network where all clients are proxied automatically, which can save time
and effort.
{{% note %}}
### Troubleshooting Transparent Mode
Incorrect transparent mode configurations are a frequent source of
error. If it doesn't work for you, try the following things:
- Open mitmproxy's event log - do you see clientconnect messages? If not, the
packets are not arriving at the proxy. One common cause is the occurrence of
ICMP redirects, which means that your machine is telling the client that
there's a faster way to the internet by contacting your router directly (see
the [transparent]({{< relref "howto-transparent"
>}}) section on how to disable them). If in doubt,
[Wireshark](https://wireshark.org/) may help you to see whether something
arrives at your machine or not.
- Make sure you have not explicitly configured an HTTP proxy on the client. This
is not needed in transparent mode.
- Re-check the instructions in the [transparent]({{< relref "howto-transparent"
>}}) section. Anything you missed?
If you encounter any other pitfalls that should be listed here, please
let us know!
{{% /note %}}
#### (b) Custom Routing
In some cases, you may need more fine-grained control of which traffic
reaches the mitmproxy instance, and which doesn't. You may, for
instance, choose only to divert traffic to some hosts into the
transparent proxy. There are a huge number of ways to accomplish this,
and much will depend on the router or packet filter you're using. In
most cases, the configuration will look like this:
{{< figure src="/schematics/proxy-modes-transparent-3.png" >}}
## Reverse Proxy
mitmproxy is usually used with a client that uses the proxy to access
the Internet. Using reverse proxy mode, you can use mitmproxy to act
like a normal HTTP server:
{{< figure src="/schematics/proxy-modes-reverse.png" >}}
There are various use-cases:
- Say you have an internal API running at <http://example.local/>. You could now
set up mitmproxy in reverse proxy mode at <http://debug.example.local/> and
dynamically point clients to this new API endpoint, which provides them with
the same data and you with debug information. Similarly, you could move your
real server to a different IP/port and set up mitmproxy in the original
place to debug and or redirect all sessions.
- Say you're a web developer working on <http://example.com/> (with a
development version running on <http://localhost:8000/>). You can modify
your hosts file so that example.com points to 127.0.0.1 and then run
mitmproxy in reverse proxy mode on port 80. You can test your app on the
example.com domain and get all requests recorded in mitmproxy.
- Say you have some toy project that should get SSL support. Simply set up
mitmproxy as a reverse proxy on port 443 and you're done (`mitmdump -p 443
-R http://localhost:80/`). Mitmproxy auto-detects TLS traffic and intercepts
it dynamically. There are better tools for this specific task, but mitmproxy
is very quick and simple way to set up an SSL-speaking server.
- Want to add a non-SSL-capable compression proxy in front of your server? You
could even spawn a mitmproxy instance that terminates SSL (`-R http://...`),
point it to the compression proxy and let the compression proxy point to a
SSL-initiating mitmproxy (`-R https://...`), which then points to the real
server. As you see, it's a fairly flexible thing.
### Host Header
In reverse proxy mode, mitmproxy automatically rewrites the Host header to match
the upstream server. This allows mitmproxy to easily connect to existing
endpoints on the open web (e.g. `mitmproxy -R https://example.com`). You can
disable this behaviour with the `keep_host_header` option.
However, keep in mind that absolute URLs within the returned document or HTTP
redirects will NOT be rewritten by mitmproxy. This means that if you click on a
link for "<http://example.com>" in the returned web page, you will be taken
directly to that URL, bypassing mitmproxy.
One possible way to address this is to modify the hosts file of your OS so that
"example.com" resolves to your proxy's IP, and then access the proxy by going
directly to example.com. Make sure that your proxy can still resolve the
original IP, or specify an IP in mitmproxy.
{{% note %}}
### Caveat: Interactive Use
Reverse Proxy mode is usually not sufficient to create a copy of an
interactive website at different URL. The HTML served to the client
remains unchanged - as soon as the user clicks on an non-relative URL
(or downloads a non-relative image resource), traffic no longer passes
through mitmproxy.
{{% /note %}}
## Upstream Proxy
If you want to chain proxies by adding mitmproxy in front of a different
proxy appliance, you can use mitmproxy's upstream mode. In upstream
mode, all requests are unconditionally transferred to an upstream proxy
of your choice.
{{< figure src="/schematics/proxy-modes-upstream.png" >}}
mitmproxy supports both explicit HTTP and explicit HTTPS in upstream
proxy mode. You could in theory chain multiple mitmproxy instances in a
row, but that doesn't make any sense in practice (i.e. outside of our
tests).
## SOCKS Proxy

66
docs/src/content/concepts-options.md Normal file
View File

@ -0,0 +1,66 @@
---
title: "Options"
menu:
concepts:
weight: 5
---
# Options
The mitmproxy tools share a common [YAML](http://yaml.org/) configuration file
located at `~/.mitmproxy/config.yaml`. This file controls **options** - typed
values that determine the behaviour of mitmproxy. The options mechanism is very
comprehensive - in fact, options control all of mitmproxy's runtime behaviour.
Most command-line flags are simply aliases for underlying options, and
interactive settings changes made in **mitmproxy** and **mitmweb** just change
values in our runtime options store. This means that almost any facet of
mitmproxy's behaviour can be controlled through options.
The canonical reference for options is the `--options` flag, which is exposed by
each of the mitmproxy tools. Passing this flag will dump an annotated YAML
configuration to console, which includes all options and their default values.
The options mechanism is extensible - third-party addons can define options that
are treated exactly like mitmproxy's own. This means that addons can also be
configured through the central configuration file, and their options will appear
in the options editors in interactive tools.
## Tools
Both **mitmproxy** and **mitmweb** have built-in editors that let you view and
manipulate the complete configuration state of mitmproxy. Values you change
interactively have immediate effect in the running instance, and can be made
persistent by saving the settings out to a YAML configuration file (please see
the specific tool's interactive help for details on how to do this).
For all tools, options can be set directly by name using the `--set` command-line
option. Please see the command-line help (`--help`) for usage.
## Example
Here is an excerpt showing the first few lines of the complete `--options`
output:
{{< highlight yaml >}}
# Add all certificates of the upstream server to the certificate chain
# that will be served to the proxy client, as extras. Type bool.
add_upstream_certs_to_client_chain: false
# Allow remote clients to connect to proxy. If set to false, client will
# not be able to connect to proxy unless it is on the same network or
# the proxyauth option is set Type bool.
allow_remote: false
# Strip out request headers that might cause the server to return
# 304-not-modified. Type bool.
anticache: false
# Try to convince servers to send us un-compressed data. Type bool.
anticomp: false
{{< /highlight >}}

102
docs/src/content/howto-ignoredomains.md Normal file
View File

@ -0,0 +1,102 @@
---
title: "Ignoring Domains"
menu:
howto:
weight: 2
---
# Ignoring Domains
There are two main reasons why you may want to exempt some traffic from
mitmproxy's interception mechanism:
- **Certificate pinning:** Some traffic is is protected using [Certificate
Pinning](https://security.stackexchange.com/questions/29988/what-is-certificate-pinning)
and mitmproxy's interception leads to errors. For example, the Twitter app,
Windows Update or the Apple App Store fail to work if mitmproxy is active.
- **Convenience:** You really don't care about some parts of the traffic and
just want them to go away. Note that mitmproxy's "Limit" option is often the
better alternative here, as it is not affected by the limitations listed
below.
If you want to peek into (SSL-protected) non-HTTP connections, check out the
**tcp_proxy** feature. If you want to ignore traffic from mitmproxy's processing
because of large response bodies, take a look at the [streaming]({{< relref "overview-features#streaming" >}}) feature.
## ignore_hosts
The `ignore_hosts` option allows you to specify a regex which is matched against
a `host:port` string (e.g. "example.com:443") of a connection. Matching hosts
are excluded from interception, and passed on unmodified.
| | |
| ------------------ | ------------------------------------------------------------------ |
| command-line alias | `--ignore-hosts regex` |
| mitmproxy option | `ignore_hosts` |
## Limitations
There are two important quirks to consider:
- **In transparent mode, the ignore pattern is matched against the IP and
ClientHello SNI host.** While we usually infer the hostname from the Host
header if the `ignore_hosts` option is set, we do not have access to this
information before the SSL handshake. If the client uses SNI however, then we
treat the SNI host as an ignore target.
- **In regular and upstream proxy mode, explicit HTTP requests are never
ignored.**\[1\] The ignore pattern is applied on CONNECT requests, which
initiate HTTPS or clear-text WebSocket connections.
## Tutorial
If you just want to ignore one specific domain, there's usually a bulletproof
method to do so:
1. Run mitmproxy or mitmdump in verbose mode (`-v`) and observe the `host:port`
information in the serverconnect messages. mitmproxy will filter on these.
2. Take the `host:port` string, surround it with ^ and $, escape all dots (.
becomes \\.) and use this as your ignore pattern:
{{< highlight none >}}
>>> mitmdump -v
127.0.0.1:50588: clientconnect
127.0.0.1:50588: request
-> CONNECT example.com:443 HTTP/1.1
127.0.0.1:50588: Set new server address: example.com:443
127.0.0.1:50588: serverconnect
-> example.com:443
^C
>>> mitmproxy --ignore-hosts ^example\.com:443$
{{< /highlight >}}
Here are some other examples for ignore
patterns:
{{< highlight none >}}
# Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
--ignore-hosts apple.com:443
# "Correct" version without false-positives:
--ignore-hosts '^(.+\.)?apple\.com:443$'
# Ignore example.com, but not its subdomains:
--ignore-hosts '^example.com:'
# Ignore everything but example.com and mitmproxy.org:
--ignore-hosts '^(?!example\.com)(?!mitmproxy\.org)'
# Transparent mode:
--ignore-hosts 17\.178\.96\.59:443
# IP address range:
--ignore-hosts 17\.178\.\d+\.\d+:443
{{< / highlight >}}
**Footnotes**
1. This stems from an limitation of explicit HTTP proxying: A single connection
can be re-used for multiple target domains - a `GET http://example.com/`
request may be followed by a `GET http://evil.com/` request on the same
connection. If we start to ignore the connection after the first request, we
would miss the relevant second one.

114
docs/src/content/howto-transparent-vms.md Normal file
View File

@ -0,0 +1,114 @@
---
title: "Transparently Proxying VMs"
menu:
howto:
weight: 3
---
# Transparently proxify virtual machines
This walkthrough illustrates how to set up transparent proxying with
mitmproxy. We use VirtualBox VMs with an Ubuntu proxy machine in this
example, but the general *Internet \<--\> Proxy VM \<--\> (Virtual)
Internal Network* setup can be applied to other setups.
## 1. Configure Proxy VM
On the proxy machine, **eth0** is connected to the internet. **eth1** is
connected to the internal network that will be proxified and configured
to use a static ip (192.168.3.1).
### VirtualBox configuration
{{< figure src="/transparent-vms/step1_vbox_eth0.png" >}}
{{< figure src="/transparent-vms/step1_vbox_eth1.png" >}}
### VM Network Configuration
{{< figure src="/transparent-vms/step1_proxy.png" >}}
## 2. Configure DHCP and DNS
We use dnsmasq to provide DHCP and DNS in our internal network. Dnsmasq is a
lightweight server designed to provide DNS (and optionally DHCP and TFTP)
services to a small-scale network. Before we get to that, we need to fix some
Ubuntu quirks: **Ubuntu \>12.04** runs an internal dnsmasq instance (listening
on loopback only) by default
[\[1\]](https://www.stgraber.org/2012/02/24/dns-in-ubuntu-12-04/). For our use
case, this needs to be disabled by changing `dns=dnsmasq` to `#dns=dnsmasq` in
**/etc/NetworkManager/NetworkManager.conf** and if on Ubuntu 16.04 or newer
running:
{{< highlight bash >}}
sudo systemctl restart NetworkManager
{{< / highlight >}}
If on Ubuntu 12.04 or 14.04 running:
{{< highlight bash >}}
sudo restart network-manager
{{< / highlight >}}
afterwards.
Now, dnsmasq can be be installed and configured:
{{< highlight bash >}}
sudo apt-get install dnsmasq
{{< / highlight >}}
Replace **/etc/dnsmasq.conf** with the following configuration:
{{< highlight none >}}
# Listen for DNS requests on the internal network
interface=eth1
# Act as a DHCP server, assign IP addresses to clients
dhcp-range=192.168.3.10,192.168.3.100,96h
# Broadcast gateway and dns server information
dhcp-option=option:router,192.168.3.1
dhcp-option=option:dns-server,192.168.3.1
{{< / highlight >}}
Apply changes:
If on Ubuntu 16.04 or newer:
{{< highlight bash >}}
sudo systemctl restart dnsmasq
{{< / highlight >}}
If on Ubuntu 12.04 or 14.04:
{{< highlight bash >}}
sudo service dnsmasq restart
{{< / highlight >}}
Your **proxied machine** in the internal virtual network should now receive an
IP address via DHCP:
{{< figure src="/transparent-vms/step2_proxied_vm.png" >}}
## 3. Redirect traffic to mitmproxy
To redirect traffic to mitmproxy, we need to add two iptables
rules:
{{< highlight bash >}}
sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 8080
sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-port 8080
{{< / highlight >}}
## 4. Run mitmproxy
Finally, we can run mitmproxy in transparent mode with
{{< highlight bash >}}
mitmproxy -T
{{< / highlight >}}
The proxied machine cannot to leak any data outside of HTTP or DNS requests. If
required, you can now [install the mitmproxy certificates on the proxied
machine]({{< relref "concepts-certificates" >}}).

Some files were not shown because too many files have changed in this diff Show More