organize examples

This commit is largely based on work by Thiago Arrais (@thiagoarrais)
and Shane Bradfield (@l33tLumberjack). I wasn't really able to get their
PR reasonably merged onto the latest master, so I reapplied their changes
manually here and did some further improvements on that.
This commit is contained in:
Maximilian Hils 2016-11-21 02:16:20 +01:00
parent f74e561524
commit 9af8f4bb31
43 changed files with 196 additions and 268 deletions

View File

@ -40,8 +40,8 @@ You can also use a script to customize exactly which responses are streamed.
Responses that should be tagged for streaming by setting their ``.stream``
attribute to ``True``:
.. literalinclude:: ../../examples/stream.py
:caption: examples/stream.py
.. literalinclude:: ../../examples/complex/stream.py
:caption: examples/complex/stream.py
:language: python
Implementation Details
@ -59,8 +59,8 @@ Modifying streamed data
If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all
chunks.
.. literalinclude:: ../../examples/stream_modify.py
:caption: examples/stream_modify.py
.. literalinclude:: ../../examples/complex/stream_modify.py
:caption: examples/complex/stream_modify.py
:language: python
.. seealso::

View File

@ -17,8 +17,8 @@ appropriate points of mitmproxy's operation. Here's a complete mitmproxy script
that adds a new header to every HTTP response before it is returned to the
client:
.. literalinclude:: ../../examples/add_header.py
:caption: :src:`examples/add_header.py`
.. literalinclude:: ../../examples/simple/add_header.py
:caption: :src:`examples/simple/add_header.py`
:language: python
All events that deal with an HTTP request get an instance of `HTTPFlow
@ -42,8 +42,8 @@ called before anything else happens. You can replace the current script object
by returning it from this handler. Here's how this looks when applied to the
example above:
.. literalinclude:: ../../examples/classes.py
:caption: :src:`examples/classes.py`
.. literalinclude:: ../../examples/simple/add_header_class.py
:caption: :src:`examples/simple/add_header_class.py`
:language: python
So here, we're using a module-level script to "boot up" into a class instance.
@ -62,13 +62,13 @@ sophisticated - replace one value with another in all responses. Mitmproxy's
<api.html#mitmproxy.models.http.HTTPResponse.replace>`_ method that takes care
of all the details for us.
.. literalinclude:: ../../examples/arguments.py
:caption: :src:`examples/arguments.py`
.. literalinclude:: ../../examples/simple/script_arguments.py
:caption: :src:`examples/simple/script_arguments.py`
:language: python
We can now call this script on the command-line like this:
>>> mitmdump -dd -s "./arguments.py html faketml"
>>> mitmdump -dd -s "./script_arguments.py html faketml"
Whenever a handler is called, mitpmroxy rewrites the script environment so that
it sees its own arguments as if it was invoked from the command-line.
@ -85,8 +85,8 @@ and mitmproxy console can place script output in the event buffer.
Here's how this looks:
.. literalinclude:: ../../examples/context_logging.py
:caption: :src:`examples/context_logging.py`
.. literalinclude:: ../../examples/simple/logging.py
:caption: :src:`examples/simple/logging.py`
:language: python
The ``ctx`` module also exposes the mitmproxy master object at ``ctx.master``
@ -126,8 +126,8 @@ It's possible to implement a concurrent mechanism on top of the blocking
framework, and mitmproxy includes a handy example of this that is fit for most
purposes. You can use it as follows:
.. literalinclude:: ../../examples/nonblocking.py
:caption: :src:`examples/nonblocking.py`
.. literalinclude:: ../../examples/complex/nonblocking.py
:caption: :src:`examples/complex/nonblocking.py`
:language: python

View File

@ -1,31 +0,0 @@
Some inline scripts may require additional dependencies, which can be installed using
`pip install mitmproxy[examples]`.
# inline script examples
add_header.py Simple script that just adds a header to every request.
change_upstream_proxy.py Dynamically change the upstream proxy
dns_spoofing.py Use mitmproxy in a DNS spoofing scenario.
dup_and_replay.py Duplicates each request, changes it, and then replays the modified request.
fail_with_500.py Turn every response into an Internal Server Error.
filt.py Use mitmproxy's filter expressions in your script.
flowwriter.py Only write selected flows into a mitmproxy dumpfile.
iframe_injector.py Inject configurable iframe into pages.
modify_form.py Modify all form submissions to add a parameter.
modify_querystring.py Modify all query strings to add a parameters.
modify_response_body.py Replace arbitrary strings in all responses
nonblocking.py Demonstrate parallel processing with a blocking script.
proxapp.py How to embed a WSGI app in a mitmproxy server
redirect_requests.py Redirect requests or directly reply to them.
stub.py Script stub with a method definition for every event.
upsidedownternet.py Rewrites traffic to turn images upside down.
# mitmproxy examples
flowbasic Basic use of mitmproxy as a library.
stickycookies An example of writing a custom proxy with mitmproxy.
# misc
read_dumpfile Read a dumpfile generated by mitmproxy.
mitmproxywrapper.py Bracket mitmproxy run with proxy enable/disable on OS X

15
examples/README.md Normal file
View File

@ -0,0 +1,15 @@
# Mitmproxy Scripting API
Mitmproxy has a powerful scripting API that allows you to control almost any aspect of traffic being
proxied. In fact, much of mitmproxys own core functionality is implemented using the exact same API
exposed to scripters (see [mitmproxy/addons](../mitmproxy/addons)).
This directory contains some examples of the scripting API. We recommend to start with the
ones in [simple/](./simple).
| :warning: | If you are browsing this on GitHub, make sure to select the git tag matching your mitmproxy version. |
|------------|------------------------------------------------------------------------------------------------------|
Some inline scripts may require additional dependencies, which can be installed using
`pip install mitmproxy[examples]`.

View File

@ -0,0 +1,19 @@
## Complex Examples
| Filename | Description |
|:-------------------------|:----------------------------------------------------------------------------------------------|
| change_upstream_proxy.py | Dynamically change the upstream proxy. |
| dns_spoofing.py | Use mitmproxy in a DNS spoofing scenario. |
| dup_and_replay.py | Duplicates each request, changes it, and then replays the modified request. |
| flowbasic.py | Basic use of mitmproxy's FlowMaster directly. |
| full_transparency_shim.c | Setuid wrapper that can be used to run mitmproxy in full transparency mode, as a normal user. |
| har_dump.py | Dump flows as HAR files. |
| mitmproxywrapper.py | Bracket mitmproxy run with proxy enable/disable on OS X |
| nonblocking.py | Demonstrate parallel processing with a blocking script |
| remote_debug.py | This script enables remote debugging of the mitmproxy _UI_ with PyCharm. |
| sslstrip.py | sslstrip-like funtionality implemented with mitmproxy |
| stickycookies | An advanced example of using mitmproxy's FlowMaster directly. |
| stream | Enable streaming for all responses. |
| stream_modify.py | Modify a streamed response body. |
| tcp_message.py | Modify a raw TCP connection |
| tls_passthrough.py | Use conditional TLS interception based on a user-defined strategy. |

1
examples/flowbasic → examples/complex/flowbasic.py Executable file → Normal file
View File

@ -35,6 +35,7 @@ class MyMaster(master.Master):
def log(self, l):
print("log", l.msg)
opts = options.Options(cadir="~/.mitmproxy/")
config = ProxyConfig(opts)
server = ProxyServer(config)

View File

@ -1,3 +1,7 @@
"""
This script implements an sslstrip-like attack based on mitmproxy.
https://moxie.org/software/sslstrip/
"""
import re
import urllib
@ -28,7 +32,7 @@ def response(flow):
flow.response.headers.pop('Public-Key-Pins', None)
# strip links in response body
flow.response.content = flow.response.content.replace('https://', 'http://')
flow.response.content = flow.response.content.replace(b'https://', b'http://')
# strip meta tag upgrade-insecure-requests in response body
csp_meta_tag_pattern = b'<meta.*http-equiv=["\']Content-Security-Policy[\'"].*upgrade-insecure-requests.*?>'

View File

@ -6,7 +6,7 @@ implement functionality similar to the "sticky cookies" option.
Heads Up: In the majority of cases, you want to use inline scripts.
"""
import os
from mitmproxy import controller, proxy
from mitmproxy import controller, proxy, master
from mitmproxy.proxy.server import ProxyServer

View File

@ -1,5 +1,6 @@
def responseheaders(flow):
"""
Enables streaming for all responses.
This is equivalent to passing `--stream 0` to mitmproxy.
"""
flow.response.stream = True

View File

@ -1,6 +0,0 @@
from mitmproxy import ctx
def start():
ctx.log.info("This is some informative text.")
ctx.log.error("This is an error.")

View File

@ -1,70 +0,0 @@
import string
import lxml.html
import lxml.etree
from mitmproxy import contentviews
from mitmproxy.utils import strutils
class ViewPigLatin(contentviews.View):
name = "pig_latin_HTML"
prompt = ("pig latin HTML", "l")
content_types = ["text/html"]
def __call__(self, data, **metadata):
if strutils.is_xml(data):
parser = lxml.etree.HTMLParser(
strip_cdata=True,
remove_blank_text=True
)
d = lxml.html.fromstring(data, parser=parser)
docinfo = d.getroottree().docinfo
def piglify(src):
words = src.split()
ret = ''
for word in words:
idx = -1
while word[idx] in string.punctuation and (idx * -1) != len(word):
idx -= 1
if word[0].lower() in 'aeiou':
if idx == -1:
ret += word[0:] + "hay"
else:
ret += word[0:len(word) + idx + 1] + "hay" + word[idx + 1:]
else:
if idx == -1:
ret += word[1:] + word[0] + "ay"
else:
ret += word[1:len(word) + idx + 1] + word[0] + "ay" + word[idx + 1:]
ret += ' '
return ret.strip()
def recurse(root):
if hasattr(root, 'text') and root.text:
root.text = piglify(root.text)
if hasattr(root, 'tail') and root.tail:
root.tail = piglify(root.tail)
if len(root):
for child in root:
recurse(child)
recurse(d)
s = lxml.etree.tostring(
d,
pretty_print=True,
doctype=docinfo.doctype
)
return "HTML", contentviews.format_text(s)
pig_view = ViewPigLatin()
def start():
contentviews.add(pig_view)
def done():
contentviews.remove(pig_view)

View File

@ -1,3 +0,0 @@
def response(flow):
flow.response.status_code = 500
flow.response.content = b""

View File

@ -1,18 +0,0 @@
"""
This example shows two ways to redirect flows to other destinations.
"""
from mitmproxy import http
def request(flow):
# pretty_host takes the "Host" header of the request into account,
# which is useful in transparent mode where we usually only have the IP
# otherwise.
# Method 1: Answer with a locally generated response
if flow.request.pretty_host.endswith("example.com"):
flow.response = http.HTTPResponse.make(200, b"Hello World", {"Content-Type": "text/html"})
# Method 2: Redirect the request to a different server
if flow.request.pretty_host.endswith("example.org"):
flow.request.host = "mitmproxy.org"

18
examples/simple/README.md Normal file
View File

@ -0,0 +1,18 @@
## Simple Examples
| Filename | Description |
|:-----------------------------|:---------------------------------------------------------------------------|
| add_header.py | Simple script that just adds a header to every request. |
| custom_contentview.py | Add a custom content view to the mitmproxy UI. |
| filter_flows.py | This script demonstrates how to use mitmproxy's filter pattern in scripts. |
| io_read_dumpfile.py | Read a dumpfile generated by mitmproxy. |
| io_write_dumpfile.py | Only write selected flows into a mitmproxy dumpfile. |
| logging.py | Use mitmproxy's logging API. |
| modify_body_inject_iframe.py | Inject configurable iframe into pages. |
| modify_form.py | Modify HTTP form submissions. |
| modify_querystring.py | Modify HTTP query strings. |
| redirect_requests.py | Redirect a request to a different server. |
| script_arguments.py | Add arguments to a script. |
| send_reply_from_proxy.py | Send a HTTP response directly from the proxy. |
| upsidedownternet.py | Turn all images upside down. |
| wsgi_flask_app.py | Embed a WSGI app into mitmproxy. |

View File

@ -0,0 +1,28 @@
"""
This example shows how one can add a custom contentview to mitmproxy.
The content view API is explained in the mitmproxy.contentviews module.
"""
from mitmproxy import contentviews
class ViewSwapCase(contentviews.View):
name = "swapcase"
# We don't have a good solution for the keyboard shortcut yet -
# you manually need to find a free letter. Contributions welcome :)
prompt = ("swap case text", "p")
content_types = ["text/plain"]
def __call__(self, data: bytes, **metadata):
return "case-swapped text", contentviews.format_text(data.swapcase())
view = ViewSwapCase()
def start():
contentviews.add(view)
def done():
contentviews.remove(view)

View File

@ -1,6 +1,8 @@
# This scripts demonstrates how to use mitmproxy's filter pattern in scripts.
# Usage: mitmdump -s "flowfilter.py FILTER"
"""
This scripts demonstrates how to use mitmproxy's filter pattern in scripts.
Usage:
mitmdump -s "flowfilter.py FILTER"
"""
import sys
from mitmproxy import flowfilter

View File

@ -3,7 +3,7 @@
# Simple script showing how to read a mitmproxy dump file
#
from mitmproxy import flow
from mitmproxy import io
from mitmproxy.exceptions import FlowReadException
import pprint
import sys

View File

@ -1,3 +1,10 @@
"""
This script how to generate a mitmproxy dump file,
as it would also be generated by passing `-w` to mitmproxy.
In contrast to `-w`, this gives you full control over which
flows should be saved and also allows you to rotate files or log
to multiple files in parallel.
"""
import random
import sys
from mitmproxy import io

View File

@ -0,0 +1,12 @@
"""
It is recommended to use `ctx.log` for logging within a script.
This goes to the event log in mitmproxy and to stdout in mitmdump.
If you want to help us out: https://github.com/mitmproxy/mitmproxy/issues/1530 :-)
"""
from mitmproxy import ctx
def start():
ctx.log.info("This is some informative text.")
ctx.log.error("This is an error.")

View File

@ -1,7 +1,9 @@
def request(flow):
if flow.request.urlencoded_form:
# If there's already a form, one can just add items to the dict:
flow.request.urlencoded_form["mitmproxy"] = "rocks"
else:
# One can also just pass new form data.
# This sets the proper content type and overrides the body.
flow.request.urlencoded_form = [
("foo", "bar")

View File

@ -0,0 +1,11 @@
"""
This example shows two ways to redirect flows to another server.
"""
def request(flow):
# pretty_host takes the "Host" header of the request into account,
# which is useful in transparent mode where we usually only have the IP
# otherwise.
if flow.request.pretty_host == "example.org":
flow.request.host = "mitmproxy.org"

View File

@ -0,0 +1,17 @@
"""
This example shows how to send a reply from the proxy immediately
without sending any data to the remote server.
"""
from mitmproxy import http
def request(flow):
# pretty_url takes the "Host" header of the request into account, which
# is useful in transparent mode where we usually only have the IP otherwise.
if flow.request.pretty_url == "http://example.com/path":
flow.response = http.HTTPResponse.make(
200, # (optional) status code
b"Hello World", # (optional) content
{"Content-Type": "text/html"} # (optional) headers
)

View File

@ -0,0 +1,16 @@
"""
This script rotates all images passing through the proxy by 180 degrees.
"""
import io
from PIL import Image
def response(flow):
if flow.response.headers.get("content-type", "").startswith("image"):
s = io.BytesIO(flow.response.content)
img = Image.open(s).rotate(180)
s2 = io.BytesIO()
img.save(s2, "png")
flow.response.content = s2.getvalue()
flow.response.headers["content-type"] = "image/png"

View File

@ -1,87 +0,0 @@
import mitmproxy
"""
This is a script stub, with definitions for all events.
"""
def start():
"""
Called once on script startup before any other events
"""
mitmproxy.ctx.log("start")
def configure(options, updated):
"""
Called once on script startup before any other events, and whenever options changes.
"""
mitmproxy.ctx.log("configure")
def clientconnect(root_layer):
"""
Called when a client initiates a connection to the proxy. Note that a
connection can correspond to multiple HTTP requests
"""
mitmproxy.ctx.log("clientconnect")
def request(flow):
"""
Called when a client request has been received.
"""
mitmproxy.ctx.log("request")
def serverconnect(server_conn):
"""
Called when the proxy initiates a connection to the target server. Note that a
connection can correspond to multiple HTTP requests
"""
mitmproxy.ctx.log("serverconnect")
def responseheaders(flow):
"""
Called when the response headers for a server response have been received,
but the response body has not been processed yet. Can be used to tell mitmproxy
to stream the response.
"""
mitmproxy.ctx.log("responseheaders")
def response(flow):
"""
Called when a server response has been received.
"""
mitmproxy.ctx.log("response")
def error(flow):
"""
Called when a flow error has occured, e.g. invalid server responses, or
interrupted connections. This is distinct from a valid server HTTP error
response, which is simply a response with an HTTP error code.
"""
mitmproxy.ctx.log("error")
def serverdisconnect(server_conn):
"""
Called when the proxy closes the connection to the target server.
"""
mitmproxy.ctx.log("serverdisconnect")
def clientdisconnect(root_layer):
"""
Called when a client disconnects from the proxy.
"""
mitmproxy.ctx.log("clientdisconnect")
def done():
"""
Called once on script shutdown, after any other events.
"""
mitmproxy.ctx.log("done")

View File

@ -1,15 +0,0 @@
import io
from PIL import Image
def response(flow):
if flow.response.headers.get("content-type", "").startswith("image"):
try:
s = io.StringIO(flow.response.content)
img = Image.open(s).rotate(180)
s2 = io.StringIO()
img.save(s2, "png")
flow.response.content = s2.getvalue()
flow.response.headers["content-type"] = "image/png"
except: # Unknown image types etc.
pass

View File

@ -40,30 +40,29 @@ def tscript(cmd, args=""):
class TestScripts(mastertest.MasterTest):
def test_add_header(self):
m, _ = tscript("add_header.py")
m, _ = tscript("simple/add_header.py")
f = tflow.tflow(resp=tutils.tresp())
m.response(f)
assert f.response.headers["newheader"] == "foo"
def test_custom_contentviews(self):
m, sc = tscript("custom_contentviews.py")
pig = contentviews.get("pig_latin_HTML")
_, fmt = pig(b"<html>test!</html>")
assert any(b'esttay!' in val[0][1] for val in fmt)
assert not pig(b"gobbledygook")
m, sc = tscript("simple/custom_contentview.py")
swapcase = contentviews.get("swapcase")
_, fmt = swapcase(b"<html>Test!</html>")
assert any(b'tEST!' in val[0][1] for val in fmt)
def test_iframe_injector(self):
with tutils.raises(ScriptError):
tscript("iframe_injector.py")
tscript("simple/modify_body_inject_iframe.py")
m, sc = tscript("iframe_injector.py", "http://example.org/evil_iframe")
m, sc = tscript("simple/modify_body_inject_iframe.py", "http://example.org/evil_iframe")
f = tflow.tflow(resp=tutils.tresp(content=b"<html>mitmproxy</html>"))
m.response(f)
content = f.response.content
assert b'iframe' in content and b'evil_iframe' in content
def test_modify_form(self):
m, sc = tscript("modify_form.py")
m, sc = tscript("simple/modify_form.py")
form_header = Headers(content_type="application/x-www-form-urlencoded")
f = tflow.tflow(req=tutils.treq(headers=form_header))
@ -76,7 +75,7 @@ class TestScripts(mastertest.MasterTest):
assert list(f.request.urlencoded_form.items()) == [(b"foo", b"bar")]
def test_modify_querystring(self):
m, sc = tscript("modify_querystring.py")
m, sc = tscript("simple/modify_querystring.py")
f = tflow.tflow(req=tutils.treq(path="/search?q=term"))
m.request(f)
@ -87,17 +86,23 @@ class TestScripts(mastertest.MasterTest):
assert f.request.query["mitmproxy"] == "rocks"
def test_arguments(self):
m, sc = tscript("arguments.py", "mitmproxy rocks")
m, sc = tscript("simple/script_arguments.py", "mitmproxy rocks")
f = tflow.tflow(resp=tutils.tresp(content=b"I <3 mitmproxy"))
m.response(f)
assert f.response.content == b"I <3 rocks"
def test_redirect_requests(self):
m, sc = tscript("redirect_requests.py")
m, sc = tscript("simple/redirect_requests.py")
f = tflow.tflow(req=tutils.treq(host="example.org"))
m.request(f)
assert f.request.host == "mitmproxy.org"
def test_send_reply_from_proxy(self):
m, sc = tscript("simple/send_reply_from_proxy.py")
f = tflow.tflow(req=tutils.treq(host="example.com", port=80))
m.request(f)
assert f.response.content == b"Hello World"
class TestHARDump:
@ -115,13 +120,13 @@ class TestHARDump:
def test_no_file_arg(self):
with tutils.raises(ScriptError):
tscript("har_dump.py")
tscript("complex/har_dump.py")
def test_simple(self):
with tutils.tmpdir() as tdir:
path = os.path.join(tdir, "somefile")
m, sc = tscript("har_dump.py", shlex.quote(path))
m, sc = tscript("complex/har_dump.py", shlex.quote(path))
m.addons.invoke(m, "response", self.flow())
m.addons.remove(sc)
@ -134,7 +139,7 @@ class TestHARDump:
with tutils.tmpdir() as tdir:
path = os.path.join(tdir, "somefile")
m, sc = tscript("har_dump.py", shlex.quote(path))
m, sc = tscript("complex/har_dump.py", shlex.quote(path))
m.addons.invoke(m, "response", self.flow(resp_content=b"foo" + b"\xFF" * 10))
m.addons.remove(sc)
@ -144,7 +149,7 @@ class TestHARDump:
assert har["log"]["entries"][0]["response"]["content"]["encoding"] == "base64"
def test_format_cookies(self):
m, sc = tscript("har_dump.py", "-")
m, sc = tscript("complex/har_dump.py", "-")
format_cookies = sc.ns.ns["format_cookies"]
CA = cookies.CookieAttrs
@ -174,7 +179,7 @@ class TestHARDump:
with tutils.tmpdir() as tdir:
path = os.path.join(tdir, "somefile")
m, sc = tscript("har_dump.py", shlex.quote(path))
m, sc = tscript("complex/har_dump.py", shlex.quote(path))
m.addons.invoke(m, "response", f)
m.addons.remove(sc)