Commit Graph

389 Commits

Author SHA1 Message Date
Joaquin
f4a6153f2e Fixing Typos 2018-05-13 17:31:20 -05:00
Thomas Kriechbaumer
e305a320a2 fix tcp example 2018-05-11 11:58:01 +02:00
Maximilian Hils
0c101a4bcc
Merge pull request #3106 from cortesi/noprint
Ditch the addon stdout wrapper
2018-05-08 15:24:02 +02:00
0xHJK
a46a317dca har_dump example cmdline invocation 2018-05-08 16:27:56 +08:00
Aldo Cortesi
af1a4ffdcd Ditch the addon stdout wrapper
This results in a 30% improvement in our core request throughput.

Fixes #3102
2018-05-08 14:26:41 +12:00
Aldo Cortesi
76a7883c8b events: document behaviour of done() more fully 2018-05-08 11:09:10 +12:00
Aldo Cortesi
9e283b2779
Merge pull request #3097 from cortesi/examp
Revamp dup_and_replay example
2018-05-05 10:05:43 +12:00
Aldo Cortesi
f5dc0aace1 Revamp dup_and_replay example
- Exposes view.add as a command
- Copes with cases where a view addon isn't present
- Avoids infinite loop caused by replaying replays

Fixes #3096
2018-05-05 09:34:22 +12:00
Aldo Cortesi
f380a77dee Remove the tick event
Mitmproxy: the tickless wonder.
2018-05-02 14:29:15 +12:00
oscure76
0e984e1442 fix Python 3.6 variable type annotations #3053 2018-04-14 16:24:41 -07:00
Thomas Kriechbaumer
fe699e555f fix example filename 2018-04-13 09:09:24 +02:00
Barry Lee
b35311441f
Typo fix 2018-03-10 23:24:56 +08:00
Aldo Cortesi
ded9423764
Merge pull request #2922 from kajojify/new_example
Replaces upsidedown with mirror reflection
2018-02-26 08:48:52 +13:00
Miroslav
3b134df77f Replace upsidedown with mirror reflection 2018-02-25 21:17:08 +02:00
luz.paz
afbb7f117b Misc. typos
Found via `codespell -q 3 -I ../mitmproxy-word-whitelist.txt`
Where whitelist contains:
```
cas
doubleclick
nd
ot
seeked
statics
te
thru
```
2018-02-24 21:45:11 -05:00
Aldo Cortesi
982508d30f All new documentation
This patch does a lot.

- Ditch sphinx in favor of hugo. This gives us complete control of the layout
and presentation of our docs. Henceforth, docs will be hosted on our website
rather than ReadTheDocs.
- Create a simple, clean doc layout and theme.
- Remove large parts of the documentaion. I've ditched anything that was a)
woefully out of date, b) too detailed, or c) too hard to maintain in the long
term.
- Huge updates to the docs themselves: completely rewrite addons documentation,
add docs for core concepts like commands and options, and revise and tweak a
lot of the existing docs.

With this patch, we're also changing the way we publish and maintain the docs.
From now on, we don't publish docs for every release. Instead, the website will
contain ONE set of docs for each major release. The online docs will be updated
if needed as minor releases are made. Docs are free to improve during minor
releases, but anything that changes behaviour sufficiently to require a doc
change warrants a new major release. This also leaves us free to progressively
update and improve docs out of step with our release cadence.

With this new scheme, I feel CI over the docs is less important. I've removed
it for now, but won't object if someone wants to add it back in.
2018-02-22 18:07:58 +13:00
Maximilian Hils
95c160ac13
Merge pull request #2884 from kajojify/hotkeys-cleanup
Hotkeys cleanup. #2877
2018-02-20 21:45:48 +01:00
Maximilian Hils
3934dbb5ad
update log_events example documentation 2018-02-20 17:15:35 +01:00
Maximilian Hils
29ef663aa1
add documentation for custom_option.py 2018-02-20 17:14:21 +01:00
Miroslav
aea6fd28c8 Hotkeys cleanup 2018-02-20 16:25:39 +02:00
Arushit Mudgal
91834f98cc Extend mypy checking, fix #2194 (#2819) 2018-02-03 21:37:33 +01:00
Aniket Panjwani
71494c6a06 Fix mistakes in examples/simple/Readme.md.
1. Delete row for script_arguments.py.
2. Add row for custom_option.py, using script_arguments.py's Description.
2018-01-29 12:06:38 -06:00
Thomas Kriechbaumer
9aae3213b9 rename TLS/SSL-related attributes
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for attributes.
2018-01-06 10:43:47 +01:00
David Dworken
04a06eb6b5 Added scanning for CSS injection and iframe injection 2017-10-17 23:39:33 -04:00
Suraj Tripathi
781369a326 fix #2477 (#2556) 2017-09-01 23:08:34 +02:00
Ujjwal Verma
6367dcab8e update streaming docs 2017-07-04 10:53:23 +02:00
Maximilian Hils
eb5f37a7d1 fix mypy annotations 2017-05-26 16:14:20 +02:00
harsh vijay
e24b4cc1b6 Extend Mypy checking to pathod
* mypy checking pathod

* initial commit , fixed errors

* tox: mypy checking to pathod

* Fixed mypy test failed

* issue was with args in custom_contentview.py

* tox: mypy checking to #2221

* follow-import=skip since we cant provide args to custom_contentview.py during mypy testing

* Lint , Typo Fixed

* code style: module import
2017-05-02 05:19:25 +05:30
harsh vijay
789fbd00d2 Fixed lint error 2017-04-29 03:54:00 +05:30
harsh vijay
6e03231d25 lint error fixed 2017-04-29 03:49:08 +05:30
harsh vijay
36118973d9 extend mypy example/simple 2017-04-29 03:26:14 +05:30
Aldo Cortesi
2a46f3851a Merge pull request #2265 from cortesi/addons
Addons and addon testing
2017-04-27 07:40:14 +12:00
Maximilian Hils
ab07b79138 Merge pull request #2270 from F1ashhimself/master
Update readme for complex examples
2017-04-26 17:15:34 +02:00
Maksim Beloborodko
90a5b90b0d Update readme for complex examples 2017-04-26 17:14:41 +03:00
Maximilian Hils
87610cc8b2 fix #2250, add type info to cookie module 2017-04-26 14:17:14 +02:00
Aldo Cortesi
5327756377 Addons and addon testing
- Fix some loading sequence bugs affecting command-line script invocation
- Allow addons to over-ride existing options (with a warning). We need this for
reloading.
- Convert har_dump to new-style arguments, fix and re-instate its test suite.
- Covnert miscelaneous other exmples to new-style args.
2017-04-26 19:56:33 +12:00
Aldo Cortesi
b72f139093 configure(options, updated) -> configure(updated)
Options are now available globally on ctx, so the first argument of configure
is redundant.
2017-04-26 11:01:27 +12:00
Aldo Cortesi
e6eeab6094 Revamp how addons work
- Addons now nest, which means that addons can manage addons. This has a number
of salutary effects - the scripts addon no longer has to poke into the global
addons list, we no longer have to replace/remove/boot-outof parent addons when
we load scripts, and this paves the way for making our top-level tools into
addons themselves.
- All addon calls are now wrapped in a safe execution environment where
exceptions are caught, and output to stdout/stderr are intercepted and turned
into logs.
- We no longer support script arguments in sys.argv - creating an option
properly is the only way to pass arguments. This means that all scripts are
always directly controllable from interctive tooling, and that arguments are
type-checked.

For now, I've disabled testing of the har dump example - it needs to be moved
to the new argument handling, and become a class addon. I'll address that in a
separate patch.
2017-04-25 22:13:44 +12:00
Maximilian Hils
482c1579f2 proxapp -> proxapp.local
Chrome doesn't like it if there's no TLD.
2017-03-26 15:02:47 +02:00
Aldo Cortesi
65f0885bd6 addon loader: add boot_into, which replaces returning from start()
While we're here, expand test coverage for addonmanager to 100%, and promote to
individual coverage.
2017-03-25 10:48:12 +13:00
Aldo Cortesi
541c1e8b9f addons: start -> load throughout 2017-03-25 10:48:12 +13:00
Aldo Cortesi
1410cbb4b6 Remove test handlers by using taddons.RecordingMaster
This also means expanding and tweaking the recording master API, which we
reflect through the current test suite
2017-03-16 18:05:57 +13:00
Aldo Cortesi
b745428b5c Enable custom options in config files
We also now ignore unknown options in config files by default, and print a
warning if verbosity is incremented.
2017-03-14 09:40:33 +13:00
Aldo Cortesi
c24f7d8e12 Optmanager: handle unknown options in value sets 2017-03-14 08:32:19 +13:00
Aldo Cortesi
0c6663d0d5 Enable custom options for addons
- Add an options parameter to the start() event. This is to be used by addons
on startup to add custom options.
- Add a running() event that is called once the proxy is up and running.
- With the new paradigm we can't log during master __init__, so add a tiny
termstatus addon to print proxy status to terminal once we're running.
2017-03-14 08:32:19 +13:00
Maximilian Hils
927b5707fe fix tcp.Address leftovers
this fixes the issue described in https://github.com/mitmproxy/mitmproxy/issues/2119#issuecomment-285067292
2017-03-08 16:18:34 +01:00
Nikhil Soni
0081d9b828 Merge branch 'master' into on-issues 2017-03-03 12:58:44 +05:30
Nikhil Soni
317d183ba4 Changes dns_spoofing example to use --keep-host-header 2017-03-03 12:34:36 +05:30
David Dworken
99b584ad7d added XSS scanner example (#1907) 2017-02-27 18:22:39 +01:00
Ujjwal Verma
e723a58af5 Remove pytz in hardump 2017-02-25 23:07:47 +05:30
Thomas Kriechbaumer
36352c9539 protobuf: coverage++ 2017-02-19 14:29:09 +01:00
Maximilian Hils
b9e31f213f .headers["host"] -> .host_header 2017-02-18 00:13:14 +01:00
Maximilian Hils
5acdd78b15 fix typo 2017-02-14 17:45:54 +01:00
Ujjwal Verma
c622622c59 Encoding fixes and tests 2017-02-05 18:59:01 +05:30
Ammonite
aaff9dfd32 Reset the host header to the correct destination 2017-01-30 23:41:47 +01:00
Ammonite
c2c6050df3 Store original host in flow metadata 2017-01-29 14:33:53 +01:00
Ammonite
0ca1916f1b Fix host extraction 2017-01-22 15:28:14 +01:00
Ammonite
e8fc4af4c6 Follow PEP-8 and add comment 2017-01-22 14:59:46 +01:00
Ammonite
93172460aa Add blank lines for lint 2017-01-21 09:39:34 +01:00
Ammonite
8aa250d679 Change class name 2017-01-20 23:48:26 +01:00
Ammonite
a55eba3b37 Get the the original header in requestheaders instead of request 2017-01-20 23:43:53 +01:00
Feei
95cca4ce75 update stream.py 2017-01-16 18:30:05 +08:00
Feei
eaaec4353d update log_events.py 2017-01-16 18:11:18 +08:00
Maximilian Hils
98a079aa69 rename logging.py example. this conflicted with Python's stdlib 2016-12-19 16:21:17 +01:00
Maximilian Hils
c4e9000021 fix #1858 2016-12-19 01:15:10 +01:00
Maximilian Hils
3e37cbd061 minor fixes 2016-12-10 12:06:33 +01:00
Maximilian Hils
293b79af91 remove lxml-dependent code 2016-12-10 10:19:05 +01:00
Maximilian Hils
45332006a3 mitmweb: 100% app test coverage, numerous fixes 2016-11-23 22:35:07 +01:00
Maximilian Hils
40f0193dda remove mitmweb auth
9b08279c7c removed the actual functionality.
we should not have a command line switch that does nothing. :)
2016-11-22 18:27:16 +01:00
Maximilian Hils
c90405253a remove stickycookie example
The recommended way to do this is mitmproxy/addons/stickycookie.py
2016-11-21 02:46:25 +01:00
Maximilian Hils
9af8f4bb31 organize examples
This commit is largely based on work by Thiago Arrais (@thiagoarrais)
and Shane Bradfield (@l33tLumberjack). I wasn't really able to get their
PR reasonably merged onto the latest master, so I reapplied their changes
manually here and did some further improvements on that.
2016-11-21 02:28:10 +01:00
Mike Fotinakis
d16a3753d7 Remove dead run_scripthooks example reference. 2016-11-17 16:33:51 -08:00
Maximilian Hils
2a2387fb32 explain host=pretty_host assignment 2016-11-09 13:11:31 +01:00
phackt
dc44465c92 fix SNI for transparent mode - #1638
In transparent mode host is set with the target server ip.
Attribute flow.request.host is used as SNI while mitmproxy is
initiating TLS handshake, so it should be set with the pretty_host.
2016-11-08 15:39:24 +01:00
Aldo Cortesi
4eb2b56dec Let's not over-ride __bool__ on connection objects
If I had a thousand years and every thesaurus in the world, I still couldn't
adequately express how much I dislike this piece of interface design.
2016-11-04 10:59:41 +13:00
Thomas Kriechbaumer
d56bbb95e2 rename logging .py to prevent import errors 2016-10-29 11:31:38 -07:00
Maximilian Hils
f26a375560 fix #1678 2016-10-27 12:55:39 -07:00
Maximilian Hils
eda1b39a74 minor fixes 2016-10-25 22:06:52 -07:00
Slobodan Mišković
39d7ba852c Include boudary=... in mutipart postData
While the HAR spec is not very explicit and their example shows just this one example: ```json
"postData": {
    "mimeType": "multipart/form-data"
}
```
Would it not make sense to include all the information necessary to parse out the post data `text`. Eg.
```json
"postData": {
           "text": "--xYzZY\r\nContent-Disposition: form-data; name=\"sort1\"\r\n\r\noldest date first\r\n--xYzZY--\r\n",
           "mimeType": "multipart/form-data; boundary=xYzZY"
         },
```
Currently, full mimeType is included only in `content-type` request header.

Elsewhere in HAR spec they include the 'extras', eg ```json
"content": {
    "mimeType": "text/html; charset=utf-8"
}
``` 
So one could argue that `mimeType` should include all information necessary to interpret the data. In case of `multipart/form-data`, as per RFC2046 http://www.ietf.org/rfc/rfc2046.txt
```
 The Content-Type field for multipart entities requires one parameter, "boundary".
```
I believe that earlier incarnations, eg `har_exporter.py` included it in the mimeType.
2016-10-24 14:34:04 -07:00
Maximilian Hils
a1a792aeac various encoding fixes, fix #1650 2016-10-22 18:47:12 -07:00
Slobodan Mišković
0526d94f4a Handle bytes in request parameters 2016-10-22 18:28:32 -07:00
Brady Law
0d0a3a51df The first argument should be the filter, then the flow. 2016-10-21 16:47:19 -07:00
Aldo Cortesi
8430f857b5 The final piece: netlib -> mitproxy.net 2016-10-20 11:56:38 +13:00
Aldo Cortesi
f45f4e677e netlib.strutils -> mitmproxy.utils.strutils 2016-10-20 10:11:58 +13:00
Aldo Cortesi
7440232f60 netlib.version -> mitmproxy.version 2016-10-20 09:20:44 +13:00
Aldo Cortesi
e73c7fe77e mitmproxy.protocol -> mitmproxy.proxy.protocol
The protocols here are compltely proxy-specific, are only used from within the
proxy module, and are not exposed to users.
2016-10-19 23:11:56 +13:00
Aldo Cortesi
24cf8da27e Move all tools into mitmproxy.tools, move models/* to top level
The primary motivation here (and for all the other moving around) is to present
a clean "front of house" to library users, and to migrate primary objects to
the top of the module hierarchy.
2016-10-19 20:26:05 +13:00
Aldo Cortesi
5a68d21e8c Remove flow module entirely, move contents to top level
mitmproxy.flow.io -> mitmproxy.io
mitmproxy.flow.export -> mitmproxy.export
2016-10-19 15:08:35 +13:00
Aldo Cortesi
7c32d4ea2a flow.state -> addons.state 2016-10-19 14:48:42 +13:00
Aldo Cortesi
22eebfd574 addons.Addons -> addonmanager, builtins -> addons 2016-10-19 14:39:39 +13:00
Aldo Cortesi
bce387a5a0 Kill flow.master - create master.Master
Also extract events into .events
2016-10-19 13:22:50 +13:00
Aldo Cortesi
87629586ae web app cleanups: tests and examples 2016-10-19 11:48:51 +13:00
Maximilian Hils
02d3d61820 fix redirect_requests.py example 2016-10-16 23:49:54 -07:00
Aldo Cortesi
c774a9fec9 python3: clean up super and __future__ 2016-10-17 18:03:07 +13:00
Aldo Cortesi
a647b30365 python3: clean up class brackets 2016-10-17 17:37:08 +13:00
Aldo Cortesi
fb22f2ff4f Zap object base class 2016-10-17 17:37:08 +13:00
Aldo Cortesi
ce98a9219e test & examples: zap six 2016-10-17 16:45:45 +13:00
Aldo Cortesi
55cb2a8547 docs: logging and the context 2016-10-16 20:26:06 +13:00
Aldo Cortesi
fb69c9c345 docs: overview, classes, arguments 2016-10-16 20:26:06 +13:00
Thomas Kriechbaumer
bb60b76af4 use flowfilter.match 2016-10-03 11:45:54 +02:00
Thomas Kriechbaumer
ba84248910 rename mitmproxy.filt -> mitmproxy.flowfilter 2016-10-03 11:45:54 +02:00
phackt
8021427ab9 Fixes - #1555 sslstrip.py flow.response.headers (#1556)
* Fixes - #1555 sslstrip.py flow.response.headers

* #1557 - add enhancements in inline script sslstrip.py with upgrade-insecure-requests stripping

* #1557 - update to match python style guide

* #1555, #1556, update to a bytes pattern
2016-09-25 19:29:26 -07:00
smill
7cd8456445 Added a description to the shim loader, and renamed it. 2016-09-14 19:25:53 +00:00
smill
fbfedbdc8f Improved error-handling / supplemented documention. 2016-09-04 01:30:27 +00:00
Aldo Cortesi
afe34e8b28 Improve the way we handle upstream errors
- Don't log a traceback for either HTTP or HTTPS DNS resolution or TCP
connection errors. These are "ordinary" errors, not mitmproxy issues.
- Ensure that the error handler is correctly called for SSL-related protocol
errors.
2016-09-01 12:32:09 +12:00
Aldo Cortesi
9306e80e65 Adjust flowbasic example for Options API changes 2016-09-01 12:11:00 +12:00
Maximilian Hils
f27028f58e introduce Response.make for simple response creation 2016-08-23 00:17:06 -07:00
Shadab Zafar
bf4425de80 Fix issue with binary content in json 2016-08-15 12:00:23 +05:30
Shadab Zafar
4f1fb43dcc Use postData field in PUT, PATCH requests too
The HAR spec isn't really clear on whether this should be the case,
but Google Chrome does this, so I think we should too.
2016-08-15 12:00:23 +05:30
Shadab Zafar
15c488225f Refactor format_cookies 2016-08-15 12:00:23 +05:30
Shadab Zafar
567cbe6cb9 Support .zhar compression 2016-08-15 12:00:23 +05:30
Shadab Zafar
ac97e5efa1 Add text field to response content 2016-08-15 12:00:23 +05:30
Shadab Zafar
9aa230707d Add serverIPAddress field 2016-08-15 12:00:23 +05:30
Shadab Zafar
7de48fc197 Add postData field 2016-08-15 12:00:23 +05:30
Shadab Zafar
b14eb57db1 Add SSL & Connect timings 2016-08-15 12:00:23 +05:30
Shadab Zafar
23b9ef799e Add a ctx.log on finish 2016-08-15 12:00:23 +05:30
Shadab Zafar
456f80d862 Open JSON file in text mode 2016-08-15 12:00:23 +05:30
Shadab Zafar
2c9240fd22 Simplify name_value 2016-08-15 12:00:23 +05:30
Shadab Zafar
e9c6563367 Fix wrong import 2016-08-15 12:00:23 +05:30
Shadab Zafar
a2a8283fa4 Improve cookies formatting 2016-08-15 12:00:23 +05:30
Shadab Zafar
55f1ffe0b1 Format Cookies according to the HAR Spec 2016-08-15 12:00:23 +05:30
Shadab Zafar
a0932af55c Remove pages object
The HAR Spec says that the field can be left out by applications that don't
group by pages.

http://www.softwareishard.com/blog/har-12-spec/#log
2016-08-15 12:00:23 +05:30
Shadab Zafar
250e4a17d0 Welcome har_dump 2016-08-15 12:00:23 +05:30
Shadab Zafar
8ba34be7ab Goodbye har_extractor 2016-08-15 12:00:23 +05:30
Maximilian Hils
fb238ad63e make the linter happy 2016-07-23 14:35:04 -07:00
Maximilian Hils
b78fea3784 add remote debug example 2016-07-23 14:02:54 -07:00
Aldo Cortesi
afda175e1c Addon iface: .configure(options) -> .configure(options, updated) 2016-07-23 15:43:55 +12:00
Aldo Cortesi
dbafe9f87b scripts: refactor some examples that keep global state
We now have a better way to do this.
2016-07-23 12:24:09 +12:00
Maximilian Hils
3602fd7a36 Merge remote-tracking branch 'origin/master' into message-body-encoding 2016-07-15 23:17:57 -07:00
Aldo Cortesi
917d51bd22 Fix HAR extractor
Thanks to @mhils
2016-07-15 15:01:35 +12:00
Aldo Cortesi
92a5076bd0 Adjust concurrent tests for start method 2016-07-15 15:00:37 +12:00
Aldo Cortesi
4ba7ce50c6 Add .start for addons and scripts
Also improve error messages, fix various unit tests
2016-07-15 14:52:34 +12:00
Aldo Cortesi
b94f5fd361 Convert examples and example tests for new-style scripts
Remove the test that just loads all the example scripts for now - it's a very
low-value test, and we need to think of something better.
2016-07-14 19:54:15 +12:00
Maximilian Hils
5d2b7c52f9 move script context to mitmproxy.ctx 2016-07-08 19:57:57 -07:00
Maximilian Hils
7c67faa8da remove script contexts 2016-07-07 23:50:55 -07:00
Maximilian Hils
c048ae1d5b remove context from all scripts 2016-07-07 20:14:52 -07:00
Maximilian Hils
76473d44e0 py3++ 2016-07-07 17:50:26 -07:00
Maximilian Hils
f259b9478f disable harparser on py3 2016-07-07 01:21:15 -07:00
Maximilian Hils
444f0a4c39 py3++ 2016-07-06 17:31:08 -07:00
Maximilian Hils
d51cf543bb remove clean_bin, clarify unicode handling 2016-07-05 15:11:32 -07:00
Maximilian Hils
dbf7cb1a44 update examples: no decoded() anymore 🎉 2016-07-02 02:01:46 -07:00
Tai Dickerson
c2d0910e59 Make har_extractor.py output HAR 1.2 spec-compliant
HAR files were failing to load in harviewer (http://www.softwareishard.com/har/viewer/) due to:
- ISO 8601 dates for startedDateTime missing timezone (http://www.softwareishard.com/blog/har-12-spec/#pages)
-- Used UTC but could add detection of default system timezone if desired
- pages object missing pageTimings (http://www.softwareishard.com/blog/har-12-spec/#pageTimings)
-- Used {} because all child fields are optional
2016-06-14 12:47:10 -04:00
Maximilian Hils
63c6660895 update examples, tests, docs 2016-06-13 18:17:09 -07:00
Aldo Cortesi
b3bf754e53 Simplify script concurrency helpers
We now have take() to prevent double-replies.
2016-06-08 11:21:38 +12:00
Aldo Cortesi
a388ddfd78 A new interface for reply
Reply is now explicit - it's no longer a callable itself. Instead, we have:

    reply.kill()            - kill the flow
    reply.ack()             - ack, but don't send anything
    reply.send(message)     - send a response

This is part of an incremental move to detach reply from our flow objects,
and unify the script and handler interfaces.
2016-06-08 10:44:20 +12:00
Aldo Cortesi
09da1febbd Shift a bunch more string-related functions to strutils 2016-06-02 13:04:19 +12:00
Aldo Cortesi
cccdc98426 Utils reorganisation: add netlib.strutils
Extract a number of string and format-related functions to netlib.strutils.
2016-06-02 12:31:41 +12:00
Maximilian Hils
2e633af5aa handle_* -> * leftovers 2016-05-29 13:01:45 -07:00
Thomas Kriechbaumer
8eeab37a07 more style cleanup
Use this to check:
flake8 --count mitmproxy netlib pathod examples test
2016-05-29 11:55:12 +02:00
Thomas Kriechbaumer
0c2c017727 change comment indentation 2016-05-29 11:04:55 +02:00