test_xss_scanner.py was utterly failing because of a trouble (bug?)
with the `monkeypatch` fixture failing to replace `gethostbyname`
with the correct mock function.
Indeed, when stepping through the code, the `gethostbyname` presumably
mocked was reported as a builtin python function. The problem could
then come from the fact that it is hard to monkeypatch builtin function
in python.
Using absolute imports seems to resolve the problem.
This patch does a lot.
- Ditch sphinx in favor of hugo. This gives us complete control of the layout
and presentation of our docs. Henceforth, docs will be hosted on our website
rather than ReadTheDocs.
- Create a simple, clean doc layout and theme.
- Remove large parts of the documentaion. I've ditched anything that was a)
woefully out of date, b) too detailed, or c) too hard to maintain in the long
term.
- Huge updates to the docs themselves: completely rewrite addons documentation,
add docs for core concepts like commands and options, and revise and tweak a
lot of the existing docs.
With this patch, we're also changing the way we publish and maintain the docs.
From now on, we don't publish docs for every release. Instead, the website will
contain ONE set of docs for each major release. The online docs will be updated
if needed as minor releases are made. Docs are free to improve during minor
releases, but anything that changes behaviour sufficiently to require a doc
change warrants a new major release. This also leaves us free to progressively
update and improve docs out of step with our release cadence.
With this new scheme, I feel CI over the docs is less important. I've removed
it for now, but won't object if someone wants to add it back in.
SSL is an outdated protocol superseeded by TLS. Although the commonly
used library is called OpenSSL, it is no reason to still use outdated
language for attributes.
- Fix some loading sequence bugs affecting command-line script invocation
- Allow addons to over-ride existing options (with a warning). We need this for
reloading.
- Convert har_dump to new-style arguments, fix and re-instate its test suite.
- Covnert miscelaneous other exmples to new-style args.
- Addons now nest, which means that addons can manage addons. This has a number
of salutary effects - the scripts addon no longer has to poke into the global
addons list, we no longer have to replace/remove/boot-outof parent addons when
we load scripts, and this paves the way for making our top-level tools into
addons themselves.
- All addon calls are now wrapped in a safe execution environment where
exceptions are caught, and output to stdout/stderr are intercepted and turned
into logs.
- We no longer support script arguments in sys.argv - creating an option
properly is the only way to pass arguments. This means that all scripts are
always directly controllable from interctive tooling, and that arguments are
type-checked.
For now, I've disabled testing of the har dump example - it needs to be moved
to the new argument handling, and become a class addon. I'll address that in a
separate patch.
- Add an options parameter to the start() event. This is to be used by addons
on startup to add custom options.
- Add a running() event that is called once the proxy is up and running.
- With the new paradigm we can't log during master __init__, so add a tiny
termstatus addon to print proxy status to terminal once we're running.
This commit is largely based on work by Thiago Arrais (@thiagoarrais)
and Shane Bradfield (@l33tLumberjack). I wasn't really able to get their
PR reasonably merged onto the latest master, so I reapplied their changes
manually here and did some further improvements on that.
In transparent mode host is set with the target server ip.
Attribute flow.request.host is used as SNI while mitmproxy is
initiating TLS handshake, so it should be set with the pretty_host.
While the HAR spec is not very explicit and their example shows just this one example: ```json
"postData": {
"mimeType": "multipart/form-data"
}
```
Would it not make sense to include all the information necessary to parse out the post data `text`. Eg.
```json
"postData": {
"text": "--xYzZY\r\nContent-Disposition: form-data; name=\"sort1\"\r\n\r\noldest date first\r\n--xYzZY--\r\n",
"mimeType": "multipart/form-data; boundary=xYzZY"
},
```
Currently, full mimeType is included only in `content-type` request header.
Elsewhere in HAR spec they include the 'extras', eg ```json
"content": {
"mimeType": "text/html; charset=utf-8"
}
```
So one could argue that `mimeType` should include all information necessary to interpret the data. In case of `multipart/form-data`, as per RFC2046 http://www.ietf.org/rfc/rfc2046.txt
```
The Content-Type field for multipart entities requires one parameter, "boundary".
```
I believe that earlier incarnations, eg `har_exporter.py` included it in the mimeType.
The primary motivation here (and for all the other moving around) is to present
a clean "front of house" to library users, and to migrate primary objects to
the top of the module hierarchy.
