From 40366fa94ec09a10f93732c0bf042a29ec866a29 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Thu, 12 Mar 2015 22:12:15 +1100 Subject: [PATCH] Virtualenv is the one and only recommended dev environment - Create a "dev" script to create a virtualenv-based dev environment - Update the docs to remove non-virtualenv install recommendations - Update osx-binaries generation to use virtualenv TODO: - The dev script or an equivalent should be made to work on Windows - We still can't remove the annoying top-level command scripts, because pyinstaller doesn't support entry points. Once it does, they can go. --- README.mkd | 61 ++++++++++++++++++++------------------ dev | 16 ++++++++++ release/osx-binaries | 70 ++++++++++++++++++++++++++------------------ 3 files changed, 90 insertions(+), 57 deletions(-) create mode 100755 dev diff --git a/README.mkd b/README.mkd index 0fdf3715e..ce2767642 100644 --- a/README.mkd +++ b/README.mkd @@ -16,8 +16,7 @@ mitmproxy.org website: [mitmproxy.org](http://mitmproxy.org). - -You can find complete directions for installing mitmproxy [here](http://mitmproxy.org/doc/install.html). +You can find complete directions for installing mitmproxy [here](http://mitmproxy.org/doc/install.html). Features @@ -41,43 +40,49 @@ only mitmdump is supported, which does not have a graphical user interface. Hacking ------- - -### Requirements - - -* [Python](http://www.python.org) 2.7.x. -* [netlib](http://pypi.python.org/pypi/netlib), version matching mitmproxy. -* Third-party packages listed in [setup.py](https://github.com/mitmproxy/mitmproxy/blob/master/setup.py) - -Optional packages for extended content decoding: - -* [PyAMF](http://www.pyamf.org/) version 0.6.1 or newer. -* [protobuf](https://code.google.com/p/protobuf/) version 2.5.0 or newer. -* [cssutils](http://cthedot.de/cssutils/) version 1.0 or newer. - -For convenience, all optional dependencies can be installed with - -`pip install "mitmproxy[contentviews]"` - -### Setting up a dev environment - -The following procedure is recommended to set up your dev environment: +To get started hacking on mitmproxy, make sure you have +[Python](http://www.python.org) 2.7.x. with +[virtualenv](https://virtualenv.pypa.io/en/latest/) installed (you can find +installation instructions for virtualenv +[here](https://virtualenv.pypa.io/en/latest/installation.html)). Then do the +following: ``` $ git clone https://github.com/mitmproxy/mitmproxy.git +$ git clone https://github.com/mitmproxy/netlib.git +$ git clone https://github.com/mitmproxy/pathod.git $ cd mitmproxy -$ pip install --src . -r requirements.txt +$ ./dev ``` -This installs the latest GitHub versions of mitmproxy, netlib and pathod into `mitmproxy/`. All other development dependencies save countershape are installed into their usual locations. +The *dev* script will create a virtualenv environment in a directory called +"venv.mitmproxy", and install all of mitmproxy's development requirements, plus +all optional modules. The primary mitmproxy components - mitmproxy, netlib and +pathod - are all installed "editable", so any changes to the source in the git +checkouts will be reflected live in the virtualenv. + +To confirm that you're up and running, activate the virtualenv, and run the +mitmproxy test suite: + +``` +$ source ../venv.mitmproxy/bin/activate +$ nosetests ./test +``` +Note that the main executables for the project - **mitmdump**, **mitmproxy** and +**mitmweb** - are all created within the virtualenv. After activating the +virtualenv, they will be on your $PATH, and you can run them like any other +command: + +```$ mitmdump --version``` + ### Testing -The test suite requires the `dev` extra requirements listed in [setup.py](https://github.com/mitmproxy/mitmproxy/blob/master/setup.py). Install these with: - -`pip install "mitmproxy[dev]"` +If you've followed the procedure above, you already have all the development +requirements installed, and you can simply run the test suite: +```nosetests ./test``` Please ensure that all patches are accompanied by matching changes in the test suite. The project maintains 100% test coverage. diff --git a/dev b/dev new file mode 100755 index 000000000..c8346af19 --- /dev/null +++ b/dev @@ -0,0 +1,16 @@ +#!/bin/sh +VENV=../venv.mitmproxy +PIP="$VENV/bin/pip --cache-dir ~/.pipcache" + +echo "This script sets up the following:" +echo "\t~/.pipcache - A pip cache directory" +echo "\t$VENV - A development virtualenv" + +mkdir -p ~/.pipcache + +virtualenv $VENV +source $VENV/bin/activate +$PIP install -r ./requirements.txt +# Re-install these to make them editable +$PIP install --editable ../netlib +$PIP install --editable ../pathod diff --git a/release/osx-binaries b/release/osx-binaries index d2af5e18f..990d047d4 100755 --- a/release/osx-binaries +++ b/release/osx-binaries @@ -2,32 +2,44 @@ # Quick and dangerous script for building OSX binaries. -# A few quirks to note, which should be re-checked every release: -# - We require the latest development version of PyInstaller. - -# To run, first install netlib and mitmproxy, then run -# -# ./release/osx-binaries -# -# From the top-level mitmproxy directory. - -usage () -{ - echo 'Usage : ./release/osx-binaries /path/to/pyinstaller.py' - echo 'Run from the top-level mitmproxy directory' - exit -} - -if [ "$1" = "" ] -then - usage -fi - +# At the moment, pyinstaller has no support for entry points, except for this +# hideous hack on the wiki: +# https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Setuptools-Entry-Point +# Once this is fixed, we can ditch the redundant command scripts. +VENV=../venv.mitmproxy +PYINST_CMD="$VENV/bin/pyinstaller -F --clean" TMPDIR=./tmp -PYINST_CMD=$1" -F --clean" CACHE="~/Library/Application Support/pyinstaller" + +if [ ! -d $VENV ] +then + echo "Failed: set up a dev environment as described in the README" + echo "and run from the top-level mitmproxy directory." + exit +fi + +source $VENV/bin/activate + +if [ ! -f $VENV/bin/pyinstaller ] + then + echo "Installing pyinstaller..." + $VENV/bin/pip install \ + --force-reinstall \ + --upgrade \ + https://github.com/pyinstaller/pyinstaller/archive/develop.zip + $VENV/bin/pip install --upgrade macholib +fi + +# readline.so is actually a symlink to a Python file, which breaks PyInstaller +# (and readline itself). Why? Who knows. Re-address this when this stupidity +# ceases to be. +echo "Removing broken readline..." +rm -f $VENV/lib/python2.7/readline.so + + +echo "Clearing caches..." rm -f dist/* rm -rf $TMPDIR rm -rf $CACHE @@ -36,14 +48,14 @@ $PYINST_CMD ./release/mitmdump.spec echo "Running mitmdump..." ./dist/mitmdump --version || exit 1 -$PYINST_CMD ./release/mitmproxy.spec -echo "Running mitmproxy..." -./dist/mitmproxy --version || exit 1 - -$PYINST_CMD ./release/mitmweb.spec -echo "Running mitmweb..." -./dist/mitmweb --version || exit 1 +#$PYINST_CMD ./release/mitmproxy.spec +#echo "Running mitmproxy..." +#./dist/mitmproxy --version || exit 1 +#$PYINST_CMD ./release/mitmweb.spec +#echo "Running mitmweb..." +#./dist/mitmweb --version || exit 1 +exit DST=osx-mitmproxy-`./dist/mitmdump --shortversion 2>&1` mkdir -p $TMPDIR/$DST cp ./dist/mitmproxy $TMPDIR/$DST