110 lines
4.6 KiB
Markdown
110 lines
4.6 KiB
Markdown
# Porting to Other Implementations
|
||
|
||
## Introduction
|
||
|
||
This document provides an overview of the test runner and how to
|
||
integrate it with other stacks. So far we have it working with
|
||
BoringSSL and some incomplete integrations with NSS and OpenSSL.
|
||
|
||
Note that supporting non-BoringSSL implementations is a work in
|
||
progress and interfaces may change in the future. Consumers should pin
|
||
to a particular revision rather than using BoringSSL’s master branch
|
||
directly. As we gain experience with other implementations, we hope to
|
||
make further improvements to portability, so please contact
|
||
davidben@google.com and ekr@rtfm.com if implementing a new shim.
|
||
|
||
|
||
## Integration Architecture
|
||
|
||
The test runner integrates with the TLS stack under test through a
|
||
“shim”: a command line program which encapsulates the stack. By
|
||
default, the shim points to the BoringSSL shim in the same source
|
||
tree, but any program can be supplied via the `-shim-path` flag. The
|
||
runner opens up a server socket and provides the shim with a `-port`
|
||
argument that points to that socket. The shim always connects to the
|
||
runner as a TCP client even when acting as a TLS server. For DTLS,
|
||
there is a small framing layer that gives packet boundaries over
|
||
TCP. The shim can also pass a variety of command line arguments which
|
||
are used to configure the stack under test. These can be found at
|
||
`test_config.cc`.
|
||
|
||
|
||
The shim reports success by exiting with a `0` error code and failure by
|
||
reporting a non-zero error code and generally sending a textual error
|
||
value to stderr. Many of the tests expect specific error string (such
|
||
as `NO_SHARED_CIPHER`) that indicates what went wrong.
|
||
|
||
|
||
## Compatibility Issues
|
||
|
||
There are a number of situations in which the runner might succeed
|
||
with some tests and not others:
|
||
|
||
* Defects in the stack under test
|
||
* Features which haven’t yet been implemented
|
||
* Failure to implement one or more of the command line flags the runner uses with the shim
|
||
* Disagreement about the right behavior/interpretation of the spec
|
||
|
||
|
||
We have implemented several features which allow implementations to ease these compatibility issues.
|
||
|
||
### Configuration File
|
||
|
||
The runner can be supplied with a JSON configuration file which is
|
||
intended to allow for a per-stack mapping. This file currently takes
|
||
two directives:
|
||
|
||
|
||
* `DisabledTests`: A JSON map consisting of the pattern matching the
|
||
tests to be disabled as the key and some sort of reason why it was
|
||
disabled as the value. The key is used as a match against the test
|
||
name. The value is ignored and is just used for documentation
|
||
purposes so you can remember why you disabled a
|
||
test. `-include-disabled` overrides this filter.
|
||
|
||
* `ErrorMap`: A JSON map from the internal errors the runner expects to
|
||
the error strings that your implementation spits out. Generally
|
||
you’ll need to map every error, but if you also provide the
|
||
` -loose-errors` flag, then every un-mapped error just gets mapped to
|
||
the empty string and treated as if it matched every error the runner
|
||
expects.
|
||
|
||
|
||
The `-shim-config` flag is used to provide the config file.
|
||
|
||
|
||
### Unimplemented Features
|
||
If the shim encounters some request from the runner that it knows it
|
||
can’t fulfill (e.g., a command line flag that it doesn’t recognize),
|
||
then it can exit with the special code `89`. Shims are recommended to
|
||
use this exit code on unknown command-line arguments.
|
||
|
||
The test runner interprets this as “unimplemented” and skips the
|
||
test. If run normally, this will cause the test runner to report that
|
||
the entire test suite failed. The `-allow-unimplemented` flag suppresses
|
||
this behavior and causes the test runner to ignore these tests for the
|
||
purpose of evaluating the success or failure of the test suite.
|
||
|
||
|
||
### Malloc Tests
|
||
|
||
The test runner can also be used to stress malloc failure
|
||
codepaths. If passed `-malloc-test=0`, the runner will run each test
|
||
repeatedly with an incrementing `MALLOC_NUMBER_TO_FAIL` environment
|
||
variable. The shim should then replace the malloc implementation with
|
||
one which fails at the specified number of calls. If there are not
|
||
enough calls to reach the number, the shim should fail with exit code
|
||
`88`. This signals to the runner that the test has completed.
|
||
|
||
See `crypto/test/malloc.cc` for an example malloc implementation.
|
||
|
||
Note these tests are slow and will hit Go's test timeout. Pass `-timeout 72h` to
|
||
avoid crashing after 10 minutes.
|
||
|
||
|
||
## Example: Running Against NSS
|
||
|
||
```
|
||
DYLD_LIBRARY_PATH=~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/lib go test -shim-path ~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/bin/nss_bogo_shim -loose-errors -allow-unimplemented -shim-config ~/dev/nss-dev/nss-sandbox/nss/external_tests/nss_bogo_shim/config.json
|
||
```
|