From a user's perspective, the test package provides two main pieces of functionality: an API for defining tests, and a command-line tool to run those tests. The structure of the package reflects this division. The code is divided into three main sections: the frontend, the backend, and the runner.
The lib/src/frontend directory contains APIs that are exposed to the user when they import package:test/test.dart. This includes core functions such as expect() and expectAsync(), test-specific matchers such as throwsA() and prints(), and annotation classes such as TestOn and Timeout. The functions that define the top-level structure of the test, such as test() and group(), are defined in lib/test.dart, but they can be thought of as frontend functions as well.
The frontend communicates with the backend using zone-scoped getters. Invoker.current provides access to the current test case to built-in matchers like completion(), for example to control when it completes. Structural functions use Declarer.current to gradually build up an in-memory representation of a test suite. The runner is in charge of setting up these variables, but the frontend never communicates with the runner directly.
The lib/src/backend directory contains classes that represent the in-memory structure of a test suite. A Suite represents a single test file, and class contains a tree of Groups, each of which contains many Tests. These classes are built using a Declarer.
The backend also contains the Invoker, which is responsible for actually running an individual test case—including tracking how many outstanding asynchronous callbacks are pending, handling exceptions, and timing out the test if it takes too long. The Invoker provides information about the status of a running test as streams and futures on a LiveTest object.
The backend provides a bridge between the frontend and the runner. The runner sets up the Declarer and starts the Invoker, which the frontend functions then communicate with directly.
The lib/src/runner directory contains the code that‘s executed when pub run test is invoked. It’s in charge of locating test files, loading them, executing them, and communicating their results to the user. It's also by far the biggest section. For more information on the runner architecture, see Lifecycle of a Test Run below.
To understand generally how the test runner works, let‘s look at an example run. When the user first invokes pub run test, the command-line arguments and configuration files are combined into a single Configuration object which is passed into the Runner class. The Runner is mostly just glue: it starts up the various components necessary for a test run, and connects them to one another. It’s also in charge of handling certain Configuration flags.
The first thing the runner starts is the Engine. The engine iterates through a test suite's tests and invokes them in order. It knows how to handle set-up and tear-down functions, and how to combine the output of multiple test suites running concurrently. It exposes its progress through a collection of getters and streams that provide access to individual LiveTests.
The runner then passes the Engine to a [Reporter][Reporter], which listens to the Engine's streams and exposes the information there to the user, usually by printing human-readable text. [CompactReporter][CompactReporter] is the default on Posix platforms, but others may be selected based on the Configuration. Nearly everything the user sees comes through the reporter. [Reporter]: https://github.com/dart-lang/test/tree/master/lib/src/runner/reporter.dart [CompactReporter]: https://github.com/dart-lang/test/tree/master/lib/src/runner/reporter/compact.dart
The Engine and Reporter can‘t do much of anything, though, without any test suites to run. The next step is to load those suites. The Loader is in charge of this part. It takes in file or directory paths and finds all the test files they contain—by default any files matching *_test.dart. It then proceeds to load each file on all the platforms specified in the Configuration that’s also supported by the test suite.
The specifics of loading suites differs based on whether the platform is a browser or the Dart VM. I‘ll cover each platform below, but for now let’s stick to what they have in common. Every platform will emit a LoadSuite, which is a synthetic Suite containing a single test that, when invoked, produces the actual Suite defined in the test file.
Wrapping the loading process in a synthetic Suite gives us the very useful invariant that all test errors occur within a Suite. Loading can fail in all sorts of ways—the code might not compile, the main() method might throw, the browser might not be installed, and so on. Locating those errors within a Suite means that the Engine and Reporter, which already know how to deal with test errors, can deal with load errors in exactly the same way. It makes the load process a little more complex, but it makes everything else a lot cleaner.
Once a Suite has been loaded, the runner does a little post-processing to make sure the Configuration is handled properly. It filters out tests whose tags don‘t match the --tags flag, or whose names don’t match the --name flag. Then it passes the resulting Suites on to the Engine and they begin to run.
Let's start with looking at how suites are loaded on the Dart VM, since the process is substantially simpler than loading them on a browser. This loading is handled by the VMPlatform, which extends the PlatformPlugin class. Eventually, we plan to support a user-accessible platform plugin API, so we model platforms as plugins to prepare for that.
In its simplest form, a PlatformPlugin‘s responsibility is just to create a StreamChannel that connects the test runner to a remote isolate—everything else is handled by helper functions. The VMPlatform uses Isolates to dynamically load its test suites, and then communicates with them using an IsolateChannel. It passes in a data: URI containing Dart code that imports the user’s code, and runs that code in the context of the serializeSuite() helper, and the PlatformPlugin superclass deserializes it on the other side using deserializeSuite().
When a test suite is serialized and deserialized, it's not just converted to and from some static representation like JSON. The Engine needs fine-grained control over the remote suite, and the [Reporter][Reporter] needs fine-grained access to the LiveTests it emits. To make this work, the helper functions use the MultiChannel class to tunnel streams for each test through the main IsolateChannel. Each test has its own virtual channel that gets a message when the test runner calls Test.load(), and that sends messages back to indicate the progress of the test.
Information about these virtual channels, as well as test names and metadata, are bundled up into a JSON object and sent over the IsolateChannel to be deserialized. The deserialization process then converts them into RunnerTests within a RunnerSuite, which the Engine can then run just like normal Tests in a normal Suite.
The BrowserPlatform class also extends PlatformPlugin, but rather than just emitting a StreamChannel and letting the plugin helpers do the rest, it takes more control over the loading process. It emits its own RunnerSuite, which allows it to expose its own Environment to enable debugging.
Whereas the VMPlatform loads each separate suite in isolation, the BrowserPlatform shares a substantial amount of resources between suites. All suites load their code from a single HTTP server, which is managed by the platform. This server provides access to Dart entrypoint wrappers for Dartium and content shell, to compiled JavaScript for other browsers, and to HTML files that bootstrap the tests.
In addition to sharing a server, when multiple suites are loaded for the same browser, they all share a tab within that browser. Each separate browser is controlled by its own BrowserManager, which uses WebSockets to communicate with Dart code running in the main frame—also known as the host.
Each browser is spawned with a tab pointing to packages/test/src/runner/browser/static/index.html, the host page. The host's code then opens a WebSocket connection to a dynamically-generated URL. This URL tells the BrowserPlatform which BrowserManager to send the WebSocket to.
To load a suite for this browser, the BrowserPlatform passes the URL for that suite's HTML file to the BrowserManager, which in turn sends it down to the host page. The host opens this HTML in an iframe, opens a StreamChannel with this iframe using Window.postMessage(). It then tunnels this channel through the WebSocket connection, again using MultiChannel, so that the BrowserManager has a direct line to the iframe where the tests are defined.
From this point forward the process is similar to VMPlatform. The iframe serializes its test suite using serializeSuite(), and the BrowserManager deserializes it using deserializeSuite(). It's then forwarded to the Loader via the BrowserPlatform.