Test Infrastructure without Batteries

This package:

  • Provides a way to test a compiler in multiple steps.

  • Provides a way to run standalone tests. A standalone test is a test that has a main method, and can be run as a standalone program.

  • Ensures all tests and implementations are free of warnings (using dartanalyzer).

This package is not:

  • A replacement for package:test. This package can be used to run package:test tests, and package:test can be viewed as the batteries that aren't included in this package.

Motivation

We want to test tool chains, for example, a Dart compiler. Depending on the tool chain, it may comprise several individual steps. For example, to test dart2js, you have these steps:

  1. Run dart2js on a Dart source file to produce a Javascript output file.

  2. Run the Javascript file from step 1 on a Javascript interpreter and report if the program threw an exception.

On the other hand, to test a Dart VM, there's only one step:

  1. Run the Dart source file in the Dart VM and report if the program threw an exception.

Similarly, to test dartanalyzer, there's also a single step:

  1. Analyze the Dart source file and report if there were any problems.

In general, a tool chain can have more steps, for example, a pub transformer.

Furthermore, multiple tool chains may share the input sources and should agree on the behavior. For example, you should be able to compile hello-world.dart with dart2js and run it on d8 and it shouldn‘t throw an exception, running hello-world.dart on the Dart VM shouldn’t throw an exception, and analysing it with dartanalyzer should report nothing.

In addition, parts of the tool chain may have been implemented in Dart and have unit tests written in Dart, for example, using package:test. We want to run these unit tests, and have noticed that compiler unit tests in general run faster when run from the same Dart VM process (due to dynamic optimizations kicking in). For this reason, it‘s convenient to have a single Dart program that runs all tests. On the other hand, when developing, it’s often convenient to run just a single test.

For this reason, we want to support running unit tests individually, or combined in one program. And we also want the Dart-based implementation to be free of problems with respect to dartanalyzer.

Test Suites

A test suite is a collection of tests. Based on the above motivation, we have two kinds of suites:

  1. Chain, a test suite for tool chains.

  2. Dart, a test suite for Dart-based unit tests.

Getting Started

  1. Create a configuration file named testing.json.

  2. Run bin/run_tests.dart.

Configuration

The test runner is configured using a JSON file. A minimal configuration file is:

{
}

Chain

A Chain suite is a suite that's designed to test a tool chain and can be used to test anything that can be divided into one or more steps.

Here a complete example of a Chain suite:

{
  "suites": [
    {
      "name": "golden",
      "kind": "Chain",
      "source": "test/golden_suite.dart",
      "path": "test/golden/",
      "status": "test/golden.status",
      "pattern": [
        "\\.dart$"
      ],
      "exclude": [
      ]
    }
  ]
}

The properties of a Chain suite are:

name: a name for the suite. For simple packages, test or the package name are good candidates. In the Dart SDK, for example, it would be things like language, corelib, etc.

kind: always Chain for this kind of suite.

source: a relative URI to a Dart program that implements the steps in the suite. See below.

path: a URI relative to the configuration file which is the root directory of all files in this suite. For now, only file URIs are supported. Each file is passed to the first step in the suite.

status: a URI relative to the configuration file which lists the status of tests.

pattern: a list of regular expressions that match file names that are tests.

exclude: a list of regular expressions that exclude files from being included in this suite.

Implementing a Chain Suite

The source property of a Chain suite is a Dart program that must provide a top-level method with this name and signature:

Future<ChainContext> createContext(Chain suite) async { ... }

A suite is expected to implement a subclass of ChainContext which defines the steps that make up the chain and return it from createContext.

A step is a subclass of Step. The input to the first step is a TestDescription. The input to step n+1 is the output of step n.

Here is an example of a suite that runs tests on the Dart VM:

import 'testing.dart';

Future<ChainContext> createContext(
    Chain suite, Map<String, String> environment) async {
  return new VmContext();
}

class VmContext extends ChainContext {
  final List<Step> steps = const <Step>[const DartVmStep()];
}

class DartVmStep extends Step<TestDescription, int, VmContext> {
  const DartVmStep();

  String get name => "Dart VM";

  Future<Result<int>> run(TestDescription input, VmContext context) async {
    StdioProcess process = await StdioProcess.run("dart", [input.file.path]);
    return process.toResult();
  }
}

main(List<String> arguments) => runMe(arguments, createContext);

An example with multiple steps in the chain can be found in the Kernel package's suite. Notice how this suite stores an AnalysisContext in its TestContext and is this way able to reuse the same AnalysisContext in all tests.

Dart

The Dart suite is for running unit tests written in Dart. Each test is a Dart program with a main method that can be run directly from the command line.

The suite generates a new Dart program which combines all the tests included in the suite, so they can all be run (in sequence) in the same process. Such tests must be co-operative and must clean up after themselves.

You can use any test-framework, for example, package:test in these individual programs, as long as the frameworks are well-behaved with respect to global state, see below.

Here is a complete example of a Dart suite:

{
  "suites": [
    {
      "name": "my-package",
      "path": "test/",
      "kind": "Dart",
      "pattern": [
        "_test\\.dart$"
      ],
      "exclude": [
        "/test/golden/"
      ]
    }
  ]
}

The properties of a Dart suite are:

name: a name for the suite. For simple packages, test or the package name are good candidates. In the Dart SDK, for example, the names could be the name of the component that‘s tested by this suite’s unit tests, for example, dart2js.

path: a URI relative to the configuration file which is the root directory of all files in this suite. For now, only file URIs are supported.

kind: always Dart for this kind of suite.

pattern: a list of regular expressions that match file names that are tests.

exclude: a list of regular expressions that exclude files from being included in this suite.

Well-Behaved Tests

The Dart suite makes certain assumptions about the tests it runs.

  • All tests use the same packages configuration file.

  • An asynchronous test returns a Future from its main.

  • Tests manages global state.

All tests are imported into the same program as individual libraries, which is why they all must use the same .packages file. The tests aren‘t concatenated, so they have the lexical scope you’d normally expect from a Dart library.

Tests are run in order. In particular, the test framework will not start the next test until any future returned from the current test's main method complete. In addition, asynchronous tests are expected to have finished all asynchronous operations when the future returned from their main method completes (with or without an error).

Tests are expected to manage global state (aka static state). Simply put: clean up after yourself. But if it‘s simpler to ensure global state is reset before running a test and not clean up afterwards, that’s also fine as long as all tests agree on how to manage their shared global state.

Configuring Analyzed Programs

By default, all tests in Dart suites are analyzed by the dartanalyzer. It is possible to exclude tests from analysis, and it's possible to add additional files to be analyzed. Here is a complete example of a Dart suite and analyzer configuration:

{
  "suites": [
    {
      "name": "my-package",
      "path": "test/",
      "kind": "Dart",
      "pattern": [
        "_test\\.dart$"
      ],
      "exclude": [
        "/test/golden/"
      ]
    }
  ],
  "analyze": {
    "uris": [
      "lib/",
    ],
    "exclude": [
      "/third_party/"
    ]
  }
}

The properties of the analyze section are:

uris: a list of URIs relative to the configuration file that should also be analyzed. For now, only file URIs are supported.

exclude: a list of regular expression that matches file names that should be excluded from analysis. For now, the files are still analyzed but diagnostics are suppressed and ignored.

Integrating with Other Test Runners

test.dart

To run the suite my_suite from test.dart, create a file named mysuite_test.dart with this content:

import 'package:async_helper/async_helper.dart' show asyncTest;

import 'package:testing/testing.dart' show run;

main(List<String> arguments) => asyncTest(run(arguments, ["my_suite"]));

package:test

To run the suite my_suite from package:test, create a file named mysuite_test.dart with this content:

import 'package:test/test.dart' show Timeout, test;

import 'package:testing/testing.dart' show run;

main() {
  test("my_suite", () => run([], ["my_suite"]),
      timeout: new Timeout(new Duration(minutes: 5)));
}