pkgs/test/doc/configuration.md

Each package may include a configuration file that applies to the package as a
whole. This file can be used to provide custom defaults for various options, to
define configuration for multiple files, and more.

The file is named `dart_test.yaml` and lives at the root of the package, next to
the package's pubspec. Like the pubspec, it's a [YAML][] file. Here's an
example:

[YAML]: http://yaml.org/

```yaml
# This package's tests are very slow. Double the default timeout.
timeout: 2x

# This is a browser-only package, so test on chrome by default.
platforms: [chrome]

tags:
  # Integration tests are even slower, so increase the timeout again.
  integration: {timeout: 2x}

  # Sanity tests are quick and verify that nothing is obviously wrong. Declaring
  # the tag allows us to tag tests with it without getting warnings.
  sanity:
```

* [Test Configuration](#test-configuration)
  * [`timeout`](#timeout)
  * [`ignore-timeouts`](#ignore-timeouts)
  * [`verbose_trace`](#verbose_trace)
  * [`chain_stack_traces`](#chain_stack_traces)
  * [`js_trace`](#js_trace)
  * [`skip`](#skip)
  * [`retry`](#retry)
  * [`test_on`](#test_on)
  * [`allow_test_randomization`](#allow_test_randomization)
  * [`allow_duplicate_test_names`](#allow_duplicate_test_names)
* [Runner Configuration](#runner-configuration)
  * [`include`](#include)
  * [`paths`](#paths)
  * [`filename`](#filename)
  * [`names`](#names)
  * [`plain_names`](#plain_names)
  * [`include_tags`](#include_tags)
  * [`exclude_tags`](#exclude_tags)
  * [`platforms`](#platforms)
  * [`concurrency`](#concurrency)
  * [`pause_after_load`](#pause_after_load)
  * [`run_skipped`](#run_skipped)
  * [`pub_serve`](#pub_serve)
  * [`reporter`](#reporter)
  * [`file_reporters`](#file_reporters)
  * [`fold_stack_frames`](#fold_stack_frames)
  * [`custom_html_template_path`](#custom_html_template_path)
* [Configuring Tags](#configuring-tags)
  * [`tags`](#tags)
  * [`add_tags`](#add_tags)
* [Configuring Platforms](#configuring-platforms)
  * [`on_os`](#on_os)
  * [`on_platform`](#on_platform)
  * [`override_platforms`](#override_platforms)
  * [`define_platforms`](#define_platforms)
  * [Browser and Node.js Settings](#browser-and-nodejs-settings)
    * [`arguments`](#arguments)
    * [`executable`](#executable)
    * [`headless`](#headless)
* [Configuration Presets](#configuration-presets)
  * [`presets`](#presets)
  * [`add_presets`](#add_presets)
* [Global Configuration](#global-configuration)

## Test Configuration

There are two major categories of configuration field: "test" and "runner". Test
configuration controls how individual tests run, while
[runner configuration](#runner-configuration) controls the test runner as a
whole. Both types of fields may be used at the top level of a configuration
file. However, because different tests can have different test configuration,
only test configuration fields may be used to [configure tags](#tags) or
[platforms](#on_platform).

### `timeout`

This field indicates how much time the test runner should allow a test to remain
inactive before it considers that test to have failed. It has three possible
formats:

* The string "none" indicates that tests should never time out.

* A number followed by a unit abbreviation indicates an exact time. For example,
  "1m" means a timeout of one minute, and "30s" means a timeout of thirty
  seconds. Multiple numbers can be combined, as in "1m 30s".

* A number followed by "x" indicates a multiple. This is applied to the default
  value of 30s.

```yaml
timeout: 1m
```

###  `ignore-timeouts`

This field disables all timeouts for all tests. This can be useful when debugging, so tests don't time out during debug sessions. It defaults to `false`.

```yaml
ignore-timeouts: true
```

### `verbose_trace`

This boolean field controls whether or not stack traces caused by errors are
trimmed to remove internal stack frames. This includes frames from the Dart core
libraries, the [`stack_trace`][stack_trace] package, and the `test` package
itself. It defaults to `false`.

[stack_trace]: https://pub.dev/packages/stack_trace

```yaml
verbose_trace: true
```

### `chain_stack_traces`

This boolean field controls whether or not stack traces are chained.
Disabling [`stack trace chaining`][stack trace chaining] will improve
performance for heavily asynchronous code at the cost of debuggability.

[stack trace chaining]: https://github.com/dart-lang/stack_trace/blob/master/README.md#stack-chains

```yaml
chain_stack_traces: false
```

### `js_trace`

This boolean field controls whether or not stack traces caused by errors that
occur while running Dart compiled to JS are converted back to Dart style. This
conversion uses the source map generated by `dart2js` to approximate the
original Dart line, column, and in some cases member name for each stack frame.
It defaults to `false`.

```yaml
js_trace: true
```

### `skip`

This field controls whether or not tests are skipped. It's usually applied to
[specific tags](#configuring-tags) rather than used at the top level. Like the
`skip` parameter for [`test()`][test], it can either be a boolean indicating
whether the tests are skipped or a string indicating the reason they're skipped.

[test]: https://pub.dev/documentation/test_api/latest/test_api/test.html

```yaml
tags:
  chrome:
    skip: "Our Chrome launcher is busted. See issue 1234."
```

This field is not supported in the
[global configuration file](#global-configuration).

### `retry`

This int field controls how many times a test is retried upon failure.

```yaml
tags:
  chrome:
    retry: 3 # Retry chrome failures 3 times.
```

This field is not supported in the
[global configuration file](#global-configuration).

### `test_on`

This field declares which platforms a test supports. It takes a
[platform selector][platform selectors] and only allows tests to run on
platforms that match the selector. It's often used with
[specific tags](#configuring-tags) to ensure that certain features will only be
tested on supported platforms.

[platform selectors]: https://github.com/dart-lang/test/tree/master/pkgs/test#platform-selectors

```yaml
tags:
  # Internet Explorer doesn't support promises yet.
  promises: {test_on: "browser && !ie"}
```

The field can also be used at the top level of the configuration file to
indicate that the entire package only supports a particular platform. If someone
tries to run the tests on an unsupported platform, the runner will print a
warning and skip that platform.

```yaml
# This package uses dart:io.
test_on: vm
```

This field is not supported in the
[global configuration file](#global-configuration).

### `allow_test_randomization`

This can be used to disable test randomization for certain tests, regardless
of the `--test-randomize-ordering-seed` configuration.

This is typically useful when a subset of your tests are order dependent, but
you want to run the other ones with randomized ordering.

```yaml
tags:
  doNotRandomize:
    allow_test_randomization: false
```

### `allow_duplicate_test_names`

This can be used to allow multiple tests in the same suite to have the same
name. This is disabled by default because it is usually an indication of a
mistake, and it causes problems with IDE integrations which run tests by name.

It can be disabled for all tests:

```yaml
allow_duplicate_test_names: true
```

Or for tagged tests only (useful if migrating to enable this):

```yaml
tags:
  allowDuplicates:
    allow_duplicate_test_names: true
```

It cannot be globally enabled or configured on the command line - this would
make it so that tests might pass on one users machine but not anothers which
should be avoided.

## Runner Configuration

Unlike [test configuration](#test-configuration), runner configuration affects
the test runner as a whole rather than individual tests. It can only be used at
the top level of the configuration file.

### `include`

This field loads another configuration file. It's useful for repositories that
contain multiple packages and want to share configuration among them. It takes a
(usually relative) `file:` URL.

If you have a repository with the following structure:

```
repo/
  dart_test_base.yaml
  package/
    test/
    dart_test.yaml
    pubspec.yaml
```

```yaml
# repo/dart_test_base.yaml
filename: "test_*.dart"
```

```yaml
# repo/package/dart_test.yaml
include: ../dart_test_base.yaml
```

...tests in the `package` directory will use configuration from both
`dart_test_base.yaml` and `dart_test.yaml`.

The local configuration file's fields take precedence over those from an
included file, so it's possible to override a base configuration.

### `paths`

This field indicates the default paths that the test runner should run. These
paths are usually directories, although single filenames may be used as well.
Paths must be relative, and they must be in URL format so that they're
compatible across operating systems. This defaults to `[test]`.

```yaml
paths: [dart/test]

paths:
- test/instantaneous
- test/fast
- test/middling
```

This field is not supported in the
[global configuration file](#global-configuration).

### `filename`

This field indicates the filename pattern that the test runner uses to find test
files in directories. All files in directories passed on the command line (or in
directories in [`paths`](#paths), if none are passed) whose basenames match this
pattern will be loaded and run as tests.

This supports the full [glob syntax][]. However, since it's only compared
against a path's basename, path separators aren't especially useful. It defaults
to `"*_test.dart"`.

```yaml
filename: "test_*.dart"
```

[glob syntax]: https://github.com/dart-lang/glob#syntax

This field is not supported in the
[global configuration file](#global-configuration).

### `names`

This field causes the runner to only run tests whose names match the given
regular expressions. A test's name must match *all* regular expressions in
`names`, as well as containing all strings in [`plain_names`](#plain_names), in
order to be run.

This is usually used in a [preset](#configuration-presets) to make it possible
to quickly select a given set of tests.

```yaml
presets:
  # Pass "-P chrome" to run only Chrome tests.
  chrome:
    names:
    - "^browser:"
    - "[Cc]hrome"
```

This field is not supported in the
[global configuration file](#global-configuration).

### `plain_names`

This field causes the runner to only run tests whose names contain the given
strings. A test's name must contain *all* strings in `plain_names`, as well as
matching all regular expressions in [`names`](#names), in order to be run.

This is usually used in a [preset](#configuration-presets) to make it possible
to quickly select a given set of tests.

```yaml
presets:
  # Pass "-P ie" to run only Internet Explorer tests.
  ie:
    plain_names:
    - "IE"
    - "Internet Explorer"
```

This field is not supported in the
[global configuration file](#global-configuration).

### `include_tags`

This field causes the runner to only run tests whose tags match the given
[boolean selector][]. If both `include_tags` and [`exclude_tags`](#exclude_tags)
are used, the exclusions take precedence.

[boolean selector]: https://github.com/dart-lang/boolean_selector/blob/master/README.md

This is usually used in a [preset](#configuration-preset) to make it possible to
quickly select a set of tests.

```yaml
presets:
  # Pass "-P windowless" to run tests that don't open browser windows.
  windowless:
    include_tags: !browser || content-shell
```

This field is not supported in the
[global configuration file](#global-configuration).

### `exclude_tags`

This field causes the runner not to run tests whose tags match the given
[boolean selector][]. If both [`include_tags`](#include_tags) and `exclude_tags`
are used, the exclusions take precedence.

This is usually used in a [preset](#configuration-preset) to make it possible to
quickly select a set of tests.

```yaml
presets:
  # Pass "-P windowless" to run tests that don't open browser windows.
  windowless:
    exclude_tags: browser && !content-shell
```

This field is not supported in the
[global configuration file](#global-configuration).

### `platforms`

This field indicates which platforms tests should run on by default. It allows
the same platform identifiers that can be passed to `--platform`. If multiple
platforms are included, the test runner will default to running tests on all of
them. This defaults to `[vm]`.

```yaml
platforms: [chrome]

platforms:
- chrome
- firefox
```

### `compilers`

This field indicates which compilers tests should be compiled with by default.
It allows the same compiler selectors that can be passed to `--compiler`. If
a given platform has no supported compiler configured, it will use its default.

```yaml
compilers: [source]

compilers:
- source
```

### `concurrency`

This field indicates the default number of test suites to run in parallel. More
parallelism can improve the overall speed of running tests up to a point, but
eventually it just adds more memory overhead without any performance gain. This
defaults to approximately half the number of processors on the current machine.
If it's set to 1, only one test suite will run at a time.

```yaml
concurrency: 3
```

### `pause_after_load`

This field indicates that the test runner should pause for debugging after each
test suite is loaded but before its tests are executed. If it's set,
[`concurrency`](#concurrency) is automatically set to 1 and
[`timeout`](#timeout) is automatically set to `none`.

This is usually used in a [preset](#configuration-presets).

```yaml
presets:
  # Pass "-P debug" to enable debugging configuration
  debug:
    pause_after_load: true
    exclude_tags: undebuggable
    reporter: expanded
```

### `run_skipped`

This field indicates that the test runner should run tests even if they're
marked as skipped.

This is usually used in a [preset](#configuration-presets).

```yaml
presets:
  # Pass "-P all" to run all tests
  debug:
    run_skipped: true
    paths: ["test/", "extra_test/"]
```

### `pub_serve`

This field indicates that the test runner should run against a `pub serve`
instance by default, and provides the port number for that instance. Note that
if there is no `pub serve` instance running at that port, running the tests will
fail by default.

```yaml
pub_serve: 8081
```

### `reporter`

This field indicates the default reporter to use. It may be set to "compact",
"expanded", or "json" (although why anyone would want to default to JSON is
beyond me). It defaults to "expanded" on Windows and "compact" everywhere else.

```yaml
reporter: expanded
```

This field is not supported in the
[global configuration file](#global-configuration).

### `file_reporters`

This field specifies additional reporters to use that will write their output to
a file rather than stdout. It should be a map of reporter names to filepaths.

```yaml
file_reporters:
  json: reports/tests.json
```

This field is not supported in the
[global configuration file](#global-configuration).

### `fold_stack_frames`

This field controls which packages' stack frames will be folded away
when displaying stack traces. Packages contained in the `except`
option will be folded. If `only` is provided, all packages not
contained in this list will be folded. By default,
frames from the `test` package and the `stream_channel`
package are folded.

```yaml
fold_stack_frames:
  except:
  - test
  - stream_channel
```

Sample stack trace, note the absence of `package:test`
and `package:stream_channel`:
```
test/sample_test.dart 7:5   main.<fn>
===== asynchronous gap ===========================
dart:async                  _Completer.completeError
test/sample_test.dart 8:3   main.<fn>
===== asynchronous gap ===========================
dart:async                  _asyncThenWrapperHelper
test/sample_test.dart 5:27  main.<fn>
```

### `custom_html_template_path`

This field specifies the path of the HTML template file to be used for tests run in an HTML environment.
Any HTML file that is named the same as the test and in the same directory will take precedence over the template.
For more information about the usage of this option see [Providing a custom HTML template](https://github.com/dart-lang/test/blob/master/README.md#providing-a-custom-html-template)

## Configuring Tags

### `tags`

The `tag` field can be used to apply [test configuration](#test-configuration)
to all tests [with a given tag][tagging tests] or set of tags. It takes a map
from tag selectors to configuration maps. These configuration maps are just like
the top level of the configuration file, except that they may not contain
[runner configuration](#runner-configuration).

[tagging tests]: https://github.com/dart-lang/test/blob/master/README.md#tagging-tests

```yaml
tags:
  # Integration tests need more time to run.
  integration:
    timeout: 1m
```

Tags may also have no configuration associated with them. The test runner prints
a warning whenever it encounters a tag it doesn't recognize, and this the best
way to tell it that a tag exists.

```yaml
# We occasionally want to use --tags or --exclude-tags on these tags.
tags:
  # A test that spawns a browser.
  browser:

  # A test that needs Ruby installed.
  ruby:
```

You can also use [boolean selector syntax][] to define configuration that
involves multiple tags. For example:

[boolean selector syntax]: https://github.com/dart-lang/boolean_selector/blob/master/README.md

```yaml
tags:
  # Tests that invoke sub-processes tend to be a little slower.
  ruby || python:
    timeout: 1.5x
```

Tag configuration is applied at whatever level the tag appears—so if a group is
tagged as `integration`, its timeout will take precedence over the suite's
timeout but not any tests'. If the group itself had a timeout declared, the
group's explicit timeout would take precedence over the tag.

If multiple tags appear at the same level, and they have conflicting
configurations, the test runner *does not guarantee* what order they'll be
resolved in. In practice, conflicting configuration is pretty unlikely and it's
easy to just explicitly specify what you want on the test itself.

This field counts as [test configuration](#test-configuration). It is not
supported in the [global configuration file](#global-configuration).

### `add_tags`

This field adds additional tags. It's technically
[test configuration](#test-configuration), but it's usually used in more
specific contexts. For example, when included in a tag's configuration, it can
be used to enable tag inheritance, where adding one tag implicitly adds another
as well. It takes a list of tag name strings.

```yaml
tags:
  # Any test that spawns a browser.
  browser:
    timeout: 2x

  # Tests that spawn specific browsers. These automatically get the browser tag
  # as well.
  chrome: {add_tags: [browser]}
  firefox: {add_tags: [browser]}
  safari: {add_tags: [browser]}
  ie: {add_tags: [browser]}
```

This field is not supported in the
[global configuration file](#global-configuration).

## Configuring Platforms

There are two different kinds of platform configuration.
[Operating system configuration](#on_os) cares about the operating system on
which test runner is running. It sets global configuration for the runner on
particular OSes. [Test platform configuration](#on_platform), on the other hand,
cares about the platform the *test* is running on (like the
[`@OnPlatform` annotation][@OnPlatform]). It sets configuration for particular
tests that are running on matching platforms.

[@OnPlatform]: https://github.com/dart-lang/test/tree/master/pkgs/test#platform-specific-configuration

### `on_os`

This field applies configuration when specific operating systems are being used.
It takes a map from operating system identifiers (the same ones that are used in
[platform selectors][]) to configuration maps that are applied on those
operating systems. These configuration maps are just like the top level of the
configuration file, and allow any fields that may be used in the context where
`on_os` was used.

```yaml
on_os:
  windows:
    # Both of these are the defaults anyway, but let's be explicit about it.
    color: false
    runner: expanded

    # My Windows machine is real slow.
    timeout: 2x

  # My Linux machine has SO MUCH RAM.
  linux:
    concurrency: 500
```

This field counts as [test configuration](#test-configuration). If it's used in
a context that only allows test configuration, it may only contain test
configuration.

### `on_platform`

This field applies configuration to tests that are run on specific platforms. It
takes a map from [platform selectors][] to configuration maps that are applied
to tests run on those platforms. These configuration maps are just like the top
level of the configuration file, except that they may not contain
[runner configuration](#runner-configuration).

```yaml
# Our code is kind of slow on Blink and WebKit.
on_platform:
  chrome || safari: {timeout: 2x}
```

**Note**: operating system names that appear in `on_platform` refer to tests
that are run on the Dart VM under that operating system. To configure all tests
when running on a particular operating system, use [`on_os`](#on_os) instead.

This field counts as [test configuration](#test-configuration).

### `override_platforms`

This field allows you to customize the settings for built-in test platforms. It
takes a map from platform identifiers to settings for those platforms. For example:

```yaml
override_platforms:
  chrome:
    # The settings to override for this platform.
    settings:
      executable: chromium
```

This tells the test runner to use the `chromium` executable for Chrome tests. It
calls that executable with the same logic and flags it normally uses for Chrome.

Each platform can define exactly which settings it supports. All browsers and
Node.js support [the same settings](#browser-and-node-js-settings), but the VM
doesn't support any settings and so can't be overridden.

### `define_platforms`

You can define new platforms in terms of old ones using the `define_platforms`
field. This lets you define variants of existing platforms without overriding
the old ones. This field takes a map from the new platform identifiers to
definitions for those platforms. For example:

```yaml
define_platforms:
  # This identifier is used to select the platform with the --platform flag.
  chromium:
    # A human-friendly name for the platform.
    name: Chromium

    # The identifier for the platform that this is based on.
    extends: chrome

    # Settings for the new child platform.
    settings:
      executable: chromium
```

Once this is defined, you can run `dart test -p chromium` and it will run
those tests in the Chromium browser, using the same logic it normally uses for
Chrome. You can even use `chromium` in platform selectors; for example, you
might pass `testOn: "chromium"` to declare that a test is Chromium-specific.
User-defined platforms also count as their parents, so Chromium will run tests
that say `testOn: "chrome"` as well.

Each platform can define exactly which settings it supports. All browsers and
Node.js support [the same settings](#browser-and-node-js-settings), but the VM
doesn't support any settings and so can't be extended.

This field is not supported in the
[global configuration file](#global-configuration).

### Browser and Node.js Settings

All built-in browser platforms, as well as the built-in Node.js platform,
provide the same settings that can be set using
[`define_platforms`](#define_platforms), which control how their executables are
invoked.

#### `arguments`

The `arguments` field provides extra arguments to the executable. It takes a
string, and parses it in the same way as the POSIX shell:

```yaml
override_platforms:
  firefox:
    settings:
      arguments: -headless
```

#### `executable`

The `executable` field tells the test runner where to look for the executable to
use to start the subprocess. It has three sub-keys, one for each supported
operating system, which each take a path or an executable name:

```yaml
define_platforms:
  chromium:
    name: Chromium
    extends: chrome

    settings:
      executable:
        linux: chromium
        mac_os: /Applications/Chromium.app/Contents/MacOS/Chromium
        windows: Chromium\Application\chrome.exe
```

Executables can be defined in three ways:

* As a plain basename, with no path separators. These executables are passed
  directly to the OS, which looks them up using the `PATH` environment variable.

* As an absolute path, which is used as-is.

* **Only on Windows**, as a relative path. The test runner will look up this
  path relative to the `LOCALAPPATA`, `PROGRAMFILES`, and `PROGRAMFILES(X86)`
  environment variables, in that order.

If a platform is omitted, it defaults to using the built-in executable location.

As a shorthand, you can also define the same executable for all operating
systems:

```yaml
define_platforms:
  chromium:
    name: Chromium
    extends: chrome

    settings:
      executable: chromium
```

#### `headless`

The `headless` field says whether or not to run the browser in headless mode.
It defaults to `true`. It's currently only supported for Chrome:

```yaml
override_platforms:
  chrome:
    settings:
      headless: false
```

Note that headless mode is always disabled when debugging.

## Configuration Presets

*Presets* are collections of configuration that can be explicitly selected on
the command-line. They're useful for quickly selecting options that are
frequently used together, for providing special configuration for continuous
integration systems, and for defining more complex logic than can be expressed
directly using command-line arguments.

Presets can be selected on the command line using the `--preset` or `-P` flag.
Any number of presets can be selected this way; if they conflict, the last one
selected wins. Only presets that are defined in the configuration file may be
selected.

### `presets`

This field defines which presets are available to select. It takes a map from
preset names to configuration maps that are applied when those presets are
selected. These configuration maps are just like the top level of the
configuration file, and allow any fields that may be used in the context where
`presets` was used.

```yaml
presets:
  # Use this when you need completely un-munged stack traces.
  debug:
    verbose_trace: false
    js_trace: true

  # Shortcut for running only browser tests.
  browser:
    paths:
    - test/runner/browser
    - test/runner/pub_serve_test.dart
```

The `presets` field counts as [test configuration](#test-configuration). It can
be useful to use it in combination with other fields for advanced preset
behavior.

```yaml
tags:
  chrome:
    skip: "Our Chrome launcher is busted. See issue 1234."

    # Pass -P force to verify that the launcher is still busted.
    presets: {force: {skip: false}}
```

### `add_presets`

This field selects additional presets. It's technically
[runner configuration](#runner-configuration), but it's usually used in more
specific contexts. For example, when included in a preset's configuration, it
can be used to enable preset inheritance, where selecting one preset implicitly
selects another as well. It takes a list of preset name strings.

```yaml
presets:
  # Shortcut for running only browser tests.
  browser:
    paths: [test/runner/browser]

  # Shortcut for running only Chrome tests.
  chrome:
    filename: "chrome_*_test.dart"
    add_presets: [browser]
```

## Global Configuration

The test runner also supports a global configuration file. On Windows, this
file's local defaults to `%LOCALAPPDATA%\DartTest.yaml`. On Unix, it defaults to
`~/.dart_test.yaml`. It can also be explicitly set using the `DART_TEST_CONFIG`
environment variable.

The global configuration file supports a subset of the fields supported by the
package-specific configuration file. In general, it doesn't support fields that
are closely tied to the structure of an individual package. Fields that are not
supported in the global configuration file say so in their documentation.