Each package may include a configuration file that applies to the package as a whole. This file can be used to provide custom defaults for various options, to define configuration for multiple files, and more.

The file is named dart_test.yaml and lives at the root of the package, next to the package‘s pubspec. Like the pubspec, it’s a YAML file. Here's an example:

# This package's tests are very slow. Double the default timeout.
timeout: 2x

# This is a browser-only package, so test on content shell by default.
platforms: [content-shell]

tags:
  # Integration tests are even slower, so increase the timeout again.
  integration: {timeout: 2x}

  # Sanity tests are quick and verify that nothing is obviously wrong. Declaring
  # the tag allows us to tag tests with it without getting warnings.
  sanity:

Test Configuration

There are two major categories of configuration field: “test” and “runner”. Test configuration controls how individual tests run, while runner configuration controls the test runner as a whole. Both types of fields may be used at the top level of a configuration file. However, because different tests can have different test configuration, only test configuration fields may be used to configure tags or platforms.

timeout

This field indicates how much time the test runner should allow a test to remain inactive before it considers that test to have failed. It has three possible formats:

  • The string “none” indicates that tests should never time out.

  • A number followed by a unit abbreviation indicates an exact time. For example, “1m” means a timeout of one minute, and “30s” means a timeout of thirty seconds. Multiple numbers can be combined, as in “1m 30s”.

  • A number followed by “x” indicates a multiple. This is applied to the default value of 30s.

timeout: 1m

verbose_trace

This boolean field controls whether or not stack traces caused by errors are trimmed to remove internal stack frames. This includes frames from the Dart core libraries, the stack_trace package, and the test package itself. It defaults to false.

verbose_trace: true

chain_stack_traces

This boolean field controls whether or not stack traces are chained. Disabling stack trace chaining will improve performance for heavily asynchronous code at the cost of debuggability.

chain_stack_traces: false

js_trace

This boolean field controls whether or not stack traces caused by errors that occur while running Dart compiled to JS are converted back to Dart style. This conversion uses the source map generated by dart2js to approximate the original Dart line, column, and in some cases member name for each stack frame. It defaults to false.

js_trace: true

skip

This field controls whether or not tests are skipped. It‘s usually applied to specific tags rather than used at the top level. Like the skip parameter for test(), it can either be a boolean indicating whether the tests are skipped or a string indicating the reason they’re skipped.

tags:
  chrome:
    skip: "Our Chrome launcher is busted. See issue 1234."

This field is not supported in the global configuration file.

retry

This int field controls how many times a test is retried upon failure.

tags:
  chrome:
    retry: 3 # Retry chrome failures 3 times.

This field is not supported in the global configuration file.

test_on

This field declares which platforms a test supports. It takes a platform selector and only allows tests to run on platforms that match the selector. It's often used with specific tags to ensure that certain features will only be tested on supported platforms.

tags:
  # Internet Explorer doesn't support promises yet.
  promises: {test_on: "browser && !ie"}

The field can also be used at the top level of the configuration file to indicate that the entire package only supports a particular platform. If someone tries to run the tests on an unsupported platform, the runner will print a warning and skip that platform.

# This package uses dart:io.
test_on: vm

This field is not supported in the global configuration file.

Runner Configuration

Unlike test configuration, runner configuration affects the test runner as a whole rather than individual tests. It can only be used at the top level of the configuration file.

paths

This field indicates the default paths that the test runner should run. These paths are usually directories, although single filenames may be used as well. Paths must be relative, and they must be in URL format so that they're compatible across operating systems. This defaults to [test].

paths: [dart/test]

paths:
- test/instantaneous
- test/fast
- test/middling

This field is not supported in the global configuration file.

filename

This field indicates the filename pattern that the test runner uses to find test files in directories. All files in directories passed on the command line (or in directories in paths, if none are passed) whose basenames match this pattern will be loaded and run as tests.

This supports the full glob syntax. However, since it‘s only compared against a path’s basename, path separators aren't especially useful. It defaults to "*_test.dart".

filename: "test_*.dart"

This field is not supported in the global configuration file.

names

This field causes the runner to only run tests whose names match the given regular expressions. A test's name must match all regular expressions in names, as well as containing all strings in plain_names, in order to be run.

This is usually used in a preset to make it possible to quickly select a given set of tests.

presets:
  # Pass "-P chrome" to run only Chrome tests.
  chrome:
    names:
    - "^browser:"
    - "[Cc]hrome"

This field is not supported in the global configuration file.

plain_names

This field causes the runner to only run tests whose names contain the given strings. A test's name must contain all strings in plain_names, as well as matching all regular expressions in names, in order to be run.

This is usually used in a preset to make it possible to quickly select a given set of tests.

presets:
  # Pass "-P ie" to run only Internet Explorer tests.
  ie:
    plain_names:
    - "IE"
    - "Internet Explorer"

This field is not supported in the global configuration file.

include_tags

This field causes the runner to only run tests whose tags match the given boolean selector. If both include_tags and exclude_tags are used, the exclusions take precedence.

This is usually used in a preset to make it possible to quickly select a set of tests.

presets:
  # Pass "-P windowless" to run tests that don't open browser windows.
  windowless:
    include_tags: !browser || content-shell

This field is not supported in the global configuration file.

exclude_tags

This field causes the runner not to run tests whose tags match the given boolean selector. If both include_tags and exclude_tags are used, the exclusions take precedence.

This is usually used in a preset to make it possible to quickly select a set of tests.

presets:
  # Pass "-P windowless" to run tests that don't open browser windows.
  windowless:
    exclude_tags: browser && !content-shell

This field is not supported in the global configuration file.

platforms

This field indicates which platforms tests should run on by default. It allows the same platform identifiers that can be passed to --platform. If multiple platforms are included, the test runner will default to running tests on all of them. This defaults to [vm].

platforms: [content_shell]

platforms:
- chrome
- firefox

concurrency

This field indicates the default number of test suites to run in parallel. More parallelism can improve the overall speed of running tests up to a point, but eventually it just adds more memory overhead without any performance gain. This defaults to approximately half the number of processors on the current machine. If it's set to 1, only one test suite will run at a time.

concurrency: 3

pause_after_load

This field indicates that the test runner should pause for debugging after each test suite is loaded but before its tests are executed. If it's set, concurrency is automatically set to 1 and timeout is automatically set to none.

This is usually used in a preset.

presets:
  # Pass "-P debug" to enable debugging configuration
  debug:
    pause_after_load: true
    exclude_tags: undebuggable
    reporter: expanded

run_skipped

This field indicates that the test runner should run tests even if they're marked as skipped.

This is usually used in a preset.

presets:
  # Pass "-P all" to run all tests
  debug:
    run_skipped: true
    paths: ["test/", "extra_test/"]

pub_serve

This field indicates that the test runner should run against a pub serve instance by default, and provides the port number for that instance. Note that if there is no pub serve instance running at that port, running the tests will fail by default.

pub_serve: 8081

reporter

This field indicates the default reporter to use. It may be set to “compact”, “expanded”, or “json” (although why anyone would want to default to JSON is beyond me). It defaults to “expanded” on Windows and “compact” everywhere else.

reporter: expanded

This field is not supported in the global configuration file.

fold_stack_frames

This field controls which packages' stack frames will be folded away when displaying stack traces. Packages contained in the except option will be folded. If only is provided, all packages not contained in this list will be folded. By default, frames from the test package and the stream_channel package are folded.

fold_stack_frames:
  except:
  - test 
  - stream_channel

Sample stack trace, note the absence of package:test and package:stream_channel:

test/sample_test.dart 7:5   main.<fn>
===== asynchronous gap ===========================
dart:async                  _Completer.completeError
test/sample_test.dart 8:3   main.<fn>
===== asynchronous gap ===========================
dart:async                  _asyncThenWrapperHelper
test/sample_test.dart 5:27  main.<fn>

Configuring Tags

tags

The tag field can be used to apply test configuration to all tests with a given tag or set of tags. It takes a map from tag selectors to configuration maps. These configuration maps are just like the top level of the configuration file, except that they may not contain runner configuration.

tags:
  # Integration tests need more time to run.
  integration:
    timeout: 1m

Tags may also have no configuration associated with them. The test runner prints a warning whenever it encounters a tag it doesn't recognize, and this the best way to tell it that a tag exists.

# We occasionally want to use --tags or --exclude-tags on these tags.
tags:
  # A test that spawns a browser.
  browser:

  # A test that needs Ruby installed.
  ruby:

You can also use boolean selector syntax to define configuration that involves multiple tags. For example:

tags:
  # Tests that invoke sub-processes tend to be a little slower.
  ruby || python:
    timeout: 1.5x

Tag configuration is applied at whatever level the tag appears—so if a group is tagged as integration, its timeout will take precedence over the suite‘s timeout but not any tests’. If the group itself had a timeout declared, the group's explicit timeout would take precedence over the tag.

If multiple tags appear at the same level, and they have conflicting configurations, the test runner does not guarantee what order they‘ll be resolved in. In practice, conflicting configuration is pretty unlikely and it’s easy to just explicitly specify what you want on the test itself.

This field counts as test configuration. It is not supported in the global configuration file.

add_tags

This field adds additional tags. It‘s technically test configuration, but it’s usually used in more specific contexts. For example, when included in a tag's configuration, it can be used to enable tag inheritance, where adding one tag implicitly adds another as well. It takes a list of tag name strings.

tags:
  # Any test that spawns a browser.
  browser:
    timeout: 2x

  # Tests that spawn specific browsers. These automatically get the browser tag
  # as well.
  chrome: {add_tags: [browser]}
  firefox: {add_tags: [browser]}
  safari: {add_tags: [browser]}
  ie: {add_tags: [browser]}

This field is not supported in the global configuration file.

Configuring Platforms

There are two different kinds of platform configuration. Operating system configuration cares about the operating system on which test runner is running. It sets global configuration for the runner on particular OSes. Test platform configuration, on the other hand, cares about the platform the test is running on (like the @OnPlatform annotation). It sets configuration for particular tests that are running on matching platforms.

on_os

This field applies configuration when specific operating systems are being used. It takes a map from operating system identifiers (the same ones that are used in platform selectors) to configuration maps that are applied on those operating systems. These configuration maps are just like the top level of the configuration file, and allow any fields that may be used in the context where on_os was used.

on_os:
  windows:
    # Both of these are the defaults anyway, but let's be explicit about it.
    color: false
    runner: expanded

    # My Windows machine is real slow.
    timeout: 2x

  # My Linux machine has SO MUCH RAM.
  linux:
    concurrency: 500

This field counts as test configuration. If it's used in a context that only allows test configuration, it may only contain test configuration.

on_platform

This field applies configuration to tests that are run on specific platforms. It takes a map from platform selectors to configuration maps that are applied to tests run on those platforms. These configuration maps are just like the top level of the configuration file, except that they may not contain runner configuration.

# Our code is kind of slow on Blink and WebKit.
on_platform:
  chrome || safari: {timeout: 2x}

Note: operating system names that appear in on_platform refer to tests that are run on the Dart VM under that operating system. To configure all tests when running on a particular operating system, use on_os instead.

This field counts as test configuration.

override_platforms

This field allows you to customize the settings for built-in test platforms. It takes a map from platform identifiers to settings for those platforms. For example:

override_platforms:
  chrome:
    # The settings to override for this platform.
    settings:
      executable: chromium

This tells the test runner to use the chromium executable for Chrome tests. It calls that executable with the same logic and flags it normally uses for Chrome.

Each platform can define exactly which settings it supports. All browsers and Node.js support the same settings, but the VM doesn‘t support any settings and so can’t be overridden.

define_platforms

You can define new platforms in terms of old ones using the define_platforms field. This lets you define variants of existing platforms without overriding the old ones. This field takes a map from the new platform identifiers to definitions for those platforms. For example:

define_platforms:
  # This identifier is used to select the platform with the --platform flag.
  chromium:
    # A human-friendly name for the platform.
    name: Chromium

    # The identifier for the platform that this is based on.
    extends: chrome

    # Settings for the new child platform.
    settings:
      executable: chromium

Once this is defined, you can run pub run test -p chromium and it will run those tests in the Chromium browser, using the same logic it normally uses for Chrome. You can even use chromium in platform selectors; for example, you might pass testOn: "chromium" to declare that a test is Chromium-specific. User-defined platforms also count as their parents, so Chromium will run tests that say testOn: "chrome" as well.

Each platform can define exactly which settings it supports. All browsers and Node.js support the same settings, but the VM doesn‘t support any settings and so can’t be extended.

This field is not supported in the global configuration file.

Browser and Node.js Settings

All built-in browser platforms, as well as the built-in Node.js platform, provide the same settings that can be set using define_platforms, which control how their executables are invoked.

arguments

The arguments field provides extra arguments to the executable. It takes a string, and parses it in the same way as the POSIX shell:

override_platforms:
  firefox:
    settings:
      arguments: -headless

executable

The executable field tells the test runner where to look for the executable to use to start the subprocess. It has three sub-keys, one for each supported operating system, which each take a path or an executable name:

define_platforms:
  chromium:
    name: Chromium
    extends: chrome

    settings:
      executable:
        linux: chromium
        mac_os: /Applications/Chromium.app/Contents/MacOS/Chromium
        windows: Chromium\Application\chrome.exe

Executables can be defined in three ways:

  • As a plain basename, with no path separators. These executables are passed directly to the OS, which looks them up using the PATH environment variable.

  • As an absolute path, which is used as-is.

  • Only on Windows, as a relative path. The test runner will look up this path relative to the LOCALAPPATA, PROGRAMFILES, and PROGRAMFILES(X86) environment variables, in that order.

If a platform is omitted, it defaults to using the built-in executable location.

As a shorthand, you can also define the same executable for all operating systems:

define_platforms:
  chromium:
    name: Chromium
    extends: chrome

    settings:
      executable: chromium

Configuration Presets

Presets are collections of configuration that can be explicitly selected on the command-line. They're useful for quickly selecting options that are frequently used together, for providing special configuration for continuous integration systems, and for defining more complex logic than can be expressed directly using command-line arguments.

Presets can be selected on the command line using the --preset or -P flag. Any number of presets can be selected this way; if they conflict, the last one selected wins. Only presets that are defined in the configuration file may be selected.

presets

This field defines which presets are available to select. It takes a map from preset names to configuration maps that are applied when those presets are selected. These configuration maps are just like the top level of the configuration file, and allow any fields that may be used in the context where presets was used.

presets:
  # Use this when you need completely un-munged stack traces.
  debug:
    verbose_trace: false
    js_trace: true

  # Shortcut for running only browser tests.
  browser:
    paths:
    - test/runner/browser
    - test/runner/pub_serve_test.dart

The presets field counts as test configuration. It can be useful to use it in combination with other fields for advanced preset behavior.

tags:
  chrome:
    skip: "Our Chrome launcher is busted. See issue 1234."

    # Pass -P force to verify that the launcher is still busted.
    presets: {force: {skip: false}}

add_presets

This field selects additional presets. It‘s technically runner configuration, but it’s usually used in more specific contexts. For example, when included in a preset's configuration, it can be used to enable preset inheritance, where selecting one preset implicitly selects another as well. It takes a list of preset name strings.

presets:
  # Shortcut for running only browser tests.
  browser:
    paths: [test/runner/browser]

  # Shortcut for running only Chrome tests.
  chrome:
    filename: "chrome_*_test.dart"
    add_presets: [browser]

Global Configuration

The test runner also supports a global configuration file. On Windows, this file's local defaults to %LOCALAPPDATA%\DartTest.yaml. On Unix, it defaults to ~/.dart_test.yaml. It can also be explicitly set using the DART_TEST_CONFIG environment variable.

The global configuration file supports a subset of the fields supported by the package-specific configuration file. In general, it doesn't support fields that are closely tied to the structure of an individual package. Fields that are not supported in the global configuration file say so in their documentation.