Add support for configuration presets.
Closes #83
R=kevmoo@google.com
Review URL: https://codereview.chromium.org//1782473005 .
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f4f6e12..36b1fe5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,10 @@
## 0.12.12
+* Add support for [test presets][]. These are defined using the `presets` field
+ in the package configuration file. They can be selected by passing `--preset`
+ or `-P`, or by using the `add_presets` field in the package configuration
+ file.
+
* Add an `on_os` field to the package configuration file that allows users to
select different configuration for different operating systems.
@@ -10,6 +15,8 @@
the `test` executable itself is running on iOS, not when it's running browser
tests on an iOS browser.
+[test presets]: https://github.com/dart-lang/test/blob/master/doc/package_config.md#configuration-presets
+
## 0.12.11+2
* Update to `shelf_web_socket` 0.2.0.
diff --git a/doc/package_config.md b/doc/package_config.md
index adf4057..07c4eca 100644
--- a/doc/package_config.md
+++ b/doc/package_config.md
@@ -43,6 +43,9 @@
* [Configuring Platforms](#configuring-platforms)
* [`on_os`](#on_os)
* [`on_platform`](#on_platform)
+* [Configuration Presets](#configuration-presets)
+ * [`presets`](#presets)
+ * [`add_preset`](#add_preset)
## Test Configuration
@@ -288,8 +291,8 @@
This field adds additional tags. It's technically
[test configuration](#test-configuration), but it's usually used in more
specific contexts. For example, when included in a tag's configuration, it can
-be used to enable tag inheritance, where adding one tag implicitly adds other as
-well. It takes a list of tag name strings.
+be used to enable tag inheritance, where adding one tag implicitly adds another
+as well. It takes a list of tag name strings.
```yaml
tags:
@@ -364,3 +367,71 @@
when running on a particular operating system, use [`on_os`](#on_os) instead.
This field counts as [test configuration](#test-configuration).
+
+## Configuration Presets
+
+*Presets* are collections of configuration that can be explicitly selected on
+the command-line. They're useful for quickly selecting options that are
+frequently used together, for providing special configuration for continuous
+integration systems, and for defining more complex logic than can be expressed
+directly using command-line arguments.
+
+Presets can be selected on the command line using the `--preset` or `-P` flag.
+Any number of presets can be selected this way; if they conflict, the last one
+selected wins. Only presets that are defined in the configuration file may be
+selected.
+
+### `presets`
+
+This field defines which presets are available to select. It takes a map from
+preset names to configuration maps that are applied when those presets are
+selected. These configuration maps are just like the top level of the
+configuration file, and allow any fields that may be used in the context where
+`presets` was used.
+
+```yaml
+presets:
+ # Use this when you need completely un-munged stack traces.
+ debug:
+ verbose_trace: false
+ js_trace: true
+
+ # Shortcut for running only browser tests.
+ browser:
+ paths:
+ - test/runner/browser
+ - test/runner/pub_serve_test.dart
+```
+
+The `presets` field counts as [test configuration](#test-configuration). It can
+be useful to use it in combination with other fields for advanced preset
+behavior.
+
+```yaml
+tags:
+ chrome:
+ skip: "Our Chrome launcher is busted. See issue 1234."
+
+ # Pass -P force to verify that the launcher is still busted.
+ presets: {force: {skip: false}}
+```
+
+### `add_presets`
+
+This field selects additional presets. It's technically
+[runner configuration](#runner-configuration), but it's usually used in more
+specific contexts. For example, when included in a preset's configuration, it
+can be used to enable preset inheritance, where selecting one preset implicitly
+selects another as well. It takes a list of preset name strings.
+
+```yaml
+presets:
+ # Shortcut for running only browser tests.
+ browser:
+ paths: [test/runner/browser]
+
+ # Shortcut for running only Chrome tests.
+ chrome:
+ filename: "chrome_*_test.dart"
+ add_presets: [browser]
+```
diff --git a/lib/src/backend/metadata.dart b/lib/src/backend/metadata.dart
index 6ff5412..91bb54d 100644
--- a/lib/src/backend/metadata.dart
+++ b/lib/src/backend/metadata.dart
@@ -140,6 +140,7 @@
// If there's no tag-specific metadata, or if none of it applies, just
// return the metadata as-is.
if (forTag == null || tags == null) return _unresolved();
+ tags = new Set.from(tags);
forTag = new Map.from(forTag);
// Otherwise, resolve the tag-specific components. Doing this eagerly means
diff --git a/lib/src/executable.dart b/lib/src/executable.dart
index 8dd546e..8c82c47 100644
--- a/lib/src/executable.dart
+++ b/lib/src/executable.dart
@@ -104,6 +104,17 @@
return;
}
+ var undefinedPresets =
+ configuration.chosenPresets
+ .where((preset) => !configuration.knownPresets.contains(preset))
+ .toList();
+ if (undefinedPresets.isNotEmpty) {
+ _printUsage("Undefined ${pluralize('preset', undefinedPresets.length)} "
+ "${toSentence(undefinedPresets.map((preset) => '"$preset"'))}.");
+ exitCode = exit_codes.usage;
+ return;
+ }
+
if (configuration.pubServeUrl != null && !_usesTransformer) {
stderr.write('''
When using --pub-serve, you must include the "test/pub_serve" transformer in
@@ -169,7 +180,7 @@
output = stderr;
}
- output.write("""$message
+ output.write("""${wordWrap(message)}
Usage: pub run test:test [files or directories...]
diff --git a/lib/src/runner/configuration.dart b/lib/src/runner/configuration.dart
index aaeb94f..b8e2ed5 100644
--- a/lib/src/runner/configuration.dart
+++ b/lib/src/runner/configuration.dart
@@ -21,6 +21,12 @@
/// A class that encapsulates the command-line configuration of the test runner.
class Configuration {
+ /// An empty configuration with only default values.
+ ///
+ /// Using this is slightly more efficient than manually constructing a new
+ /// configuration with no arguments.
+ static final empty = new Configuration._();
+
/// The usage string for the command-line arguments.
static String get usage => args.usage;
@@ -107,6 +113,14 @@
List<TestPlatform> get platforms => _platforms ?? [TestPlatform.vm];
final List<TestPlatform> _platforms;
+ /// The set of presets to use.
+ ///
+ /// Any chosen presets for the parent configuration are added to the chosen
+ /// preset sets for child configurations as well.
+ ///
+ /// Note that the order of this set matters.
+ final Set<String> chosenPresets;
+
/// Only run tests whose tags match this selector.
///
/// When [merge]d, this is intersected with the other configuration's included
@@ -142,17 +156,21 @@
forTag: mapMap(tags, value: (_, config) => config.metadata),
onPlatform: mapMap(onPlatform, value: (_, config) => config.metadata));
- /// The set of tags that have been declaredin any way in this configuration.
+ /// The set of tags that have been declared in any way in this configuration.
Set<String> get knownTags {
if (_knownTags != null) return _knownTags;
var known = includeTags.variables.toSet()
..addAll(excludeTags.variables)
..addAll(addTags);
- tags.forEach((selector, config) {
+
+ for (var selector in tags.keys) {
known.addAll(selector.variables);
- known.addAll(config.knownTags);
- });
+ }
+
+ for (var configuration in _children) {
+ known.addAll(configuration.knownTags);
+ }
_knownTags = new UnmodifiableSetView(known);
return _knownTags;
@@ -166,6 +184,40 @@
/// configuration fields, but that isn't enforced.
final Map<PlatformSelector, Configuration> onPlatform;
+ /// Configuration presets.
+ ///
+ /// These are configurations that can be explicitly selected by the user via
+ /// the command line. Preset configuration takes precedence over the base
+ /// configuration.
+ ///
+ /// This is guaranteed not to have any keys that match [chosenPresets]; those
+ /// are resolved when the configuration is constructed.
+ final Map<String, Configuration> presets;
+
+ /// All preset names that are known to be valid.
+ ///
+ /// This includes presets that have already been resolved.
+ Set<String> get knownPresets {
+ if (_knownPresets != null) return _knownPresets;
+
+ var known = presets.keys.toSet();
+ for (var configuration in _children) {
+ known.addAll(configuration.knownPresets);
+ }
+
+ _knownPresets = new UnmodifiableSetView(known);
+ return _knownPresets;
+ }
+ Set<String> _knownPresets;
+
+ /// All child configurations of [this] that may be selected under various
+ /// circumstances.
+ Iterable<Configuration> get _children sync* {
+ yield* tags.values;
+ yield* onPlatform.values;
+ yield* presets.values;
+ }
+
/// Parses the configuration from [args].
///
/// Throws a [FormatException] if [args] are invalid.
@@ -177,7 +229,97 @@
/// a [FormatException] if its contents are invalid.
factory Configuration.load(String path) => load(path);
- Configuration({
+ factory Configuration({
+ bool help,
+ bool version,
+ bool verboseTrace,
+ bool jsTrace,
+ bool skip,
+ String skipReason,
+ PlatformSelector testOn,
+ bool pauseAfterLoad,
+ bool color,
+ String packageRoot,
+ String reporter,
+ int pubServePort,
+ int concurrency,
+ Timeout timeout,
+ Pattern pattern,
+ Iterable<TestPlatform> platforms,
+ Iterable<String> paths,
+ Glob filename,
+ Iterable<String> chosenPresets,
+ BooleanSelector includeTags,
+ BooleanSelector excludeTags,
+ Iterable addTags,
+ Map<BooleanSelector, Configuration> tags,
+ Map<PlatformSelector, Configuration> onPlatform,
+ Map<String, Configuration> presets}) {
+ _unresolved() => new Configuration._(
+ help: help,
+ version: version,
+ verboseTrace: verboseTrace,
+ jsTrace: jsTrace,
+ skip: skip,
+ skipReason: skipReason,
+ testOn: testOn,
+ pauseAfterLoad: pauseAfterLoad,
+ color: color,
+ packageRoot: packageRoot,
+ reporter: reporter,
+ pubServePort: pubServePort,
+ concurrency: concurrency,
+ timeout: timeout,
+ pattern: pattern,
+ platforms: platforms,
+ paths: paths,
+ filename: filename,
+ chosenPresets: chosenPresets,
+ includeTags: includeTags,
+ excludeTags: excludeTags,
+ addTags: addTags,
+
+ // Make sure we pass [chosenPresets] to the child configurations as
+ // well. This ensures that
+ tags: _withChosenPresets(tags, chosenPresets),
+ onPlatform: _withChosenPresets(onPlatform, chosenPresets),
+ presets: _withChosenPresets(presets, chosenPresets));
+
+ if (chosenPresets == null) return _unresolved();
+ chosenPresets = new Set.from(chosenPresets);
+
+ if (presets == null) return _unresolved();
+ presets = new Map.from(presets);
+
+ var knownPresets = presets.keys.toSet();
+
+ var merged = chosenPresets.fold(Configuration.empty, (merged, preset) {
+ if (!presets.containsKey(preset)) return merged;
+ return merged.merge(presets.remove(preset));
+ });
+
+ var result = merged == Configuration.empty
+ ? _unresolved()
+ : _unresolved().merge(merged);
+
+ // Make sure the configuration knows about presets that were selected and
+ // thus removed from [presets].
+ result._knownPresets = result.knownPresets.union(knownPresets);
+
+ return result;
+ }
+
+ static Map<Object, Configuration> _withChosenPresets(
+ Map<Object, Configuration> map, Set<String> chosenPresets) {
+ if (map == null || chosenPresets == null) return map;
+ return mapMap(map, value: (_, config) => config.change(
+ chosenPresets: config.chosenPresets.union(chosenPresets)));
+ }
+
+ /// Creates new Configuration.
+ ///
+ /// Unlike [new Configuration], this assumes [presets] is already resolved.
+ Configuration._({
bool help,
bool version,
bool verboseTrace,
@@ -196,11 +338,13 @@
Iterable<TestPlatform> platforms,
Iterable<String> paths,
Glob filename,
+ Iterable<String> chosenPresets,
BooleanSelector includeTags,
BooleanSelector excludeTags,
Iterable addTags,
Map<BooleanSelector, Configuration> tags,
- Map<PlatformSelector, Configuration> onPlatform})
+ Map<PlatformSelector, Configuration> onPlatform,
+ Map<String, Configuration> presets})
: _help = help,
_version = version,
_verboseTrace = verboseTrace,
@@ -221,13 +365,13 @@
_platforms = _list(platforms),
_paths = _list(paths),
_filename = filename,
+ chosenPresets = new Set.from(chosenPresets ?? []),
includeTags = includeTags ?? BooleanSelector.all,
excludeTags = excludeTags ?? BooleanSelector.none,
- addTags = addTags?.toSet() ?? new Set(),
- tags = tags == null ? const {} : new Map.unmodifiable(tags),
- onPlatform = onPlatform == null
- ? const {}
- : new Map.unmodifiable(onPlatform) {
+ addTags = new UnmodifiableSetView(addTags?.toSet() ?? new Set()),
+ tags = _map(tags),
+ onPlatform = _map(onPlatform),
+ presets = _map(presets) {
if (_filename != null && _filename.context.style != p.style) {
throw new ArgumentError(
"filename's context must match the current operating system, was "
@@ -235,24 +379,33 @@
}
}
- /// Returns a [input] as a list or `null`.
+ /// Returns a [input] as an unmodifiable list or `null`.
///
/// If [input] is `null` or empty, this returns `null`. Otherwise, it returns
/// `input.toList()`.
static List _list(Iterable input) {
if (input == null) return null;
- input = input.toList();
+ input = new List.unmodifiable(input);
if (input.isEmpty) return null;
return input;
}
+ /// Returns an modifiable copy of [input] or an empty unmodifiable map.
+ static Map _map(Map input) {
+ if (input == null) return const {};
+ return new Map.unmodifiable(input);
+ }
+
/// Merges this with [other].
///
/// For most fields, if both configurations have values set, [other]'s value
/// takes precedence. However, certain fields are merged together instead.
/// This is indicated in those fields' documentation.
Configuration merge(Configuration other) {
- return new Configuration(
+ if (this == Configuration.empty) return other;
+ if (other == Configuration.empty) return this;
+
+ var result = new Configuration(
help: other._help ?? _help,
version: other._version ?? _version,
verboseTrace: other._verboseTrace ?? _verboseTrace,
@@ -271,12 +424,84 @@
platforms: other._platforms ?? _platforms,
paths: other._paths ?? _paths,
filename: other._filename ?? _filename,
+ chosenPresets: chosenPresets.union(other.chosenPresets),
includeTags: includeTags.intersection(other.includeTags),
excludeTags: excludeTags.union(other.excludeTags),
addTags: other.addTags.union(addTags),
- tags: mergeMaps(tags, other.tags,
- value: (config1, config2) => config1.merge(config2)),
- onPlatform: mergeMaps(onPlatform, other.onPlatform,
- value: (config1, config2) => config1.merge(config2)));
+ tags: _mergeConfigMaps(tags, other.tags),
+ onPlatform: _mergeConfigMaps(onPlatform, other.onPlatform),
+ presets: _mergeConfigMaps(presets, other.presets));
+
+ // Make sure the merged config preserves any presets that were chosen and
+ // discarded.
+ result._knownPresets = knownPresets.union(other.knownPresets);
+ return result;
}
+
+ /// Returns a copy of this configuration with the given fields updated.
+ ///
+ /// Note that unlike [merge], this has no merging behavior—the old value is
+ /// always replaced by the new one.
+ Configuration change({
+ bool help,
+ bool version,
+ bool verboseTrace,
+ bool jsTrace,
+ bool skip,
+ String skipReason,
+ PlatformSelector testOn,
+ bool pauseAfterLoad,
+ bool color,
+ String packageRoot,
+ String reporter,
+ int pubServePort,
+ int concurrency,
+ Timeout timeout,
+ Pattern pattern,
+ Iterable<TestPlatform> platforms,
+ Iterable<String> paths,
+ Glob filename,
+ Iterable<String> chosenPresets,
+ BooleanSelector includeTags,
+ BooleanSelector excludeTags,
+ Iterable addTags,
+ Map<BooleanSelector, Configuration> tags,
+ Map<PlatformSelector, Configuration> onPlatform,
+ Map<String, Configuration> presets}) {
+ return new Configuration(
+ help: help ?? _help,
+ version: version ?? _version,
+ verboseTrace: verboseTrace ?? _verboseTrace,
+ jsTrace: jsTrace ?? _jsTrace,
+ skip: skip ?? _skip,
+ skipReason: skipReason ?? this.skipReason,
+ testOn: testOn ?? this.testOn,
+ pauseAfterLoad: pauseAfterLoad ?? _pauseAfterLoad,
+ color: color ?? _color,
+ packageRoot: packageRoot ?? _packageRoot,
+ reporter: reporter ?? _reporter,
+ pubServePort: pubServePort ?? pubServeUrl?.port,
+ concurrency: concurrency ?? _concurrency,
+ timeout: timeout ?? this.timeout,
+ pattern: pattern ?? this.pattern,
+ platforms: platforms ?? _platforms,
+ paths: paths ?? _paths,
+ filename: filename ?? _filename,
+ chosenPresets: chosenPresets ?? this.chosenPresets,
+ includeTags: includeTags ?? this.includeTags,
+ excludeTags: excludeTags ?? this.excludeTags,
+ addTags: addTags ?? this.addTags,
+ tags: tags ?? this.tags,
+ onPlatform: onPlatform ?? this.onPlatform,
+ presets: presets ?? this.presets);
+ }
+
+ /// Merges two maps whose values are [Configuration]s.
+ ///
+ /// Any overlapping keys in the maps have their configurations merged in the
+ /// returned map.
+ Map<Object, Configuration> _mergeConfigMaps(Map<Object, Configuration> map1,
+ Map<Object, Configuration> map2) =>
+ mergeMaps(map1, map2,
+ value: (config1, config2) => config1.merge(config2));
}
diff --git a/lib/src/runner/configuration/args.dart b/lib/src/runner/configuration/args.dart
index 8ce1d1f..5b96912 100644
--- a/lib/src/runner/configuration/args.dart
+++ b/lib/src/runner/configuration/args.dart
@@ -60,6 +60,10 @@
defaultsTo: 'vm',
allowed: allPlatforms.map((platform) => platform.identifier).toList(),
allowMultiple: true);
+ parser.addOption("preset",
+ abbr: 'P',
+ help: 'The configuration preset(s) to use.',
+ allowMultiple: true);
parser.addOption("concurrency",
abbr: 'j',
help: 'The number of concurrent test suites run.',
@@ -156,6 +160,7 @@
(value) => new Timeout.parse(value)),
pattern: pattern,
platforms: ifParsed('platform')?.map(TestPlatform.find),
+ chosenPresets: ifParsed('preset'),
paths: options.rest.isEmpty ? null : options.rest,
includeTags: includeTags,
excludeTags: excludeTags);
diff --git a/lib/src/runner/configuration/load.dart b/lib/src/runner/configuration/load.dart
index 93cc729..9a9778c 100644
--- a/lib/src/runner/configuration/load.dart
+++ b/lib/src/runner/configuration/load.dart
@@ -27,7 +27,7 @@
var source = new File(path).readAsStringSync();
var document = loadYamlNode(source, sourceUrl: p.toUri(path));
- if (document.value == null) return new Configuration();
+ if (document.value == null) return Configuration.empty;
if (document is! Map) {
throw new SourceSpanFormatException(
@@ -75,14 +75,8 @@
var timeout = _parseValue("timeout", (value) => new Timeout.parse(value));
- var addTags = _getList("add_tags", (tagNode) {
- _validate(tagNode, "Tags must be strings.", (value) => value is String);
- _validate(
- tagNode,
- "Invalid tag. Tags must be (optionally hyphenated) Dart identifiers.",
- (value) => value.contains(anchoredHyphenatedIdentifier));
- return tagNode.value;
- });
+ var addTags = _getList("add_tags",
+ (tagNode) => _parseIdentifierLike(tagNode, "Tag name"));
var tags = _getMap("tags",
key: (keyNode) => _parseNode(keyNode, "tags key",
@@ -108,6 +102,10 @@
keyNode.span, _source);
}, value: (valueNode) => _nestedConfig(valueNode, "on_os value"));
+ var presets = _getMap("presets",
+ key: (keyNode) => _parseIdentifierLike(keyNode, "presets key"),
+ value: (valueNode) => _nestedConfig(valueNode, "presets value"));
+
var config = new Configuration(
verboseTrace: verboseTrace,
jsTrace: jsTrace,
@@ -117,7 +115,8 @@
timeout: timeout,
addTags: addTags,
tags: tags,
- onPlatform: onPlatform);
+ onPlatform: onPlatform,
+ presets: presets);
var osConfig = onOS[currentOS];
return osConfig == null ? config : config.merge(osConfig);
@@ -135,7 +134,8 @@
_disallow("platforms");
_disallow("paths");
_disallow("filename");
- return new Configuration();
+ _disallow("add_presets");
+ return Configuration.empty;
}
var reporter = _getString("reporter");
@@ -166,13 +166,17 @@
var filename = _parseValue("filename", (value) => new Glob(value));
+ var chosenPresets = _getList("add_presets",
+ (presetNode) => _parseIdentifierLike(presetNode, "Preset name"));
+
return new Configuration(
reporter: reporter,
pubServePort: pubServePort,
concurrency: concurrency,
platforms: platforms,
paths: paths,
- filename: filename);
+ filename: filename,
+ chosenPresets: chosenPresets);
}
/// Throws an exception with [message] if [test] returns `false` when passed
@@ -253,6 +257,15 @@
value: (_, valueNode) => value(valueNode));
}
+ String _parseIdentifierLike(YamlNode node, String name) {
+ _validate(node, "$name must be a string.", (value) => value is String);
+ _validate(
+ node,
+ "$name must be an (optionally hyphenated) Dart identifier.",
+ (value) => value.contains(anchoredHyphenatedIdentifier));
+ return node.value;
+ }
+
/// Asserts that [node] is a string, passes its value to [parse], and returns
/// the result.
///
@@ -287,7 +300,7 @@
/// nested configuration. It defaults to [_runnerConfig].
Configuration _nestedConfig(YamlNode node, String name,
{bool runnerConfig}) {
- if (node == null || node.value == null) return new Configuration();
+ if (node == null || node.value == null) return Configuration.empty;
_validate(node, "$name must be a map.", (value) => value is Map);
var loader = new _ConfigurationLoader(node, _source,
diff --git a/pubspec.yaml b/pubspec.yaml
index 8b5bb40..61d16e4 100644
--- a/pubspec.yaml
+++ b/pubspec.yaml
@@ -1,5 +1,5 @@
name: test
-version: 0.12.12-dev
+version: 0.12.12
author: Dart Team <misc@dartlang.org>
description: A library for writing dart unit tests.
homepage: https://github.com/dart-lang/test
diff --git a/test/runner/configuration/configuration_test.dart b/test/runner/configuration/configuration_test.dart
index 366d6c0..8fe9ba6 100644
--- a/test/runner/configuration/configuration_test.dart
+++ b/test/runner/configuration/configuration_test.dart
@@ -229,25 +229,43 @@
});
});
- group("for tags", () {
- test("merges each nested configuration", () {
- var merged = new Configuration(
- tags: {
- new BooleanSelector.parse("foo"):
- new Configuration(verboseTrace: true),
- new BooleanSelector.parse("bar"): new Configuration(jsTrace: true)
- }
- ).merge(new Configuration(
- tags: {
- new BooleanSelector.parse("bar"): new Configuration(jsTrace: false),
- new BooleanSelector.parse("baz"): new Configuration(skip: true)
- }
- ));
+ group("for sets", () {
+ test("if neither is defined, preserves the default", () {
+ var merged = new Configuration().merge(new Configuration());
+ expect(merged.addTags, isEmpty);
+ expect(merged.chosenPresets, isEmpty);
+ });
- expect(merged.tags[new BooleanSelector.parse("foo")].verboseTrace,
- isTrue);
- expect(merged.tags[new BooleanSelector.parse("bar")].jsTrace, isFalse);
- expect(merged.tags[new BooleanSelector.parse("baz")].skip, isTrue);
+ test("if only the old configuration's is defined, uses it", () {
+ var merged = new Configuration(
+ addTags: new Set.from(["foo", "bar"]),
+ chosenPresets: new Set.from(["baz", "bang"]))
+ .merge(new Configuration());
+
+ expect(merged.addTags, unorderedEquals(["foo", "bar"]));
+ expect(merged.chosenPresets, equals(["baz", "bang"]));
+ });
+
+ test("if only the new configuration's is defined, uses it", () {
+ var merged = new Configuration().merge(new Configuration(
+ addTags: new Set.from(["foo", "bar"]),
+ chosenPresets: new Set.from(["baz", "bang"])));
+
+ expect(merged.addTags, unorderedEquals(["foo", "bar"]));
+ expect(merged.chosenPresets, equals(["baz", "bang"]));
+ });
+
+ test("if both are defined, unions them", () {
+ var older = new Configuration(
+ addTags: new Set.from(["foo", "bar"]),
+ chosenPresets: new Set.from(["baz", "bang"]));
+ var newer = new Configuration(
+ addTags: new Set.from(["blip"]),
+ chosenPresets: new Set.from(["qux"]));
+ var merged = older.merge(newer);
+
+ expect(merged.addTags, unorderedEquals(["foo", "bar", "blip"]));
+ expect(merged.chosenPresets, equals(["baz", "bang", "qux"]));
});
});
@@ -286,26 +304,123 @@
});
});
- group("for onPlatform", () {
+ group("for config maps", () {
test("merges each nested configuration", () {
var merged = new Configuration(
+ tags: {
+ new BooleanSelector.parse("foo"):
+ new Configuration(verboseTrace: true),
+ new BooleanSelector.parse("bar"): new Configuration(jsTrace: true)
+ },
onPlatform: {
- new PlatformSelector.parse("vm"): new Configuration(verboseTrace: true),
- new PlatformSelector.parse("chrome"): new Configuration(jsTrace: true)
+ new PlatformSelector.parse("vm"):
+ new Configuration(verboseTrace: true),
+ new PlatformSelector.parse("chrome"):
+ new Configuration(jsTrace: true)
+ },
+ presets: {
+ "bang": new Configuration(verboseTrace: true),
+ "qux": new Configuration(jsTrace: true)
}
).merge(new Configuration(
+ tags: {
+ new BooleanSelector.parse("bar"): new Configuration(jsTrace: false),
+ new BooleanSelector.parse("baz"): new Configuration(skip: true)
+ },
onPlatform: {
- new PlatformSelector.parse("chrome"): new Configuration(jsTrace: false),
+ new PlatformSelector.parse("chrome"):
+ new Configuration(jsTrace: false),
new PlatformSelector.parse("firefox"): new Configuration(skip: true)
+ },
+ presets: {
+ "qux": new Configuration(jsTrace: false),
+ "zap": new Configuration(skip: true)
}
));
+ expect(merged.tags[new BooleanSelector.parse("foo")].verboseTrace,
+ isTrue);
+ expect(merged.tags[new BooleanSelector.parse("bar")].jsTrace, isFalse);
+ expect(merged.tags[new BooleanSelector.parse("baz")].skip, isTrue);
+
expect(merged.onPlatform[new PlatformSelector.parse("vm")].verboseTrace,
isTrue);
expect(merged.onPlatform[new PlatformSelector.parse("chrome")].jsTrace,
isFalse);
expect(merged.onPlatform[new PlatformSelector.parse("firefox")].skip,
isTrue);
+
+ expect(merged.presets["bang"].verboseTrace, isTrue);
+ expect(merged.presets["qux"].jsTrace, isFalse);
+ expect(merged.presets["zap"].skip, isTrue);
+ });
+ });
+
+ group("for presets", () {
+ test("automatically resolves a matching chosen preset", () {
+ var configuration = new Configuration(
+ presets: {"foo": new Configuration(verboseTrace: true)},
+ chosenPresets: ["foo"]);
+ expect(configuration.presets, isEmpty);
+ expect(configuration.chosenPresets, equals(["foo"]));
+ expect(configuration.knownPresets, equals(["foo"]));
+ expect(configuration.verboseTrace, isTrue);
+ });
+
+ test("resolves a chosen presets in order", () {
+ var configuration = new Configuration(
+ presets: {
+ "foo": new Configuration(verboseTrace: true),
+ "bar": new Configuration(verboseTrace: false)
+ },
+ chosenPresets: ["foo", "bar"]);
+ expect(configuration.presets, isEmpty);
+ expect(configuration.chosenPresets, equals(["foo", "bar"]));
+ expect(configuration.knownPresets, unorderedEquals(["foo", "bar"]));
+ expect(configuration.verboseTrace, isFalse);
+
+ configuration = new Configuration(
+ presets: {
+ "foo": new Configuration(verboseTrace: true),
+ "bar": new Configuration(verboseTrace: false)
+ },
+ chosenPresets: ["bar", "foo"]);
+ expect(configuration.presets, isEmpty);
+ expect(configuration.chosenPresets, equals(["bar", "foo"]));
+ expect(configuration.knownPresets, unorderedEquals(["foo", "bar"]));
+ expect(configuration.verboseTrace, isTrue);
+ });
+
+ test("ignores inapplicable chosen presets", () {
+ var configuration = new Configuration(
+ presets: {},
+ chosenPresets: ["baz"]);
+ expect(configuration.presets, isEmpty);
+ expect(configuration.chosenPresets, equals(["baz"]));
+ expect(configuration.knownPresets, equals(isEmpty));
+ });
+
+ test("resolves presets through merging", () {
+ var configuration = new Configuration(presets: {
+ "foo": new Configuration(verboseTrace: true)
+ }).merge(new Configuration(chosenPresets: ["foo"]));
+
+ expect(configuration.presets, isEmpty);
+ expect(configuration.chosenPresets, equals(["foo"]));
+ expect(configuration.knownPresets, equals(["foo"]));
+ expect(configuration.verboseTrace, isTrue);
+ });
+
+ test("preserves known presets through merging", () {
+ var configuration = new Configuration(presets: {
+ "foo": new Configuration(verboseTrace: true)
+ }, chosenPresets: ["foo"])
+ .merge(new Configuration());
+
+ expect(configuration.presets, isEmpty);
+ expect(configuration.chosenPresets, equals(["foo"]));
+ expect(configuration.knownPresets, equals(["foo"]));
+ expect(configuration.verboseTrace, isTrue);
});
});
});
diff --git a/test/runner/configuration/platform_test.dart b/test/runner/configuration/platform_test.dart
index f9bef94..373a08c 100644
--- a/test/runner/configuration/platform_test.dart
+++ b/test/runner/configuration/platform_test.dart
@@ -180,7 +180,7 @@
test.shouldExit(0);
});
- test("doesn't OS-specific configuration on a non-matching OS", () {
+ test("doesn't apply OS-specific configuration on a non-matching OS", () {
d.file("dart_test.yaml", JSON.encode({
"on_os": {
otherOS: {"filename": "test_*.dart"}
diff --git a/test/runner/configuration/presets_test.dart b/test/runner/configuration/presets_test.dart
new file mode 100644
index 0000000..6491ad7
--- /dev/null
+++ b/test/runner/configuration/presets_test.dart
@@ -0,0 +1,450 @@
+// Copyright (c) 2016, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+@TestOn("vm")
+
+import 'dart:convert';
+
+import 'package:scheduled_test/descriptor.dart' as d;
+import 'package:scheduled_test/scheduled_stream.dart';
+import 'package:scheduled_test/scheduled_test.dart';
+
+import 'package:test/src/util/exit_codes.dart' as exit_codes;
+import 'package:test/src/util/io.dart';
+
+import '../../io.dart';
+
+void main() {
+ useSandbox();
+
+ group("presets", () {
+ test("don't do anything by default", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"}
+ }
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ runTest(["test.dart"]).shouldExit(0);
+ });
+
+ test("can be selected on the command line", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"}
+ }
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ var test = runTest(["-P", "foo", "test.dart"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "-1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+ });
+
+ test("multiple presets can be selected", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"},
+ "bar": {"paths": ["test.dart"]}
+ }
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ var test = runTest(["-P", "foo,bar"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "-1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+ });
+
+ test("the latter preset takes precedence", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"},
+ "bar": {"timeout": "30s"}
+ }
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ runTest(["-P", "foo,bar", "test.dart"]).shouldExit(0);
+
+ var test = runTest(["-P", "bar,foo", "test.dart"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "-1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+ });
+
+ test("a preset takes precedence over the base configuration", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"}
+ },
+ "timeout": "30s"
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ var test = runTest(["-P", "foo", "test.dart"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "-1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "30s"}
+ },
+ "timeout": "00s"
+ })).create();
+
+ runTest(["-P", "foo", "test.dart"]).shouldExit(0);
+ });
+
+ test("a nested preset is activated", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "tags": {
+ "foo": {
+ "presets": {
+ "bar": {"timeout": "0s"}
+ },
+ },
+ }
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test 1", () => new Future.delayed(Duration.ZERO), tags: "foo");
+ test("test 2", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ var test = runTest(["-P", "bar", "test.dart"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "+1 -1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "30s"}
+ },
+ "timeout": "00s"
+ })).create();
+
+ runTest(["-P", "foo", "test.dart"]).shouldExit(0);
+ });
+ });
+
+ group("add_presets", () {
+ test("selects a preset", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"}
+ },
+ "add_presets": ["foo"]
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ var test = runTest(["test.dart"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "-1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+ });
+
+ test("applies presets in selection order", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"},
+ "bar": {"timeout": "30s"}
+ },
+ "add_presets": ["foo", "bar"]
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ runTest(["test.dart"]).shouldExit(0);
+
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"timeout": "0s"},
+ "bar": {"timeout": "30s"}
+ },
+ "add_presets": ["bar", "foo"]
+ })).create();
+
+ var test = runTest(["test.dart"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "-1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+ });
+
+ test("allows preset inheritance via add_presets", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"add_presets": ["bar"]},
+ "bar": {"timeout": "0s"}
+ }
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () => new Future.delayed(Duration.ZERO));
+ }
+ """).create();
+
+ var test = runTest(["-P", "foo", "test.dart"]);
+ test.stdout.expect(containsInOrder([
+ "-1: test",
+ "-1: Some tests failed."
+ ]));
+ test.shouldExit(1);
+ });
+
+ test("allows circular preset inheritance via add_presets", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {
+ "foo": {"add_presets": ["bar"]},
+ "bar": {"add_presets": ["foo"]}
+ }
+ })).create();
+
+ d.file("test.dart", """
+ import 'dart:async';
+
+ import 'package:test/test.dart';
+
+ void main() {
+ test("test", () {});
+ }
+ """).create();
+
+ runTest(["-P", "foo", "test.dart"]).shouldExit(0);
+ });
+ });
+
+ group("errors", () {
+ group("presets", () {
+ test("rejects an invalid preset type", () {
+ d.file("dart_test.yaml", '{"presets": {12: null}}').create();
+
+ var test = runTest([]);
+ test.stderr.expect(containsInOrder([
+ "presets key must be a string",
+ "^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+
+ test("rejects an invalid preset name", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {"foo bar": null}
+ })).create();
+
+ var test = runTest([]);
+ test.stderr.expect(containsInOrder([
+ "presets key must be an (optionally hyphenated) Dart identifier.",
+ "^^^^^^^^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+
+ test("rejects an invalid preset map", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": 12
+ })).create();
+
+ var test = runTest([]);
+ test.stderr.expect(containsInOrder([
+ "presets must be a map",
+ "^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+
+ test("rejects an invalid preset configuration", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "presets": {"foo": {"timeout": "12p"}}
+ })).create();
+
+ var test = runTest([]);
+ test.stderr.expect(containsInOrder([
+ "Invalid timeout: expected unit",
+ "^^^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+
+ test("rejects runner configuration in a non-runner context", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "tags": {
+ "foo": {
+ "presets": {"bar": {"filename": "*_blorp.dart"}}
+ }
+ }
+ })).create();
+
+ var test = runTest([]);
+ test.stderr.expect(containsInOrder([
+ "filename isn't supported here.",
+ "^^^^^^^^^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+
+ test("fails if an undefined preset is passed", () {
+ var test = runTest(["-P", "foo"]);
+ test.stderr.expect(consumeThrough(contains('Undefined preset "foo".')));
+ test.shouldExit(exit_codes.usage);
+ });
+
+ test("fails if an undefined preset is added", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "add_presets": ["foo", "bar"]
+ })).create();
+
+ var test = runTest([]);
+ test.stderr.expect(consumeThrough(contains(
+ 'Undefined presets "foo" and "bar".')));
+ test.shouldExit(exit_codes.usage);
+ });
+
+ test("fails if an undefined preset is added in a nested context", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "on_os": {
+ currentOS.identifier: {
+ "add_presets": ["bar"]
+ }
+ }
+ })).create();
+
+ var test = runTest([]);
+ test.stderr.expect(consumeThrough(contains('Undefined preset "bar".')));
+ test.shouldExit(exit_codes.usage);
+ });
+ });
+
+ group("add_presets", () {
+ test("rejects an invalid list type", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "add_presets": "foo"
+ })).create();
+
+ var test = runTest(["test.dart"]);
+ test.stderr.expect(containsInOrder([
+ "add_presets must be a list",
+ "^^^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+
+ test("rejects an invalid preset type", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "add_presets": [12]
+ })).create();
+
+ var test = runTest(["test.dart"]);
+ test.stderr.expect(containsInOrder([
+ "Preset name must be a string",
+ "^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+
+ test("rejects an invalid preset name", () {
+ d.file("dart_test.yaml", JSON.encode({
+ "add_presets": ["foo bar"]
+ })).create();
+
+ var test = runTest(["test.dart"]);
+ test.stderr.expect(containsInOrder([
+ "Preset name must be an (optionally hyphenated) Dart identifier.",
+ "^^^^^^^^^"
+ ]));
+ test.shouldExit(exit_codes.data);
+ });
+ });
+ });
+}
diff --git a/test/runner/configuration/tags_test.dart b/test/runner/configuration/tags_test.dart
index cddbf76..5fd432e 100644
--- a/test/runner/configuration/tags_test.dart
+++ b/test/runner/configuration/tags_test.dart
@@ -158,7 +158,7 @@
test.shouldExit(exit_codes.data);
});
- test("rejects an inavlid tag map", () {
+ test("rejects an invalid tag map", () {
d.file("dart_test.yaml", JSON.encode({
"tags": 12
})).create();
@@ -219,7 +219,7 @@
var test = runTest(["test.dart"]);
test.stderr.expect(containsInOrder([
- "Tags must be strings",
+ "Tag name must be a string",
"^^"
]));
test.shouldExit(exit_codes.data);
@@ -232,7 +232,7 @@
var test = runTest(["test.dart"]);
test.stderr.expect(containsInOrder([
- "Invalid tag. Tags must be (optionally hyphenated) Dart identifiers.",
+ "Tag name must be an (optionally hyphenated) Dart identifier.",
"^^^^^^^^^"
]));
test.shouldExit(exit_codes.data);
diff --git a/test/runner/runner_test.dart b/test/runner/runner_test.dart
index 9eb92cb..5c37bf0 100644
--- a/test/runner/runner_test.dart
+++ b/test/runner/runner_test.dart
@@ -62,6 +62,7 @@
-p, --platform The platform(s) on which to run the tests.
$_browsers
+-P, --preset The configuration preset(s) to use.
-j, --concurrency=<threads> The number of concurrent test suites run.
(defaults to "$_defaultConcurrency")
