The test runner supports a JSON reporter which provides a machine-readable representation of the test runner‘s progress. This reporter is intended for use by IDEs and other tools to present a custom view of the test runner’s operation without needing to parse output intended for humans.
Note that the test runner is highly asynchronous, and users of this protocol shouldn‘t make assumptions about the ordering of events beyond what’s explicitly specified in this document. It's possible for events from multiple tests to be intertwined, for a single test to emit an error after it completed successfully, and so on.
Pass the --reporter json command-line flag to the test runner to activate the JSON reporter.
pub run test --reporter json <path-to-test-file>
The JSON stream will be emitted via standard output. It will be a stream of JSON objects, separated by newlines.
See json_reporter.schema.json for a formal description of the protocol schema. See test/runner/json_reporter_test.dart for some sample output.
The protocol emitted by the JSON reporter is considered part of the public API of the test package, and is subject to its semantic versioning restrictions. In particular:
No new feature will be added to the protocol without increasing the test package's minor version number.
No breaking change will be made to the protocol without increasing the test package's major version number.
The following changes are not considered breaking. This is not necessarily a comprehensive list.
Adding a new attribute to an existing object.
Adding a new type of any object with a type parameter.
Adding new test state values.
Each major type of JSON object used by the protocol is described by a class. Classes have names which are referred to in this document, but are not used as part of the protocol. Classes have typed attributes, which refer to the types and names of attributes in the JSON objects. If an attribute's type is another class, that refers to a nested object. The special type List<...> indicates a JSON list of the given type.
Classes can “extend” one another, meaning that the subclass has all the attributes of the superclass. Concrete subclasses can be distinguished by the specific value of their type attribute. Classes may be abstract, indicating that only their subclasses will ever be used.
abstract class Event {
// The type of the event.
//
// This is always one of the subclass types listed below.
String type;
// The time (in milliseconds) that has elapsed since the test runner started.
int time;
}
This is the root class of the protocol. All root-level objects emitted by the JSON reporter will be subclasses of Event.
class StartEvent extends Event {
String type = "start";
// The version of the JSON reporter protocol being used.
//
// This is a semantic version, but it reflects only the version of the
// protocol—it's not identical to the version of the test runner itself.
String protocolVersion;
// The version of the test runner being used.
String runnerVersion;
}
A single start event is emitted before any other events. It indicates that the test runner has started running.
class GroupEvent extends Event {
String type = "group";
/// Metadata about the group.
Group group;
}
A group event is emitted before any TestStartEvents for tests in a given group. This is the only event that contains the full metadata about a group; future events will refer to the group by its opaque ID.
This includes the implicit group at the root of each suite, which has a null name. However, it does not include implicit groups for the virtual suites generated to represent loading test files.
The group should be considered skipped if group.metadata.skip is true. When a group is skipped, a single TestStartEvent will be emitted for a test within that group that will also be skipped.
class TestStartEvent extends Event {
String type = "testStart";
// Metadata about the test that started.
Test test;
}
An event emitted when a test begins running. This is the only event that contains the full metadata about a test; future events will refer to the test by its opaque ID.
The test should be considered skipped if test.metadata.skip is true.
class PrintEvent extends Event {
String type = "print";
// The ID of the test that printed a message.
int testID;
// The message that was printed.
String message;
}
A PrintEvent indicates that a test called print() and wishes to display output.
class ErrorEvent extends Event {
String type = "error";
// The ID of the test that experienced the error.
int testID;
// The result of calling toString() on the error object.
String error;
// The error's stack trace, in the stack_trace package format.
String stackTrace;
// Whether the error was a TestFailure.
bool isFailure;
}
A ErrorEvent indicates that a test encountered an uncaught error. Note that this may happen even after the test has completed, in which case it should be considered to have failed.
If a test is asynchronous, it may encounter multiple errors, which will result in multiple ErrorEvents.
class TestDoneEvent extends Event {
String type = "testDone";
// The ID of the test that completed.
int testID;
// The result of the test.
String result;
// Whether the test's result should be hidden.
bool hidden;
}
An event emitted when a test completes. The result attribute indicates the result of the test:
"success" if the test had no errors.
"failure" if the test had a TestFailure but no other errors.
"error" if the test had an error other than a TestFailure.
If the test encountered an error, the TestDoneEvent will be emitted after the corresponding ErrorEvent.
The hidden attribute indicates that the test's result should be hidden and not counted towards the total number of tests run for the suite. This is true for virtual tests created for loading test suites, setUpAll(), and tearDownAll(). Only successful tests will be hidden.
Note that it's possible for a test to encounter an error after completing. In that case, it should be considered to have failed, but no additional TestDoneEvent will be emitted. If a previously-hidden test encounters an error after completing, it should be made visible.
class DoneEvent extends Event {
String type = "done";
// Whether all tests succeeded (or were skipped).
bool success;
}
An event indicating the result of the entire test run. This will be the final event emitted by the reporter.
class Test {
// An opaque ID for the test.
int id;
// The name of the test, including prefixes from any containing groups.
String name;
// The IDs of groups containing this test, in order from outermost to
// innermost.
List<int> groupIDs;
// The test's metadata, including metadata from any containing groups.
Metadata metadata;
}
A single test case. The test‘s ID is unique in the context of this test run. It’s used elsewhere in the protocol to refer to this test without including its full representation.
Most tests will have at least one group ID, representing the implicit root group. However, some may not; these should be treated as having no group metadata.
class Group {
// An opaque ID for the group.
int id;
// The name of the group, including prefixes from any containing groups.
String? name;
// The ID of the group's parent group, unless it's the root group.
int? parentID;
// The group's metadata, including metadata from any containing groups.
Metadata metadata;
}
A group containing test cases. The group‘s ID is unique in the context of this test run. It’s used elsewhere in the protocol to refer to this test without including its full representation.
The implicit group at the root of each test suite has null name and parentID attributes.
class Metadata {
// Whether the test case will be skipped by the test runner.
bool skip;
// The reason the test case is skipped, if the user provided it.
String? skipReason;
}
The metadata attached to a test by a user.