The test runner supports a JSON reporter which provides a machine-readable representation of the test runner‘s progress. This reporter is intended for use by IDEs and other tools to present a custom view of the test runner’s operation without needing to parse output intended for humans.
Note that the test runner is highly asynchronous, and users of this protocol shouldn‘t make assumptions about the ordering of events beyond what’s explicitly specified in this document. It's possible for events from multiple tests to be intertwined, for a single test to emit an error after it completed successfully, and so on.
Pass the --reporter json
command-line flag to the test runner to activate the JSON reporter.
pub run test --reporter json <path-to-test-file>
The JSON stream will be emitted via standard output. It will be a stream of JSON objects, separated by newlines.
See json_reporter.schema.json
for a formal description of the protocol schema. See test/runner/json_reporter_test.dart
for some sample output.
The protocol emitted by the JSON reporter is considered part of the public API of the test
package, and is subject to its semantic versioning restrictions. In particular:
No new feature will be added to the protocol without increasing the test package's minor version number.
No breaking change will be made to the protocol without increasing the test package's major version number.
The following changes are not considered breaking. This is not necessarily a comprehensive list.
Adding a new attribute to an existing object.
Adding a new type of any object with a type
parameter.
Adding new test state values.
Each major type of JSON object used by the protocol is described by a class. Classes have names which are referred to in this document, but are not used as part of the protocol. Classes have typed attributes, which refer to the types and names of attributes in the JSON objects. If an attribute's type is another class, that refers to a nested object. The special type List<...>
indicates a JSON list of the given type.
Classes can “extend” one another, meaning that the subclass has all the attributes of the superclass. Concrete subclasses can be distinguished by the specific value of their type
attribute. Classes may be abstract, indicating that only their subclasses will ever be used.
abstract class Event { // The type of the event. // // This is always one of the subclass types listed below. String type; // The time (in milliseconds) that has elapsed since the test runner started. int time; }
This is the root class of the protocol. All root-level objects emitted by the JSON reporter will be subclasses of Event
.
class StartEvent extends Event { String type = "start"; // The version of the JSON reporter protocol being used. // // This is a semantic version, but it reflects only the version of the // protocol—it's not identical to the version of the test runner itself. String protocolVersion; // The version of the test runner being used. String runnerVersion; }
A single start event is emitted before any other events. It indicates that the test runner has started running.
class AllSuitesEvent { String type = "allSuites"; /// The total number of suites that will be loaded. int count; }
A single suite count event is emitted once the test runner knows the total number of suites that will be loaded over the course of the test run. Because this is determined asynchronously, its position relative to other events (except StartEvent
) is not guaranteed.
class SuiteEvent extends Event { String type = "suite"; /// Metadata about the suite. Suite suite; }
A suite event is emitted before any GroupEvent
s for groups in a given test suite. This is the only event that contains the full metadata about a suite; future events will refer to the suite by its opaque ID.
class GroupEvent extends Event { String type = "group"; /// Metadata about the group. Group group; }
A group event is emitted before any TestStartEvent
s for tests in a given group. This is the only event that contains the full metadata about a group; future events will refer to the group by its opaque ID.
This includes the implicit group at the root of each suite, which has a null
name. However, it does not include implicit groups for the virtual suites generated to represent loading test files.
If the group is skipped, a single TestStartEvent
will be emitted for a test within the group, followed by a TestDoneEvent
marked as skipped. The group.metadata
field should not be used for determining whether a group is skipped.
class TestStartEvent extends Event { String type = "testStart"; // Metadata about the test that started. Test test; }
An event emitted when a test begins running. This is the only event that contains the full metadata about a test; future events will refer to the test by its opaque ID.
If the test is skipped, its TestDoneEvent
will have skipped
set to true
. The test.metadata
should not be used for determining whether a test is skipped.
class MessageEvent extends Event { String type = "print"; // The ID of the test that printed a message. int testID; // The type of message being printed. String messageType; // The message that was printed. String message; }
A MessageEvent
indicates that a test emitted a message that should be displayed to the user. The messageType
field indicates the precise type of this message. Different message types should be visually distinguishable.
A message of type “print” comes from a user explicitly calling print()
.
A message of type “skip” comes from a test, or a section of a test, being skipped. A skip message shouldn't be considered the authoritative source that a test was skipped; the TestDoneEvent.skipped
field should be used instead.
class ErrorEvent extends Event { String type = "error"; // The ID of the test that experienced the error. int testID; // The result of calling toString() on the error object. String error; // The error's stack trace, in the stack_trace package format. String stackTrace; // Whether the error was a TestFailure. bool isFailure; }
A ErrorEvent
indicates that a test encountered an uncaught error. Note that this may happen even after the test has completed, in which case it should be considered to have failed.
If a test is asynchronous, it may encounter multiple errors, which will result in multiple ErrorEvent
s.
class TestDoneEvent extends Event { String type = "testDone"; // The ID of the test that completed. int testID; // The result of the test. String result; // Whether the test's result should be hidden. bool hidden; // Whether the test (or some part of it) was skipped. bool skipped; }
An event emitted when a test completes. The result
attribute indicates the result of the test:
"success"
if the test had no errors.
"failure"
if the test had a TestFailure
but no other errors.
"error"
if the test had an error other than a TestFailure
.
If the test encountered an error, the TestDoneEvent
will be emitted after the corresponding ErrorEvent
.
The hidden
attribute indicates that the test's result should be hidden and not counted towards the total number of tests run for the suite. This is true for virtual tests created for loading test suites, setUpAll()
, and tearDownAll()
. Only successful tests will be hidden.
Note that it's possible for a test to encounter an error after completing. In that case, it should be considered to have failed, but no additional TestDoneEvent
will be emitted. If a previously-hidden test encounters an error after completing, it should be made visible.
class DoneEvent extends Event { String type = "done"; // Whether all tests succeeded (or were skipped). bool success; }
An event indicating the result of the entire test run. This will be the final event emitted by the reporter.
class Test { // An opaque ID for the test. int id; // The name of the test, including prefixes from any containing groups. String name; // The ID of the suite containing this test. int suiteID; // The IDs of groups containing this test, in order from outermost to // innermost. List<int> groupIDs; // The (1-based) line on which the test was defined, or `null`. int line; // The (1-based) column on which the test was defined, or `null`. int column; // The URL for the file in which the test was defined, or `null`. String url; // This field is deprecated and should not be used. Metadata metadata; }
A single test case. The test‘s ID is unique in the context of this test run. It’s used elsewhere in the protocol to refer to this test without including its full representation.
Most tests will have at least one group ID, representing the implicit root group. However, some may not; these should be treated as having no group metadata.
The line
, column
, and url
fields indicate the location the test()
function was called to create this test. They‘re treated as a unit: they’ll either all be null
or they'll all be non-null
. The URL is always absolute, and may be a package:
URL.
class Suite { // An opaque ID for the group. int id; // The platform on which the suite is running. String? platform; // The path to the suite's file. String path; }
A test suite corresponding to a loaded test file. The suite‘s ID is unique in the context of this test run. It’s used elsewhere in the protocol to refer to this suite without including its full representation.
A suite‘s platform is one of the platforms that can be passed to the --platform
option, or null
if there is no platform (for example if the file doesn’t exist at all). Its path is either absolute or relative to the root of the current package.
class Group { // An opaque ID for the group. int id; // The name of the group, including prefixes from any containing groups. String? name; // The ID of the suite containing this group. int suiteID; // The ID of the group's parent group, unless it's the root group. int? parentID; // The number of tests (recursively) within this group. int testCount; // The (1-based) line on which the group was defined, or `null`. int line; // The (1-based) column on which the group was defined, or `null`. int column; // The URL for the file in which the group was defined, or `null`. String url; // This field is deprecated and should not be used. Metadata metadata; }
A group containing test cases. The group‘s ID is unique in the context of this test run. It’s used elsewhere in the protocol to refer to this group without including its full representation.
The implicit group at the root of each test suite has null
name
and parentID
attributes.
The line
, column
, and url
fields indicate the location the group()
function was called to create this group. They‘re treated as a unit: they’ll either all be null
or they'll all be non-null
. The URL is always absolute, and may be a package:
URL.
class Metadata { bool skip; String? skipReason; }
The metadata class is deprecated and should not be used.