Report tests that pass on retry as flaky (#2801) A test that failed and then passed on retry was reported inconsistently. A `setUpAll` or `tearDownAll` stayed in `Engine.failed` because the code that removes a test from `failed` when a later attempt passes was skipped for callbacks that are modeled as tests in the runner, but aren't what the user considers a test case. These can cause failures when there is an error, but don't count as success when they run without exception. The summaries then reported a failure even though the overall run succeeded, for example `::error::2 tests passed, 1 failed.` from the GitHub reporter or `+2 -1: All tests passed!` from the compact and expanded reporters. Regular tests that passed on retry were counted as passed with no indication that they had failed. - Always remove a test from `failed` when it completes successfully, and track tests that passed after a failed attempt in new `LiveSuite.flaky` and `Engine.flaky` sets. A test that fails and is then skipped on retry moves from `failed` to `skipped` instead of being counted in both. - Add an `AttemptCounter` utility for reporters to track attempts without changing `LiveTest` or the protocol between the runner and test isolates. - compact, expanded, failures-only: show `[E] (attempt N of M)` for tests with `retry`, except for errors after the final attempt, and add `(N flaky)` to the final line. The expanded reporter also lists flaky tests. - github: title flaky test groups `⚠️ <name> (passed on attempt N of M)` and tests that failed every attempt `(failed after N attempts)`. Print output and errors in the order they were emitted for all tests, add a flaky count to the summary, and use a `::warning::` annotation when the only problems are flaky tests. - json: add `testDone.retried` and bump the protocol to 0.1.2. A synthetic test that passes on retry is now hidden, as the protocol documents. A retried `tearDownAll` doesn't re-run its callbacks, so it always passes on retry; that is tracked separately in https://github.com/dart-lang/test/issues/2800.
Welcome! package:test is the standard testing library for Dart and Flutter. If you have questions about Dart testing, please see the docs for package:test. package:test_api and package:test_core are implementation details and generally not user-facing.
package:checks is a relatively new library for expressing test expectations. It's a more modern version of package:matcher and features a literate API.
| Package | Description | Issues | Version |
|---|---|---|---|
| checks | A framework for checking values against expectations and building custom expectations. | ||
| fake_async | Fake asynchronous events such as timers and microtasks for deterministic testing. | ||
| matcher | Support for specifying test expectations via an extensible Matcher class. Also includes a number of built-in Matcher implementations for common cases. | ||
| test | A full featured library for writing and running Dart tests across platforms. | ||
| test_api | The user facing API for structuring Dart tests and checking expectations. | ||
| test_core | A basic library for writing tests and running them on the VM. | ||
| test_descriptor | An API for defining and verifying files and directory structures. | ||
| test_process | Test processes: starting; validating stdout and stderr; checking exit code. |