Report tests that pass on retry as flaky (#2801)

A test that failed and then passed on retry was reported inconsistently.
A `setUpAll` or `tearDownAll` stayed in `Engine.failed` because the code
that removes a test from `failed` when a later attempt passes was skipped
for callbacks that are modeled as tests in the runner, but aren't what the
user considers a test case. These can cause failures when there is an
error, but don't count as success when they run without exception.

The summaries then reported a failure even though the overall run
succeeded, for example `::error::2 tests passed, 1 failed.` from the
GitHub reporter or `+2 -1: All tests passed!` from the compact and
expanded reporters. Regular tests that passed on retry were counted as
passed with no indication that they had failed.

- Always remove a test from `failed` when it completes successfully, and
  track tests that passed after a failed attempt in new
  `LiveSuite.flaky` and `Engine.flaky` sets. A test that fails and is
  then skipped on retry moves from `failed` to `skipped` instead of
  being counted in both.
- Add an `AttemptCounter` utility for reporters to track attempts
  without changing `LiveTest` or the protocol between the runner and
  test isolates.
- compact, expanded, failures-only: show `[E] (attempt N of M)` for
  tests with `retry`, except for errors after the final attempt, and add
  `(N flaky)` to the final line. The expanded reporter also lists flaky
  tests.
- github: title flaky test groups `⚠️ <name> (passed on attempt N of M)`
  and tests that failed every attempt `(failed after N attempts)`. Print
  output and errors in the order they were emitted for all tests, add a
  flaky count to the summary, and use a `::warning::` annotation when
  the only problems are flaky tests.
- json: add `testDone.retried` and bump the protocol to 0.1.2. A
  synthetic test that passes on retry is now hidden, as the protocol
  documents.

A retried `tearDownAll` doesn't re-run its callbacks, so it always
passes on retry; that is tracked separately in
https://github.com/dart-lang/test/issues/2800.
21 files changed
tree: dcda45187408d2153768be077a2d46858a65805e
  1. .github/
  2. integration_tests/
  3. pkgs/
  4. tool/
  5. .gitattributes
  6. .gitignore
  7. analysis_options.yaml
  8. CODEOWNERS
  9. CONTRIBUTING.md
  10. mono_repo.yaml
  11. pubspec.yaml
  12. README.md
README.md

Dart CI OpenSSF Scorecard

What's here?

Welcome! package:test is the standard testing library for Dart and Flutter. If you have questions about Dart testing, please see the docs for package:test. package:test_api and package:test_core are implementation details and generally not user-facing.

package:checks is a relatively new library for expressing test expectations. It's a more modern version of package:matcher and features a literate API.

Packages

PackageDescriptionIssuesVersion
checksA framework for checking values against expectations and building custom expectations.issuespub package
fake_asyncFake asynchronous events such as timers and microtasks for deterministic testing.issuespub package
matcherSupport for specifying test expectations via an extensible Matcher class. Also includes a number of built-in Matcher implementations for common cases.issuespub package
testA full featured library for writing and running Dart tests across platforms.issuespub package
test_apiThe user facing API for structuring Dart tests and checking expectations.pub package
test_coreA basic library for writing tests and running them on the VM.pub package
test_descriptorAn API for defining and verifying files and directory structures.issuespub package
test_processTest processes: starting; validating stdout and stderr; checking exit code.issuespub package