Document how to read the dashboard (#14577) * Document how to read the dashboard * address comments
diff --git a/dev/devicelab/README.md b/dev/devicelab/README.md index ed93e80..dd3e1c3 100644 --- a/dev/devicelab/README.md +++ b/dev/devicelab/README.md
@@ -7,15 +7,91 @@ the tests are referred to as "tasks" in the API, but since we primarily use it for testing, this document refers to them as "tests". -If you have access to Google's internal network, you can see the continuous -build results from the master branch at <http://go/flutter-dashboard/build.html>. -(There is currently no public view of this data, unfortunately.) +Build results are available at https://flutter-dashboard.appspot.com. -# Prerequisites +# Reading the dashboard -You must set the `ANDROID_HOME` environment variable to run tests on Android. If -you have a local build of the Flutter engine, then you have a copy of the -Android SDK at `.../engine/src/third_party/android_tools/sdk`. +## The build page + +The build page is accessible at https://flutter-dashboard.appspot.com/build.html. +This page reports the health of build servers, called _agents_, and the statuses +of build tasks. + +### Agents + +A green agent is considered healthy and ready to receive new tasks to build. A +red agent is broken and does not receive new tasks. + +In the example below, the dashboard shows that the `linux2` agent is broken and +requires attention. All other agents are healthy. + + + +### Tasks + +The table below the agent statuses displays the statuses of build tasks. Task +statuses are color-coded. The following statuses are available: + +**New task** (light blue): the task is waiting for an agent to pick it up and +start the build. + +**Task is running** (spinning blue): an agent is currently building the task. + +**Task succeeded** (green): an agent reported a successful completion of the +task. + +**Task is flaky** (yellow): the task was attempted multiple time, but only the +latest attempt succeeded (we currently only try twice). + +**Task failed** (red): the task failed all of the attempts. + +**Task underperformed** (orange): currently not used. + +**Task was skipped** (transparent): the task is not scheduled for a build. This +usually happens when a task is removed from `manifest.yaml` file. + +**Task status unknown** (purple): currently not used. + +In addition to color-coding, a task may display a question mark. This means +that the task was marked as flaky manually. The status of such task is ignored +when considering whether the build is broken or not. For example, if a flaky +task fails, GitHub will not prevent PR submissions. However, if the latest +status of a non-flaky task is red, all pending PRs will contain a warning about +the broken build and recommend caution when submitting. + +Legend: + + + +The example below shows that commit `e122d5d` caused a wide-spread breakage, +which was fixed by `bdc6f10`. It also shows that Travis, AppVeyor and Chrome +Infra (left-most tasks) decided to skip building these commits. Hovering over +a cell will pop up a tooltip containing the name of the broken task. Clicking +on the cell will open the log file in a new browser tab (only visible to core +contributors as of today). + + + +## Why is a task stuck on "new task" status? + +The dashboard aggregates build results from multiple build environments, +including Travis, AppVeyor, Chrome Infra, and devicelab. While devicelab +tests every commit that goes into the `master` branch, other environments +may skip some commits. For example, Travis and AppVeyor will only test the +_last_ commit of a PR that's merged into the `master` branch. Chrome Infra may +skip commits when they come in too fast. + +## How the devicelab runs the tasks + +The devicelab agents have a small script installed on them that continuously +asks the CI server for tasks to run. When the server finds a suitable task for +an agent it reserves that task for the agent. If the task succeeds, the agent +reports the success to the server and the dashboard shows that task in green. +If the task fails, the agent reports the failure to the server, the server +increments the counter counting the number of attempts it took to run the task +and puts the task back in the pool of available tasks. If a task does not +succeed after a certain number of attempts (as of this writing the limit is 2), +the task is marked as failed and is displayed using red color on the dashboard. # Running tests locally @@ -24,6 +100,12 @@ CI environment runs them. These commands are also useful when you need to reproduce a CI test failure locally. +## Prerequisites + +You must set the `ANDROID_HOME` environment variable to run tests on Android. If +you have a local build of the Flutter engine, then you have a copy of the +Android SDK at `.../engine/src/third_party/android_tools/sdk`. + To run a test, use option `-t` (`--task`): ```sh @@ -61,15 +143,6 @@ # Reproducing broken builds locally -If a commit caused a test to fail, -[the dashboard](http://go/flutter-dashboard/build.html) (requires access to the -Google network, sorry) might look something like this: - - - -The red circle tells you that a test failed. The number inside tells you how -many times the devicelab attempted to run the test before giving up on it. - To reproduce the breakage locally `git checkout` the corresponding Flutter revision. Note the name of the test that failed. In the example above the failing test is `flutter_gallery__transition_perf`. This name can be passed to
diff --git a/dev/devicelab/images/agent-statuses.png b/dev/devicelab/images/agent-statuses.png new file mode 100644 index 0000000..d0b3dc5 --- /dev/null +++ b/dev/devicelab/images/agent-statuses.png Binary files differ
diff --git a/dev/devicelab/images/broken-test.png b/dev/devicelab/images/broken-test.png index 6450f65..cf6db90 100644 --- a/dev/devicelab/images/broken-test.png +++ b/dev/devicelab/images/broken-test.png Binary files differ
diff --git a/dev/devicelab/images/legend.png b/dev/devicelab/images/legend.png new file mode 100644 index 0000000..c9bd9a0 --- /dev/null +++ b/dev/devicelab/images/legend.png Binary files differ
