This directory is created for gathering all tools doing gardening in one place. Every tool or script, big or small should go here, and over time, hopefully we will have a useful collection of tools that support every part of the gardening work.
The current (working) tools are:
All the tools have been created in an add-hoc manner, thus they solve specific tasks that may not match your workflow fully. Feel free to add functionality to these. Below is a detailed description of each of the tools.
The results tool should be the primary tool for looking at failures and updating of status files. The tool fetches results.logs generated for each invocation of test.py on either the build bots or the CQ and matches the actual result for each test against the status files in your repository. Use it by calling:
dart results.dart get failures <argument>
The arguments can be one of the following:
get failures <result.log> : for a local result.log file. get failures <uri_to_result_log> : for direct links to result.logs. get failures <uri_try_bot> : for links to try bot builders. get failures <commit_number> <patchset> : for links to try bot builders (see example below). get failures <builder> : for a builder name. get failures <builder> <build_number> : for a builder and build number. get failures <builder_group> : for a builder group.
Some common workflows are listed below.
If a CQ job fails then you will receive an email about try jobs failing. There is a direct link in the email to the builder that observed a failure:
https://ci.chromium.org/swarming/task/<swarm_task_id>?server=chromium-swarm.appspot.com
Copy the link and paste it in the command below to see all failures for that builder:
dart tools/gardening/bin/results.dart get failures <url> All result logs fetched. Calling test.py to find status files for the configuration and the expectation for 18298 tests. Estimated time remaining is 1 seconds... FAILED: analyzer/test/generated/strong_mode_kernel_test Result: RuntimeError Expected: {Slow, Pass} Configuration: mode=release, arch=x64, compiler=none, runtime=vm, checked, system=windows, use-sdk, builder-tag=win7 To run locally (if you have the right architecture and runtime): tools/test.py --mode=release --arch=x64 --compiler=none --runtime=vm --checked --system=windows --use-sdk --builder-tag=win7 pkg/analyzer/test/generated/strong_mode_kernel_test
In the PolyGerrit interface under Tryjobs, for each try builder, you can obtain the link by right-clicking a builder-link and use copy link address.
To get all failures from all builders in a try run, you can use the following command:
dart tools/gardening/bin/results.dart get failures <CL number> <patch set>
The CL number is shown at the top of the page, and in the URL, and the patch set number is shown in the try jobs pane.
There are a few ways to find failures on build bots.
To find failures for an entire builder group use the following:
dart tools/gardening/bin/results.dart get failures <group>
The groups are shown on the main waterfall: vm,vm-kernel,analyzer...
To find failures for the latest build for a single build bot, use the following:
dart tools/gardening/bin/results.dart get failures <builder_name>
If you want to check failures on a specific build on a builder, use the build number:
dart tools/gardening/bin/results.dart get failures <builder_name> <build_no>
Remember, that statuses are tested against status files in your current repository. As a result, previous errors may not be reported, if the status files have changed.
You can find failures directly from a result log from a run of test.py. You can pass the URL linking to a result log, or you can pass the file name of a local result log on the command line.
dart tools/gardening/bin/results.dart get failures <url>
Finally, you can read the result log from a file by passing the file as so:
dart tools/gardening/bin/results.dart get failures <file>
For now, the tool does not suggest any updates to status files, however, the tool can help to verify that changes to the status files has been done correctly.
If a build is failing, the results tool will show which tests failed and how their results differed from the expectations.
Update the status files appropriately, save the status files and run the tool again with the same arguments as before. If the changes are correct, the tool will now report that all test passes.
Note, that changed test files are not rerun, and new status annotations are not picked up
If you, for some reason, would like to have a result.log output for a test.py invocation, just pass the flag --write-result-log to test.py. The default output-directory is logs, therefore the file would be written to logs/result.log. Be aware that existing files are overwritten.
To change the output directory, use the --output-directory option.
If a test appears in multiple sections whose conditions are simultaneously true, the test‘s expectations is the union of the applicable entries. It’s often unintended and undesirable to have overlapping sections for this reason. To find overlapping sections, run:
dart tools/gardening/bin/status_overlapping.dart <suite>
or
dart tools/gardening/bin/status_overlapping.dart <suite> <test>
Example output can be:
... ~/dart-sdk/sdk/tests/lib_2/lib_2_analyzer.status 5: [ $compiler == dart2analyzer ] 9: mirrors/deferred_mirrors_metadata_test: [Fail] 10: mirrors/deferred_type_test: [StaticWarning, OK] 11: mirrors/generic_f_bounded_mixin_application_test: [StaticWarning] 12: mirrors/mirrors_nsm_mismatch_test: [StaticWarning, OK] 13: mirrors/mirrors_nsm_test: [StaticWarning, OK] 14: mirrors/mirrors_nsm_test/dart2js: [StaticWarning, OK] 17: mirrors/repeated_private_anon_mixin_app_test: [StaticWarning, OK] 19: [ $compiler == dart2analyzer && $strong ] 21: mirrors/deferred_mirrors_metadata_test: [StaticWarning] 22: mirrors/deferred_type_test: [CompileTimeError, OK] 23: mirrors/generic_f_bounded_mixin_application_test: [CompileTimeError] 24: mirrors/mirrors_nsm_mismatch_test: [CompileTimeError, OK] 25: mirrors/mirrors_nsm_test: [CompileTimeError, OK] 26: mirrors/mirrors_nsm_test/dart2js: [CompileTimeError, OK] 28: mirrors/repeated_private_anon_mixin_app_test: [CompileTimeError, OK] ...
Notice that the sections overlap, but there may not be any relationship between entries in a section or file, ex. line 9 and 21. This is easier to see if overlapping entries are grouped by tests. To divide overlapping sections and entries into tests, use --print-test.
This tool compares a test log of a build step with previous builds. This is particularly useful to detect changes over time, such as flakiness of failures, timeouts or to find the build where a test started failing. The tool can be used for a single builder (vertically) or a builder-group (horizontally and vertically).
The tool displays the result of the latest 10 runs, which can be modified by the option --run-count=X where X is the number of latest results to display.
Some typical use-cases are shown below.
Say a builder is going red, which means, for a step in that builder, some tests have not met their expectation. One can then find stdout log of that step and pass to the tool, as such:
Usage:
dart bin/compare_failures.dart <stdio-url>
where <stdio-url> is a url for a test log (“https://.../logs/stdio”) from the buildbot. Here is an example:
dart bin/compare_failures.dart https://uberchromegw.corp.google.com/i/client.dart/builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked%20vm%20tests/logs/stdio Errors for /builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked vm tests : none-vm-checked release_x64 standalone/io/http_server_response_test 6786: Pass / Crash 6785: / -- OK -- 6784: / -- OK -- 6783: / -- OK -- 6782: / -- OK -- 6781: / -- OK -- 6780: / -- OK -- 6779: / -- OK -- 6778: / -- OK -- 6777: / -- OK -- 6776: / -- OK --
Sometimes it may be that a group of builders started to turn red. To find if there is any commonality between failing test, use the following command:
Usage:
dart bin/compare_failures.dart <build-group>
where <build-group> is the name of the build-group as shown in the console. Here is an example:
dart bin/compare_failures.dart analyzer Timeouts for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests : none-vm-checked release_x64 pkg/analyzer/test/generated/strong_mode_driver_test vm 7106: 0:00:21.086108 7105: 0:00:19.712000 7104: 0:01:00.774077 7103: 0:00:19.315931 7102: 0:00:21.380137 7101: 0:00:16.947695 7100: 0:00:13.226322 7099: 0:00:15.951636 7098: 0:00:19.665966 7097: 0:00:15.654565 7096: 0:00:24.775477 Errors for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests : none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test 7106: Pass Slow / RuntimeError 7105: Pass Slow / RuntimeError 7104: Pass Slow / RuntimeError 7103: / -- OK -- 7102: / -- OK -- 7101: / -- OK -- 7100: / -- OK -- 7099: / -- OK -- 7098: / -- OK -- 7097: / -- OK -- 7096: / -- OK -- Errors for /builders/analyzer-win7-release-strong-be/builds/3716/steps/analyzer unit tests : none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test 3716: Pass Slow / RuntimeError 3715: Pass Slow / RuntimeError 3714: Pass Slow / RuntimeError 3713: / -- OK -- 3712: / -- OK -- 3711: / -- OK -- 3710: / -- OK -- 3709: / -- OK -- 3708: / -- OK -- 3707: / -- OK -- 3706: / -- OK -- No errors found for the 52 remaining bots.
Collects the configurations for all status files in the ‘tests’ folder that mention one of the test names given as argument. This is useful to see what the expectation of a particular test/group of tests have on different configurations.
Usage:
dart bin/status_summary.dart <test-name1> [<test-name2> ...]
where <test-nameX> are test names like language/arithmetic_test or function_subtype_typearg2_test.
Say that the test function_subtype_typearg2_test suddenly started to fail with a RuntimeError on Safari in the dart2js safari builder group. It may be that similar behavior was already spotted in the dart2js chrome builder group and had been resolved (the status file was changed). One could then use run the following command:
dart bin/status_summary.dart function_subtype_typearg2_test function_subtype_typearg2_test file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language/language_dart2js.status Crash [ $compiler == dart2js && $dart2js_with_kernel && $host_checked ] NoSuchMethodError: The method 'hasSubclass' was called on null. Crash [ $compiler == dart2js && $dart2js_with_kernel && $minified ] NoSuchMethodError: The method 'hasSubclass' was called on null. file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language_strong/language_strong.status RuntimeError [ $compiler == dartdevc && $runtime != none ] Issue 29920
The output would then indicate if there are other expectations for the test function_subtype_typearg2_test in other configurations.
Collects the test results for all build bots in [buildGroups] (defined in lib/src/buildbot_data.dart) for tests that mention one of the test names given as argument.
Usage:
dart bin/current_summary.dart <test-name1> [<test-name2> ...]
where <test-nameX> are test names like language/arithmetic_test.
The results are currently pulled from the second to last build since the last build might not have completed yet.
Say that a test function_subtype_typearg2_test is behaving strange and a few builders have started to turn red, and you suspect this particular test for being responsible. Running the following will provide an answer:
dart bin/current_summary.dart function_subtype_typearg2_test Fetching "/builders/vm-mac-debug-simdbc64-be/builds/-1/steps/vm tests" + 33 more ... Fetching "/builders/app-linux-debug-x64-be/builds/-1/steps/vm tests" + 2 more ... Fetching "/builders/vm-kernel-linux-release-x64-be/builds/-1/steps/front-end tests" + 7 more ... Fetching "/builders/vm-win-debug-ia32-russian-be/builds/-1/steps/vm tests" + 11 more ... Fetching "/builders/vm-noopt-simarm64-mac-be/builds/-1/steps/test vm" + 9 more ... Fetching "/builders/vm-linux-product-x64-be/builds/-1/steps/vm tests" + 2 more ... Fetching "/builders/vm-linux-debug-x64-reload-be/builds/-1/steps/vm tests" + 11 more ... Fetching "/builders/dart2js-linux-d8-hostchecked-unittest-1-5-be/builds/-1/steps/dart2js-d8 tests" + 54 more ... Fetching "/builders/dart2js-linux-d8-minified-1-5-be/builds/-1/steps/dart2js-d8 tests" + 79 more ... Fetching "/builders/dart2js-linux-jsshell-1-4-be/builds/-1/steps/dart2js-jsshell tests" + 39 more ... Fetching "/builders/analyzer-mac10.11-release-be/builds/-1/steps/analyze tests" + 53 more ... Fetching "/builders/dart2js-linux-chromeff-1-4-be/builds/-1/steps/dart2js-chrome tests" + 79 more ... Fetching "/builders/dart2js-linux-drt-1-2-be/builds/-1/steps/dart2js-drt tests" + 39 more ... Fetching "/builders/dart2js-mac10.11-safari-1-3-be/builds/-1/steps/dart2js-safari tests" + 29 more ... Fetching "/builders/dart2js-win8-ie11-1-4-be/builds/-1/steps/dart2js ie11 tests" + 111 more ... Fetching "/builders/pkg-mac10.11-release-be/builds/-1/steps/package unit tests" + 5 more ... language/function_subtype_typearg2_test pass: none-vm vm-mac-debug-simdbc64-be/vm tests pass: none-vm-checked vm-mac-debug-simdbc64-be/checked vm tests ... language_2/function_subtype_typearg2_test pass: none-vm vm-mac-debug-simdbc64-be/vm tests pass: none-vm-checked vm-mac-debug-simdbc64-be/checked vm tests pass: none-vm vm-mac-release-simdbc64-be/vm tests pass: none-vm-checked vm-mac-release-simdbc64-be/checked vm tests pass: none-vm vm-linux-debug-x64-be/vm tests fail: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests ...
The above output has been truncated (shown as ...), to better indicate how the result of a test is shown. One can then grep for failing tests, as so:
dart bin/current_summary.dart --group d8-minified language_2 | grep "fail:" -B 5 -A 5 pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests language_2/function_subtype_typearg2_test pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests fail: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests language_2/class_codegen_test pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests -- pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests ...
Luci is a tool made to query luci/logdog for information. The tool can be used to find the information about build-bots, build groups, build-details and commits across builds. There is nothing statically entered in the code files regarding the bots, thus this tool has all current information.
Usage:
dart bin/luci.dart <command>
To find help about each of the sub-tools, use help as <command>.
The primary build-bots are those build-bots shown in the console view. To get them as list:
dart bin/luci.dart --build-bots ddc-win-release-stable pkg-linux-release-dev dart2js-mac10.11-safari-3-3-be vm-mac-debug-x64-be vm-win-debug-ia32-russian-stable vm-win-debug-ia32-be dart2js-linux-drt-1-2-stable dart-sdk-linux-be pkg-win7-release-stable vm-kernel-mac-release-x64-be dart2js-linux-d8-hostchecked-unittest-5-5-stable ...
Similarly, all build bots can be found by:
dart bin/luci.dart --build-bots-all
The build groups can be shown as such:
dart bin/luci.dart --build-groups safari vm dart-sdk vm-kernel ddc vm-product analyzer vm-misc vm-reload dart2js-jsshell vm-app pub-pkg chrome vm-precomp dart2js-windows dart2js-d8-minified dart2js-linux dart2js-d8-hostchecked misc
This should match the view of the console.
The tool can show all the builders in a specific group:
dart bin/luci.dart --builders-in-group <group>
where <group> is one of the groups shown above.
To find all the builders of the chrome builder group, use the following:
dart bin/luci.dart --builders-in-group chrome dart2js-mac10.11-chrome-be dart2js-linux-drt-1-2-be dart2js-linux-drt-2-2-be dart2js-linux-drt-csp-minified-be
To find detailed information about a build on a build-bot, the following command can be used. This essentially gives a view similar to the html-page that shows information about a specific build, since it will give information about the steps, files and commits.
Usage:
dart bin/luci.dart --build-details <builder> <number>
where <builder> is the name of the builder and <number> is the build number. It can be used in the following way:
dart bin/luci.dart --build-details vm-linux-debug-x64-reload-be 2971
The buildbot assigns a build number to each build on a builder. To find the build numbers corresponding to a specific commit, use the --builds-with-commit option as follows:
Usage:
dart bin/luci.dart --builds-with-commit <commit_hash>
where <commit_hash> is the full hash of the specific commit. Since all bots have to be checked, and one normally wants to investigate the latest commits, the cache is only 15 minutes.
To run it, fx, on the hash 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c, one can run the following:
dart bin/luci.dart --builds-with-commit 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c 2017-09-18 14:42:04.977786 Info: Sorry - this is going to take some time, since we have to look into all 25 latest builds for all bots for client client.dart. Subsequent queries run faster if caching is not turned off... The commit '4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c is used in the following builds: dart2js-mac10.11-safari-3-3-be: #6821 https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-3-3-be/6821 vm-mac-debug-x64-be: #13007 https://luci-milo.appspot.com/buildbot/client.dart/vm-mac-debug-x64-be/13007 vm-win-debug-ia32-be: #5270 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-be/5270 dart-sdk-linux-be: #15544 https://luci-milo.appspot.com/buildbot/client.dart/dart-sdk-linux-be/15544 vm-kernel-mac-release-x64-be: #1441 https://luci-milo.appspot.com/buildbot/client.dart/vm-kernel-mac-release-x64-be/1441 ddc-mac-release-be: #972 https://luci-milo.appspot.com/buildbot/client.dart/ddc-mac-release-be/972 vm-win-product-x64-be: #8433 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-product-x64-be/8433 analyze-linux-be: #3696 https://luci-milo.appspot.com/buildbot/client.dart/analyze-linux-be/3696 vm-win-debug-x64-be: #5096 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-x64-be/5096 vm-win-debug-ia32-russian-be: #3836 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-russian-be/3836 dart2js-mac10.11-safari-1-3-be: #6730 https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-1-3-be/6730 ...