This directory is created for gathering all tools doing gardening in one place. Every tool or script, big or small should go here, and over time, hopefully we will have a useful collection of tools that support every part of the gardening work.
The current (working) tools are:
All the tools have been created in an add-hoc manner, thus they solve specific tasks that may not match your workflow fully. Feel free to add functionality to these. Below is a detailed description of each of the tools.
This tool compares a test log of a build step with previous builds. This is particularly useful to detect changes over time, such as flakiness of failures, timeouts or to find the build where a test started failing. The tool can be used for a single builder (vertically) or a builder-group (horizontally and vertically).
The tool displays the result of the latest 10 runs, which can be modified by the option --run-count=X where X is the number of latest results to display.
Some typical use-cases are shown below.
Say a builder is going red, which means, for a step in that builder, some tests have not met their expectation. One can then find stdout log of that step and pass to the tool, as such:
Usage:
dart bin/compare_failures.dart <stdio-url>
where <stdio-url> is a url for a test log (“https://.../logs/stdio”) from the buildbot. Here is an example:
dart bin/compare_failures.dart https://uberchromegw.corp.google.com/i/client.dart/builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked%20vm%20tests/logs/stdio Errors for /builders/vm-linux-release-x64-optcounter-threshold-be/builds/6786/steps/checked vm tests : none-vm-checked release_x64 standalone/io/http_server_response_test 6786: Pass / Crash 6785: / -- OK -- 6784: / -- OK -- 6783: / -- OK -- 6782: / -- OK -- 6781: / -- OK -- 6780: / -- OK -- 6779: / -- OK -- 6778: / -- OK -- 6777: / -- OK -- 6776: / -- OK --
Sometimes it may be that a group of builders started to turn red. To find if there is any commonality between failing test, use the following command:
Usage:
dart bin/compare_failures.dart <build-group>
where <build-group> is the name of the build-group as shown in the console. Here is an example:
dart bin/compare_failures.dart analyzer Timeouts for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests : none-vm-checked release_x64 pkg/analyzer/test/generated/strong_mode_driver_test vm 7106: 0:00:21.086108 7105: 0:00:19.712000 7104: 0:01:00.774077 7103: 0:00:19.315931 7102: 0:00:21.380137 7101: 0:00:16.947695 7100: 0:00:13.226322 7099: 0:00:15.951636 7098: 0:00:19.665966 7097: 0:00:15.654565 7096: 0:00:24.775477 Errors for /builders/analyzer-win7-release-be/builds/7106/steps/analyzer unit tests : none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test 7106: Pass Slow / RuntimeError 7105: Pass Slow / RuntimeError 7104: Pass Slow / RuntimeError 7103: / -- OK -- 7102: / -- OK -- 7101: / -- OK -- 7100: / -- OK -- 7099: / -- OK -- 7098: / -- OK -- 7097: / -- OK -- 7096: / -- OK -- Errors for /builders/analyzer-win7-release-strong-be/builds/3716/steps/analyzer unit tests : none-vm-checked release_x64 pkg/analyzer/test/generated/non_error_resolver_kernel_test 3716: Pass Slow / RuntimeError 3715: Pass Slow / RuntimeError 3714: Pass Slow / RuntimeError 3713: / -- OK -- 3712: / -- OK -- 3711: / -- OK -- 3710: / -- OK -- 3709: / -- OK -- 3708: / -- OK -- 3707: / -- OK -- 3706: / -- OK -- No errors found for the 52 remaining bots.
Collects the configurations for all status files in the ‘tests’ folder that mention one of the test names given as argument. This is useful to see what the expectation of a particular test/group of tests have on different configurations.
Usage:
dart bin/status_summary.dart <test-name1> [<test-name2> ...]
where <test-nameX> are test names like language/arithmetic_test or function_subtype_typearg2_test.
Say that the test function_subtype_typearg2_test suddenly started to fail with a RuntimeError on Safari in the dart2js safari builder group. It may be that similar behavior was already spotted in the dart2js chrome builder group and had been resolved (the status file was changed). One could then use run the following command:
dart bin/status_summary.dart function_subtype_typearg2_test function_subtype_typearg2_test file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language/language_dart2js.status Crash [ $compiler == dart2js && $dart2js_with_kernel && $host_checked ] NoSuchMethodError: The method 'hasSubclass' was called on null. Crash [ $compiler == dart2js && $dart2js_with_kernel && $minified ] NoSuchMethodError: The method 'hasSubclass' was called on null. file:///usr/local/google/home/mkroghj/dart-sdk/sdk/tests/language_strong/language_strong.status RuntimeError [ $compiler == dartdevc && $runtime != none ] Issue 29920
The output would then indicate if there are other expectations for the test function_subtype_typearg2_test in other configurations.
Collects the test results for all build bots in [buildGroups] (defined in lib/src/buildbot_data.dart) for tests that mention one of the test names given as argument.
Usage:
dart bin/current_summary.dart <test-name1> [<test-name2> ...]
where <test-nameX> are test names like language/arithmetic_test.
The results are currently pulled from the second to last build since the last build might not have completed yet.
Say that a test function_subtype_typearg2_test is behaving strange and a few builders have started to turn red, and you suspect this particular test for being responsible. Running the following will provide an answer:
dart bin/current_summary.dart function_subtype_typearg2_test Fetching "/builders/vm-mac-debug-simdbc64-be/builds/-1/steps/vm tests" + 33 more ... Fetching "/builders/app-linux-debug-x64-be/builds/-1/steps/vm tests" + 2 more ... Fetching "/builders/vm-kernel-linux-release-x64-be/builds/-1/steps/front-end tests" + 7 more ... Fetching "/builders/vm-win-debug-ia32-russian-be/builds/-1/steps/vm tests" + 11 more ... Fetching "/builders/vm-noopt-simarm64-mac-be/builds/-1/steps/test vm" + 9 more ... Fetching "/builders/vm-linux-product-x64-be/builds/-1/steps/vm tests" + 2 more ... Fetching "/builders/vm-linux-debug-x64-reload-be/builds/-1/steps/vm tests" + 11 more ... Fetching "/builders/dart2js-linux-d8-hostchecked-unittest-1-5-be/builds/-1/steps/dart2js-d8 tests" + 54 more ... Fetching "/builders/dart2js-linux-d8-minified-1-5-be/builds/-1/steps/dart2js-d8 tests" + 79 more ... Fetching "/builders/dart2js-linux-jsshell-1-4-be/builds/-1/steps/dart2js-jsshell tests" + 39 more ... Fetching "/builders/analyzer-mac10.11-release-be/builds/-1/steps/analyze tests" + 53 more ... Fetching "/builders/dart2js-linux-chromeff-1-4-be/builds/-1/steps/dart2js-chrome tests" + 79 more ... Fetching "/builders/dart2js-linux-drt-1-2-be/builds/-1/steps/dart2js-drt tests" + 39 more ... Fetching "/builders/dart2js-mac10.11-safari-1-3-be/builds/-1/steps/dart2js-safari tests" + 29 more ... Fetching "/builders/dart2js-win8-ie11-1-4-be/builds/-1/steps/dart2js ie11 tests" + 111 more ... Fetching "/builders/pkg-mac10.11-release-be/builds/-1/steps/package unit tests" + 5 more ... language/function_subtype_typearg2_test pass: none-vm vm-mac-debug-simdbc64-be/vm tests pass: none-vm-checked vm-mac-debug-simdbc64-be/checked vm tests ... language_2/function_subtype_typearg2_test pass: none-vm vm-mac-debug-simdbc64-be/vm tests pass: none-vm-checked vm-mac-debug-simdbc64-be/checked vm tests pass: none-vm vm-mac-release-simdbc64-be/vm tests pass: none-vm-checked vm-mac-release-simdbc64-be/checked vm tests pass: none-vm vm-linux-debug-x64-be/vm tests fail: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests ...
The above output has been truncated (shown as ...), to better indicate how the result of a test is shown. One can then grep for failing tests, as so:
dart bin/current_summary.dart --group d8-minified language_2 | grep "fail:" -B 5 -A 5 pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests language_2/function_subtype_typearg2_test pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests fail: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests pass: dart2js-d8-checked dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup-checked tests language_2/class_codegen_test pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests -- pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-with-kernel-d8 tests pass: dart2js-d8 dart2js-linux-d8-minified-1-5-be/dart2js-d8-fast-startup tests ...
Luci is a tool made to query luci/logdog for information. The tool can be used to find the information about build-bots, build groups, build-details and commits across builds. There is nothing statically entered in the code files regarding the bots, thus this tool has all current information.
Usage:
dart bin/luci.dart <command>
To find help about each of the sub-tools, use help as <command>.
The primary build-bots are those build-bots shown in the console view. To get them as list:
dart bin/luci.dart --build-bots ddc-win-release-stable pkg-linux-release-dev dart2js-mac10.11-safari-3-3-be vm-mac-debug-x64-be vm-win-debug-ia32-russian-stable vm-win-debug-ia32-be dart2js-linux-drt-1-2-stable dart-sdk-linux-be pkg-win7-release-stable vm-kernel-mac-release-x64-be dart2js-linux-d8-hostchecked-unittest-5-5-stable ...
Similarly, all build bots can be found by:
dart bin/luci.dart --build-bots-all
The build groups can be shown as such:
dart bin/luci.dart --build-groups safari vm dart-sdk vm-kernel ddc vm-product analyzer vm-misc vm-reload dart2js-jsshell vm-app pub-pkg chrome vm-precomp dart2js-windows dart2js-d8-minified dart2js-linux dart2js-d8-hostchecked misc
This should match the view of the console.
The tool can show all the builders in a specific group:
dart bin/luci.dart --builders-in-group <group>
where <group> is one of the groups shown above.
To find all the builders of the chrome builder group, use the following:
dart bin/luci.dart --builders-in-group chrome dart2js-mac10.11-chrome-be dart2js-linux-drt-1-2-be dart2js-linux-drt-2-2-be dart2js-linux-drt-csp-minified-be
To find detailed information about a build on a build-bot, the following command can be used. This essentially gives a view similar to the html-page that shows information about a specific build, since it will give information about the steps, files and commits.
Usage:
dart bin/luci.dart --build-details <builder> <number>
where <builder> is the name of the builder and <number> is the build number. It can be used in the following way:
dart bin/luci.dart --build-details vm-linux-debug-x64-reload-be 2971
The buildbot assigns a build number to each build on a builder. To find the build numbers corresponding to a specific commit, use the --builds-with-commit option as follows:
Usage:
dart bin/luci.dart --builds-with-commit <commit_hash>
where <commit_hash> is the full hash of the specific commit. Since all bots have to be checked, and one normally wants to investigate the latest commits, the cache is only 15 minutes.
To run it, fx, on the hash 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c, one can run the following:
dart bin/luci.dart --builds-with-commit 4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c 2017-09-18 14:42:04.977786 Info: Sorry - this is going to take some time, since we have to look into all 25 latest builds for all bots for client client.dart. Subsequent queries run faster if caching is not turned off... The commit '4d55a6779e6430c382bf0e0e4b8c0d61bee5c92c is used in the following builds: dart2js-mac10.11-safari-3-3-be: #6821 https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-3-3-be/6821 vm-mac-debug-x64-be: #13007 https://luci-milo.appspot.com/buildbot/client.dart/vm-mac-debug-x64-be/13007 vm-win-debug-ia32-be: #5270 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-be/5270 dart-sdk-linux-be: #15544 https://luci-milo.appspot.com/buildbot/client.dart/dart-sdk-linux-be/15544 vm-kernel-mac-release-x64-be: #1441 https://luci-milo.appspot.com/buildbot/client.dart/vm-kernel-mac-release-x64-be/1441 ddc-mac-release-be: #972 https://luci-milo.appspot.com/buildbot/client.dart/ddc-mac-release-be/972 vm-win-product-x64-be: #8433 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-product-x64-be/8433 analyze-linux-be: #3696 https://luci-milo.appspot.com/buildbot/client.dart/analyze-linux-be/3696 vm-win-debug-x64-be: #5096 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-x64-be/5096 vm-win-debug-ia32-russian-be: #3836 https://luci-milo.appspot.com/buildbot/client.dart/vm-win-debug-ia32-russian-be/3836 dart2js-mac10.11-safari-1-3-be: #6730 https://luci-milo.appspot.com/buildbot/client.dart/dart2js-mac10.11-safari-1-3-be/6730 ...