| commit | 185d60754d54568607defc270b0af1c8f66d978c | [log] [tgz] |
|---|---|---|
| author | Konstantin Shcheglov <scheglov@google.com> | Mon Aug 25 12:44:42 2025 -0700 |
| committer | Commit Queue <dart-scoped@luci-project-accounts.iam.gserviceaccount.com> | Mon Aug 25 12:44:42 2025 -0700 |
| tree | 610f9f032c5296e530f9da8f40b138db67bead8a | |
| parent | 6ce015e01e281f27bdbc451a654a84d30a21ac6b [diff] |
Fine. A/B testing tool.
Introduce a CLI under `pkg/analyzer/tool/fine/ab_mutate` to compare the
conservative analyzer pipeline with the fine-grained dependencies
pipeline across seeded mutation chains. The tool applies deterministic
source mutations, collects diagnostics and timing for each engine, and
writes reproducible artifacts. Runs stop on divergence, exception, or
when a diagnostics cap is exceeded.
Key pieces:
- Engines: A = current state with conservative analysis without fine dependencies (not full rebuild without caching, but current coarse-grained incremental); B = incremental with `withFineDependencies`. A/B are warmed once, then performance counters are reset so the first mutation is incremental.
- Baseline: compute diagnostics for both engines and require equality before
running chains. The per-run diagnostics cap is `max(baseline*1.20, +50)`
unless explicitly set.
- Mutations (deterministic discovery & selection):
- `insert_unit_header_comment` (trivia only)
- `rename_local_variable` (declaration + local references in block)
- `remove_last_formal_parameter` (robust comma/group handling)
- `swap_top_level_functions` (preserves raw text/comments)
- `toggle_return_type_nullability` (named return types)
- Selection & editing:
- Separate `SiteSelector` context for parsed/resolved discovery so A/B stay
cold; edits are applied via `OverlayResourceProvider`.
- RNG is derived from a SHA-256–based 32-bit seed over structured parts for
stable, reproducible picks.
Outputs (for reproducibility and analysis):
- `manifest.json`, `files.json` (paths + SHA-256), and per-run `run_summary`.
- `baseline/diagnostics_{A,B}.json`, performance dumps, and a compare file.
- Per step: `mutation.json`, `before.dart`, `after.dart`, `patch.diff`,
`diagnostics_{A,B}.json`, `metrics_{A,B}.json`, performance, equality check,
and `diverge_details.json` when A≠B.
- Chain summary includes end reason, step list, per-kind counts, and p50/p90
speedup (A_time / B_time).
API & model additions:
- `Mutation`, `MutationEdit`, `MutationResult`, and `HarnessDiagnostic`
(normalized with stable `key()`); diagnostics use `ErrorProcessor` and skip
`TODO`s.
- Utility helpers for file discovery (skips generated/build/tool dirs), JSON
writing, deterministic seeding, and speedup stats.
Minor supporting change:
- Extend `FunctionAstVisitor` with `functionDeclaration` and
`methodDeclaration` callbacks and visit overrides. These are used by
mutation discovery (e.g., collecting bodies/identifiers for renames).
This adds an end-to-end, reproducible A/B harness for measuring
correctness and performance impacts of fine-grained dependencies on
realistic, evolving code, with minimal intrusion into the analyzer
proper.
Change-Id: I068138d6aa201ef9a79f884f0f4638d02c796b29
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/446480
Reviewed-by: Johnni Winther <johnniwinther@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Dart is:
Approachable: Develop with a strongly typed programming language that is consistent, concise, and offers modern language features like null safety and patterns.
Portable: Compile to ARM, x64, or RISC-V machine code for mobile, desktop, and backend. Compile to JavaScript or WebAssembly for the web.
Productive: Make changes iteratively: use hot reload to see the result instantly in your running app. Diagnose app issues using DevTools.
Dart's flexible compiler technology lets you run Dart code in different ways, depending on your target platform and goals:
Dart Native: For programs targeting devices (mobile, desktop, server, and more), Dart Native includes both a Dart VM with JIT (just-in-time) compilation and an AOT (ahead-of-time) compiler for producing machine code.
Dart Web: For programs targeting the web, Dart Web includes both a development time compiler (dartdevc) and a production time compiler (dart2js).
Dart is free and open source.
See LICENSE and PATENT_GRANT.
Visit dart.dev to learn more about the language, tools, and to find codelabs.
Browse pub.dev for more packages and libraries contributed by the community and the Dart team.
Our API reference documentation is published at api.dart.dev, based on the stable release. (We also publish docs from our beta and dev channels, as well as from the primary development branch).
If you want to build Dart yourself, here is a guide to getting the source, preparing your machine to build the SDK, and building.
There are more documents in our repo at docs.
The easiest way to contribute to Dart is to file issues.
You can also contribute patches, as described in Contributing.
Future plans for Dart are included in the combined Dart and Flutter roadmap on the Flutter wiki.