[CFE] Improve [StringCanonicalizer] implementation

The current [StringCanonicalizer] implementation has some issues:

  * It hangs on to large [Uint8List]/[String] objects in it's cache
    => This requires users (such as analyzer) to clear the cache

  * Has api that works on dynamic input (which is assumed to be String
    or List<int> / Uint8List)
    => Call sites have typing information we loose when doing the call

  * It uses dynamic [] calls to compare input bytes with cached bytes /
    input strings with cached strings
    => Dynamic calls come with overhead
    => Will cause substring generation for every character comparison
    => Will compare bytes with strings (which doesn't make sense)

To address these issues we

  * Use the canonicalized [String] to compare against instead of the
    (much larger) source strings, thereby no longer hanging on to large
    strings in the canonicalizer cache (it's still an issue with
    [Uint8List]s though)

  * Make seperate API for canonicalization of strings, sub-strings or
    sub-utf8-bytes and use it from the token implementation.

  * For canonicalization of strings use String.== (instead of
    char-by-char comparison)

  * For canonicalization of sub-strings use String.charCodeAt instead of
    [] (which creates substrings)

  * Seperate out cache node entries into two classes and reduce memory
    consumption of the nodes that represent strings by 16 bytes (it
    does an additional `is` check on lookups in the cache, but that is
    better than paying for dynamic calls on the payload - which
    causes the compiler to do implicit checks)

=> This CL reduces RAM consumption and makes CFE scan/scan_bytes benchmarks a little faster.


Change-Id: I157c298d26d25ac5da82c32eedfa270a590156f0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/255121
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>
3 files changed
tree: 1dd9f20d877d3299cdecc2b5071c32deca6d7399
  1. .dart_tool/
  2. .github/
  3. benchmarks/
  4. build/
  5. docs/
  6. pkg/
  7. runtime/
  8. samples/
  9. samples-dev/
  10. samples_2/
  11. sdk/
  12. tests/
  13. third_party/
  14. tools/
  15. utils/
  16. .clang-format
  17. .gitattributes
  18. .gitconfig
  19. .gitignore
  20. .gn
  21. .mailmap
  22. .style.yapf
  23. .vpython
  25. BUILD.gn
  26. CHANGELOG.md
  27. codereview.settings
  29. DEPS
  31. OWNERS
  33. PRESUBMIT.py
  34. README.dart-sdk
  35. README.md
  36. sdk_args.gni
  37. SECURITY.md


A client-optimized language for fast apps on any platform

Dart is:

  • Optimized for UI: Develop with a programming language specialized around the needs of user interface creation.

  • Productive: Make changes iteratively: use hot reload to see the result instantly in your running app.

  • Fast on all platforms: Compile to ARM & x64 machine code for mobile, desktop, and backend. Or compile to JavaScript for the web.

Dart's flexible compiler technology lets you run Dart code in different ways, depending on your target platform and goals:

  • Dart Native: For programs targeting devices (mobile, desktop, server, and more), Dart Native includes both a Dart VM with JIT (just-in-time) compilation and an AOT (ahead-of-time) compiler for producing machine code.

  • Dart Web: For programs targeting the web, Dart Web includes both a development time compiler (dartdevc) and a production time compiler (dart2js).

Dart platforms illustration

License & patents

Dart is free and open source.


Using Dart

Visit dart.dev to learn more about the language, tools, and to find codelabs.

Browse pub.dev for more packages and libraries contributed by the community and the Dart team.

Our API reference documentation is published at api.dart.dev, based on the stable release. (We also publish docs from our beta and dev channels, as well as from the primary development branch).

Building Dart

If you want to build Dart yourself, here is a guide to getting the source, preparing your machine to build the SDK, and building.

There are more documents on our wiki.

Contributing to Dart

The easiest way to contribute to Dart is to file issues.

You can also contribute patches, as described in Contributing.