tree 1dd9f20d877d3299cdecc2b5071c32deca6d7399
parent d35332351079d10077fcacb0d820b2265119fe72
author Martin Kustermann <kustermann@google.com> 1660583218 +0000
committer Commit Bot <commit-bot@chromium.org> 1660583218 +0000

[CFE] Improve [StringCanonicalizer] implementation

The current [StringCanonicalizer] implementation has some issues:

  * It hangs on to large [Uint8List]/[String] objects in it's cache
    => This requires users (such as analyzer) to clear the cache
       frequently

  * Has api that works on dynamic input (which is assumed to be String
    or List<int> / Uint8List)
    => Call sites have typing information we loose when doing the call

  * It uses dynamic [] calls to compare input bytes with cached bytes /
    input strings with cached strings
    => Dynamic calls come with overhead
    => Will cause substring generation for every character comparison
    => Will compare bytes with strings (which doesn't make sense)

To address these issues we

  * Use the canonicalized [String] to compare against instead of the
    (much larger) source strings, thereby no longer hanging on to large
    strings in the canonicalizer cache (it's still an issue with
    [Uint8List]s though)

  * Make seperate API for canonicalization of strings, sub-strings or
    sub-utf8-bytes and use it from the token implementation.

  * For canonicalization of strings use String.== (instead of
    char-by-char comparison)

  * For canonicalization of sub-strings use String.charCodeAt instead of
    [] (which creates substrings)

  * Seperate out cache node entries into two classes and reduce memory
    consumption of the nodes that represent strings by 16 bytes (it
    does an additional `is` check on lookups in the cache, but that is
    better than paying for dynamic calls on the payload - which
    causes the compiler to do implicit checks)

=> This CL reduces RAM consumption and makes CFE scan/scan_bytes benchmarks a little faster.

TEST=ci

Change-Id: I157c298d26d25ac5da82c32eedfa270a590156f0
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/255121
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Jens Johansen <jensj@google.com>