UTF-8 decoder using a state machine.

Two-pass decoder: the first pass scans through the input to compute
the length of the resulting string and which decoder to use, and the
second pass does the actual decoding.

The same decoder is used for both one-shot and chunked decoding, and
both with and without allowMalformed. If there is an error in the input
and allowMalformed is true, it starts over with a general decoder that
supports malformed input and allocates space as it goes along.

JS targets go directly to the general decoder, as the two-pass approach
is not beneficial here.

Three pieces of the decoder are designed to be pluggable by patches to
optimize the performance further:
- scan, running the first pass of the conversion.
- decode8, decoding Latin1 data into a OneByteString.
- decode16, decoding arbitrary data into a TwoByteString.

Improves decoding speed, especially for complex input (many multi-byte
characters). Observed speed increases are approximately:
 - dart2js: up to 40%
 - VM JIT:  up to 260%
 - VM AOT:  up to 130%

The constant overhead of calling the UTF-8 decoder is also significantly
reduced for dart2js.

Code size for dart2js is slightly reduced compared to the old decoder.

ASCII inputs currently see a slight speed decrease for VM targets, which
will be fixed in https://dart-review.googlesource.com/c/sdk/+/145460

This is part of the implementation of the breaking change described at
https://github.com/dart-lang/sdk/issues/41100

Closes https://github.com/dart-lang/sdk/issues/28832
Closes https://github.com/dart-lang/sdk/issues/31954

Ideas for further improvements to the decoder are collected in
https://github.com/dart-lang/sdk/issues/41734

Change-Id: I3c5bb84e8d6783231680a9d34d6c38e8a28ab112
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/142025
Reviewed-by: Stephen Adams <sra@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
12 files changed
tree: 376b3183de1be2d2592cd3946ffc0441c4e87051
  1. .dart_tool/
  2. .github/
  3. benchmarks/
  4. build/
  5. client/
  6. docs/
  7. pkg/
  8. runtime/
  9. samples/
  10. samples-dev/
  11. sdk/
  12. sdk_nnbd/
  13. tests/
  14. third_party/
  15. tools/
  16. utils/
  17. .clang-format
  18. .gitattributes
  19. .gitconfig
  20. .gitignore
  21. .gn
  22. .mailmap
  23. .packages
  24. .style.yapf
  25. .vpython
  26. AUTHORS
  27. BUILD.gn
  28. CHANGELOG.md
  29. codereview.settings
  30. CONTRIBUTING.md
  31. DEPS
  32. LICENSE
  33. PATENT_GRANT
  34. PRESUBMIT.py
  35. README.dart-sdk
  36. README.md
  37. sdk_args.gni
  38. WATCHLISTS
README.md

Dart

A client-optimized language for fast apps on any platform

Dart is:

  • Optimized for UI: Develop with a programming language specialized around the needs of user interface creation

  • Productive: Make changes iteratively: use hot reload to see the result instantly in your running app

  • Fast on all platforms: Compile to ARM & x64 machine code for mobile, desktop, and backend. Or compile to JavaScript for the web

Dart's flexible compiler technology lets you run Dart code in different ways, depending on your target platform and goals:

  • Dart Native: For programs targeting devices (mobile, desktop, server, and more), Dart Native includes both a Dart VM with JIT (just-in-time) compilation and an AOT (ahead-of-time) compiler for producing machine code.

  • Dart Web: For programs targeting the web, Dart Web includes both a development time compiler (dartdevc) and a production time compiler (dart2js).

Dart platforms illustration

License & patents

Dart is free and open source.

See LICENSE and PATENT_GRANT.

Using Dart

Visit the dart.dev to learn more about the language, tools, getting started, and more.

Browse pub.dev for more packages and libraries contributed by the community and the Dart team.

Building Dart

If you want to build Dart yourself, here is a guide to getting the source, preparing your machine to build the SDK, and building.

There are more documents on our wiki.

Contributing to Dart

The easiest way to contribute to Dart is to file issues.

You can also contribute patches, as described in Contributing.