commit 91c628d63aa5cd9c6165413946bb9f1171131f60
author Martin Kustermann <kustermann@google.com> Tue Mar 02 18:57:02 2021 +0000
committer commit-bot@chromium.org <commit-bot@chromium.org> Tue Mar 02 18:57:02 2021 +0000
tree 4c23dc3c000881cf23fb08dd29e096964aecfecf
parent 60732555f7c709fc14a4f04b2b4f6683c711534f

[vm/concurrency] Final support for hot-reload of multi-isolate groups

This is the initial implementation of hot reload with multi-isolate
groups.

Implementation:

As before, when a service API call triggers a reload it will be routed
as an OOB message to a specific isolate (**). As opposed to before, that
isolate has now to coordinate with all other isolates, ensuring that it
"owns" the reload and all other isolates are waiting in a state that
allows reload.

This is implemented as a [ReloadOperationScope] which first participates
in other reloads (if there are any) and then owns the reload. It will
send a new kind of service message to all other registered isolates. All
of them have to check in before reload can proceed. If a new isolate
is about to join the group, it will participate when registering the
isolate. If an old isolate wants to die, it will participate when
unregistering the isolate.

This means that in addition to the existing StackOverFlow checks that
can process OOB messages and therefore reload, we'll have isolate
registration and unregistration as well as a new
Isolate::kCheckForReload OOB message handler where an isolate can
participate in a reload.

We consider the isolate group to be reloadable if the main isolate has
loaded the program and set the root library. Helper isolates don't need
to load any more kernel code and only initialize core libraries, so it's
fine to reload them during this time.

(**) The reason we continue to send reload service API calls to any
isolate in an isolate group is that re-loading might involve calling out
to the embedder's tag handler. Doing so currently requires an active
isolate.

If we allowed a subset of dart_api.h (the subset needed by the tag
handler) to be used only with an active IsolateGroup instead of an
active Isolate we could remove this requirement.

Edge cases:

There's various edge cases to consider: The main edge case is, we currently
maintain an upper limit to the number of isolates executing in parallel
(to ensure each can have big enough chunk of new space, i.e. TLAB).

If there are more isolates with active work they are waiting until one
of the exiting ones "yields". To ensure progress, if any such actively
running isolate gets a request to participate in a reload, it will mark
its own thread as "blocked" and therefore "yields", so another isolate
can make progress until all isolates are participating and the reload
can start.

Marking an isolate as "blocked" happens by exiting that isolate. It will
free up it's TLAB, decrease active mutator count and (if running on VM's
thread pool) also temporarily increase the thread pool size.

The side-effect of this is that it will use one pthread per isolate
during reload. In the future we can extend this first implementation, by
specially handling isolates that don't have a message handler running.
Doing so would require careful consideration to avoid races.

Testing:

In order to test this we use a small helper framework for reload tests.
The helper framework will, similar to real world reload e.g. in flutter,
will spawn a subprocess. It will use the service API to trigger reloads
in this subproces.

To synchronize between the reload driver and the application being
reloaded it allows watching for events to be printed to stdout/stderr.

The reload test itself can be written - similar to multitests - with
annotations such as `// @include-in-relload-0` in them. The testing
framework will then generate multiple application versions that all get
compiled to kernel.

For simplicity we generate the kernel using the standalone VM with
`--snapshot-kind=kernel` and avoid using the incremental compiler.

There are 4 different tests exercising different aspects of
multi-isolate reload:

  vm/dart_2/isolates/reload_active_stack_test:
    Performs a reload while a fixed number of isolates have an active
    stack, thereby ensuring e.g. that all frames of all isolate mutator
    stacks get deoptimized, ...

  vm/dart_2/isolates/reload_no_active_stack_test:
    Similar to the test above, but instead of having an active stack the
    isolates can yield to the event loop, possibly be even descheduled

  vm/dart_2/isolates/reload_many_isolates_test:
    Similar to the test above, but this test uses many more isolates.

  vm/dart_2/isolates/reload_many_isolates_live_and_die_test:
    Performs a reload where isolates get spawned and die all the time.
    There are always P isolates alive at any given point in time, each
    of them spawns children when their parent has died.
    Performing a reload catches isolates as various stages of their
    lifecycle and can therefore cover a lot of corner cases.

TEST=vm/dart_2/isolates/reload_*_test.dart

Issue https://github.com/dart-lang/sdk/issues/36097

Change-Id: I97039b4084de040b7f2e22f5832a40d57ba398d5
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/187461
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Alexander Aprelev <aam@google.com>