tree 3c34f52badb472a23879c49f8569596c58fb939c
parent 924a33c0f5705ecf504e9a75c7c11b40c50e410d
author Tess Strickland <sstrickl@google.com> 1688631407 +0000
committer Commit Queue <dart-scoped@luci-project-accounts.iam.gserviceaccount.com> 1688631407 +0000

[vm] Tweak handling of rare types with non-null instance type arguments.

The CFE may return super-bounded types (types that are a strict
supertype of the rare type) for certain type positions like the right
hand sides of `is` and `as` checks, so we can't assume that all types
involving a given class is a subtype of its rare type. Note that this is
only for uses that do not involve instantiation, as all instances of a
class must have runtime types that are a subtype of its rare type.

Keeping this in mind, here are the changes being made in this CL.

-----

In the flow graph builder when producting unoptimized code, we
previously assumed that if the checked type is not equal to the rare
type, then the code can't use _simpleInstanceOf, which only performs
class ID checks.  However, if the checked type is a supertype of the
rare type, we can, because any instance is guaranteed to be a subtype of
the checked type.

Note that this didn't cause incorrect behavior, just more work than
needed in some cases.

-----

In the generation of optimized type testing stubs, we assumed that the
null type argument vector (TAV) is a possible instance TAV for any
instance. However, if the rare type for a class has a non-null TAV, then
no instance of that class can have a null TAV and we can skip the null
check. (We keep it in the DEBUG case, but just to immediately trigger
a breakpoint if seen.)

Again, no incorrect behavior caused here previously, just an unnecessary
check that we now remove in some cases.

------

Also when generating optimized type testing stubs, when checking for the
null TAV prior to instance type argument checks, the code assumed that
it was possible to have to check type arguments if the checked type was
a super type of the rare type for the instance's class.  However, that's
backwards: if the checked type is a super type, then the runtime type of
any instance of the class is guaranteed to be a subtype and we shouldn't
be checking type arguments at all.

Thus, add an ASSERT that checks this, and instead only generate the code
that goes to the runtime if the null TAV is seen, as the runtime type of
the instance is the rare type and the rare type is guaranteed to not be
a subtype of the checked type.

Again, the previous version of this wouldn't have caused incorrect
behavior either, because the VM should never generate type arguments
checking in the type testing stub if the checked type is a supertype of
the class's rare type. Thus, that branch in the code generation was
just dead code.

-----

TEST=vm/cc/TTS, ci (especially DEBUG bots)

Issue: https://github.com/dart-lang/sdk/issues/52848
Cq-Include-Trybots: luci.dart.try:vm-aot-linux-debug-x64-try,vm-linux-debug-x64-try
Change-Id: I86d159693d6218f88dd1f04dd34cba702744b4d2
Fixed: 52848
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/312500
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Tess Strickland <sstrickl@google.com>