[vm, gc] Mark through new-space.
- Initial and final marking no longer visit all of new-space, reducing the STW pause for major GC.
- A scavenge during concurrent marking must forward / filter objects in the marking worklist that are moved / collected, increasing the STW pause for minor GC.
- Unreachable intergenerational cycles and weak references are collected in the next mark-sweep instead of first requiring enough scavenges to promote the whole cycle or weak target into old-space.
- Artificial minor GCs are no longer needed to avoid memory leaks from back-to-back major GCs.
- reachabilityBarrier is now just a count of major GCs.
TEST=ci
Change-Id: I1e653c9b5d3e02e45b280302c832157a75788db6
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/345350
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Siva Annamalai <asiva@google.com>
diff --git a/runtime/docs/gc.md b/runtime/docs/gc.md
index fc55bba..c7169ee 100644
--- a/runtime/docs/gc.md
+++ b/runtime/docs/gc.md
@@ -57,17 +57,9 @@
All objects have a bit in their header called the mark bit. At the start of a collection cycle, all objects have this bit clear.
-During the marking phase, the collector visits each of the root pointers. If the target object is an old-space object and its mark bit is clear, the mark bit is set and the target added to the marking stack (grey set). The collector then removes and visits objects in the marking stack, marking more old-space objects and adding them to the marking stack, until the marking stack is empty. At this point, all reachable objects have their mark bits set and all unreachable objects have their mark bits clear.
+During the marking phase, the collector visits each of the root pointers. If the target object has its mark bit clear, the mark bit is set and the target added to the marking stack (grey set). The collector then removes and visits objects in the marking stack, marking more objects and adding them to the marking stack, until the marking stack is empty. At this point, all reachable objects have their mark bits set and all unreachable objects have their mark bits clear.
-During the sweeping phase, the collector visits each old-space object. If the mark bit is clear, the object's memory is added to a [free list](https://github.com/dart-lang/sdk/blob/main/runtime/vm/heap/freelist.h) to be used for future allocations. Otherwise the object's mark bit is cleared. If every object on some page is unreachable, the page is released to the OS.
-
-### New-Space as Roots
-
-We do not mark new-space objects, and pointers to new-space objects are ignored; instead all objects in new-space are treated as part of the root set.
-
-This has the advantage of making collections of the two spaces more independent. In particular, the concurrent marker never needs to dereference any memory in new-space, avoiding several data race issues, and avoiding the need to pause or otherwise synchronize with the concurrent marker when starting a scavenge.
-
-It has the disadvantage that no single collection will collect all garbage. An unreachable old-space object that is referenced by an unreachable new-space object will not be collected until a scavenge first collects the new-space object, and unreachable objects that have a generation-crossing cycle will not be collected until the whole subgraph is promoted into old-space. The growth policy must be careful to ensure it doesn't perform old-space collections without interleaving new-space collections, such as when the program performs mostly large allocation that go directly to old-space, or old-space can accumulate such floating garbage and grow without bound.
+During the sweeping phase, the collector visits each object. If the mark bit is clear, the object's memory is added to a [free list](https://github.com/dart-lang/sdk/blob/main/runtime/vm/heap/freelist.h) to be used for future allocations. Otherwise the object's mark bit is cleared. If every object on some page is unreachable, the page is released to the OS.
## Mark-Compact
@@ -103,17 +95,17 @@
```c++
enum HeaderBits {
...
- kOldAndNotMarkedBit, // Incremental barrier target.
+ kNotMarkedBit, // Incremental barrier target.
kNewBit, // Generational barrier target.
- kOldBit, // Incremental barrier source.
+ kAlwaysSetBit, // Incremental barrier source.
kOldAndNotRememberedBit, // Generational barrier source.
...
};
static constexpr intptr_t kGenerationalBarrierMask = 1 << kNewBit;
-static constexpr intptr_t kIncrementalBarrierMask = 1 << kOldAndNotMarkedBit;
+static constexpr intptr_t kIncrementalBarrierMask = 1 << kNotMarkedBit;
static constexpr intptr_t kBarrierOverlapShift = 2;
-COMPILE_ASSERT(kOldAndNotMarkedBit + kBarrierOverlapShift == kOldBit);
+COMPILE_ASSERT(kNotMarkedBit + kBarrierOverlapShift == kAlwaysSetBit);
COMPILE_ASSERT(kNewBit + kBarrierOverlapShift == kOldAndNotRememberedBit);
StorePointer(ObjectPtr source, ObjectPtr* slot, ObjectPtr target) {
diff --git a/runtime/platform/atomic.h b/runtime/platform/atomic.h
index 390aa62..13176ce 100644
--- a/runtime/platform/atomic.h
+++ b/runtime/platform/atomic.h
@@ -146,6 +146,27 @@
std::atomic<T> value_;
};
+template <typename T>
+static inline T LoadRelaxed(const T* ptr) {
+ static_assert(sizeof(std::atomic<T>) == sizeof(T));
+ return reinterpret_cast<const std::atomic<T>*>(ptr)->load(
+ std::memory_order_relaxed);
+}
+
+template <typename T>
+static inline T LoadAcquire(const T* ptr) {
+ static_assert(sizeof(std::atomic<T>) == sizeof(T));
+ return reinterpret_cast<const std::atomic<T>*>(ptr)->load(
+ std::memory_order_acquire);
+}
+
+template <typename T>
+static inline void StoreRelease(T* ptr, T value) {
+ static_assert(sizeof(std::atomic<T>) == sizeof(T));
+ reinterpret_cast<std::atomic<T>*>(ptr)->store(value,
+ std::memory_order_release);
+}
+
} // namespace dart
#endif // RUNTIME_PLATFORM_ATOMIC_H_
diff --git a/runtime/vm/app_snapshot.cc b/runtime/vm/app_snapshot.cc
index fe4eedb..b885272 100644
--- a/runtime/vm/app_snapshot.cc
+++ b/runtime/vm/app_snapshot.cc
@@ -875,8 +875,8 @@
tags = UntaggedObject::ClassIdTag::update(class_id, tags);
tags = UntaggedObject::SizeTag::update(size, tags);
tags = UntaggedObject::CanonicalBit::update(is_canonical, tags);
- tags = UntaggedObject::OldBit::update(true, tags);
- tags = UntaggedObject::OldAndNotMarkedBit::update(true, tags);
+ tags = UntaggedObject::AlwaysSetBit::update(true, tags);
+ tags = UntaggedObject::NotMarkedBit::update(true, tags);
tags = UntaggedObject::OldAndNotRememberedBit::update(true, tags);
tags = UntaggedObject::NewBit::update(false, tags);
raw->untag()->tags_ = tags;
diff --git a/runtime/vm/compiler/runtime_api.cc b/runtime/vm/compiler/runtime_api.cc
index 5a33326..e982c589 100644
--- a/runtime/vm/compiler/runtime_api.cc
+++ b/runtime/vm/compiler/runtime_api.cc
@@ -360,6 +360,8 @@
TranslateOffsetInWordsToHost(instance_size)) |
dart::UntaggedObject::ClassIdTag::encode(cid) |
dart::UntaggedObject::NewBit::encode(true) |
+ dart::UntaggedObject::AlwaysSetBit::encode(true) |
+ dart::UntaggedObject::NotMarkedBit::encode(true) |
dart::UntaggedObject::ImmutableBit::encode(
ShouldHaveImmutabilityBitSet(cid));
}
@@ -378,8 +380,7 @@
const word UntaggedObject::kOldAndNotRememberedBit =
dart::UntaggedObject::kOldAndNotRememberedBit;
-const word UntaggedObject::kOldAndNotMarkedBit =
- dart::UntaggedObject::kOldAndNotMarkedBit;
+const word UntaggedObject::kNotMarkedBit = dart::UntaggedObject::kNotMarkedBit;
const word UntaggedObject::kImmutableBit = dart::UntaggedObject::kImmutableBit;
diff --git a/runtime/vm/compiler/runtime_api.h b/runtime/vm/compiler/runtime_api.h
index fdd0615..5c6fbe0 100644
--- a/runtime/vm/compiler/runtime_api.h
+++ b/runtime/vm/compiler/runtime_api.h
@@ -418,7 +418,7 @@
static const word kCanonicalBit;
static const word kNewBit;
static const word kOldAndNotRememberedBit;
- static const word kOldAndNotMarkedBit;
+ static const word kNotMarkedBit;
static const word kImmutableBit;
static const word kSizeTagPos;
static const word kSizeTagSize;
@@ -1498,6 +1498,8 @@
static const word kBytesPerCardLog2;
static word card_table_offset();
+ static word original_top_offset();
+ static word original_end_offset();
};
class Heap : public AllStatic {
diff --git a/runtime/vm/compiler/runtime_offsets_extracted.h b/runtime/vm/compiler/runtime_offsets_extracted.h
index 8164f90..52bc367 100644
--- a/runtime/vm/compiler/runtime_offsets_extracted.h
+++ b/runtime/vm/compiler/runtime_offsets_extracted.h
@@ -199,6 +199,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x1c;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x20;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -913,6 +915,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -1629,6 +1633,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x1c;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x20;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -2342,6 +2348,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -3061,6 +3069,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -3777,6 +3787,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -4494,6 +4506,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x1c;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x20;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -5209,6 +5223,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -5921,6 +5937,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x1c;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x20;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -6627,6 +6645,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -7335,6 +7355,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x1c;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x20;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -8040,6 +8062,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -8751,6 +8775,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -9459,6 +9485,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -10168,6 +10196,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x1c;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x20;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -10875,6 +10905,8 @@
static constexpr dart::compiler::target::word
GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word Page_original_top_offset = 0x38;
+static constexpr dart::compiler::target::word Page_original_end_offset = 0x40;
static constexpr dart::compiler::target::word
CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word ICData_NumArgsTestedMask = 0x3;
@@ -11604,6 +11636,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x1c;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x20;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -12395,6 +12431,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -13193,6 +13233,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -13987,6 +14031,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -14781,6 +14829,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -15577,6 +15629,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x1c;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x20;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -16369,6 +16425,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -17157,6 +17217,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x1c;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x20;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -17939,6 +18003,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -18728,6 +18796,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -19513,6 +19585,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -20298,6 +20374,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -21085,6 +21165,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x4;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x10;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x1c;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x20;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x8;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
@@ -21868,6 +21952,10 @@
static constexpr dart::compiler::target::word
AOT_GrowableObjectArray_type_arguments_offset = 0x8;
static constexpr dart::compiler::target::word AOT_Page_card_table_offset = 0x20;
+static constexpr dart::compiler::target::word AOT_Page_original_top_offset =
+ 0x38;
+static constexpr dart::compiler::target::word AOT_Page_original_end_offset =
+ 0x40;
static constexpr dart::compiler::target::word
AOT_CallSiteData_arguments_descriptor_offset = 0x10;
static constexpr dart::compiler::target::word AOT_ICData_NumArgsTestedMask =
diff --git a/runtime/vm/compiler/runtime_offsets_list.h b/runtime/vm/compiler/runtime_offsets_list.h
index 3a62082..75cddea 100644
--- a/runtime/vm/compiler/runtime_offsets_list.h
+++ b/runtime/vm/compiler/runtime_offsets_list.h
@@ -160,6 +160,8 @@
FIELD(GrowableObjectArray, length_offset) \
FIELD(GrowableObjectArray, type_arguments_offset) \
FIELD(Page, card_table_offset) \
+ FIELD(Page, original_top_offset) \
+ FIELD(Page, original_end_offset) \
FIELD(CallSiteData, arguments_descriptor_offset) \
FIELD(ICData, NumArgsTestedMask) \
FIELD(ICData, NumArgsTestedShift) \
@@ -310,7 +312,6 @@
FIELD(Thread, return_async_not_future_stub_offset) \
FIELD(Thread, return_async_star_stub_offset) \
FIELD(Thread, return_async_stub_offset) \
- \
FIELD(Thread, object_null_offset) \
FIELD(Thread, predefined_symbols_address_offset) \
FIELD(Thread, resume_pc_offset) \
@@ -324,7 +325,6 @@
FIELD(Thread, stack_overflow_shared_with_fpu_regs_entry_point_offset) \
FIELD(Thread, stack_overflow_shared_with_fpu_regs_stub_offset) \
FIELD(Thread, stack_overflow_shared_without_fpu_regs_entry_point_offset) \
- \
FIELD(Thread, stack_overflow_shared_without_fpu_regs_stub_offset) \
FIELD(Thread, store_buffer_block_offset) \
FIELD(Thread, suspend_state_await_entry_point_offset) \
@@ -406,7 +406,6 @@
kNumberOfCpuRegisters - 1, [](Register reg) { \
return (kDartAvailableCpuRegs & (1 << reg)) != 0; \
}) \
- \
SIZEOF(AbstractType, InstanceSize, UntaggedAbstractType) \
SIZEOF(ApiError, InstanceSize, UntaggedApiError) \
SIZEOF(Array, header_size, UntaggedArray) \
diff --git a/runtime/vm/compiler/stub_code_compiler.cc b/runtime/vm/compiler/stub_code_compiler.cc
index db0b2a5..37d78dd 100644
--- a/runtime/vm/compiler/stub_code_compiler.cc
+++ b/runtime/vm/compiler/stub_code_compiler.cc
@@ -1870,7 +1870,7 @@
const Register kSrcFrame = SuspendStubABI::kSrcFrameReg;
const Register kDstFrame = SuspendStubABI::kDstFrameReg;
Label alloc_slow_case, alloc_done, init_done, resize_suspend_state,
- old_gen_object, call_dart;
+ remember_object, call_dart;
#if defined(TARGET_ARCH_ARM) || defined(TARGET_ARCH_ARM64)
SPILLS_LR_TO_FRAME({}); // Simulate entering the caller (Dart) frame.
@@ -1966,11 +1966,6 @@
}
#endif
- __ LoadFromOffset(
- kTemp, Address(FPREG, kSavedCallerPcSlotFromFp * target::kWordSize));
- __ StoreToOffset(
- kTemp, FieldAddress(kSuspendState, target::SuspendState::pc_offset()));
-
if (kSrcFrame == THR) {
__ PushRegister(THR);
}
@@ -1982,6 +1977,11 @@
__ PopRegister(THR);
}
+ __ LoadFromOffset(
+ kTemp, Address(FPREG, kSavedCallerPcSlotFromFp * target::kWordSize));
+ __ StoreToOffset(
+ kTemp, FieldAddress(kSuspendState, target::SuspendState::pc_offset()));
+
#ifdef DEBUG
{
// Verify that kSuspendState matches :suspend_state in the copied stack
@@ -2008,8 +2008,13 @@
}
// Write barrier.
- __ BranchIfBit(kSuspendState, target::ObjectAlignment::kNewObjectBitPosition,
- ZERO, &old_gen_object);
+ __ AndImmediate(kTemp, kSuspendState, target::kPageMask);
+ __ LoadFromOffset(kTemp, Address(kTemp, target::Page::original_top_offset()));
+ __ CompareRegisters(kSuspendState, kTemp);
+ __ BranchIf(UNSIGNED_LESS, &remember_object);
+ // Assumption: SuspendStates are always on non-image pages.
+ // TODO(rmacnak): Also check original_end if we bound TLABs to smaller than a
+ // heap page.
__ Bind(&call_dart);
if (call_suspend_function) {
@@ -2085,7 +2090,7 @@
__ PopRegister(kArgument); // Restore argument.
__ Jump(&alloc_done);
- __ Bind(&old_gen_object);
+ __ Bind(&remember_object);
__ Comment("Old gen SuspendState slow case");
if (!call_suspend_function) {
// Save kArgument which contains the return value
@@ -2559,6 +2564,7 @@
SuspendStateFpOffset()));
__ MoveRegister(CallingConventions::kReturnReg, kDestination);
+ EnsureIsNewOrRemembered();
__ Ret();
__ Bind(&alloc_slow_case);
diff --git a/runtime/vm/compiler/stub_code_compiler_arm.cc b/runtime/vm/compiler/stub_code_compiler_arm.cc
index aff561e..19191bd 100644
--- a/runtime/vm/compiler/stub_code_compiler_arm.cc
+++ b/runtime/vm/compiler/stub_code_compiler_arm.cc
@@ -37,11 +37,14 @@
// WARNING: This might clobber all registers except for [R0], [THR] and [FP].
// The caller should simply call LeaveStubFrame() and return.
void StubCodeCompiler::EnsureIsNewOrRemembered() {
- // If the object is not remembered we call a leaf-runtime to add it to the
- // remembered set.
+ // If the object is not in an active TLAB, we call a leaf-runtime to add it to
+ // the remembered set and/or deferred marking worklist. This test assumes a
+ // Page's TLAB use is always ascending.
Label done;
- __ tst(R0, Operand(1 << target::ObjectAlignment::kNewObjectBitPosition));
- __ BranchIf(NOT_ZERO, &done);
+ __ AndImmediate(TMP, R0, target::kPageMask);
+ __ LoadFromOffset(TMP, Address(TMP, target::Page::original_top_offset()));
+ __ CompareRegisters(R0, TMP);
+ __ BranchIf(UNSIGNED_GREATER_EQUAL, &done);
{
LeafRuntimeScope rt(assembler,
@@ -1680,16 +1683,16 @@
__ b(&skip_marking, ZERO);
{
- // Atomically clear kOldAndNotMarkedBit.
+ // Atomically clear kNotMarkedBit.
Label retry, done;
__ PushList((1 << R2) | (1 << R3) | (1 << R4)); // Spill.
__ AddImmediate(R3, R0, target::Object::tags_offset() - kHeapObjectTag);
// R3: Untagged address of header word (ldrex/strex do not support offsets).
__ Bind(&retry);
__ ldrex(R2, R3);
- __ tst(R2, Operand(1 << target::UntaggedObject::kOldAndNotMarkedBit));
+ __ tst(R2, Operand(1 << target::UntaggedObject::kNotMarkedBit));
__ b(&done, ZERO); // Marked by another thread.
- __ bic(R2, R2, Operand(1 << target::UntaggedObject::kOldAndNotMarkedBit));
+ __ bic(R2, R2, Operand(1 << target::UntaggedObject::kNotMarkedBit));
__ strex(R4, R2, R3);
__ cmp(R4, Operand(1));
__ b(&retry, EQ);
@@ -1713,7 +1716,6 @@
__ Bind(&done);
__ clrex();
__ PopList((1 << R2) | (1 << R3) | (1 << R4)); // Unspill.
- __ Ret();
}
Label add_to_remembered_set, remember_card;
diff --git a/runtime/vm/compiler/stub_code_compiler_arm64.cc b/runtime/vm/compiler/stub_code_compiler_arm64.cc
index 4e3b2df..c633767 100644
--- a/runtime/vm/compiler/stub_code_compiler_arm64.cc
+++ b/runtime/vm/compiler/stub_code_compiler_arm64.cc
@@ -36,10 +36,14 @@
// WARNING: This might clobber all registers except for [R0], [THR] and [FP].
// The caller should simply call LeaveStubFrame() and return.
void StubCodeCompiler::EnsureIsNewOrRemembered() {
- // If the object is not remembered we call a leaf-runtime to add it to the
- // remembered set.
+ // If the object is not in an active TLAB, we call a leaf-runtime to add it to
+ // the remembered set and/or deferred marking worklist. This test assumes a
+ // Page's TLAB use is always ascending.
Label done;
- __ tbnz(&done, R0, target::ObjectAlignment::kNewObjectBitPosition);
+ __ AndImmediate(TMP, R0, target::kPageMask);
+ __ LoadFromOffset(TMP, Address(TMP, target::Page::original_top_offset()));
+ __ CompareRegisters(R0, TMP);
+ __ BranchIf(UNSIGNED_GREATER_EQUAL, &done);
{
LeafRuntimeScope rt(assembler, /*frame_size=*/0,
@@ -1989,21 +1993,20 @@
__ b(&skip_marking, ZERO);
{
- // Atomically clear kOldAndNotMarkedBit.
+ // Atomically clear kNotMarkedBit.
Label retry, done;
__ PushRegisters(spill_set);
__ AddImmediate(R3, R0, target::Object::tags_offset() - kHeapObjectTag);
// R3: Untagged address of header word (atomics do not support offsets).
if (TargetCPUFeatures::atomic_memory_supported()) {
- __ LoadImmediate(TMP, 1 << target::UntaggedObject::kOldAndNotMarkedBit);
+ __ LoadImmediate(TMP, 1 << target::UntaggedObject::kNotMarkedBit);
__ ldclr(TMP, TMP, R3);
- __ tbz(&done, TMP, target::UntaggedObject::kOldAndNotMarkedBit);
+ __ tbz(&done, TMP, target::UntaggedObject::kNotMarkedBit);
} else {
__ Bind(&retry);
__ ldxr(R2, R3, kEightBytes);
- __ tbz(&done, R2, target::UntaggedObject::kOldAndNotMarkedBit);
- __ AndImmediate(R2, R2,
- ~(1 << target::UntaggedObject::kOldAndNotMarkedBit));
+ __ tbz(&done, R2, target::UntaggedObject::kNotMarkedBit);
+ __ AndImmediate(R2, R2, ~(1 << target::UntaggedObject::kNotMarkedBit));
__ stxr(R4, R2, R3, kEightBytes);
__ cbnz(&retry, R4);
}
diff --git a/runtime/vm/compiler/stub_code_compiler_ia32.cc b/runtime/vm/compiler/stub_code_compiler_ia32.cc
index d53eccf..9124b57 100644
--- a/runtime/vm/compiler/stub_code_compiler_ia32.cc
+++ b/runtime/vm/compiler/stub_code_compiler_ia32.cc
@@ -35,11 +35,14 @@
// WARNING: This might clobber all registers except for [EAX], [THR] and [FP].
// The caller should simply call LeaveFrame() and return.
void StubCodeCompiler::EnsureIsNewOrRemembered() {
- // If the object is not remembered we call a leaf-runtime to add it to the
- // remembered set.
+ // If the object is not in an active TLAB, we call a leaf-runtime to add it to
+ // the remembered set and/or deferred marking worklist. This test assumes a
+ // Page's TLAB use is always ascending.
Label done;
- __ testl(EAX, Immediate(1 << target::ObjectAlignment::kNewObjectBitPosition));
- __ BranchIf(NOT_ZERO, &done);
+ __ AndImmediate(ECX, EAX, target::kPageMask);
+ __ LoadFromOffset(ECX, Address(ECX, target::Page::original_top_offset()));
+ __ CompareRegisters(EAX, ECX);
+ __ BranchIf(UNSIGNED_GREATER_EQUAL, &done);
{
LeafRuntimeScope rt(assembler,
@@ -1414,15 +1417,14 @@
__ j(ZERO, &skip_marking);
{
- // Atomically clear kOldAndNotMarkedBit.
+ // Atomically clear kNotMarkedBit.
Label retry, done;
__ movl(EAX, FieldAddress(EBX, target::Object::tags_offset()));
__ Bind(&retry);
__ movl(ECX, EAX);
- __ testl(ECX, Immediate(1 << target::UntaggedObject::kOldAndNotMarkedBit));
+ __ testl(ECX, Immediate(1 << target::UntaggedObject::kNotMarkedBit));
__ j(ZERO, &done); // Marked by another thread.
- __ andl(ECX,
- Immediate(~(1 << target::UntaggedObject::kOldAndNotMarkedBit)));
+ __ andl(ECX, Immediate(~(1 << target::UntaggedObject::kNotMarkedBit)));
// Cmpxchgq: compare value = implicit operand EAX, new value = ECX.
// On failure, EAX is updated with the current value.
__ LockCmpxchgl(FieldAddress(EBX, target::Object::tags_offset()), ECX);
diff --git a/runtime/vm/compiler/stub_code_compiler_riscv.cc b/runtime/vm/compiler/stub_code_compiler_riscv.cc
index 534560d..5bd9423 100644
--- a/runtime/vm/compiler/stub_code_compiler_riscv.cc
+++ b/runtime/vm/compiler/stub_code_compiler_riscv.cc
@@ -36,11 +36,14 @@
// WARNING: This might clobber all registers except for [A0], [THR] and [FP].
// The caller should simply call LeaveStubFrame() and return.
void StubCodeCompiler::EnsureIsNewOrRemembered() {
- // If the object is not remembered we call a leaf-runtime to add it to the
- // remembered set.
+ // If the object is not in an active TLAB, we call a leaf-runtime to add it to
+ // the remembered set and/or deferred marking worklist. This test assumes a
+ // Page's TLAB use is always ascending.
Label done;
- __ andi(TMP2, A0, 1 << target::ObjectAlignment::kNewObjectBitPosition);
- __ bnez(TMP2, &done);
+ __ AndImmediate(TMP, A0, target::kPageMask);
+ __ LoadFromOffset(TMP, Address(TMP, target::Page::original_top_offset()));
+ __ CompareRegisters(A0, TMP);
+ __ BranchIf(UNSIGNED_GREATER_EQUAL, &done);
{
LeafRuntimeScope rt(assembler, /*frame_size=*/0,
@@ -1801,18 +1804,18 @@
__ beqz(TMP, &skip_marking);
{
- // Atomically clear kOldAndNotMarkedBit.
+ // Atomically clear kNotMarkedBit.
Label done;
__ PushRegisters(spill_set);
__ addi(T3, A1, target::Object::tags_offset() - kHeapObjectTag);
// T3: Untagged address of header word (amo's do not support offsets).
- __ li(TMP2, ~(1 << target::UntaggedObject::kOldAndNotMarkedBit));
+ __ li(TMP2, ~(1 << target::UntaggedObject::kNotMarkedBit));
#if XLEN == 32
__ amoandw(TMP2, TMP2, Address(T3, 0));
#else
__ amoandd(TMP2, TMP2, Address(T3, 0));
#endif
- __ andi(TMP2, TMP2, 1 << target::UntaggedObject::kOldAndNotMarkedBit);
+ __ andi(TMP2, TMP2, 1 << target::UntaggedObject::kNotMarkedBit);
__ beqz(TMP2, &done); // Was already clear -> lost race.
__ lx(T4, Address(THR, target::Thread::marking_stack_block_offset()));
diff --git a/runtime/vm/compiler/stub_code_compiler_x64.cc b/runtime/vm/compiler/stub_code_compiler_x64.cc
index 69b714e..1d65261 100644
--- a/runtime/vm/compiler/stub_code_compiler_x64.cc
+++ b/runtime/vm/compiler/stub_code_compiler_x64.cc
@@ -39,11 +39,14 @@
// WARNING: This might clobber all registers except for [RAX], [THR] and [FP].
// The caller should simply call LeaveStubFrame() and return.
void StubCodeCompiler::EnsureIsNewOrRemembered() {
- // If the object is not remembered we call a leaf-runtime to add it to the
- // remembered set.
+ // If the object is not in an active TLAB, we call a leaf-runtime to add it to
+ // the remembered set and/or deferred marking worklist. This test assumes a
+ // Page's TLAB use is always ascending.
Label done;
- __ testq(RAX, Immediate(1 << target::ObjectAlignment::kNewObjectBitPosition));
- __ BranchIf(NOT_ZERO, &done);
+ __ AndImmediate(TMP, RAX, target::kPageMask);
+ __ LoadFromOffset(TMP, Address(TMP, target::Page::original_top_offset()));
+ __ CompareRegisters(RAX, TMP);
+ __ BranchIf(UNSIGNED_GREATER_EQUAL, &done);
{
LeafRuntimeScope rt(assembler, /*frame_size=*/0,
@@ -1910,7 +1913,7 @@
__ j(ZERO, &skip_marking);
{
- // Atomically clear kOldAndNotMarkedBit.
+ // Atomically clear kNotMarkedBit.
Label retry, done;
__ pushq(RAX); // Spill.
__ pushq(RCX); // Spill.
@@ -1919,11 +1922,10 @@
__ Bind(&retry);
__ movq(RCX, RAX);
- __ testq(RCX, Immediate(1 << target::UntaggedObject::kOldAndNotMarkedBit));
+ __ testq(RCX, Immediate(1 << target::UntaggedObject::kNotMarkedBit));
__ j(ZERO, &done); // Marked by another thread.
- __ andq(RCX,
- Immediate(~(1 << target::UntaggedObject::kOldAndNotMarkedBit)));
+ __ andq(RCX, Immediate(~(1 << target::UntaggedObject::kNotMarkedBit)));
// Cmpxchgq: compare value = implicit operand RAX, new value = RCX.
// On failure, RAX is updated with the current value.
__ LockCmpxchgq(FieldAddress(TMP, target::Object::tags_offset()), RCX);
diff --git a/runtime/vm/dart_api_impl_test.cc b/runtime/vm/dart_api_impl_test.cc
index ab3c036..178f73a 100644
--- a/runtime/vm/dart_api_impl_test.cc
+++ b/runtime/vm/dart_api_impl_test.cc
@@ -1766,9 +1766,6 @@
TransitionNativeToVM transition(thread);
EXPECT_EQ(40, peer8);
EXPECT_EQ(41, peer16);
- GCTestHelper::CollectOldSpace();
- EXPECT_EQ(40, peer8);
- EXPECT_EQ(41, peer16);
GCTestHelper::CollectNewSpace();
EXPECT_EQ(80, peer8);
EXPECT_EQ(82, peer16);
@@ -3283,8 +3280,6 @@
{
TransitionNativeToVM transition(thread);
EXPECT(peer == 0);
- GCTestHelper::CollectOldSpace();
- EXPECT(peer == 0);
GCTestHelper::CollectNewSpace();
EXPECT(peer == 42);
}
@@ -4062,8 +4057,6 @@
}
{
TransitionNativeToVM transition(thread);
- GCTestHelper::CollectOldSpace();
- EXPECT(peer == 0);
GCTestHelper::CollectNewSpace();
EXPECT(peer == 42);
}
@@ -4087,8 +4080,6 @@
}
{
TransitionNativeToVM transition(thread);
- GCTestHelper::CollectOldSpace();
- EXPECT(peer == 0);
GCTestHelper::CollectNewSpace();
EXPECT(peer == 42);
}
@@ -4176,8 +4167,6 @@
}
{
TransitionNativeToVM transition(thread);
- GCTestHelper::CollectOldSpace();
- EXPECT(peer == 0);
GCTestHelper::CollectNewSpace();
EXPECT(peer == 42);
ASSERT(delete_on_finalization == nullptr);
@@ -4212,8 +4201,8 @@
TEST_CASE(DartAPI_WeakPersistentHandleExternalAllocationSize) {
Heap* heap = IsolateGroup::Current()->heap();
- EXPECT(heap->ExternalInWords(Heap::kNew) == 0);
- EXPECT(heap->ExternalInWords(Heap::kOld) == 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew), 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
Dart_WeakPersistentHandle weak1 = nullptr;
const intptr_t kWeak1ExternalSize = 1 * KB;
{
@@ -4241,20 +4230,26 @@
}
{
TransitionNativeToVM transition(thread);
- GCTestHelper::CollectOldSpace();
- EXPECT(heap->ExternalInWords(Heap::kNew) ==
- (kWeak1ExternalSize + kWeak2ExternalSize) / kWordSize);
- // Collect weakly referenced string, and promote strongly referenced string.
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew),
+ (kWeak1ExternalSize + kWeak2ExternalSize) / kWordSize);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
+ // Collect weakly referenced string.
GCTestHelper::CollectNewSpace();
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew),
+ kWeak2ExternalSize / kWordSize);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
+ // Promote strongly referenced string.
GCTestHelper::CollectNewSpace();
- EXPECT(heap->ExternalInWords(Heap::kNew) == 0);
- EXPECT(heap->ExternalInWords(Heap::kOld) == kWeak2ExternalSize / kWordSize);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew), 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld),
+ kWeak2ExternalSize / kWordSize);
}
Dart_DeletePersistentHandle(strong_ref);
{
TransitionNativeToVM transition(thread);
GCTestHelper::CollectOldSpace();
- EXPECT(heap->ExternalInWords(Heap::kOld) == 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew), 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
}
Dart_DeleteWeakPersistentHandle(weak1);
Dart_DeleteWeakPersistentHandle(weak2);
@@ -4262,8 +4257,8 @@
TEST_CASE(DartAPI_FinalizableHandleExternalAllocationSize) {
Heap* heap = IsolateGroup::Current()->heap();
- EXPECT(heap->ExternalInWords(Heap::kNew) == 0);
- EXPECT(heap->ExternalInWords(Heap::kOld) == 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew), 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
const intptr_t kWeak1ExternalSize = 1 * KB;
{
Dart_EnterScope();
@@ -4285,20 +4280,26 @@
}
{
TransitionNativeToVM transition(thread);
- GCTestHelper::CollectOldSpace();
- EXPECT(heap->ExternalInWords(Heap::kNew) ==
- (kWeak1ExternalSize + kWeak2ExternalSize) / kWordSize);
- // Collect weakly referenced string, and promote strongly referenced string.
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew),
+ (kWeak1ExternalSize + kWeak2ExternalSize) / kWordSize);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
+ // Collect weakly referenced string.
GCTestHelper::CollectNewSpace();
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew),
+ kWeak2ExternalSize / kWordSize);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
+ // Promote strongly referenced string.
GCTestHelper::CollectNewSpace();
- EXPECT(heap->ExternalInWords(Heap::kNew) == 0);
- EXPECT(heap->ExternalInWords(Heap::kOld) == kWeak2ExternalSize / kWordSize);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew), 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld),
+ kWeak2ExternalSize / kWordSize);
}
Dart_DeletePersistentHandle(strong_ref);
{
TransitionNativeToVM transition(thread);
GCTestHelper::CollectOldSpace();
- EXPECT(heap->ExternalInWords(Heap::kOld) == 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kNew), 0);
+ EXPECT_EQ(heap->ExternalInWords(Heap::kOld), 0);
}
}
@@ -4753,20 +4754,6 @@
Dart_ExitScope();
}
- {
- TransitionNativeToVM transition(thread);
- GCTestHelper::CollectOldSpace();
- }
-
- {
- Dart_EnterScope();
- // Old space collection should not affect old space objects.
- EXPECT(!Dart_IsNull(AsHandle(weak1)));
- EXPECT(!Dart_IsNull(AsHandle(weak2)));
- EXPECT(!Dart_IsNull(AsHandle(weak3)));
- Dart_ExitScope();
- }
-
Dart_DeleteWeakPersistentHandle(strong_weak);
Dart_DeleteWeakPersistentHandle(weak1);
Dart_DeleteWeakPersistentHandle(weak2);
diff --git a/runtime/vm/heap/become.cc b/runtime/vm/heap/become.cc
index 875798a..e219084 100644
--- a/runtime/vm/heap/become.cc
+++ b/runtime/vm/heap/become.cc
@@ -27,8 +27,7 @@
tags = UntaggedObject::SizeTag::update(size, tags);
tags = UntaggedObject::ClassIdTag::update(kForwardingCorpse, tags);
bool is_old = (addr & kNewObjectAlignmentOffset) == kOldObjectAlignmentOffset;
- tags = UntaggedObject::OldBit::update(is_old, tags);
- tags = UntaggedObject::OldAndNotMarkedBit::update(is_old, tags);
+ tags = UntaggedObject::NotMarkedBit::update(true, tags);
tags = UntaggedObject::OldAndNotRememberedBit::update(is_old, tags);
tags = UntaggedObject::NewBit::update(!is_old, tags);
diff --git a/runtime/vm/heap/freelist.cc b/runtime/vm/heap/freelist.cc
index b3767b9..1be1306 100644
--- a/runtime/vm/heap/freelist.cc
+++ b/runtime/vm/heap/freelist.cc
@@ -25,8 +25,8 @@
tags = UntaggedObject::SizeTag::update(size, tags);
tags = UntaggedObject::ClassIdTag::update(kFreeListElement, tags);
ASSERT((addr & kNewObjectAlignmentOffset) == kOldObjectAlignmentOffset);
- tags = UntaggedObject::OldBit::update(true, tags);
- tags = UntaggedObject::OldAndNotMarkedBit::update(true, tags);
+ tags = UntaggedObject::AlwaysSetBit::update(true, tags);
+ tags = UntaggedObject::NotMarkedBit::update(true, tags);
tags = UntaggedObject::OldAndNotRememberedBit::update(true, tags);
tags = UntaggedObject::NewBit::update(false, tags);
result->tags_ = tags;
@@ -40,6 +40,29 @@
// writable.
}
+FreeListElement* FreeListElement::AsElementNew(uword addr, intptr_t size) {
+ ASSERT(size >= kObjectAlignment);
+ ASSERT(Utils::IsAligned(size, kObjectAlignment));
+
+ FreeListElement* result = reinterpret_cast<FreeListElement*>(addr);
+
+ uword tags = 0;
+ tags = UntaggedObject::SizeTag::update(size, tags);
+ tags = UntaggedObject::ClassIdTag::update(kFreeListElement, tags);
+ ASSERT((addr & kNewObjectAlignmentOffset) == kNewObjectAlignmentOffset);
+ tags = UntaggedObject::AlwaysSetBit::update(true, tags);
+ tags = UntaggedObject::NotMarkedBit::update(true, tags);
+ tags = UntaggedObject::OldAndNotRememberedBit::update(false, tags);
+ tags = UntaggedObject::NewBit::update(true, tags);
+ result->tags_ = tags;
+
+ if (size > UntaggedObject::SizeTag::kMaxSizeTag) {
+ *result->SizeAddress() = size;
+ }
+ result->set_next(nullptr);
+ return result;
+}
+
void FreeListElement::Init() {
ASSERT(sizeof(FreeListElement) == kObjectAlignment);
ASSERT(OFFSET_OF(FreeListElement, tags_) == Object::tags_offset());
diff --git a/runtime/vm/heap/freelist.h b/runtime/vm/heap/freelist.h
index e43981e..c010055 100644
--- a/runtime/vm/heap/freelist.h
+++ b/runtime/vm/heap/freelist.h
@@ -37,6 +37,7 @@
}
static FreeListElement* AsElement(uword addr, intptr_t size);
+ static FreeListElement* AsElementNew(uword addr, intptr_t size);
static void Init();
diff --git a/runtime/vm/heap/heap.cc b/runtime/vm/heap/heap.cc
index 29a5bff..19de6f1 100644
--- a/runtime/vm/heap/heap.cc
+++ b/runtime/vm/heap/heap.cc
@@ -49,7 +49,6 @@
new_space_(this, max_new_gen_semi_words),
old_space_(this, max_old_gen_words),
read_only_(false),
- last_gc_was_old_space_(false),
assume_scavenge_will_fail_(false),
gc_on_nth_allocation_(kNoForcedGarbageCollection) {
UpdateGlobalMaxUsed();
@@ -58,8 +57,6 @@
old_weak_tables_[sel] = new WeakTable();
}
stats_.num_ = 0;
- stats_.state_ = kInitial;
- stats_.reachability_barrier_ = 0;
}
Heap::~Heap() {
@@ -140,7 +137,7 @@
}
// All GC tasks finished without allocating successfully. Collect both
// generations.
- CollectMostGarbage(GCReason::kOldSpace, /*compact=*/false);
+ CollectOldSpaceGarbage(thread, GCType::kMarkSweep, GCReason::kOldSpace);
addr = old_space_.TryAllocate(size, is_exec);
if (addr != 0) {
return addr;
@@ -157,7 +154,7 @@
return addr;
}
// Before throwing an out-of-memory error try a synchronous GC.
- CollectAllGarbage(GCReason::kOldSpace, /*compact=*/true);
+ CollectOldSpaceGarbage(thread, GCType::kMarkCompact, GCReason::kOldSpace);
WaitForSweeperTasksAtSafepoint(thread);
}
uword addr = old_space_.TryAllocate(size, is_exec, PageSpace::kForceGrowth);
@@ -232,9 +229,6 @@
}
if (old_space_.ReachedHardThreshold()) {
- if (last_gc_was_old_space_) {
- CollectNewSpaceGarbage(thread, GCType::kScavenge, GCReason::kFull);
- }
CollectGarbage(thread, GCType::kMarkSweep, GCReason::kExternal);
} else {
CheckConcurrentMarking(thread, GCReason::kExternal, 0);
@@ -480,7 +474,6 @@
#if defined(SUPPORT_TIMELINE)
PrintStatsToTimeline(&tbes, reason);
#endif
- last_gc_was_old_space_ = false;
}
if (type == GCType::kScavenge && reason == GCReason::kNewSpace) {
if (old_space_.ReachedHardThreshold()) {
@@ -547,7 +540,6 @@
isolate->catch_entry_moves_cache()->Clear();
},
/*at_safepoint=*/true);
- last_gc_was_old_space_ = true;
assume_scavenge_will_fail_ = false;
}
}
@@ -567,19 +559,8 @@
}
}
-void Heap::CollectMostGarbage(GCReason reason, bool compact) {
- Thread* thread = Thread::Current();
- CollectNewSpaceGarbage(thread, GCType::kScavenge, reason);
- CollectOldSpaceGarbage(
- thread, compact ? GCType::kMarkCompact : GCType::kMarkSweep, reason);
-}
-
void Heap::CollectAllGarbage(GCReason reason, bool compact) {
Thread* thread = Thread::Current();
-
- // New space is evacuated so this GC will collect all dead objects
- // kept alive by a cross-generational pointer.
- CollectNewSpaceGarbage(thread, GCType::kEvacuate, reason);
if (thread->is_marking()) {
// If incremental marking is happening, we need to finish the GC cycle
// and perform a follow-up GC to purge any "floating garbage" that may be
@@ -624,19 +605,6 @@
return;
case PageSpace::kDone:
if (old_space_.ReachedSoftThreshold()) {
- // New-space objects are roots during old-space GC. This means that even
- // unreachable new-space objects prevent old-space objects they
- // reference from being collected during an old-space GC. Normally this
- // is not an issue because new-space GCs run much more frequently than
- // old-space GCs. If new-space allocation is low and direct old-space
- // allocation is high, which can happen in a program that allocates
- // large objects and little else, old-space can fill up with unreachable
- // objects until the next new-space GC. This check is the
- // concurrent-marking equivalent to the new-space GC before
- // synchronous-marking in CollectMostGarbage.
- if (last_gc_was_old_space_) {
- CollectNewSpaceGarbage(thread, GCType::kScavenge, GCReason::kFull);
- }
StartConcurrentMarking(thread, reason);
}
return;
diff --git a/runtime/vm/heap/heap.h b/runtime/vm/heap/heap.h
index 36898d9..95b0db4 100644
--- a/runtime/vm/heap/heap.h
+++ b/runtime/vm/heap/heap.h
@@ -54,20 +54,6 @@
kNumWeakSelectors
};
- // States for a state machine that represents the worst-case set of GCs
- // that an unreachable object could survive before begin collected:
- // a new-space object that is involved with a cycle with an old-space object
- // is copied to survivor space, then promoted during concurrent marking,
- // and finally proven unreachable in the next round of old-gen marking.
- // We ignore the case of unreachable-but-not-yet-collected objects being
- // made reachable again by allInstances.
- enum LeakCountState {
- kInitial = 0,
- kFirstScavenge,
- kSecondScavenge,
- kMarkingStart,
- };
-
// Pattern for unused new space and swept old space.
static constexpr uint8_t kZapByte = 0xf3;
@@ -120,18 +106,10 @@
// Collect a single generation.
void CollectGarbage(Thread* thread, GCType type, GCReason reason);
- // Collect both generations by performing a scavenge followed by a
- // mark-sweep. This function may not collect all unreachable objects. Because
- // mark-sweep treats new space as roots, a cycle between unreachable old and
- // new objects will not be collected until the new objects are promoted.
- // Verification based on heap iteration should instead use CollectAllGarbage.
- void CollectMostGarbage(GCReason reason = GCReason::kFull,
- bool compact = false);
-
- // Collect both generations by performing an evacuation followed by a
- // mark-sweep. If incremental marking was in progress, perform another
- // mark-sweep. This function will collect all unreachable objects, including
- // those in inter-generational cycles or stored during incremental marking.
+ // Collect both generations by performing a mark-sweep. If incremental marking
+ // was in progress, perform another mark-sweep. This function will collect all
+ // unreachable objects, including those in inter-generational cycles or stored
+ // during incremental marking.
void CollectAllGarbage(GCReason reason = GCReason::kFull,
bool compact = false);
@@ -290,7 +268,7 @@
}
#endif // PRODUCT
- intptr_t ReachabilityBarrier() { return stats_.reachability_barrier_; }
+ intptr_t ReachabilityBarrier() { return old_space_.collections(); }
IsolateGroup* isolate_group() const { return isolate_group_; }
bool is_vm_isolate() const { return is_vm_isolate_; }
@@ -310,8 +288,6 @@
intptr_t num_;
GCType type_;
GCReason reason_;
- LeakCountState state_; // State to track finalization of GCed object.
- intptr_t reachability_barrier_; // Tracks reachability of GCed objects.
class Data : public ValueObject {
public:
@@ -388,7 +364,6 @@
// This heap is in read-only mode: No allocation is allowed.
bool read_only_;
- bool last_gc_was_old_space_;
bool assume_scavenge_will_fail_;
static constexpr intptr_t kNoForcedGarbageCollection = -1;
diff --git a/runtime/vm/heap/heap_test.cc b/runtime/vm/heap/heap_test.cc
index cc1ff25..811d5a0 100644
--- a/runtime/vm/heap/heap_test.cc
+++ b/runtime/vm/heap/heap_test.cc
@@ -980,9 +980,9 @@
WeakProperty_Generations(kNew, kNew, kNew, true, true, true);
WeakProperty_Generations(kNew, kNew, kOld, true, true, true);
WeakProperty_Generations(kNew, kNew, kImm, true, true, true);
- WeakProperty_Generations(kNew, kOld, kNew, false, false, true);
- WeakProperty_Generations(kNew, kOld, kOld, false, false, true);
- WeakProperty_Generations(kNew, kOld, kImm, false, false, true);
+ WeakProperty_Generations(kNew, kOld, kNew, false, true, true);
+ WeakProperty_Generations(kNew, kOld, kOld, false, true, true);
+ WeakProperty_Generations(kNew, kOld, kImm, false, true, true);
WeakProperty_Generations(kNew, kImm, kNew, false, false, false);
WeakProperty_Generations(kNew, kImm, kOld, false, false, false);
WeakProperty_Generations(kNew, kImm, kImm, false, false, false);
@@ -1061,7 +1061,7 @@
FLAG_early_tenuring_threshold = 100; // I.e., off.
WeakReference_Generations(kNew, kNew, true, true, true);
- WeakReference_Generations(kNew, kOld, false, false, true);
+ WeakReference_Generations(kNew, kOld, false, true, true);
WeakReference_Generations(kNew, kImm, false, false, false);
WeakReference_Generations(kOld, kNew, true, true, true);
WeakReference_Generations(kOld, kOld, false, true, true);
@@ -1132,7 +1132,7 @@
FLAG_early_tenuring_threshold = 100; // I.e., off.
WeakArray_Generations(kNew, kNew, true, true, true);
- WeakArray_Generations(kNew, kOld, false, false, true);
+ WeakArray_Generations(kNew, kOld, false, true, true);
WeakArray_Generations(kNew, kImm, false, false, false);
WeakArray_Generations(kOld, kNew, true, true, true);
WeakArray_Generations(kOld, kOld, false, true, true);
@@ -1203,7 +1203,7 @@
FLAG_early_tenuring_threshold = 100; // I.e., off.
FinalizerEntry_Generations(kNew, kNew, true, true, true);
- FinalizerEntry_Generations(kNew, kOld, false, false, true);
+ FinalizerEntry_Generations(kNew, kOld, false, true, true);
FinalizerEntry_Generations(kNew, kImm, false, false, false);
FinalizerEntry_Generations(kOld, kNew, true, true, true);
FinalizerEntry_Generations(kOld, kOld, false, true, true);
diff --git a/runtime/vm/heap/marker.cc b/runtime/vm/heap/marker.cc
index 19c5d6e..2f326db 100644
--- a/runtime/vm/heap/marker.cc
+++ b/runtime/vm/heap/marker.cc
@@ -30,10 +30,12 @@
MarkingVisitorBase(IsolateGroup* isolate_group,
PageSpace* page_space,
MarkingStack* marking_stack,
+ MarkingStack* new_marking_stack,
MarkingStack* deferred_marking_stack)
: ObjectPointerVisitor(isolate_group),
page_space_(page_space),
work_list_(marking_stack),
+ new_work_list_(new_marking_stack),
deferred_work_list_(deferred_marking_stack),
marked_bytes_(0),
marked_micros_(0),
@@ -51,7 +53,6 @@
static bool IsMarked(ObjectPtr raw) {
ASSERT(raw->IsHeapObject());
- ASSERT(raw->IsOldObject());
return raw->untag()->IsMarked();
}
@@ -64,10 +65,9 @@
ObjectPtr raw_key = cur_weak->untag()->key();
// Reset the next pointer in the weak property.
cur_weak->untag()->next_seen_by_gc_ = WeakProperty::null();
- if (raw_key->IsImmediateOrNewObject() || raw_key->untag()->IsMarked()) {
+ if (raw_key->IsImmediateObject() || raw_key->untag()->IsMarked()) {
ObjectPtr raw_val = cur_weak->untag()->value();
- if (!raw_val->IsImmediateOrNewObject() &&
- !raw_val->untag()->IsMarked()) {
+ if (!raw_val->IsImmediateObject() && !raw_val->untag()->IsMarked()) {
more_to_mark = true;
}
@@ -87,33 +87,63 @@
void DrainMarkingStackWithPauseChecks() {
do {
- ObjectPtr raw_obj;
- while (work_list_.Pop(&raw_obj)) {
- const intptr_t class_id = raw_obj->GetClassId();
+ ObjectPtr obj;
+ while (work_list_.Pop(&obj)) {
+ if (obj->IsNewObject()) {
+ Page* page = Page::Of(obj);
+ uword top = page->original_top();
+ uword end = page->original_end();
+ uword addr = static_cast<uword>(obj);
+ if (top <= addr && addr < end) {
+ new_work_list_.Push(obj);
+ if (UNLIKELY(page_space_->pause_concurrent_marking())) {
+ work_list_.Flush();
+ new_work_list_.Flush();
+ deferred_work_list_.Flush();
+ page_space_->YieldConcurrentMarking();
+ }
+ continue;
+ }
+ }
+
+ const intptr_t class_id = obj->GetClassId();
+ ASSERT(class_id != kIllegalCid);
+ ASSERT(class_id != kFreeListElement);
+ ASSERT(class_id != kForwardingCorpse);
intptr_t size;
if (class_id == kWeakPropertyCid) {
- size = ProcessWeakProperty(static_cast<WeakPropertyPtr>(raw_obj));
+ size = ProcessWeakProperty(static_cast<WeakPropertyPtr>(obj));
} else if (class_id == kWeakReferenceCid) {
- size = ProcessWeakReference(static_cast<WeakReferencePtr>(raw_obj));
+ size = ProcessWeakReference(static_cast<WeakReferencePtr>(obj));
} else if (class_id == kWeakArrayCid) {
- size = ProcessWeakArray(static_cast<WeakArrayPtr>(raw_obj));
+ size = ProcessWeakArray(static_cast<WeakArrayPtr>(obj));
} else if (class_id == kFinalizerEntryCid) {
- size = ProcessFinalizerEntry(static_cast<FinalizerEntryPtr>(raw_obj));
+ size = ProcessFinalizerEntry(static_cast<FinalizerEntryPtr>(obj));
} else if (sync && concurrent_ && class_id == kSuspendStateCid) {
// Shape changing is not compatible with concurrent marking.
- deferred_work_list_.Push(raw_obj);
- size = raw_obj->untag()->HeapSize();
+ deferred_work_list_.Push(obj);
+ size = obj->untag()->HeapSize();
} else {
- size = raw_obj->untag()->VisitPointersNonvirtual(this);
+ size = obj->untag()->VisitPointersNonvirtual(this);
}
- marked_bytes_ += size;
+ if (!obj->IsNewObject()) {
+ marked_bytes_ += size;
+ }
if (UNLIKELY(page_space_->pause_concurrent_marking())) {
+ work_list_.Flush();
+ new_work_list_.Flush();
+ deferred_work_list_.Flush();
page_space_->YieldConcurrentMarking();
}
}
} while (ProcessPendingWeakProperties());
+
+ ASSERT(work_list_.IsLocalEmpty());
+ // In case of scavenge before final marking.
+ new_work_list_.Flush();
+ deferred_work_list_.Flush();
}
void DrainMarkingStack() {
@@ -142,34 +172,57 @@
bool ProcessMarkingStack(intptr_t remaining_budget) {
do {
// First drain the marking stacks.
- ObjectPtr raw_obj;
- while (work_list_.Pop(&raw_obj)) {
- const intptr_t class_id = raw_obj->GetClassId();
+ ObjectPtr obj;
+ while (work_list_.Pop(&obj)) {
+ if (sync && concurrent_ && obj->IsNewObject()) {
+ Page* page = Page::Of(obj);
+ uword top = page->original_top();
+ uword end = page->original_end();
+ uword addr = static_cast<uword>(obj);
+ if (top <= addr && addr < end) {
+ new_work_list_.Push(obj);
+ // We did some work routing this object, but didn't look at any of
+ // its slots.
+ intptr_t size = kObjectAlignment;
+ remaining_budget -= size;
+ if (remaining_budget < 0) {
+ return true; // More to mark.
+ }
+ continue;
+ }
+ }
+
+ const intptr_t class_id = obj->GetClassId();
+ ASSERT(class_id != kIllegalCid);
+ ASSERT(class_id != kFreeListElement);
+ ASSERT(class_id != kForwardingCorpse);
intptr_t size;
if (class_id == kWeakPropertyCid) {
- size = ProcessWeakProperty(static_cast<WeakPropertyPtr>(raw_obj));
+ size = ProcessWeakProperty(static_cast<WeakPropertyPtr>(obj));
} else if (class_id == kWeakReferenceCid) {
- size = ProcessWeakReference(static_cast<WeakReferencePtr>(raw_obj));
+ size = ProcessWeakReference(static_cast<WeakReferencePtr>(obj));
} else if (class_id == kWeakArrayCid) {
- size = ProcessWeakArray(static_cast<WeakArrayPtr>(raw_obj));
+ size = ProcessWeakArray(static_cast<WeakArrayPtr>(obj));
} else if (class_id == kFinalizerEntryCid) {
- size = ProcessFinalizerEntry(static_cast<FinalizerEntryPtr>(raw_obj));
+ size = ProcessFinalizerEntry(static_cast<FinalizerEntryPtr>(obj));
} else if (sync && concurrent_ && class_id == kSuspendStateCid) {
// Shape changing is not compatible with concurrent marking.
- deferred_work_list_.Push(raw_obj);
- size = raw_obj->untag()->HeapSize();
+ deferred_work_list_.Push(obj);
+ size = obj->untag()->HeapSize();
} else {
if ((class_id == kArrayCid) || (class_id == kImmutableArrayCid)) {
- size = raw_obj->untag()->HeapSize();
+ size = obj->untag()->HeapSize();
if (size > remaining_budget) {
- work_list_.Push(raw_obj);
+ work_list_.Push(obj);
return true; // More to mark.
}
}
- size = raw_obj->untag()->VisitPointersNonvirtual(this);
+ size = obj->untag()->VisitPointersNonvirtual(this);
}
- marked_bytes_ += size;
+ if (!obj->IsNewObject()) {
+ marked_bytes_ += size;
+ }
remaining_budget -= size;
if (remaining_budget < 0) {
return true; // More to mark.
@@ -222,8 +275,7 @@
ObjectPtr raw_key =
LoadCompressedPointerIgnoreRace(&raw_weak->untag()->key_)
.Decompress(raw_weak->heap_base());
- if (raw_key->IsHeapObject() && raw_key->IsOldObject() &&
- !raw_key->untag()->IsMarked()) {
+ if (raw_key->IsHeapObject() && !raw_key->untag()->IsMarked()) {
// Key was white. Enqueue the weak property.
ASSERT(IsMarked(raw_weak));
delayed_.weak_properties.Enqueue(raw_weak);
@@ -239,8 +291,7 @@
ObjectPtr raw_target =
LoadCompressedPointerIgnoreRace(&raw_weak->untag()->target_)
.Decompress(raw_weak->heap_base());
- if (raw_target->IsHeapObject() && raw_target->IsOldObject() &&
- !raw_target->untag()->IsMarked()) {
+ if (raw_target->IsHeapObject() && !raw_target->untag()->IsMarked()) {
// Target was white. Enqueue the weak reference. It is potentially dead.
// It might still be made alive by weak properties in next rounds.
ASSERT(IsMarked(raw_weak));
@@ -271,9 +322,11 @@
}
void ProcessDeferredMarking() {
- ObjectPtr raw_obj;
- while (deferred_work_list_.Pop(&raw_obj)) {
- ASSERT(raw_obj->IsHeapObject() && raw_obj->IsOldObject());
+ TIMELINE_FUNCTION_GC_DURATION(Thread::Current(), "ProcessDeferredMarking");
+
+ ObjectPtr obj;
+ while (deferred_work_list_.Pop(&obj)) {
+ ASSERT(obj->IsHeapObject());
// We need to scan objects even if they were already scanned via ordinary
// marking. An object may have changed since its ordinary scan and been
// added to deferred marking stack to compensate for write-barrier
@@ -290,11 +343,13 @@
// encounters it during ordinary marking. This is in the same spirit as
// the eliminated write barrier, which would have added the newly written
// key and value to the ordinary marking stack.
- intptr_t size = raw_obj->untag()->VisitPointersNonvirtual(this);
+ intptr_t size = obj->untag()->VisitPointersNonvirtual(this);
// Add the size only if we win the marking race to prevent
// double-counting.
- if (TryAcquireMarkBit(raw_obj)) {
- marked_bytes_ += size;
+ if (TryAcquireMarkBit(obj)) {
+ if (!obj->IsNewObject()) {
+ marked_bytes_ += size;
+ }
}
}
}
@@ -303,6 +358,7 @@
// after this will trigger an error.
void FinalizeMarking() {
work_list_.Finalize();
+ new_work_list_.Finalize();
deferred_work_list_.Finalize();
MournFinalizerEntries();
// MournFinalizerEntries inserts newly discovered dead entries into the
@@ -362,7 +418,7 @@
static bool ForwardOrSetNullIfCollected(ObjectPtr parent,
CompressedObjectPtr* slot) {
ObjectPtr target = slot->Decompress(parent->heap_base());
- if (target->IsImmediateOrNewObject()) {
+ if (target->IsImmediateObject()) {
// Object not touched during this GC.
return false;
}
@@ -381,6 +437,7 @@
void Flush(GCLinkedLists* global_list) {
work_list_.Flush();
+ new_work_list_.Flush();
deferred_work_list_.Flush();
delayed_.FlushInto(global_list);
}
@@ -392,6 +449,7 @@
void AbandonWork() {
work_list_.AbandonWork();
+ new_work_list_.AbandonWork();
deferred_work_list_.AbandonWork();
delayed_.Release();
}
@@ -399,35 +457,43 @@
void FinalizeIncremental(GCLinkedLists* global_list) {
work_list_.Flush();
work_list_.Finalize();
+ new_work_list_.Flush();
+ new_work_list_.Finalize();
deferred_work_list_.Flush();
deferred_work_list_.Finalize();
delayed_.FlushInto(global_list);
}
+ GCLinkedLists* delayed() { return &delayed_; }
+
private:
- void PushMarked(ObjectPtr raw_obj) {
- ASSERT(raw_obj->IsHeapObject());
- ASSERT(raw_obj->IsOldObject());
+ void PushMarked(ObjectPtr obj) {
+ ASSERT(obj->IsHeapObject());
// Push the marked object on the marking stack.
- ASSERT(raw_obj->untag()->IsMarked());
- work_list_.Push(raw_obj);
+ ASSERT(obj->untag()->IsMarked());
+ work_list_.Push(obj);
}
- static bool TryAcquireMarkBit(ObjectPtr raw_obj) {
+ static bool TryAcquireMarkBit(ObjectPtr obj) {
if (!sync) {
- raw_obj->untag()->SetMarkBitUnsynchronized();
+ obj->untag()->SetMarkBitUnsynchronized();
return true;
} else {
- return raw_obj->untag()->TryAcquireMarkBit();
+ return obj->untag()->TryAcquireMarkBit();
}
}
DART_FORCE_INLINE
- void MarkObject(ObjectPtr raw_obj) {
- // Fast exit if the raw object is immediate or in new space. No memory
- // access.
- if (raw_obj->IsImmediateOrNewObject()) {
+ void MarkObject(ObjectPtr obj) {
+ if (obj->IsImmediateObject()) {
+ return;
+ }
+
+ if (sync && concurrent_ && obj->IsNewObject()) {
+ if (TryAcquireMarkBit(obj)) {
+ PushMarked(obj);
+ }
return;
}
@@ -441,30 +507,31 @@
// was allocated after the concurrent marker started. It can read either a
// zero or the header of an object allocated black, both of which appear
// marked.
- if (raw_obj->untag()->IsMarkedIgnoreRace()) {
+ if (obj->untag()->IsMarkedIgnoreRace()) {
return;
}
- intptr_t class_id = raw_obj->GetClassId();
+ intptr_t class_id = obj->GetClassId();
ASSERT(class_id != kFreeListElement);
if (sync && UNLIKELY(class_id == kInstructionsCid)) {
// If this is the concurrent marker, this object may be non-writable due
// to W^X (--write-protect-code).
- deferred_work_list_.Push(raw_obj);
+ deferred_work_list_.Push(obj);
return;
}
- if (!TryAcquireMarkBit(raw_obj)) {
+ if (!TryAcquireMarkBit(obj)) {
// Already marked.
return;
}
- PushMarked(raw_obj);
+ PushMarked(obj);
}
PageSpace* page_space_;
MarkerWorkList work_list_;
+ MarkerWorkList new_work_list_;
MarkerWorkList deferred_work_list_;
GCLinkedLists delayed_;
uintptr_t marked_bytes_;
@@ -477,11 +544,11 @@
typedef MarkingVisitorBase<false> UnsyncMarkingVisitor;
typedef MarkingVisitorBase<true> SyncMarkingVisitor;
-static bool IsUnreachable(const ObjectPtr raw_obj) {
- if (raw_obj->IsImmediateOrNewObject()) {
+static bool IsUnreachable(const ObjectPtr obj) {
+ if (obj->IsImmediateObject()) {
return false;
}
- return !raw_obj->untag()->IsMarked();
+ return !obj->untag()->IsMarked();
}
class MarkingWeakVisitor : public HandleVisitor {
@@ -491,8 +558,8 @@
void VisitHandle(uword addr) override {
FinalizablePersistentHandle* handle =
reinterpret_cast<FinalizablePersistentHandle*>(addr);
- ObjectPtr raw_obj = handle->ptr();
- if (IsUnreachable(raw_obj)) {
+ ObjectPtr obj = handle->ptr();
+ if (IsUnreachable(obj)) {
handle->UpdateUnreachable(thread()->isolate_group());
}
}
@@ -503,17 +570,10 @@
void GCMarker::Prologue() {
isolate_group_->ReleaseStoreBuffers();
- if (heap_->stats_.state_ == Heap::kSecondScavenge) {
- heap_->stats_.state_ = Heap::kMarkingStart;
- }
+ marking_stack_.PushAll(new_marking_stack_.PopAll());
}
-void GCMarker::Epilogue() {
- if (heap_->stats_.state_ == Heap::kMarkingStart) {
- heap_->stats_.state_ = Heap::kInitial;
- heap_->stats_.reachability_barrier_ += 1;
- }
-}
+void GCMarker::Epilogue() {}
enum RootSlices {
kIsolate = 0,
@@ -526,10 +586,6 @@
root_slices_started_ = 0;
root_slices_finished_ = 0;
root_slices_count_ = kNumFixedRootSlices;
- new_page_ = heap_->new_space()->head();
- for (Page* p = new_page_; p != nullptr; p = p->next()) {
- root_slices_count_++;
- }
weak_slices_started_ = 0;
}
@@ -549,17 +605,6 @@
visitor, ValidationPolicy::kDontValidateFrames);
break;
}
- default: {
- Page* page;
- {
- MonitorLocker ml(&root_slices_monitor_);
- page = new_page_;
- ASSERT(page != nullptr);
- new_page_ = page->next();
- }
- TIMELINE_FUNCTION_GC_DURATION(Thread::Current(), "ProcessNewSpace");
- page->VisitObjectPointers(visitor);
- }
}
MonitorLocker ml(&root_slices_monitor_);
@@ -627,8 +672,23 @@
for (intptr_t i = 0; i < size; i++) {
if (table->IsValidEntryAtExclusive(i)) {
// The object has been collected.
- ObjectPtr raw_obj = table->ObjectAtExclusive(i);
- if (raw_obj->IsHeapObject() && !raw_obj->untag()->IsMarked()) {
+ ObjectPtr obj = table->ObjectAtExclusive(i);
+ if (obj->IsHeapObject() && !obj->untag()->IsMarked()) {
+ if (cleanup != nullptr) {
+ cleanup(reinterpret_cast<void*>(table->ValueAtExclusive(i)));
+ }
+ table->InvalidateAtExclusive(i);
+ }
+ }
+ }
+ table =
+ heap_->GetWeakTable(Heap::kNew, static_cast<Heap::WeakSelector>(sel));
+ size = table->size();
+ for (intptr_t i = 0; i < size; i++) {
+ if (table->IsValidEntryAtExclusive(i)) {
+ // The object has been collected.
+ ObjectPtr obj = table->ObjectAtExclusive(i);
+ if (obj->IsHeapObject() && !obj->untag()->IsMarked()) {
if (cleanup != nullptr) {
cleanup(reinterpret_cast<void*>(table->ValueAtExclusive(i)));
}
@@ -643,18 +703,18 @@
TIMELINE_FUNCTION_GC_DURATION(thread, "ProcessRememberedSet");
// Filter collected objects from the remembered set.
StoreBuffer* store_buffer = isolate_group_->store_buffer();
- StoreBufferBlock* reading = store_buffer->TakeBlocks();
+ StoreBufferBlock* reading = store_buffer->PopAll();
StoreBufferBlock* writing = store_buffer->PopNonFullBlock();
while (reading != nullptr) {
StoreBufferBlock* next = reading->next();
// Generated code appends to store buffers; tell MemorySanitizer.
MSAN_UNPOISON(reading, sizeof(*reading));
while (!reading->IsEmpty()) {
- ObjectPtr raw_object = reading->Pop();
- ASSERT(!raw_object->IsForwardingCorpse());
- ASSERT(raw_object->untag()->IsRemembered());
- if (raw_object->untag()->IsMarked()) {
- writing->Push(raw_object);
+ ObjectPtr obj = reading->Pop();
+ ASSERT(!obj->IsForwardingCorpse());
+ ASSERT(obj->untag()->IsRemembered());
+ if (obj->untag()->IsMarked()) {
+ writing->Push(obj);
if (writing->IsFull()) {
store_buffer->PushBlock(writing, StoreBuffer::kIgnoreThreshold);
writing = store_buffer->PopNonFullBlock();
@@ -676,9 +736,9 @@
void VisitPointers(ObjectPtr* first, ObjectPtr* last) override {
for (ObjectPtr* current = first; current <= last; current++) {
- ObjectPtr raw_obj = *current;
- ASSERT(raw_obj->IsHeapObject());
- if (raw_obj->IsOldObject() && !raw_obj->untag()->IsMarked()) {
+ ObjectPtr obj = *current;
+ ASSERT(obj->IsHeapObject());
+ if (!obj->untag()->IsMarked()) {
// Object has become garbage. Replace it will null.
*current = Object::null();
}
@@ -900,6 +960,7 @@
: isolate_group_(isolate_group),
heap_(heap),
marking_stack_(),
+ new_marking_stack_(),
deferred_marking_stack_(),
global_list_(),
visitors_(),
@@ -947,8 +1008,9 @@
ResetSlices();
for (intptr_t i = 0; i < num_tasks; i++) {
ASSERT(visitors_[i] == nullptr);
- SyncMarkingVisitor* visitor = new SyncMarkingVisitor(
- isolate_group_, page_space, &marking_stack_, &deferred_marking_stack_);
+ SyncMarkingVisitor* visitor =
+ new SyncMarkingVisitor(isolate_group_, page_space, &marking_stack_,
+ &new_marking_stack_, &deferred_marking_stack_);
visitors_[i] = visitor;
if (i < (num_tasks - 1)) {
@@ -988,7 +1050,7 @@
"IncrementalMarkWithUnlimitedBudget");
SyncMarkingVisitor visitor(isolate_group_, page_space, &marking_stack_,
- &deferred_marking_stack_);
+ &new_marking_stack_, &deferred_marking_stack_);
int64_t start = OS::GetCurrentMonotonicMicros();
visitor.DrainMarkingStack();
int64_t stop = OS::GetCurrentMonotonicMicros();
@@ -1012,7 +1074,7 @@
"IncrementalMarkWithSizeBudget");
SyncMarkingVisitor visitor(isolate_group_, page_space, &marking_stack_,
- &deferred_marking_stack_);
+ &new_marking_stack_, &deferred_marking_stack_);
int64_t start = OS::GetCurrentMonotonicMicros();
visitor.ProcessMarkingStack(size);
int64_t stop = OS::GetCurrentMonotonicMicros();
@@ -1031,7 +1093,7 @@
"IncrementalMarkWithTimeBudget");
SyncMarkingVisitor visitor(isolate_group_, page_space, &marking_stack_,
- &deferred_marking_stack_);
+ &new_marking_stack_, &deferred_marking_stack_);
int64_t start = OS::GetCurrentMonotonicMicros();
visitor.ProcessMarkingStackUntil(deadline);
int64_t stop = OS::GetCurrentMonotonicMicros();
@@ -1051,7 +1113,8 @@
: ObjectVisitor(), ObjectPointerVisitor(IsolateGroup::Current()) {}
void VisitObject(ObjectPtr obj) override {
- if (obj->IsNewObject() || obj->untag()->IsMarked()) {
+ if (obj->untag()->IsMarked()) {
+ current_ = obj;
obj->untag()->VisitPointers(this);
}
}
@@ -1059,10 +1122,11 @@
void VisitPointers(ObjectPtr* from, ObjectPtr* to) override {
for (ObjectPtr* ptr = from; ptr <= to; ptr++) {
ObjectPtr obj = *ptr;
- if (obj->IsHeapObject() && obj->IsOldObject() &&
- !obj->untag()->IsMarked()) {
- FATAL("Verifying after marking: Not marked: *0x%" Px " = 0x%" Px "\n",
- reinterpret_cast<uword>(ptr), static_cast<uword>(obj));
+ if (obj->IsHeapObject() && !obj->untag()->IsMarked()) {
+ OS::PrintErr("object=0x%" Px ", slot=0x%" Px ", value=0x%" Px "\n",
+ static_cast<uword>(current_), reinterpret_cast<uword>(ptr),
+ static_cast<uword>(obj));
+ failed_ = true;
}
}
}
@@ -1073,14 +1137,21 @@
CompressedObjectPtr* to) override {
for (CompressedObjectPtr* ptr = from; ptr <= to; ptr++) {
ObjectPtr obj = ptr->Decompress(heap_base);
- if (obj->IsHeapObject() && obj->IsOldObject() &&
- !obj->untag()->IsMarked()) {
- FATAL("Verifying after marking: Not marked: *0x%" Px " = 0x%" Px "\n",
- reinterpret_cast<uword>(ptr), static_cast<uword>(obj));
+ if (obj->IsHeapObject() && !obj->untag()->IsMarked()) {
+ OS::PrintErr("object=0x%" Px ", slot=0x%" Px ", value=0x%" Px "\n",
+ static_cast<uword>(current_), reinterpret_cast<uword>(ptr),
+ static_cast<uword>(obj));
+ failed_ = true;
}
}
}
#endif
+
+ bool failed() const { return failed_; }
+
+ private:
+ ObjectPtr current_;
+ bool failed_ = false;
};
void GCMarker::MarkObjects(PageSpace* page_space) {
@@ -1097,6 +1168,7 @@
int64_t start = OS::GetCurrentMonotonicMicros();
// Mark everything on main thread.
UnsyncMarkingVisitor visitor(isolate_group_, page_space, &marking_stack_,
+ &new_marking_stack_,
&deferred_marking_stack_);
visitor.set_concurrent(false);
ResetSlices();
@@ -1128,9 +1200,9 @@
// Visitors may or may not have already been created depending on
// whether we did some concurrent marking.
if (visitor == nullptr) {
- visitor =
- new SyncMarkingVisitor(isolate_group_, page_space,
- &marking_stack_, &deferred_marking_stack_);
+ visitor = new SyncMarkingVisitor(isolate_group_, page_space,
+ &marking_stack_, &new_marking_stack_,
+ &deferred_marking_stack_);
visitors_[i] = visitor;
}
@@ -1176,9 +1248,19 @@
if (FLAG_verify_after_marking) {
VerifyAfterMarkingVisitor visitor;
heap_->VisitObjects(&visitor);
+ if (visitor.failed()) {
+ FATAL("verify after marking");
+ }
}
Epilogue();
}
+void GCMarker::PruneWeak(Scavenger* scavenger) {
+ scavenger->PruneWeak(&global_list_);
+ for (intptr_t i = 0, n = FLAG_marker_tasks; i < n; i++) {
+ scavenger->PruneWeak(visitors_[i]->delayed());
+ }
+}
+
} // namespace dart
diff --git a/runtime/vm/heap/marker.h b/runtime/vm/heap/marker.h
index af323cb..b9755bb 100644
--- a/runtime/vm/heap/marker.h
+++ b/runtime/vm/heap/marker.h
@@ -51,6 +51,8 @@
intptr_t marked_words() const { return marked_bytes_ >> kWordSizeLog2; }
intptr_t MarkedWordsPerMicro() const;
+ void PruneWeak(Scavenger* scavenger);
+
private:
void Prologue();
void Epilogue();
@@ -69,11 +71,11 @@
IsolateGroup* const isolate_group_;
Heap* const heap_;
MarkingStack marking_stack_;
+ MarkingStack new_marking_stack_;
MarkingStack deferred_marking_stack_;
GCLinkedLists global_list_;
MarkingVisitorBase<true>** visitors_;
- Page* new_page_;
Monitor root_slices_monitor_;
RelaxedAtomic<intptr_t> root_slices_started_;
intptr_t root_slices_finished_;
@@ -85,6 +87,9 @@
friend class ConcurrentMarkTask;
friend class ParallelMarkTask;
+ friend class Scavenger;
+ template <bool sync>
+ friend class MarkingVisitorBase;
DISALLOW_IMPLICIT_CONSTRUCTORS(GCMarker);
};
diff --git a/runtime/vm/heap/page.h b/runtime/vm/heap/page.h
index d820a91..af21f28 100644
--- a/runtime/vm/heap/page.h
+++ b/runtime/vm/heap/page.h
@@ -126,6 +126,14 @@
return Utils::RoundUp(sizeof(Page), kObjectAlignment,
kNewObjectAlignmentOffset);
}
+ // These are "original" in the sense that they reflect TLAB boundaries when
+ // the TLAB was acquired, not the current boundaries. An object between
+ // original_top and top may still be in use by Dart code that has eliminated
+ // write barriers.
+ uword original_top() const { return LoadAcquire(&top_); }
+ uword original_end() const { return LoadRelaxed(&end_); }
+ static intptr_t original_top_offset() { return OFFSET_OF(Page, top_); }
+ static intptr_t original_end_offset() { return OFFSET_OF(Page, end_); }
// Warning: This does not work for objects on image pages because image pages
// are not aligned. However, it works for objects on large pages, because
@@ -190,6 +198,8 @@
void Acquire(Thread* thread) {
ASSERT(owner_ == nullptr);
owner_ = thread;
+ ASSERT(thread->top() == 0);
+ ASSERT(thread->end() == 0);
thread->set_top(top_);
thread->set_end(end_);
thread->set_true_end(end_);
@@ -199,7 +209,7 @@
owner_ = nullptr;
uword old_top = top_;
uword new_top = thread->top();
- top_ = new_top;
+ StoreRelease(&top_, new_top);
thread->set_top(0);
thread->set_end(0);
thread->set_true_end(0);
diff --git a/runtime/vm/heap/pages.cc b/runtime/vm/heap/pages.cc
index 96015fe..1f07232 100644
--- a/runtime/vm/heap/pages.cc
+++ b/runtime/vm/heap/pages.cc
@@ -1102,6 +1102,7 @@
}
bool can_verify;
+ SweepNew();
if (compact) {
Compact(thread);
set_phase(kDone);
@@ -1144,6 +1145,20 @@
}
}
+void PageSpace::SweepNew() {
+ // TODO(rmacnak): Run in parallel with SweepExecutable.
+ TIMELINE_FUNCTION_GC_DURATION(Thread::Current(), "SweepNew");
+
+ GCSweeper sweeper;
+ intptr_t free = 0;
+ for (Page* page = heap_->new_space()->head(); page != nullptr;
+ page = page->next()) {
+ page->Release();
+ free += sweeper.SweepNewPage(page);
+ }
+ heap_->new_space()->set_freed_in_words(free >> kWordSizeLog2);
+}
+
void PageSpace::SweepLarge() {
TIMELINE_FUNCTION_GC_DURATION(Thread::Current(), "SweepLarge");
diff --git a/runtime/vm/heap/pages.h b/runtime/vm/heap/pages.h
index 44b58cc..a497137 100644
--- a/runtime/vm/heap/pages.h
+++ b/runtime/vm/heap/pages.h
@@ -345,6 +345,8 @@
bool IsObjectFromImagePages(ObjectPtr object);
+ GCMarker* marker() const { return marker_; }
+
private:
// Ids for time and data records in Heap::GCStats.
enum {
@@ -405,6 +407,7 @@
void FreePages(Page* pages);
void CollectGarbageHelper(Thread* thread, bool compact, bool finalize);
+ void SweepNew();
void SweepLarge();
void Sweep(bool exclusive);
void ConcurrentSweep(IsolateGroup* isolate_group);
diff --git a/runtime/vm/heap/pointer_block.cc b/runtime/vm/heap/pointer_block.cc
index 4885b47..e9c05c3 100644
--- a/runtime/vm/heap/pointer_block.cc
+++ b/runtime/vm/heap/pointer_block.cc
@@ -69,7 +69,7 @@
}
template <int BlockSize>
-typename BlockStack<BlockSize>::Block* BlockStack<BlockSize>::TakeBlocks() {
+typename BlockStack<BlockSize>::Block* BlockStack<BlockSize>::PopAll() {
MonitorLocker ml(&monitor_);
while (!partial_.IsEmpty()) {
full_.Push(partial_.Pop());
@@ -78,6 +78,16 @@
}
template <int BlockSize>
+void BlockStack<BlockSize>::PushAll(Block* block) {
+ while (block != nullptr) {
+ Block* next = block->next();
+ block->set_next(nullptr);
+ PushBlockImpl(block);
+ block = next;
+ }
+}
+
+template <int BlockSize>
void BlockStack<BlockSize>::PushBlockImpl(Block* block) {
ASSERT(block->next() == nullptr); // Should be just a single block.
if (block->IsFull()) {
@@ -237,7 +247,8 @@
return full_.length() + partial_.length();
}
-void StoreBuffer::VisitObjectPointers(ObjectPointerVisitor* visitor) {
+template <int BlockSize>
+void BlockStack<BlockSize>::VisitObjectPointers(ObjectPointerVisitor* visitor) {
for (Block* block = full_.Peek(); block != nullptr; block = block->next()) {
block->VisitObjectPointers(visitor);
}
diff --git a/runtime/vm/heap/pointer_block.h b/runtime/vm/heap/pointer_block.h
index d68a0a2..2bb8aec0e 100644
--- a/runtime/vm/heap/pointer_block.h
+++ b/runtime/vm/heap/pointer_block.h
@@ -103,7 +103,8 @@
Block* PopNonEmptyBlock();
// Pops and returns all non-empty blocks as a linked list (owned by caller).
- Block* TakeBlocks();
+ Block* PopAll();
+ void PushAll(Block* blocks);
// Discards the contents of all non-empty blocks.
void Reset();
@@ -112,6 +113,8 @@
Block* WaitForWork(RelaxedAtomic<uintptr_t>* num_busy, bool abort);
+ void VisitObjectPointers(ObjectPointerVisitor* visitor);
+
protected:
class List {
public:
@@ -274,8 +277,6 @@
// action).
bool Overflowed();
intptr_t Size();
-
- void VisitObjectPointers(ObjectPointerVisitor* visitor);
};
typedef StoreBuffer::Block StoreBufferBlock;
diff --git a/runtime/vm/heap/scavenger.cc b/runtime/vm/heap/scavenger.cc
index c1e0be1..9028ad8 100644
--- a/runtime/vm/heap/scavenger.cc
+++ b/runtime/vm/heap/scavenger.cc
@@ -14,6 +14,7 @@
#include "vm/flags.h"
#include "vm/heap/become.h"
#include "vm/heap/gc_shared.h"
+#include "vm/heap/marker.h"
#include "vm/heap/pages.h"
#include "vm/heap/pointer_block.h"
#include "vm/heap/safepoint.h"
@@ -114,14 +115,14 @@
}
DART_FORCE_INLINE
-static uword ReadHeaderRelaxed(ObjectPtr raw_obj) {
- return reinterpret_cast<std::atomic<uword>*>(UntaggedObject::ToAddr(raw_obj))
+static uword ReadHeaderRelaxed(ObjectPtr obj) {
+ return reinterpret_cast<std::atomic<uword>*>(UntaggedObject::ToAddr(obj))
->load(std::memory_order_relaxed);
}
DART_FORCE_INLINE
-static void WriteHeaderRelaxed(ObjectPtr raw_obj, uword header) {
- reinterpret_cast<std::atomic<uword>*>(UntaggedObject::ToAddr(raw_obj))
+static void WriteHeaderRelaxed(ObjectPtr obj, uword header) {
+ reinterpret_cast<std::atomic<uword>*>(UntaggedObject::ToAddr(obj))
->store(header, std::memory_order_relaxed);
}
@@ -358,24 +359,17 @@
DART_FORCE_INLINE
void ScavengePointer(ObjectPtr* p) {
// ScavengePointer cannot be called recursively.
- ObjectPtr raw_obj = *p;
+ ObjectPtr obj = *p;
- if (raw_obj->IsImmediateOrOldObject()) {
+ if (obj->IsImmediateOrOldObject()) {
return;
}
- ObjectPtr new_obj = ScavengeObject(raw_obj);
+ ObjectPtr new_obj = ScavengeObject(obj);
// Update the reference.
- if (!new_obj->IsNewObject()) {
- // Setting the mark bit above must not be ordered after a publishing store
- // of this object. Note this could be a publishing store even if the
- // object was promoted by an early invocation of ScavengePointer. Compare
- // Object::Allocate.
- reinterpret_cast<std::atomic<ObjectPtr>*>(p)->store(
- static_cast<ObjectPtr>(new_obj), std::memory_order_release);
- } else {
- *p = new_obj;
+ *p = new_obj;
+ if (new_obj->IsNewObject()) {
// Update the store buffer as needed.
ObjectPtr visiting_object = visiting_old_object_;
if (visiting_object != nullptr &&
@@ -388,25 +382,18 @@
DART_FORCE_INLINE
void ScavengeCompressedPointer(uword heap_base, CompressedObjectPtr* p) {
// ScavengePointer cannot be called recursively.
- ObjectPtr raw_obj = p->Decompress(heap_base);
+ ObjectPtr obj = p->Decompress(heap_base);
// Could be tested without decompression.
- if (raw_obj->IsImmediateOrOldObject()) {
+ if (obj->IsImmediateOrOldObject()) {
return;
}
- ObjectPtr new_obj = ScavengeObject(raw_obj);
+ ObjectPtr new_obj = ScavengeObject(obj);
// Update the reference.
- if (!new_obj->IsNewObject()) {
- // Setting the mark bit above must not be ordered after a publishing store
- // of this object. Note this could be a publishing store even if the
- // object was promoted by an early invocation of ScavengePointer. Compare
- // Object::Allocate.
- reinterpret_cast<std::atomic<CompressedObjectPtr>*>(p)->store(
- static_cast<CompressedObjectPtr>(new_obj), std::memory_order_release);
- } else {
- *p = new_obj;
+ *p = new_obj;
+ if (new_obj->IsNewObject()) {
// Update the store buffer as needed.
ObjectPtr visiting_object = visiting_old_object_;
if (visiting_object != nullptr &&
@@ -417,27 +404,27 @@
}
DART_FORCE_INLINE
- ObjectPtr ScavengeObject(ObjectPtr raw_obj) {
+ ObjectPtr ScavengeObject(ObjectPtr obj) {
// Fragmentation might cause the scavenge to fail. Ensure we always have
// somewhere to bail out to.
ASSERT(thread_->long_jump_base() != nullptr);
- uword raw_addr = UntaggedObject::ToAddr(raw_obj);
+ uword raw_addr = UntaggedObject::ToAddr(obj);
// The scavenger is only expects objects located in the from space.
ASSERT(from_->Contains(raw_addr));
// Read the header word of the object and determine if the object has
// already been copied.
- uword header = ReadHeaderRelaxed(raw_obj);
+ uword header = ReadHeaderRelaxed(obj);
ObjectPtr new_obj;
if (IsForwarding(header)) {
// Get the new location of the object.
new_obj = ForwardedObj(header);
} else {
- intptr_t size = raw_obj->untag()->HeapSize(header);
+ intptr_t size = obj->untag()->HeapSize(header);
ASSERT(IsAllocatableInNewSpace(size));
uword new_addr = 0;
// Check whether object should be promoted.
- if (!Page::Of(raw_obj)->IsSurvivor(raw_addr)) {
+ if (!Page::Of(obj)->IsSurvivor(raw_addr)) {
// Not a survivor of a previous scavenge. Just copy the object into the
// to space.
new_addr = TryAllocateCopy(size);
@@ -446,12 +433,7 @@
// This object is a survivor of a previous scavenge. Attempt to promote
// the object. (Or, unlikely, to-space was exhausted by fragmentation.)
new_addr = page_space_->TryAllocatePromoLocked(freelist_, size);
- if (LIKELY(new_addr != 0)) {
- // If promotion succeeded then we need to remember it so that it can
- // be traversed later.
- promoted_list_.Push(UntaggedObject::FromAddr(new_addr));
- bytes_promoted_ += size;
- } else {
+ if (UNLIKELY(new_addr == 0)) {
// Promotion did not succeed. Copy into the to space instead.
scavenger_->failed_to_promote_ = true;
new_addr = TryAllocateCopy(size);
@@ -471,19 +453,9 @@
if (new_obj->IsOldObject()) {
// Promoted: update age/barrier tags.
uword tags = static_cast<uword>(header);
- tags = UntaggedObject::OldBit::update(true, tags);
tags = UntaggedObject::OldAndNotRememberedBit::update(true, tags);
tags = UntaggedObject::NewBit::update(false, tags);
- // Setting the forwarding pointer below will make this tenured object
- // visible to the concurrent marker, but we haven't visited its slots
- // yet. We mark the object here to prevent the concurrent marker from
- // adding it to the mark stack and visiting its unprocessed slots. We
- // push it to the mark stack after forwarding its slots.
- tags = UntaggedObject::OldAndNotMarkedBit::update(
- !thread_->is_marking(), tags);
- // release: Setting the mark bit above must not be ordered after a
- // publishing store of this object. Compare Object::Allocate.
- new_obj->untag()->tags_.store(tags, std::memory_order_release);
+ new_obj->untag()->tags_.store(tags, std::memory_order_relaxed);
}
intptr_t cid = UntaggedObject::ClassIdTag::decode(header);
@@ -493,7 +465,14 @@
// Try to install forwarding address.
uword forwarding_header = ForwardingHeader(new_obj);
- if (!InstallForwardingPointer(raw_addr, &header, forwarding_header)) {
+ if (InstallForwardingPointer(raw_addr, &header, forwarding_header)) {
+ if (new_obj->IsOldObject()) {
+ // If promotion succeeded then we need to remember it so that it can
+ // be traversed later.
+ promoted_list_.Push(new_obj);
+ bytes_promoted_ += size;
+ }
+ } else {
ASSERT(IsForwarding(header));
if (new_obj->IsOldObject()) {
// Abandon as a free list element.
@@ -611,11 +590,11 @@
typedef ScavengerVisitorBase<true> ParallelScavengerVisitor;
static bool IsUnreachable(ObjectPtr* ptr) {
- ObjectPtr raw_obj = *ptr;
- if (raw_obj->IsImmediateOrOldObject()) {
+ ObjectPtr obj = *ptr;
+ if (obj->IsImmediateOrOldObject()) {
return false;
}
- uword raw_addr = UntaggedObject::ToAddr(raw_obj);
+ uword raw_addr = UntaggedObject::ToAddr(obj);
uword header = *reinterpret_cast<uword*>(raw_addr);
if (IsForwarding(header)) {
*ptr = ForwardedObj(header);
@@ -798,14 +777,7 @@
Scavenger::Scavenger(Heap* heap, intptr_t max_semi_capacity_in_words)
: heap_(heap),
max_semi_capacity_in_words_(max_semi_capacity_in_words),
- scavenging_(false),
- gc_time_micros_(0),
- collections_(0),
- scavenge_words_per_micro_(kConservativeInitialScavengeSpeed),
- idle_scavenge_threshold_in_words_(0),
- external_size_(0),
- failed_to_promote_(false),
- abort_(false) {
+ scavenge_words_per_micro_(kConservativeInitialScavengeSpeed) {
ASSERT(heap != nullptr);
// Verify assumptions about the first word in objects which the scavenger is
@@ -870,18 +842,18 @@
void VisitPointers(ObjectPtr* from, ObjectPtr* to) override {
for (ObjectPtr* ptr = from; ptr <= to; ptr++) {
- ObjectPtr raw_obj = *ptr;
- RELEASE_ASSERT_WITH_MSG(raw_obj->untag()->IsRemembered(), msg_);
- RELEASE_ASSERT_WITH_MSG(raw_obj->IsOldObject(), msg_);
+ ObjectPtr obj = *ptr;
+ RELEASE_ASSERT_WITH_MSG(obj->untag()->IsRemembered(), msg_);
+ RELEASE_ASSERT_WITH_MSG(obj->IsOldObject(), msg_);
- RELEASE_ASSERT_WITH_MSG(!raw_obj->untag()->IsCardRemembered(), msg_);
- if (raw_obj.GetClassId() == kArrayCid) {
+ RELEASE_ASSERT_WITH_MSG(!obj->untag()->IsCardRemembered(), msg_);
+ if (obj.GetClassId() == kArrayCid) {
const uword length =
- Smi::Value(static_cast<UntaggedArray*>(raw_obj.untag())->length());
+ Smi::Value(static_cast<UntaggedArray*>(obj.untag())->length());
RELEASE_ASSERT_WITH_MSG(!Array::UseCardMarkingForAllocation(length),
msg_);
}
- in_store_buffer_->Add(raw_obj);
+ in_store_buffer_->Add(obj);
}
}
@@ -910,28 +882,27 @@
to_(to),
msg_(msg) {}
- void VisitObject(ObjectPtr raw_obj) override {
- if (raw_obj->IsPseudoObject()) return;
- RELEASE_ASSERT_WITH_MSG(raw_obj->IsOldObject(), msg_);
+ void VisitObject(ObjectPtr obj) override {
+ if (obj->IsPseudoObject()) return;
+ RELEASE_ASSERT_WITH_MSG(obj->IsOldObject(), msg_);
RELEASE_ASSERT_WITH_MSG(
- raw_obj->untag()->IsRemembered() == in_store_buffer_->Contains(raw_obj),
- msg_);
+ obj->untag()->IsRemembered() == in_store_buffer_->Contains(obj), msg_);
- visiting_ = raw_obj;
- is_remembered_ = raw_obj->untag()->IsRemembered();
- is_card_remembered_ = raw_obj->untag()->IsCardRemembered();
+ visiting_ = obj;
+ is_remembered_ = obj->untag()->IsRemembered();
+ is_card_remembered_ = obj->untag()->IsCardRemembered();
if (is_card_remembered_) {
RELEASE_ASSERT_WITH_MSG(!is_remembered_, msg_);
- RELEASE_ASSERT_WITH_MSG(Page::Of(raw_obj)->progress_bar_ == 0, msg_);
+ RELEASE_ASSERT_WITH_MSG(Page::Of(obj)->progress_bar_ == 0, msg_);
}
- raw_obj->untag()->VisitPointers(this);
+ obj->untag()->VisitPointers(this);
}
void VisitPointers(ObjectPtr* from, ObjectPtr* to) override {
for (ObjectPtr* ptr = from; ptr <= to; ptr++) {
- ObjectPtr raw_obj = *ptr;
- if (raw_obj->IsHeapObject() && raw_obj->IsNewObject()) {
+ ObjectPtr obj = *ptr;
+ if (obj->IsHeapObject() && obj->IsNewObject()) {
if (is_card_remembered_) {
if (!Page::Of(visiting_)->IsCardRemembered(ptr)) {
FATAL(
@@ -940,8 +911,8 @@
"slot's card is not remembered. Consider using rr to watch the "
"slot %p and reverse-continue to find the store with a missing "
"barrier.\n",
- msg_, static_cast<uword>(visiting_),
- static_cast<uword>(raw_obj), ptr);
+ msg_, static_cast<uword>(visiting_), static_cast<uword>(obj),
+ ptr);
}
} else if (!is_remembered_) {
FATAL("%s: Old object %#" Px " references new object %#" Px
@@ -949,10 +920,10 @@
"not in any store buffer. Consider using rr to watch the "
"slot %p and reverse-continue to find the store with a missing "
"barrier.\n",
- msg_, static_cast<uword>(visiting_),
- static_cast<uword>(raw_obj), ptr);
+ msg_, static_cast<uword>(visiting_), static_cast<uword>(obj),
+ ptr);
}
- RELEASE_ASSERT_WITH_MSG(to_->Contains(UntaggedObject::ToAddr(raw_obj)),
+ RELEASE_ASSERT_WITH_MSG(to_->Contains(UntaggedObject::ToAddr(obj)),
msg_);
}
}
@@ -963,8 +934,8 @@
CompressedObjectPtr* from,
CompressedObjectPtr* to) override {
for (CompressedObjectPtr* ptr = from; ptr <= to; ptr++) {
- ObjectPtr raw_obj = ptr->Decompress(heap_base);
- if (raw_obj->IsHeapObject() && raw_obj->IsNewObject()) {
+ ObjectPtr obj = ptr->Decompress(heap_base);
+ if (obj->IsHeapObject() && obj->IsNewObject()) {
if (is_card_remembered_) {
if (!Page::Of(visiting_)->IsCardRemembered(ptr)) {
FATAL(
@@ -973,8 +944,8 @@
"slot's card is not remembered. Consider using rr to watch the "
"slot %p and reverse-continue to find the store with a missing "
"barrier.\n",
- msg_, static_cast<uword>(visiting_),
- static_cast<uword>(raw_obj), ptr);
+ msg_, static_cast<uword>(visiting_), static_cast<uword>(obj),
+ ptr);
}
} else if (!is_remembered_) {
FATAL("%s: Old object %#" Px " references new object %#" Px
@@ -982,10 +953,10 @@
"not in any store buffer. Consider using rr to watch the "
"slot %p and reverse-continue to find the store with a missing "
"barrier.\n",
- msg_, static_cast<uword>(visiting_),
- static_cast<uword>(raw_obj), ptr);
+ msg_, static_cast<uword>(visiting_), static_cast<uword>(obj),
+ ptr);
}
- RELEASE_ASSERT_WITH_MSG(to_->Contains(UntaggedObject::ToAddr(raw_obj)),
+ RELEASE_ASSERT_WITH_MSG(to_->Contains(UntaggedObject::ToAddr(obj)),
msg_);
}
}
@@ -1025,6 +996,7 @@
TIMELINE_FUNCTION_GC_DURATION(Thread::Current(), "Prologue");
heap_->isolate_group()->ReleaseStoreBuffers();
+ heap_->isolate_group()->FlushMarkingStacks();
if (FLAG_verify_store_buffer) {
heap_->WaitForSweeperTasksAtSafepoint(Thread::Current());
@@ -1033,7 +1005,13 @@
// Need to stash the old remembered set before any worker begins adding to the
// new remembered set.
- blocks_ = heap_->isolate_group()->store_buffer()->TakeBlocks();
+ blocks_ = heap_->isolate_group()->store_buffer()->PopAll();
+ GCMarker* marker = heap_->old_space()->marker();
+ if (marker != nullptr) {
+ mark_blocks_ = marker->marking_stack_.PopAll();
+ new_blocks_ = marker->new_marking_stack_.PopAll();
+ deferred_blocks_ = marker->deferred_marking_stack_.PopAll();
+ }
UpdateMaxHeapCapacity();
@@ -1140,23 +1118,12 @@
NoSafepointScope no_safepoint;
// TODO(rmacnak): Investigate collecting a history of idle period durations.
- intptr_t used_in_words = UsedInWords();
+ intptr_t used_in_words = UsedInWords() + freed_in_words_;
intptr_t external_in_words = ExternalInWords();
// Normal reason: new space is getting full.
bool for_new_space = (used_in_words >= idle_scavenge_threshold_in_words_) ||
(external_in_words >= idle_scavenge_threshold_in_words_);
- // New-space objects are roots during old-space GC. This means that even
- // unreachable new-space objects prevent old-space objects they reference
- // from being collected during an old-space GC. Normally this is not an
- // issue because new-space GCs run much more frequently than old-space GCs.
- // If new-space allocation is low and direct old-space allocation is high,
- // which can happen in a program that allocates large objects and little
- // else, old-space can fill up with unreachable objects until the next
- // new-space GC. This check is the idle equivalent to the
- // new-space GC before synchronous-marking in CollectMostGarbage.
- bool for_old_space = heap_->last_gc_was_old_space_ &&
- heap_->old_space()->ReachedIdleThreshold();
- if (!for_new_space && !for_old_space) {
+ if (!for_new_space) {
return false;
}
@@ -1252,6 +1219,7 @@
kWeakTables,
kProgressBars,
kRememberLiveTemporaries,
+ kPruneWeak,
kNumWeakSlices,
};
@@ -1276,10 +1244,23 @@
// Restore write-barrier assumptions.
heap_->isolate_group()->RememberLiveTemporaries();
break;
+ case kPruneWeak: {
+ GCMarker* marker = heap_->old_space()->marker();
+ if (marker != nullptr) {
+ marker->PruneWeak(this);
+ }
+ } break;
default:
UNREACHABLE();
}
}
+
+ GCMarker* marker = heap_->old_space()->marker();
+ if (marker != nullptr) {
+ Prune(&mark_blocks_, &marker->marking_stack_);
+ Prune(&new_blocks_, &marker->marking_stack_);
+ Prune(&deferred_blocks_, &marker->deferred_marking_stack_);
+ }
}
void Scavenger::MournWeakHandles() {
@@ -1295,8 +1276,8 @@
while (scan_ != nullptr) {
uword resolved_top = scan_->resolved_top_;
while (resolved_top < scan_->top_) {
- ObjectPtr raw_obj = UntaggedObject::FromAddr(resolved_top);
- resolved_top += ProcessObject(raw_obj);
+ ObjectPtr obj = UntaggedObject::FromAddr(resolved_top);
+ resolved_top += ProcessObject(obj);
}
scan_->resolved_top_ = resolved_top;
@@ -1315,12 +1296,8 @@
while (promoted_list_.Pop(&obj)) {
VisitingOldObject(obj);
ProcessObject(obj);
- if (obj->untag()->IsMarked()) {
- // Complete our promise from ScavengePointer. Note that marker cannot
- // visit this object until it pops a block from the mark stack, which
- // involves a memory fence from the mutex, so even on architectures
- // with a relaxed memory model, the marker will see the fully
- // forwarded contents of this object.
+ // Black allocation.
+ if (thread_->is_marking() && obj->untag()->TryAcquireMarkBit()) {
thread_->MarkingStackAddObject(obj);
}
}
@@ -1431,16 +1408,16 @@
intptr_t size = table->size();
for (intptr_t i = 0; i < size; i++) {
if (table->IsValidEntryAtExclusive(i)) {
- ObjectPtr raw_obj = table->ObjectAtExclusive(i);
- ASSERT(raw_obj->IsHeapObject());
- uword raw_addr = UntaggedObject::ToAddr(raw_obj);
+ ObjectPtr obj = table->ObjectAtExclusive(i);
+ ASSERT(obj->IsHeapObject());
+ uword raw_addr = UntaggedObject::ToAddr(obj);
uword header = *reinterpret_cast<uword*>(raw_addr);
if (IsForwarding(header)) {
// The object has survived. Preserve its record.
- raw_obj = ForwardedObj(header);
+ obj = ForwardedObj(header);
auto replacement =
- raw_obj->IsNewObject() ? replacement_new : replacement_old;
- replacement->SetValueExclusive(raw_obj, table->ValueAtExclusive(i));
+ obj->IsNewObject() ? replacement_new : replacement_old;
+ replacement->SetValueExclusive(obj, table->ValueAtExclusive(i));
} else {
// The object has been collected.
if (cleanup != nullptr) {
@@ -1489,6 +1466,117 @@
/*at_safepoint=*/true);
}
+void Scavenger::Forward(MarkingStack* marking_stack) {
+ ASSERT(abort_);
+
+ class ReverseMarkStack : public ObjectPointerVisitor {
+ public:
+ explicit ReverseMarkStack(IsolateGroup* group)
+ : ObjectPointerVisitor(group) {}
+
+ void VisitPointers(ObjectPtr* first, ObjectPtr* last) override {
+ for (ObjectPtr* p = first; p <= last; p++) {
+ ObjectPtr obj = *p;
+#if defined(DEBUG)
+ if (obj->IsNewObject()) {
+ uword header = ReadHeaderRelaxed(obj);
+ ASSERT(!IsForwarding(header));
+ }
+#endif
+ if (obj->IsForwardingCorpse()) {
+ // Promoted object was pushed to mark list but reversed.
+ *p = reinterpret_cast<ForwardingCorpse*>(UntaggedObject::ToAddr(obj))
+ ->target();
+ }
+ }
+ }
+#if defined(DART_COMPRESSED_POINTERS)
+ void VisitCompressedPointers(uword heap_base,
+ CompressedObjectPtr* first,
+ CompressedObjectPtr* last) override {
+ UNREACHABLE();
+ }
+#endif
+ };
+
+ ReverseMarkStack visitor(heap_->isolate_group());
+ marking_stack->VisitObjectPointers(&visitor);
+}
+
+void Scavenger::Prune(MarkingStackBlock** source, MarkingStack* marking_stack) {
+ ASSERT(!abort_);
+ TIMELINE_FUNCTION_GC_DURATION(Thread::Current(), "PruneMarkingStack");
+ MarkingStackBlock* reading;
+ MarkingStackBlock* writing = marking_stack->PopNonFullBlock();
+ for (;;) {
+ {
+ MutexLocker ml(&space_lock_);
+ reading = *source;
+ if (reading == nullptr) break;
+ *source = reading->next();
+ }
+ // Generated code appends to marking stacks; tell MemorySanitizer.
+ MSAN_UNPOISON(reading, sizeof(*reading));
+ while (!reading->IsEmpty()) {
+ ObjectPtr obj = reading->Pop();
+ ASSERT(obj->IsHeapObject());
+ if (obj->IsNewObject()) {
+ uword header = ReadHeaderRelaxed(obj);
+ if (!IsForwarding(header)) continue;
+ obj = ForwardedObj(header);
+ }
+ ASSERT(!obj->IsForwardingCorpse());
+ ASSERT(!obj->IsFreeListElement());
+ writing->Push(obj);
+ if (writing->IsFull()) {
+ marking_stack->PushBlock(writing);
+ writing = marking_stack->PopNonFullBlock();
+ }
+ }
+ reading->Reset();
+ marking_stack->PushBlock(reading);
+ }
+ marking_stack->PushBlock(writing);
+}
+
+void Scavenger::PruneWeak(GCLinkedLists* deferred) {
+ ASSERT(!abort_);
+ TIMELINE_FUNCTION_GC_DURATION(Thread::Current(), "PruneWeak");
+ PruneWeak(&deferred->weak_properties);
+ PruneWeak(&deferred->weak_references);
+ PruneWeak(&deferred->weak_arrays);
+ PruneWeak(&deferred->finalizer_entries);
+}
+
+template <typename Type, typename PtrType>
+void Scavenger::PruneWeak(GCLinkedList<Type, PtrType>* list) {
+ PtrType weak = list->Release();
+ while (weak != Object::null()) {
+ PtrType next;
+ if (weak->IsOldObject()) {
+ ASSERT(weak->GetClassId() == Type::kClassId);
+ next = weak->untag()->next_seen_by_gc_.Decompress(weak->heap_base());
+ weak->untag()->next_seen_by_gc_ = Type::null();
+ list->Enqueue(weak);
+ } else {
+ uword header = ReadHeaderRelaxed(weak);
+ if (IsForwarding(header)) {
+ weak = static_cast<PtrType>(ForwardedObj(header));
+ ASSERT(weak->GetClassId() == Type::kClassId);
+ next = weak->untag()->next_seen_by_gc_.Decompress(weak->heap_base());
+ weak->untag()->next_seen_by_gc_ = Type::null();
+ list->Enqueue(weak);
+ } else {
+ // Collected in this scavenge.
+ ASSERT(weak->GetClassId() == Type::kClassId);
+ next = weak->untag()->next_seen_by_gc_.Decompress(weak->heap_base());
+ }
+ }
+
+ weak = next;
+ }
+}
+
// Returns whether the object referred to in `slot` was GCed this GC.
template <bool parallel>
bool ScavengerVisitorBase<parallel>::ForwardOrSetNullIfCollected(
@@ -1607,6 +1695,9 @@
Page* page = Page::Of(thread->top() - 1);
intptr_t allocated;
{
+ if (thread->is_marking()) {
+ thread->DeferredMarkLiveTemporaries();
+ }
MutexLocker ml(&space_lock_);
allocated = page->Release(thread);
}
@@ -1661,6 +1752,7 @@
abort_ = false;
root_slices_started_ = 0;
weak_slices_started_ = 0;
+ freed_in_words_ = 0;
intptr_t abandoned_bytes = 0; // TODO(rmacnak): Count fragmentation?
SpaceUsage usage_before = GetCurrentUsage();
intptr_t promo_candidate_words = 0;
@@ -1684,18 +1776,6 @@
ReverseScavenge(&from);
bytes_promoted = 0;
} else {
- if (type == GCType::kEvacuate) {
- if (heap_->stats_.state_ == Heap::kInitial ||
- heap_->stats_.state_ == Heap::kFirstScavenge) {
- heap_->stats_.state_ = Heap::kSecondScavenge;
- }
- } else {
- if (heap_->stats_.state_ == Heap::kInitial) {
- heap_->stats_.state_ = Heap::kFirstScavenge;
- } else if (heap_->stats_.state_ == Heap::kFirstScavenge) {
- heap_->stats_.state_ = Heap::kSecondScavenge;
- }
- }
if ((ThresholdInWords() - UsedInWords()) < KBInWords) {
// Don't scavenge again until the next old-space GC has occurred. Prevents
// performing one scavenge per allocation as the heap limit is approached.
@@ -1795,12 +1875,9 @@
// Reset the ages bits in case this was a promotion.
uword from_header = static_cast<uword>(to_header);
- from_header = UntaggedObject::OldBit::update(false, from_header);
from_header =
UntaggedObject::OldAndNotRememberedBit::update(false, from_header);
from_header = UntaggedObject::NewBit::update(true, from_header);
- from_header =
- UntaggedObject::OldAndNotMarkedBit::update(false, from_header);
WriteHeaderRelaxed(from_obj, from_header);
@@ -1846,6 +1923,26 @@
heap_->old_space()->ResetProgressBars();
+ GCMarker* marker = heap_->old_space()->marker();
+ if (marker != nullptr) {
+ marker->marking_stack_.PushAll(mark_blocks_);
+ mark_blocks_ = nullptr;
+ marker->marking_stack_.PushAll(new_blocks_);
+ new_blocks_ = nullptr;
+ marker->deferred_marking_stack_.PushAll(deferred_blocks_);
+ deferred_blocks_ = nullptr;
+ // Not redundant with the flush at the beginning of the scavenge because
+ // the scavenge workers may add promoted objects to the mark stack.
+ heap_->isolate_group()->FlushMarkingStacks();
+
+ Forward(&marker->marking_stack_);
+ ASSERT(marker->new_marking_stack_.IsEmpty());
+ Forward(&marker->deferred_marking_stack_);
+ }
+
+ // Restore write-barrier assumptions. Must occur after mark list fixups.
+ heap_->isolate_group()->RememberLiveTemporaries();
+
// Don't scavenge again until the next old-space GC has occurred. Prevents
// performing one scavenge per allocation as the heap limit is approached.
heap_->assume_scavenge_will_fail_ = true;
diff --git a/runtime/vm/heap/scavenger.h b/runtime/vm/heap/scavenger.h
index 2d19bb8..c61452f 100644
--- a/runtime/vm/heap/scavenger.h
+++ b/runtime/vm/heap/scavenger.h
@@ -29,6 +29,10 @@
class ObjectSet;
template <bool parallel>
class ScavengerVisitorBase;
+class GCMarker;
+template <typename Type, typename PtrType>
+class GCLinkedList;
+struct GCLinkedLists;
class SemiSpace {
public:
@@ -155,7 +159,7 @@
intptr_t UsedInWords() const {
MutexLocker ml(&space_lock_);
- return to_->used_in_words();
+ return to_->used_in_words() - freed_in_words_;
}
intptr_t CapacityInWords() const {
MutexLocker ml(&space_lock_);
@@ -217,6 +221,8 @@
ASSERT(external_size_ >= 0);
}
+ void set_freed_in_words(intptr_t value) { freed_in_words_ = value; }
+
// The maximum number of Dart mutator threads we allow to execute at the same
// time.
static intptr_t MaxMutatorThreadCount() {
@@ -232,6 +238,12 @@
return to_->head();
}
+ void Prune(MarkingStackBlock** from, MarkingStack* to);
+ void Forward(MarkingStack* stack);
+ void PruneWeak(GCLinkedLists* delayed);
+ template <typename Type, typename PtrType>
+ void PruneWeak(GCLinkedList<Type, PtrType>* list);
+
private:
// Ids for time and data records in Heap::GCStats.
enum {
@@ -294,25 +306,29 @@
intptr_t max_semi_capacity_in_words_;
// Keep track whether a scavenge is currently running.
- bool scavenging_;
+ bool scavenging_ = false;
bool early_tenure_ = false;
- RelaxedAtomic<intptr_t> root_slices_started_;
- RelaxedAtomic<intptr_t> weak_slices_started_;
+ RelaxedAtomic<intptr_t> root_slices_started_ = {0};
+ RelaxedAtomic<intptr_t> weak_slices_started_ = {0};
StoreBufferBlock* blocks_ = nullptr;
+ MarkingStackBlock* mark_blocks_ = nullptr;
+ MarkingStackBlock* new_blocks_ = nullptr;
+ MarkingStackBlock* deferred_blocks_ = nullptr;
- int64_t gc_time_micros_;
- intptr_t collections_;
+ int64_t gc_time_micros_ = 0;
+ intptr_t collections_ = 0;
static constexpr int kStatsHistoryCapacity = 4;
RingBuffer<ScavengeStats, kStatsHistoryCapacity> stats_history_;
intptr_t scavenge_words_per_micro_;
- intptr_t idle_scavenge_threshold_in_words_;
+ intptr_t idle_scavenge_threshold_in_words_ = 0;
// The total size of external data associated with objects in this scavenger.
- RelaxedAtomic<intptr_t> external_size_;
+ RelaxedAtomic<intptr_t> external_size_ = {0};
+ intptr_t freed_in_words_ = 0;
- RelaxedAtomic<bool> failed_to_promote_;
- RelaxedAtomic<bool> abort_;
+ RelaxedAtomic<bool> failed_to_promote_ = {false};
+ RelaxedAtomic<bool> abort_ = {false};
// Protects new space during the allocation of new TLABs
mutable Mutex space_lock_;
diff --git a/runtime/vm/heap/sweeper.cc b/runtime/vm/heap/sweeper.cc
index 03438a4..ff9a3da 100644
--- a/runtime/vm/heap/sweeper.cc
+++ b/runtime/vm/heap/sweeper.cc
@@ -15,6 +15,48 @@
namespace dart {
+intptr_t GCSweeper::SweepNewPage(Page* page) {
+ ASSERT(!page->is_image());
+ ASSERT(!page->is_old());
+ ASSERT(!page->is_executable());
+
+ uword start = page->object_start();
+ uword end = page->object_end();
+ uword current = start;
+ intptr_t free = 0;
+ while (current < end) {
+ ObjectPtr raw_obj = UntaggedObject::FromAddr(current);
+ ASSERT(Page::Of(raw_obj) == page);
+ uword tags = raw_obj->untag()->tags_.load(std::memory_order_relaxed);
+ intptr_t obj_size = raw_obj->untag()->HeapSize(tags);
+ if (UntaggedObject::IsMarked(tags)) {
+ // Found marked object. Clear the mark bit and update swept bytes.
+ raw_obj->untag()->ClearMarkBitUnsynchronized();
+ ASSERT(IsAllocatableInNewSpace(obj_size));
+ } else {
+ uword free_end = current + obj_size;
+ while (free_end < end) {
+ ObjectPtr next_obj = UntaggedObject::FromAddr(free_end);
+ tags = next_obj->untag()->tags_.load(std::memory_order_relaxed);
+ if (UntaggedObject::IsMarked(tags)) {
+ // Reached the end of the free block.
+ break;
+ }
+ // Expand the free block by the size of this object.
+ free_end += next_obj->untag()->HeapSize(tags);
+ }
+ obj_size = free_end - current;
+#if defined(DEBUG)
+ memset(reinterpret_cast<void*>(current), Heap::kZapByte, obj_size);
+#endif // DEBUG
+ FreeListElement::AsElementNew(current, obj_size);
+ free += obj_size;
+ }
+ current += obj_size;
+ }
+ return free;
+}
+
bool GCSweeper::SweepPage(Page* page, FreeList* freelist) {
ASSERT(!page->is_image());
// Large executable pages are handled here. We never truncate Instructions
diff --git a/runtime/vm/heap/sweeper.h b/runtime/vm/heap/sweeper.h
index de932a6..2def2d5 100644
--- a/runtime/vm/heap/sweeper.h
+++ b/runtime/vm/heap/sweeper.h
@@ -34,6 +34,8 @@
// last marked object.
intptr_t SweepLargePage(Page* page);
+ intptr_t SweepNewPage(Page* page);
+
// Sweep the large and regular sized data pages.
static void SweepConcurrent(IsolateGroup* isolate_group);
};
diff --git a/runtime/vm/image_snapshot.cc b/runtime/vm/image_snapshot.cc
index a15ccae..b738e45 100644
--- a/runtime/vm/image_snapshot.cc
+++ b/runtime/vm/image_snapshot.cc
@@ -596,8 +596,8 @@
}
static constexpr uword kReadOnlyGCBits =
- UntaggedObject::OldBit::encode(true) |
- UntaggedObject::OldAndNotMarkedBit::encode(false) |
+ UntaggedObject::AlwaysSetBit::encode(true) |
+ UntaggedObject::NotMarkedBit::encode(false) |
UntaggedObject::OldAndNotRememberedBit::encode(true) |
UntaggedObject::NewBit::encode(false);
diff --git a/runtime/vm/isolate.cc b/runtime/vm/isolate.cc
index 929420f..63cf72f 100644
--- a/runtime/vm/isolate.cc
+++ b/runtime/vm/isolate.cc
@@ -2706,6 +2706,10 @@
thread_registry()->ReleaseStoreBuffers();
}
+void IsolateGroup::FlushMarkingStacks() {
+ thread_registry()->FlushMarkingStacks();
+}
+
void Isolate::RememberLiveTemporaries() {
if (mutator_thread_ != nullptr) {
mutator_thread_->RememberLiveTemporaries();
diff --git a/runtime/vm/isolate.h b/runtime/vm/isolate.h
index 5b3f1d3..d36a562 100644
--- a/runtime/vm/isolate.h
+++ b/runtime/vm/isolate.h
@@ -593,6 +593,7 @@
// Prepares all threads in an isolate for Garbage Collection.
void ReleaseStoreBuffers();
+ void FlushMarkingStacks();
void EnableIncrementalBarrier(MarkingStack* marking_stack,
MarkingStack* deferred_marking_stack);
void DisableIncrementalBarrier();
diff --git a/runtime/vm/object.cc b/runtime/vm/object.cc
index 17003bb..f0a3f6c 100644
--- a/runtime/vm/object.cc
+++ b/runtime/vm/object.cc
@@ -1634,8 +1634,8 @@
UntaggedObject::ClassIdTag::update(kTypedDataInt8ArrayCid, 0);
new_tags = UntaggedObject::SizeTag::update(leftover_size, new_tags);
const bool is_old = obj.ptr()->IsOldObject();
- new_tags = UntaggedObject::OldBit::update(is_old, new_tags);
- new_tags = UntaggedObject::OldAndNotMarkedBit::update(is_old, new_tags);
+ new_tags = UntaggedObject::AlwaysSetBit::update(true, new_tags);
+ new_tags = UntaggedObject::NotMarkedBit::update(true, new_tags);
new_tags =
UntaggedObject::OldAndNotRememberedBit::update(is_old, new_tags);
new_tags = UntaggedObject::NewBit::update(!is_old, new_tags);
@@ -1658,8 +1658,8 @@
uword new_tags = UntaggedObject::ClassIdTag::update(kInstanceCid, 0);
new_tags = UntaggedObject::SizeTag::update(leftover_size, new_tags);
const bool is_old = obj.ptr()->IsOldObject();
- new_tags = UntaggedObject::OldBit::update(is_old, new_tags);
- new_tags = UntaggedObject::OldAndNotMarkedBit::update(is_old, new_tags);
+ new_tags = UntaggedObject::AlwaysSetBit::update(true, new_tags);
+ new_tags = UntaggedObject::NotMarkedBit::update(true, new_tags);
new_tags =
UntaggedObject::OldAndNotRememberedBit::update(is_old, new_tags);
new_tags = UntaggedObject::NewBit::update(!is_old, new_tags);
@@ -2809,8 +2809,8 @@
tags = UntaggedObject::SizeTag::update(size, tags);
const bool is_old =
(address & kNewObjectAlignmentOffset) == kOldObjectAlignmentOffset;
- tags = UntaggedObject::OldBit::update(is_old, tags);
- tags = UntaggedObject::OldAndNotMarkedBit::update(is_old, tags);
+ tags = UntaggedObject::AlwaysSetBit::update(true, tags);
+ tags = UntaggedObject::NotMarkedBit::update(true, tags);
tags = UntaggedObject::OldAndNotRememberedBit::update(is_old, tags);
tags = UntaggedObject::NewBit::update(!is_old, tags);
tags = UntaggedObject::ImmutableBit::update(
@@ -26828,12 +26828,10 @@
dst.set_pc(src.pc());
// Trigger write barrier if needed.
if (dst.ptr()->IsOldObject()) {
- if (!dst.untag()->IsRemembered()) {
- dst.untag()->EnsureInRememberedSet(thread);
- }
- if (thread->is_marking()) {
- thread->DeferredMarkingStackAddObject(dst.ptr());
- }
+ dst.untag()->EnsureInRememberedSet(thread);
+ }
+ if (thread->is_marking()) {
+ thread->DeferredMarkingStackAddObject(dst.ptr());
}
}
return dst.ptr();
diff --git a/runtime/vm/object_graph_copy.cc b/runtime/vm/object_graph_copy.cc
index d22c629..a2b2be6 100644
--- a/runtime/vm/object_graph_copy.cc
+++ b/runtime/vm/object_graph_copy.cc
@@ -230,8 +230,8 @@
tags = UntaggedObject::SizeTag::update(size, tags);
tags = UntaggedObject::ClassIdTag::update(cid, tags);
- tags = UntaggedObject::OldBit::update(false, tags);
- tags = UntaggedObject::OldAndNotMarkedBit::update(false, tags);
+ tags = UntaggedObject::AlwaysSetBit::update(true, tags);
+ tags = UntaggedObject::NotMarkedBit::update(true, tags);
tags = UntaggedObject::OldAndNotRememberedBit::update(false, tags);
tags = UntaggedObject::CanonicalBit::update(false, tags);
tags = UntaggedObject::NewBit::update(true, tags);
diff --git a/runtime/vm/raw_object.cc b/runtime/vm/raw_object.cc
index e4c5788..0f6ed84 100644
--- a/runtime/vm/raw_object.cc
+++ b/runtime/vm/raw_object.cc
@@ -51,12 +51,6 @@
if (!NewBit::decode(tags)) {
FATAL("New object missing kNewBit: %" Px "\n", tags);
}
- if (OldBit::decode(tags)) {
- FATAL("New object has kOldBit: %" Px "\n", tags);
- }
- if (OldAndNotMarkedBit::decode(tags)) {
- FATAL("New object has kOldAndNotMarkedBit: %" Px "\n", tags);
- }
if (OldAndNotRememberedBit::decode(tags)) {
FATAL("New object has kOldAndNotRememberedBit: %" Px "\n", tags);
}
@@ -64,9 +58,6 @@
if (NewBit::decode(tags)) {
FATAL("Old object has kNewBit: %" Px "\n", tags);
}
- if (!OldBit::decode(tags)) {
- FATAL("Old object missing kOldBit: %" Px "\n", tags);
- }
}
const intptr_t class_id = ClassIdTag::decode(tags);
if (!isolate_group->class_table()->IsValidIndex(class_id)) {
diff --git a/runtime/vm/raw_object.h b/runtime/vm/raw_object.h
index ccb9964..1c4aa2f 100644
--- a/runtime/vm/raw_object.h
+++ b/runtime/vm/raw_object.h
@@ -162,9 +162,9 @@
enum TagBits {
kCardRememberedBit = 0,
kCanonicalBit = 1,
- kOldAndNotMarkedBit = 2, // Incremental barrier target.
+ kNotMarkedBit = 2, // Incremental barrier target.
kNewBit = 3, // Generational barrier target.
- kOldBit = 4, // Incremental barrier source.
+ kAlwaysSetBit = 4, // Incremental barrier source.
kOldAndNotRememberedBit = 5, // Generational barrier source.
kImmutableBit = 6,
kReservedBit = 7,
@@ -178,9 +178,9 @@
};
static constexpr intptr_t kGenerationalBarrierMask = 1 << kNewBit;
- static constexpr intptr_t kIncrementalBarrierMask = 1 << kOldAndNotMarkedBit;
+ static constexpr intptr_t kIncrementalBarrierMask = 1 << kNotMarkedBit;
static constexpr intptr_t kBarrierOverlapShift = 2;
- COMPILE_ASSERT(kOldAndNotMarkedBit + kBarrierOverlapShift == kOldBit);
+ COMPILE_ASSERT(kNotMarkedBit + kBarrierOverlapShift == kAlwaysSetBit);
COMPILE_ASSERT(kNewBit + kBarrierOverlapShift == kOldAndNotRememberedBit);
// The bit in the Smi tag position must be something that can be set to 0
@@ -244,14 +244,13 @@
class CardRememberedBit
: public BitField<uword, bool, kCardRememberedBit, 1> {};
- class OldAndNotMarkedBit
- : public BitField<uword, bool, kOldAndNotMarkedBit, 1> {};
+ class NotMarkedBit : public BitField<uword, bool, kNotMarkedBit, 1> {};
class NewBit : public BitField<uword, bool, kNewBit, 1> {};
class CanonicalBit : public BitField<uword, bool, kCanonicalBit, 1> {};
- class OldBit : public BitField<uword, bool, kOldBit, 1> {};
+ class AlwaysSetBit : public BitField<uword, bool, kAlwaysSetBit, 1> {};
class OldAndNotRememberedBit
: public BitField<uword, bool, kOldAndNotRememberedBit, 1> {};
@@ -278,41 +277,34 @@
// Support for GC marking bit. Marked objects are either grey (not yet
// visited) or black (already visited).
- static bool IsMarked(uword tags) { return !OldAndNotMarkedBit::decode(tags); }
- bool IsMarked() const {
- ASSERT(IsOldObject());
- return !tags_.Read<OldAndNotMarkedBit>();
- }
+ static bool IsMarked(uword tags) { return !NotMarkedBit::decode(tags); }
+ bool IsMarked() const { return !tags_.Read<NotMarkedBit>(); }
bool IsMarkedIgnoreRace() const {
- ASSERT(IsOldObject());
- return !tags_.ReadIgnoreRace<OldAndNotMarkedBit>();
+ return !tags_.ReadIgnoreRace<NotMarkedBit>();
}
void SetMarkBit() {
- ASSERT(IsOldObject());
ASSERT(!IsMarked());
- tags_.UpdateBool<OldAndNotMarkedBit>(false);
+ tags_.UpdateBool<NotMarkedBit>(false);
}
void SetMarkBitUnsynchronized() {
- ASSERT(IsOldObject());
ASSERT(!IsMarked());
- tags_.UpdateUnsynchronized<OldAndNotMarkedBit>(false);
+ tags_.UpdateUnsynchronized<NotMarkedBit>(false);
}
void SetMarkBitRelease() {
- ASSERT(IsOldObject());
ASSERT(!IsMarked());
- tags_.UpdateBool<OldAndNotMarkedBit, std::memory_order_release>(false);
+ tags_.UpdateBool<NotMarkedBit, std::memory_order_release>(false);
}
void ClearMarkBit() {
- ASSERT(IsOldObject());
ASSERT(IsMarked());
- tags_.UpdateBool<OldAndNotMarkedBit>(true);
+ tags_.UpdateBool<NotMarkedBit>(true);
+ }
+ void ClearMarkBitUnsynchronized() {
+ ASSERT(IsMarked());
+ tags_.UpdateUnsynchronized<NotMarkedBit>(true);
}
// Returns false if the bit was already set.
DART_WARN_UNUSED_RESULT
- bool TryAcquireMarkBit() {
- ASSERT(IsOldObject());
- return tags_.TryClear<OldAndNotMarkedBit>();
- }
+ bool TryAcquireMarkBit() { return tags_.TryClear<NotMarkedBit>(); }
// Canonical objects have the property that two canonical objects are
// logically equal iff they are the same object (pointer equal).
@@ -339,6 +331,10 @@
ASSERT(IsOldObject());
tags_.UpdateBool<OldAndNotRememberedBit>(true);
}
+ void ClearRememberedBitUnsynchronized() {
+ ASSERT(IsOldObject());
+ tags_.UpdateUnsynchronized<OldAndNotRememberedBit>(true);
+ }
DART_FORCE_INLINE
void EnsureInRememberedSet(Thread* thread) {
@@ -705,16 +701,17 @@
void CheckHeapPointerStore(ObjectPtr value, Thread* thread) {
uword source_tags = this->tags_;
uword target_tags = value->untag()->tags_;
- if (((source_tags >> kBarrierOverlapShift) & target_tags &
- thread->write_barrier_mask()) != 0) {
- if (value->IsNewObject()) {
+ uword overlap = (source_tags >> kBarrierOverlapShift) & target_tags &
+ thread->write_barrier_mask();
+ if (overlap != 0) {
+ if ((overlap & kGenerationalBarrierMask) != 0) {
// Generational barrier: record when a store creates an
// old-and-not-remembered -> new reference.
EnsureInRememberedSet(thread);
- } else {
+ }
+ if ((overlap & kIncrementalBarrierMask) != 0) {
// Incremental barrier: record when a store creates an
- // old -> old-and-not-marked reference.
- ASSERT(value->IsOldObject());
+ // any -> not-marked reference.
if (ClassIdTag::decode(target_tags) == kInstructionsCid) {
// Instruction pages may be non-writable. Defer marking.
thread->DeferredMarkingStackAddObject(value);
@@ -733,9 +730,10 @@
Thread* thread) {
uword source_tags = this->tags_;
uword target_tags = value->untag()->tags_;
- if (((source_tags >> kBarrierOverlapShift) & target_tags &
- thread->write_barrier_mask()) != 0) {
- if (value->IsNewObject()) {
+ uword overlap = (source_tags >> kBarrierOverlapShift) & target_tags &
+ thread->write_barrier_mask();
+ if (overlap != 0) {
+ if ((overlap & kGenerationalBarrierMask) != 0) {
// Generational barrier: record when a store creates an
// old-and-not-remembered -> new reference.
if (this->IsCardRemembered()) {
@@ -743,10 +741,10 @@
} else if (this->TryAcquireRememberedBit()) {
thread->StoreBufferAddObject(static_cast<ObjectPtr>(this));
}
- } else {
+ }
+ if ((overlap & kIncrementalBarrierMask) != 0) {
// Incremental barrier: record when a store creates an
// old -> old-and-not-marked reference.
- ASSERT(value->IsOldObject());
if (ClassIdTag::decode(target_tags) == kInstructionsCid) {
// Instruction pages may be non-writable. Defer marking.
thread->DeferredMarkingStackAddObject(value);
@@ -1801,6 +1799,7 @@
friend class MarkingVisitorBase;
template <bool>
friend class ScavengerVisitorBase;
+ friend class Scavenger;
};
// WeakArray is special in that it has a pointer field which is not
@@ -3549,6 +3548,7 @@
friend class MarkingVisitorBase;
template <bool>
friend class ScavengerVisitorBase;
+ friend class Scavenger;
friend class FastObjectCopy; // For OFFSET_OF
friend class SlowObjectCopy; // For OFFSET_OF
};
@@ -3581,6 +3581,7 @@
friend class MarkingVisitorBase;
template <bool>
friend class ScavengerVisitorBase;
+ friend class Scavenger;
friend class ObjectGraph;
friend class FastObjectCopy; // For OFFSET_OF
friend class SlowObjectCopy; // For OFFSET_OF
@@ -3694,6 +3695,7 @@
friend class MarkingVisitorBase;
template <bool>
friend class ScavengerVisitorBase;
+ friend class Scavenger;
friend class ObjectGraph;
};
diff --git a/runtime/vm/runtime_entry.cc b/runtime/vm/runtime_entry.cc
index 6803163..2d128b2 100644
--- a/runtime/vm/runtime_entry.cc
+++ b/runtime/vm/runtime_entry.cc
@@ -527,25 +527,18 @@
uword /*ObjectPtr*/ object_in,
Thread* thread) {
ObjectPtr object = static_cast<ObjectPtr>(object_in);
- // The allocation stubs will call this leaf method for newly allocated
- // old space objects.
- RELEASE_ASSERT(object->IsOldObject());
// If we eliminate a generational write barriers on allocations of an object
// we need to ensure it's either a new-space object or it has been added to
// the remembered set.
//
- // NOTE: We use reinterpret_cast<>() instead of ::RawCast() to avoid handle
+ // NOTE: We use static_cast<>() instead of ::RawCast() to avoid handle
// allocations in debug mode. Handle allocations in leaf runtimes can cause
// memory leaks because they will allocate into a handle scope from the next
// outermost runtime code (to which the generated Dart code might not return
// in a long time).
bool add_to_remembered_set = true;
- if (object->untag()->IsRemembered()) {
- // Objects must not be added to the remembered set twice because the
- // scavenger's visitor is not idempotent.
- // Might already be remembered because of type argument store in
- // AllocateArray or any field in CloneContext.
+ if (object->IsNewObject()) {
add_to_remembered_set = false;
} else if (object->IsArray()) {
const intptr_t length = Array::LengthOf(static_cast<ArrayPtr>(object));
diff --git a/runtime/vm/tagged_pointer.h b/runtime/vm/tagged_pointer.h
index 9b50522..6632f5a 100644
--- a/runtime/vm/tagged_pointer.h
+++ b/runtime/vm/tagged_pointer.h
@@ -311,6 +311,12 @@
#define DEFINE_COMPRESSED_POINTER(klass, base) \
class Compressed##klass##Ptr : public Compressed##base##Ptr { \
public: \
+ Compressed##klass##Ptr* operator->() { \
+ return this; \
+ } \
+ const Compressed##klass##Ptr* operator->() const { \
+ return this; \
+ } \
explicit Compressed##klass##Ptr(klass##Ptr uncompressed) \
: Compressed##base##Ptr(uncompressed) {} \
const klass##Ptr& operator=(const klass##Ptr& other) { \
diff --git a/runtime/vm/thread.cc b/runtime/vm/thread.cc
index 1071751..f54c685 100644
--- a/runtime/vm/thread.cc
+++ b/runtime/vm/thread.cc
@@ -850,6 +850,11 @@
UntaggedObject::kIncrementalBarrierMask;
}
+void Thread::MarkingStackFlush() {
+ isolate_group()->marking_stack()->PushBlock(marking_stack_block_);
+ marking_stack_block_ = isolate_group()->marking_stack()->PopEmptyBlock();
+}
+
void Thread::DeferredMarkingStackRelease() {
MarkingStackBlock* block = deferred_marking_stack_block_;
deferred_marking_stack_block_ = nullptr;
@@ -861,6 +866,13 @@
isolate_group()->deferred_marking_stack()->PopEmptyBlock();
}
+void Thread::DeferredMarkingStackFlush() {
+ isolate_group()->deferred_marking_stack()->PushBlock(
+ deferred_marking_stack_block_);
+ deferred_marking_stack_block_ =
+ isolate_group()->deferred_marking_stack()->PopEmptyBlock();
+}
+
Heap* Thread::heap() const {
return isolate_group_->heap();
}
@@ -954,7 +966,7 @@
for (; first != last + 1; first++) {
ObjectPtr obj = *first;
// Stores into new-space objects don't need a write barrier.
- if (obj->IsImmediateOrNewObject()) continue;
+ if (obj->IsImmediateObject()) continue;
// To avoid adding too much work into the remembered set, skip large
// arrays. Write barrier elimination will not remove the barrier
@@ -983,13 +995,16 @@
switch (op_) {
case Thread::RestoreWriteBarrierInvariantOp::kAddToRememberedSet:
- obj->untag()->EnsureInRememberedSet(current_);
+ if (obj->IsOldObject()) {
+ obj->untag()->EnsureInRememberedSet(current_);
+ }
if (current_->is_marking()) {
current_->DeferredMarkingStackAddObject(obj);
}
break;
case Thread::RestoreWriteBarrierInvariantOp::kAddToDeferredMarkingStack:
// Re-scan obj when finalizing marking.
+ ASSERT(current_->is_marking());
current_->DeferredMarkingStackAddObject(obj);
break;
}
@@ -1022,8 +1037,7 @@
// Dart frames preceding an exit frame to the store buffer or deferred
// marking stack.
void Thread::RestoreWriteBarrierInvariant(RestoreWriteBarrierInvariantOp op) {
- ASSERT(IsAtSafepoint() || OwnsGCSafepoint());
- ASSERT(IsDartMutatorThread());
+ ASSERT(IsAtSafepoint() || OwnsGCSafepoint() || this == Thread::Current());
const StackFrameIterator::CrossThreadPolicy cross_thread_policy =
StackFrameIterator::kAllowCrossThreadIteration;
diff --git a/runtime/vm/thread.h b/runtime/vm/thread.h
index 33ec525..5bf9089 100644
--- a/runtime/vm/thread.h
+++ b/runtime/vm/thread.h
@@ -1396,8 +1396,10 @@
void MarkingStackRelease();
void MarkingStackAcquire();
+ void MarkingStackFlush();
void DeferredMarkingStackRelease();
void DeferredMarkingStackAcquire();
+ void DeferredMarkingStackFlush();
void set_safepoint_state(uint32_t value) { safepoint_state_ = value; }
void EnterSafepointUsingLock();
diff --git a/runtime/vm/thread_registry.cc b/runtime/vm/thread_registry.cc
index e97c166..98e95c0 100644
--- a/runtime/vm/thread_registry.cc
+++ b/runtime/vm/thread_registry.cc
@@ -105,6 +105,19 @@
}
}
+void ThreadRegistry::FlushMarkingStacks() {
+ MonitorLocker ml(threads_lock());
+ Thread* thread = active_list_;
+ while (thread != nullptr) {
+ if (!thread->BypassSafepoints() && thread->is_marking()) {
+ thread->MarkingStackFlush();
+ thread->DeferredMarkingStackFlush();
+ ASSERT(thread->is_marking());
+ }
+ thread = thread->next_;
+ }
+}
+
void ThreadRegistry::AddToActiveListLocked(Thread* thread) {
ASSERT(thread != nullptr);
ASSERT(threads_lock()->IsOwnedByCurrentThread());
diff --git a/runtime/vm/thread_registry.h b/runtime/vm/thread_registry.h
index 7ec348b..24b9468 100644
--- a/runtime/vm/thread_registry.h
+++ b/runtime/vm/thread_registry.h
@@ -37,6 +37,7 @@
void ReleaseStoreBuffers();
void AcquireMarkingStacks();
void ReleaseMarkingStacks();
+ void FlushMarkingStacks();
// Concurrent-approximate number of active isolates in the active_list
intptr_t active_isolates_count() { return active_isolates_count_.load(); }
