[incremental compiler] Don't process dill libraries twice

An internal benchmark via kernel worker of lots of outline
calculations in worker mode with reuse and the incremental compiler
(and lots of dill loaded dependencies) goes from ~160 seconds to ~150 seconds.

Change-Id: I80afa10caacef5e14e569928a1d421bd3e8ab342
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/104003
Reviewed-by: Kevin Millikin <kmillikin@google.com>
Commit-Queue: Jens Johansen <jensj@google.com>
diff --git a/pkg/front_end/lib/src/fasta/incremental_compiler.dart b/pkg/front_end/lib/src/fasta/incremental_compiler.dart
index 16d1187..99cba9b 100644
--- a/pkg/front_end/lib/src/fasta/incremental_compiler.dart
+++ b/pkg/front_end/lib/src/fasta/incremental_compiler.dart
@@ -885,9 +885,14 @@
       }
     }
 
-    userBuilders?.forEach(addBuilderAndInvalidateUris);
     if (userCode != null) {
+      // userCode already contains the builders from userBuilders.
       userCode.loader.builders.forEach(addBuilderAndInvalidateUris);
+    } else {
+      // userCode was null so we explicitly have to add the builders from
+      // userBuilders (which cannot be null as we checked initially that one of
+      // them was non-null).
+      userBuilders.forEach(addBuilderAndInvalidateUris);
     }
 
     recordInvalidatedImportUrisForTesting(invalidatedImportUris);