TS 5.1.3 Regression: TSC Compiler throws FATAL ERROR: Reached heap limit Allocation failed

[virtuallyunknown]

Bug Report

🔎 Search Terms

tsc, compiler, heap limit, crash, memory

🕗 Version & Regression Information

• This is a crash
• This changed between versions 5.0.4 and 5.1.3

⏯ Repository Link

https://github.com/virtuallyunknown/tsc-error

💻 Code

To reproduce the error, clone the repository and run the build script or just npx tsc.

git clone https://github.com/virtuallyunknown/tsc-error.git && cd tsc-error
npm install
npx tsc

🙁 Actual behavior

Running the typescript compiler (tsc) from project root causes the build to fail and produce this error:

<--- Last few GCs --->

[43355:0x6bd0ae0]     9563 ms: Scavenge 4038.1 (4116.6) -> 4037.2 (4117.8) MB, 4.5 / 0.0 ms  (average mu = 0.538, current mu = 0.113) allocation failure; 
[43355:0x6bd0ae0]     9572 ms: Scavenge 4039.0 (4117.8) -> 4038.2 (4138.6) MB, 8.4 / 0.0 ms  (average mu = 0.538, current mu = 0.113) allocation failure; 
[43355:0x6bd0ae0]    10371 ms: Mark-Compact 4052.2 (4138.6) -> 4048.8 (4152.1) MB, 794.9 / 0.0 ms  (average mu = 0.390, current mu = 0.040) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xbf9890 node::Abort() [node]
 2: 0xaf6859  [node]
 3: 0xddaf80 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0xddb336 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 5: 0xfd9db5  [node]
 6: 0xfed4e5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xfc924f v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xfca297 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xfa99fa v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x13bd76f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x18443f9  [node]
Aborted (core dumped)

I also tried typescript@next, but that doesn't fix it. I also tried this command:

NODE_OPTIONS=--max-old-space-size=28000 npx tsc --noEmit --diagnostics --extendedDiagnostics --incremental false

but it crashes with a different error after a while.

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

🙂 Expected behavior

Running the typescript compiler (tsc) command compiles the build successfully after 1-2 seconds, just like it did in 5.0.4.

[zhiyan114]

Can confirm. Not only does it crash, but it also cause the transpile time to increase by ~x6 before the crash happens.

[Andarist]

bisected to #53672 , cc @ahejlsberg

[Andarist]

The problem originates in @sapphire/shapeshift and its ArrayValidator. A reduced test case is this:

type UnshiftTuple<T extends [...any[]]> = T extends [T[0], ...infer Tail] ? Tail : never;
type ExpandSmallerTuples<T extends [...any[]]> = T extends [T[0], ...infer Tail] ? T | ExpandSmallerTuples<Tail> : [];
type Shift<A extends Array<any>> = ((...args: A) => void) extends (...args: [A[0], ...infer R]) => void ? R : never;
type GrowExpRev<A extends Array<any>, N extends number, P extends Array<Array<any>>> = A['length'] extends N ? A : GrowExpRev<[...A, ...P[0]][N] extends undefined ? [...A, ...P[0]] : A, N, Shift<P>>;
type GrowExp<A extends Array<any>, N extends number, P extends Array<Array<any>>> = [...A, ...A][N] extends undefined ? GrowExp<[...A, ...A], N, [A, ...P]> : GrowExpRev<A, N, P>;
type Tuple<T, N extends number> = number extends N ? Array<T> : N extends 0 ? [] : N extends 1 ? [T] : GrowExp<[T], N, [[]]>;

declare class ArrayValidator<T extends unknown[], I = T[number]> {
    lengthRange<S extends number, E extends number>(start: S, endBefore: E): ArrayValidator<Exclude<ExpandSmallerTuples<UnshiftTuple<[...Tuple<I, E>]>>, ExpandSmallerTuples<UnshiftTuple<[...Tuple<I, S>]>>>>;
}

You can use this TS playground to verify this. Just wait for a moment - you will see that the browser tab crashes.

I reduced this to the lengthRange method but in the regression test you might also want to include other methods from this class from the same length* "family". They all do complex things with arrays/tuples.

[belgattitude]

Might be related to #54348 as well ? Edit: nope that one is fixed

[ahejlsberg]

The issue appears to be runaway type instantiation as the compiler is attempting to compute variance information. That logic relies on our various depth limiters to catch infinite recursion, but apparently that doesn't happen fast enough here.

A quick workaround for now is to change the declaration of the Tuple type to

type Tuple<T, N extends number> = N extends number ? number extends N ? Array<T> : N extends 0 ? [] : N extends 1 ? [T] : GrowExp<[T], N, [[]]> : never;

That is, introduce a sanity check that N is actually a number and resolve to never if not. This would seem unnecessary since N is constrained to number, but variance computation introduces special marker types that (by design) aren't related to anything else.

[fatcerberus]

Just wait for a moment - you will see that the browser tab crashes.

Isolated tabs are a wonderful thing. I still remember the days when this kind of thing would have taken out the entire browser.

Still a bit alarming that any JS code is able to outright crash a process (and thereby cause a DoS), though.

[favna]

Many thanks for the fix that we can apply @ahejlsberg! We'll release an update for @sapphire/shapeshift ASAP so I think this issue can be closed.

Edit: I have released @sapphire/shapeshift v3.9.2 that resolves this issue

[virtuallyunknown]

You guys are legends for fixing this so quickly, thanks to everyone involved!

If my understanding is correct, we now wait for the next TS release where this is backported, or just use the nightly version?

[JstnMcBrd]

I also encountered this issue in my node discord.js project after upgrading from TypeScript 5.0.4 to 5.1.3. Enabling the skipLibCheck option has resolved it for now. Looking forward to the fix!

[favna]

I also encountered this issue in my node discord.js project after upgrading from TypeScript 5.0.4 to 5.1.3. Enabling the skipLibCheck option has resolved it for now. Looking forward to the fix!

As an @sapphire/shapeshift update has been rolled out you can update the transitive dependency with your package manager of choice. Once you have updated the transitive dependency you should be able to remove skipLibCheck.

npm
I cannot find anything for npm to do a transitive dependency update so you'll have to provide overrides until @discordjs/builders releases an update (and subsequently discord.js). If someone knows a way to update transitive dependencies with npm please do correct me.
In your package.json:

{
  "overrides": {
    "@sapphire/shapeshift": "3.9.2"
  }
}

Then run npm install. You can verify the change by checking your package-lock.json

Yarn v1
Yarn v1 (deprecated, recommend updating to v3) does not have a way to update transitive dependencies so a similar strategy is required as npm:
In your package.json:

{
  "resolutions": {
    "@sapphire/shapeshift": "3.9.2"
  }
}

Then run yarn install. You can verify the change by checking your yarn.lock

Yarn v3
Yarn v3 can update transitive dependencies so run the command:

yarn up -R @sapphire/shapeshift

pnpm
pnpm can also update transitive dependencies so run the command:

pnpm --recursive update @sapphire/[email protected]