fix(BREAKING CHANGE): dereference caching to prevent infinite loops on circular schemas (#380)

We've come across an interesting case with one of our customers OpenAPI definitions where due to the amount of circular references they are using it ends up sending `$RefParser.dereference()` into an infinite loop that has unfortunately been causing us some pain.

The following is a video of running the `dereference.circular=ignore` test in this PR without my eventual fix:

https://github.com/user-attachments/assets/077129bd-997a-40b7-aa57-8129bd7df87f

I've killed the process after 20 seconds but we've seen cases where it can run for almost an hour trying to dereference this circular API definition. In investigating this we've identified a couple issues:

1. When dereferencing this specific schema the event loop gets blocked in the process preventing the `options.timeoutMs` check and exception from ever getting hit.
    * We were able to resolve this by adding a supplementary timeout check when the dereference crawler processes each property within an object.
2. The dereference cache isn't being fully taken advantage of.

In investigating the cache issues we noticed a couple more issues:

#### Core issues with `Set` caches

The `Set` objects that are used for `parents` and `processedObjects` don't appear to be fully utilized because these are often setting objects, and `Set` does not do object deuping:

```js
const set = new Set();
set.add({ type: 'string' });
set.add({ type: 'string' });

console.log({ set: Array.from(set), has: set.has({ type: 'string'} )})

> {set: Array(2), has: false}
```

I'm not convinced that any of the `.has()` checks being executed on these stores are currently working and I made an attempt at pulling in [flatted](https://npm.im/flatted)[^1] to create consistent and unique keys off of these values however a couple unit tests broken in unexpected ways and ended up moving on. I would love to spend some more time investigating this because I think in extreme cases like ours we could really improve memory usage during dereferencing.

#### The dereference cache is only being used for non-circular objects

After crawling to dereferencing an object and either updating the original schema or resetting it to the  original `$ref` if `dereference.circular=ignore` is set that resulting object is saved into the dereferenced cache `Map`. This map is referenced at the beginning of the `dereference$Ref` function however the cache is only utilized if the found object is **not** circular.

I was unable to uncover a reason why this was the case but without this cache being utilized this dereferencer would continuously, even despite `dereference.circular=ignore` being configured, crawl and re-dereference objects it had already seen. Changing this logic to always return the cache if we found something brought dereferencing our use case down from ∞ to less than a second.

https://github.com/user-attachments/assets/4d38a619-7b0b-4ec8-8f10-a3e728855a84

<sub>Ignore the logs in this video, it was recorded while I was still working on this fix.</sub>

[^1]: Because `JSON.stringify()` cannot serialize circular objects.

BREAKING CHANGE: dereference caching to prevent infinite loops on circular schemas

Author: erunion

lib/dereference.ts

@@ -69,11 +69,8 @@ function crawl<S extends object = JSONSchema, O extends ParserOptions<S> = Parse
     circular: false,
   };
 
-  if (options && options.timeoutMs) {
-    if (Date.now() - startTime > options.timeoutMs) {
-      throw new TimeoutError(options.timeoutMs);
-    }
-  }
+  checkDereferenceTimeout<S, O>(startTime, options);
+
   const derefOptions = (options.dereference || {}) as DereferenceOptions;
   const isExcludedPath = derefOptions.excludedPathMatcher || (() => false);
 
@@ -98,6 +95,8 @@ function crawl<S extends object = JSONSchema, O extends ParserOptions<S> = Parse
         result.value = dereferenced.value;
       } else {
         for (const key of Object.keys(obj)) {
+          checkDereferenceTimeout<S, O>(startTime, options);
+
           const keyPath = Pointer.join(path, key);
           const keyPathFromRoot = Pointer.join(pathFromRoot, key);
 
@@ -214,7 +213,17 @@ function dereference$Ref<S extends object = JSONSchema, O extends ParserOptions<
   const $refPath = url.resolve(shouldResolveOnCwd ? url.cwd() : path, $ref.$ref);
 
   const cache = dereferencedCache.get($refPath);
-  if (cache && !cache.circular) {
+
+  if (cache) {
+    // If the object we found is circular we can immediately return it because it would have been
+    // cached with everything we need already and we don't need to re-process anything inside it.
+    //
+    // If the cached object however is _not_ circular and there are additional keys alongside our
+    // `$ref` pointer here we should merge them back in and return that.
+    if (cache.circular) {
+      return cache;
+    }
+
     const refKeys = Object.keys($ref);
     if (refKeys.length > 1) {
       const extraKeys = {};
@@ -294,6 +303,23 @@ function dereference$Ref<S extends object = JSONSchema, O extends ParserOptions<
   return dereferencedObject;
 }
 
+/**
+ * Check if we've run past our allowed timeout and throw an error if we have.
+ *
+ * @param startTime - The time when the dereferencing started.
+ * @param options
+ */
+function checkDereferenceTimeout<S extends object = JSONSchema, O extends ParserOptions<S> = ParserOptions<S>>(
+  startTime: number,
+  options: O,
+): void {
+  if (options && options.timeoutMs) {
+    if (Date.now() - startTime > options.timeoutMs) {
+      throw new TimeoutError(options.timeoutMs);
+    }
+  }
+}
+
 /**
  * Called when a circular reference is found.
  * It sets the {@link $Refs#circular} flag, executes the options.dereference.onCircular callback,

test/specs/circular-extensive/circular-extensive.spec.ts

@@ -0,0 +1,107 @@
+import { describe, it } from "vitest";
+import $RefParser from "../../../lib/index.js";
+import helper from "../../utils/helper.js";
+import path from "../../utils/path.js";
+
+import { expect } from "vitest";
+
+describe("Schema with an extensive amount of circular $refs", () => {
+  it("should dereference successfully", async () => {
+    const circularRefs = new Set<string>();
+
+    const parser = new $RefParser<Record<string, any>>();
+    const schema = await parser.dereference(path.rel("test/specs/circular-extensive/schema.json"), {
+      dereference: {
+        onCircular: (ref: string) => circularRefs.add(ref),
+      },
+    });
+
+    // Ensure that a non-circular $ref was dereferenced.
+    expect(schema.components?.schemas?.ArrayOfMappedData).toStrictEqual({
+      type: "array",
+      items: {
+        type: "object",
+        properties: {
+          mappingTypeName: { type: "string" },
+          sourceSystemValue: { type: "string" },
+          mappedValueID: { type: "string" },
+          mappedValue: { type: "string" },
+        },
+        additionalProperties: false,
+      },
+    });
+
+    // Ensure that a circular $ref **was** dereferenced.
+    expect(circularRefs).toHaveLength(23);
+    expect(schema.components?.schemas?.Customer?.properties?.customerNode).toStrictEqual({
+      type: "array",
+      items: {
+        type: "object",
+        properties: {
+          customerNodeGuid: expect.any(Object),
+          customerGuid: expect.any(Object),
+          nodeId: expect.any(Object),
+          customerGu: expect.any(Object),
+        },
+        additionalProperties: false,
+      },
+    });
+  });
+
+  it("should dereference successfully with `dereference.circular` is `ignore`", async () => {
+    const circularRefs = new Set<string>();
+
+    const parser = new $RefParser<Record<string, any>>();
+    const schema = await parser.dereference(path.rel("test/specs/circular-extensive/schema.json"), {
+      dereference: {
+        onCircular: (ref: string) => circularRefs.add(ref),
+        circular: "ignore",
+      },
+    });
+
+    // Ensure that a non-circular $ref was dereferenced.
+    expect(schema.components?.schemas?.ArrayOfMappedData).toStrictEqual({
+      type: "array",
+      items: {
+        type: "object",
+        properties: {
+          mappingTypeName: { type: "string" },
+          sourceSystemValue: { type: "string" },
+          mappedValueID: { type: "string" },
+          mappedValue: { type: "string" },
+        },
+        additionalProperties: false,
+      },
+    });
+
+    // Ensure that a circular $ref was **not** dereferenced.
+    expect(circularRefs).toHaveLength(23);
+    expect(schema.components?.schemas?.Customer?.properties?.customerNode).toStrictEqual({
+      type: "array",
+      items: {
+        $ref: "#/components/schemas/CustomerNode",
+      },
+    });
+  });
+
+  it('should throw an error if "options.dereference.circular" is false', async () => {
+    const parser = new $RefParser();
+
+    try {
+      await parser.dereference(path.rel("test/specs/circular-extensive/schema.json"), {
+        dereference: { circular: false },
+      });
+
+      helper.shouldNotGetCalled();
+    } catch (err) {
+      expect(err).to.be.an.instanceOf(ReferenceError);
+      expect(err.message).to.contain("Circular $ref pointer found at ");
+      expect(err.message).to.contain(
+        "specs/circular-extensive/schema.json#/components/schemas/AssignmentExternalReference/properties/assignment/oneOf/0",
+      );
+
+      // $Refs.circular should be true
+      expect(parser.$refs.circular).to.equal(true);
+    }
+  });
+});