Possible CPU usage spikes

[timkendall]

Just upgraded our large GraphQL service to apollo-server-koa@2 and am seeing inconsistent CPU spikes all over the place. This seems to have caused inconsistent query response times (sometime queries are fast and sometimes they take many seconds).

Haven't had a chance to dig in yet but figured it was worth reporting now. Worth noting that we rely heavily on remote schema stitching.

Mega Spikes:

[timkendall]

Graphql.js 14.0.0 may fix this it seems, see:

• Fix memory leak in buildSchema/extendSchema graphql/graphql-js#1417
• Prevent infinite recursion on invalid unions graphql/graphql-js#1427

Unfortunately Graphql.js 14.0.0 has not fixed it yet, we are experiencing multiple crashes every couple of hours, and to this day we have not been able to find the source of the issue, no errors, no warnings, just the CPU spikes up, causes a domino effect in all of our ECS instances, 5 minutes later, everything goes back to normal, until the next series of crashes. Completely giving up on anything Node.js related.

[timkendall]

Ugh thats gnarly. @Human-a what version of node are you guys on (we're on v10.9.0)?

@timkendall Yes, it surely is gnarly. We are on v8.12.0
I have also contacted AWS support (we have the business support plan) and they are looking into the issue to find the source of the issue. I will reply back here if they give me something useful, you may find it useful too.

[ZhimaoL]

I am wondering have u guys solved the CPU problems? I tried to run apollo server as API gateway (simple logic in apollo server) on kubernetes, and experienced a very high CPU usage. Especially the socket connections with redis pubsub, the CPU will reach 1.5 core quickly. Do you guys have any related experiences? @Human-a @timbotnik

@ZhimaoL In our case, the issue was not related to GraphQL, it was caused by another library we were using, which was causing infinite loops, and because NodeJS is rather dumb, we could not see any errors or warnings. All I did was to use the library on the client side, and bam, there was the error on our console.

The library that was causing the issue: https://github.com/TinyNova/deep-omit-by
We fixed the infinite loop it by doing so JSON.parse(JSON.stringify(ourObject))

I hope this helps. I would suggest looking elsewhere for the source of your CPU spikes first.

[AndryKwiatow]

Hi, guys. Do you have any update in this? I'm experiencing the same in my EC2 instance

[glasser]

There is not enough information here to usefully investigate. (For example, at least one of the cases above was "a user had an infinite loop in their code".)