TypeScript slow to respond when switching between projects

[dgieselaar]

Issue Type: Performance Issue

For Kibana, we have a TypeScript setup with nested tsconfig.json files. Over the course of the last few months, I and others have started seeing serious latency (10s+) when resolving type information (either via suggestions or code navigation). We've also tried the same setup in Webstorm, and after some initial delay due to indexing, feedback from TypeScript is almost instant in WS.

Our setup (where ./ is the root directory of the Kibana repo):

• ./tsconfig.json: type checks ./src/**/*
• ./x-pack/tsconfig.json: extends the root tsconfig.json and type checks ./x-pack/**/*
• We use yarn workspaces, so npm modules can be resolved from both ./node_modules and ./x-pack/node_modules

The latency issues I have been experiencing are mostly when I'm working on a file that belongs to the x-pack project, and I want to resolve type information for an npm module. "Local" files are pretty fast to respond with type information (sub-second), but for npm modules, feedback consistently takes around 10 seconds. I've inspected the logs, and seemingly the delay happens because every time type information for an npm module is requested, the TypeScript service starts the root project, and closes it again after the tab is closed. I've taken the following steps to reproduce the issue:

• Check out elastic/kibana@e14dcb2 (this was the most recent commit when I reproduced the issue)
• Run yarn kbn:bootstrap in the root of the directory
• Open the Kibana repo in VS Code
• Open x-pack/legacy/plugins/apm/public/components/shared/ApmHeader/index.tsx
• Observe that it takes around 30s for the TypeScript server to start responding with type information
• Open x-pack/legacy/plugins/apm/public/components/shared/ApmHeader/DatePicker/index.tsx via the import statement.
• Observe that the file is opened (near-)instantly.
• Open react from DatePicker/index.tsx, again via the import statement
• Observe that the spinner ("Initializing TS/JS language features") starts
• Observe that it takes around 10s for the file (node_modules/@types/react/index.d.ts) to open

It's important to note that this doesn't seem to happen once, but every time type information from the inactive project is needed. That means that initially, type info for DatePicker is instant, but is delayed for react. After hovering over react, type information for DatePicker is suddenly delayed.

I've experimented with splitting up the two projects into smaller ones which seems to resolve the issue (PR here), but maybe it's possible that it could be fixed in VS Code as well. I'm optimistic that this is possible given the fact that Webstorm can display instant feedback.

Logs:
ts-logs.txt

VS Code version: Code 1.39.2 (6ab598523be7a800d7f3eb4d92d7ab9a66069390, 2019-10-15T15:33:00.827Z)
OS version: Darwin x64 18.7.0

System Info

Item	Value
CPUs	Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz (12 x 2900)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: enabled oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: enabled skia_deferred_display_list: disabled_off skia_renderer: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: disabled_off webgl: enabled webgl2: enabled
Load (avg)	2, 2, 2
Memory (System)	32.00GB (7.88GB free)
Process Argv
Screen Reader	no
VM	0%

Process Info

CPU %	Mem MB	   PID	Process
   14	   164	 47107	code main
    5	    98	 47108	   gpu-process
    5	    98	 47498	   shared-process

Workspace Info

|  Window (index.tsx — kibana-backup)
|    Folder (kibana-backup): more than 24161 files
|      File types: js(5297) ts(5054) md(949) tsx(887) json(634) scss(579)
|                  png(435) map(347) asciidoc(302) snap(202)
|      Conf files: package.json(98) tsconfig.json(36) webpack.config.js(4)
|                  grunt.js(2) settings.json(1) gulp.js(1)
|                  jsconfig.json(1);

Extensions: none

[DanielRosenwasser]

Thank you very much for reporting on this! I believe I've run into similar issues as well. @amcasey's been working on perf for a bunch of different projects and he's currently got his hands full, but may be able to look into this soon. Just wanted to be up-front about expectations here.

[dgieselaar]

Thanks for the update @DanielRosenwasser, let me know if there's anything we can do to help.

[DanielRosenwasser]

btw, this might be a more-actionable version of #34783.

[amcasey]

@dgieselaar Thanks for the details! I'm having a bit of trouble with yarn kbn:bootstrap, so I'll have to pick this up next week. While I'm working on that, could you possibly try VS Code Insiders with typescript.tsserver.maxTsServerMemory: 4000 (see microsoft/vscode#82630)? It is unlikely to affect the startup delay, but may eliminate the delays you're seeing when switching between projects. Thanks!

[dgieselaar]

@amcasey It doesn't make a lot of difference, from what I can tell. I increased it to 8gb as well. Something I didn't notice before is that while a regular hover is pretty much instant, holding down ⌘ when hovering seems to trigger the project switching.

If you're having trouble running yarn kbn:bootstrap, try running yarn first. Would be happy to assist if you can let me know whatever output you get from the bootstrap command.

[amcasey]

@dgieselaar Thanks for the update. Your log didn't indicate memory issues, but sometimes it crashes before writing that error, so it seemed worth a shot.

Assuming that the mapping is straightforward (i.e. that command-click on Mac = ctrl-click on Windows), you're probably seeing speculative go-to-definition execution when your cursor passes over an identifier while you're holding command. That's consistent with other reports of GTD causing this issue.

I'll give kbn:bootstrap another shot today and reach out if I can't get it working. Thanks!

[amcasey]

@dgieselaar Still no luck. Some ignorant questions:

1. Are there things that are expected to be on the path? tsc seems expected to run TS 3.5.3, for example. Running plain yarn also seems to try to build some native components - are there particular compilers or tools that I need installed?
2. Is there a preferred OS? I've been trying on MacOS.
3. Are yarn kbn bootstrap (recommended in the contributor guidelines) and yarn kbn:bootstrap (from your steps) equivalent? It seems like kbn:bootstrap might not be installing dependencies.
4. Are you expecting the project to build cleanly at elastic/kibana@e14dcb2? I'm getting errors, but I think they're about modules that would have been created during the yarn steps that I'm having trouble with.

Thanks,
Andrew

Edit: I switched to Windows, installed [email protected] globally, and ran yarn kbn bootstrap (no colon) and it seems to have worked.

Edit 2: I can repro the issue as original described.

[amcasey]

Once you've got ApmHeader and DatePicker open, you can flip between them and ctrl-mouse over 'react' to see a delay - crazy.

[amcasey]

The repro starts with just kibana/x-pack/tsconfig.json open but, as soon as you open kibana/node_modules/@types/react/index.d.ts, kibana/tsconfig.json gets loaded too. The second initialization is for that project. Probably, GTD should never trigger loading of another project. The first step is for editors to specify which project they're opening the file for. After that, there may be some cleanup required for project references (I think we call the feature "declaration mapping").

[dgieselaar]

Thanks for looking into this Andrew, and glad you reproduced it. I am away for our conf this week so a little low on bandwidth, but let me know if you need anything else. Op di 5 nov. 2019 18:53 schreef Andrew Casey <[email protected]>:

…

The repro starts with just kibana/x-pack/tsconfig.json open but, as soon as you open ***@***.***/react/index.d.ts, kibana/tsconfig.json gets loaded too. The second initialization is for that project. Probably, GTD should never trigger loading of another project. The first step is for editors to specify which project they're opening the file for. After that, there may be some cleanup required for project references (I think we call the feature "declaration mapping"). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34843?email_source=notifications&email_token=AACWDXECFW7KMXQGFGCYCOTQSIBPTA5CNFSM4JHCMDE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDEYB5A#issuecomment-550076660>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACWDXGEWDGXXW34UJLRKMLQSIBPTANCNFSM4JHCMDEQ> .

[amcasey]

@dgieselaar I've got a reduced repro, so I don't think there'll be anything for you to do until there's a fix to test. Thanks again for the detailed steps!

[amcasey]

@dgieselaar Turns out I do have a question: do you ever build that root tsconfig.json or is it just a base file that others extend? If the latter, you can probably eliminate the delay just by renaming it to tsconfig.base.json (for example).

Here's what's going on:

1. Open a file in a/b/c/tsconfig.json
2. GTD on a module name that comes from node_modules
3. Open a/node_modules/target.d.ts
4. Walk up the directory hierarchy from a/node_modules/target.d.ts looking for a containing tsconfig.json
5. Find a/tsconfig.json and open that
6. Pause the world while that project loads

I found (4) pretty surprising, but it's by design and some customers rely on it. We think there might be a special case where we shouldn't do it for node_modules specifically, but if you happened to go to a non-node_modules module in an analogous location, then you would still end up doing another project load.

By renaming your root tsconfig.json to anything other than "tsconfig.json", you prevent (5) and, thus, (6). It will fall back on opening the target file in an already open project (i.e. the one from (1)).

[amcasey]

@walterra @monfera @spong @timroes @sqren @smith, it sounds like you might be having similar problems. Are you also jumping to a module from node_modules? Also, are you using composite projects (i.e. project references)? Thanks!

[timroes]

@amcasey we're currently all at the same conf, so with a bit longer response times as usual. @spalger pinging you since you're not yet on the list, and best familiar with our build setup. Do you think you could help out here next week? (Also I'm busy upgrading to TS 3.7 ;P)

[dgieselaar]

@amcasey We're not using project references AFAIK. I see you've opened a draft PR, that's awesome! Is it in a state that I can give it a try as well to see if it improves things?

[amcasey]

@dgieselaar Are all of you on the same team/project? I can't tell how many repros I'm working with here.

You can try the PR. We're still discussing some of the implications, but it should be usable. However, for your particular project, I'd still recommend just renaming your root tsconfig.json to tsconfig.base.json, assuming that's an option.

[dgieselaar]

@amcasey thanks! Yeah, all the same repo, I should have clarified 😅. Will give it a shot. Renaming the tsconfig files is something I'm doing locally with a scripted process, but to do it for the whole of Kibana, our operations team probably should have a look at it. Will take it up with them.

[amcasey]

Code review is definitely a good idea, but it should just be a single file rename, plus updates to a dozen or so extends properties, no? That seemed to work when I tried it locally, though I didn't run the the tests or anything.

[dgieselaar]

Yeah there's a bunch of things related to type checking, publishing types etc that I'm not familiar with. Plus this piece is owned by the operations team. But I'll get back to it :) Thanks for your help so far.

[amcasey]

@dgieselaar Any luck with renaming that root tsconfig file?

[amcasey]

Please file a new issue if you're still seeing this.