Completions stop working during heavy usage

[chris-armstrong]

I'm using Onivim2 as my day-to-day editor (it's pretty good!), and a recurrent problem I'm finding is that the completion suggestions will stop working at some point after starting it up, requiring a restart of Onivim2 to get them working again. I've tried executing the 'Typescript: Restart TS Server' option, but this doesn't seem to work.

I'm running completions on TypeScript files. I ran onivim2 with logging enabled, and found this at the time of its failure which suggestions the VSCode extension host died:

[INFO]  [1345.894s] Exthost.Transport : Connection closed.
[WARN]  [1345.894s] Exthost.Client : Unhandled message: Exthost_Protocol.Message.Incoming.Disconnected
[INFO]  [1345.895s] Oni2.Extension.ClientStore : Extension host process exited with exit status: 0 and signal: 6
[INFO]  [1345.896s] Oni2.Store.dispatch : dispatch: KeyboardInput {isText = false; input = "<BS>"}
[INFO]  [1345.897s] Oni2.Store.dispatch : dispatch: (LanguageSupport (Feature_LanguageSupport.Hover (KeyPressed "<BS>")))
[INFO]  [1345.897s] Oni2.Store.dispatch : dispatch: (Buffers
   Update {update = <opaque>; oldBuffer = <opaque>; newBuffer = <opaque>;
     triggerKey = (Some "<BS>")})
[INFO]  [1345.898s] Oni2.Store.dispatch : dispatch: (Vim
   Feature_Vim.ModeChanged {allowAnimation = true; mode = <opaque>;
     subMode = <opaque>; effects = <opaque>})
[ERROR] [1345.914s] Oni2.Extension.ClientStore : Stream.write: bad file descriptor

The last error, Stream.write bad file descriptor continues to occur after this point (I could find no instances of it before).

I know this only suggests the extension host received a SIGABRT (maybe an out of memory condition?), so I realise I'll probably have to dig deeper. Can anyone suggest how I might capture some more information about the extension host connection or from the extension host itself to debug this, or any other pointers?

[bryphe]

Thanks for logging the issue @chris-armstrong and grabbing the logs!

using Onivim2 as my day-to-day editor (it's pretty good!)

This is great to hear 😄 Sounds like you're exercising it thoroughly.

I know this only suggests the extension host received a SIGABRT (maybe an out of memory condition?), so I realise I'll probably have to dig deeper.

I believe your assessment is accurate - my suspicion is that we're hitting the heap memory limit of Node (suggesting some memory leak). Another possibility is we're hitting the open file descriptor limit. I'm going to take a look and see if I see anything odd in the memory profiler.

Some extra data points that might be helpful:

• What version of Onivim were you running? The 0.5.2 stable, or the nightly build?
• What platform are you running on? Windows, Linux, OSX?

Also I've upgraded our Node version for the extension host in #3012 - Node 12 -> 14 goes from the v8 Javascript engine version v7.x to v8.x (with several performance improvements), but I don't think this is the root issue.

[chris-armstrong]

I'm using the nightly build on MacOS X. Is there any debugging or profiling I can run locally? Node.js is more my domain than Reason :-) and it would be nice to get to the bottom of this.

[bryphe]

Definitely - we can actually run the node debug tools - if you do the following:

1. Open Activity Monitor to get the PID of the extension host process:

2. Run node debug -p <process id>

3. Open up Chrome and go to chrome://inspect'

That should show the extension host process available for inspection:

And then we can use the performance profiling / memory allocation tools:

I do think there may be a memory leak in the completions or signature help that could be getting us into this state - when I run the profiler I'm seeing some retained objects there:

[chris-armstrong]

When I profile the memory, I'm seeing the same object tree on SuggestAdapter - the `ExtHostLanguageFeatures` class seems to keep a cache of suggestion items, which AFAICT aren't removed until `$releaseCompletionItems` is called - is it possible there needs to be a message back to the extension host telling it when it can let go of suggestions? https://github.com/onivim/vscode-exthost/blob/master/src/vs/workbench/api/common/extHostLanguageFeatures.ts

[bryphe]

@chris-armstrong - ah, nice investigation, thanks for profiling too! I was thinking the same thing. We're missing those $release* APIs which are needed to clean up the _cached suggest items. Working on a fix in #3016

Looks like the signature help / codelens may also be affected

[chris-armstrong]

Thanks! Looking forward to the fix for this, apart from buggy vim-surround support, this is probably the main blocker for me day-to-day with Onivim2

[chris-armstrong]

This is behaving so much better now that the #3016 fix is in. I've profiled it again and the memory usage is much more stable and not growing.

Happy for this to be closed.

[bryphe]

Awesome news! Thanks for the update and all your help investigating @chris-armstrong - I'll close this out!