Number of processes increases every time remote window is reloaded about vscode HOT 8 OPEN

SSH connection to AWS Linux x64 6.2.0-1010-aws specifically running Ubuntu 22.04.

from vscode.

This does not seem file watcher process specific as the other process typescript/lib/tsserver.js is also increasing each time. Moving to some remote folks for chiming in, I am not sure how sessions are tracked and processes recycled.

from vscode.

Nice script
I tested on WSL and all looks fine there. No increase of watchers after a reload.

from vscode.

@jcready what kind of remote is this? SSH? Docker? WSL? CLI?

from vscode.

Same issue.

Version: 1.88.1
Commit: e170252
Date: 2024-04-10T17:42:52.765Z (2 wks ago)
Electron: 28.2.8
ElectronBuildId: 27744544
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Darwin x64 23.4.0

remote ssh on:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

the scripts output:

from vscode.

I can reproduce that after window reload extension host and file watcher processes pile up when connected to a SSH remote:

Unfortunately this does not reproduce with our test resolver where I see 1 file watcher and extension host per window reload.

from vscode.

One interesting observation is that the PID of "remote agent" seems to stay the same, indicating that this process is kept alive and thus child processes such as file watcher and extension host do not just terminate.

I wonder if this disposable never getting disposed could contribute to it:

Btw this reproduces for older versions, so its not a regression.

from vscode.

@connor4312

We have a number of mechanisms in place for dealing with this scenario and it looks like all are failing. See below for what 1, 2 and 3 are.

In a perfect world, we would have sufficient time on the client machine to be able to forward the final messages sent via 1. and 2. from the renderer process. But that doesn't always work (sometimes the renderer crashes), so at least 3. would be important to get right (i.e. closing the sockets to the server when the renderer reloads / the WebSocket is closed). We would preferably get all 3 working OK like they work in the case without a resolver.

By the way, to try out core remoting support without a resolver you can use (I just checked again and 1. and 2. work without a resolver):

./scripts/code-server.sh --without-connection-token
./scripts/code.sh --folder-uri=vscode-remote://localhost:9888/Users/alex/src/working_dir

1. Sending ProtocolMessageType.Disconnect on the management connection

The call stack looks something like this:

• lifecycleService.onDidShutdown
• RemoteAgentConnection.dispose
• PersistentConnection.dispose
• Client<TContext = string>.dispose
• PersistentProtocol.sendDisconnect
• The buffered write is flushed so WebSocket.send should be called immediately and should "make it" before the window goes down.

2. Sending a Terminate and a ProtocolMessageType.Disconnect on the remote extension host connection

• lifecycleService.onDidShutdown
• _doStopExtensionHosts going first to the remote extension host before the local one
• ExtensionHostManager.dispose
• sending Terminate, which is a concept extension hosts have irrespective of remote
• PersistentProtocol.sendDisconnect
• The buffered write is flushed so WebSocket.send should be called immediately and should "make it" before the window goes down.

3. Listening to socket close events on the server side

On the server side we listen to socket close events and then install a 3hrs timeout to wait for reconnections.

• For the management connection, this happens here. As soon as this fires, we log as INFO "The client has disconnected, will wait for reconnection...". So seeing this log is a good indication that we're getting a socket close event. See below in the server log, this log message doesn't appear.
• For the remote extension host, this happens here.
• 3hrs is a lot, so to support cases in which 1. & 2. failed, whenever a new succesful connection occurs (like in this repro with a window reload), we loop through all connections, and for those who are on the 3hrs wait time, we shorten their wait time to 5min. This happens here.

Server log

This is when reloading around 3 times. Note there is no "The client has disconnected, will wait for reconnection..." message.

2024-05-03 11:31:55.228 [info] 




2024-05-03 11:31:55.386 [info] Extension host agent started.
2024-05-03 11:31:55.446 [info] [<unknown>][c0889be6][ManagementConnection] New connection established.
2024-05-03 11:31:55.446 [info] [<unknown>][1bc96464][ExtensionHostConnection] New connection established.
2024-05-03 11:31:55.571 [info] [<unknown>][1bc96464][ExtensionHostConnection] <83418> Launched Extension Host Process.
2024-05-03 11:31:55.630 [warning] Skipping extension /Users/alex/.vscode-server-insiders/extensions/ms-vscode.js-debug-1.72.1 in favour of the builtin extension /Users/alex/.vscode-server-insiders/cli/servers/Insiders-a1f2ea3162c704a5890c35dde4df6ee46283d957/server/extensions/ms-vscode.js-debug.
2024-05-03 11:36:55.391 [info] New EH opened, aborting shutdown
2024-05-03 11:37:11.365 [info] [<unknown>][c5315465][ExtensionHostConnection] New connection established.
2024-05-03 11:37:11.366 [info] [<unknown>][d37d0172][ManagementConnection] New connection established.
2024-05-03 11:37:11.368 [info] [<unknown>][c5315465][ExtensionHostConnection] <10544> Launched Extension Host Process.
2024-05-03 11:37:11.428 [warning] Skipping extension /Users/alex/.vscode-server-insiders/extensions/ms-vscode.js-debug-1.72.1 in favour of the builtin extension /Users/alex/.vscode-server-insiders/cli/servers/Insiders-a1f2ea3162c704a5890c35dde4df6ee46283d957/server/extensions/ms-vscode.js-debug.
2024-05-03 11:42:11.822 [info] [<unknown>][1bc96464][ExtensionHostConnection] <83418> Extension Host Process exited with code: 0, signal: null.
2024-05-03 11:58:27.323 [info] [<unknown>][d7222e53][ManagementConnection] New connection established.
2024-05-03 11:58:27.325 [info] [<unknown>][e4939ad8][ExtensionHostConnection] New connection established.
2024-05-03 11:58:27.330 [info] [<unknown>][e4939ad8][ExtensionHostConnection] <21783> Launched Extension Host Process.
2024-05-03 11:58:27.358 [warning] Skipping extension /Users/alex/.vscode-server-insiders/extensions/ms-vscode.js-debug-1.72.1 in favour of the builtin extension /Users/alex/.vscode-server-insiders/cli/servers/Insiders-a1f2ea3162c704a5890c35dde4df6ee46283d957/server/extensions/ms-vscode.js-debug.
2024-05-03 12:03:27.691 [info] [<unknown>][c5315465][ExtensionHostConnection] <10544> Extension Host Process exited with code: 0, signal: null.

from vscode.