Comments (29)
@silversword411 im not sure if thats the case?
because my other post here about duplicate agents fixes this exact issue #6205 (comment)
if it notices duplicate connects/agents, it disconnects the first one lets the second one through
altho it could not closing the connect properly i suppose?
from meshcentral.
something happened between 11 and 12 from the looks of the graph
going to sound like a DAFT one, can you disable/remove the autoBackup and restart and monitor?
and the fact it looks like its loading itself over and over again in the pic doesnt look good either?
from meshcentral.
going to sound like a DAFT one, can you disable/remove the autoBackup and restart and monitor?
I disabled autobackup, rebooted the server, and watched the server work.
from meshcentral.
Sometimes this error appears in the logs:
-------- 6/16/2024, 9:33:59 PM ---- 1.1.24 --------
(node:55552) Warning: An error event has already been emitted on the socket. Please use the destroy method on the socket while handling a 'clientError' event.
(Use node --trace-warnings ...
to show where the warning was created)
but I don't think it's related to the problem
from meshcentral.
@sheshko-as that issue has been around for about a year,
it first popped up when we had to move to node 14 and upgraded expressjs
havent been able to track down what line is causing it yet but i dont think its effecting you
UNLESS the timestamp of the event is WHEN you notice memory being increased?
from meshcentral.
going to sound like a DAFT one, can you disable/remove the autoBackup and restart and monitor?
It didn't help
from meshcentral.
Server Error Log:
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
<--- Last few GCs --->
[89911:0x5ff48b0] 17700692 ms: Mark-sweep 4047.0 (4138.1) -> 4034.2 (4141.1) MB, 2816.1 / 0.0 ms (average mu = 0.346, current mu = 0.030) allocation failure; scavenge might not succeed
[89911:0x5ff48b0] 17705461 ms: Mark-sweep 4050.1 (4141.1) -> 4037.5 (4144.4) MB, 4691.7 / 0.0 ms (average mu = 0.167, current mu = 0.016) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
1: 0xb9c310 node::Abort() [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
2: 0xaa27ee [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
3: 0xd73eb0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
4: 0xd74257 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
5: 0xf515d5 [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
6: 0xf63aad v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
7: 0xf3e19e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
8: 0xf3f567 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
9: 0xf2076a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
10: 0x12e599f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
-------- 6/19/2024, 9:02:28 PM ---- 1.1.24 --------
11: 0x17125f9 [/usr/bin/node]
from meshcentral.
It also didn't help "Useful config.js settings":
"AgentsInRAM": false,
"AgentUpdateBlockSize": 2048,
"agentUpdateSystem": 1,
"noAgentUpdate": 1,
"WsCompression": false,
"AgentWsCompression": false,
https://ylianst.github.io/MeshCentral/meshcentral/debugging/
from meshcentral.
Are you using some kind of VPN/proxy between agent and server?
Can you monitor the ws connections between agent and server? By default they will stay up once established for 24hrs...but I've seen VPN and other networking software/proxies prematurely close ws connections.
When mesh realizes its connection is dead it stands up a new connection but I think the old isn't cleaned....memory leak.
from meshcentral.
we need to find out what you did/happens when the memory starts to climb
did you connect to a certain device?
did loaads of devices connect?
did someone stop a recording session and its saving from memory to file?
was the loads of login attempts?
have u tried switching from mongodb to just nedb or say mysql? incase its a database issue
from meshcentral.
Are you using some kind of VPN/proxy between agent and server?
We don't use it, agents connect directly by domain name
from meshcentral.
we need to find out what you did/happens when the memory starts to climb did you connect to a certain device? did loaads of devices connect? did someone stop a recording session and its saving from memory to file? was the loads of login attempts? have u tried switching from mongodb to just nedb or say mysql? incase its a database issue
I'm trying to monitor all this, but I don't see any pattern yet. The database is the only thing that I have not changed yet, I will try to change it to postgresql, for example. There are specifics: we have a lot of client computers with a frozen C disk through the Shadow Defender program, as well as a lot of computers that work without disks through one vhd image (for example, one vhd disk can be on 30-40 PCs at once) for these computers, it is worth deleting in the group settings when the computer is offline.
from meshcentral.
@sheshko-as wow that does sound mad/complex!
i mean you could be duplicating the meshids if you are using VHD images!?
and that could be causing an issue/confusion of the server!
so maybe yeh set a few groups to delete themselves when they disconnect and see if it helps?
from meshcentral.
@sheshko-as wow that does sound mad/complex! i mean you could be duplicating the meshids if you are using VHD images!? and that could be causing an issue/confusion of the server! so maybe yeh set a few groups to delete themselves when they disconnect and see if it helps?
All groups that use one VHD image per group are configured to be deleted after the computer becomes offline.
from meshcentral.
@sheshko-as hmmm and they vanish ok?
this is getting very stange now as i liteally have no idea whats causing the memory leak/increase
you COULD try running node --trace-warnings node_modules/meshcentral
and see if that every displays any output before it crashes
it could very well be the --trace-warnings
issue,
also you could try using the latest node 20.15.0 as the was push for a fix to do with misleaking messages
nodejs/node#51204
from meshcentral.
hmmm and they vanish ok?
yes
from meshcentral.
you COULD try running
node --trace-warnings node_modules/meshcentral
and see if that every displays any output before it crashes
it could very well be the--trace-warnings
issue,
also you could try using the latest node 20.15.0 as the was push for a fix to do with misleaking messages
nodejs/node#51204
Okay, I'll do everything, I'll write you back based on the result
from meshcentral.
@sheshko-as no worries!
in theory the node 20.13+ will fix the Warning: An error event has already been emitted on the socket. Please use the destroy method on the socket while handling a 'clientError' event.
messages
(explains why im not seeing them anymore as i moved to node 20)
but i will downgrade my setup to node 18 and monitor see if it increases ram wise too
from meshcentral.
@sheshko-as no worries! in theory the node 20.13+ will fix the
Warning: An error event has already been emitted on the socket. Please use the destroy method on the socket while handling a 'clientError' event.
messages (explains why im not seeing them anymore as i moved to node 20) but i will downgrade my setup to node 18 and monitor see if it increases ram wise too
updated to 20.15, did not help, the problem was repeated today
from meshcentral.
@sheshko-as which issue sorry? u mean the memory increase/crash?
did you see when it started increasing and if you did anything like connect to a computer?
in theory the An error event has already been emitted
should vanish with the latest LTS of node 20
from meshcentral.
did you see when it started increasing and if you did anything like connect to a computer?
I'm trying to figure out what action causes uncontrolled RAM growth to begin, but it's not working yet.
from meshcentral.
u mean the memory increase/crash?
yes
from meshcentral.
I think I found it: the problem is due to relay sessions, for many users, when opening a lot of RDP, several MeshRouter windows open, and not one, as expected. I think the problem occurs when meshcentral users run multiple copies of the MeshCentralRouter application due to an error when clicking the RDP button in the browser. If the user has only one instance of the MeshCentralRouter program running, RAM does not grow. I'll try to find the reason why this is happening.
from meshcentral.
OK that's an interesting theory!
So its opening multiple meshcentralrouter sessions that seems to increase the ram on the server side!
I'll have to test myself, sadly I don't have 32 comps that have rdp. But I suppose I could open 32 remote desktops and see if the memory starts increasing!
from meshcentral.
do you think a low-quality VPS can cause this kind of RAM leak behavior?
from meshcentral.
@sheshko-as sorry ive not had chance to test yet, been working on the android app, and also poorly (full of cold/flu again!)
in theory ur VPS u have is already a good spec, 4 core, 16gb ram! so i cant see the VPS being the issue.
from meshcentral.
I changed the VPS provider again, to the best one in our country, the problem does not manifest itself for a day.
This problem also no longer occurs: #6127
I'll run the tests for a couple more days and let you know the result.
from meshcentral.
I think the problem is related to this error: #6127
If I return obj.user._id, then an error occurs when memory leaks: meshuser.js:792
If you delete "obj." in "obj.user._id", the memory leak occurs until it ends on the server.
It was also noticed: on the computers of the meshcentral admins, for some reason, several instances of MeshCentralRouter open when working in the browser and clicking on the RDP connection button. If the administrator notices that several windows are open and closes all but one, there will be a sharp decrease in RAM consumption on the server.
from meshcentral.
One thing to try is using the master branch, which includes a fix for the other issue and see if the memory leak happens still?
- Stop meshcentral.
npm install Ylianst/MeahCentral
- Start meshcentral
from meshcentral.
Related Issues (20)
- Add a security Layer for connection to a PC or Terminal
- No DNS published for info.meshcentral.com
- upable to connect web socket,click to reconnect HOT 1
- upable to connect web socket,click to reconnect HOT 2
- Mobile view blank screen HOT 8
- Show output for run command of multiple PCs in the same place
- Using Docker Compose leads into an upgrading process even if the internal-upgrade-flag is explicitely deactivated HOT 5
- All remote sessions are disconnected when a new user, previously unknown to MeshCentral, logs in to MeshCentral via LDAP HOT 6
- MeshCentral SSO via Microsoft Azure creates a new user at MeshCentral instead of using the existing, same user from the local LDAP AD with the authorization groups HOT 5
- How can i activte Only 2FA Login for Accounts HOT 1
- Let's Encrypt Certification Failed to update HOT 1
- Add some feature of frp. HOT 2
- Connection from mobile device no lock option for Windows machines. HOT 7
- 7 Day Power State color change HOT 1
- Installers Manual Revision HOT 1
- Cira connection shows on wrong devices HOT 3
- MeshCtrl.js does not output requested data. HOT 5
- Pulling Utilities-sidebar just works fine in one direction HOT 6
- If AllowAccountReset is set to false, the user can not change the password even if it expires so he can not log in anymore HOT 4
- Troubleshooting Slow File Uploads in Remote MeshCentral Server HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from meshcentral.