Server hanging or stalled
Hi guys, been out with the boat a few days, updated the SignalK server and plugins to the latest version before departure.
Everything seems to work fine, and a few hours later the server seems really slow, and after another hour it stops receiving data from my instruments.
Connected are, Autopilot, wind, log, depth sounder, GPS and AIS.
All except the GPS and the AIS are NMEA183 low speed, the GPS and the AIS are high speed.
Everything comes alive again after reboot, if I only restart SIgnalK everyting also comes alive but I think it is alive a bit shorter.
Not really a Linux guy so I do not really know how to look for e.g. a memory leak.
Some kind of leak seems like the logical explanation to me.
I could see that a node process was eating basically all CPU (using top).
(And yes I should have written down the Node version)
A Raspberry Pi, not the latest and greatest is driving this.
Any tips on what I should gather to aid mor skilled guys in an investigation?
Will not see the boat again until friday at the earliest so will not be able to supply data until then or later.
I live in central European timezone 🙂
11 Replies
Any issues in the log?
Turn off plugins one-by-one or turn off all and enable one-by-one & monitor server.
Hav not seen any issues in the log (that I found)
Agree with your suggestion about plugins, but also they have not changed or been reconfigured.
Some updated...
What version node? "node - V". I had high CPU receiving AMA with v20
What is AMA in your system?
lrwxrwxrwx 1 root root 7 Jul 4 17:14 ttyOP_ama2gps -> ttyAMA2
serial uart on AMA2 was definitely slowing things down, didn't check the others individually.
So a ”usb serial gps”, AMA can be anything…
Ah., OK. Just what openplotter auto names them, internal UART2 on pins 27 & 28.
Sorry for not beeing responsive, it's been pooring down rain and howling winds --> not been to the boat.
Will get you the info as soon as I get there.
I do believe that Node is 22.14 LTS, but will verify that.
Confirmed Node is 22.14
Also env is:
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.12.20+rpt-rpi-v8
Architecture: ARM64
HW is Raspberry Pi 3 Model B Rev 1.2
One thing that I have tried to get working is Victron SmartShunt over Bluetooth, bluetooth seems dodgy on the Pi 3 model B HW, any experience to share?
Saw some Bluetooth error messages in log window when on the boat, but the harvested logs does not contain that info.
I guess I need to change log level...
See about disabling all plugins. That should be your first step if there is nothing obvious. Also both ”not changed” and ”updated” per your earlier comment can not be both true..
Correct Teppo, what I meant was that I did not change any config, I only updated software and my first thought was memory leak (too many years with software that too often had these leaks) and if that was the case it might have been introduced by the software update.
The bluetooth attempts was before the latest upgrade.
One path to investigate might be to downgrade, any thoughts on that?
I realize this is a goose chase, Yesterday I disabled the bluetooth plugins and removed a no longer used plugin, that is it I think.
Did not have time to sit at the dock for hours and wait for the system to stall/hang if that happens without the Bluetooth plugins.
If I disable all plugins, would the next step be to enable them one by one and that way see if the problematic thing can be identified?
I have four USB/COM ports attached, they have not created problems before and they are still the same no config changes --> I'm assuming that these are not causing any problems.
@Tore Dahl regarding Smart Shunt on 3B
Don't recommend the GATT connection if that's what you're using-- thinking about deprecating the GATT option entirely for all Victron Battery Monitors. Best to get the encryption key from Victron Connect app.
How much ram does your 3B have? I've seen nodeJS get hungrier as it matures. I used to be able to run SK on a Pi Zero (512mb) but no more. 1 Gb doesn't seem to be enough to satisfy nodeJS these days.
As for the error messages through version 1.2.3 most were spurious, coming as a result of the plugin trying to communicate with an unconfigured device that wasn't in range.
Once the plugin is configured, you should set the Scan Interval to 0 and restart.