Single Coder server instance crashing with 20ish or so workspaces

We have been giving Coder a trial and have seen really great improvements to DX. But we did run into an issue with 20 or so workspaces running seemingly crashed the server instance. @Théo asked to upload some info we found regarding the issue. From working with our Platform Engineer, here's what we can share:
Workspaces would be temporarily accessible when the Coder Embedded Relay (CER) was green, but if CER went red and then back to green shortly after, all workspaces became inaccessible again until each of the individual workspaces were restarted. This became a cycle about halfway through debugging; CER green, I restart my WS and access it, CER goes red, CER goes green, I restart my WS to access it...
Unfortunately those logs would have hit the lifecycle policy on the 4th.
During the incident, the Coder Embedded Relay / DERP portions of Coder's health page were rapidly flipping between green and red. See screenshot.
No description
7 Replies
Codercord
Codercord3w ago
<#1307071966766694461>
Category
Help needed
Product
Coder OSS (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
Phorcys
Phorcys3w ago
@Fattyacid i see has this happened multiple times?
Fattyacid
FattyacidOP3w ago
I can check but believe it was isolated to that specific event
Phorcys
Phorcys3w ago
okay, this really shouldn't happen but without logs or a way to reproduce i'm afraid we can't really identify the root cause
Fattyacid
FattyacidOP3w ago
Ok. We'll continue to keep an eye out. It's good to hear that we should be fine with our numbers. If it pops up again, we'll be sure to grab the logs.
Phorcys
Phorcys3w ago
sounds good! closing this issue :)
Codercord
Codercord3w ago
@Phorcys closed the thread.
Want results from more Discord servers?
Add your server