Single Coder server instance crashing with 20ish or so workspaces
We have been giving Coder a trial and have seen really great improvements to DX. But we did run into an issue with 20 or so workspaces running seemingly crashed the server instance.
@Théo asked to upload some info we found regarding the issue.
From working with our Platform Engineer, here's what we can share:
@Théo asked to upload some info we found regarding the issue.
From working with our Platform Engineer, here's what we can share:
Workspaces would be temporarily accessible when the Coder Embedded Relay (CER) was green, but if CER went red and then back to green shortly after, all workspaces became inaccessible again until each of the individual workspaces were restarted. This became a cycle about halfway through debugging; CER green, I restart my WS and access it, CER goes red, CER goes green, I restart my WS to access it...
Unfortunately those logs would have hit the lifecycle policy on the 4th.
During the incident, the Coder Embedded Relay / DERP portions of Coder's health page were rapidly flipping between green and red. See screenshot.
