backup/restore workspaces in k8s
We have deployed Coder with Kubernetes and we are trying to figure out how others may be backing up/restoring a workspace's persistent volume. We use rook-ceph for our storage solution and can create rbd volumeshapshots for backups. However, we are finding it difficult to restore them in an event workspace's pvc is deleted as workspaces are spun up as Pods and after creation you can't tie it to a different pvc. Any ideas are appreciated.
12 Replies
<#1206653225038774402>
Category
Help needed
Product
Coder OSS (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
@Bjorn Robertsson may have some thoughts here.
Coder doesn't allow binding a new workspace to an existing or restored PVC.@asmii I think this not a Coder limitaion but a k8s one. If its possible in k89s there should be a way to do it in a Coder template.
@Atif , the problem is at the moment when I restart the workspace, a new pvc is created and it overrides the restored one
This is strange. Can you share your template? I've been using Coder in k8s for about a year now with no issues. As long as a workspace isn't deleted, the PVC is always there and thus, workspaces can be shut down, restarted and also upgraded at any time and all the current work stored in the volume is there as it should be.
@Scott , I just identified the issue. I added the lifecycle: ignore_changes parameter to the PVC Terraform resource in the template, and it's working fine now.
Thanks!
That's great.
So a backup restore with Valero works as expected.
Interested in contributing a basic minimal tutorial on how other members can do this?
(@demba)
this is strange. this should not be necessary. we are using PVC for over 2 years with coder
I 2nd the idea of writing up a document/tutorial for restoring Coder. Restoring to a particular state with the PVC as backed up would be a desired outcome. Some of the non-permanence of PVC's might relate to count/for_each behaviour (or lack thereof) and usually if you restore point-in-time PVC and Database, the PVC/PV state and labels should match the Coder Workspace's.
Since Velero provides namespace type backup and could limit the backup/restore to a few components, the terraform state, PVC vs Database might be skewed also.
I will prepare a complete guide at the beginning of next week, on how to restore volumes on Coder-Workspaces . It will include all the key steps to help others easily reproduce the process.
This is great. Thank you ❤️
thanks a lot :-)