Coder.com•2mo ago

🐳 Workspace DIND — “directory not empty” error when stopping workspace automatically

Hey everyone, I’m running a self-hosted Coder (latest version) setup, with a Docker-in-Docker (DIND) workspace template using sysbox-runc. I’ve configured automatic start/stop for my workspaces. When the workspace stops, if there are still containers running inside Docker, I get this error and the stop fails:

Error deleting container: ... directory not empty, driver "overlay2": unlinkat /var/lib/docker/overlay2/...

It looks like the workspace container can’t fully unmount overlay2 layers because dockerd is killed before it has time to clean up. Here’s my setup: - Terraform workspace template (using the Docker provider) - Workspace container launched with sysbox-runc - Docker-in-Docker started in startup_script via sudo dockerd & Question: 👉 What’s the recommended way (in a Docker-based, self-hosted setup) to gracefully stop dockerd and its inner containers when Coder stops or autostops the workspace? Ideally I’d like a built-in way (like a “preStop” hook or a shutdown script) to ensure DIND cleans up properly and avoid this overlay2 error. Any best practices or official guidance for handling Docker-in-Docker shutdown inside Coder workspaces would be much appreciated 🙏

5 Replies

Codercord•2mo ago

<#1432628344779964416>

Category

Bug report

Product

Coder (v2)

Platform

Linux

Logs

Please post any relevant logs/error messages.

MonnomcjoOP•4w ago

Nobody?

noel•3w ago

This is what I use. Ignore lifecycle is key:

resource "docker_volume" "workspace" {
  name = "coder-${data.coder_workspace.me.id}"
  lifecycle {
    ignore_changes = all
  }
}

# Docker image resource - use sha256 digest directly in name to force specific version
resource "docker_image" "workspace" {
  name = "ghcr.io/sonicaapp/coder:latest"
  
  # Always pull the latest by using timestamp that changes every apply
  pull_triggers = [timestamp()]
  
  # Never keep locally - always pull fresh
  keep_locally = false
  
  # Force remove any old versions
  force_remove = true
}

# Workspace container with sysbox runtime
resource "docker_container" "workspace" {
  count    = data.coder_workspace.me.start_count
  # Use latest digest to ensure we get the newest version
  image    = docker_image.workspace.repo_digest
  name     = "coder-${data.coder_workspace_owner.me.name}-${lower(data.coder_workspace.me.name)}"
  hostname = data.coder_workspace.me.name
  
  # This is the key - use sysbox runtime for proper Docker-in-Docker
  runtime = "sysbox-runc"
  
  # Sysbox needs more time to clean up nested overlay2 filesystems
  stop_timeout = 300  # 5 minutes for complex cleanup
  
  # Use SIGKILL after timeout - sysbox containers need forceful termination
  stop_signal = "SIGKILL"
  
  env = [
    "CODER_AGENT_TOKEN=${coder_agent.main.token}",
    "DOPPLER_TOKEN=${var.doppler_token}",
    "GITHUB_TOKEN=${var.github_token}",
    "CLAUDE_CODE_OAUTH_TOKEN=${var.claude_code_oauth_token}",
    "DOCKERHUB_USERNAME=${var.dockerhub_username}",
    "DOCKERHUB_TOKEN=${var.dockerhub_token}",
    "AWS_ACCESS_KEY_ID=${var.aws_access_key_id}",
    "AWS_SECRET_ACCESS_KEY=${var.aws_secret_access_key}"
  ]
  
  # Mount workspace volume
  volumes {
    volume_name    = docker_volume.workspace.name
    container_path = "/home/coder"
  }
  
  # Run the Coder agent - curl is already in our custom image
  command = ["sh", "-c", "${coder_agent.main.init_script}"]
  
  # Force removal settings for sysbox containers
  remove_volumes = true
  must_run       = false
  
  # Attach and remove policies for sysbox
  attach = false
  logs   = false
  rm     = true  # Automatically remove container when it exits
  
  lifecycle {
    ignore_changes = all
    
    # Accept that sysbox containers may leave overlay2 remnants
    # These are cleaned up by Docker's garbage collection
  }
}

resource "docker_volume" "workspace" {
  name = "coder-${data.coder_workspace.me.id}"
  lifecycle {
    ignore_changes = all
  }
}

# Docker image resource - use sha256 digest directly in name to force specific version
resource "docker_image" "workspace" {
  name = "ghcr.io/sonicaapp/coder:latest"
  
  # Always pull the latest by using timestamp that changes every apply
  pull_triggers = [timestamp()]
  
  # Never keep locally - always pull fresh
  keep_locally = false
  
  # Force remove any old versions
  force_remove = true
}

# Workspace container with sysbox runtime
resource "docker_container" "workspace" {
  count    = data.coder_workspace.me.start_count
  # Use latest digest to ensure we get the newest version
  image    = docker_image.workspace.repo_digest
  name     = "coder-${data.coder_workspace_owner.me.name}-${lower(data.coder_workspace.me.name)}"
  hostname = data.coder_workspace.me.name
  
  # This is the key - use sysbox runtime for proper Docker-in-Docker
  runtime = "sysbox-runc"
  
  # Sysbox needs more time to clean up nested overlay2 filesystems
  stop_timeout = 300  # 5 minutes for complex cleanup
  
  # Use SIGKILL after timeout - sysbox containers need forceful termination
  stop_signal = "SIGKILL"
  
  env = [
    "CODER_AGENT_TOKEN=${coder_agent.main.token}",
    "DOPPLER_TOKEN=${var.doppler_token}",
    "GITHUB_TOKEN=${var.github_token}",
    "CLAUDE_CODE_OAUTH_TOKEN=${var.claude_code_oauth_token}",
    "DOCKERHUB_USERNAME=${var.dockerhub_username}",
    "DOCKERHUB_TOKEN=${var.dockerhub_token}",
    "AWS_ACCESS_KEY_ID=${var.aws_access_key_id}",
    "AWS_SECRET_ACCESS_KEY=${var.aws_secret_access_key}"
  ]
  
  # Mount workspace volume
  volumes {
    volume_name    = docker_volume.workspace.name
    container_path = "/home/coder"
  }
  
  # Run the Coder agent - curl is already in our custom image
  command = ["sh", "-c", "${coder_agent.main.init_script}"]
  
  # Force removal settings for sysbox containers
  remove_volumes = true
  must_run       = false
  
  # Attach and remove policies for sysbox
  attach = false
  logs   = false
  rm     = true  # Automatically remove container when it exits
  
  lifecycle {
    ignore_changes = all
    
    # Accept that sysbox containers may leave overlay2 remnants
    # These are cleaned up by Docker's garbage collection
  }
}

MonnomcjoOP•3w ago

Perfect! thx! it's ok now.

Gaming

Programming

🐳 Workspace DIND — “directory not empty” error when stopping workspace automatically

Did you find this page helpful?