Rootless docker: permission denied - help me debug?

Hello, I have set up coder.com on dev and now I am getting ready for prod. I am struggling with setting up rootless docker containers though:

Linux debian-zaci 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

I followed https://docs.docker.com/engine/security/rootless/

I tried fiddling with main.tf adding

provider "docker" {
  host="unix:///run/user/1002/docker.sock"
}

provider "docker" {
  host="unix:///run/user/1002/docker.sock"
}

provider "docker" {
  host="unix:///run/user/1002/docker.sock"
}

provider "docker" {
  host="unix:///run/user/1002/docker.sock"
}

to the default docker container template, but I am still getting:

2024-02-05 16:45:33.337+01:00 Error: Error pinging Docker server: Got permission denied while trying to connect to the Docker daemon socket at unix:///run/user/1002/docker.sock: Get "http://%2Frun%2Fuser%2F1002%2Fdocker.sock/_ping": dial unix /run/user/1002/docker.sock: connect: permission denied
2024-02-05 16:45:33.337+01:00 on main.tf line 19, in provider "docker":
2024-02-05 16:45:33.337+01:00   19: provider "docker" {
2024-02-05 16:45:33.337+01:00
2024-02-05 16:45:33.337+01:00

2024-02-05 16:45:33.337+01:00 Error: Error pinging Docker server: Got permission denied while trying to connect to the Docker daemon socket at unix:///run/user/1002/docker.sock: Get "http://%2Frun%2Fuser%2F1002%2Fdocker.sock/_ping": dial unix /run/user/1002/docker.sock: connect: permission denied
2024-02-05 16:45:33.337+01:00 on main.tf line 19, in provider "docker":
2024-02-05 16:45:33.337+01:00   19: provider "docker" {
2024-02-05 16:45:33.337+01:00
2024-02-05 16:45:33.337+01:00

2024-02-05 16:45:33.337+01:00 Error: Error pinging Docker server: Got permission denied while trying to connect to the Docker daemon socket at unix:///run/user/1002/docker.sock: Get "http://%2Frun%2Fuser%2F1002%2Fdocker.sock/_ping": dial unix /run/user/1002/docker.sock: connect: permission denied
2024-02-05 16:45:33.337+01:00 on main.tf line 19, in provider "docker":
2024-02-05 16:45:33.337+01:00   19: provider "docker" {
2024-02-05 16:45:33.337+01:00
2024-02-05 16:45:33.337+01:00

2024-02-05 16:45:33.337+01:00 Error: Error pinging Docker server: Got permission denied while trying to connect to the Docker daemon socket at unix:///run/user/1002/docker.sock: Get "http://%2Frun%2Fuser%2F1002%2Fdocker.sock/_ping": dial unix /run/user/1002/docker.sock: connect: permission denied
2024-02-05 16:45:33.337+01:00 on main.tf line 19, in provider "docker":
2024-02-05 16:45:33.337+01:00   19: provider "docker" {
2024-02-05 16:45:33.337+01:00
2024-02-05 16:45:33.337+01:00

I must have missed a step or haven't got enough information.

Docker Documentation

Run the Docker daemon as a non-root user (Rootless mode)

Codercord•2/5/24, 5:05 PM

<#1204110515333496912>

**Category**

Help needed

**Product**

Coder OSS (v2)

**Platform**

Linux

Logs

Please post any relevant logs/error messages.

AAlbertLookalikeQualify Hello, I have set up coder.com on dev and now I am getting ready for prod. I am ...

Phorcys•2/5/24, 9:24 PM

you have to run your Coder instance with the docker group

Phorcys•2/5/24, 9:24 PM

are you running Coder within Docker ?

PPhorcys you have to run your Coder instance with the docker group

Phorcys•2/5/24, 9:25 PM

basically you have to run the codercoder process as coder:dockercoder:docker, where dockerdocker is the GID of the dockerdocker group

bamhm182 (BytePen)•2/7/24, 1:22 AM

Not quite. He's running as rootless. The user doesn't need to be in the docker group, but the Coder user does need to have permissions to access the socket.

Check the permissions on the socket and compare it against the id of whoever is running coder.

Bbamhm182 (BytePen)Not quite. He's running as rootless. The user doesn't need to be in the docker g...

Phorcys•2/7/24, 6:54 AM

oh alright, I thought that even with rootless you'd have a group dedicated to docker

Phorcys•2/7/24, 6:55 AM

well i mean there's nothing preventing you from doing it that way, no ?

Phorcys•2/7/24, 6:55 AM

because otherwise you'd need to give rwx permissions to everyone, which kind of sucks

Phorcys•2/7/24, 6:55 AM

or you'd use ACLs I guess

AlbertLookalikeQualifyOP•2/7/24, 7:54 AM

I am quietly observing, trying to make sense of it. Thank you for your input to both

AlbertLookalikeQualifyOP•2/7/24, 8:32 AM

Well, I made sure the coder and the user initiating coder cli commands both have access to the socket:

$ ls -l /run/user/1002/docker.sock
srw-rw---T 1 albert coder 0 Feb  6 09:15 /run/user/1002/docker.sock

$ ls -l /run/user/1002/docker.sock
srw-rw---T 1 albert coder 0 Feb  6 09:15 /run/user/1002/docker.sock

The issue is exactly the same upon

templates push

templates push

 Error pinging Docker server: Got permission denied while trying to connect to the Docker daemon socket at unix:///run/user/1002/docker.sock: Get "http://%2Frun%2Fuser%2F1002%2Fdocker.sock/_ping": dial unix /run/user/1002/docker.sock: connect: permission denied
2024-02-07 09:27:51.951+01:00 on main.tf line 19, in provider "docker":
2024-02-07 09:27:51.951+01:00   19: provider "docker" {
2024-02-07 09:27:51.951+01:00
2024-02-07 09:27:51.951+01:00

 Error pinging Docker server: Got permission denied while trying to connect to the Docker daemon socket at unix:///run/user/1002/docker.sock: Get "http://%2Frun%2Fuser%2F1002%2Fdocker.sock/_ping": dial unix /run/user/1002/docker.sock: connect: permission denied
2024-02-07 09:27:51.951+01:00 on main.tf line 19, in provider "docker":
2024-02-07 09:27:51.951+01:00   19: provider "docker" {
2024-02-07 09:27:51.951+01:00
2024-02-07 09:27:51.951+01:00

I am new to this so if I might be making some silly mistake.

AlbertLookalikeQualifyOP•2/7/24, 9:28 AM

I did try to give all users permission to use the socket, to test it and I am still getting the same error:

$ ls -l /run/user/1002/docker.sock
srw-rw-rw- 1 albert coder 0 Feb  7 09:57 /run/user/1002/docker.sock

$ ls -l /run/user/1002/docker.sock
srw-rw-rw- 1 albert coder 0 Feb  7 09:57 /run/user/1002/docker.sock

AlbertLookalikeQualifyOP•2/7/24, 9:34 AM

just to confirm, I haven't noticed any difference between:

coder : coder docker

coder : coder docker

and

coder : docker

coder : docker

The issue persisted in both cases.

AAlbertLookalikeQualify just to confirm, I haven't noticed any difference between: ``` coder : coder doc...

Phorcys•2/7/24, 10:22 AM

are you running Coder as a sustemd service ?

bamhm182 (BytePen)•2/7/24, 10:41 AM

Who is user 1002? Albert?

I suspect it may be an issue with the folder, as nobody else should be able to access that folder besides user 1002.

Here's how I would do it (assuming I chose to have Coder running directly on the box and not in a container):

Create a user codercoder with no special groups. Say he got uid 1010
Enable linger for that user (IIRC, it's just enable-linger coderenable-linger coder) There should now be a /run/user/1010/docker.sock
Create a systemd service to run Coder and ensure User=coderUser=coder and Group=coderGroup=coder are in there
If user albert needs to mess with the Coder containers, do so like sudo -u coder podman ps --allsudo -u coder podman ps --all

You might also need to add an entry for the codercoder user in /etc/{subuid,subgid}. Confirm that something like sudo -u coder run --rm -it alpine ashsudo -u coder run --rm -it alpine ash works, then confirm that you don't see that container with sudo docker ps --allsudo docker ps --all and that you do see it with sudo -u coder docker ps --allsudo -u coder docker ps --all. Coder shouldn't be in any groups other than their own.

PPhorcys well i mean there's nothing preventing you from doing it that way, no ?

bamhm182 (BytePen)•2/7/24, 10:43 AM

You could give whatever user the docker group, but then that user can choose to use /run/docker.sock/run/docker.sock (root) or /run/user/uid/docker.sock/run/user/uid/docker.sock (rootless). If you don't want Coder to be able to run root containers, you don't want to do that.

Bbamhm182 (BytePen)You could give whatever user the docker group, but then that user can choose to ...

Phorcys•2/7/24, 11:56 AM

oh no that theory was in the case where you're only running rootless, mb

PPhorcys are you running Coder as a sustemd service ?

AlbertLookalikeQualifyOP•2/7/24, 12:00 PM

yes, but as a a user albert.

Bbamhm182 (BytePen)Who is user 1002? Albert? I suspect it may be an issue with the folder, as nobo...

AlbertLookalikeQualifyOP•2/7/24, 12:02 PM

yes, 1002 is albert.

I also want to run coder on the box and not in the container. But in combination with rootless docker.

I will try this. Maybe I should really switch to podman, I was told it is better supported.

bamhm182 (BytePen)•2/7/24, 12:27 PM

I have been using Podman, and it's fine. There was a patch I needed to apply to the terraform provider to get it to work because the PR wasn't merged into the main provider, so I'm currently using my own provider, but if and when the main docker terraform provider gets the patch, I will be switching back to that.

bamhm182 (BytePen)•2/7/24, 12:32 PM

This is the patch:
https://github.com/kreuzwerker/terraform-provider-docker/pull/583

This is my fork:
https://github.com/bamhm182/terraform-provider-docker
This is my provider:
https://registry.terraform.io/providers/bamhm182/docker/latest

I wouldn't necessarily depend on it because I don't have any plans to support it outside of for personal reasons and you don't really have a reason to trust me. That said, it looks like the official provider hasn't been updated for quite some time, and I don't know if the original dev wants to keep supporting it, so...

GitHub

Ignore error message case in comparison for ignorable messages. by ...

In regards to #542 ignore the case of error messages when filtering ignorable error messages.

GitHub

GitHub - bamhm182/terraform-provider-docker: Terraform Docker provider

Terraform Docker provider. Contribute to bamhm182/terraform-provider-docker development by creating an account on GitHub.

Terraform Registry

bamhm182 (BytePen)•2/7/24, 12:33 PM

All of that said, I'm not totally convinced that it's a docker problem, and I feel like using Podman will probably have the same issues as docker rootless.

bamhm182 (BytePen)•2/7/24, 12:37 PM

This page gives an example of how you could disable the rootful docker socket. I would troubleshoot by disabling the main docker socket and confirming that you can still run docker containers. If you can, then your user should be using the rootless docker socket, and in that case, it is super weird that it says coder can't use it...

https://docs.docker.com/engine/security/rootless/#install

Docker Documentation

Run the Docker daemon as a non-root user (Rootless mode)

bamhm182 (BytePen)•2/7/24, 12:38 PM

Also, in that same session, it shows the DOCKER_HOST env variable. I would set that to your expected socket as well.

Rootless docker: permission denied - help me debug?

Similar Threads

Similar Threads

Similar Threads