Deion Docker Rootless fails to enable plugins on Docker v25.

git bisect is done: <a class="commit-link" data-hovercard-type="commit" data-hover

PR <a class="issue-link js-issue-link" data-error-text="Failed

Hmm, looks like the vieux/sshfs plugin <a href="https://github.com/vieux/docker-volume

Thanks for checking that, <a class="user-mention notranslate" data-hovercard-type="use

v25 Regression: Docker-Rootless Fails to Enable Plugins,about moby/moby

Comments (17)

christopher-besch commented on June 18, 2024 1

Out of curiosity, what is the output of sudo cat /proc/sys/kernel/unprivileged_userns_clone

chris@debian:~$ sudo cat /proc/sys/kernel/unprivileged_userns_clone
[sudo] password for chris:
1
chris@debian:~$

from moby.

christopher-besch commented on June 18, 2024 1

git bisect is done:
7d08d84 is the first bad commit
commit 7d08d84
Author: Akihiro Suda [email protected]
Date: Tue Jan 16 23:56:04 2024 +0900

dockerd-rootless.sh: set `rootlesskit --state-dir=DIR`

Now the state dir is set to `${XDG_RUNTIME_DIR}/dockerd-rootless`.

This is similar to `${XDG_RUNTIME_DIR}/containerd-rootless` used in nerdctl:
https://github.com/containerd/nerdctl/blob/v1.7.2/extras/rootless/containerd-rootless.sh#L35

Prior to this commit, the state dir was unset and a random dir under `/tmp` was used.
(e.g., `/tmp/rootlesskit1869901982`)

Signed-off-by: Akihiro Suda <[email protected]>

contrib/dockerd-rootless.sh | 3 +++
1 file changed, 3 insertions(+)

from moby.

AkihiroSuda commented on June 18, 2024 1

#47558

from moby.

corhere commented on June 18, 2024

Hmm, looks like the vieux/sshfs plugin is authored incorrectly, making it incompatible with rootless docker. Managed plugins should not be requesting a bind-mount of the (default path to the rootful) unmanaged plugin directory.

This is not a v25 regression; that plugin appears to never have been compatible with rootless mode. I can reproduce the issue on docker:23.0.6-dind-rootless.

/ $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 5.787GiB
 Name: ef1780590d4f
 ID: c81669c8-7d5e-48aa-9f21-6aa19e8aa773
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. Systemd is required to enable cgroups in rootless-mode.
/ $ docker plugin install vieux/sshfs
Plugin "vieux/sshfs" is requesting the following privileges:
 - network: [host]
 - mount: [/var/lib/docker/plugins/]
 - mount: []
 - device: [/dev/fuse]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from vieux/sshfs
Digest: sha256:1d3c3e42c12138da5ef7873b97f7f32cf99fb6edde75fa4f0bcf9ed277855811
52d435ada6a4: Complete
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker/plugins/" to rootfs at "/mnt/state": stat /var/lib/docker/plugins/: no such file or directory: unknown

There is nothing we can do.

from moby.

christopher-besch commented on June 18, 2024

Thanks for checking that, @corhere!

I tested it myself and vieux/sshfs does not show the problem, though ghcr.io/ibm/docker-logdna does. It is authored correctly. See: https://github.com/IBM/docker_logdna/blob/main/plugin/config.json

{
    "description": "logdna plugin",
    "documentation": "https://github.com/ibm/docker_logdna",
    "entrypoint": [
        "/docker_logdna"
    ],
    "network": {
        "type": "host"
    },
    "interface": {
        "types": [
            "docker.logdriver/1.0"
        ],
        "socket": "logdna.sock"
    }
}

Installing the plugin on Debian 12 with Docker v24.0.9 works. Installing it on Docker 25.0.1 does not (as tested in the issue description). This is the log for v24.0.9:

chris@debian:~$ docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
Plugin "ghcr.io/ibm/docker-logdna:1.0.1" is requesting the following privileges:
 - network: [host]
Do you grant the above permissions? [y/N] y
1.0.1: Pulling from ghcr.io/ibm/docker-logdna
Digest: sha256:b4b73b2e41dd9a23cc5279b0a9aca53e0e54f9084600de1bc53cb193fd585d26
cd173d625410: Complete
Installed plugin ghcr.io/ibm/docker-logdna:1.0.1

Here is the history for v24.0.9:

    1  ps --version
    2  iptables --version
    3  sudo apt install iptables
    4  iptables --version
    5  sudo iptables --version
    6  git --version
    7  sudo apt install git
    8  wget
    9  wget https://download.docker.com/linux/static/stable/x86_64/docker-24.0.9.tgz
   10  tar xzvf docker-24.0.9.tgz
   11  sudo cp docker/* /usr/bin/
   12  sudo dockerd &
   13  sudo docker run --rm hello-world
   14  top
   15  pgrep dockerd
   16  sudo reboot
   17  sudo apt install -y uidmap
   18  sudo apt install -y dbus-user-session
   19  sudo apt install -y fuse-overlayfs
   20  slirp4netns --version
   21  sudo apt install slirp4netns
   22  slirp4netns --version
   23  sudo apt install docker-ce-rootless-extras
   24  wget https://download.docker.com/linux/static/stable/x86_64/docker-rootless-extras-24.0.9.tgz
   25  ls
   26  tar xzvf docker-rootless-extras-24.0.9.tgz
   27  docker-rootless-extras/dockerd-rootless-setuptool.sh install
   28  sudo cp docker-rootless-extras/* /usr/bin/
   29  dockerd-rootless-setuptool.sh install
   30  systemctl --user start docker
   31  systemctl --user enable docker
   32  sudo loginctl enable-linger $(whoami)
   33  sudo reboot
   34  docker run --rm hello-world
   35  docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
   36  docker plugin install vieux/sshfs
   37  docker --version
   38  history

Installing vieux/sshfs indeed fails.

This is the docker version:

Client:
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.20.13
 Git commit:        2936816
 Built:             Thu Feb  1 00:47:46 2024
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.13
  Git commit:       fca702d
  Built:            Thu Feb  1 00:49:16 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit499189580
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

And this is docker info:

Client:
 Version:    24.0.9
 Context:    rootless
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-17-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 30.4GiB
 Name: debian
 ID: 5ebb78f4-3851-40e7-9897-19130167d40e
 Docker Root Dir: /home/chris/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

If there is anything I could test or do to help, just let me know.

Thanks
Chris

from moby.

christopher-besch commented on June 18, 2024

I stumbled through the moby contribution guides and tried a few more combinations. So far I haven't managed to reproduce the problem in the moby dev container. I also haven't managed to get enabling the plugin to work on the debian12 vm. These are a few versions I've tried:

Doesn't enable on Debian12 VM

Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

Doesn't enable on Debian12 VM

Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.7.12
  GitCommit:        71909c1814c544ac47ab91d2e8b84718e517bb99
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

Works on moby dev container

Client:
 Version:           25.0.0
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:01 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          dev
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab079ec13d17257f86fa92db8d7f24802f1
  Built:            Mon Feb  5 14:13:34 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.12
  GitCommit:        71909c1814c544ac47ab91d2e8b84718e517bb99
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /home/test-user/.docker/run/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

Works on moby dev container

Client:
 Version:           25.0.2.m
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629222
 Built:             Mon Feb  5 13:06:34 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          dev
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab079ec13d17257f86fa92db8d7f24802f1
  Built:            Mon Feb  5 13:22:21 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.12
  GitCommit:        71909c1814c544ac47ab91d2e8b84718e517bb99
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    vpnkit
  PortDriver:       builtin
  StateDir:         /home/test-user/.docker/run/dockerd-rootless
 vpnkit:
  Version:          7f0eff0dd99b576c5474de53b4454a157c642834

I tried different dockerd versions, different docker cli versions, different containerd versions and using vpnkit vs slirp4netns. But the problem stays.
I don't know if that is any help, but it's all I got for now.

from moby.

christopher-besch commented on June 18, 2024

Just to rule out any problems with apt or the Debian repos I tried installing v25.0.1 without apt and the plugin can't be installed. So this is not a Debian issue.

chris@debian:~$ docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
Plugin "ghcr.io/ibm/docker-logdna:1.0.1" is requesting the following privileges:
 - network: [host]
Do you grant the above permissions? [y/N] y
1.0.1: Pulling from ghcr.io/ibm/docker-logdna
Digest: sha256:b4b73b2e41dd9a23cc5279b0a9aca53e0e54f9084600de1bc53cb193fd585d26
cd173d625410: Complete
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/etc/hosts" to rootfs at "/etc/hosts": mount /etc/hosts:/etc/hosts (via /proc/self/fd/6), flags: 0x5021: operation not permitted: unknown

History:

    1  ls
    2  history
    3  sudo apt install iptables
    4  sudo apt install git
    5  wget https://download.docker.com/linux/static/stable/x86_64/docker-25.0.1.tgz
    6  wget https://download.docker.com/linux/static/stable/x86_64/docker-rootless-extras-25.0.1.tgz
    7  ls
    8  tar xf docker-25.0.1.tgz
    9  tar xf docker-rootless-extras-25.0.1.tgz
   10  sudo cp docker/* /usr/bin/
   11  sudo docker-rootless-extras/* /usr/bin
   12  sudo cp docker-rootless-extras/* /usr/bin
   13  sudo dockerd &
   14  sudo docker run --rm hello-world
   15  sudo apt install -y uidmap
   16  sudo apt install -y dbus-user-session fuse-overlayfs
   17  sudo apt install slirp4netns -y
   18  dockerd-rootless-setuptool.sh install
   19  docker run --rm hello-world
   20  docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
   21  history

from moby.

corhere commented on June 18, 2024

mount /etc/hosts:/etc/hosts (via /proc/self/fd/6), flags: 0x5021: operation not permitted

Interesting... What happens if you do docker run --rm --net=host hello-world on the affected install?

from moby.

christopher-besch commented on June 18, 2024

Interesting... What happens if you do docker run --rm --net=host hello-world on the affected install?

The normal hello-world container starts as expected.

I'm currently in the process of doing a git bisect by compiling docker in the dev container and then copying the binary to a separate vm. That actually works and this is my progress:

good: v24.0.9 fca702d
bad: v25.0.1 71fa3ab
good: 8d9a40a

from moby.

cpuguy83 commented on June 18, 2024

Out of curiosity, what is the output of sudo cat /proc/sys/kernel/unprivileged_userns_clone

from moby.

corhere commented on June 18, 2024

The daemon managed-plugin code hasn't really changed between v24 and v25. I did notice a couple differences between the docker version output of the working v24.0.9 and broken v25 installs which might be pertinent.

Version	runc	rootlesskit
24.0.9	1.1.12	1.1.1
25.0.1	1.1.11	2.0.0

The only difference between runc 1.1.11 and 1.1.12 is the CVE fix. It's possible, albeit unlikely, that the change also unintentionally fixed an issue with bind-mounting /etc/hosts in rootless. Should be easy enough to rule out by trying to install the plugin on rootless v25 with runc 1.1.12 installed.

I have a feeling that the upgrade of rootlesskit to v2.0.0 is related.

(cc @AkihiroSuda)

from moby.

christopher-besch commented on June 18, 2024

These are the tests I ran:

good fca702d
bad 71fa3ab
good 8d9a40a
good f10e182
good f7ddfdb
good afe2819
good b7042cb
good 60ed73b
bad d25aa32
bad 66325f7
good 1fd6829
bad 83de55b
bad 7d08d84
good ae9fbcb

from moby.

christopher-besch commented on June 18, 2024

Should be easy enough to rule out by trying to install the plugin on rootless v25 with runc 1.1.12 installed.

I just ruled that out. The problem persists with these versions:

Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:46 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /run/user/1000/dockerd-rootless
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

from moby.

AkihiroSuda commented on June 18, 2024

ghcr.io/ibm/docker-logdna

What is this plugin? Where is the source? Who built it?

from moby.

AkihiroSuda commented on June 18, 2024

Does it work if you downgrade RootlessKit binary to v1.1 ?

from moby.

corhere commented on June 18, 2024

What is this plugin? Where is the source? Who built it?

All that information has been provided in #47248 (comment)

Though I don't think it matters. The error logs clearly show that runc can't bind-mount the host's /etc/hosts file in a rootless environment. Managed plugins with "network": {"type": "host"} happen to be affected but unless there's some special-casing for the moby containerd namespace somewhere it seems unlikely that only plugins are affected, let alone a specific plugin.

from moby.

mahiso commented on June 18, 2024

Does it work if you downgrade RootlessKit binary to v1.1 ?

Can confirm, that it works with Rootlesskit v1.1.1

 Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:14:26 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:26 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /tmp/rootlesskit2257322869
 slirp4netns:
  Version:          1.0.1
  GitCommit:        6a7b16babc95b6a3056b33fb45b74a6f62262dd4

from moby.

v25 Regression: Docker-Rootless Fails to Enable Plugins about moby HOT 17 CLOSED

Comments (17)

Doesn't enable on Debian12 VM

Doesn't enable on Debian12 VM

Works on moby dev container

Works on moby dev container

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent