`docker exec` seems to be broken on UmbrelOS 1.0 (RPi) about umbrel HOT 3 CLOSED

I can confirm that this is solved in UmbrelOS 1.1 👍

Upgrading to rugpi v0.6.5 did the job.
Seems like it was fixed in rugpi, by using pivot_root instead of chroot (like docker does).

Thanks @lukechilds and thanks @koehlma 😄

from umbrel.

Adding details of my diagnosis, for further information and to help debugging:

Setup:

# start a container and exec a new process in it
sudo docker run -d --rm --name docker_test busybox sleep 1000
sudo docker exec -d docker_test sleep 1000

# find out the pid of the main process and the subprocess
main_pid=$(sudo docker inspect docker_test | jq -r '.[] | .State.Pid')
main_ppid=$(sudo cat /proc/${main_pid}/status | grep PPid | awk '{print $2}')
sub_pid=$(pgrep -P $main_ppid  | grep -v ${main_pid})

Now, look at the root mounts of the processes:

$ sudo cat /proc/${main_pid}/mounts | head -1 
overlay / overlay rw,relatime,lowerdir=/var/lib/docker/ ... ... /work 0 0

( truncated long output... Root is mounted to the container's overlay, with all the layers listed)

$ sudo cat /proc/${sub_pid}/mounts | head -1
/dev/root / ext4 ro,relatime 0 0

For the subprocess, root is mounted as ext4 (a physical disk partition?). This happens for me only on umbrel 1.0. Repeating the same commands on other linux distros, I get the same result as I got above for the main pain process above (root mounted to the container's overlay).

Note that the container's layer fs IS in fact mounted, but not at the root:

$ layerhash=$(sudo cat /proc/${main_pid}/mounts | head -1 | grep -o 'upperdir=.*/diff' | awk -F/ '{print $(NF-1)}')
$ sudo cat /proc/${lower_pid}/mounts | grep $layerhash
overlay /run/rugpi/mounts/data/overlay/root overlay rw,relatime,lowerdir=/ ... ... /work 0 0

Looks like the same overlay that is mounted as root in the main process is now mounted at /run/rugpi/mounts/data/overlay/root, but not chrooted.

This fact opens the way for a workaround that I can use until this is solved (I posted it in the community website as a way to use the LND cli on my Rpi 4 Umbrel 1.0 :
https://community.umbrel.com/t/why-is-docker-compose-not-working-anymore-with-umbrel-1-0/15756/32 )

from umbrel.

Heya @AmitAronovitch thanks so much for taking the time to dig into this and post your findings, really appreciate it!

I can confirm the issue doesn't occur on our amd64 builds for Umbrel Home, it looks like it's to do with the build system we use for Raspberry Pi builds and the way it mounts the filesystem as you pointed out.

Working on a fix for the next umbrelOS release, thanks again! 💜

from umbrel.