catatonit keeps a mount directory open, preventing unmounting about catatonit HOT 6 OPEN

Does this happen with catatonit v0.1.6 and later? We added can de to close all fds >=3 a while ago (see #14 and #12) and so it seems unlikely that we would have any fds open as a result. Pre-0.1.6 it was possible for us to keep files open that were passed to the container process, which looks similar to what you're describing.

(FWIW, catatonit is a very simple program, it only opens /dev/tty and possibly /proc/self/fd and closes them before doing a signal passing loop.)

Also, if no containers are running there should be no catatonit process -- catatonit is a pid1 for containers.

from catatonit.

$ catatonit --version
tini version 0.1.7_catatonit

It appears that podman is using the -P option of catatonit as per user.slice -> podman-pause-0db74da4.scope:

$ systemctl status
● raspberrypi3a
    State: running
    Units: 305 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
  systemd: 252.17-1~deb12u1
   CGroup: /
           ├─init.scope
           │ └─1 /sbin/init
           ├─system.slice
             snip
           └─user.slice
             └─user-1000.slice
               ├─session-1.scope
                 snip
               └─[email protected]
                 ├─init.scope
                 │ ├─714 /lib/systemd/systemd --user
                 │ └─715 "(sd-pam)"
                 ├─session.slice
                 │ └─dbus.service
                 │   └─841 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
                 └─user.slice
                   └─podman-pause-0db74da4.scope
                     └─829 catatonit -P

Catatonit has several objects reported by lsof, where /mnt/wd is the mount that catatonit keeps busy even if no other processes are using it. And that mount is created after the podman pause service is running, suggesting to me that catatonit responds to some notification about the filesystem, and maybe sets its current working directory there...?

$ lsof | fgrep catatonit
COMMAND     PID   TID TASKCMD               USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME
catatonit   829                               pi  cwd       DIR               0,40        0   18989662 /mnt/wd
catatonit   829                               pi  rtd       DIR              179,2     4096          2 /
catatonit   829                               pi  txt       REG              179,2   600176      34185 /usr/bin/catatonit
catatonit   829                               pi    0u      CHR                1,3      0t0          5 /dev/null
catatonit   829                               pi    1u      CHR                1,3      0t0          5 /dev/null
catatonit   829                               pi    2u      CHR                1,3      0t0          5 /dev/null
catatonit   829                               pi    3u  a_inode               0,13        0       1060 [signalfd]

from catatonit.

I said:

... where /mnt/wd is the mount that catatonit keeps busy even if no other processes are using it. And that mount is created after the podman pause service is running...

I'm no longer sure about that claim regarding the order of things.

What I have definitely just seen is that if the podman catatonit -P starts after the cifs mount is already in place, the catatonit process has the cwd entry for the mounted directory. That is, I don't have evidence that catatonit is actively responding to filesystem changes -- I do have evidence that when runs it does something special with the existing cifs mount...

from catatonit.

Ah, the issue is that catatonit is being run inside the mount (by podman presumably). I guess we could change directory to / but if the process is being run with the mount as the root of the mount namespace there's not much we can do. If anything this seems like a podman issue.

What does ls -ld /proc/$(pgrep catatonit)/root output?

from catatonit.

I'll have to check if podman is being used from the mount directory. That will require a reboot and there's a big build ongoing, so it will have to wait until tomorrow...

$ ls -ld /proc/$(pgrep catatonit)/root
lrwxrwxrwx 1 pi pi 0 Nov  4 02:09 /proc/1375/root -> /

from catatonit.

I believe your suspicions are correct. It appears that the first time (in the user-specific systemd slice) that a user runs podman it creates a "hidden" container running catatonit -P with cwd being the directory from which podman was run. Subsequent use of podman from other directories (or other simultaneous logins by the user) has no effect on the hidden container. This container does not show up in podman ps, and it persists until the user-specific slice terminates.

I will file an issue with the podman project. It seems to me that the hidden container should have its cwd in the user's home directory, or some suitable /var area.

from catatonit.