Starting containers fails with EACCESS to bash/sh for select users about enroot HOT 13 CLOSED

Are you sharing your containers with other users?
It looks like they don't have permissions to the sh binary inside this specific container (not the host one)

The home directory is on a network file system

Is selinux disabled for all users?
Maybe you have squashing enabled on your network file system which could alter permissions.

from enroot.

Are you sharing your containers with other users?

No, they are tied to the individual users. (using ~/.local/enroot or /run/user/###/enroot and stuff)

It looks like they don't have permissions to the sh binary inside this specific container (not the host one)

the unpacked container stuffies in the local directories seem to have ok permissions; how can I debug that?

The home directory is on a network file system
is selinux disabled for all users?

It is globally disabled

Maybe you have squashing enabled on your network file system which could alter permissions.

that's an interesting Idea. GPFS does funny things with permissions, but I configured it such that it respects chmod.

However, It works for my normal user but not for others. My user has no priviliges (except sudo, but that does not seem to play a role)

from enroot.

Is GPFS mounted directly or exported through say NFS?
It could be ACLs or something like that too.

You can try executing said binary outside the container to make sure it's enroot related:
~/.local/enroot/share/<container>/bin/sh
If your users can't run it from outside, the problem is probably not coming from enroot

My user has no priviliges (except sudo, but that does not seem to play a role)

Just to make sure, there are not using sudo to start the container right?

from enroot.

Is GPFS mounted directly or exported through say NFS?
It could be ACLs or something like that too.

Directly. I'll look. ... Oh dear:

$ mmgetacl /$GPFS/home/$ME/.local/share/enroot/cuda/bin/bash
#NFSv4 ACL
#owner:$ME
#group:$GROUP
special:owner@:rw--:allow:Inherited
 (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (X)DELETE    (X)DELETE_CHILD (X)CHOWN        (-)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED

special:group@:r---:allow:Inherited
 (X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (-)DELETE    (-)DELETE_CHILD (-)CHOWN        (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED

user:$ME:rwx-:allow:Inherited
 (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (X)DELETE    (X)DELETE_CHILD (X)CHOWN        (X)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED

$ mmgetacl /$GPFS/home/$USER/.local/share/enroot/cuda-test/bin/bash
#NFSv4 ACL
#owner:$USER
#group:$GROUP
special:owner@:rw--:allow:Inherited
 (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (X)DELETE    (X)DELETE_CHILD (X)CHOWN        (-)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED

special:group@:r---:allow:Inherited
 (X)READ/LIST (-)WRITE/CREATE (-)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (-)DELETE    (-)DELETE_CHILD (-)CHOWN        (-)EXEC/SEARCH (-)WRITE_ACL (-)WRITE_ATTR (-)WRITE_NAMED

user:$USER:rw--:allow:Inherited
 (X)READ/LIST (X)WRITE/CREATE (X)APPEND/MKDIR (X)SYNCHRONIZE (X)READ_ACL  (X)READ_ATTR  (X)READ_NAMED
 (X)DELETE    (X)DELETE_CHILD (X)CHOWN        (-)EXEC/SEARCH (-)WRITE_ACL (X)WRITE_ATTR (X)WRITE_NAMED

This is NFSv4-Acl format, bu the last blocks each say that: my files have execute by me, theirs dont.

from enroot.

Ok, its an ACL problem.

My home directory is set to give all files r-x bit for me. but this is not visible in ls.

BUT: If I chmod +x bash the bash in $USERs enroot dir, the x bit gets set and the following happens:

[$USER ~]$ ls -al .local/share/enroot/cuda-test/bin/bash
-rwxr-x--x 1 $USER GROUP 1563432 Jun 23  2020 .local/share/enroot/cuda-test/bin/bash
[$USER ~]$ .local/share/enroot/cuda-test/bin/bash
[$USER ~]$ #inside bash
[$USER ~]$ exit
[$USER ~]$ enroot start cuda-test bash
enroot-switchroot: failed to execute: /bin/bash: Permission denied

note that this time it says /bin/bash and not /bin/sh as in the first post:

[$USER ~]$ enroot start cuda-test sh
enroot-switchroot: failed to execute: /bin/bash: Permission denied

How does unsquahsfs handle permissions?

Do you want this open for future reference or shall I close?

from enroot.

Squashfs doesn't support ACL iirc, but this shouldn't matter in this instance.
Unsquashing the image will apply the same permissions as found in the image, which should indirectly apply the correct ACLs.
I'm not sure why this would work for you but not the other users, maybe you have different umasks?

There could be a GPFS issue lurking too, could be similar to this if not supported correctly.

from enroot.

Squashfs doesn't support ACL iirc, but this shouldn't matter in this instance.
Unsquashing the image will apply the same permissions as found in the image, which should indirectly apply the correct ACLs.

That's what I hoped.
It's curious, tho, I thought the effect of unsquashing was, permission-wise equivalent to create/chmod, but it is not, apparrently.

I'm not sure why this would work for you but not the other users, maybe you have different umasks?

No, different ACL inheritance rules on parent dirs. mine is older and has different rules.
I might “rebase” the users' enroot directories…

I'll have a look how things are after GFPS ACL changes and report back.

There could be a GPFS issue lurking too, could be similar to this if not supported correctly.

That would be .. ungood.
I'll watch out for that. (That said, GPFS acls are not the same as posix acls and are not accessed in the same way)

from enroot.

Ok, it is an ACL problem.
From my POV this can be closed unless you want it open for documentation purposes.

from enroot.

I think we can close

from enroot.

Thanks for looking into it!

from enroot.

Thanks to you for your analysis :)

from enroot.

This is the only result I could find for this issue, so I'm posting the resolution here. I needed to change the directory for storing enroot data. Apparently the drive was not mounted with "exec" and was causing the execution to fail. Remounting with this:

sudo mount -o remount,exec /mnt/localdisk

Resolved the issue for me.

from enroot.

Yes, this is a not atypical reason for this error

from enroot.