Code Monkey home page Code Monkey logo

fuse-overlayfs's Introduction

fuse-overlayfs

An implementation of overlay+shiftfs in FUSE for rootless containers.

Usage:

$ fuse-overlayfs -o lowerdir=lowerdir/a:lowerdir/b,upperdir=up,workdir=workdir merged

Specify a different UID/GID mapping:

$ fuse-overlayfs -o uidmapping=0:10:100:100:10000:2000,gidmapping=0:10:100:100:10000:2000,lowerdir=lowerdir/a:lowerdir/b,upperdir=up,workdir=workdir merged

Requirements:

If you are not using the static build as explained in the next chapter, your system needs libfuse > v3.2.1.

  • On Fedora: dnf install fuse3-devel
  • On Ubuntu > v19.04: apt install libfuse3-dev

Also, please note that, when using fuse-overlayfs from a user namespace (for example, when using rootless podman) a Linux Kernel > v4.18.0 is required.

Static Build:

This project provides a convenient way to automatically perform a static build using a container. The result is a self-contained binary without dependencies, that can be copied across hosts.

  1. Install buildah as explained here;

  2. Both the build and deploy host require the special device /dev/fuse; there are a few ways to obtain it:

  • install fuse2 or fuse3 using the package manager of your choice (dnf, apt, pacman, etc): the install script will usually create the device automatically; or
  • manually create the device with the command mknod /dev/fuse -m 0666 c 10 229 (see this code)
  1. Clone this repository, and switch to the top-level folder containing the file "Containerfile.static.ubuntu";

  2. Launch the build with the command (note the single dot . at the end):

buildah bud -v $PWD:/build/fuse-overlayfs -t fuse-overlayfs -f ./Containerfile.static.ubuntu .
  1. Copy the resulting binary to your host:
sudo cp fuse-overlayfs /usr/bin/

fuse-overlayfs's People

Contributors

3xx0 avatar akihirosuda avatar asafkahlon avatar cptpcrd avatar dhedberg avatar eamsden avatar ffontaine avatar flouthoc avatar frecks avatar giuseppe avatar gqgs avatar leorochael avatar ltalirz avatar mgoltzsche avatar mskarbek avatar nalind avatar natefoo avatar nwf avatar plopresti avatar rhatdan avatar saschagrunert avatar sectorsize512 avatar sgotand avatar sipsma avatar svattic avatar thiell avatar tomsweeneyredhat avatar tuupertunut avatar tylarb avatar vbatts avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fuse-overlayfs's Issues

Errors when building an image

Description

Does anyone have an idea why I'm getting errors like call to execv failed (No such file or directory) and sed: command not found?

This is the full log: https://gist.github.com/oblitum/163d9c5666a0b4106c421e90e55b778e

It doesn't happen with docker, only with podman, which I run rootless.

Original issue: https://github.com/archlinux/archlinux-docker/issues/19

Steps to reproduce the issue:

  1. podman build -t archlinux/oblitum .
  2. FROM archlinux/base:latest
    
    RUN pacman -Syu --noconfirm && pacman -S --noconfirm \
        aria2 aspell aspell-en aspell-pt atool autofs base-devel bat bind-tools \
        boost cabextract clamav clang cryptsetup devtools dhcpcd dictd dosfstools \
        dvd+rw-tools easy-rsa ethtool exfat-utils extra-cmake-modules fd ffmpeg \
        fzf gcc-fortran gdb geoip geoip-database-extra gifsicle git go gtop htop \
        hub hugo hyperfine iftop imagemagick inetutils iotop iperf3 iproute2 \
        iputils iw jfsutils jdk-openjdk jq languagetool lastpass-cli libqalculate \
        logrotate lsd lsof luarocks man-db man-pages mediainfo moreutils msmtp \
        mtools ncdu neomutt neovim netctl newsboat nfs-utils ninja nmap nodejs \
        node-gyp ntfs-3g ntp openmp openmpi openssh p7zip parallel parted \
        pax-utils peg pigz postgresql powertop proxychains-ng psmisc pygmentize \
        python python-pip python-pynvim python-send2trash python-setuptools \
        python-virtualenv python2 python2-pip python2-setuptools \
        python2-virtualenv r radare2 rclone reflector reiserfsprogs ripgrep rtv \
        ruby rubygems rustup sdcv setconf skim smartmontools strace syncthing \
        termtosvg testdisk texlive-langchinese texlive-langcyrillic \
        texlive-langextra texlive-langgreek texlive-langjapanese \
        texlive-langkorean texlive-most tig tldr tmux tokei tor traceroute \
        translate-shell transmission-cli trash-cli typescript ufw unrar usbutils \
        valgrind vi vint w3m watchexec weechat wget whois woff2 xclip \
        xdg-user-dirs xdg-utils xfsprogs yarn youtube-dl zsh \
     && pacman -Sc --noconfirm \
     && rm -rf /tmp/* \
     && ln -sf /usr/share/zoneinfo/America/Sao_Paulo /etc/localtime
    
    RUN groupadd -g 1000 francisco \
     && useradd -g francisco -G wheel -u 1000 -ms /bin/zsh francisco \
     && sed -i -e 's/^# %wheel ALL=(ALL) NOPASSWD: ALL$/%wheel ALL=(ALL) NOPASSWD: ALL/' /etc/sudoers \
     && sed -i -e 's/^#\(IgnorePkg\s*\)=\s*$/\1 = pacman-mirrorlist libc++ libc++abi libc++experimental/' /etc/pacman.conf \
     && sed -i -e 's/^#Color$/Color/' /etc/pacman.conf \
     && su francisco -c "git clone --depth 1 https://aur.archlinux.org/yay.git /tmp/yay && cd /tmp/yay && makepkg -cs" \
     && cd /tmp/yay && pacman -U --noconfirm *.pkg.tar.xz && cd .. && rm -rf yay \
     && su francisco -c "git clone --depth 1 https://aur.archlinux.org/libc++.git /tmp/libc++ && cd /tmp/libc++ && makepkg -cs --skippgpcheck --nocheck" \
     && cd /tmp/libc++ && pacman -U --noconfirm *.pkg.tar.xz && cd .. && rm -rf libc++
    
    USER francisco
    WORKDIR /home/francisco
    
    COPY --chown=francisco:francisco dotfiles .
    
    RUN xdg-user-dirs-update \
     && chmod 700 .gnupg \
     && chmod 600 .gnupg/gpg-agent.conf \
     && git config status.showUntrackedFiles no \
     && git clone https://github.com/tmux-plugins/tpm .tmux/plugins/tpm \
     && curl -fsSL https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash \
     && (curl -fsSL https://github.com/rbenv/rbenv-installer/raw/master/bin/rbenv-installer | bash || true) \
     && rustup toolchain install stable \
     && rustup default stable \
     && rm -rf /tmp/* .cache
    
    RUN yay -S --removemake --noconfirm \
        antibody-bin bitlbee-libpurple cargo-watch ccls dict-gcide \
        dict-wikt-en-all dua-cli dust goobook-git grv mdcat megatools \
        modd neo4j-community ovpngen pandoc-bin patat-bin pgformatter-git \
        ptpython purescript-bin purple-hangouts-hg python-grip ranger-git \
        sfnt2woff speedometer stardict-oald stardict-wordnet starship-bin \
        stig uni2ascii universal-ctags-git we-get-git xsv \
     && yay -Sc --noconfirm \
     && rm -rf /tmp/* .cache .cargo/git .cargo/registry
    
    RUN rustup component add clippy rust-src rustfmt \
     && yarn global add mathjs \
     && yarn global add commitizen \
     && yarn global add dockerfile-language-server-nodejs \
     && yarn global add esy \
     && yarn global add bs-platform \
     && yarn global add purescript-language-server \
     && yarn global add spago \
     && pip install --user tmuxp==1.3.2 \
     && pip install --user pgcli \
     && gem install sqlint \
     && rm -rf /tmp/* .cache
    
    COPY --chown=bitlbee:bitlbee bitlbee /var/lib/bitlbee
    
    CMD ["tmux", "new", "-s", "TMUX"]

Describe the results you received:

I get lots of call to execv failed (No such file or directory) on post-transaction hooks and /bin/sh: sed: command not found, when sed is in fact part of the base image.

Describe the results you expected:

Building the image with docker runs without problems. It also used to work with podman, but for some reason it stopped working lately.

Output of rpm -q buildah or apt list buildah:

❯ pacman -Qi buildah
Name            : buildah
Version         : 1.10.1-2
Description     : A tool which facilitates building OCI images
Architecture    : x86_64
URL             : https://github.com/containers/buildah
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : ostree  runc  skopeo  slirp4netns
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 24.99 MiB
Packager        : Bartłomiej Piotrowski <[email protected]>
Build Date      : Sun 01 Sep 2019 04:45:27 PM -03
Install Date    : Sun 01 Sep 2019 07:02:25 PM -03
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Output of buildah version:

❯ buildah version
Version:         1.10.1
Go Version:      go1.12.9
Image Spec:      1.0.1
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  v0.7.1
Git Commit:      8c1c2c58
Built:           Sun Sep  1 16:45:49 2019
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

❯ podman version
Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.8
OS/Arch:            linux/amd64

Output of cat /etc/*release:

❯ cat /etc/*release
Arch Linux release
LSB_VERSION=1.4
DISTRIB_ID=Arch
DISTRIB_RELEASE=rolling
DISTRIB_DESCRIPTION="Arch Linux"
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux

Output of uname -a:

❯ uname -a
Linux leibniz 5.2.14-arch2-1-ARCH containers/buildah#1 SMP PREEMPT Thu Sep 12 10:42:38 UTC 2019 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

❯ cat ~/.config/containers/storage.conf
[storage]
  driver = "overlay"
  runroot = "/run/user/1000"
  graphroot = "/home/francisco/.local/share/containers/storage"
  [storage.options]
    size = ""
    remap-uids = ""
    remap-gids = ""
    ignore_chown_errors = ""
    remap-user = ""
    remap-group = ""
    ostree_repo = ""
    skip_mount_home = ""
    mount_program = "/usr/bin/fuse-overlayfs"
    mountopt = ""
    [storage.options.thinpool]
      autoextend_percent = ""
      autoextend_threshold = ""
      basesize = ""
      blocksize = ""
      directlvm_device = ""
      directlvm_device_force = ""
      fs = ""
      log_level = ""
      min_free_space = ""
      mkfsarg = ""
      mountopt = ""
      use_deferred_deletion = ""
      use_deferred_removal = ""
      xfs_nospace_max_retries = ""

dnf transaction fails inside Fedora container and corrupts container

/kind bug

Description

Doing a dnf install inside a Fedora container or a Silverblue toolbox results in failed transaction and corrupted container.

Steps to reproduce the issue:

  1. podman --log-level=debug run -ti --restart=no --rm fedora

  2. Inside container: dnf install jq (jq is an example, it could be dnf upgrade)

Describe the results you received:

DNF transaction failed. If the command used is dnf upgrade the system is corrupted (library files from the packages in the transaction are deleted).

INFO[0000] running as rootless                          
DEBU[0000] Initializing boltdb state at /var/home/yann/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/yann/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/1000                     
DEBU[0000] Using static dir /var/home/yann/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/yann/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/fedora:latest" 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] exporting opaque data as blob "sha256:ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] exporting opaque data as blob "sha256:ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] Got mounts: []                               
DEBU[0000] Got volumes: []                              
DEBU[0000] Using slirp4netns netmode                    
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] exporting opaque data as blob "sha256:ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] created container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0000] container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" has work directory "/var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata" 
DEBU[0000] container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" has run directory "/tmp/1000/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata" 
DEBU[0000] New container created "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0000] container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" has CgroupParent "/libpod_parent/libpod-325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] overlay: mount_data=lowerdir=/var/home/yann/.local/share/containers/storage/overlay/l/A3V3TYSMAI2J2KG2KDT3Q66D2D,upperdir=/var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/diff,workdir=/var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/work,context="system_u:object_r:container_file_t:s0:c189,c381" 
DEBU[0000] mounted container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" at "/var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/merged" 
DEBU[0000] Created root filesystem for container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 at /var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Created OCI spec for container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 at /var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/config.json 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[-c 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 -u 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 -n jovial_jang -r /usr/bin/runc -b /var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata -p /tmp/1000/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/pidfile --exit-dir /run/user/1000/libpod/tmp/exits --conmon-pidfile /tmp/1000/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/yann/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 --socket-dir-path /run/user/1000/libpod/tmp/socket -t -l k8s-file:/var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/ctr.log --log-level debug --syslog]"
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/libpod_parent: permission denied 
DEBU[0000] Received container pid: 19072                
DEBU[0000] Created container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 in OCI runtime 
DEBU[0000] Attaching to container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0000] connecting to socket /run/user/1000/libpod/tmp/socket/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/attach 
DEBU[0000] Received a resize event: {Width:238 Height:55} 
DEBU[0000] Starting container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 with command [/bin/bash] 
DEBU[0000] Started container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0000] Enabling signal proxying                     
[root@325b9572ad86 /]# dnf install jq
Fedora Modular 30 - x86_64                                                                                                                                                                                    1.1 MB/s | 2.7 MB     00:02    
Fedora Modular 30 - x86_64 - Updates                                                                                                                                                                          959 kB/s | 3.0 MB     00:03    
Fedora 30 - x86_64 - Updates                                                                                                                                                                                  5.0 MB/s |  19 MB     00:03    
Fedora 30 - x86_64                                                                                                                                                                                             12 MB/s |  70 MB     00:05    
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                                   Architecture                                           Version                                                       Repository                                               Size
==============================================================================================================================================================================================================================================
Installing:
 jq                                                        x86_64                                                 1.6-2.fc30                                                    fedora                                                  168 k
Installing dependencies:
 oniguruma                                                 x86_64                                                 6.9.2-2.fc30                                                  updates                                                 193 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  2 Packages

Total download size: 361 k
Installed size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): oniguruma-6.9.2-2.fc30.x86_64.rpm                                                                                                                                                                      665 kB/s | 193 kB     00:00    
(2/2): jq-1.6-2.fc30.x86_64.rpm                                                                                                                                                                               462 kB/s | 168 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         193 kB/s | 361 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Installing       : oniguruma-6.9.2-2.fc30.x86_64                                                                                                                                                                                        1/2 
Error unpacking rpm package oniguruma-6.9.2-2.fc30.x86_64
  Installing       : jq-1.6-2.fc30.x86_64                                                                                                                                                                                                 2/2 
error: unpacking of archive failed on file /usr/lib/.build-id/1c/7588d6da78dd5888d79f988ab594f6b5abeeb5;5d5d2bf0: cpio: utime
error: oniguruma-6.9.2-2.fc30.x86_64: install failed

Error unpacking rpm package jq-1.6-2.fc30.x86_64
  Verifying        : oniguruma-6.9.2-2.fc30.x86_64                                                                                                                                                                                        1/2 
  Verifying        : jq-1.6-2.fc30.x86_64                                                                                                                                                                                                 2/2 

Failed:
  oniguruma-6.9.2-2.fc30.x86_64                                                                                              jq-1.6-2.fc30.x86_64                                                                                             

Error: Transaction failed
[root@325b9572ad86 /]# exit
DEBU[0471] Checking container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 status... 
DEBU[0471] Cleaning up container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0471] Network is already cleaned up, skipping...   
DEBU[0471] unmounted container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0471] Successfully cleaned up container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0471] Container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 storage is already unmounted, skipping...

Describe the results you expected:

I expect a successful installation of jq (or upgrade of the system if using dnf upgrade).

Additional information you deem important (e.g. issue happens only occasionally):

Issue did not happen a month ago. I don't know exactly when the problem appeared though.

Output of podman version:

Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.0.0-dev, commit: 164df8af4e62dc759c312eab4b97ea9fb6b5f1fc'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 10731966464
  MemTotal: 16551071744
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 8342466560
  SwapTotal: 8342466560
  arch: amd64
  cpus: 8
  hostname: work-laptop
  kernel: 5.2.9-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 58m 49.5s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/yann/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/yann/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /tmp/1000
  VolumePath: /var/home/yann/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.4.4-4.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora Silverblue 30.20190821.0.

hanging in the merged directory

carrying over #19 (comment)

Reproducer:

in one terminal:

[vbatts@getdown] {master} ~$ mkdir /tmp/{upper,workdir,merged}
[vbatts@getdown] {master} ~$ fuse-overlayfs -f -o lowerdir=$HOME,upperdir=/tmp/upper,workdir=/tmp/workdir /tmp/merged
UID=unchanged
GID=unchanged
UPPERDIR=/tmp/upper
WORKDIR=/tmp/workdir
LOWERDIR=/home/vbatts
MOUNTPOINT=/tmp/merged

in the other terminal:

cd /tmp/merged

Description

on the cd /tmp/merged, since I happen to have a large enough $HOME folder and a $PS1 that looks for a .git folder, with this fix now cd /tmp/merged causes fuse-overlayfs to freeze when my $PS1 scans for git changes. I can't crtl+c, only killall -9 fuse-overlayfs and umount /tmp/merged to cleanup.

More info:

[vbatts@getdown] {master} ~$ du -sh ~/
75G     /home/vbatts/
[vbatts@getdown] {master} ~$ find ~/ -type d | wc -l
177465
[vbatts@getdown] {master} ~$ echo $PS1
\[\033[0;94m\][\u@\h]\[\033[0m\]$(git branch &>/dev/null;\ if [ $? -eq 0 ]; then \ echo "$(echo `git status` | grep "nothing to commit" > /dev/null 2>&1; \ if [ "$?" -eq "0" ]; then \ # @4 - Clean repository - nothing to commit echo "\[\033[0;32m\]"$(__git_ps1 " (%s)"); \ else \ # @5 - Changes to working tree echo "\[\033[0;91m\]"$(__git_ps1 " {%s}"); \ fi) \[\033[1;94m\]\w\[\033[0m\]\$ "; \ else \ # @2 - Prompt when not in GIT repo echo " \[\033[0;94m\]\w\[\033[0m\]\$ "; \ fi)
[vbatts@getdown] {master} ~$ git ls-files | wc -l
243

Strace https://vbatts.fedorapeople.org/fuse-overlayfs_issue21.0.dump
(the command looked like sudo strace -o dump -f ./fuse-overlayfs -f -o lowerdir=$HOME,upperdir=/tmp/upper,workdir=/tmp/workdir /tmp/merged since strace without sudo got an EPERM on mount)

my work around

Setting export PS1="$> " allows me to cd into that directory, so it's something in the git status doing a directory traversal that is freezing up the fuse-overlayfs.

podman commit overlooks something?

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

  1. Run debian:10 in a container; install texlive and texlive-luatex. Create example.tex with the following content:
\documentclass{article}
\begin{document}
\end{document}

Run texhash (optionally convince yourself that this does not change the environment).

  1. Commit the container.

  2. Still in the container, run lualatex example.

  3. Create and run a container from the image you just commited. Run lualatex example.

Describe the results you received:
In the original container the compilation proceeds without problems. In the container started from the commit, the following message occurs:

luaotfload | db : Reload initiated (formats: otf,ttf,ttc); reason: "File not found: lmroman10-regular.".))

Describe the results you expected:
The container and the copy created from the commit behave identically.

Additional information you deem important (e.g. issue happens only occasionally):
The problem also occurs when building the image from a Dockerfile, i.e. to avoid the error one always has to run texhash in a fresh container, even if texhash is run during image creation. I cannot find an error in the file system of the freshly started container, and I am clueless as to what changes when texhash is run, apart from some timestamps.

Output of podman version:

podman version 1.5.1

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 7826296832
  MemTotal: 14639816704
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 34359734272
  SwapTotal: 34359734272
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: podstation
  kernel: 5.2.13-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 3h 33m 15.46s (Approximately 0.12 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /var/home/poduser/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/poduser/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 7
  RunRoot: /run/user/1000
  VolumePath: /var/home/poduser/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-3.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora Silverblue 30:

● ostree://fedora:fedora/30/x86_64/silverblue
                   Version: 30.20190912.0 (2019-09-12T20:42:50Z)
                    Commit: 6eab3e7e6afb27f6c3271c85242d744cba11287b39c08484b68d64cf85c01f9f

buildah run loses setuid bit with pacman

Description

This is really weird...

So if I install sudo on an Arch container with buildah run, it loses the setuid bit. Doing the same in a scratch container created with podman keeps it, however.

Steps to reproduce the issue:

  1. podman pull archlinux/base
  2. buildah from --name suid-test-work archlinux/base
  3. buildah run suid-test-work -- pacman -Sy --noconfirm sudo (NOTE: I'm aware -Sy is terrible without -u, but this isn't a production system and adding -u just makes it take longer to test.)
  4. buildah run suid-test-work -- ls -l /usr/bin/sudo

Describe the results you received:

-rwxr-xr-x 1 root root 140600 Jan 13  2019 /usr/bin/sudo

The setuid bit is not set.

Describe the results you expected:

-rwsr-xr-x 1 root root 140600 Jan 13  2019 /usr/bin/sudo

This is the result I get if I do podman run --rm -it archlinux/base, then run the pacman install from there.

Output of rpm -q buildah or apt list buildah:

buildah-1.11.2-2.git0bafbfe.fc30.x86_64

Output of buildah version:

Version:         1.11.2
Go Version:      go1.12.9
Image Spec:      1.0.1
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Wed Dec 31 18:00:00 1969
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Not entirely relevant but still somewhat I guess:

Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of cat /etc/*release:

Fedora release 30 (Thirty)
NAME=Fedora
VERSION="30.20190930.0 (Workstation Edition)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30.20190930.0 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
OSTREE_VERSION=30.20190930.0
Fedora release 30 (Thirty)
Fedora release 30 (Thirty)

Output of uname -a:

Linux hp-fedora 5.2.17-200.fc30.x86_64 containers/buildah#1 SMP Mon Sep 23 13:42:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver 
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""

# Set to skip a PRIVATE bind mount on the storage home directory.  Only supported by
# certain container storage drivers
skip_mount_home = "false"

Output of fuse-overlayfs --version:

fusermount3 version: 3.6.2
fuse-overlayfs: version 0.6.3
FUSE library version 3.6.2
using FUSE kernel interface version 7.29

(IMO this section should be in the template...)

Error with locating libfuse3.so.3

Receiving this error when trying to run the built fuse-overlayfs executable:

fuse-overlayfs: error while loading shared libraries: libfuse3.so.3: cannot open shared object file: No such file or directory

I have no problem with libfuse on its own, so I'm guessing fuse-overlayfs just isn't locating the library correctly? Not sure what to make of it.

EL6 I/O error with fuse-overlayfs 0.4.1

On Fedora 30, when using fuse-overlayfs-0.4.1-0.dev.git1ff7c64.fc30.x86_64 from updates-testing, installing packages in a CentOS 6 container results in an I/O error.

$ podman run -it --rm centos:6
[root@15e67895f0ba /]# yum --disablerepo \* install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
Loaded plugins: fastestmirror, ovl
Setting up Install Process
Examining /var/tmp/yum-root-zLjjSS/epel-release-latest-6.noarch.rpm: epel-release-6-8.noarch
Marking /var/tmp/yum-root-zLjjSS/epel-release-latest-6.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:6-8 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================
 Package                        Arch                     Version                Repository                                       Size
======================================================================================================================================
Installing:
 epel-release                   noarch                   6-8                    /epel-release-latest-6.noarch                    22 k

Transaction Summary
======================================================================================================================================
Install       1 Package(s)

Total size: 22 k
Installed size: 22 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : epel-release-6-8.noarch                                                                                            1/1
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/yum/rpmtrans.py", line 444, in callback
    self._instCloseFile(  bytes, total, h )
  File "/usr/lib/python2.6/site-packages/yum/rpmtrans.py", line 526, in _instCloseFile
    self.base.history.trans_data_pid_end(pid, state)
  File "/usr/lib/python2.6/site-packages/yum/history.py", line 868, in trans_data_pid_end
    """, ('TRUE', self._tid, pid, state))
  File "/usr/lib/python2.6/site-packages/yum/sqlutils.py", line 168, in executeSQLQmark
    return cursor.execute(query, params)
sqlite3.OperationalError: disk I/O error
error: python callback <bound method RPMTransaction.callback of <yum.rpmtrans.RPMTransaction instance at 0x1875ab8>> failed, aborting!
[root@15e67895f0ba /]# echo $?      
1
[root@15e67895f0ba /]# rpm -q epel-release
epel-release-6-8.noarch

The rpm is installed afterwards, but yum exits with a status of 1. registry.access.redhat.com/rhel6 has the same issue. This is as a non-root user. The problem is not reproducible as root. It appears to be limited to EL6, because I cannot reproduce it with CentOS 7 or RHEL 7 containers. Downgrading to fuse-overlayfs-0.3-10.dev.gita7c8295.fc30.x86_64 fixes the problem.

Other installed packages:

  • podman-1.4.0-2.fc30.x86_64
  • container-selinux-2.106-1.gitfc7111d.fc30.noarch
  • containers-common-0.1.37-0.gite079f9d.fc30.x86_64
  • containernetworking-plugins-0.7.5-1.fc30.x86_64

empty file created depending on the number of CPUs when writeback != 0

containerd/fuse-overlayfs-snapshotter#2

$ git clone https://github.com/AkihiroSuda/containerd-fuse-overlayfs
$ cd containerd-fuse-overlayfs
$ git checkout 5e1e2cf980ba0c12e7d1ddd8a6170dc36b9a1938
$ go test -c . && sudo taskset --cpu-list 1 ./containerd-fuse-overlayfs.test -test.root -test.v -test.run TestFUSEOverlayFS/LayerFileupdate 
--- FAIL: TestFUSEOverlayFS (0.01s)                                                                                                                    
    --- FAIL: TestFUSEOverlayFS/LayerFileupdate (0.04s)                                                                                                
        issues.go:62: Check snapshots failed: directory diff between /tmp/snapshot-suite-fuse-overlayfs-348405887/work/flat737854418 and /tmp/snapshot-
suite-fuse-overlayfs-348405887/work/check097464327                                                                                                     
            ~ /root/.bashrc(mode: 644, uid: 0, gid: 0) -> /root/.bashrc(mode: 644, uid: 0, gid: 0)                                                     
                                                                                                                                                       
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/continuity/fs/fstest.CheckDirectoryEqual                     
                /home/suda/go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/continuity/fs/fstest/compare.go:5
...

The file metadata are same, but the updated file has zero length. (containerd/fuse-overlayfs-snapshotter#2 (comment))

The issue doesn't happen when

  • More number of CPUs are assigned
  • Running with ptrace (which changes the scheduling sigfinicantly)
  • writeback=0 option is added

fuse-overlayfs version: c9bbc94
kernel: 5.3.0-1004-gcp #4-Ubuntu (Ubuntu 19.10)

v0.6 breaks postgres in rootless podman

Postgres does not longer run properly in rootless podman with fuse-overlayfs v0.6. It works with v0.5.2.
Now it fails with

waiting for server to start....2019-08-25 16:52:40.287 UTC [45] LOG:  could not bind Unix address "/var/run/postgresql/.s.PGSQL.5432": Input/output error
2019-08-25 16:52:40.287 UTC [45] HINT:  Is another postmaster already running on port 5432? If not, remove socket file "/var/run/postgresql/.s.PGSQL.5432" and retry.
2019-08-25 16:52:40.287 UTC [45] WARNING:  could not create Unix-domain socket in directory "/var/run/postgresql"
2019-08-25 16:52:40.287 UTC [45] FATAL:  could not create any Unix-domain sockets

How to reproduce:

podman run --rm -it postgres

tag a release?

even if just an 0.1, but something to landmark package by.

Invalid cross-device link when remove a file in merged directory

My host os is CentOS 7, and I backport fuse3 to it, and compile fuse-overlayfs in the master branch with commit 91bb401. The error is occured by following commands:

# ls lower
a  aaa  aaa1  aaa2
# ls upper
b  bbb  bbb1  bbb2
# fuse-overlayfs -o lowerdir=lower,upperdir=upper,workdir=work merged
# ls merged/
a  aaa  aaa1  aaa2  b  bbb  bbb1  bbb2
# rm -rf merged/aaa
rm: cannot remove ‘merged/aaa’: Invalid cross-device link
# rm -rf merged/bbb
rm: cannot remove ‘merged/bbb’: Invalid cross-device link

lower and upper is encrypt by gocryptfs or encfs, gocryptfs and encfs are fuse filesystem which use fuse high level apis. Take a look at rfjakob/gocryptfs#290 and vgough/encfs#542.
I have test it out that files in lower directory can not be modified, and files in upper directory can be modified.
Fedora 29 is the same with CentOS 7.

Spurious files in a directory deleted in upper layer

Given a directory structure:

/dir1/dir2/foo

when dir1 gets removed and then in another layer dir1 and dir1/dir2 get recreated, the file foo is not correctly hidden as it should be. This seems to be an extension of the problem reported in containers/podman#3021, which is marked as fixed, but it only seems to be fixed for the case when the file is in the deleted directory. When the file is in a subdirectory of the deleted directory, the issue is still present.

It can be reproduced using podman by building the following Dockerfile and checking for existence of the file foo using podman run as unpriviledged user.

FROM busybox

RUN mkdir -p /dir1/dir2
RUN touch /dir1/dir2/foo
RUN rm -r /dir1
RUN mkdir -p /dir1/dir2

There are actually two variants of the problem.

  1. With the dockerfile above, the whole dir1 gets marked with a whiteout file.
  2. When the above dockerfile is altered so that the last two RUN instructions are merged into one, dir1 gets marked as opaque (this only occurs when building with docker, not buildah).

fuse-overlayfs seems to handle both cases incorrectly, whereas kernel overlay works as expected (using podman as root or using docker)

Dockerfile build fails

Building the library using builda bud as explained in the docs

`buildah bud -t fuse-overlayfs -f ./Dockerfile.static .`

fails at this line with error:

[...]
Installing /build/libfuse/doc/mount.fuse3.8 to /usr/share/man/man8
Installing /build/libfuse/build/meson-private/fuse3.pc to /usr/lib64/pkgconfig
Running custom install script '/build/libfuse/util/install_helper.sh /etc /usr/bin /usr/lib/udev/rules.d true'
mknod: /dev/fuse: Operation not permitted
FAILED: meson-install
/usr/bin/meson install --no-rebuild
ninja: build stopped: subcommand failed.

As far as I understand the container runtime (rightfully) prevents this call to mknod. If run in privileged mode, the call would probably succeed, but that is probably unsafe and will likely modify the host; on the other hand, this might be desired (in my case, the host already has FUSE2 installed, so it makes no difference).

Instead, I simply "tricked" the script into believing that the device already exists by adding the line touch /dev/fuse before this one. With this workaround the build completes successfully.

Further problems I encountered:

  • this command does not seem to work: fuse-overlayfs: upperdir not specified; as far as I understand, this is expected.
  • it might be helpful to newcomers to explain what to do after the build (grab the binary from the image and install it somewhere in the PATH on the host).

buildah bud fails due to missing file system in the container

Failing case: missing mount binding to the parent container

#docker run -it --privileged quay.io/buildah/stable
[root@e8c64993a35b /]# echo 'FROM alpine' > Dockerfile
[root@e8c64993a35b /]# echo 'RUN whoami' >> Dockerfile
[root@e8c64993a35b /]# buildah bud -t foo .
STEP 1: FROM alpine
Getting image source signatures
Copying blob 9d48c3bd43c5 done
Copying config 9617696764 done
Writing manifest to image destination
Storing signatures
STEP 2: RUN whoami
process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1
                                                                                                      error building at STEP "RUN whoami": exit status 1
[root@e8c64993a35b /]# 

Passing case: mount binding enabled to the parent container

#docker run -it --privileged -v containers:/var/lib/containers quay.io/buildah/stable
[root@c4975399f13e /]# echo 'FROM alpine' > Dockerfile
[root@c4975399f13e /]# echo 'RUN whoami' >> Dockerfile
[root@c4975399f13e /]# buildah bud -t foo .
STEP 1: FROM alpine
STEP 2: RUN whoami
root
STEP 3: COMMIT foo
Getting image source signatures
Copying blob 03901b4a2ea8 skipped: already exists
Copying blob 652252567dcf done
Copying config bbf36d740b done
Writing manifest to image destination
Storing signatures
bbf36d740bc9b1defd3efb7cce0900b70f7a532863b7bc49d89c8a0c44a4d1f6
[root@c4975399f13e /]# 

So far so good. Now I converted this cli invocation into a tekton pipeline as follows (only the task steps are shown that is most relevant here)

  steps:
    - name: assemble-extract-step
      image: gireeshpunathil/foo
      command: ["/bin/bash"]
      args:
        - -c
        - "somescript.sh"
      env:
      securityContext:
        privileged: true
      volumeMounts:
        - name: varlibcontainers
          mountPath: /var/lib/containers
  volumes:
    - name: varlibcontainers

The image gireeshpunathil/foo is essentially a thin layer on top of quay.io/buildah/stable.

This works as expected with minikube. (just like second case above)

This fails in minishift (just like first case above)

With my limited experience with minishift, I believe this has something to do with either:

  • the mount bind is not properly assigned processed with the container.
  • sufficient privilege has not yet been provided

As I am not very sure about the root cause, and as the issue is manifesting as a buildah command failure, I am raising it here. Please advise, thanks in advance!

Additional info - queried from the faulty container:

# mount | grep container
/dev/mapper/rhel-root on /var/lib/containers type xfs (rw,relatime,attr2,inode64,noquota)
# env | grep BUILDAH
_BUILDAH_STARTED_IN_USERNS=
BUILDAH_ISOLATION=chroot

excerpt from strace:

newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/bd2ae87e47f432e463e519a14a27376510799b51d5a1be793297f22a760a2bf2/merged/bin/bash", 0xc000785d78, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
# ls -lrt /var/lib/containers/storage/overlay/bd2ae87e47f432e463e519a14a27376510799b51d5a1be793297f22a760a2bf2/merged/
total 0
drwx------ 2 root root  6 Aug 27 04:03 sys
drwx------ 2 root root  6 Aug 27 04:03 proc
drwx------ 2 root root  6 Aug 27 04:03 dev
d--x--x--x 3 root root 21 Aug 27 04:03 run
d--x--x--x 2 root root 38 Aug 27 04:03 etc
#

So! /usr and and many other system folders are missing! how this could be!

All other required info for the bug tracker (system information etc.) are precisely the same as in containers/buildah#1814, so not duplicating here.

Be less noisy

Hello,

maybe it makes sense to remove this output to stderr:

fprintf (stderr, "uid=%s\n", lo.uid_str ? : "unchanged");

It is kind of noisy since its printed on every invocation and echoes parameters. Maybe only show it if some debugging flag is specified?

Cheers

ihucos

Spurious files in a directory marked as opaque

#136 is currently fixed for the case when the directory is marked with a whiteout file (I tested this), but it is still reproducible in the second variant where the directory is marked opaque. Reproducer:

FROM busybox

RUN mkdir -p /dir1/dir2
RUN touch /dir1/dir2/foo
RUN rm -r /dir1 && mkdir -p /dir1

The image has to be built with docker in this case, buildah doesn't generate opaque directories. In this case the /dir1 gets marked as opaque. When run with unpriviledged podman (using fuse-overlayfs from current master branch), the foo file is still visible.

umount needs root

description of the problem
Seems like umount needs root permissions

steps to reproduce the problem

mkdir -p lowerdir/{a,b}
mkdir {merged,up,workdir}
fuse-overlayfs -o lowerdir=lowerdir/a:lowerdir/b,upperdir=up,workdir=workdir merged

# Permission denied
umount merged 

# works
sudo umount merged 

with actived -o allow_other I'm getting this (sudo works again):

umount failed: Operation not permitted

environment

  • Debian 9 on VirtualBox 6 on Windows 10 on AMD 64
  • kernel: 4.19.0-0.bpo.2-amd64 #1 SMP Debian 4.19.16-1~bpo9+1 (2019-02-07) x86_64 GNU/Linux
  • fuse3 3.4.1-1~bpo9+1 (own backport from Debian 10)
  • fuse-overlayfs v0.3 (built from GIT) using FUSE kernel interface version 7.27
  • umount from util-linux 2.29.2 (libmount 2.29.2: selinux, btrfs, assert, debug)

support for removing files

Built from 1c72a1a

for lowerdir files, there is a whiteout needed when an unlink happens. Unlinking currently works for files created in the upperdir.

in one terminal:

[vbatts@poly] {master} ~$ mkdir /tmp/{upper,workdir,merged}
[vbatts@poly] {master} ~$ fuse-overlayfs -f -o lowerdir=$HOME,upperdir=/tmp/upper,workdir=/tmp/workdir /tmp/merged
UID=unchanged
GID=unchanged
UPPERDIR=/tmp/upper
WORKDIR=/tmp/workdir
LOWERDIR=/home/vbatts
MOUNTPOINT=/tmp/merged

in the other terminal:

[vbatts@poly] {master} /tmp/merged$ ls
bin/  Documents/  Downloads/  go1.10/  gopath/  memfetch.tgz  Projects/  src/  tmp/  tmp.984XhpMqlB/
[vbatts@poly] {master} /tmp/merged$ touch file 
[vbatts@poly] {master} /tmp/merged$ stat file
  File: file
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 46h/70d Inode: 2744322     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/  vbatts)   Gid: ( 1000/  vbatts)
Context: system_u:object_r:fusefs_t:s0
Access: 2018-09-26 11:07:09.558930372 -0400
Modify: 2018-09-26 11:07:09.558930372 -0400
Change: 2018-09-26 11:07:09.558930372 -0400
 Birth: -
[vbatts@poly] {master} /tmp/merged$ rm file
[vbatts@poly] {master} /tmp/merged$ rm memfetch.tgz
rm: cannot remove 'memfetch.tgz': Operation not supported

`do_rm` fails to remove directories in upper layer (if whiteouts present?)

Test case:

container="$( buildah from -q debian:10-slim )"
buildah run "${container}" sh -ec 'apt-get update -qq && apt-get install -qqy time'
buildah run "${container}" rm -rf /usr/share/zoneinfo

Removing directories in an upper layer fails with, e.g.,

/bin/rm: cannot remove '/usr/share/zoneinfo/Australia/Adelaide': Is a directory

where /usr/share/zoneinfo/Australia/Adelaide is actually a file, not a directory.

gh-89 introduced a bug such that do_rm might not call empty_dir and not remove .wh.* files from a folder.

Used builds:

  • /tmp/fuse-overlayfs-7bc2dd9: release 0.4.2
  • /tmp/fuse-overlayfs-95e1d01: gh-88
  • /tmp/fuse-overlayfs-bd9eb8d: gh-89
  • /tmp/fuse-overlayfs-9c77da7: gh-91
  • /tmp/fuse-overlayfs-4e32e89: gh-92

Full test case:

buildah_fuse_overlayfs() {
  local fuse_overlayfs_bin
  fuse_overlayfs_bin="${1}"
  shift
  buildah --runroot=/tmp/containers-tmp --root=/tmp/containers-tmp --storage-opt=overlay.mount_program="${fuse_overlayfs_bin}" "${@}"
}
for fuse_overlayfs_bin in \
    /tmp/fuse-overlayfs-7bc2dd9 \
    /tmp/fuse-overlayfs-95e1d01 \
    /tmp/fuse-overlayfs-bd9eb8d \
    /tmp/fuse-overlayfs-9c77da7 \
    /tmp/fuse-overlayfs-4e32e89 \
    ; do
  echo "${fuse_overlayfs_bin}"
  container="$( buildah_fuse_overlayfs "${fuse_overlayfs_bin}" from -q debian:10-slim )"
  buildah_fuse_overlayfs "${fuse_overlayfs_bin}" run "${container}" sh -ec 'apt-get update -qq && apt-get install -qqy time >/dev/null 2>&1'
  buildah_fuse_overlayfs "${fuse_overlayfs_bin}" run "${container}" rm -rf /usr/share/zoneinfo || true
  buildah_fuse_overlayfs "${fuse_overlayfs_bin}" rm  "${container}" >/dev/null
done
/tmp/fuse-overlayfs-7bc2dd9
/tmp/fuse-overlayfs-95e1d01
/tmp/fuse-overlayfs-bd9eb8d
/bin/rm: cannot remove '/usr/share/zoneinfo/Australia/Adelaide': Is a directory
error while running runtime: exit status 1
ERRO[0000] exit status 1                                
/tmp/fuse-overlayfs-9c77da7
/bin/rm: cannot remove '/usr/share/zoneinfo/EST': Is a directory
error while running runtime: exit status 1
ERRO[0000] exit status 1                                
/tmp/fuse-overlayfs-4e32e89
/bin/rm: cannot remove '/usr/share/zoneinfo/Australia/NSW': Is a directory
error while running runtime: exit status 1
ERRO[0000] exit status 1                                

Partially reverting 605e17f, i.e., removing 605e17f#diff-2045016cb90d1e65d71c2407a2570927R2607 fixes the issue.
However, I'm not sure this would be correct. If I understand do_rm, count_dir_entries and empty_dir correctly, the change from that commit makes sense. Which would imply that there is an inconsistency between node->children (whiteout) nodes and actual present .wh.* files.

(... and that's as far as I can analyze the problem without reading all of the code ;). )

fuse-overlayfs frequently gets stuck forever waiting for I/O

fuse-overlayfs --version output:

fuse-overlayfs: version 0.3
FUSE library version 3.4.2
using FUSE kernel interface version 7.27
fusermount3 version: 3.4.2

In short, whenever I try to build plan9port inside my toolbox, fuse-overlayfs gets stuck waiting on I/O and never finishes:

$ ps -A -o comm,pid,stat | rg fuse-overlay
fuse-overlayfs  26946 Ss

No amount of Ctrl-C'ing will stop it; I have to instead stop the container and manually kill the process.

I killed it with pkill -SIGABRT to get a core dump, here's the stack trace:

                #0  0x00007f6a43558c57 openat64 (libc.so.6)
                #1  0x0000000000407ac8 ovl_do_open (fuse-overlayfs)
                #2  0x0000000000407db7 ovl_create (fuse-overlayfs)
                #3  0x00007f6a4366a0ab n/a (libfuse3.so.3)
                #4  0x00007f6a4366c8e6 n/a (libfuse3.so.3)
                #5  0x00007f6a43667c6b fuse_session_loop (libfuse3.so.3)
                #6  0x0000000000402c1e main (fuse-overlayfs)
                #7  0x00007f6a43490f33 __libc_start_main (libc.so.6)
                #8  0x0000000000402f8e _start (fuse-overlayfs)

and with debug symbols:

(gdb) bt
#0  0x00007f6a43558c57 in openat64 () from /lib64/libc.so.6
#1  0x0000000000407ac8 in ovl_do_open (req=req@entry=0x18f4990, parent=parent@entry=26679616, 
    name=name@entry=0x7f6a43439048 "9c.14754.ryan.out", flags=164417, mode=mode@entry=33206)
    at main.c:2457
#2  0x0000000000407db7 in ovl_create (req=0x18f4990, parent=26679616, 
    name=0x7f6a43439048 "9c.14754.ryan.out", mode=33206, fi=0x7ffc8ee14e80) at main.c:2596
#3  0x00007f6a4366a0ab in ?? () from /lib64/libfuse3.so.3
#4  0x0000000000008241 in ?? ()
#5  0x0000000000000000 in ?? ()
(gdb) 

symlinks are broken

Hi,
I was testing fuse-overlayfs when I noticed some issues when using symlinks. Sometimes symlinks stay in upper but are not visible from merged. It's also reproducible if you create a symlink that doesn't point to any existing file:

[sthiell@sh-101-60 /merged/lntest]$ ln -s non_existant_file link
[sthiell@sh-101-60 /merged/lntest]$ ls -l
total 0
[sthiell@sh-101-60 /merged/lntest]$ ls -lisa
total 8
144115205440289463 4 drwxr-xr-x 2 sthiell group 4096 Oct 31 14:12 .
144116351290596352 4 drwx--x--x 9 sthiell group 4096 Oct 31 14:12 ..
[sthiell@sh-101-60 /merged/lntest]$ ls -l /upper/lntest/
total 0
lrwxrwxrwx 1 root root 17 Oct 31 14:12 link -> non_existant_file
[sthiell@sh-101-60 /merged/lntest]$ cd ..
[sthiell@sh-101-60 /merged]$ rmdir lntest
rmdir: failed to remove ‘lntest’: Directory not empty
[sthiell@sh-101-60 /merged]$ rm -rf lntest
rm: cannot remove ‘lntest’: Directory not empty

v0.5 breaks postgres in rootless podman

Postgres does not longer run properly in rootless podman with fuse-overlayfs v0.5. It works with v0.4.1.
Now it fails with

waiting for server to start....2019-08-06 16:39:14.659 UTC [44] LOG:  could not set permissions of file "/var/run/postgresql/.s.PGSQL.5432": No such device or address
2019-08-06 16:39:14.659 UTC [44] WARNING:  could not create Unix-domain socket in directory "/var/run/postgresql"
2019-08-06 16:39:14.659 UTC [44] FATAL:  could not create any Unix-domain sockets

How to reproduce:

podman run --rm -it postgres

This is on
Linux linux1 5.2.6-arch1-1-ARCH #1 SMP PREEMPT Sun Aug 4 14:58:49 UTC 2019 x86_64 GNU/Linux
But I tested it with older kernels too (some 5.1.x)

UPDATE
Poor mans bisect with makepkg shows this:

1395753e1a17da68f35ab05b805878b9da0972b1 broken
9e20d96bbf2de0f32abb7ab8eb9bd02f11510c9e broken
62cbd9977c5be5968e0c1d404c26c5a346a486f2 broken
0b0c7a3a01dae4de65ba79016a111d794c1d1719 broken
468cb46773dc70349d511e1f5d4748d792e979b3 ok
241fecf92b9540266affb950bcf29866bcd0167c ok
d092552b76732bbf2f2f0bdbd4cf458f13ded330 ok
eb509974efe9e21a122a93a6cc16379ae66ef288 ok
d9c5215e33f3cee938b263b80d3c8f628234b9e5 ok

so 0b0c7a3 it is .

fuse-overlay needs to support readonly lower with no upper.

Containers storage, has a fix that allows additional-stores to be mounted read/only, but this requires the mounting of multi-layer lowers without an upper directory. Currently fuse-overlay does not support this and is has to be disabled from this feature.

Minimum required kernel version

Hi,

When trying to use the static build across hosts, we only managed to properly use the binary with relatively new kernels (in our config it works with 4.15.0-66).

On machines with older kernels (we tested 4.04 and also below, on Ubuntu and CentOS systems), the merged directory didn't work:

[salia@'h:ofs-test ] $ls -laht
ls: cannot access 'merged': Invalid argument
total 20K
drwxr-xr-x  3 salia domain users 4.0K Nov 11 15:36 workdir
drwxr-xr-x  6 salia domain users 4.0K Nov 11 15:36 .
drwxr-xr-x  2 salia domain users 4.0K Nov 11 15:36 upperdir
drwxr-xr-x  2 salia domain users 4.0K Nov 11 15:36 lowerdir
drwxrwxrwt 13 root  root         4.0K Nov 11 15:36 ..
d?????????  ? ?     ?               ?            ? merged

Is there some required minimum kernel version for using the statically built fuse-overlayfs binary?

Thanks,

Andras

Segfault on rename (?)

Apparently, fuse-overlayfs can segfault on rename under some (to be identified) condition.

Reproducer on current Silverblue (F30, commit 06ef84d208a1c29ce18a00b51d4ef5cd101365037f580cde0bbed423b2c051e3):

  1. Create and enter some container (in my case, toolbox create and toolbox enter)
  2. Run sudo rm /usr/share/glib-2.0/schemas/gschemas.compiled
  3. Run sudo glib-compile-schemas /usr/share/glib-2.0/schemas/
  4. The final rename fails and Bus error is written.
$ sudo glib-compile-schemas /usr/share/glib-2.0/schemas/
Warning: Schema “org.gnome.system.locale” has path “/system/locale/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy” has path “/system/proxy/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.http” has path “/system/proxy/http/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.https” has path “/system/proxy/https/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.ftp” has path “/system/proxy/ftp/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.socks” has path “/system/proxy/socks/”.  Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Failed to rename file “/usr/share/glib-2.0/schemas/gschemas.compiled.49TX0Z” to “/usr/share/glib-2.0/schemas/gschemas.compiled”: g_rename() failed: Software caused connection abort
Bus error

From the logs:

May 01 12:38:48 nuc audit[7529]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=2 subj=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 pid=7529 comm="fuse-overlayfs" exe="/usr/bin/fuse-overlayfs" sig=11 res=1
May 01 12:38:48 nuc kernel: fuse-overlayfs[7529]: segfault at 10 ip 0000000000406a34 sp 00007ffe2cc756c0 error 4 in fuse-overlayfs[402000+9000]
May 01 12:38:48 nuc kernel: Code: 89 fd be 3c b0 40 00 48 8d 7c 24 40 48 8d 42 01 48 89 05 27 89 00 00 31 c0 e8 48 bb ff ff eb 39 66 0f 1f 44 00 00 48 8b 43 10 <8b> 70 10 41 b8 00 01 00 00 48 8d 4c 24 60 bf 01 00 00 00 e8 94 b7
May 01 12:38:48 nuc systemd[1]: Started Process Core Dump (PID 7687/UID 0).
May 01 12:38:48 nuc audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@22-7687-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 01 12:38:49 nuc audit[7686]: ANOM_ABEND auid=1000 uid=100000 gid=100000 ses=2 subj=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 pid=7686 comm="glib-compile-sc" exe="/usr/bin/glib-compile-schemas" sig=7 res=1
May 01 12:38:49 nuc systemd-coredump[7688]: Process 7529 (fuse-overlayfs) of user 1000 dumped core.
                                            
                                            Stack trace of thread 7529:
                                            #0  0x0000000000406a34 get_node_up (fuse-overlayfs)
                                            #1  0x0000000000408c42 ovl_rename (fuse-overlayfs)
                                            #2  0x00007f8baf7908e6 n/a (libfuse3.so.3)
                                            #3  0x00007f8baf78bc6b fuse_session_loop (libfuse3.so.3)
                                            #4  0x0000000000402c1e main (fuse-overlayfs)
                                            #5  0x00007f8baf5b4f33 __libc_start_main (libc.so.6)
                                            #6  0x0000000000402f8e _start (fuse-overlayfs)

NULL node->children passed to hash_lookup

Using fuse-overlayfs to back a rootless container, I seem to be able to reliably trigger the following crash in v0.6.1. The make process inside the container runs

450   openat(AT_FDCWD, "/opt/build/build_dir/target-arm_xscale_musl_eabi/linux-kirkwood/linux-4.14.138/.vermagic", O_RDONLY <unfinished ...>

and fuse-overlayfs dies with

Program received signal SIGSEGV, Segmentation fault.
0x000000010ad12e40 in safe_hasher (table=table@entry=0x0, key=key@entry=0x7fffe618ba20) at hash.c:251                                                       
251       size_t n = table->hasher (key, table->n_buckets);
(gdb) bt
#0  0x000000010ad12e40 in safe_hasher (table=table@entry=0x0, key=key@entry=0x7fffe618ba20) at hash.c:251                                                   
#1  0x000000010ad13688 in hash_lookup (table=0x0, entry=0x7fffe618ba20) at hash.c:263                                                                       
#2  0x000000010ad0b684 in do_lookup_file (lo=0x7fffe618bf50, parent=<optimized out>, name=0x7fffbe1c0038 "toolchain-arm_xscale_gcc-7.4.0_musl_eabi")        
    at main.c:1739
#3  0x000000010ad0c268 in ovl_lookup (req=0x1145fca40, parent=70405, name=0x7fffbe1c0038 "toolchain-arm_xscale_gcc-7.4.0_musl_eabi") at main.c:1863         
#4  0x00007fffbe68c724 in do_lookup (req=<optimized out>, nodeid=<optimized out>, inarg=<optimized out>) at ../lib/fuse_lowlevel.c:1039                     
#5  0x00007fffbe68edfc in fuse_session_process_buf_int (se=0x113c5d390, buf=0x7fffe618bea0, ch=<optimized out>) at ../lib/fuse_lowlevel.c:2579              
#6  0x00007fffbe687bf4 in fuse_session_loop (se=0x113c5d390) at ../lib/fuse_loop.c:34                                                                       
#7  0x000000010ad03e94 in main (argc=<optimized out>, argv=<optimized out>) at main.c:5054      

Beyond being able to say that pnode->children is NULL, I'm at something of a loss.

musl build failure

FTBFS with musl libc

cc -DHAVE_CONFIG_H -I.    -I . -I /builddir/fuse-overlayfs-0.3/lib -I/usr/include/fuse3 -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe    -MT fuse_overlayfs-main.o -MD -MP -MF .deps/fuse_overlayfs-main.Tpo -c -o fuse_overlayfs-main.o `test -f 'main.c' || echo './'`main.c
main.c:39:10: fatal error: error.h: No such file or directory
 #include <error.h>
          ^~~~~~~~~
compilation terminated.
make[2]: *** [Makefile:571: fuse_overlayfs-main.o] Error 1

https://travis-ci.org/void-linux/void-packages/jobs/498513645

Are remote / network filesystems supported as upperdir?

Hi,

In kernel overlay there are restrictions to explicitly prohibit network based / remote filesystems (e.g. CephFS) as upperdir: https://github.com/torvalds/linux/blob/b41dae061bbd722b9d7fa828f35d22035b218e18/fs/overlayfs/super.c#L754

I was wondering about the implementation of fuse-overlayfs, whether it would support such filesystems? In particular whether it would support filesystems where dcache is not necessarily up-to-date. (https://lwn.net/Articles/649115/ - "More than just a cache" section)

Many thanks,

Andras

Subtle errors when running from inside a chroot

When trying to run fuse-overlayfs from inside a chroot, there are hard to pin down, subtle errors.
This is some output of installing firefox inside arch linux with pacman.

(10/15) Creating temporary files...
Setting default ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Setting access ACL "u::rwx,g::r-x,g:wheel:r-x,g:adm:r-x,m::r-x,o::r-x" on /var/log/journal failed: Invalid argument
Cannot set file attribute for '/var/log/journal', value=0x00800000, mask=0x00800000, ignoring: Function not implemented
Cannot set file attribute for '/var/log/journal/remote', value=0x00800000, mask=0x00800000, ignoring: Function not implemented
error: command failed to execute correctly
(11/15) Reloading device manager configuration...
  Skipped: Device manager is not running.
(12/15) Arming ConditionNeedsUpdate...
(13/15) Updating the info directory file...
(14/15) Updating the desktop file MIME type cache...
(15/15) Updating the MIME type database...

When using apt, there are errors about failed GPG checks, that I can not reproduce, it's kind of difficult to pin down.

This is interesting for nested containers. With unionfs-fuse I also get apparently the same kind of error: rpodgorny/unionfs-fuse#84 . But much more often.

Let me know if I should try to find an reproducible example or give me some direction to investigate.

chmod a-rw results in zombie directory

I noticed this issue when I was trying to run makepkg in an overlayfs.

Here's how it can be reproduced:

[/tmp]$ mkdir overlay && cd $_
[/tmp/overlay]$ mkdir lower upper work merged
[/tmp/overlay]$ fuse-overlayfs --debug -o rw,async,exec,dev,suid,nonempty,default_permissions,allow_other,use_ino,lowerdir=./lower,upperdir=./upper,workdir=./work ./merged
uid=unchanged
uid=unchanged
upperdir=/tmp/overlay/upper
workdir=./work
lowerdir=./lower
mountpoint=./merged
FUSE library version: 3.7.0
[/tmp/overlay]$ mkdir merged/test && chmod a-rw $_
[/tmp/overlay]$ rm -rf merged/test
rm: cannot remove 'merged/test': Permission denied
[/tmp/overlay]$ ll merged upper
merged:
ls: merged/test: Permission denied
total 0
drwxr-xr-x 3 frederick frederick 60 Oct 23 04:18 .
d--x--x--x 2 frederick frederick 40 Oct 23 04:18 test

upper:
total 0
drwxr-xr-x 3 frederick frederick  60 Oct 23 04:18 .
drwxr-xr-x 6 frederick frederick 120 Oct 23 04:18 ..
d--x--x--x 2 frederick frederick  40 Oct 23 04:18 test
[/tmp/overlay]$ rm -rf merged/test
[/tmp/overlay]$ ll merged upper
merged:
ls: merged/test: Permission denied
total 0
drwxr-xr-x 3 frederick frederick 60 Oct 23 04:18 .
d--x--x--x 2 frederick frederick 40 Oct 23 04:18 test

upper:
total 0
drwxr-xr-x 3 frederick frederick  60 Oct 23 04:18 .
drwxr-xr-x 6 frederick frederick 120 Oct 23 04:18 ..
d--x--x--x 2 frederick frederick  40 Oct 23 04:18 test

/tmp/overlay/merged/test in this example cannot be deleted and it somehow shows and disappears randomly in my experiment. chmod a+rw also fails.

Changelog

It would be great to see a changelog or release notes to have a distilled look into updates.

roadmap

Hi @giuseppe ,

I found this repo through the singularity thread apptainer/singularity#1207 and wanted to understand a bit more the status of this implementation. is there already a PoC of this working with a container runtime like podman or is that still far in the future?

Most of the infrastructure we use at CERN and elsewhere is based on RHEL so the ubuntu patches are not in play for us -- so this is very interesting to us

setattr fails with readonly /proc

setattr fails if /proc is mounted as readonly. It used to work at some point but now that symlinks are handled through /proc/self/fd it will fail with EROFS.

AFAIK it's not easily fixable with the current logic since we're opening the fd on the symlink early on
and syscalls like utimensat won't work on symlink fds.

fuse-overlayfs/main.c

Lines 3309 to 3317 in 1faf62c

case S_IFLNK:
cleaned_up_fd = TEMP_FAILURE_RETRY (openat (dirfd, node->path, O_PATH|O_NOFOLLOW|O_NONBLOCK));
if (cleaned_up_fd < 0)
{
fuse_reply_err (req, errno);
return;
}
sprintf (path, "/proc/self/fd/%d", cleaned_up_fd);
break;

Is there a reason not to rely on things like utimensat(dirfd,...), fchownat, fchmodat instead?

centos7 libfuse?

looks like the fuse-devel-2.9.2-10.el7.x86_64 on centos7 is not new enough or not found for building.

checking for uint8_t... yes
checking for ptrdiff_t... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for FUSE... no
configure: error: *** libfuse not found
vbatts@centos7-2503932173 ~/src/github.com/containers/fuse-overlayfs$ rpm -q fuse-devel
fuse-devel-2.9.2-10.el7.x86_64

rootless podman breaks extended filesystem attributes

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Listing extended filesystem attributes for a directory is broken in rootless podman but works with sudo podman.

Steps to reproduce the issue:

  1. podman run --rm -it centos:centos7.6.1810

  2. yum install -y attr && getfattr / (this fails with Is a directory)

  3. Then try sudo podman run --rm -it centos:centos7.6.1810 and run the command above. This does not raise an error.

Describe the results you received:

Is a directory

Describe the results you expected:

No error in either scenario (or at least the same behavior)
Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 1.4.4

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.0.0-dev, commit: 164df8af4e62dc759c312eab4b97ea9fb6b5f1fc'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 5509607424
  MemTotal: 29436071936
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 17172848640
  SwapTotal: 17179865088
  arch: amd64
  cpus: 8
  hostname: jah-m4800.redacted.local
  kernel: 5.2.7-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 48h 57m 26.72s (Approximately 2.00 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/jah/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/jah/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 21
  RunRoot: /tmp/1000
  VolumePath: /home/jah/.local/share/containers/storage/volumes

fusermount3: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf

I'm trying to package this application for Debian (cf. http://bugs.debian.org/917889). While trying it out, I'm running into this issue:

./fuse-overlayfs -o lowerdir=/ -o upperdir=/tmp/upper/ -o workdir=/tmp/overlay-workdir/ /tmp/overlay
UID=unchanged
GID=unchanged
UPPERDIR=/tmp/upper
WORKDIR=/tmp/overlay-workdir/
LOWERDIR=/
MOUNTPOINT=/tmp/overlay
fusermount3: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf
./fuse-overlayfs: cannot mount: Operation not permitted

After editing /etc/fuse.conf to enable the option user_allow_other, the program appears to work. Unfortunately, this is not the default setting in Debian. My questions here are:

  1. Is there any way to modify the program to work without editing /etc/fuse.conf?
  2. What are the security ramifications for enabling this option?
  3. Does RHEL8 enable this option by default? - If no, is the superuser expected to do that modification himself?

Thank you

Failure to invalidate cache on chmod

On the host, we begin:

veloci% ls -l /dozer/scratch/openwrt/images/darknetgw/upper/tmp/stage-perl/usr/lib/perl5/5.28/CORE/regexp.h
-r--r--r-- 1 nwf nwf 36363 Sep 29 18:24 /dozer/scratch/openwrt/images/darknetgw/upper/tmp/stage-perl/usr/lib/perl5/5.28/CORE/regexp.h  

Inside the overlayfs mount, inside an unprivileged user container, we run:

bash-5.0$ ls -l '/opt/build/tmp/stage-perl/usr/lib/perl5/5.28/CORE/uconfig.h'
-r--r--r-- 1 daemon 1 162891 Sep 29 18:23 /opt/build/tmp/stage-perl/usr/lib/perl5/5.28/CORE/uconfig.h
bash-5.0$ chmod 644 '/opt/build/tmp/stage-perl/usr/lib/perl5/5.28/CORE/regexp.h'
bash-5.0$ ls -l '/opt/build/tmp/stage-perl/usr/lib/perl5/5.28/CORE/uconfig.h'
-r--r--r-- 1 daemon 1 162891 Sep 29 18:23 /opt/build/tmp/stage-perl/usr/lib/perl5/5.28/CORE/uconfig.h

However, on the host, the file actually has been modified:

veloci% ls -l /dozer/scratch/openwrt/images/darknetgw/upper/tmp/stage-perl/usr/lib/perl5/5.28/CORE/regexp.h
-rw-r--r-- 1 nwf nwf 36363 Sep 29 18:24 /dozer/scratch/openwrt/images/darknetgw/upper/tmp/stage-perl/usr/lib/perl5/5.28/CORE/regexp.h

Multiple flock calls on the same filename deadlocks

When flock calls are made from multiple processes for the same filename fuse-overlayfs deadlocks.

This happens reliably when running multiple git clones or go get from a rootless container in Fedora Silverblue:

● ostree://fedora-workstation:fedora/30/x86_64/silverblue
                   Version: 30.20190530.0 (2019-05-30T00:39:27Z)
                BaseCommit: 8980028d1ac8f8180b29233be4704d0ff3c8e01d6833d149dc460919a63c39a7

Backtrace of hanging /usr/bin/fuse-overlayfs:

#0  0x00007f68d23a647b in flock () from target:/lib64/libc.so.6
#1  0x00000000004031be in ovl_flock ()
#2  0x00007f68d24b71ba in ?? () from target:/lib64/libfuse3.so.3
#3  0x00007f68d24b9908 in ?? () from target:/lib64/libfuse3.so.3
#4  0x00007f68d24b4c2b in fuse_session_loop () from target:/lib64/libfuse3.so.3
#5  0x0000000000402c56 in main ()

It's probably related to the following section in the flock(2) manpage:

If a process uses open(2) (or similar) to obtain more than one file descriptor for the same file, these file descriptors are treated independently by flock(). An attempt to lock the file using one of these file descriptors may be denied by a lock that the calling process has already placed via another file descriptor.

Maybe related to #67

How to reproduce

Running the following program in a rootless container concurrently hangs that container.

fuse-overlay-flock.c

#include <sys/file.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char* argv[])
{
	int fd;
	fd = open(argv[1], O_CREAT|O_WRONLY, 0644);
	flock(fd, LOCK_EX);
	dprintf(fd, "foo\n");
	flock(fd, LOCK_UN);
	close(fd);
	printf("done\n");
	return 0;
}

Dockerfile:

FROM registry.fedoraproject.org/fedora:30
RUN dnf install -y gcc
COPY fuse-overlay-flock.c /
RUN gcc -o fuse-overlay-flock fuse-overlay-flock.c
ENTRYPOINT [ "/usr/bin/bash" ]
CMD [ "-c", "for i in $(seq 1 10); do ( /fuse-overlay-flock testfile & ); done" ]

Command:

podman build -t localhost/fuse-overlay-bug:latest
podman run --rm -it localhost/fuse-overlay-bug:latest

Actual result

$ podman run --rm -it localhost/fuse-overlay-bug:latest
done

The container does not print the other 'done' messages and never exits. It will only stop when the relevant fuse-overlayfs process is killed.

Expected result

$ podman run --rm -it localhost/fuse-overlay-bug:latest
done
done
done
done
done
done
done
done
done
done
$

Static build fails

<3 project.

Static build fails with:

 ---> Running in 5ce3b3418daf
Cloning into 'libfuse'...
The Meson build system
Version: 0.50.0
Source dir: /build/libfuse
Build dir: /build/libfuse/build
Build type: native build
Project name: libfuse3
Project version: 3.5.0
Appending LDFLAGS from environment: '-lpthread'
Appending LDFLAGS from environment: '-lpthread'
Native C compiler: cc (gcc 8.3.1 "cc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)")

meson.build:1:0: ERROR: Unknown compiler(s): ['c++', 'g++', 'clang++', 'pgc++']
The follow exceptions were encountered:
Running "c++ --version" gave "[Errno 2] No such file or directory: 'c++': 'c++'"
Running "g++ --version" gave "[Errno 2] No such file or directory: 'g++': 'g++'"
Running "clang++ --version" gave "[Errno 2] No such file or directory: 'clang++': 'clang++'"
Running "pgc++ --version" gave "[Errno 2] No such file or directory: 'pgc++': 'pgc++'"

A full log can be found at /build/libfuse/build/meson-logs/meson-log.txt
The command '/bin/sh -c git clone https://github.com/libfuse/libfuse &&     cd libfuse &&     mkdir build &&     cd build &&     LDFLAGS="-lpthread" meson --prefix /usr -D default_library=static .. &&     ninja &&     ninja install' returned a non-zero code: 1

moving to rootless-containers

Would you be interested in moving this project to https://github.com/rootless-containers? This project is going to be one of the recommended deployments of rootless containers (on new enough kernels) and given that it's primary use is for rootless containers it seems a bit odd to have it in https://github.com/containers (given that the project mostly hosts cri-o-related projects).

WDYT @giuseppe? (Obviously you'd still have the same admin rights for rootless-containers as you have here -- there would be no practical difference from that standpoint).

static link option

for interesting portability use-cases, it would be nice to statically link fuse-overlayfs, but this would require a libfuse.a which many distros do not provide.
(I was building here with LDFLAGS="-static" ./configure && make)

"warning given when extracting ... (Can't restore time)"

This is hard to test but I'm pretty sure this wasn't present until I upgraded to 0.6.3-2.0.dev.git46c0f8e from Fedora 30 testing vs the stable 0.6.2-2.git67a4afe. When I try to install anything onto an archlinux/base image running in podman (e.g. pacman -Syu):

:: Processing package changes...
( 1/37) upgrading tzdata                                                [########################################] 100%
warning: warning given when extracting /usr/share/zoneinfo-posix (Can't restore time)
( 2/37) upgrading iana-etc                                              [########################################] 100%
( 3/37) upgrading bash                                                  [########################################] 100%
warning: warning given when extracting /usr/bin/sh (Can't restore time)
( 4/37) upgrading openssl                                               [########################################] 100%
warning: warning given when extracting /etc/ssl/misc/tsget (Can't restore time)
warning: warning given when extracting /usr/lib/libcrypto.so (Can't restore time)
warning: warning given when extracting /usr/lib/libssl.so (Can't restore time)
warning: warning given when extracting /usr/share/man/man1/c_rehash.1ssl.gz (Can't restore time)
warning: warning given when extracting /usr/share/man/man1/openssl-asn1parse.1ssl.gz (Can't restore time)
warning: warning given when extracting /usr/share/man/man1/openssl-c_rehash.1ssl.gz (Can't restore time)
warning: warning given when extracting /usr/share/man/man1/openssl-ca.1ssl.gz (Can't restore time)

This happens for every single file.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.