OS: Fedora 37 (yes, I know..)

Expected behavior: Podman brings up containers like magic and everything works

What happens:
Followed instructions, podman brings up pod

Mimir pod: runs
Everything else: fails

output of podman pod log grafana-stack-pod-0 gives this repetitively:

3067b393d981 failed parsing config: failed to parse configFile /mnt/config/tempo-local-config.yaml: yaml: unmarshal errors: 3067b393d981 line 3: field metrics_generator_enabled not found in type app.Config b1a53055dacc GF_PATHS_DATA='/var/lib/grafana' is not writable. b1a53055dacc You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later f21e8af3ef40 open /tmp/loki/wal/00000002: permission denied 3067b393d981 line 52: field index_downsample_bytes not found in type common.BlockConfig f21e8af3ef40 error initialising module: ingester b1a53055dacc mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied f21e8af3ef40 github.com/grafana/dskit/modules.(*Manager).initModule 3067b393d981 line 53: field encoding not found in type common.BlockConfig f21e8af3ef40 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122 3067b393d981 line 56: field encoding not found in type wal.Config f21e8af3ef40 github.com/grafana/dskit/modules.(*Manager).InitModuleServices f21e8af3ef40 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 b1a53055dacc GF_PATHS_DATA='/var/lib/grafana' is not writable. b1a53055dacc You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
and

f21e8af3ef40 main.main f21e8af3ef40 /src/loki/cmd/loki/main.go:105 f21e8af3ef40 runtime.main f21e8af3ef40 /usr/local/go/src/runtime/proc.go:250 f21e8af3ef40 runtime.goexit f21e8af3ef40 /usr/local/go/src/runtime/asm_amd64.s:1594 f21e8af3ef40 open /tmp/loki/wal/00000002: permission denied f21e8af3ef40 error initialising module: ingester f21e8af3ef40 github.com/grafana/dskit/modules.(*Manager).initModule f21e8af3ef40 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122 f21e8af3ef40 github.com/grafana/dskit/modules.(*Manager).InitModuleServices f21e8af3ef40 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 f21e8af3ef40 github.com/grafana/loki/pkg/loki.(*Loki).Run f21e8af3ef40 /src/loki/pkg/loki/loki.go:422 f21e8af3ef40 main.main f21e8af3ef40 /src/loki/cmd/loki/main.go:105 f21e8af3ef40 runtime.main f21e8af3ef40 /usr/local/go/src/runtime/proc.go:250 f21e8af3ef40 runtime.goexit f21e8af3ef40 /usr/local/go/src/runtime/asm_amd64.s:1594 f21e8af3ef40 open /tmp/loki/wal/00000002: permission denied f21e8af3ef40 error initialising module: ingester f21e8af3ef40 github.com/grafana/dskit/modules.(*Manager).initModule f21e8af3ef40 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122 f21e8af3ef40 github.com/grafana/dskit/modules.(*Manager).InitModuleServices f21e8af3ef40 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 f21e8af3ef40 github.com/grafana/loki/pkg/loki.(*Loki).Run f21e8af3ef40 /src/loki/pkg/loki/loki.go:422

There seems to be some expectation on the part of the other containers needing permissions to external directories. This isn't great, since as I understand it, Podman is supposed to allow containers to run without root privileges. This is a core part of its security model.

I did try running it under an account with privileges on my test machine, however under those conditions tempo failed to download and start at all, the other containers did come up.

I'm at a bit of a loss as I still don't fully understand the relationship between containers and the hosts filesystem, and finding information on how to troubleshoot them is proving quite difficult.

Hi, thanks for the info, however my install pretty much fails at the first step after pulling down the git files:-

[mike@oracle-nginx grafana-stack-podman]$ podman play kube grafana-stack.yaml Error: grafana-stack.yaml: error encountered while bringing up pod grafana-stack-pod-0: building local pause image: error building at STEP "COPY /usr/libexec/podman/catatonit /catatonit": error storing "/usr/libexec/podman/catatonit": error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/home/mike/.local/share/containers/storage/overlay/e8789f080a5a923fc6251f111a74c5cde33bdc5ce6c9378afcea3e6f4d58e97b/merged", rootPrefix:"/home/mike/.local/share/containers/storage/overlay/e8789f080a5a923fc6251f111a74c5cde33bdc5ce6c9378afcea3e6f4d58e97b/merged", Directory:"/", preservedDirectory:"/home/mike/.local/share/containers/storage/overlay/e8789f080a5a923fc6251f111a74c5cde33bdc5ce6c9378afcea3e6f4d58e97b/merged", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap{}, GIDMap:[]idtools.IDMap{}, DefaultDirOwner:(*idtools.IDPair)(0xc0006b5fe0), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:true, NoOverwriteDirNonDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting extended attributes on "/catatonit": error setting value of extended attribute "security.ima" on "/catatonit": operation not permitted [mike@oracle-nginx grafana-stack-podman]$

This is on a Centos 9 install ?