Distro tests fail within nixos about grype HOT 11 OPEN

I think this issue has the same root cause as anchore/syft#2894, which is more obviously a bug in Syft.

from grype.

Hi @JordanFaust,

Would you mind providing more detail in your repro steps? I have never used Nix before, and I don't know how to get into a situation where I can run nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype. Can you provide a Dockerfile or set of steps that can take me from "this machine has never run nix before at all" to "I can watch this error happen?" I'd prefer a Dockerfile specifically if that's possible.

I tried the following:

1. Run docker run -ti ghcr.io/nixos/nix - this puts me at a shell in Nix
2. In the container run nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype

This fails with so error: experimental Nix feature 'nix-command' is disabled; add '--extra-experimental-features nix-command' to enable it. I've tried a few things, but I haven't gotten as far as the error you're seeing.

Thanks!

from grype.

Unfortunately it looks like this is a specific issue for NixOS and not nix the package manager. The above runs a container with a different OS and just has the nix CLI. I'm not sure the best way to test that in a container. I'm not sure what the specific issue is but something about these tests are failing when being built within NixOS. If you have any ideas on what could be the source I can try it out myself and try building from there.

from grype.

I wasn't able to reproduce this issue:

FROM ubuntu:22.04

# install prerequisites
RUN apt-get update && apt-get install -y curl sudo xz-utils

# create a non-root user (recommended for running Nix)
RUN useradd -m nixuser && echo 'nixuser ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

# switch to the non-root user
USER nixuser
WORKDIR /home/nixuser

# install Nix
RUN curl -L https://nixos.org/nix/install | sh

# set up Nix environment
ENV USER nixuser
ENV HOME /home/nixuser
ENV PATH /home/nixuser/.nix-profile/bin:/home/nixuser/.nix-profile/sbin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/run/current-system/sw/bin:/home/nixuser/.local/bin:$PATH
RUN . /home/nixuser/.nix-profile/etc/profile.d/nix.sh

# enable experimental features in Nix configuration
RUN mkdir -p /home/nixuser/.config/nix
RUN echo "experimental-features = nix-command flakes" >> /home/nixuser/.config/nix/nix.conf

Then:

$ docker build -t localhost/nix:latest .
$ docker run --rm -it localhost/nix:latest

And from within the container, everything seems to work:

$ docker run --rm -it localhost/nix:latest
nixuser@78cbeeb35219:~$ nix profile install nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
nixuser@78cbeeb35219:~$ grype alpine:latest
 ✔ Vulnerability DB                [updated]
 ✔ Parsed image                                                                                                                        sha256:ace17d5d883e9ea5a21138d0608d60aa2376c68f616c55b0b7e73fba6d8556a3
 ✔ Cataloged contents                                                                                                                         a0264d60f80df12bc1e6dd98bae6c43debe6667c0ba482711f0d806493467a46
   ├── ✔ Packages                        [15 packages]
   ├── ✔ File digests                    [80 files]
   ├── ✔ File metadata                   [80 locations]
   └── ✔ Executables                     [17 executables]
 ✔ Scanned for vulnerabilities     [14 vulnerability matches]
   ├── by severity: 0 critical, 0 high, 12 medium, 0 low, 0 negligible (2 unknown)
   └── by status:   2 fixed, 12 not-fixed, 0 ignored
NAME           INSTALLED   FIXED-IN  TYPE  VULNERABILITY   SEVERITY
busybox        1.36.1-r15            apk   CVE-2023-42366  Medium
busybox        1.36.1-r15            apk   CVE-2023-42365  Medium
busybox        1.36.1-r15            apk   CVE-2023-42364  Medium
busybox        1.36.1-r15            apk   CVE-2023-42363  Medium
busybox-binsh  1.36.1-r15            apk   CVE-2023-42366  Medium
busybox-binsh  1.36.1-r15            apk   CVE-2023-42365  Medium
busybox-binsh  1.36.1-r15            apk   CVE-2023-42364  Medium
busybox-binsh  1.36.1-r15            apk   CVE-2023-42363  Medium
libcrypto3     3.1.4-r5    3.1.4-r6  apk   CVE-2024-2511   Unknown
libssl3        3.1.4-r5    3.1.4-r6  apk   CVE-2024-2511   Unknown
ssl_client     1.36.1-r15            apk   CVE-2023-42366  Medium
ssl_client     1.36.1-r15            apk   CVE-2023-42365  Medium
ssl_client     1.36.1-r15            apk   CVE-2023-42364  Medium
ssl_client     1.36.1-r15            apk   CVE-2023-42363  Medium
A newer version of grype is available for download: 0.77.4 (installed version is 0.76.0)

Are there other specific details that I'm missing from this setup?

from grype.

I don't believe there is an issue for nix the package manager. This specifically is an issue with NixOS and something specific about the nature of the distro tests that are failing within that hermetic build environment. I will need to dig into what the distro tests are actually doing to simulate the different distros. I'm assuming something in the test setup is causing issues within the nixos build environment. I can try and track this down if you can help me understand these tests a little better.

from grype.

Thanks @JordanFaust !

What the "distro" tests are doing is testing Grype's ability to detect what distro the image it's scanning is based on.

The tests are defined here:

My guess is that something in here is broken by the hermetic build environment.

The way the tests essentially work is:

1. There's a test fixture that looks the identifying files on the distro, for example https://github.com/anchore/grype/blob/main/grype/distro/test-fixtures/os/alpine/etc/os-release looks like /etc/os-release from an Alpine image.
2. The test makes a source and file resolver from this directory - the source and file resolver are grype's abstraction over file I/O.
3. The tests exercise Grype's distro identification code by passing it the resolver.

They're failing at require.NotNil(t, release, "empty linux release info"), which means that given the file resolver, Grype couldn't identify the distro. I suspect this is because something about the hermetic build prevents Grype from doing the necessary file I/O or something like that.

Creating the file resolver is happening over in Syft code here: https://github.com/anchore/syft/blob/2d318cffaaca07a8b3e87d68f80f881153a305b8/syft/source/directorysource/directory_source.go#L135

which calls: https://github.com/anchore/syft/blob/2d318cffaaca07a8b3e87d68f80f881153a305b8/syft/internal/fileresolver/directory.go#L40

I don't see anything in there that I think NixOS should be blocking - the chroot-named functions don't seem to be actually making a chroot system call, just returning a file resolver rooted outside the current directory.

Thanks for taking a look! Let me know if there's any other help I can be.

from grype.

Hello all, I help maintain the syft and grype packages in nixpkgs. I'm looking into this as part of our Zero Hydra (our ci system) Failures process for the next stable release.

https://hydra.nixos.org/build/259554448/nixlog/3
This is the tests running within the build sandbox.

I've tried just running go as normal on my NixOS machine and also get the issue
I have trimmed the logs as they're nothing new.

$ go test -v -run 'Test_NewDistroFromRelease_Coverage' ./grype/distro/...
=== RUN   Test_NewDistroFromRelease_Coverage
=== RUN   Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine
    distro_test.go:254:
        	Error Trace:	/home/jk/projects/personal/grype/grype/distro/distro_test.go:254
        	Error:      	Expected value not to be nil.
        	Test:       	Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine
        	Messages:   	empty linux release info
#...
=== RUN   Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
    distro_test.go:254:
        	Error Trace:	/home/jk/projects/personal/grype/grype/distro/distro_test.go:254
        	Error:      	Expected value not to be nil.
        	Test:       	Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
        	Messages:   	empty linux release info
=== NAME  Test_NewDistroFromRelease_Coverage
    distro_test.go:272:    defined: archlinux
#...
    distro_test.go:272:    defined: debian
    distro_test.go:277: distro coverage incomplete (defined=19, coverage=0)
--- FAIL: Test_NewDistroFromRelease_Coverage (0.01s)
    --- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/alpine (0.00s)
#...
    --- FAIL: Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard (0.00s)
FAIL
FAIL	github.com/anchore/grype/grype/distro	0.013s
FAIL

Interestingly in a container I don't have the issue

docker run -it nixos/nix nix develop --experimental-features "nix-command flakes" nixpkgs/92d295f588631b0db2da509f381b4fb1e74173c5#grype
Will put you in a shell with the dependencies used at build time

You can then run phases="unpackPhase patchPhase" genericBuild to get you into a situation where the source code is available in $PWD
Then you can run go test -v -run 'Test_NewDistroFromRelease_Coverage' ./grype/distro/...

And I'm not getting the issue inside the nixos/nix docker container.

Is there an easy way to make the logs visible so I can check the errors during the linux.IdentifyRelease() call? Just seeing nil doesn't narrow it down much.
If the logs aren't easily visible I'll probably make it return the error

from grype.

Thanks for taking a look @06kellyjac!

I haven't found an easy way to enable Grype's normal logging during a test run, so it might make sense to do a quick local change to return the error.

from grype.

Here's what's needed to log within tests (today... we can probably make this better):

import (
	"github.com/anchore/go-logger"
	"github.com/anchore/go-logger/adapter/discard"
	"github.com/anchore/go-logger/adapter/logrus"
	"github.com/anchore/grype/internal/log"
	"github.com/anchore/syft/syft"
	...
)
...
for _, test := range tests {
		t.Run(test.fixture, func(t *testing.T) {
			l, err := logrus.New(
				logrus.Config{
					EnableConsole: true,
					Level:         logger.TraceLevel,
				},
			)
			syft.SetLogger(l)
			log.Set(l)

			require.NoError(t, err)
			require.NotNil(t, l)

			// run your test...

			syft.SetLogger(discard.New())
			log.Set(discard.New())
		})
}

Another observation, the link to the logs doesn't seem to be running tests with -v, which means that stdout will be suppressed. So you'll need to make both changes to see the log output in your CI environment.

from grype.

Running locally with those changes:

I don't see anything super useful here from my understanding. Unless maybe it sees "ignoring /" and that ignores everything? But I wouldn't expect that as the test fixture dir is meant to be the root from that point right?

I don't see any of the error traces so unless the logging isn't set up right I guess it's not running into problems there

=== RUN   Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
[0000]  WARN no explicit name and version provided for directory source, deriving artifact ID from the given path (which is not ideal)
[0000] DEBUG ignoring system mountpoint mountpoint=/
[0000] DEBUG ignoring system mountpoint mountpoint=/run
[0000] DEBUG ignoring system mountpoint mountpoint=/dev
[0000] DEBUG ignoring system mountpoint mountpoint=/dev/shm
[0000] DEBUG ignoring system mountpoint mountpoint=/proc
[0000] DEBUG ignoring system mountpoint mountpoint=/run/wrappers
[0000] DEBUG ignoring system mountpoint mountpoint=/sys
[0000] DEBUG ignoring system mountpoint mountpoint=/run/user/1000
[0000] TRACE indexing filetree path=test-fixtures/os/chainguard
    distro_test.go:268:
        	Error Trace:	/home/jk/projects/personal/grype/grype/distro/distro_test.go:268
        	Error:      	Expected value not to be nil.
        	Test:       	Test_NewDistroFromRelease_Coverage/test-fixtures/os/chainguard
        	Messages:   	empty linux release info
=== NAME  Test_NewDistroFromRelease_Coverage

Just running locally in my case I've got a tmpfs on root. Not 100% sure for the CI.
There's a non-insignificant amount of NixOS users following this pattern to identify where their state is.

https://grahamc.com/blog/erase-your-darlings/
https://github.com/nix-community/impermanence

from grype.

[0000] DEBUG ignoring system mountpoint mountpoint=/

ha! Agreed, that one seems a tad impactful (that will ignore the entire filesystem I believe). Syft doesn't catalog within certain filesystem mounts, so it seems like there is something peculiar with the system mounts (or something that needs to be corrected in syft).

Can you confirm on a system that is exhibiting the bad behavior what the filesystem types for each mount? Would you be willing to post the output of df from that system? (or a portion of it?)

edit: I do see a hint from the links you posted that we know the answer already...

If you have a lot of RAM, you could skip the erase step and make / a tmpfs.

from grype.