What would you like to be added:

A lot of yardstick commands accept something like DESCRIPTION or ID, but it's not obvious what this string is. Example:

❯ yardstick label explore --help
Usage: yardstick label explore [OPTIONS] DESCRIPTION

  interact with an label results for a single image scan

Options:
  -y, --year-max-limit TEXT       max year to include in comparison (relative
                                  to the CVE ID)
  -c, --derive-year-from-cve-only
                                  only use the CVE ID year-max-limit
  --help                          Show this message and exit.

In this case, DESCRIPTION is really a result set UUID obtained by running yardstick result list and looking for the image+tool combo the user is interested in. But nothing about DESCRIPTION makes it obvious what to pass in our how to get it.

Why is this needed:

The use of generic parameter names like ID and DESCRIPTION in the docs makes it very hard to come up to speed in yardstick.

Additional context:

What happened:

Running git clone [email protected]:anchore/yardstick ytest && cd ytest && make bootstrap && poetry install && make all fails because it can't install an old enough grype on main.

What you expected to happen:

All the validation targets should pass on a clean checkout of main.

How to reproduce it (as minimally and precisely as possible):

git clone [email protected]:anchore/yardstick ytest && cd ytest && make bootstrap && poetry install && make all

Anything else we need to know?:

Hypothesis

This is likely because the requested version of grype, https://github.com/anchore/grype/releases/tag/v0.27.0, only has a zip for the darwin_arm64 image, but has a .dmg for darwin_amd64.

Later releases of grype, such as https://github.com/anchore/grype/releases/tag/v0.50.2, have a .tar.gz for both versions.

Stacktrace of failure:

one
[info] checking github for release tag='v0.27.0'
[info] fetching release script for tag='v0.27.0'
anchore/grype info checking GitHub for tag 'v0.27.0'
anchore/grype info found version: 0.27.0 for v0.27.0/darwin/arm64
Traceback (most recent call last):
  File "/private/tmp/ytest/tests/cli/venv/bin/yardstick", line 8, in <module>
    sys.exit(cli())
  File "/private/tmp/ytest/tests/cli/venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/private/tmp/ytest/tests/cli/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/private/tmp/ytest/tests/cli/venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/private/tmp/ytest/tests/cli/venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/private/tmp/ytest/tests/cli/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/private/tmp/ytest/tests/cli/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/private/tmp/ytest/tests/cli/venv/lib/python3.9/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/private/tmp/ytest/src/yardstick/cli/result.py", line 45, in capture_results
    capture.result_set(
  File "/private/tmp/ytest/src/yardstick/capture.py", line 164, in result_set
    scan_config = one(scan_request, producer_state=producer_data_path, profiles=profiles)
  File "/private/tmp/ytest/src/yardstick/capture.py", line 105, in one
    match_results, raw_json = run_scan(config=scan_config, profile=profile_obj)
  File "/private/tmp/ytest/src/yardstick/capture.py", line 32, in run_scan
    tool = tool_cls.install(version=config.tool_version, path=path, use_cache=not reinstall, **kwargs)
  File "/private/tmp/ytest/src/yardstick/tool/grype.py", line 207, in install
    tool_obj = cls._install_from_installer(version=version, path=path, use_cache=use_cache, **kwargs)
  File "/private/tmp/ytest/src/yardstick/tool/grype.py", line 65, in _install_from_installer
    os.chmod(f"{path}/grype", 0o755)
FileNotFoundError: [Errno 2] No such file or directory: '.yardstick/tools/grype/v0.27.0/grype'
Exited with 1
make[1]: *** [test] Error 1
make: *** [cli] Error 2

Environment:

• Output of yardstick version: yardstick 0.0.0 (using a dev build off main)
• OS (e.g: cat /etc/os-release or similar): M1 macbook pro

What would you like to be added:

When I run yardstick label explore <some-uuid> I should be able to quickly find any matches that are not labeled.

Why is this needed:

Otherwise, when labeling results for an image with a lot of matches, many of which are already labeled, searching for unlabeled matches involves a lot of additional scrolling.

Additional context:

What happened:

Running yardstick label remove 73bd6255-d663-4a07-8eca-a7555902898f from test/quality directory in Grype doesn't remove the label with that ID, but does log [INFO] removed 1 labels and exit zero

❯ yardstick label remove 73bd6255-d663-4a07-8eca-a7555902898f                                                                                                                               
/Users/willmurphy/work/grype1373/test/quality/venv/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
2023-07-26 10:06:59,182 [INFO] removed 1 labels
❯ echo $? 
0
❯ jq . ./vulnerability-match-labels/labels/registry.access.redhat.com+ubi8@sha256:68fecea0d255ee253acbf0c860eaebb7017ef5ef007c25bee9eeffd29ce85b29/73bd6255-d663-4a07-8eca-a7555902898f.json
{
  "ID": "73bd6255-d663-4a07-8eca-a7555902898f",
  "effective_cve": "CVE-2020-26137",
  "image": {
    "exact": "registry.access.redhat.com/ubi8@sha256:68fecea0d255ee253acbf0c860eaebb7017ef5ef007c25bee9eeffd29ce85b29"
  },
  "label": "TP",
  "package": {
    "name": "urllib3",
    "version": "1.24.2"
  },
  "timestamp": "2022-09-15T19:43:11-04:00",
  "user": "wagoodman",
  "vulnerability_id": "GHSA-wqvq-5m8c-6g24"
}

What you expected to happen:

I expected the label to be removed or, if something is wrong, the log of an error and a non-zero exit code.

How to reproduce it (as minimally and precisely as possible):

git clone [email protected]:anchore/grype --recurse-submodules /tmp/repro-yardstick-issue
cd /tmp/repro-yardstick-issue
cd ./test/quality
. venv/bin/activate
yardstick label remove 73bd6255-d663-4a07-8eca-a7555902898f
cd ./vulnerability-match-labels
git status

If the removal succeeded, the last git status should have printed the diff indicating that the label was removed, specifically the file at labels/registry.access.redhat.com+ubi8@sha256:68fecea0d255ee253acbf0c860eaebb7017ef5ef007c25bee9eeffd29ce85b29/73bd6255-d663-4a07-8eca-a7555902898f.json should have been deleted.

Anything else we need to know?:

Environment:

• Output of yardstick version:

❯ yardstick version
/Users/willmurphy/work/grype1373/test/quality/venv/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
yardstick 0.7.0 (/Users/willmurphy/work/grype1373/test/quality/venv/lib/python3.9/site-packages)

• OS (e.g: cat /etc/os-release or similar):

❯ uname -m -s -r -v
Darwin 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000 arm64

What would you like to be added:

Right now, yardstick can capture the difference between different tool outputs, but it has no notion of a label provider, and tools that compare yardstick results to known labels implement two mechanisms on their own:

1. A mechanism to fetch some labels to use in comparison. (grype uses git submodules for this mechanism.)
2. A mechanism to run a quality gate, for example so that too great a deviation from known labels fails a job in CI. Grype's gate.py is an example.

The core request is: a yardstick.yaml file should be able to configure two additional things: where do my labels come from? And what comparison methods and thresholds are considered success vs failure.

One way we've discussed this is: a label source and a comparator should be sort of like "tools" in yardstick's config model.

Why is this needed:

Right now, we're maintaining a fair amount of python code in different repos that fetches labels and runs different comparisons. All of these comparison codes fetch labels and run comparisons in similar, but not identical ways. We should promote this duplicated code into a new configurable section in yardstick.

Additional context:

Today you can find the same match more than one way, when labeling these matches there tends to be one match that is "most correct" and others that "are technically correct". For instance, what if in one version of grype we don't consider RPM epoch but in a future version we do consider the epoch... the label for the match with the epoch included is more correct, but that doesn't mean that the original label without the epoch is wrong.

The problem with this situation is that without choosing to label the match as either TP or FP then quality gates will start failing over time due to a high number of indeterminate matches. An indeterminate match is a match that is either:

• unlabeled
• has been explicitly labeled as unclear
• has multiple labels that conflict (TP and FP labels for example)

There are a couple paths forward here:

1. don't count unclear labels as indeterminate. The advantage is that gating can still pass, but if the behavior regresses and the pairing TP match is not made (but the unclear match is) then the F1 score goes down, which is ultimately a good behavior.
2. add the concept of a TP alias label, where the label can stand in as a TP for another label, but if it's missing it won't count as a FN. This has the advantage of allowing the F1 score to remain unchanged in either matching scenario (even if a match is "better" or "not as good" if at least one is found then it does not matter, nothing will be counted against you).

What would you like to be added:

Support for tools that are both SBOM generators and vulnerability scanners. Currently we could work around this by adding two unique entries for the tool, and this may be the easiest path forward for now, but would be nice to discuss if there is some design enhancement we can make for a single tool to support both.

What would you like to be added:

Right now, you can installed tools from a release or a pushed commit somewhere on GH. We should enable yardstick to "install" a tool by building it from a path.

Why is this needed:

Running make quality in grype has 2 unexpected behaviors right now:

1. If your local commit isn't pushed to a remote, it will fail
2. If you have uncommitted local changes, the tests will succeed, but will use grype built without those changes (because yardstick downloaded the last commit from GitHub.)

Number 2 could be especially frustrating to debug, and might surprise contributors.