nexb / container-inspector Goto Github PK

For each of the major distro/package managers (rpm, dep, apk, etc) collect not only the installed packages but also the set of files installed by these.

Right now we are using tarfile directly to keep symlinks. The story on symlinks extractions vs. empty is not entirely clear.

There is also some concerns about how scancode.io uses image extraction. See nexB/scancode.io#251

We should revisit extraction across the board to consider using extractcode

See also nexB/extractcode#6 for links.

Traceback (most recent call last):
  File "./repos", line 11, in <module>
    load_entry_point('conan', 'console_scripts', 'repos')()
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/home/jqbx34/tools/conan-develop/src/conan/cli.py", line 198, in conanv11
    registry.populate(loc)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 136, in populate
    repo.load_manifest(rd)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 247, in load_manifest
    image = Image.load_image_config(config_file)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 473, in load_image_config
    config, warns = merge_configs(ccnf, cnf)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 528, in merge_configs
    return merge_update_mappings(container_config, config, mapping=OrderedDict)
  File "/home/jqbx34/tools/conan-develop/src/conan/utils.py", line 198, in merge_update_mappings
    assert isinstance(value2, Mapping)
AssertionError

AssertionError as value2 returned None

Beyond collecting the presence of a package as installed (and get the basic name/version) we should also collect the corresponding metadata

These can be added as LABEL from a Dockerfile https://github.com/Tekki/docker-perl-test/blob/e52c397f9daf04ca89c416b76b610da5ae1cc15a/p16/Dockerfile#L24 and were initially specified in http://label-schema.org/rc1/ and are now specified at https://github.com/opencontainers/image-spec/blob/master/annotations.md

Some key/values are predefined:
https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys :

• org.opencontainers.image.created: date/timestamp
• org.opencontainers.image.authors: contact details of the people or organization responsible for the image (freeform string)
• org.opencontainers.image.url: homepage URL
• org.opencontainers.image.documentation: documentation URL
• org.opencontainers.image.source: VCS/download URL
• org.opencontainers.image.version: version ideally semver and mapped to a VCS repo tag
• org.opencontainers.image.revision: VCS revision e.g. git commit in org.opencontainers.image.source
• org.opencontainers.image.vendor: author (e.g. created_by)
• org.opencontainers.image.licenses: Should be an SPDX License Expression
• org.opencontainers.image.ref.name: unclear... "Name of the reference for a target"
• org.opencontainers.image.title: brief summary or "name" of the image?
• org.opencontainers.image.description: long description

These are some examples:

• json.txt
• config.json.zip

Instead extract and rebuild a rootfs and collect package on this rootfs without running this in a rootfs

I was looking around for a way to inspect (docker) container images on android (using termux, without root) and discovered this tool.

initially i ran into funny issues with intbitset due to python 3.11 (inveniosoftware-contrib/intbitset#81), but if you update intbitset to 3.0.2, the tool runs quite fine in termux! :)

unfortunately this tool cannot download images, but for that i used https://github.com/NotGlop/docker-drag.

Skopeo's https://github.com/containers/skopeo docker-archive: format is not exactly the docker save format and uses links for layer.tar tarballs ... See https://github.com/containers/skopeo/blob/cc3ddf4804995ab69606885c83a9990b84d1de36/docs/skopeo.1.md#L38

OCI images should be supported too. See https://github.com/containers/image/blob/56ddc22a8b0b1cef6fc1c5f321285fbde7f4274b/docs/containers-transports.5.md#oci-archivepathtag and https://github.com/opencontainers/image-spec/blob/daf51542d53813621913dde9bf438bd9c1fe32c8/spec.md

When running the tests from the PyPI tarball, on openSUSE Tumbleweed, I get two failures looking for ubuntu-artful.txt-distro-expected.json in https://github.com/nexB/container-inspector/tree/main/tests/data/distro/os-release/ubuntu , but it doesnt exist.

This occurs irrespective of whether I add /etc/os-release from https://software.opensuse.org/package/openSUSE-release to the build VM.

[   32s] =================================== FAILURES ===================================
[   32s] _________________ TestDistro.test_distro_from_os_release_file __________________
[   32s] 
[   32s] self = <test_distro.TestDistro testMethod=test_distro_from_os_release_file>
[   32s] 
[   32s]     def test_distro_from_os_release_file(self):
[   32s]         test_dir = self.get_test_loc('distro/os-release')
[   32s]     
[   32s]         for test_file in resource_iter(test_dir, with_dirs=False):
[   32s]             if test_file.endswith('-expected.json'):
[   32s]                 continue
[   32s]             expected = test_file + '-distro-expected.json'
[   32s]             result = Distro.from_os_release_file(test_file).to_dict()
[   32s] >           check_expected(result, expected, regen=False)
[   32s] 
[   32s] tests/test_distro.py:41: 
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] 
[   32s] result = {'architecture': None, 'bug_report_url': 'https://bugs.launchpad.net/ubuntu/', 'build_id': None, 'cpe_name': None, ...}
[   32s] expected = '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-distro-expected.json'
[   32s] regen = False
[   32s] 
[   32s]     def check_expected(result, expected, regen=False):
[   32s]         """
[   32s]         Check equality between a result collection and an expected JSON file.
[   32s]         Regen the expected file if regen is True.
[   32s]         """
[   32s]         if regen:
[   32s]             with open(expected, 'w') as ex:
[   32s]                 ex.write(json.dumps(result, indent=2))
[   32s]     
[   32s] >       with open(expected) as ex:
[   32s] E       FileNotFoundError: [Errno 2] No such file or directory: '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-distro-expected.json'
[   32s] 
[   32s] tests/utilities.py:21: FileNotFoundError
[   32s] _______________________ TestDistro.test_parse_os_release _______________________
[   32s] 
[   32s] self = <test_distro.TestDistro testMethod=test_parse_os_release>
[   32s] 
[   32s]     def test_parse_os_release(self):
[   32s]         test_dir = self.get_test_loc('distro/os-release')
[   32s]     
[   32s]         for test_file in resource_iter(test_dir, with_dirs=False):
[   32s]             if test_file.endswith('expected.json'):
[   32s]                 continue
[   32s]             expected = test_file + '-expected.json'
[   32s]             result = parse_os_release(test_file)
[   32s] >           check_expected(result, expected, regen=False)
[   32s] 
[   32s] tests/test_distro.py:31: 
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] 
[   32s] result = {'BUG_REPORT_URL': 'https://bugs.launchpad.net/ubuntu/', 'HOME_URL': 'https://www.ubuntu.com/', 'ID': 'ubuntu', 'ID_LIKE': 'debian', ...}
[   32s] expected = '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-expected.json'
[   32s] regen = False
[   32s] 
[   32s]     def check_expected(result, expected, regen=False):
[   32s]         """
[   32s]         Check equality between a result collection and an expected JSON file.
[   32s]         Regen the expected file if regen is True.
[   32s]         """
[   32s]         if regen:
[   32s]             with open(expected, 'w') as ex:
[   32s]                 ex.write(json.dumps(result, indent=2))
[   32s]     
[   32s] >       with open(expected) as ex:
[   32s] E       FileNotFoundError: [Errno 2] No such file or directory: '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-expected.json'
[   32s] 
[   32s] tests/utilities.py:21: FileNotFoundError

Those changes e4f4472
introduced the need to have manifest.json + repositories in the root to be properly detected as Docker but the repositories is not always present.

Only scans dockerfiles with name Dockerfile.

Doesn't scan dockerfiles like container-inspector.dockerfile.

From nexB/scancode.io#471

While scanning docker://quay.io/wire/alpine-deps I get this :

_extract_tar: skipping link with missing target: <TarInfo 'bin/arch' at 0x7ffa99193d00>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)
File "/usr/local/lib/python3.9/tarfile.py", line 2418, in _find_link_target raise KeyError("linkname %r not found" % linkname) KeyError: "linkname 'bin//bin/busybox' not found" _extract_tar: skipping link with missing target: <TarInfo 'bin/ash' at 0x7ffa99193dc0>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)
File "/usr/local/lib/python3.9/tarfile.py", line 2418, in _find_link_target raise KeyError("linkname %r not found" % linkname) KeyError: "linkname 'bin//bin/busybox' not found" _extract_tar: skipping link with missing target: <TarInfo 'bin/base64' at 0x7ffa99193b80>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)
File "/usr/local/lib/python3.9/tarfile.py", line 2418, in _find_link_target raise KeyError("linkname %r not found" % linkname) KeyError: "linkname 'bin//bin/busybox' not found" _extract_tar: skipping link with missing target: <TarInfo 'bin/bbconfig' at 0x7ffa99193e80>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)

To enhance the reporting capabilities of Conan and the rpm inventory it generates, it would be helpful to collect the SHA1 checksums of these rpm files and output them to the csv.

This may be a separate set of commands that needs to be run on the container, or the rpm utility may have some of this functionality.

See http://label-schema.org/ and an example here https://github.com/CentOS/sig-cloud-instance-images/blob/45cccc8f40650e860852cfe9054eefa5a4c0dc54/docker/Dockerfile

This /usr/lib/os-release

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

was not detected correctly in rootfs pipeline.

1. we should ensure that we can get to this path too
2. we should be more flexible with the paths (and leading segments)
3. in all cases, checking for system-installed packages should not be dependent on the detected distro. We should try Debian, RPM, etc. and all supported types in all FS and/or layers analyzed in a scancode.io pipeline (which impacts the code here, in scancode-toolkit and in scancode.io)

This is badly missing for now.

See https://github.com/opencontainers/image-spec/blob/master/layer.md#opaque-whiteout

See also nexB/scancode.io#163

Resource for a directory is created with is_file = True, instead of is_file = False