Traceback (most recent call last):
  File "./repos", line 11, in <module>
    load_entry_point('conan', 'console_scripts', 'repos')()
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jqbx34/tools/conan-develop/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/home/jqbx34/tools/conan-develop/src/conan/cli.py", line 198, in conanv11
    registry.populate(loc)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 136, in populate
    repo.load_manifest(rd)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 247, in load_manifest
    image = Image.load_image_config(config_file)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 473, in load_image_config
    config, warns = merge_configs(ccnf, cnf)
  File "/home/jqbx34/tools/conan-develop/src/conan/image_v11.py", line 528, in merge_configs
    return merge_update_mappings(container_config, config, mapping=OrderedDict)
  File "/home/jqbx34/tools/conan-develop/src/conan/utils.py", line 198, in merge_update_mappings
    assert isinstance(value2, Mapping)
AssertionError

AssertionError as value2 returned None

When running the tests from the PyPI tarball, on openSUSE Tumbleweed, I get two failures looking for ubuntu-artful.txt-distro-expected.json in https://github.com/nexB/container-inspector/tree/main/tests/data/distro/os-release/ubuntu , but it doesnt exist.

This occurs irrespective of whether I add /etc/os-release from https://software.opensuse.org/package/openSUSE-release to the build VM.

[   32s] =================================== FAILURES ===================================
[   32s] _________________ TestDistro.test_distro_from_os_release_file __________________
[   32s] 
[   32s] self = <test_distro.TestDistro testMethod=test_distro_from_os_release_file>
[   32s] 
[   32s]     def test_distro_from_os_release_file(self):
[   32s]         test_dir = self.get_test_loc('distro/os-release')
[   32s]     
[   32s]         for test_file in resource_iter(test_dir, with_dirs=False):
[   32s]             if test_file.endswith('-expected.json'):
[   32s]                 continue
[   32s]             expected = test_file + '-distro-expected.json'
[   32s]             result = Distro.from_os_release_file(test_file).to_dict()
[   32s] >           check_expected(result, expected, regen=False)
[   32s] 
[   32s] tests/test_distro.py:41: 
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] 
[   32s] result = {'architecture': None, 'bug_report_url': 'https://bugs.launchpad.net/ubuntu/', 'build_id': None, 'cpe_name': None, ...}
[   32s] expected = '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-distro-expected.json'
[   32s] regen = False
[   32s] 
[   32s]     def check_expected(result, expected, regen=False):
[   32s]         """
[   32s]         Check equality between a result collection and an expected JSON file.
[   32s]         Regen the expected file if regen is True.
[   32s]         """
[   32s]         if regen:
[   32s]             with open(expected, 'w') as ex:
[   32s]                 ex.write(json.dumps(result, indent=2))
[   32s]     
[   32s] >       with open(expected) as ex:
[   32s] E       FileNotFoundError: [Errno 2] No such file or directory: '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-distro-expected.json'
[   32s] 
[   32s] tests/utilities.py:21: FileNotFoundError
[   32s] _______________________ TestDistro.test_parse_os_release _______________________
[   32s] 
[   32s] self = <test_distro.TestDistro testMethod=test_parse_os_release>
[   32s] 
[   32s]     def test_parse_os_release(self):
[   32s]         test_dir = self.get_test_loc('distro/os-release')
[   32s]     
[   32s]         for test_file in resource_iter(test_dir, with_dirs=False):
[   32s]             if test_file.endswith('expected.json'):
[   32s]                 continue
[   32s]             expected = test_file + '-expected.json'
[   32s]             result = parse_os_release(test_file)
[   32s] >           check_expected(result, expected, regen=False)
[   32s] 
[   32s] tests/test_distro.py:31: 
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] 
[   32s] result = {'BUG_REPORT_URL': 'https://bugs.launchpad.net/ubuntu/', 'HOME_URL': 'https://www.ubuntu.com/', 'ID': 'ubuntu', 'ID_LIKE': 'debian', ...}
[   32s] expected = '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-expected.json'
[   32s] regen = False
[   32s] 
[   32s]     def check_expected(result, expected, regen=False):
[   32s]         """
[   32s]         Check equality between a result collection and an expected JSON file.
[   32s]         Regen the expected file if regen is True.
[   32s]         """
[   32s]         if regen:
[   32s]             with open(expected, 'w') as ex:
[   32s]                 ex.write(json.dumps(result, indent=2))
[   32s]     
[   32s] >       with open(expected) as ex:
[   32s] E       FileNotFoundError: [Errno 2] No such file or directory: '/home/abuild/rpmbuild/BUILD/container-inspector-32.0.1/tests/data/distro/os-release/ubuntu/ubuntu-artful.txt-expected.json'
[   32s] 
[   32s] tests/utilities.py:21: FileNotFoundError

Skopeo's https://github.com/containers/skopeo docker-archive: format is not exactly the docker save format and uses links for layer.tar tarballs ... See https://github.com/containers/skopeo/blob/cc3ddf4804995ab69606885c83a9990b84d1de36/docs/skopeo.1.md#L38

OCI images should be supported too. See https://github.com/containers/image/blob/56ddc22a8b0b1cef6fc1c5f321285fbde7f4274b/docs/containers-transports.5.md#oci-archivepathtag and https://github.com/opencontainers/image-spec/blob/daf51542d53813621913dde9bf438bd9c1fe32c8/spec.md

See also nexB/scancode.io#163

To enhance the reporting capabilities of Conan and the rpm inventory it generates, it would be helpful to collect the SHA1 checksums of these rpm files and output them to the csv.

This may be a separate set of commands that needs to be run on the container, or the rpm utility may have some of this functionality.

For each of the major distro/package managers (rpm, dep, apk, etc) collect not only the installed packages but also the set of files installed by these.

Only scans dockerfiles with name Dockerfile.

Doesn't scan dockerfiles like container-inspector.dockerfile.

This is badly missing for now.

These can be added as LABEL from a Dockerfile https://github.com/Tekki/docker-perl-test/blob/e52c397f9daf04ca89c416b76b610da5ae1cc15a/p16/Dockerfile#L24 and were initially specified in http://label-schema.org/rc1/ and are now specified at https://github.com/opencontainers/image-spec/blob/master/annotations.md

Some key/values are predefined:
https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys :

• org.opencontainers.image.created: date/timestamp
• org.opencontainers.image.authors: contact details of the people or organization responsible for the image (freeform string)
• org.opencontainers.image.url: homepage URL
• org.opencontainers.image.documentation: documentation URL
• org.opencontainers.image.source: VCS/download URL
• org.opencontainers.image.version: version ideally semver and mapped to a VCS repo tag
• org.opencontainers.image.revision: VCS revision e.g. git commit in org.opencontainers.image.source
• org.opencontainers.image.vendor: author (e.g. created_by)
• org.opencontainers.image.licenses: Should be an SPDX License Expression
• org.opencontainers.image.ref.name: unclear... "Name of the reference for a target"
• org.opencontainers.image.title: brief summary or "name" of the image?
• org.opencontainers.image.description: long description

These are some examples:

• json.txt
• config.json.zip

Right now we are using tarfile directly to keep symlinks. The story on symlinks extractions vs. empty is not entirely clear.

There is also some concerns about how scancode.io uses image extraction. See nexB/scancode.io#251

We should revisit extraction across the board to consider using extractcode

See also nexB/extractcode#6 for links.

From nexB/scancode.io#471

While scanning docker://quay.io/wire/alpine-deps I get this :

_extract_tar: skipping link with missing target: <TarInfo 'bin/arch' at 0x7ffa99193d00>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)
File "/usr/local/lib/python3.9/tarfile.py", line 2418, in _find_link_target raise KeyError("linkname %r not found" % linkname) KeyError: "linkname 'bin//bin/busybox' not found" _extract_tar: skipping link with missing target: <TarInfo 'bin/ash' at 0x7ffa99193dc0>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)
File "/usr/local/lib/python3.9/tarfile.py", line 2418, in _find_link_target raise KeyError("linkname %r not found" % linkname) KeyError: "linkname 'bin//bin/busybox' not found" _extract_tar: skipping link with missing target: <TarInfo 'bin/base64' at 0x7ffa99193b80>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)
File "/usr/local/lib/python3.9/tarfile.py", line 2418, in _find_link_target raise KeyError("linkname %r not found" % linkname) KeyError: "linkname 'bin//bin/busybox' not found" _extract_tar: skipping link with missing target: <TarInfo 'bin/bbconfig' at 0x7ffa99193e80>: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/container_inspector/utils.py", line 120, in extract_tar target = tarball._find_link_target(tarinfo)

See http://label-schema.org/ and an example here https://github.com/CentOS/sig-cloud-instance-images/blob/45cccc8f40650e860852cfe9054eefa5a4c0dc54/docker/Dockerfile

Those changes e4f4472
introduced the need to have manifest.json + repositories in the root to be properly detected as Docker but the repositories is not always present.

Procedures:

• Announce on Gitter/Element, in a specific channel if available, to notify of the org migration process including the scheduled start time and the expected completion time.
• Follow the following procedures instruction for the org migration process
  • 1. Transfer the nexB repo to the aboutcode-org organization from Settings -> Danger Zone -> Transfer ownership
  • 2. Set up CI for that repo on Azure Devops
    • Log into the nexB organization on Azure Devops
    • Go to the project that was moved
    • On the left hand menu, click on Pipelines, and then delete the existing pipeline
    • On the same page, click New pipeline, and then click Github, then select the repo that was transferred to aboutcode-org
    • Complete adding the pipeline
• Announce on Gitter/Element, in a specific channel if available, to notify the completion of the org migration.
• Update org-level permissions: Check org-level permissions for the source repo and update the repo-level permissions on the migrated repo
• Check if there are any old references with the previous org (documentation, reference links) that need to be updated.
• Push a new release about the org change with the updated references

Resource for a directory is created with is_file = True, instead of is_file = False
https://github.com/nexB/container-inspector/blob/3c799050f1a00161f468265127ccc533c983b869/src/container_inspector/image.py#L1038

Instead extract and rebuild a rootfs and collect package on this rootfs without running this in a rootfs

This /usr/lib/os-release

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

was not detected correctly in rootfs pipeline.

1. we should ensure that we can get to this path too
2. we should be more flexible with the paths (and leading segments)
3. in all cases, checking for system-installed packages should not be dependent on the detected distro. We should try Debian, RPM, etc. and all supported types in all FS and/or layers analyzed in a scancode.io pipeline (which impacts the code here, in scancode-toolkit and in scancode.io)

See https://github.com/opencontainers/image-spec/blob/master/layer.md#opaque-whiteout

Beyond collecting the presence of a package as installed (and get the basic name/version) we should also collect the corresponding metadata

I was looking around for a way to inspect (docker) container images on android (using termux, without root) and discovered this tool.

initially i ran into funny issues with intbitset due to python 3.11 (inveniosoftware-contrib/intbitset#81), but if you update intbitset to 3.0.2, the tool runs quite fine in termux! :)

unfortunately this tool cannot download images, but for that i used https://github.com/NotGlop/docker-drag.