[root@test-change module]# /root/.local/bin/repo2module -s stable . modules.yaml
Traceback (most recent call last):
File "/root/.local/bin/repo2module", line 11, in
load_entry_point('repo2module==0.1', 'console_scripts', 'repo2module')()
File "/root/.local/lib/python3.6/site-packages/repo2module-0.1-py3.6.egg/repo2module/cli.py", line 122, in cli
File "/root/.local/lib/python3.6/site-packages/repo2module-0.1-py3.6.egg/repo2module/cli.py", line 65, in get_source_packages
TypeError: 'NoneType' object is not subscriptable

I am making an issue as we discussed the module build workflow with @mcurlej at devconf.

Since PR #16,
we are depending on libmodulemd instead of python3-libmodulemd which is incorrect. Please see the discussion in PR #16 for the reasoning.

Once python3-libmodulemd is available for EPEL8 (should be available in EPEL8.3), we should fix our dependency.
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1860401

Could we somewhat cooperate on creating an appropriate mock configuration
for the created local repository? This is just a rough idea/brainstorm,
"this" can be implemented in Mock directly, or anywhere else.

Currently, I tried to call bld2repo (with Red hat internal Koji server)
and then edited an existing (EL8) mock configuration and I'm getting
various errors (not being able to reproduce the build eventually).

Currently I'm getting:

...
No available modular metadata for modular package 'xmvn-subst-....module+....', it cannot be installed on the system
No available modular metadata for modular package 'xz-java-...module+.....', it cannot be installed on the system
Error: No available modular metadata for modular package
``

Here's the patch:

diff --git a/dir2module/dir2module/dir2module.py b/dir2module/dir2module/dir2module.py
index 6643a83..e639bba 100755
--- a/dir2module/dir2module/dir2module.py
+++ b/dir2module/dir2module/dir2module.py
@@ -113,8 +113,8 @@ class Module(ModuleBase):
         for pkglicense in self.licenses:
             mod_stream.add_content_license(pkglicense)
 
-        for nevra in self.package_nevras:
-            mod_stream.add_rpm_artifact(nevra)
+#        for nevra in self.package_nevras:
+#            mod_stream.add_rpm_artifact(nevra)
 
         dependencies = Modulemd.Dependencies()
         for depname, depstream in self.requires.items():

Apparently nevras is an empty list, so add_rpm_artifact dies.

I'm hacking yes, but if the list is null, I don't need the code right?

Can you provide a proper fix?

We currently skip some tests that don't work on either RHEL 8 or Fedora. We should fix them.

Hi
I'm trying to use this repo and not sure what should be the way to utilise it in RHEL 8
I've tried yum install modulemd_tools but get Error: Unable to find a match: modulemd_tools

Should I build it from source or is there a way to pull it like any other package?

/cc @ppisar
Based on #53 re-factoring of the whole modulemd-tools package, I discovered some issues with modulemd_add_platform tool, see:

modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:77:17: F821 undefined name 'hexvalue'
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:211:24: F821 undefined name 'indent_configurations'
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:211:48: F821 undefined name 'indent_context'
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:214:43: F821 undefined name 'indent_configurations'
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:215:37: F821 undefined name 'indent_context'
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:220:25: F841 local variable 'this_context_is_old_platform' is assigned to but never used
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:238:25: F821 undefined name 'indent_configurations'
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:243:17: F841 local variable 'indent_context' is assigned to but never used
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:258:13: F841 local variable 'indent_configurations' is assigned to but never used
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:445:86: F821 undefined name 'e'
modulemd_tools/modulemd_add_platform/modulemd_add_platform.py:463:21: F524 '...'.format(...) is missing argument(s) for placeholder(s): 1

Because the module-build-macros SRPM is the central piece for building module package, it would be useful to have a simple tool, which would provide the module-build-macros SRPM out of modulemd YAML. Having the SRPM at hand, it would allow me to build modular packages by hand in Mock.

Basically, is should be some simple command line tool, which calls the KojiModuleBuilder.get_disttag_srpm

May be I should contact the fedpkg developers, because calling fedpkg srpm in directory with module YAML file could be interesting variant.

Originally requested here, but I figured this might be better place.

The modulemd-generate-macros script expects, that some mandatory attributes that are described as AUTOMATIC, are filled in the modulemd YAML file (such as name, stream, version, context, and arch). Otherwise, it fails, please see

https://github.com/rpm-software-management/modulemd-tools/tree/master/modulemd-generate-macros#wrong-modulemd-yaml-file

We can add a feature checking whether the passed modulemd YAML file is a part of a GIT repository and if yes, generating those values (well, at least some of them).

I have a question about install docker-ce in offline case:

there are two vms, one is online, another is not. I try to download the rpm of docker-ce in the online one and install it on the offline vm.

but met the following error:

[root@localhost repo]#  dnf module install foo:devel
Last metadata expiration check: 0:01:53 ago on Sun 23 Oct 2022 11:30:35 PM EDT.
Unable to resolve argument foo:devel
No match for package docker-ce
Unable to resolve argument foo:devel
No match for package docker-ce-cli
Error: Problems in request:
broken groups or modules: foo:devel, foo:devel

Step to reproduce

on the online one:

1. add docker-ce repo

sudo yum install -y yum-utils
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

2. download rpm

yum makecache
yum install --downloadonly --installroot=/var/rpm/repo-installroot \
	--releasever=8 --downloaddir=/var/rpm/repo docker-ce docker-ce-cli containerd.io docker-compose-plugin

3. create repo

createrepo --database /var/rpm/repo

add module feature to rpm downloaded, following the instruction Creating a module repository from a set of RPM packages in README.md :

[root@localhost repo]# cd /var/rpm/repo
[root@localhost repo]# dir2module foo:devel:123:f32:x86_64 -m "My example module" --force --dir .
ERROR: RPM does not have `modularitylabel` header set: ./libxkbcommon-0.9.1-1.el8.x86_64.rpm
ERROR: RPM does not have `modularitylabel` header set: ./xkeyboard-config-2.28-1.el8.noarch.rpm
ERROR: RPM does not have `modularitylabel` header set: ./acl-2.2.53-1.el8.1.x86_64.rpm
// a lot of same kind of error, omit for brevity
Created foo:devel:123:f32:x86_64.modulemd.yaml
Created foo:devel:123:f32:x86_64.modulemd-defaults.yaml


[root@localhost repo]# createrepo_mod .
Directory walk started
Directory walk done - 197 packages
Temporary output repo path: ./.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finished

[root@localhost repo]# ls repodata/ |grep modules
ce7589fb16229ae9f5c31f2ba47ce60748c4d098e94dd13efd964b3c6f5748c6-modules.yaml.gz

copy the yum repo to /etc/yum.repos.d/ dir:

[root@localhost repo]# cat /etc/yum.repos.d/docker-offline.repo
[docker-offline]
name=My local foo module
baseurl=file:///var/rpm/repo
gpgcheck=0
enabled=1

and check module info:

[root@localhost repo]# dnf module info foo:devel
My local foo module                                                                                          52 MB/s | 277 kB     00:00
Name             : foo
Stream           : devel [d][a]
Version          : 123
Context          : f32
Architecture     :
Profiles         : common [d]
Default profiles : common
Repo             : docker-offline
Summary          : My example module
Description      : This module has been generated using dir2module tool
Requires         :
Artifacts        : acl-0:2.2.53-1.el8.1.x86_64
                 : audit-libs-0:3.0.7-2.el8.2.x86_64
                 : basesystem-0:11-5.el8.noarch
                 : bash-0:4.4.20-4.el8_6.x86_64
                 // a lot of list, no error at the end
[root@localhost repo]# ls |grep modulemd
foo:devel:123:f32:x86_64.modulemd-defaults.yaml
foo:devel:123:f32:x86_64.modulemd.yaml
[root@localhost repo]# pwd
/var/rpm/repo

4. install the docker on the offline vm (I disconnect the internet at this point of time)

the error I can't figure out:

[root@localhost repo]#  dnf module install foo:devel
Last metadata expiration check: 0:01:22 ago on Sun 23 Oct 2022 11:30:35 PM EDT.
Unable to resolve argument foo:devel
No match for package docker-ce
Unable to resolve argument foo:devel
No match for package docker-ce-cli
Error: Problems in request:
broken groups or modules: foo:devel, foo:devel

python2-six-1.11.0-6.module+el8.4.0+9287+299307c7 needs python2-libs-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm for build but the following command does not download it:

$ bld2repo --build-id 1444241 \
    --koji-host "https://brewhub.engineering.redhat.com/brewhub" \
    --koji-storage-host "http://download.devel.redhat.com/brewroot"

... which prevents the package from being successfully built in mock:

$ mock -r rhel-8-x86_64 --quiet --installdeps" "/tmp/csmockiodxf3_u/python2-six-1.11.0-6.module+el8.4.0+9287+299307c7.src.rpm" "--addrepo" "file:///tmp/csmockiodxf3_u/local-build-repo"
Error: 
 Problem: package python2-tkinter-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64 requires python2 = 2.7.18-2.module+el8.4.0+9193+f3daf6ef, but none of the providers can be installed
  - conflicting requests
  - nothing provides python2-libs(x86-64) = 2.7.18-2.module+el8.4.0+9193+f3daf6ef needed by python2-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64
(try to add '--skip-broken' to skip uninstallable packages

The code eats in order of magnitude more IO than the optimized createrepo_c run on large repositories.

In copr, we run:
/usr/bin/createrepo_mod /var/lib/copr/public_html/results/iucar/cran/fedora-31-x86_64 --database --ignore-lock --local-sqlite --cachedir /tmp/ --workers 8 --recycle-pkglist --update --skip-stat --pkglist /var/lib/copr/public_html/results/iucar/cran/fedora-31-x86_64/.copr-createrepo-pkglist

There is a missing python3-click-man package for EPEL8 and therefore we cannot
generate the manpages on the fly. They are temporarily committed into git repo.
Once RHBZ 1900423 gets resolved, just uncomment the manpage generation in the specfile.
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1900423

I imagine that every build system that supports building modules needs to have some piece of code determining what module RPMs can be built simultaneously and which need to wait and be built after some prerequisites. At this moment it is (AFAIK) trivially determined by buildorder.

The problem is, that build systems still need to parse the modulemd (i.e. directly depend on python3-libmodulemd) and implement this basic functionality by themselves. Therefore I think, that modulemd-tools is the right place to put such code.

I am suggesting to create a script called modulemd-builds or more explicitly modulemd-build-batches that will parse a modulemd YAML file and return output in this format.

[
  [
    {
      "name": "jss",
      "repository": "http://foo.bar/jss.git",
      "ref": "master"
    }
  ],
  [
    {
      "name": "tomcatjss",
      "repository": "http://foo.bar/tomcat.git",
      "ref": "master"
    },
    {
      "name": "ldapjdk",
      "repository": "http://foo.bar/ldapjdk.git",
      "ref": "master"
    }
  ],
  [
    {
      "name": "pki-core",
      "repository": "http://foo.bar/pki-core.git",
      "ref": "master"
    },
    {
      "name": "dogtag-pki",
      "repository": "http://foo.bar/dogtag-pki.git",
      "ref": "master"
    }
  ]
]

So, the format it JSON having a list of batches where each batch is represented by a list of packages. The batches need to be built after each other if and only if all the builds from the previous batch succeeded. Packages within a batch can be submitted in random order or in parallel. Each package is represented by a dictionary of all the parameters required for building it.

Do you think it is worth it having such script? Do you like the output format?