Code Monkey home page Code Monkey logo

ebuildtester's Introduction

Introduction

This is a tool to test a Gentoo ebuild and its dependencies. The idea is that the ebuild is emerged in a clean (and current) stage3 Docker container.

Documentation Status GitHub Actions Join the chat at https://gitter.im/ebuildtester/ebuildtester Get it from the Snap Store

Requirements

Using require Docker and FUSE.

If you plan to use specific storage driver options (by passing --storage-opt), be aware that these are specific to the configured Docker storage driver. Refer to the Docker documentation about storage drivers for more information. Particularly see the list of options per storage driver.

System-wide configuration of the storage driver used by Docker is done in /etc/docker/daemon.json. For example, to select the devicemapper storage driver, specify:

{
  "storage-driver": "devicemapper"
}

Usage

We are going to assume that the user has a local git clone of the portage tree in

/usr/local/git/gentoo

We have added a new ebuild and would like to verify that the build dependencies are all correct. We can build the package (ATOM) with:

ebuildtester --portage-dir /usr/local/git/gentoo \
  --atom ATOM \
  --use USE1 USE2

where we have specified two USE flags, USE1 and USE2. The ebuildtester command will now create a docker container and start installing the ATOM. All specified dependencies will be installed as well.

Command line arguments

The command understands the following command line arguments:

usage: ebuildtester [-h] [--version] [--atom ATOM [ATOM ...]] [--binhost BINHOST] [--live-ebuild]
                    [--manual] --portage-dir PORTAGE_DIR [--overlay-dir OVERLAY_DIR] [--update]
                    [--install-basic-packages] [--threads N] [--use USE [USE ...]]
                    [--global-use GLOBAL_USE [GLOBAL_USE ...]] [--unmask ATOM] [--unstable]
                    [--gcc-version VER] [--python-single-target PYTHON_SINGLE_TARGET]
                    [--python-targets PYTHON_TARGETS] [--rm] [--storage-opt STORAGE_OPT [STORAGE_OPT ...]]
                    [--with-X] [--with-vnc] [--profile PROFILE] [--features FEATURES [FEATURES ...]]
                    [--docker-image DOCKER_IMAGE] [--docker-command DOCKER_COMMAND] [--pull]
                    [--show-options] [--ccache CCACHE_DIR] [--batch]

A dockerized approach to test a Gentoo package within a clean stage3.

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --atom ATOM [ATOM ...]
                        The package atom(s) to install
  --binhost BINHOST     Binhost URI
  --live-ebuild         Unmask the live ebuild of the atom
  --manual              Install package manually
  --portage-dir PORTAGE_DIR
                        The local portage directory
  --overlay-dir OVERLAY_DIR
                        Add overlay dir (can be used multiple times)
  --update              Update container before installing atom
  --install-basic-packages
                        Install basic packages after container starts
  --threads N           Use N (default 8) threads to build packages
  --use USE [USE ...]   The use flags for the atom
  --global-use GLOBAL_USE [GLOBAL_USE ...]
                        Set global USE flag
  --unmask ATOM         Unmask atom (can be used multiple times)
  --unstable            Globally 'unstable' system, i.e. ~amd64
  --gcc-version VER     Use gcc version VER
  --python-single-target PYTHON_SINGLE_TARGET
                        Specify a PYTHON_SINGLE_TARGET
  --python-targets PYTHON_TARGETS
                        Specify a PYTHON_TARGETS
  --rm                  Remove container after session is done
  --storage-opt STORAGE_OPT [STORAGE_OPT ...]
                        Storage driver options for all volumes (same as Docker param)
  --with-X              Globally enable the X USE flag
  --with-vnc            Install VNC server to test graphical applications
  --profile PROFILE     The profile to use (default = default/linux/amd64/23.0)
  --features FEATURES [FEATURES ...]
                        Set FEATURES, see https://wiki.gentoo.org/wiki/FEATURES (default = ['-sandbox',
                        '-usersandbox', 'userfetch'])
  --docker-image DOCKER_IMAGE
                        Specify the docker image to use (default = gentoo/stage3)
  --docker-command DOCKER_COMMAND
                        Specify the docker command
  --pull                Download latest docker image
  --show-options        Show currently selected options and defaults
  --ccache CCACHE_DIR   Path to mount that contains ccache cache
  --batch               Do not drop into interactive shell

ebuildtester's People

Contributors

derpdays avatar georgek avatar gitter-badger avatar guillaumeseren avatar jonasstein avatar linxon avatar nicolasbock avatar nicoroeser avatar oxr463 avatar parona-source avatar stkw0 avatar waebbl avatar waffle-iron avatar xp-vit avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

ebuildtester's Issues

Add developers documentation

The current documentation of the project is nice, but it seems to only target users. New developers who would like to contribute to the project might need some documentation to get started (requirements, build process, testing, etc).

`snap` cannot be installed

On Ubuntu 22.04 getting the following:

error: cannot perform the following tasks:
- Run configure hook of "ebuildtester" snap if present (run hook "configure": 
-----
Please run

snap connect ebuildtester:docker-executables docker:docker-executables

in order to use the ebuildtester command.
-----)

If I try snap connect ebuildtester:docker-executables docker:docker-executables, I get:

error: snap "ebuildtester" is not installed

[Enhancement] Adding --mask option

Another enhancement issue :)
Suppose the use case, where you have to test an ebuild against several unslotted versions of another package, of which some, but not all are stable, but you still want to have a fully updated system, according to the restrictions this would apply. If you now run ebuildtester with the --update flag set, you have no way to perform this test, because the update pulls in the latest stable package.
Therefore, I was thinking about adding a --mask option. I already started on the implementation of it.
What do you think about this?

Error profile file not found

I've tried ebuildteter through the following command:

buildtester --portage-dir /usr/portage\
                  --overlay-dir /home/rafael/src/gentoo\
                  --use R boost imaging python qt5 rendering views\
                  --unmask =sci-libs/vtk-9.0.1\
                  --with-X\
                  --atom =sci-libs/vtk-9.0.1

And get the following error:

 f9d51f eselect profile set default/linux/amd64/17.1
2022-01-10 15:25:50,838 - f9d51f (stderr): sed: can't read /var/db/repos/gentoo/profiles/profiles.desc: No such file or directory
2022-01-10 15:25:50,840 - f9d51f (stderr): !!! Error: default/linux/amd64/17.1 is not a valid profile for amd64
2022-01-10 15:25:50,840 - f9d51f (stderr): exiting

Any idea what could be wrong in here?

Add support for distcc

Hey there,
I am using ebuildtester again and it is nice the support for ccache,
but my laptop is only 4 threads and I normally use distcc to speed up compilation (when it works).

I think it could nice to have this feature in ebuildtester, what do you think ?

I will try to prepare a small patch for this.

Add ccache to default image ?

Hello,
I know ebuildtester use the default stage3 image, so maybe it is an issue for gentoo BTS,
but if the ccache support could already installed in the container,
because installing ccache take a little while (and will not be cached :/ ).

Add libvrt backend

Containeres cannot be used to build ebuilds that require kernel
modifications or special kernel modules. Add a libvirt bases backened to
allow for virtual machines.

Move tester.log to a predefined directory

Instead of creating tester.log on the working directory that ebuildtester is executed it may be good to store logs in some other place (in XDG standard directory for logs probably). The problem I have with the current behaviour is that sometimes I run ebuidltester in my fork of gentoo and if I forgot to check it sometimes i add & push tester.log too. Another inconvenience is that sometimes I execute tester.log in some folder where I was in a different directory and I forgot to remove it and I end up having tester.log files in different places.

--global-use option not used

I was just wondering why boost-build in a container didn't have the python use flag enabled, although I added --global-use python to the command.
Grepping the files, I noticed, that it's defined in parse.py, but it's currently not used anywhere.

Permission denied when running ebuildtester (snap)

There seems to be a permissions issue when running ebuildtester installed through snap:

2022-06-23 07:29:03,778 - creating container
Traceback (most recent call last):
  File "/snap/ebuildtester/160/bin/ebuildtester", line 8, in <module>
    sys.exit(main())
  File "/snap/ebuildtester/160/lib/python3.8/site-packages/ebuildtester/main.py", line 21, in main
    container = Docker(
  File "/snap/ebuildtester/160/lib/python3.8/site-packages/ebuildtester/docker.py", line 21, in __init__
    repo_names = self._get_repo_names(overlay_dirs)
  File "/snap/ebuildtester/160/lib/python3.8/site-packages/ebuildtester/docker.py", line 244, in _get_repo_names
    with open(os.path.join(o, "profiles/repo_name"), "r") as f:
PermissionError: [Errno 13] Permission denied: '/home/rafael/src/gentoo-science/profiles/repo_name'

ls /home/rafael/src/gentoo-science/profiles/repo_name -lah produces:

-rw-rw-r-- 1 rafael rafael 8 Jun 22 12:15 /home/rafael/src/gentoo-science/profiles/repo_name

The ebuildtester installed following https://ebuildtester.readthedocs.io/en/latest/contributing.html# works fine on the same invokation.

Missing server use flag for tigervnc

The tigervnc package installed via the new --with-X command line argument is missing the server USE flag. Without it, the VNC server is not built.

Allow to specify docker image

Allowing to specify the docker image will allow to use docker images for other architectures and test packages with qemu. Or maybe it will be better to specify the architecture ¿?

Ebuildtester does not get installed through snap

Trying to install ebuildtester through snap in Ubuntu 22.04 produces the following error:

error: cannot perform the following tasks:
- Run configure hook of "ebuildtester" snap if present (run hook "configure": 
-----
Please run

snap connect ebuildtester:docker-executables docker:docker-executables

in order to use the ebuildtester command.
-----)

Issuing snap connect ebuildtester:docker-executables docker:docker-executables produces:

error: snap "ebuildtester" is not installed

Enable "live ebuilds"

Live ebuilds need to be enabled by adding

ATOM **

to /etc/portage/package.accept_keywords.

binhost issue with default Gentoo binhost

Now Gentoo stage3 provides a default binhost. If getbinpkg FEATURE is enabled portage will try to use first the default binhost, which requires setting up the keys. Because of that currently if trying to use ebuildtester with a binhost it gives the next error:

2023-11-21 12:06:58,184 - 8d60ee (stderr): !!!
2023-11-21 12:06:58,184 - 8d60ee (stderr): gpg: keyblock resource '/etc/portage/gnupg/pubring.kbx': No such file or directory
2023-11-21 12:06:58,184 - 8d60ee (stderr): [GNUPG:] ERROR add_keyblock_resource 33587281
2023-11-21 12:06:58,184 - 8d60ee (stderr): [GNUPG:] PLAINTEXT 74 0
2023-11-21 12:06:58,184 - 8d60ee (stderr): [GNUPG:] NEWSIG
2023-11-21 12:06:58,184 - 8d60ee (stderr): gpg: Signature made Sat Nov  4 17:21:50 2023 -00
2023-11-21 12:06:58,184 - 8d60ee (stderr): gpg:                using RSA key 534E4209AB49EEE1C19D96162C44695DB9F6043D
2023-11-21 12:06:58,184 - 8d60ee (stderr): [GNUPG:] ERROR keydb_search 33554445
2023-11-21 12:06:58,184 - 8d60ee (stderr): [GNUPG:] ERROR keydb_search 33554445
2023-11-21 12:06:58,184 - 8d60ee (stderr): [GNUPG:] ERRSIG 2C44695DB9F6043D 1 10 01 1699118510 9 534E4209AB49EEE1C19D96162C44695DB9F6043D
2023-11-21 12:06:58,184 - 8d60ee (stderr): [GNUPG:] NO_PUBKEY 2C44695DB9F6043D
2023-11-21 12:06:58,184 - 8d60ee (stderr): gpg: Can't check signature: No public key
2023-11-21 12:06:58,185 - 8d60ee (stderr): gpg: can't create `/etc/portage/gnupg/random_seed': No such file or directory
2023-11-21 12:06:58,185 - 8d60ee (stderr): !!! Invalid binary package: '/var/cache/binpkgs/dev-python/sip/sip-6.7.12-3.gpkg.tar.partial', GPG verify failed

Cannot accept licenses

2022-06-23 07:39:56,996 - fbfa8d (stdout): Calculating dependencies
2022-06-23 07:39:56,996 - fbfa8d (stdout):  * IMPORTANT: 1 news items need reading for repository 'science'.
2022-06-23 07:39:56,997 - fbfa8d (stdout):  * IMPORTANT: 13 news items need reading for repository 'gentoo'.
2022-06-23 07:39:56,997 - fbfa8d (stdout):  * Use eselect news read to view new items.
2022-06-23 07:39:56,997 - fbfa8d (stdout): 
2022-06-23 07:39:59,803 - fbfa8d (stdout): ... done!
2022-06-23 07:39:59,808 - fbfa8d (stderr): 
2022-06-23 07:39:59,808 - fbfa8d (stderr): !!! All ebuilds that could satisfy "sci-libs/dcmtk:0=" have been masked.
2022-06-23 07:39:59,809 - fbfa8d (stderr): !!! One of the following masked packages is required to complete your request:
2022-06-23 07:39:59,809 - fbfa8d (stderr): - sci-libs/dcmtk-3.6.7::gentoo (masked by: OFFIS license(s))
2022-06-23 07:39:59,809 - fbfa8d (stderr): A copy of the 'OFFIS' license is located at '/var/db/repos/gentoo/licenses/OFFIS'.
2022-06-23 07:39:59,809 - fbfa8d (stderr): 
2022-06-23 07:39:59,809 - fbfa8d (stderr): - sci-libs/dcmtk-3.6.5-r1::gentoo (masked by: OFFIS license(s), ~amd64 keyword)
2022-06-23 07:39:59,809 - fbfa8d (stderr): - sci-libs/dcmtk-3.6.5::gentoo (masked by: OFFIS license(s))
2022-06-23 07:39:59,809 - fbfa8d (stderr): 
2022-06-23 07:39:59,809 - fbfa8d (stderr): (dependency required by "sci-libs/itk-5.1.0::science" [ebuild])
2022-06-23 07:39:59,809 - fbfa8d (stderr): (dependency required by "=sci-libs/itk-5.1.0" [argument])
2022-06-23 07:39:59,809 - fbfa8d (stderr): For more information, see the MASKED PACKAGES section in the emerge
2022-06-23 07:39:59,809 - fbfa8d (stderr): man page or refer to the Gentoo Handbook.
2022-06-23 07:39:59,809 - fbfa8d (stderr):

I have tried giving --unmask =sci-libs/dcmtk-3.6.7 to ebuildtester, but it falls back into manual. Is there any way to automate this?

failure creating docker container

ebuildtester --portage-dir /var/db/repos/danidev --atom "gui-apps/packettracer"

2023-11-13 15:24:04,065 - logging at /tmp/ebuildtester-gui-apps-packettracer.log
2023-11-13 15:24:04,065 - *** please note that all necessary licenses will be accepted ***
2023-11-13 15:24:04,065 - creating container
2023-11-13 15:24:04,065 - creating docker container with: docker create --tty --cap-add CAP_SYS_ADMIN --cap-add CAP_MKNOD --cap-add CAP_NET_ADMIN --security-opt apparmor:unconfined --device /dev/fuse --workdir /root --volume /var/db/repos/danidev:/var/db/repos/gentoo --volume /var/db/repos/danidev/distfiles:/var/cache/distfiles --volume /var/db/repos/danidev/packages:/var/cache/binpkgs gentoo/stage3
docker: unknown server OS: .
See 'docker create --help'.
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.11/ebuildtester", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.11/site-packages/ebuildtester/main.py", line 23, in main
    container = Docker(
                ^^^^^^^
  File "/usr/lib/python3.11/site-packages/ebuildtester/docker.py", line 26, in __init__
    self._create_container(docker_image, local_portage,
  File "/usr/lib/python3.11/site-packages/ebuildtester/docker.py", line 184, in _create_container
    raise Exception("failure creating docker container")
Exception: failure creating docker container

equery l docker

 * Searching for docker ...
[IP-] [  ] app-containers/docker-24.0.5:0

equery l docker-clli

 * Searching for docker-cli ...
[IP-] [  ] app-containers/docker-cli-24.0.5:0

What about removing base_packages?

Personally I found it a waste of time to merge them as I almost never use it and the last time that Gentoo updated the default PYTHON_TARGETS the container failed to execute because there were conflicts when merging those base_packages.

README file inaccurate with regard to Docker storage driver

The README document tells users that the Device Mapper storage driver needs to be used in the Docker installation. Luckily (because people may prefer other storage drivers), this is not generally true. Instead each storage driver comes with a set of supported storage options. (Comments like #88 (comment) confirm this.)

The note about the requirement has been added in commit 9085de7 as part of pull request #89. This was a reaction to issue #88.

A while later, commit cf1fe37 has been merged in pull request #107. That commit, among other things, removes the hardcoded storage option size=50G. Storage options can since then be specified by users. The README document has not been modified accordingly.

I use ebuildtester with the Docker BTRFS storage driver without problems. I usually do not pass storage options. Even if I did, whether it worked or not would depend on the combination of storage driver and options, as noted above.

Therefore I’d like to suggest a modification of the README file. The misleading information about the Device Mapper storage driver being required should be removed or changed. (When I first tried ebuildtester, I was unsure whether I would able to use it at all since my setup did not fit that documented requirement.)

Idea: work on chroot?

Should the container work on a chroot system like
docker ... -vol /data/gentoo-stable-chroot
where /data/gentoo-stable-chroot is a btrfs snapshot?
This could help with #21

Unfortunately a growing toolchain docker + chroot + btrfs... makes everything more complicate too.

Manpage?

It would be pretty cool if ebuildtester shipped with a manpage.

Nosetests appears to do this in a very python-centric way that would sidestep the need to use groff macro packages directly (see the following files). It appears to hook into Sphinx. Here's a ticket discussing various approaches. Perhaps a similar approach could be used for ebuildtester?

I'm willing to learn more of sphinx to make this happen - let me know if this sounds like a good idea.

fix: update default profile to 23.0

Currently the latest gentoo stage3 docker image now uses merged-usr by default, this means that the current default profile (which is meant to be split-usr) does not always work correctly for some ebuilds which have logic specifically for split-usr since they expect a split usr/bin and bin.

I believe we should update the default profile to 23.0 since 17.1 will stop being supported eventually and to resolve the aforementioned issue.

Support different arch profiles

Now we are capable of specifying non-amd64 docker images, but if in accordance to it we try to use a non-amd64 profile we get an error like: ebuildtester: error: argument --profile: invalid choice: 'default/linux/arm/17.0' (choose from 'default/linux/amd64/17.1', 'default/linux/amd64/17.1/systemd')

It will be cool to try to improve the support for non-amd64 containers

Error while creating container

I've tried to use this tool to test some ebuilds but it failed with nice python traceback:

ebuildtester --manual --portage-dir /usr/portage --overlay-dir /home/soltys/workspace/soltys-overlay
2018-07-09 15:38:47,133 - creating container
Using default tag: latest
latest: Pulling from gentoo/stage3-amd64
457fc2d084ea: Pull complete 
Digest: sha256:c731620786d8d0d222fe9656d99ea81ec2e3ec342c39ca3e37913c384b6a3470
Status: Downloaded newer image for gentoo/stage3-amd64:latest
2018-07-09 15:42:54,708 - creating docker container with: docker create --tty --cap-add SYS_ADMIN --device /dev/fuse --storage-opt size=50G --workdir /root --volume /usr/portage:/usr/portage --volume /usr/portage/distfiles:/usr/portage/distfiles --volume=/home/soltys/workspace/soltys-overlay:/var/lib/overlays/soltys-overlay gentoo/stage3-amd64
2018-07-09 15:42:57,869 - container id 21e1f0f8f7bad81fabcd55d30f9cd119a09977efa8f1a756f565e2b3f00e1e51
2018-07-09 15:42:59,598 - setting Gentoo profile to default/linux/amd64/17.0
2018-07-09 15:42:59,599 - 21e1f0 eselect profile set default/linux/amd64/17.0
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/ebuildtester", line 11, in <module>
    load_entry_point('ebuildtester==0.1.10', 'console_scripts', 'ebuildtester')()
  File "/usr/lib64/python2.7/site-packages/ebuildtester/main.py", line 21, in main
    [os.path.abspath(p) for p in options.options.overlay_dir])
  File "/usr/lib64/python2.7/site-packages/ebuildtester/docker.py", line 25, in __init__
    self._set_profile()
  File "/usr/lib64/python2.7/site-packages/ebuildtester/docker.py", line 159, in _set_profile
    self.execute("eselect profile set %s" % options.options.profile)
  File "/usr/lib64/python2.7/site-packages/ebuildtester/docker.py", line 64, in execute
    os.waitid(os.P_PID, stdout_reader, os.WEXITED)
AttributeError: 'module' object has no attribute 'waitid'```

It looks like this tool requires at least Python 3.3 as according to [Python 3 os module docs](https://docs.python.org/3/library/os.html#os.waitid)

Could you make it work also on Python 2.x or make it impossible to install this on Python 2.x?

Brainstorming

  • is it better to run in a chroot?
  • which files/folders should be outside of a container anyway? (/var /usr/portage make.conf...)
  • if chroot, how to reuse things intelligent?
  • provide a stage5 with KDE stable?
  • how to sync the chroot? rsync?
  • how to reset the chroot? btrfs snapshot? rsync, rdiffbackup...?

--global-use not used

Version: 0.1.29

The useflags are not added to make.conf or to $USE. The command line was:

ebuildtester \
  --update \
  --pull \
  --threads 1 \
  --portage-dir "$(portageq get_repo_path / gentoo)" \
  --unstable \
  --overlay-dir ~/Projekte/guru \
  --with-X \
  --global-use "elogind" \
  --atom '=media-gfx/glimpse-0.2.0' '=dev-lang/rust-bin-1.51.0' 'sys-kernel/gentoo-sources-5.11.15'

I tried without --width-X, same result.

Error caused by --storage-opt

I've just installed Docker. Haven't done any configuration of it other than to add myself to docker group. Running as non-root:

ebuildtester --portage-dir /usr/portage/ --manual
2018-09-23 20:01:29,308 - creating container
Using default tag: latest
latest: Pulling from gentoo/stage3-amd64
8ba66ace1774: Pull complete 
Digest: sha256:80145371b9973ab35387caa7aa6c4350a1fbf5b60fe411254e3a78351e0e46ed
Status: Downloaded newer image for gentoo/stage3-amd64:latest
2018-09-23 20:05:27,135 - creating docker container with: docker create --tty --cap-add SYS_ADMIN --device /dev/fuse --storage-opt size=50G --workdir /root --volume /usr/portage:/usr/portage --volume /usr/portage/distfiles:/usr/portage/distfiles gentoo/stage3-amd64
Error response from daemon: --storage-opt is supported only for overlay over xfs with 'pquota' mount option
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.6/ebuildtester", line 11, in <module>
    load_entry_point('ebuildtester==0.1.10', 'console_scripts', 'ebuildtester')()
  File "/usr/lib64/python3.6/site-packages/ebuildtester/main.py", line 21, in main
    [os.path.abspath(p) for p in options.options.overlay_dir])
  File "/usr/lib64/python3.6/site-packages/ebuildtester/docker.py", line 23, in __init__
    zip(overlay_dirs, overlay_mountpoints))
  File "/usr/lib64/python3.6/site-packages/ebuildtester/docker.py", line 135, in _create_container
    raise Exception("failure creating docker container")
Exception: failure creating docker container

ebuildtester.docker.ExecuteFailure: failed command "eselect profile set default/linux/amd64/17.1"

ebuildtester --portage-dir /mnt/src/gentoo --atom dev-libs/fyaml
2020-05-21 19:09:41,640 - creating container
2020-05-21 19:09:41,641 - creating docker container with: docker create --tty --cap-add SYS_ADMIN --device /dev/fuse --workdir /root --volume /mnt/src/gentoo:/var/db/repos/gentoo --volume /mnt/src/gentoo/distfiles:/var/cache/distfiles --volume /mnt/src/gentoo/packages:/var/cache/binpkgs gentoo/stage3-amd64
Unable to find image 'gentoo/stage3-amd64:latest' locally
latest: Pulling from gentoo/stage3-amd64
02a68cf0a387: Pulling fs layer
02a68cf0a387: Verifying Checksum
02a68cf0a387: Download complete
02a68cf0a387: Pull complete
Digest: sha256:6f11ceb375b6f5c33ad923ccc38dbae15c717a1dfc613cb3f479ea1f4e76109b
Status: Downloaded newer image for gentoo/stage3-amd64:latest
2020-05-21 19:13:37,542 - container id 3b82a4cc314af39cc37dc3d7d6dd0475ccfb57b4e60cfd4028f01030de485850
2020-05-21 19:13:39,766 - setting Gentoo profile to default/linux/amd64/17.1
2020-05-21 19:13:39,767 - 3b82a4 eselect profile set default/linux/amd64/17.1
2020-05-21 19:13:40,905 - 3b82a4 (stderr): sed: can't read /var/db/repos/gentoo/profiles/profiles.desc: No such file or directory
2020-05-21 19:13:40,907 - 3b82a4 (stderr): !!! Error: default/linux/amd64/17.1 is not a valid profile for amd64
2020-05-21 19:13:40,908 - 3b82a4 (stderr): exiting
2020-05-21 19:13:41,017 - running in container 3b82a4cc314af39cc37dc3d7d6dd0475ccfb57b4e60cfd4028f01030de485850
Traceback (most recent call last):
  File "/usr/bin/ebuildtester", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/ebuildtester/main.py", line 19, in main
    container = Docker(
  File "/usr/lib/python3.8/site-packages/ebuildtester/docker.py", line 28, in __init__
    self._set_profile()
  File "/usr/lib/python3.8/site-packages/ebuildtester/docker.py", line 188, in _set_profile
    self.execute("eselect profile set %s" % options.options.profile)
  File "/usr/lib/python3.8/site-packages/ebuildtester/docker.py", line 91, in execute
    raise ExecuteFailure("failed command \"%s\"" % (cmd))
ebuildtester.docker.ExecuteFailure: failed command "eselect profile set default/linux/amd64/17.1"

Forking for usage with podman

Am going to try to get this to work using podman instead of docker, mostly because podman is nicer to integrate with the system and can work with an unprivileged account.

Just wanted to let you know about my endeavor in case someone else is also interested and if this could someday be merged here.

Error response from daemon: cgroups: cannot find cgroup mount destination: unknown

I'm facing this error from some time ago. Seems that it doesn't runs well without systemd. Currently I'm using the next workaround

mkdir /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Error traceback

2019-10-05 14:01:31,428 - container id 972b069b7468a8f21330de99c8cf2faf2309b4bfed15459f7c91af36616760a4
Error response from daemon: cgroups: cannot find cgroup mount destination: unknown
Error: failed to start containers: 972b069b7468a8f21330de99c8cf2faf2309b4bfed15459f7c91af36616760a4
Traceback (most recent call last):
  File "/home/david/.local/bin/ebuildtester", line 11, in <module>
    load_entry_point('ebuildtester==0.1.15.dev5+g1d8d30d', 'console_scripts', 'ebuildtester')()
  File "/home/david/.local/lib64/python3.6/site-packages/ebuildtester-0.1.15.dev5+g1d8d30d-py3.6.egg/ebuildtester/main.py", line 21, in main
    def wont_run(self, reason):
  File "/home/david/.local/lib64/python3.6/site-packages/ebuildtester-0.1.15.dev5+g1d8d30d-py3.6.egg/ebuildtester/docker.py", line 24, in __init__
  File "/home/david/.local/lib64/python3.6/site-packages/ebuildtester-0.1.15.dev5+g1d8d30d-py3.6.egg/ebuildtester/docker.py", line 184, in _start_container
Exception: failure creating docker container

Experiments with libpod/podman

Hello

I started to experiment with ebuildtester and possibly using it with podman/libpod, because it supports rootless containers with user namespaces. The CLI is equivalent to dockers, so it should ideally be a drop-in replacement.

Here is what I did:

  • Installed and configured libpod with the crun runner (the default runc does not support cgroups v2 yet)
  • removed the SYS_ADMIN cap
  • created a symlink docker -> podman

Results:

  • It seems to work at least in parts. The container gets created and starts. It can install gentoolkit etc.
  • I get a lot of messages on stderr: Unable to unshare: EPERM (for FEATURES="ipc-sandbox network-sandbox")
  • It breaks on package glibc because that checks if /dev/pts is mounted with gid=5 which it isn't in the container.

So what do you think? is this worth investigating more?

"Unable to configure loopback interface: Operation not permitted" during run

I'm seeing following message almost constantly appearing in logs when running ebuildtester:
58a1e0 (stderr): Unable to configure loopback interface: Operation not permitted

My assumption based on investigations that it's happening when portage tries to configure lo interface:
_configure_loopback_interface() in lib/portage/process.py

And fails due to some reason.

Also, I verified that when I run "ifconfig -a" inside ebuild code, I see that lo is not configured correctly.

This issue prevents successful tests of ebuilds that consume some network services on lo, e.g. when tests require local version of DB running.

Support for PYTHON_SINGLE_TARGET and PYTHON_TARGETS

Hello,
Thank you for this useful project, I have been using it more and more to test my packages.

Yesterday I wanted to test python3_8 and did not find a way without patching the source,
so I did add those options so I will open a PR.

`docker: unknown server OS` error when launching ebuildtester (snap)

Invoking:

 ebuildtester --atom =sci-libs/itk-5.1.0 --portage-dir /home/rafael/src/gentoo\
 --overlay-dir /home/rafael/src/gentoo-science\
 --unmask =sci-libs/itk-5.1.0\
 --unmask =sci-libs/dcmtk-3.6.7 \
 --features ccache\
 --ccache /home/rafael/.var/cache/systole-cache

produces:

2022-06-23 12:36:22,488 - logging at /home/rafael/snap/ebuildtester/175/ebuildtester-sci-libs-itk.log
2022-06-23 12:36:22,488 - *** please note that all necessary licenses will be accepted ***
2022-06-23 12:36:22,488 - creating container
2022-06-23 12:36:22,488 - creating docker container with: docker-wrapper.sh create --tty --cap-add CAP_SYS_ADMIN --cap-add CAP_MKNOD --cap-add CAP_NET_ADMIN --security-opt apparmor:unconfined --device /dev/fuse --workdir /root --volume /home/rafael/src/gentoo:/var/db/repos/gentoo --volume /home/rafael/src/gentoo/distfiles:/var/cache/distfiles --volume /home/rafael/src/gentoo/packages:/var/cache/binpkgs --volume=/home/rafael/.var/cache/systole-cache:/var/tmp/ccache --volume=/home/rafael/src/gentoo-science:/var/lib/overlays/science gentoo/stage3
docker: unknown server OS: .
See 'docker create --help'.
Traceback (most recent call last):
  File "/snap/ebuildtester/175/bin/ebuildtester", line 8, in <module>
    sys.exit(main())
  File "/snap/ebuildtester/175/lib/python3.8/site-packages/ebuildtester/main.py", line 23, in main
    container = Docker(
  File "/snap/ebuildtester/175/lib/python3.8/site-packages/ebuildtester/docker.py", line 26, in __init__
    self._create_container(docker_image, local_portage,
  File "/snap/ebuildtester/175/lib/python3.8/site-packages/ebuildtester/docker.py", line 184, in _create_container
    raise Exception("failure creating docker container")
Exception: failure creating docker container
rafael@precision-5820-tower:/root$

incorrect version splitting

I found another issue with how the version splitting of an atom currently is working, but haven't found a quick fix for it. My python skills are still lacking behind :/

Whenever a package name contains a hyphen, the atom.py file splits at this position to get the version string, so for example using ebuildtester --atom dev-util/boost-build it will see the build part as a version string and therefore __str__ gives wrong results which then get into the files in /etc/portage/package.*.

Also the lines in the atom constructor where you split at = sign seem to be wrong, as the last line of the output below indicates. If there's no = or >= given on the command line it seems to truncate the last char of the package name. I think the splitting needs to be guarded for cases where the atom actually starts with = or >=.

See for example the following output snippet:

waebbl@artus ~/bugs.gentoo.org/667728 $ ebuildtester --atom dev-util/boost-build --portage-dir /usr/portage --update no --threads 1 --global-use python --profile default/linux/amd64/17.0
2018-10-08 15:59:30,532 - creating container
Using default tag: latest
latest: Pulling from gentoo/stage3-amd64
Digest: sha256:a2ffc73ac6e8b4dd830365b3b686e7c2a591fa5c3d432100c45a65e98308d122
Status: Image is up to date for gentoo/stage3-amd64:latest
2018-10-08 15:59:32,207 - creating docker container with: docker create --tty --cap-add SYS_ADMIN --device /dev/fuse --storage-opt size=110G --workdir /root --volume /usr/portage:/usr/portage --volume /usr/portage/distfiles:/usr/portage/distfiles gentoo/stage3-amd64
2018-10-08 15:59:34,033 - container id 498109aa4e18cab851304abb45b447d2a8ac434632d643ba7f6ed3d39cf5f898
2018-10-08 15:59:34,985 - setting Gentoo profile to default/linux/amd64/17.0
2018-10-08 15:59:34,985 - 498109 eselect profile set default/linux/amd64/17.0
2018-10-08 15:59:36,052 - tweaking portage settings
2018-10-08 15:59:36,053 - 498109 echo FEATURES=\"-sandbox -usersandbox\" >> /etc/portage/make.conf
2018-10-08 15:59:36,264 - 498109 echo MAKEOPTS=\"-j1\" >> /etc/portage/make.conf
2018-10-08 15:59:36,458 - 498109 mkdir -p /etc/portage/repos.conf
2018-10-08 15:59:36,661 - enabling test feature for [Atom("=dev-util/boost-build")]
2018-10-08 15:59:36,661 - 498109 mkdir -p /etc/portage/env
2018-10-08 15:59:36,870 - 498109 echo "=dev-util/boost-build tester.conf" >> /etc/portage/package.env
2018-10-08 15:59:37,074 - 498109 echo "FEATURES=\"test splitdebug\"" > /etc/portage/env/tester.conf
2018-10-08 15:59:37,273 - unmasking [Atom("=dev-util/boost-build")]
2018-10-08 15:59:37,273 - 498109 mkdir -p /etc/portage/package.accept_keywords
2018-10-08 15:59:37,475 - 498109 echo "=dev-util/boost-build" ~amd64 >> /etc/portage/package.accept_keywords/testbuild
2018-10-08 15:59:37,681 - unmasking additional atoms
2018-10-08 15:59:37,682 - skipping update
2018-10-08 15:59:37,682 - installing basic packages: [Atom("app-portage/gentoolkit"), Atom("app-portage/flaggie")]
2018-10-08 15:59:37,682 - 498109 emerge --verbose app-portage/gentoolkit app-portage/flaggie
2018-10-08 15:59:38,229 - 498109 (stderr): --- Invalid atom in /etc/portage/package.env: =dev-util/boost-build
2018-10-08 15:59:38,251 - 498109 (stderr): --- Invalid atom in /etc/portage/package.accept_keywords/testbuild: =dev-util/boost-buil

License clarification

The setup.py mentions BSD, but could we include the license in the repository directly?

Enable tests

Enable the test feature for the ATOMs:

  • Create /etc/portage/env
  • Add ATOM atom.conf to /etc/portage/package.env
  • Add FEATURES='test' to /etc/portage/env/atom.conf

Add config file

Add config file to allow, at least, the next:

  • Configure multiple binhosts (or just enable the official one)
  • Define portage-dir
  • Define one or more overlay-dir
  • Define ccache dir (or disable ccache)
  • Set unstable by default
  • Use --pull by default
  • Use --update by default

Any flag passed by command line should override any defined configuration

0.1.23 release?

Would you like to release 0.1.23? It contains nice fixes and improvements. Shall we add it to Gentoo tree.

BTW, Thanks for the great tool.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.