Problem description

The XML configuration allows to specify the attribute

<repository ... imageinclude="true|false"/>

This should cause the repo to be persistently configured in the image

Actual behaviour

not implemented

Not sure if you really need that script. Can't it replaced by bumpversion?

For example:

$ bumpversion patch
$ git log --no-pager log -n1 --oneline
[SHA1] Bump version: 8.10.0 → 8.10.1

$ bumpversion minor
...
$ git log --no-pager log -n1 --oneline
[SHA1] Bump version: 8.10.0 → 8.11.0

$ git tag
v8.10.1
v8.11.0

kiwi uses the underscore (_) as directory separator because slash (/) is an invalid character for a shell variable. The volume information is turned into shell variables of the profile environment which introduces that limitation.

This means users can't specify an underscore in the mountpoint= and/or name= attributes. This limitation is bad because the underscore is a quite common character for volume names and allowed in all supported volume management systems.

In order to fix this limitation the way the kiwi initrd code handles the volume information needs to change first

For details watch:

• kiwi/xml_state::get_volumes()
• search for kiwi_LVM_ in kiwi/boot/functions.sh
• search for kiwi_LVM_ in kiwi/boot/arch/*/oemboot/suse-repart

Dear @schaefi ,
when kiwi builds the image, the rootfs has the LABEL=ROOT on blkid,
however our puppets want it to be LABEL="/"

Would it be possible to add some option so we can explicitly specify the LABEL= inside the config.xml ?
And what hook would be the best to use at this current stage ?
We are currently using the kiwi legacy (v7), where I leveraged the preCallInit.sh to enable the SELinux for RHEL6.

---mail snippet Aleksa Sarai [email protected] ---

I just want to point out that I'm not entirely sure why we generate LXC
configurations (Docker hasn't used the LXC execdriver as the default
for about 2 years, and support for it was entirely dropped in Docker
1.10 -- a month ago).

Problem description

editbootcondig=/absolute/path/to/script

causes kiwi to append the path to the image description instead of using the absolute path as it is

Debian based distros (xUbuntu) using the apt package manager

Most of your code contains exception handling which is a good idea as it supports EAFP (Easier to Ask for Forgiveness than Permission).

However, you code contains lots of check for Exception which is, well, a dangerous idea. ;) I guess this is done because it's a new project.

Let me explain why it's dangerous with the following code:

def foo(**kwargs):
   try:
      x = kwargs["x"]
      y = kwargs["y"]
      return x + y
   except Exception as err:
     print("Something bad happen! :-(")

# Example 1:
foo(x=1, y=5)
# -> will return 6

# Example 2:
foo(x='2', y='5')
# -> will return '25'

# Example 3:
foo(x='3', y=6)
# -> Something bad happen! :-(

# Example 4:
foo(a=2, y=10)
# -> Something bad happen! :-(

As you can see, due to Python's duck typing capabilities, example 1 and 2 works just out of the box. It's because of the plus operator is implemented for both types. However, the other examples fail with very different reasons:

• Example 3 fails, because x and y has different types and you can't add an integer with a string. You would get this error message:

  TypeError: unsupported operand type(s) for +: 'int' and 'str'
  

• Example 4 fails, because x is missing in the kwargs dictionary. You would get this error message:

  KeyError: 'x'
  

The examples above show, if you use the very general class Exception you will catch all errors regardless if it has something to do with your code. This mask errors which would be raised if you would use a more specific exception.

The solution to the above code would be to use one of the specific hierarchy of exception classes. Depending on what you are expect in your code, you could catch the KeyError exception. If that's not enough, write your own exception class and derive it from Exception (or something else).

I would recommend to check for specific exceptions only. Don't be too general as that leads to errors which are really hard to debug. Let other exceptions bubble up and deal with them in other places appropriately. That way, if something goes wrong it will be displayed. Or, as Python says it in the this module:

Errors should never pass silently.

:-))

Problem description

Inclucing a user id attribute in <user> section causes a KiwiCommandError exception. For instance, a user definition like:

<user name="vagrant" password="vagrant" pwdformat="plain" home="/home/vagrant" id="1000"/>

In kiwi/system/setup.py the user id given by user.get_id() if not None should be converted into string, as options in users.py are expected to be a string list.

Problem description

This is probably a long term issue

Currently, the code base has got different "API documentation": Some functions and methods have docstrings, others comments, and others don't have any documentation at all.

I think, this situation can be improved. 😃

Reasons

I see the following important points why I think this is important:

• It helps contributors to find its way through the code.
• It encourages good coding style.
• It makes interactive development easier (due to introspection).
• It makes it easier to create API documentation automatically (see Sphinx).

Possible Solution

I think, a good docstring should contain the following parts:

• Short summary of the function
• Parameters and their datatypes
• Return values and the datatype(s)
• Possible exceptions

In openSUSE/docmanager#7 I've suggested Sphinx style:

def add(a, b):
    """Adds two numbers

    :param int a: first argument
    :param int b: second argument
    :return:      sum of a and b
    :rtype:       int
    """
    return a+b

Here are a few specific ideas, taken from Brett Slatkins book "Effective Python":

• Documenting Modules
  Each module should have a top-level docstring. It should contain a single sentence describing the module's purpose. The paragraph that follow should contain the details that all users of the module should know about its operation.
• Documenting Classes
  Each class should have a class-level docstring. The first line describes the purpose of the class. Paragraphs that follow discuss important details. Also important public attributes and methods should be described in the class-level docstring. If possible or applicable, also provide guidance to subclass.
• Documenting Functions
  Each public function and method should have a docstring. This follows the same pattern as modules and classes. Additionally, there are also some special cases that are important to know:
  • If your function has no arguments and a simple return value, a single sentence description is probably good enough.
  • If your function doesn't return anything, it's better to leave out any mention of the return value instead of saying "returns None".
  • If you don't expect your function to raise an exception during normal operation, don't mention that fact.
  • If your function accepts a variable number of arguments or keyword-arguments, use *args and **kwargs in the documented list of arguments to describe their purpose.
  • If your function has arguments with default values, those defaults should be mentioned.
  • If your function is a generator, then your docstring should describe what the generator yields when it's iterated.

support for the arm architecture has not been added to this version of kiwi. This is a must have since the legacy kiwi version provides support. However there are some technical aspects which needs to be clarified with the arm team at SUSE first

This version of kiwi has no support for aci containers. The legacy kiwi version provides this and a backport is possible. However the aci support has been provided by an external contribution and I need to clarify first if there is a real demand for it with regards to docker being the "number 1" container system at the moment

Problem description

Actual behaviour

Steps to reproduce the behaviour

Problem description

When building a kiwi image inside OBS, I get the following traceback:

[ 187s] [ ERROR ]: 15:59:00 | Unexpected error: [ 187s] Traceback (most recent call last): [ 187s] File "/usr/bin/kiwi", line 9, in <module> [ 187s] load_entry_point('kiwi==8.14.1', 'console_scripts', 'kiwi-ng')() [ 187s] File "/usr/lib/python3.4/site-packages/kiwi/kiwi.py", line 55, in main [ 187s] App() [ 187s] File "/usr/lib/python3.4/site-packages/kiwi/app.py", line 35, in __init__ [ 187s] task_class().process() [ 187s] File "/usr/lib/python3.4/site-packages/kiwi/tasks/system_create.py", line 90, in process [ 187s] result = image_builder.create() [ 187s] File "/usr/lib/python3.4/site-packages/kiwi/builder/disk.py", line 256, in create [ 187s] self.boot_image.prepare() [ 187s] File "/usr/lib/python3.4/site-packages/kiwi/boot/image/kiwi.py", line 62, in prepare [ 187s] profile = Profile(self.boot_xml_state) [ 187s] File "/usr/lib/python3.4/site-packages/kiwi/system/profile.py", line 46, in __init__ [ 187s] self.__packages_marked_for_deletion_to_profile() [ 187s] File "/usr/lib/python3.4/site-packages/kiwi/system/profile.py", line 329, in __packages_marked_for_deletion_to_profile [ 187s] self.xml_state.get_to_become_deleted_packages() [ 187s] TypeError: sequence item 6: expected str instance, NoneType found [ 187s] [ INFO ]: 15:59:00 | Cleaning up SystemPrepare instance

When inserting a print inside the profile.py (/usr/lib/python3.4/site-packages/kiwi/system/profile.py) inside the python3-kiwi.tar.bz2 package, it looks like an issue with the "make" package that is specified in the section in the initrd config.xml (/usr/lib/python3.4/site-packages/kiwi/boot/arch/x86_64/oemboot/suse-SLES12/config.xml):
<packages type="delete"> <package name="dracut"/> <package name="cracklib-dict-full"/> <package name="fillup"/> <package name="gdbm"/> <package name="info"/> <package name="insserv"/> <package name="make"/> <package name="mingetty"/> <package name="Mesa"/> <package name="sles-release"/> <package name="pam"/> <package name="pam-modules"/> <package name="perl"/> <package name="perl-Bootloader"/> <package name="permissions"/> <package name="python"/> <package name="python-base"/> </packages>

The issue occurs in the following code:
def __packages_marked_for_deletion_to_profile(self): # kiwi_delete self.dot_profile['kiwi_delete'] = ' '.join( self.xml_state.get_to_become_deleted_packages() )

This is the content of self.xml_state.get_to_become_deleted_packages() during build:
[ 187s] ['dracut', 'cracklib-dict-full', 'fillup', 'gdbm', 'info', 'insserv', None, 'mingetty', 'Mesa', 'sles-release', 'pam', 'pam-modules', 'perl', 'perl-Bootloader', 'permissions', 'python', 'python-base', 'make', 'module-init-tools', 'make']

Somehow the sixth value in the list has been replaced by 'None'. At the None positition is actually 'make' according to config.xml. It also looks like 'make' has been inserted inside the list multiple times (see above).

System info:
OBS: 2.7 beta (2.6.91.git20160511.e60af6b)
Kiwi: 8.14.1
Host OS: SLES12 SP1

Any ideas?

Steps to reproduce the behaviour

Building any image in OBS.

Problem description

Kiwi8 appliance lack /etc/hosts (and a few other files perhaps).

Actual behaviour

/etc/hosts is left as /etc/hosts.rpmnew when centos package "setup" is installed.

Steps to reproduce the behaviour

Create appliance (perhaps CentOS is the only install that exhibits the behavior), observe that /etc/hosts is non-existent, with /etc/hosts.rpmnew present.

Kiwi7 logs showed logic that moved /etc/hosts.rpmnew back into /etc/hosts.

support for the ppc architecture is missing. The good news are, k0da is working on it 👍

I've found a bug when you have a system with RAID devices:

FAIL: test.unit.raid_device_test.TestRaidDevice.test_create_degraded_raid
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/repos/GH/kiwi-1/.tox/3.4/lib/python3.4/site-
packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/local/repos/GH/kiwi-1/.tox/3.4/lib/python3.4/site-
packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/local/repos/GH/kiwi-1/test/unit/raid_device_test.py", line 47, in 
test_create_degraded_raid
    '/dev/some-device', 'missing'
  File "/local/repos/GH/kiwi-1/.tox/3.4/lib/python3.4/site-
packages/mock/mock.py", line 948, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/local/repos/GH/kiwi-1/.tox/3.4/lib/python3.4/site-
packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "<string>", line 2, in raise_from
AssertionError: Expected call: run(['mdadm', '--create', '--run', '/dev/md0', 
'--level', '1', '--raid-disks', '2', '/dev/some-device', 'missing'])
Actual call: run(['mdadm', '--create', '--run', '/dev/md3', '--level', '1', 
'--raid-disks', '2', '/dev/some-device', 'missing'])

(was: Remove bin/kiwi (autogenerated))

At the moment, there is a bin/kiwi. This script is automatically generated by running setup.py by the console_script statement.

Hi! I want to test new kiwi version, but I can't download package from OBS.
http://download.opensuse.org/repositories/Virtualization:/Appliances:/Builder/openSUSE_Leap_42.1 - 404 Not Found

Problem description

Currently kiwi 8 creates a directory in /var/cache and puts downloaded rpms in there. We are wanting to contain all the files that gets created during the kiwi appliance creation process within the targetdir so that we manage them appropriately. Please provide an option to override the default behavior.

Actual behaviour

Steps to reproduce the behaviour

verbatim from kiwi google groups:

Actually the package manager cache is to share data for all builds on
the building host, thus I had planned to keep this as a fixed location.
However it is possible to use another location for it, there is just
no switch to change from the default place implemented

If you need that please open an issue on the github issue page

Regards,
Marcus

Again just one of my crazy ideas. 😁 To be honest, I'm not 100% sure if this is really a good idea, so treat it as non-important and something for the future. Feel free to close it. 😄

The basic idea is to use more (sub)packages. Packages in Python are directories which contains a __init__.py file.

For example, you have lots of filesystem_*.py files. The same applies to bootloader*.py, container*.py, and others. For me, that cries for refactoring it into (sub)packages.

Let's assume for the moment, the idea has any value and we want to refactor all the filesystem*.py files. In that case, we could do this:

1. Create the directory filesystems (or use the singular form, doesn't matter for the time being).
2. Move filesystem.py to filesystems/__init__.py (or better leave __init__.py empty and use a different name).
3. Move the other filesystem_<NAME>.py files to filesystems/<NAME>.py.

There you have it, a filesystems (sub)package! :-)

What do we gain?

• The kiwi directory doesn't look so crowded after the restructuring.
• Maybe a little bit easier to type ("Readability counts"¹).
• Creates a filesystems "namespace" instead of packing everything into a file ("Namespaces are one honking great idea -- let's do more of those!"¹)
• No special file naming schema is needed, just add everything under filesystems and you're done.
• Probably easier to collect all the relevant files under the filesystems directory (think of a plugin systems).
• Easier to extend.

¹ cited from import this

Disadvantages:

• All(?) tests need to be examined and probably rewritten
• Problem with circular dependencies? I don't know.
• Other things...?

Ok, that's all. ;-)

I found a blog post entry on GitHub about Issue and Pull Request Templates which links to the help page.

Basically, you create a .github/ directory and add some files which acts as templates for how to contribute to the project, for issues and pull requests.

It's a minor issue, but I think it's really worth.

Hi!

It would be great to add custom menu entries, but I don't know how exactly do this. Now I have some patches for kiwi7 to add some custom entries for iso image. But I think that add this functional to xml description would be easy.

Requirements file(s) (usually requirements.txt) are used to setup your virtual environment. These are used for Travis, Tox, and setup.py.

Usually I split them into two files:

• the requirements.txt file for basic requirements which is need to run the software, and
• and some extended requirements for development (for example dev-requirements.txt or something else)

In Kiwi8, it appears that the vmconfig-entries are ignored.

The following works in Kiwi7:
`

            <!--
                Configure four total virtual CPUs, with 2 per socket.
                The guest OS will see two processors with two cores.
                -->
            <vmconfig-entry>numvcpus = "4"</vmconfig-entry>
            <vmconfig-entry>cpuid.coresPerSocket = "2"</vmconfig-entry>
        </machine>`

This results in the following in the output VMX file of Kiwi8:
numvcpus = "4" cpuid.coresPerSocket = "2"

In Kiwi8, these values aren't present.

Mark Scott.

Problem description

I really want to see overlay fs support for oem image type.
Additionally I think that there is no reason to use kiwi initrd for it (as kiwi 7 does), because it is possible to write overlay config to fstab: https://wiki.archlinux.org/index.php/Overlay_filesystem

Actual behaviour

No support for overlay.

Feel free to ask for help :)

root on btrfs requests to have /boot on separate partition. Makes kind of sense but then /boot have btrfs, which doesn't makes sense at all.

Either btrfs shouldn't request /boot or ext[23] should be used there

Fedora uses the DNF package manager which is a yum successor

The legay version of kiwi provides the KIWIRuntimeChecker module which does a bunch of pre checks before the actual build starts. This version of kiwi does not implement this

Problem description

The default compression in mksquashfs is gzip. The other compression
(xz) gives a size reduction of almost 10%.

Actual behaviour

gzip compression by default, faster in filesystem creation but larger result than with xz

Solution suggestion

call mksquashfs with -comp xz

Problem description

legacy kiwi code base contains support for the vdi image format. It has turned out the format is used by the buildservice team. Thus it should be available in the next generation kiwi

Actual behaviour

format="vdi" results in an unknown format type

There is the section which is used to create the config. file in a kiwi pxe image setup. So far the code to create the file from the section data is missing in this version of kiwi

There is the question if the support for this is still required or if we better add a good documentation with an example. I vote for documenting it

for buildservice builds we need the following information

rpm --root $path -qa --qf \"$query\" &> $idest/$name.packages
rpm --root $path -Va &> $idest/$name.verified

still missing in this version of kiwi

Problem description

building a gce image with format="gce" failed to build due to a broken image reference file

Actual behaviour

build fails

"cp: stderr: cp: cannot stat ‘/tmp/myimage/LimeJeOS-Leap-42.1.raw’: No such file or directory"

Steps to reproduce the behaviour

build with format="gce" enabled

Problem description

When building a Cloud Foundry stemcell we need to have the grub entries password protected for editing, but at the same time they need to be bootable without the password. This can be achieved by adding --unrestricted to all the menuentries.

Actual behaviour

We currently have to add the --unrestricted option to etc/grub.d/10_linux like this:

sed -i 's/CLASS="--class gnu-linux --class gnu --class os"/CLASS="--class gnu-linux --class gnu --class os --unrestricted"/' etc/grub.d/10_linux

(Sorry, accidentally opened this issue on the openSUSE/kiwi tracker at first. Maybe that one should be disabled?)

When trying to boot a hybrid ISO with persistent storage that was created with firmware="bios" in qemu with the read-only ISO in a virtio drive, LinuxRC tries to partition the disk with fdisk (which does nothing) and consequently waits for the non-existent persistent partition to appear.

In the firmware="bios" case, the devices /dev/vda and /dev/vda2 exist and LinuxRC tries to create the persistent partition as /dev/vda1. This fails because the device is read-only.

Possible workaround: Check /sys/class/block/$(basename $device)/ro for whether the device is actually writable, and disable persistence if it is not.

Problem description

We have a new feature in Tumbleweed and SLES12SP2 grub2 which allows us to switch to text output with a simple key press. This is useful for environments, where grub2 runs in uEFI mode, finds graphical output and emits its menu there, leaving the serial console completely unused.

Actual behaviour

We want to have these bits in the grub.cfg if we're targeting an EFI system:

[before the gfxmenu module gets loaded]
echo "Please press 't' to show the boot menu on this console"

[after all other menu entries]

On EFI systems we can only have graphics or serial, so allow the user

to switch between the two

hiddenentry 'Text mode' --hotkey 't' {
set textmode=true
terminal_output console
}

Steps to reproduce the behaviour

Try to boot with "qemu-system-x86_64 -serial mon:stdio -bios ovmf " and verify that there's nothing useful on the serial console. With the changes, you should get a message saying "Please press 't' to show the boot menu on this console" and be able to press the "t" key on serial to switch to text mode and thus have the menu shown on both graphical and serial output.

If kiwi doesn't use gfxmenu, this whole thing is moot, since they we already see the menu on both outputs.

There is a nice tool which is named bumpversion. Can be used to raise versions with one command.

See example: https://github.com/openSUSE/docmanager/blob/develop/.bumpversion.cfg

Problem description

The documentation for kiwi consists out of the manual pages and the api documentation.
However there is no mechanism in place to create the schema documentation

Solution suggestion

It would be nice to implement something like:

• add inline schema documentation in the kiwi/schema/kiwi.rnc file using ReST markup
• write an application which is able to extract the ReST markup from the RelaxNG schema in a form that python-sphinx can handle it. Ideally there is already a module in place which could do that but I could not find anything in the sphinx area for this
• integrate that source in the doc/ processing and auto publish like the rest of the docs

Problem description

Schema has at least a definition for an <except> section that is a legacy from the previous kiwi version (v7.x). It should be deleted.

Problem description

References to: https://github.com/openSUSE/kiwi/issues/574

Just came to my mind, how about to make kiwi (v7 if possible & v8) to wait for the loopback (/dev/loopX) device before attempting to write the bootloader section in it?
This way the person who builds the image does not need to restart the whole build process again, but instead, he could clearly see the evidence and intervene (e.g. to clean-up loopback devices).

And in automation (e.g. Docker / Jenkins jobs, ...), it would be really nice if the jobs could actually wait (with the timeout ofc) for the loopback device to be free. That way we would not be afraid to run Kiwi in parallel without being afraid of the loopback device exhaustion.

Regarding the modular approach, it would be really nice in such scenario, when one builds the image, but Kiwi fails to write the bootloader (due to grub vs. FS-type/feature problem, free loopback device timeouts, etc..), and then one could simply do something like kiwi --root path/to/image/root-tree --write-grub path/to/image.raw instead of waiting the relatively long process until he gets the image.raw file.

Actual behaviour

If no free loop device exists kiwi stops processing with an exception

Steps to reproduce the behaviour

Run a build with no free loop available on the build system

Problem description

Calling system prepare or build does not allow do add/delete packages as it is possible in the legacy kiwi version

Actual behaviour

An additional system update call is needed to add/delete packages

I think, the (most?) Pythonic way is to use tox to build all the necessary targets.

One of the benefits is that it fits well into a virtual environment.

When Kiwi is considered as stable enough, you could think of publishing the documentation to github.io as an idea. The doc for Kiwi would then be accessible through http://suse.github.io/kiwi.

All the necessary steps are described in the article Creating Project Pages manually.

Probably there are methods to make it semi-automatic.

Update: I've found travis-sphinx which could be useful.

Problem description

The Path class in kiwi/path.py uses Command to create and remove directories. The implementation uses shell commands which isn't very pythonic.

Suggested Solution

This is just a crazy idea and I don't know if it is useful at all (read: more cons than pros).

To avoid calling shell commands in Path we could use the py module. This contains the py.path.local class which is an object oriented view on files and directories (see documentation for more information.)

To get the functionality on openSUSE install the python3-py package. It comes also with some other nice features which could be beneficial for KIWI.

Advantages

• OOP view on files and directories which simplifies some code
• No more os.path.join, just path / file
• It's pythonic! 😀

Downsides

• Code needs to be rewritten at some places
• Test cases needs to be refactored
• Another dependency

@schaefi Do you think this is worth the effort?

The legacy version of kiwi provides an OVF writer module which is incomplete and broken in many parts of the created configuration file. Actually it should not be the task of the image builder to create the ovf metadata

There is the openovf project which seems also outdated and due to a missing upstream alternative this version of kiwi does not provide ovf support

At the moment only a kiwi initrd can be used to drive the very first boot. The kiwi initrd contains a lot of features which are e.g not required to boot a simple cloud instance. For this purpose it would be nice to
add a BootImageDracut class which uses a dracut generic initrd

Problem description

Guests which use SELinux (such as Fedora and Red Hat Enterprise Linux) require that each file has a correct SELinux label.

Actual behaviour

kiwi does not produce images with labels on files

Solution pointer

http://libguestfs.org/virt-builder.1.html#selinux

fixfiles should be our friend

Due to the ovf issue #14 there is also no support for the vagrant format for the virtualbox provider

The vagrant format for the libvirt provider (with qcow2 as disk format) could be implemented but only if there is a demand for it. So far I'm under the impression that there are not many users for this target

Because of this reason vagrant is missing in this version of kiwi

With rpm-ostree, you can create versioned filesystem trees to provide atomic, reliable upgrades. "git-like" management of appliances would be pretty sweet.