Collaborate with KIWI? about osbuild HOT 9 CLOSED

From my perspective, I think it might be worth considering if we can look at breaking out common classes that would be reusable across both tools into a common module so that we're not rewriting everything, even if the final image construction philosophy is somewhat different.

KIWI's "tasks" feel a lot like osbuild's "stages" in that they are aspects of a build that are composable and can be used in theoretically any order.

As the developer for livecd-tools, I'd also like to see if there's something I can contribute to this to make better, more unified, and reliable tooling.

from osbuild.

sorry for the late reply

KIWI's "tasks" feel a lot like osbuild's "stages" in that they are aspects of a build that are composable and can be used in theoretically any order.

tasks vs. stages have different responsibilities between the tools but yes from a design perspective I think osbuild shares the same ideas we have with kiwi. In the kiwi case the idea on the second attempt (kiwi-ng) was also to provide a python API such that everybody can use the parts of the implementation that matters for the project. So if there are projects that uses python and need parts of an imaging process, e.g filesystem creation, partition setup, loop management, etc etc the idea was that those people can just use "from kiwi import ..." and don't need to re-implement things again. From an API perspective this is documented here: https://osinside.github.io/kiwi/api.html

The implementation of the tasks in an image build process in kiwi are done using these interfaces. I guess on that API osbuild and kiwi already have some duplication. I can imagine that it will be not so easy from either side to perform any changes here because the best result you can get is the same as you had before ;) a working system

Maybe we can think of a collaboration for a common library that combines the best parts out of both projects into kind of a "libimage" project. Both, osbuild and kiwi could then in their own pace think about using/refactoring.

The biggest hurdle I see is as usual time and manpower. My time is currently very limited due to job tasks that keeps me busy. I won't have an issue to motivate myself though and would also enjoy to work together with the osbuild people

Just my2cents

Thanks

from osbuild.

Thanks Dan

As a followup on the idea to combine our efforts; here would be a simple example how you could use the loop code from kiwi:

import logging

from kiwi.storage.loop_device import LoopDevice
from kiwi.filesystem import FileSystem

loop_provider = LoopDevice(
    '/tmp/mydata.ext4', 100
)
loop_provider.create()

filesystem = FileSystem(
    'ext4', loop_provider, '/tmp/__some_data'
)

filesystem.create_on_device('LABEL')
filesystem.sync_data()

from osbuild.

For reference, the integration test build matrix:

• https://build.opensuse.org/project/show/Virtualization:Appliances:Images:Testing_x86
• https://build.opensuse.org/project/show/Virtualization:Appliances:Images:Testing_arm
  (s390 and ppc in the loop)

If you see overlap in your goals it would be great to collaborate.

Thanks much

from osbuild.

Hey!

Sorry that there isn't any high-level documentation yet. Here's the gist: osbuild tries to build images in a way that is

• comprehensible: It should be easy to see what happened during each step of the image building process. This is why osbuild has stages that encapsulate some modification of a filesystem tree. Their only input is a JSON record that fully describes configuration. The idea is that it’s easy to inspect which files a stage added, removed or modified by diffing it with the tree from the previous stage.This allows us to find out easily what various tools we’re using to build images are actually doing.
• reproducible: While we probably won’t get to bit-by-bit reproducibility for a long time to come, our aim is for a pipeline to specify the build process in such detail that the same pipeline will always produce a functionally equivalent image. We’re running stages and assemblers in a container that shares nothing with the host. This includes /dev, which is why we have that elaborate loopback API.
• extendable: We think there’s a lot of room for experimentation in this space (immutable images, new kinds of package sources, keeping built tools off the target image, getting rid of rpm scriptlets, etc). We hope that osbuild enables this kind of experimentation by allowing to plug stages together in different ways or insert new ones into existing pipelines.

I have looked at kiwi before and found that it isn’t as focussed on isolation and encodes knowledge about the full build process in code. I agree that there’s quite some overlap, probably mostly in the tools and libraries we’re using. There’s definitely potential to discuss and collaborate :)

from osbuild.

Thanks for the details. I agree to what you are saying. In kiwi we are focused on implementing full processes as tasks, e.g "build me this image" or "create me this new image root". The implementation of the process is based on api's e.g for handling filesystems, storage devices, image formats, bootloaders, etc etc.. which are all written as loosely coupled interface classes that can be used in a different scope. That's what I tried to express in the little code example above

Your approach and logic sounds interesting to me. However I think in order to implement your logic you would also need much of the core functionality to build an image. Things like dealing with loop devices, create and manage partition tables, setup bootloaders, handling of containers, etc etc will come to your hand and needs to be implemented. At that level there will be a lot of code duplication and I thought you might be interested in letting this done by an api class of kiwi.

I agree you would create a dependency to another project but maybe it is worth the effort as you could concentrate more on your logic being comprehensible/reproducible/extendable instead of adding all the ground work in addition.

I think there is a great chance to work together :)

from osbuild.

This is not really an issue, in the sense that we can do anything to fix it. More discussion is needed, but probably in a forum with higher bandwidth.

from osbuild.

This is not really an issue, in the sense that we can do anything to fix it. More discussion is needed, but probably in a forum with higher bandwidth.

Do we have a forum with higher bandwidth? For kiwi, we have kiwi:matrix.org... You're free to join us there to discuss if you want...

from osbuild.

@Conan-Kudo @schaefi @larskarlitski @dcermak Is there any following update for this proposal?

from osbuild.