I built a bit of tooling to package things up for torcx. Is there any clarity yet on whether torcx has a place in coreos' future? I'd like to either polish that up and share, or switch to whatever the new OS will use for the purpose.

It looks like we should tweak torcx-generator logic a bit more so that it doesn't globally fail on the first image that is not found or fails to unpack/propagate.

Instead we should:

• handle each archive separately (this should also be parallelized later)
• unpack and propagate per-image
• collect all errors and keep-going over all images
• if any error was encountered during apply, do not seal the system

This should help leaving the system in a more useful and simpler to debug state, while still marking the apply task as failed.

I submitted the initial service unit I'm using in #34. As part of system integration, we need to define to better define the "early-boot" story and capture it in devdoc.

I'm currently using this very weak anchoring:

WantedBy=sysinit.target
RequiredBy=basic.target

Other datapoints:

• torcx.service is a oneshot unit, so it'll block targets progression till the final sealing is done (or the service fails)
• some addon services handled via torcx will be part of the dependency graph (docker, containerd, rkt-api) so it would be better to extract them as early as possible
• torcx-apply doesn't fetch, so it doesn't strictly need a network'd target
• torcx-apply doesn't depends on dbus service/socket
• torcx-apply needs local-fs to be in place, in particular it relies on /tmp, /var and /run
• some users may provide images via a NAS volume, thus depending on remote-fs.target. I think this usecase can and should captured via an user drop-in

The outcome of this ticket would likely be an upper and lower bounds (as in targets and services) to place torcx.service in-between.

/cc @euank @crawford @marineam

Binaries installed via torcx are propagated into their own directory outside of normal path.

I'm currently installing the following /etc/profile.d/torcx-path.sh dropin via ignition:

# torcx drop-in: add torcx-bindir to PATH

export PATH="/var/run/torcx/bin:${PATH}"

But this integration could be improved to let systemd also be aware of it. Another open question is whether this should be prefixed or suffixed on top of existing path?

/cc @dm0-

The docker fetching code isn't that necessary, and the OCI spec is still beta.

This is an umbrella issue to improve the testing status. Things that need to happen:

• configure travis #32
• add unit testing, mainly to cover:
  • JSON handling #59
  • stores in-order walking
  • metadata production
• add functional testing (via rkt), mainly to cover: #66
  • CLI stability
  • apply phase: unpack / propagate / seal
• investigate integration testing via kola

localhost core # /go/src/github.com/coreos/torcx/bin/amd64/torcx profile use-image --name casey torcx-dockersuite-com.coreos.cl:com.coreos.cl
ERRO[0000] could not write new profile: stat /etc/torcx/profiles/casey: no such file or directory 

(needs .json suffix)

torcx is currently a dynamic binary, but we should investigate if it is possible to bring it back to a plain-static binary.

After initial investigation, the cgo bits seems to be coming from two of our dependencies:

• containers/image: it links agains libgpgme. This can however be disabled via a buildtag (containers_image_openpgp)
• viper: its dynamic configuration brings in cgo for inotify support. This doesn't seem to be configurable, but needs further investigation. We don't strictly rely on it, so we can possibly get rid of it alltogether.

Other: we may still need cgo for our own internal purposes (eg. mounting), in which case we'll just admit that torcx has to be a dynamic binary. This needs to be re-assed once we are close to features completeness.

torcx currently mounts its own tmpfs for unpacking purposes.

There are a bunch things to be re-considered in that regard:

• moving out of tmpfs: concern expressed by @philips and @dm0-, as we go forward this will increse memory pressure (plus, CL doesn't setup swap). However, at early-boot (generator) we may not have yet FS set up
• tmpfs options: we should expose options to tune tmpfs parameters to limit resource usage, in a format compatible with mount/systemd-mount. This is a low-priority wishlist for the moment and can be easily exposed via config.json.

We were aware of tmpfs concerns while designing this so torcx runtime paths are still easily switchable to somewhere else, but I don't have a clean alternative solution relying on local-fs right now. Going in this direction may require:

1. fiddling with local-fs.target so that it is reached in initrd before pivot_root happens and generators are run. @crawford is looking into the feasibility of this.
2. moving part of torcx into initrd, and make all mountpoints available for initrd (similar to what ignition does). This is similar to the above and adds some more complexity, so we opted away from it.

I'm also not sure what are the implications of PXE-booting for such discussion (do we have any guarantees of a RW storage in there?).

As per subject. The rationale is that:

• it makes clearer to user what's the format, and help editors with syntax highlighter
• it avoids scanning and picking up unrelated files (dotfiles, editor tempfiles, etc)

After this, the profile name will be as basename with .json suffix stripped.

There is a strange behavior with my custom image:
/usr/lib64/systemd/system-generators/torcx-generator[511]: time="2018-06-04T08:52:05Z" level=error msg="failed to unpack: gzip: invalid header" image=docker-custom reference=com.coreos.cl
It happens if I create archive like this:

archive.tgz:
  bin/
  ./torcx

When I add an extra dir like this:

archive.tgz:
  docker-custom/
     bin/
     .torcx/

generator extracts it fine but with this extra dir in path. How to fix it? Is there any guide how to prepare images?
b.t.w. I can extract both archives using tar without issues

Right now torcx-generator seals the system only at the end of a sucessful apply. This is used both to signal that a system has been properly set up, and to avoid performing propagation tasks again if the systemd gets daemon-reloaded.

This however doesn't work too well in the case where the generator fails for whatever reason and a later daemon-reload is performed. In this case, torcx will try to unpack addons again possibly on top of some already existing assets.

As this is a failure that we can't anyway solve at runtime, we should instead write a transaction-start entry to the FS to signal that torcx already run on this boot-instance (even if later failed).

First I want to start to say that we are still trying to reproduce the issue. Because we had this on production systems we did not had the luxury to do a lot of investigation before executing the workaround. A reboot solves the issue, so that is done as soon as possible after we are triggered with this issue.

The issue we have:
We have about 200 coreos systems which are automatically updated with new coreos releases. During the day the new coreos image is installed. Then we have the locksmith with the reboot strategy=reboot and a window between 02:00 and 04:00 AM.

During that reboot we see that there are issues unmounting /run/torcx/unpack.
Aug 03 02:05:26 slnc7r1306 systemd[1]: run-torcx-unpack.mount: Mount process exited, code=exited, status=32/n/a
Aug 03 02:05:26 slnc7r1306 systemd[1]: Failed unmounting /run/torcx/unpack.

then the shutdown continues and coreos is booting from the updated partition.

After the reboot docker wil not start and we can't do anything with docker. It will generates torcx error messages. After a reboot of the the system it works fine.

TO ADD
- log the exact message we see when running into this issue
- show states of the two torcx services
more?
TO ADD

Until now we encountered this issue randomly over the 200 systems. Not during every new coreos update, but during the most of the updates we random have 1 or 2 systems running into this issue.

I hope you can help us to find / solve this issue.

To my disappointment i learned that the torcx demo videos that has been published shows things not available to the users.

Lucab informed me there is no "userland" in the OS so one cannot reproduce the demo videos.

lucab [9:39 AM] 
there is no userland in the OS, yes

[9:39] 
but the generator is always running (edited)

https://asciinema.org/a/115034
https://asciinema.org/a/127682

Is there a ETA for when userland is added or does the user have to build CoreOS themselves to get the commands shown in the videos above?

Is it possible to override the version of the utility stored in the vendor image? F.e. I need to update the containerd verison?

Currently, torcx is meant to propagate all relevant assets (binaries, units, etc) found in the addon images by default.

This means that it scans the unpack directory for the usual paths (/bin/, /usr/bin/, /lib/systemd/system/`, etc.) and propagates everything found in there. This is done in order to make building torcx images via ebuild as easy as possible.

However, when building images with binaries+libs, I realized that spurious binaries may get dragged in (e.g. libseccomp bringing scmp_sys_resolver) and multiple copies may end up in different archives, causing conflicts/issues in profiles with multiple images built this way.

An alternative approach would be to reserve for ourself some dedicated paths for lookups (e.g. /torcx/bin/ for binaries, /torcx/service/ for services, etc.) where to symlink/copy assets to be explicitely propagated.

This is the usual implicit-vs-explicit story where I made an initial call for the implicit side but I now have some doubts. Moving to an explicit reserved path means more logic in the building phase, thus /cc @euank @crawford for inputs.

I think it would be a great idea to be able to ship kernel modules in torcx modules. Currently one can package the .ko file and insmod that in a systemd service, but complex modules (like ZFS on Linux or WireGuard) are a pain because of inter-module dependencies only properly handled by modprobe. modprobe doesn't work since the modules are not in the right path to be picked up by modprobe.

If we could extend torcx to symlink modules (difficult due to R/O /lib) or get modprobe to pick up torcx module that would be pretty cool. I don't have a very concrete plan for this yet, but just wanted to throw this out here to start a discussion which will hopefully lead to a better experience when deploying kernel modules on CoreOS.

The default BinDir is under /var/run/torcx/bin, and we rely implicitely on the fact that it is a tmpfs. While this is fine in the general case, there are other details which may depend on OS setup; in particular, /var/run may be nosuid+noexec by default.

As such, we should instead mount our own tmpfs for BinDir, allowing suid and exec. It also seems a good idea to turn it to RO while sealing the system, once the "propagate" phase is done.

From looking at the code, it appears that torcx properly handles a trailing newline when reading next-profile, but forgets to add one when writing it.

Archives may contain systemd units which need to be propagate into the system. To make this ephemeral, we'd use the runtime systemd directory /run/systemd/system/. Symlinking is preferred (but not sure if it works), otherwise copying to there is fine as well.

This is a tracking issue for an initial inclusion of torcx in the alpha channel of container linux.

This specifically tracks phase 0 with the scope of: the single default version of docker is packaged and shipped via torcx for the alpha channel.

This intentionally punts on packaging multiple versions or supporting the use of torcx beyond this one version.

The primary pieces to get ironed out here are building the docker package for torcx and adding that, plus torcx itself, to the alpha image.

@dm0- @lucab: this is meant as a tracking issue for the next alpha. If it's either too early or the wrong focus to try and shoot for this, please feel free to re-evaluate as appropriate.

The torcx directory on coreos/docs nonexistent as of commit coreos/docs@9390ff6

Is there a technical reason why 'next-profile' doesn't support multiple profiles to be stacked, e.g.:

• tectonic
• addon1
• addon2
• ...

The recorded warning is:

....... level=warning msg="no next profile: profile \"tectonic\\naddon1\\naddon2\" not found"

It would be easier to just append to next-profile, instead of creating a new combined custom profile with the contents from 'tectonic.json' and the additional addons to be layered into the new system.

Thank you for giving this consideration.

This is a feature request to add an additional sub-level to the store. Such stores will contain a version selector embedded in their part, which will be used by the generator to match against the currently booted OS version.

torcx currently processes a single profile, containing an ordered list of image names+references. This can be either a vendor (RO) or a user (RW) profile, but switching to a user profile makes opting into vendor updates/changes harder to track.

This problem is spelled out in open-questions docs where we proposed either some arbitrary chaining (via includes directive) or just a flat profile but introducing well-known vendor references.

After some discussion here we distill a middle-ground solution, consisting of an implicit 2-layers profile overlaying: the lower layer is the vendor one, which is always applied if present; the upper layer is an optional user layer, which is merged (overriding) on top of the vendor one.
The final result is an ordered list of images where:

• the one specified in vendor profile comes first (lower priority)
• an image specified in the user profile overrides the same if present in vendor profile
• an empty/nil version tag in the user profile means that the archive

This introduces some implicit/merging state, so few thoughts come to mind:

• when we record the profile which we just applied, do we store the user profile or final flattened list? (i.e. are we more interested in making this re-usable later as-is or more in capturing the full current state?) - IMHO the latter
• nil-versioning an image can become stale (eg. when we remove archives from the distribution), should the merging alogrithm be relaxed to skip such cases? - IMHO yes, but reduces strictness

Brainstorming ticket, open for feedback! /cc @euank @crawford @dm0- @squeed

This is a bit funny, because it means that torcx packages now leave a permanent footprint on the system, but we couldn't think of any better alternative.

Hi,

I'm trying the new 1492.1.0 alpha since I had problems running stable and beta on our Xenserver 7.1 (NMI problem) to setup a k8s test cluster.

I'm running the coreos installer with the cloudconfig I used before, but when I need to restart dockerd with systemd it fails with the following message:

Aug 03 10:03:15 nestene systemd[1]: Starting Docker Application Container Engine...
Aug 03 10:03:15 nestene env[5633]: /usr/bin/env: '/run/torcx/bin/dockerd': No such file or directory
Aug 03 10:03:15 nestene systemd[1]: docker.service: Main process exited, code=exited, status=127/n/a
Aug 03 10:03:15 nestene systemd[1]: Failed to start Docker Application Container Engine.

Is there a simple way to disable torcx or to fix this ?

Thanks

In the initial phase torcx was in under an experimental private repo, which should now go public.

Things to refresh before toggling the switch:

• reformat README to be user-friendly
• include a short demo/recording
• align devdocs with latest changes regarding store/archives format
• check that licenses/copyrights are in place

(This issue popped up during offline discussions with @lucab, so I'll write up here not to forget about it. At the moment for me this issue is not the highest priority.)

Torcx doesn't seem to work well with nspawn containers. Normally torcx wants /etc/torcx/next-profile to be specified for loading a user-defined profile. When the system then reboots, the next profile gets detected, tarball gets unpacked. But in nspawn containers, no matter how many times a container reboots, the next-profile never gets detected. Nothing but the predefined system-wide profile docker works for now. OTOH the next-profile mechanism works fine in a pure virtual machine.

It's also not trivial for me to define system-wide profile, because the docker profile is defined in coreos-overlay, which 3rd-party users won't be able to touch.

See also kinvolk/kube-spawn#105 (comment)

here was an issue

This captures an implicit requirement from #46 (comment).

In order to reason about addons vendored with an OS images (which are arbitrarily updated, added or removed), it would be better to have the vendor profile (and a list of all available images name+reference) published out-of-band alongside with the OS.

In that way, higher level drivers (humans or operator) can verify that the final configuration of vendor + custom addons satisfies their requirements.

/cc @dm0- @euank

With the two-layer profile model, we'd like to have the vendor profile vary depending on the host platform. Since the vendor profile is written to read-only /usr, we can't modify the vendor.json file directly. We probably don't want to use the user profile configuration in /etc, since users would overwrite it when configuring the system with their own torcx profile.

For a concrete example, GCE systems have several systemd services that are only installed on GCE images in the writeable OEM partition mounted at /usr/share/oem. (Currently, they are in a rkt ACI.) We'd like these GCE agent services to continue to be run by default in torcx as if they were in the vendor profile, but only on GCE systems, where /usr/share/torcx must be the same as every other platform.

So we'd like to move the nodes in our kube cluster to docker 17.03 so we can implement some pod security policies that 1.12 doesn't support and 18.03 is problematic because it's not certified by kubernetes.

I see that 17.03 seems to be referenced with some stability in the manifests returned from: https://tectonic-torcx.release.core-os.net/manifests/$COREOS_RELEASE_BOARD/$COREOS_RELEASE_VERSION/torcx_manifest.json

However that 17.03 tarball isn't packaged with the release so we need to prefetch it (for the release version) pre torcx and stick it into /usr/share/torcx/store ... no problem we could do that with ignition we know what ami we are requesting. However we currently use the default locksmith/update engine which means that 17.03 tarball's contents could drift as we move from version to version.

I'm not a huge systemd guy, but the docs say that all generators are run in parallel, so if I wrote a binary to snag the proper tarball and put it into place is there a way to delay the torcx run until after that happens?

I can't be the first person to run into this? Is the solution to pin the version of coreos, deliver the proper tarball via ignition and manage the upgrades through the upgrade operator (and have it upgrade the tarball before it reboots the node?)...

Capturing a comment from coreos/coreos-overlay#2567:

There are other minor annoyances like the generator writing errors to the journal every daemon-reload.

This likely needs to happen after #50, so that we just emit an info message when the sealing process was already started.

Hi!
Is it possible to add an opportunity to set torcx bin directory via config.json?

Hi! I've created a torcx image for the conntrack binary, but it doesn't appear in the /bin directory.
manifest.json:

{
  "kind": "image-manifest-v0",
  "value": {
    "bin": [
      "/bin/conntrack"
    ]
  }
}

/etc/torcx/profiles/conntrack.json:

{
  "kind": "profile-manifest-v0",
  "value": {
    "images": [
      {
        "name": "conntrack",
        "reference": "com.example.test"
      }
    ]
  }
}

/etc/torcx/next-profile:
conntrack

image file:
/var/lib/torcx/store/conntrack:com.example.test.torcx.tgz

sudo journalctl -xe:

level=debug msg="image unpacked" image=conntrack path=/run/torcx/unpack/conntrack reference=com.example.test
level=debug msg="binaries propagated" assets="[/bin/conntrack]" image=conntrack path=/run/torcx/unpack/conntrack reference=com.example.test
level=debug msg="profile applied" sealed profile=/run/torcx/profile.json upper profile=conntrack

ls /run/torcx/unpack/conntrack/bin:
conntrack

ls -la /bin:

-rwxr-xr-x.   1 root root    43072 Jul 28 18:43  cksum
-rwxr-xr-x.   1 root root    10112 Jul 28 18:26  clear
-rwxr-xr-x.   1 root root    34792 Jul 28 18:26  cmp
-rwxr-xr-x.   1 root root    30664 Jul 28 19:48  col
-rwxr-xr-x.   1 root root    14280 Jul 28 19:48  colcrt
-rwxr-xr-x.   1 root root    22472 Jul 28 19:48  colrm
-rwxr-xr-x.   1 root root    51152 Jul 28 19:48  column
-rwxr-xr-x.   1 root root    47200 Jul 28 18:43  comm
lrwxrwxrwx.   1 root root        6 Jul 28 21:06  containerd -> docker
lrwxrwxrwx.   1 root root        6 Jul 28 21:06  containerd-shim -> docker
-rwxr-xr-x.   1 root root    34784 Jul 28 20:01  coredumpctl
-rwxr-xr-x.   1 root root  6481792 Jul 28 19:53  coreos-cloudinit
-rwxr-xr-x.   1 root root    29599 Jul 28 20:05  coreos-install
-rwxr-xr-x.   1 root root  4401976 Jul 28 19:37  coreos-metadata

which conntrack:
which: no conntrack in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin)

There is no conntrack binary(((

What am I doing wrong?

Before moving to generator, torcx apply used to have its own service whose status was used to signal sucsess/failure of torcx itself. As we don't have such a service, there is no quick way for the user to know when torcx setup fails.

CL should probably provide a small helper service/target which check for the presence of the seal file under /run/metadata/torcx and report success/failure appropriately. Bonus point, some way of bridging logs from the the generator run (they are already in journald, just not attached to any service).

@dm0- assigning this to you to solve as you think it suits better. I think it can be done with a single bash test-ifelse-exit, but we can also add a torcx status if you need some more complex logic.

Hi,

Is there any way to mount and use docker's socket into a container? I tried it but when I call docker I get this message:

The program docker is managed by torcx, which did not run.

I read this but it doesn't say anything about the socket but it is a transient unit
https://groups.google.com/forum/#!topic/coreos-user/ilC6l0X5Hnw

Thank you!

After spinning up a new server on Google Cloud Platform using CoreOS current stable image, I am unable to download some images using google/cloud-sdk image:

core@server ~ $ docker run --net=host -v /home/core/.config:/root/.config -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker google/cloud-sdk gcloud docker -- pull gcr.io/<valid full path>
The program docker is managed by torcx, which did not run.