How to categorize this issue?

/area os robustness
/kind enhancement
/priority normal
/os suse-chost

What would you like to be added:
Currently, this extension is supporting two different bootstrap files format - script and cloud-init.
On some cloud providers, only one of them is supported, but there might be a Gardener landscape with providers that do not support common format.

This extension can be enhanced to support format per worker pool, e.g. via .machine.image.providerConfig. In this way, the very same controller will be able to generate bootstrap files for both script and cloud-init enabled platforms.

Why is this needed:
To use a single controller registration in a gardener landscape, with cloud providers that do not support common bootstrap format.

Hello,

We are seeing more and more nodes not being able to join the shoot cluster in reasonable time (under the 20mins default time)...only suse based OS are affected.

Sample error:
task "Waiting until shoot worker nodes have been reconciled" failed: Error while waiting for Worker shoot--hc-can-ns2--407-e3-hdl/407-e3-hdl to become ready: error during reconciliation: Error reconciling worker: Failed while waiting for all machine deployments to be ready: 'machine(s) failed: 1 error occurred: "shoot--hc-can-ns2--407-e3-hdl-iq-large-v2-z1-6d59f-76dbq": Machine shoot--hc-can-ns2--407-e3-hdl-iq-large-v2-z1-6d59f-76dbq failed to join the cluster in 20m0s minutes.'

Logs from an affected node:
2022-06-07T17:56:01.328267+00:00 localhost cloud-init[1521]: Cloud-init v. 21.2-8.51.1 running 'modules:final' at Tue, 07 Jun 2022 17:55:37 +0000. Up 31.54 seconds.
2022-06-07T17:56:01.328446+00:00 localhost cloud-init[1521]: Cloud-init v. 21.2-8.51.1 finished at Tue, 07 Jun 2022 17:56:01 +0000. Datasource DataSourceEc2Local. Up 55.11 seconds
2022-06-07T17:56:01.340849+00:00 localhost download-cloud-config.sh[2243]: Could not retrieve the shoot access secret with name cloud-config-downloader with bootstrap token
2022-06-07T17:56:01.341162+00:00 localhost systemd[1]: cloud-config-downloader.service: Main process exited, code=exited, status=1/FAILURE
2022-06-07T17:56:01.341792+00:00 localhost systemd[1]: cloud-config-downloader.service: Unit entered failed state.
2022-06-07T17:56:01.341959+00:00 localhost systemd[1]: cloud-config-downloader.service: Failed with result 'exit-code'.
2022-06-07T17:56:01.363888+00:00 localhost systemd[1]: Started Execute cloud user/final scripts.
2022-06-07T17:56:01.364931+00:00 localhost systemd[1]: Reached target Cloud-init target.

The symptom is that the affected nodes are unable to retrieve the shoot access secret with name cloud-config-downloader with bootstrap token. This is required as part of the “download-cloud-config.sh” bootstrap. One of the bad nodes had the text “<<BOOTSTRAP_TOKEN>>” inside the file that is supposed to contain the actual token (/var/lib/cloud-config-downloader/credentials/bootstrap-token).

Userdata (snippet) from good node:
mkdir -p '/var/lib/cloud-config-downloader/credentials'
cat << EOF > '/var/lib/cloud-config-downloader/credentials/bootstrap-token'
383273.kfxv0fjyl4l67vrz
EOF

Userdata (snippet) from bad node:
mkdir -p '/var/lib/cloud-config-downloader/credentials'
cat << EOF | base64 -d > '/var/lib/cloud-config-downloader/credentials/bootstrap-token'
PDxCT09UU1RSQVBfVE9LRU4+Pg==
EOF

gardener/machine-controller-manager#351
gardener/gardener#3898

“In the proposed new flow the worker controller (here MCM) creates a temporary bootstrap-token for each created vm.
If the featureFlag BootstrapTokenForVMs is enabled a file with the content "<<BOOTSTRAP_TOKEN>>" is added to the operatingsystem-config. It is passed to the worker controller. The worker controller generates a temporary token for every new worker instance(node). It replaces the "<<BOOTSTRAP_TOKEN>>" string with the created token and adds it to the user-data placed on the vm on startup.
The cloud-config-downloader(original user-data) will then refer to the new temporary bootstrap-token in the kubelet-bootstrap script.”

(I see a note in #41 about not being able to successfully test a requirement specifically relating to suse os, not sure if this is related but the majority occurrence is on suse os).

According to the diagram in one of the links pasted above, MCM is supposed to write the token directly to the fs...meaning the userdata will always contain the placeholder text (<<BOOTSTRAP_TOKEN>>) but this is not the case; the userdata has the actual token.

Please clarify, thanks.

What would you like to be added:
MemoryOne OS allows using cost-effective non-volatile storage-class memory (SCM) and other high-performance storage devices to transparently replace expensive server DRAM and expand beyond their capacity limits. Read more here

In order to support MemoryOne based machines with the gardener, jeos/chost extension needs to generate following cloud-config for chost-memoryone[name to be decided] operating system.

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0
--==BOUNDARY==
Content-Type: text/x-vsmp; section=vsmp
system_memory=6x
mem_topology=2
--==BOUNDARY==
Content-Type: text/x-shellscript
#!/bin/bash
​
zypper addrepo https://download.opensuse.org/repositories/Archiving/SLE_15_SP1/Archiving.repo
zypper --non-interactive --no-gpg-checks install bzip2
​
mkdir /run/vsmp
wget "http://www.scalemp.com/downloads/others/EC2/latest/vsmp_installer.sh" -O /run/vsmp/vsmp_installer.sh
chmod 755 /run/vsmp/vsmp_installer.sh
/run/vsmp/vsmp_installer.sh in -tq
​
<STANDARD_CHOST_USERDATA_HERE>
​
--==BOUNDARY==

It would be nice to have the system_memory and mem_topology parameters configurable via shoot-resource unless it adds unnecessary complexity.

Currently if you download the official OpenSUSE JeOS image, you won't find the required packages installed there:

• docker
• socat
• jq
• nfs-client

Here is the list of packages installed in public JeOS.

From my perspective it's worth to add the code, which ensures that the above packages are installed.

In Kubernetes, pods can request and make use of memory in hugepages since a long time (Beta since K8S 1.10, GA since 1.14)

apiVersion: v1
kind: Pod
metadata:
  name: huge-pages-example
spec:
  containers:
  - name: example
    image: fedora:latest
    resources:
      limits:
        hugepages-2Mi: 100Mi
        hugepages-1Gi: 2Gi

Gardener is currently lacking the capability to configure the hugepages on operating system level.
Ideally, the configuration should be possible on Gardener in a uniform way across all operating systems.

Other K8S offerings, for example Amazon EKS, offer possibilities to call OS commands before the kubelet of a node is started (see preBootstrapCommands below).

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: test-hugepages
  region: eu-central-1
managedNodeGroups:
  - name: my-nodes-group
    instanceType: c5d.large
    desiredCapacity: 1
    minSize: 1
    maxSize: 2  
    volumeSize: 50
    volumeType: gp2
    preBootstrapCommands:
      - sudo echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

Such a configuration would enable hugepage configuration in a completely generic way.

As an alternative, a configuration that is handled by the OS-specific controllers of Gardener could offer the same functionality in a more controlled way.

See also the same issue for GardenLinux: gardener/gardener-extension-os-gardenlinux#58

What happened:

# disable the default log rotation
cat <<EOF > /etc/docker/daemon.json
{
  "log-level": "warn",
  "log-driver": "json-file"
}
EOF

https://github.com/gardener/gardener-extension-os-suse-jeos/blob/7a5050311b38edf709af3b1c432eab15721a238e/pkg/generator/templates/script/suse-jeos.template#L10

causes No such file or directory error message.

What you expected to happen:

Ensure that /etc/docker is created before redirecting to a path, or run cat after https://github.com/gardener/gardener-extension-os-suse-jeos/blob/7a5050311b38edf709af3b1c432eab15721a238e/pkg/generator/templates/script/suse-jeos.template#L37

/cc @vpnachev

How to categorize this issue?

/area os
/kind enhancement
/priority 3
/os suse-chost

What would you like to be added:
We would like to introduce a mutating webhook server which will mutate the MachineClass objects by introducing a new field called NodeTemplate or by overwriting it in the MachineClass based on the values supplied via providerConfig attribute in the worker section of the Shoot.

Why is this needed:
MCM & CA have been introduced with generic scale from zero feature which will now scale a nodeGroup from zero on any cloud provider. This is enabled using a new field in the MachineClass called NodeTemplate which is used by CA to calculate the node capacity to scale new nodes to accommodate pending pods.

This feature needs to be extended to offer the scale from zero capability to the ScaleMP team (which uses Suse cHost Operating System) because their runtime node capacities (cpu, memory & gpu) are different from the capacities offered by the cloud provider (meaning, the resource capacity of these machines will be higher than what we get from the cloudprofile). So the above technique doesn't work in a straight forward way. There should be a way to get the runtime node capacities of these ScaleMP machines & update the same in MachineClass' NodeTemplate field. So that CA rightly calculates the node template of the new nodes to be brought up based on the runtime node capacity (and not the ones as per the cloudProfile)

One way suggested by @rfranzke as per his comment is to introduce a mutating webhook server in the OS extension. See here

This ticket is being opened so as to understand the scope of this idea as well as implementation.

Notes:

• Tracking Umbrella issue of Generic Scale from zero here
• Tracking issue for this in CA here

cc: @dkistner @himanshu-kun

How to categorize this issue?

/area os usability
/kind enhancement
/priority 3
/os suse-chost

What would you like to be added:
Implement the requirements from https://github.com/gardener/gardener/blob/master/docs/proposals/10-shoot-additional-container-runtimes.md#design-details.

Why is this needed:
To enable containerd as CRI for SuSE CHost. Also needed for the docker deprecation gardener/gardener#4110

How to categorize this issue?

/area os
/kind enhancement
/priority 3
/os suse-chost

What would you like to be added:
Replace packr with native Go embed

Why is this needed:
Less external dependencies.

We observed several times that when using existing VPC on AWS with JeOS worker nodes, the VPC attributes EnableDnsHostnames and EnableDnsSupport needs to be true. Otherwise the worker nodes fail to join.
We could write down this specific detail.

What happened:

Note: this bug report is to investigate and document the findings for projectcalico/calico#2973. It is as of now unclear which component is to blame.

Problem is that the readiness check for felix fails (container in the calico-node pod) on nodes where calico typha is not running. Felix appears to work fine on nodes where typha is running locally. The problem does not appear to show up when typha is not used.

What is strange is that networking in the cluster appears to work fine and all the routes (at least in a small cluster) are created correctly by felix

What you expected to happen:

The readiness check for felix succeeds.

How to reproduce it (as minimally and precisely as possible):

Create a 2 node cluster on Azure with SuSE JeOS as operating system and calico typha. The readiness check will fail on the calico-node pod on the host where typha is not running.

Anything else we need to know?:

Environment:

• SUSE Linux Enterprise Server 15 SP1 4.12.14-197.21-default
• Azure

How to categorize this issue?

/area os
/kind enhancement
/os suse-chost

What would you like to be added:

For this extension, when using an OS image of type memoryone-chost, it is possible to supply configuration parameters by means of an OperatingSystemConfiguration.

apiVersion: memoryone-chost.os.extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfiguration
memoryTopology: "2"
systemMemory: "6x"

Right now, this configuration only supports the fields memoryTopology and systemMemory but both fields are strings. This comes with two drawbacks:

• Should someone provide a non-string input to one of the fields (i.e. an unquoted 2 for memoryTopology which will be treated as an integer), the extension will fail to unmarshal this YAML into its corresponding go structs. This happens fairly late in tthe Shoot reconcilation process ultimately causing it to fail and affects new as well as existing clusters
• These fields can be used to inject arbitrary values by just seperating them via ; (something like memoryTopology: "2;feature_enable=0x4" - we already saw that in the wild). Since these values ultimately end up in the user-data that gets supplied to the cloud-providers, not sanitizing the input is a potential security risk.

As a result, this extensions should provide an input validation that will reject an input that is invalid or potentially malicious so that it gets caught early and does not interfere with the reconcilation of Shoot clusters.

Why is this needed:

explained above

From https://github.com/gardener/gardener-extensions/issues/122

All make commands require various dev resources to be available:

• golang
• helm
• various gnu utils such as tar, gzip, tr, base64, cut, find, grep, xargs.
• gingo
• linters.

I think that the most reasonable way to get reproducible builds and things such as make generate is to run those commands in a pre-build container image.

What happened:
The docker service on the nodes is configured with

# cat /etc/docker/daemon.json
{
  "log-level": "warn",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

which enables the log rotation from docker side. On the other hand we have also logrotate which will try to rotate the same container log files - https://github.com/gardener/gardener/blob/v1.0.4/charts/seed-operatingsystemconfig/original/templates/logrotate/docker.conf.
Now it is very likely to hit an issue caused by conflict between docker log rotation and logrotate.
Basically currently logrotate relies that the docker log rotation is not enabled.

Environment:

• Gardener version (if relevant): v1.0.4
• Extension version: v1.3.0
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

What happened:

There is no autostart enabled for the docker service nor socket activation available in JeOS.

Generated compute instance user_data is the following:

... ... ...
until zypper -q install -y docker wget socat jq nfs-client; [ $? -ne 7 ]; do sleep 1; done
systemctl daemon-reload
ip link set dev eth0 mtu 1460
grep -q '^MTU' /etc/sysconfig/network/ifcfg-eth0 && sed -i 's/^MTU.*/MTU=1460/' /etc/sysconfig/network/ifcfg-eth0 || echo 'MTU=1460' >> /etc/sysconfig/network/ifcfg-eth0
ln -s /usr/bin/docker /bin/docker
ln -s /bin/ip /usr/bin/ip
if [ ! -s /etc/hostname ]; then hostname > /etc/hostname; fi
systemctl restart docker
systemctl enable 'cloud-config-downloader.service' && systemctl restart 'cloud-config-downloader.service'

It is executed only on the very first boot, and docker service is not activated:

# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: http://docs.docker.com

Besides other dependencies, e.g. kubelet rely on docker socket activation:

# systemctl cat kubelet
# /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet daemon
Documentation=https://kubernetes.io/docs/admin/kubelet
After=docker.service
Wants=docker.socket rpc-statd.service
... ... ...

However JeOS docker package doesn't contain the docker.socket unit file and this dependency just doesn't work.

What you expected to happen:

• Add docker.service into kubelet Wants directive
• Or add socket activated docker.socket unit file:

[Unit]
Requires=torcx.target
After=torcx.target
Description=Docker Socket for the API
PartOf=docker.service

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

How to reproduce it (as minimally and precisely as possible):

reboot jeos based VM

Anything else we need to know?:

Not sure why template tests contain docker service in the Units var. Therefore in tests this behavior looks file and docker service is enabled.

/cc @vpnachev @rfranzke