The OCP installation / bootstrapping process can be quite taxing for the target host, especially if the target host is running on a shared zSystems / LinuxONE server. Collecting system performance data while the OCP installation is running might help in uncovering and addressing potential bottlenecks.

The relative links to other documentation files in the main DOCUMENTATION.md markdown file are broken and need to be fixed.

The automatic compilation of binaries using 'go' should make use of a proper GOPATH setting. Currently all non-standard go libraries / binaries are bing put into ~root/go which is not ideal.

Currently some of the long-running playbook tasks (especially the cluster installation task) do not display any meaningful message on the console besides the default 'FAILED - RETRYING: wait until... (XX retries left).' message. This is not particularly helpful for the user and potentially misleading due to the inclusion of the 'FAILED' keyword in the message.

A better alternative would be to display a custom message to assure the user that everything is alright and things are progressing nicely. For this, the following Ansible callback plugin could be used: https://docs.ansible.com/ansible/latest/collections/community/general/diy_callback.html

For examples on how to customize the messages for 'until' loops with the help of this plugin see here: ansible/ansible#32584

The playbooks should support Fedora 36 as KVM host OS (in addition to Fedora 35, RHEL 8.x and RHEL 9.0).

The playbooks need to address the following:

• add ~root/go path to list of artifacts that are to be cleaned up
• change all usage of 'yes' and 'no' in playbooks (YAML files) to 'true' and 'false' respectively (for compatibility with YAML linters)

The .gitattributes file of this repo should be updated to allow GH to detect Markdown documents properly.

I am unable to install pre-release (aka nightly) builds of OCP. The OpenShift installer fails with this error message:

fatal: [myhost]: FAILED! => changed=true
 cmd:
 - /usr/local/bin/openshift-install
 - create
 - manifests
 - --dir=/root/ocp4-workdir
 delta: ‘0:00:00.054399’
 end: ‘2022-06-30 09:12:53.846745’
 msg: non-zero return code
 rc: 3
 start: ‘2022-06-30 09:12:53.792346’
 stderr: ‘level=error msg=failed to fetch Master Machines: failed to load asset “Install Config”: failed to create install config: invalid “install-config.yaml” file: platform: Invalid value: “libvirt”: must specify one of the platforms (alibabacloud, aws, azure, baremetal, gcp, ibmcloud, none, nutanix, openstack, ovirt, powervs, vsphere)’
 stderr_lines: <omitted>
 stdout: ‘’
 stdout_lines: <omitted>

Ideally the playbooks support the installation of official OCP releases as well as pre-release builds.

THe backup of the original /etc/resolv.conf file that is created by the playbooks gets overwritten in subsequent playbook runs.

The existing documentation seemingly doesn't answer some questions users have about the playbooks or the OpenShift installation in general, e.g.:

• is offline installation (air-gapped) supported (answer: no, not when using these playbooks)
• can I install multiple OCP clusters on the same KVM host (answer: no, not when using these playbooks)
• can I install an OCP cluster that spans multiple KVM hosts (answer: no, this topology setup is not supported by OCP libvirt / KVM IPI)
• can I use a different type of networking setup, e.g. macvtap or openvswitch (answer: no, different networking options are not supported by OCP libvirt / KVM IPI)
• does the OCP cluster installed on my KVM host survive reboots of the host (answer: yes, use the included playbooks to start the cluster nodes after a host reboot)

A dedicated FAQ document could be helpful for these types of questions.

A CODEOWNERS file needs to be established for this repository in order to streamline future contributions by the community.

The 'butane' binary (see also: https://coreos.github.io/butane/) can be used to generate OpenShift-compliant Ignition Configs. For developers tweaking RHCOS / OpenShift cluster node settings in a live environment this might be handy to have available on the KVM host out-of-the-box.

Red Hat OpenShift Container Platform supports multiple internal network types out of the box: OpenShiftSDN and OVNKubernetes with OpenShiftSDN being the default. Users should be able to override the default and install a cluster using network type OVNKubernetes.

The built-in sanity checks are currently accounting for a CPU overcommit factor of 2. On IBM Z / LinuxONE the max recommended CPU overcommit factor for KVM is 10. This should be changed.

Installing the latest Python3 packages as well as updating existing Python3 packages proves complicated as there are cross-dependencies between packages (pip <-> pyOpenSSL) which can easily break.

Hence the playbooks should lock down all Python3 packages that are installed via pip.

The ubi8/python-38 image will reach end-of-life in 03/23 (see also: https://catalog.redhat.com/software/containers/ubi8/python-38/5dde9cacbed8bd164a0af24a).

So moving the base image to ubi8/python-39 far ahead of time seems like a good idea.

The following things should be addressed to improve the development workflow and to streamline the contribution process:

• enable DCO bot usage (https://github.com/probot/dco)
• make sure YAML files are properly detected as such via corresponding .gitattributes settings

The Python package openshift is broken for version 0.13.0 - it cannot be installed:

pip3 install -U openshift
Collecting openshift
  Downloading https://files.pythonhosted.org/packages/12/1d/914bfdcc8e1e3b9b88b050d77b934146831b5fe320ae238e3e1813e40fa5/openshift-0.13.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-fk8w5it6/openshift/setup.py", line 49, in <module>
        install_requires=extract_requirements('requirements.txt'),
      File "/tmp/pip-build-fk8w5it6/openshift/setup.py", line 36, in extract_requirements
        with open(filename, 'r') as requirements_file:
    FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-fk8w5it6/openshift/

The playbooks need to exclude this specific package version until this has been resolved.

Red Hat OCM (https://console.redhat.com/openshift/) keeps track of all OCP clusters installed for a given Red Hat account / user ID. Upon cluster deletion on the target KVM LPAR (e.g. as part of the cleanup playbook) the cluster will be put in a 'stale' state, still lingering in the OCM UI. With many clusters setup and torn down using these playbooks the OCM inventory will inevitably end up littered with stale clusters long gone.

Using the ocm CLI tooling, stale clusters can be removed from the main OCM view automatically. The cleanup playbook should make use of this if OCM integration is configured by the user.

RHEL 8.5 has a known bug within the OS package libvirt-libs (see also: https://bugzilla.redhat.com/show_bug.cgi?id=2038812) which prevents the playbooks from working properly. Also, RHEL 8.5 is not supported by OCP 4.9.

Up until now the playbooks have been installing the latest version of all prerequisite OS-level and Python packages with every playbooks run. This could potentially lead to unstable systems as newer versions of libraries with know bugs / incompatibilities might be pulled in (as is the case for libvirt on RHEL 8.5, see also: #12).

For enhanced ease-of-use bash completion for the installed OpenShift client tooling (e.g. oc) should be added to the playbooks.

While the playbooks do have IBM Z / LinuxONE as the primary target for installing OCP, the solution potentially works on different architectures as well. Implement preliminary support for these architectures.

Setting up dedicated infrastructure nodes in addition to standard worker nodes is optional and can be enabled / disabled via the configuration variable openshift_setup_dedicated_infra_nodes.

If however dedicated infrastructure nodes are enabled they are not being included in the backend configuration sections of the KVM host's haproxy configuration. This should be fixed.

The httpd service must run for the duration of the OCP installation (to provide cached RHCOS images) but it's actual enablement state (whether the service is started automatically at boot or not) should not be affected by the playbooks.

The playbook documentation incorrectly refers to the required OCP pull secret file to be named '.ocp_pull_secret' whereas the playbooks do lookup the file '.ocp4_pull_secret'. This needs to be fixed.

The OpenShift OCP installation allows to specify advanced configuration settings for the OpenShift network, e.g.: https://docs.openshift.com/container-platform/4.10/installing/installing_bare_metal/installing-bare-metal-network-customizations.html

The playbooks should allow the user to specify these settings on a per-host level and apply them to the OpenShift installer configuration accordingly.

The unarchive task for helm uses the include option that has only been introduced in Ansible 2.11. As users may be limited to using older versions of Ansible (especially 2.9) this is a problem and should be avoided / refactored to make use of the exclude option instead.

When running the setup_host.yml playbook the installation of the Python3 prerequisites fails with this error message:

     Downloading https://files.pythonhosted.org/packages/99/f2/b71b9b5b2400fffac7d42c560ac89f302c4d8e328337b2f05f0a4d9e590d/bcrypt-4.0.0.tar.gz
        Complete output from command python setup.py egg_info:

                =============================DEBUG ASSISTANCE==========================
                If you are seeing an error here please try the following to
                successfully install cryptography:

                Upgrade to the latest pip and try again. This will fix errors for most
                users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
                =============================DEBUG ASSISTANCE==========================

        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-build-t2zo6ly4/bcrypt/setup.py", line 11, in <module>
            from setuptools_rust import RustExtension
        ModuleNotFoundError: No module named 'setuptools_rust'

        ----------------------------------------

    :stderr: WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead.
    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t2zo6ly4/bcrypt/

This is likely due to an outdated pip3 installation on the target host.

OpenShift can be installed in so-called 'fips mode' (see: https://docs.openshift.com/container-platform/4.11/installing/installing-fips.html). The playbooks should support this on a per-host basis.

The OCP cluster tuning playbook should be extended to include the following:

• replace the RPS enablement with RFS enablement (increased performance)
• add cluster worker node tuning profile that disables the usage of THPs

Enabling the dnsmasq module in NetworkManager and subsequently restarting NetworkManager results in /etc/resolv.conf to be overwritten. This will inevitably delete all pre-existing nameserver settings, potentially rendering the target KVM host to be unable to resolve hostnames.

There are some inconsistencies in the playbooks, e.g.:

• host architecture references are sometimes hardcoded (s390x) -> these should be replaced with the corresponding Ansible variable
• formatting of the playbooks is inconsistent (placement of blank lines etc.)
• typos in documentation bits (comments, README files)

The playbooks can be improved in various areas:

• proper handling of task loops

• fix sanity check for disk size of directory / mountpoint /var/lib/libvirt

Some of the 3rd-party packages installed by the playbooks should be updated to recent versions:

• go
• terraform
• IBM Cloud provider plugin for terraform
• helm

Additionally the default OpenShift cluster network type is changed from OpenShiftSDN to OVNKubernetes for compatibility with future OpenShift versions.

The host.yml.template file should be updated to reflect the current version of OCP (which is 4.9.x).

The Python package installation step that is part of the 'setup_host' playbook is still not working properly under all conditions. This should be fixed.

If a dnsmasq configuration named 'openshift.conf' for NetworkManager exists it should be overwritten completely by the playbooks to prevent possible cluster bootstrapping issues.

The required Python3 packages cannot be installed on the target KVM host if the host is running RHEL 8.5 (or older). The setup_host.yml playbook fails with this error:

...
distutils.errors.DistutilsPlatformError: Rust 1.54.0 does not match extension requirement >=1.56.0
...

The prerequisite bcrypt package requires at least rust 1.56 installed whereas RHEL 8.5 only provides rust 1.54.
Due to this, any RHEL releases older than 8.6 should be excluded from the playbooks.

The kubernetes.core.k8s Ansible plugin depends on the third-party package 'jsonpatch' to be available on the Ansible host. This can be resolved by installing the RHEL OS-level package 'python3-jsonpatch'.

For technical reasons the versions of the OpenShift installer and the OCP release that is to be installed on the KVM host must match (or should be fairly close to each other). Tests have shown these combinations to work:

• installer: release-4.9, OCP: 4.10.x, 4.9.x, 4.8.x
• installer: release-4.8, OCP: 4.8.x, 4.9.x
• installer: release-4.7, OCP: 4.7.x

The initial assumption that the OpenShift installer 'release-4.9' is capable of installing OCP release going back to 4.7.x (and older) is wrong. The installer generates CoreOS Ignition files as part of the installation process that are used to bootstrap the cluster nodes. It seems that OCP 4.7.x for instance is unable to deal with Ignition files spec'd with version 3.2.0 (generated by OpenShift installer versions newer that 4.8'.

The playbooks should allow the user to setup a new OpenShift cluster with a custom cluster network MTU size (as opposed to using the cluster's built-in default MTU size).

If no custom MTU size is specified the default value will be used.

The playbooks should support RHEL 9.0 as it has been GA-ed earlier this year: https://access.redhat.com/announcements/6958409

One of the latest changes introduced a bug to the way some of the prerequisite binaries are being installed using 'go'. The 'go' executable itself is not included in the PATH variable.

The relative links to other documentation files in the main DOCUMENTATION.md markdown file are broken and need to be fixed.

The included host configuration template file should be modified to point to OCP 4.10.

This repository could benefit from a pull request template that contains the following sections:

• PR description
• Link to parent issue (using 'Fixes:' feature)
• DCO string

This template would be helpful in the contribution process.

The 'cleanup_ocp_install' playbooks fails at the 'get ClusterVersion object' task due to the cluster not being operable after a failed installation attempt. This needs to be fixed.

The playbooks in this repository can be run from within a Docker container as described in the main documentation. This Docker container can be used in a Tekton pipeline workflow setup quiet easily to facilitate installing remote clusters from a centralized OCP cluster instance. Add examples for those pipelines and tasks.

In order to improve the out-of-the-box user experience it would be helpful to have the 'helm' binaries present on the KVM host, similar to the OpenShift client binaries.