osbuild / image-builder Goto Github PK

View Code? Open in Web Editor NEW

43.0 19.0 43.0 19.68 MB

Image Builder service for console.redhat.com

License: Apache License 2.0

Makefile 0.54% Go 88.12% Shell 9.34% Python 2.01%

image-builder's Introduction

Image Builder

Image Builder serves as an HTTP API on top of Osbuild Composer, and serves as the backend for Image Builder Frontend.

Image Builder is intended to be run within the console.redhat.com platform.

Project

Website: https://www.osbuild.org
Bug Tracker: https://github.com/osbuild/image-builder/issues
Matrix: #image-builder on fedoraproject.org
Mailing List: [email protected]
Changelog: https://github.com/osbuild/image-builder/releases

OpenAPI spec

The latest OpenAPI specification:

Contributing

Please refer to the hacking guide to learn more.

Build

Project building and testing project is based on predefined make targets:

make build to trigger Go builder

Run Tests

To run the tests locally just call

make unit-tests

Before pushing something for a pull request you should run this check to avoid problems with required github actions

make push-check

Installation

To run the project use make run target

Repository

web: https://github.com/osbuild/image-builder
https: https://github.com/osbuild/image-builder.git
ssh: [email protected]:osbuild/image-builder.git

License

Apache-2.0
See LICENSE file for details.

image-builder's People

Contributors

Stargazers

Watchers

Forkers

croissanne major atodorov yuxisun1217 ashcrow jkozol msehnout ondrejbudai chloenayon thozza achilleas-k aaronh88 yih-redhat lavocatt matoval kingsleyzissou ldjebran acidburn0zzz terrorizer1980 teg jrusz lucasgarfield supakeen diaasami miyunari regexowl lzap sarroutbi ochosi mgold1234 aadhikar bcl mshriver avitova ezr-ondrej tkoscieln mvo5 amirfefer elkoniu mlibanori adamwill schuellerf avisiedo

image-builder's Issues

Need pre-check dos2unix package in test/cases/api.sh

Once we migrate to GitLab the runners will come with some epositories pre-configured sometimes.

If want this to be a robust check then remove/disable everything under /etc/yum.repos.d before subscribing but still keep the rpm -q dos2unix check. That's a pre-condition and we want to fail if the pre-condition isn't met.

Start expanding api design

What kind of customer data do we want to pass to this service?

Can we use access.redhat.com for package lists?

I had a quick look at how access.redhat.com displays package lists with the downloadable images, and it seems they use an API that I wonder if we could steal?

Add constraints that account_number can't be an empty string/varchar to db and identity verification mw

I think account_number should always be set (maybe worth to double check). Add a table constraint for this to make sure.

org_id can theoretically be empty, so no constraints are necessary there.

If the identityheader doesn't have an account number, fail gracefully/return 404 with a good msg.

build edge image hang due to udevadm

I was testing Edge related APIs against this PR: #193
And when I tried to build a commit image, it hangs at udevadm step.

Reproduce steps:

Send a request to build Edge commit image and upload to s3.

[root@vm02-iso image-builder]# curl -w %{http_code} -H 'x-rh-identity:  ${identity}' -H 'Content-Type: application/json' -d "@data.json" http://localhost:8086/api/image-builder/v1.0/compose
{"id":"a6193303-a3ff-442c-bb61-dd7f618d6d7a"}
201
[root@vm02-iso image-builder]# curl -w %{http_code} -H 'x-rh-identity:  ${identity}' http://localhost:8086/api/image-builder/v1.0/composes/a6193303-a3ff-442c-bb61-dd7f618d6d7a
{"image_status":{"status":"building"}}
200

After hours the status is still building, then check the osbuild log:

[root@vm02-iso image-builder]# pgrep osbuild-worker
919938
[root@vm02-iso image-builder]# pstree -al 919938
osbuild-worker -unix /run/osbuild-composer/job.socket
  ├─osbuild /usr/bin/osbuild --store /var/cache/osbuild-worker/osbuild-store --output-directory /var/cache/osbuild-worker/output/555fcc00-e663-4f96-a5a8-fa1fda042c56-877396296 --json - --export assembler
  │   ├─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
  │   │   └─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
  │   │       └─org.osbuild.rhe /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
  │   │           └─org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm
  │   │               └─rpm --verbose --root /run/osbuild/tree --define _pkgverify_level none --install /tmp/manifest.06sulvm9
  │   │                   └─sh /var/tmp/rpm-tmp.17hjYk 1
  │   │                       └─udevadm trigger --sysname-match=hw_random --settle
  │   └─4*[{osbuild}]
  └─6*[{osbuild-worker}]

kill the udevadm process and the building continues and will succeed soon.

[root@vm02-iso image-builder]# pkill udevadm
[root@vm02-iso image-builder]# pstree -al 919938
osbuild-worker -unix /run/osbuild-composer/job.socket
  ├─osbuild /usr/bin/osbuild --store /var/cache/osbuild-worker/osbuild-store --output-directory /var/cache/osbuild-worker/output/555fcc00-e663-4f96-a5a8-fa1fda042c56-877396296 --json - --export assembler
  │   ├─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
  │   │   └─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
  │   │       └─org.osbuild.rhe /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
  │   │           └─org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm
  │   │               └─rpm --verbose --root /run/osbuild/tree --define _pkgverify_level none --install /tmp/manifest.06sulvm9
  │   │                   └─sh /var/tmp/rpm-tmp.OGCbPk 0 0
  │   │                       └─update-mime-dat -n /usr/share/mime
  │   └─4*[{osbuild}]
  └─6*[{osbuild-worker}]

I can always reproduce this bug, and @achilleas-k is aware of this issue, opening this issue to track it.

Input validation against the openapi spec

Currently the input is pretty much unvalidated. If the json body or query params of the request can get parsed into the golang struct it's seen as valid.

So on the one hand I think we need to expand our openapi spec to describe parameters and content better, and then validate the request based on that spec.

Potentially using https://pkg.go.dev/github.com/getkin/kin-openapi/openapi3filter ?

scrollbar in wizard?

This broke recently.

Versioning discussion

We have to tackle this sooner rather than later, so I have a few points that warrant discussion:

semver

Clouddot uses semantic versioning. I say we do our very best to only make major version changes, just so we only have one way to increase a version.

each version has their own package

Because each version has it's own spec file, we need to pull the generated api and the handlers from the server package into their own packages. I propose having internal/v1, internal/v2, etc....

Handlers will be duplicated, but I think that's fine to keep maintainability.

shared code

We should try to keep utilities shared between these packages (the database, logger etc...) common as much as we can to avoid duplication. I think having the database common makes sense since any migration needs to be backwards compatible anyway.

The clouddot guidelines state to support old versions as long as is 'reasonably possible', and I think the moment we have to bend over backwards in these shared parts in order to keep an older version, we should drop that older version.

Versioned IB -> versioned composer (CO)

I'd like to start out with keeping the cloudapi client shared between versions, and try having all versions of IB work against a single version of CO. If this turns out to be very difficult, we'll have to start adding multiple versions to the cloudapi client.

The default user on `ami` image type changed since RHEL-8.5

The ami image type have been redefined since RHEL-8.5 based on the official RHEL EC2 images (osbuild/osbuild-composer#1607). As a result, the username of the default user created by cloud-init changed from cloud-user to ec2-user.

This required a change in the Cloud API test case (api.sh) in osbuild-composer:
https://github.com/osbuild/osbuild-composer/blob/53109945efb4ff648982fc6c52af723dede63bd5/test/cases/api.sh#L391-L396

An equivalent change will be needed in the api.sh test case of image-builder, once it will start to be tested with RHEL-8.5:

image-builder/test/cases/api.sh

Lines 137 to 139 in 5a7b6ca

    
           "rhel-8.2" | "rhel-8.3" | "rhel-8.4") 
        
             DISTRO="rhel-84" 
        
             SSH_USER="cloud-user"

Actually address gosec violations regarding files

Currently we're translating query parameters into a file path. And we're not doing it in the safest way in internal/server/distribution.go.

Currently it's not too bad however as it walks a specified directory and sees if the path matches the one generated from the query params, and then loads in specific fields from the json structure, but still we should do it properly and sanatize those params.

Update rhel package lists and distribution names

dist is now rhel8.4

Expose `/ready` for the iqe test framework

The iqe test framework could check the connection to composer, and general readiness, and include it in qe metrics for clouddot.

Improve authorization/ no cross-user compose access

/composes returns a list of composes done by a specific user, which is taken from the identity header.

/compose/{id} returns the status of a compose, however technically if you know a compose id of a different user, you could retrieve the status of a compose that isn't yours. This doesn't contain any secrets, but might contain some identifying information.

When querying a compose status, make sure that the compose belongs to the user. Either the account_number should match, or the org_id (if present) should match.

Replace fedora-32 distribution with fedora-33

Also generate the correct package lists etc...

Add documentation/utility script on how to rebuild the packages lists

OpenAPI Spec link is broken

Not able to access the latest API Spec link under README.md OpenAPI spec section

https://github.com/osbuild/image-builder/blob/main/internal/server/api.yaml

Filter requests based on orgid

in the identity header check make sure they've got a certain org_id

Un-disable gosec G304

#54 (once it lands) includes a gosec override for G304. I think we could amend the code to not need this override.

image-builder port name is too long

Deployment "image-builder" is invalid: spec.template.spec.containers[0].ports[0].name: Invalid value: "image-builder-api": must be no more than 15 characters

Return 400 Bad Request instead of 500 when the distribution isn't supported

subscription test cases failed randomly

Sometimes the subscription test failed. The VM is not registered after boot up. e.g.
https://osbuildci.cloud.paas.psi.redhat.com/job/osbuild/job/image-builder/job/main/79/execution/node/65/log/

17:07:30  ++ ssh -oStrictHostKeyChecking=no -i ./keypair.pem [email protected] sudo subscription-manager identity
17:07:30  ++ grep 'org ID'
17:07:30  This system is not yet registered. Try 'subscription-manager register --help' for more information.
17:07:30  + subscribe_org_id=

Perhaps need to wait or loop retry several times to verify it.

Use jackc/tern for database migration

Currently we depend on lib/pq for migrations and jackc for the actual db operations. It makes sense to move the migrations over to jackc/tern. Furthermore lib/pq doesn't support sslmode prefer, which should be the sensible default.

metrics: ensure compose 500 are genuine

Verify that we don't consider user error as a failure, only true internal errors.

This might mostly be a composer issue.

Make it easier to run the integration tests locally

The unit tests with go test ./... are pretty straightforward.

The integration tests (cmd/image-builder-tests) require composer up and running, and also a container of image-builder to test the container variant; so they are harder to run.

Either have a script somewhere or a make target before running go test -tags=integration. And document it.

Prometheus metrics

As a first track number of 500s per endpoint (vs total amount of requests).

Update distributions and verify we're using the correct repository urls

For now they're copied in during the container build.

For the stage environment we probably don't want to include fedora, and just have rhel (which versions of rhel though?).

RFE: provide Azure account organization in AzureUploadStatus

In order to link a user to their uploaded Azure image the image-builder-frontend needs the organization that the user has authorized Image Builder on.

The link would follow this pattern: https://portal.azure.com/#@{organization url}/resource/subscriptions/{subscription id}/resourceGroups/{resource group}/providers/Microsoft.Compute/images/{image name}/overview

Currently, only the organization url is missing.

Move integration tests towards `schutzbot/run_tests.sh`

Instead of calling image-builder-tests, let's remove that binary, and just start the container and use curl and jq to tests the endpoints.

Rename references to account_id/accountId to account_number

In CRC org_id is used to refer to a number that organization admins have access to really. And account_number is the shared number over an organization.

So let's rename account_id/accountId references to account_number (but keep org_id as is). To keep it consistent and avoid future confusion. Migrate the db as well, since we currently have a single table without any references to account_id, we can just rename it to account_number.

distribution/*json files aren't shipped in the container, they should be

Use `gce-byos` image type with GCP

Once the new gce-byos image type is introduced in osbuild-composer via osbuild/osbuild-composer#1365 and the version is deployed in the staging / production, Image Builder should use it by default for images intended for GCP.

In addition to changing the image type, Image Builder must pass additional repositories to the osbuild-composer. These repositories contain GCP guest tools and SDK. Their content is as follows:

{
    "name": "google-compute-engine",
    "baseurl": "https://packages.cloud.google.com/yum/repos/google-compute-engine-el8-x86_64-stable",
    "gpgkey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQENBGA9EFkBCAC1ilzST0wns+uwZyEA5IVtYeyAuXTaQUEAd70SqIlQpDd4EyVi\nx3SCanQIu8dG9Zq3+x28WBb2OuXP9oc06ybOWdu2m7N5PY0BUT4COA36JV/YrxmN\ns+5/M+YnDHppv63jgRIOkzXzXNo6SwTsl2xG9fKB3TS0IMvBkWdw5PGrBM5GghRc\necgoSAAwRbWJXORHGKVwlV6tOxQZ/xqA08hPJneMfsMFPOXsitgGRHoXjlUWLVeJ\n70mmIYsC/pBglIwCzmdD8Ee39MrlSXbuXVQiz38iHfnvXYpLEmgNXKzI0DH9tKg8\n323kALzqaJlLFOLJm/uVJXRUEfKS3LhVZQMzABEBAAG0UVJhcHR1cmUgQXV0b21h\ndGljIFNpZ25pbmcgS2V5IChjbG91ZC1yYXB0dXJlLXNpZ25pbmcta2V5LTIwMjEt\nMDMtMDEtMDhfMDFfMDkucHViKYkBKAQTAQgAHAUCYD0QWQkQ/uqRaTB+oHECGwMF\nCQPDCrACGQEAAHtlCACxSWMp3yRcLmsHhxGDt59nhSNXhouWiNePSMe5vETQA/lh\nip9Zx/NPRCa4q5jpIDBlEYOg67YanztcjSWGSI35Xblq43H4uLSxh4PtKzZMo+Uj\n8n2VNHOZXBdGcsODcU3ynF64r7eTQevUe2aU0KN2o656O3HrE4itOVKYwnnkmNsk\nG45b9b7DJnsQ6WPszUc8lNhsa2gBI6vfLl68vjj7PlWw030BM/RoMEPpoOApohHo\nsfnNhxJmE1AxwBkMEzyo2kZhPZGh85LDnDbAvjSFKqYSPReKmRFjLlo3DPVHZ/de\nQn6noHbgUChLo21FefhlZO6tysrb283MWMIyY/YSuQENBGA9EFkBCADcdO/Aw1qu\ndZORZCNLz3vTiQSFcUFYyScfJJnwUsg8fy0kgg9olFY0GK5icT6n/shc1RlIpuqr\nOQYBZgtK3dSZfOAXE2N20HUvC+nrKKuXXX+jcM/X1kHxwX5tG6fB1fyNH0p/Qqsz\nEfYRHJu0Y4PonTYIslITnEzlN4hUN6/mx1+mWPl4P4R7/h6+p7Q2jtaClEtddF0e\neOf16Ma5S8fff80uZCLJoVu3lOXCT22oCf7qmH2XddmqGisUScqwmbmuv30tdQed\nn+8njKo2pfpVF1Oa67CWRXdKTknuZybxI9Ipcivy8CISL2Do0uzij7SR7keVf7G1\nQ3K3iJ0wn6mDABEBAAGJAR8EGAEIABMFAmA9EFkJEP7qkWkwfqBxAhsMAAA/3Af9\nFJ2hEp2144fzgtNWHOVFv27hsrO7wYFZwoic9lHSl4iEw8mJc/3kEXdg9Vf9m1zb\nG/kZ6slmzpfv7zDAdN3h3HT0B1yrb3xXzRX0zhOYAbQSUnc6DemhDZoDWt/wVceK\nfzvebB9VTDzRBUVzxCduvY6ij0p2APZpnTrznvCPoCHkfzBMC3Zyk1FueiPTPoP1\n9M0BProMy8qDVSkFr0uX3PM54hQN6mGRQg5HVVBxUNaMnn2yOQcxbQ/T/dKlojdp\nRmvpGyYjfrvyExE8owYn8L7ly2N76GcY6kiN1CmTnCgdrbU0SPacm7XbxTYlQHwJ\nCEa9Hf4/nuiBaxwXKuc/y5kBDQRfyX5eAQgA0z1F3ZDbtOe1/j90k1cQsyaVNjJ/\nrVGpinUnVWpmxnmBSDXKfxBsDRoXW9GtQWx7NUlmGW88IeHevqd5OAAc1TDvkaTL\nv2gcfROWjp+XPBsx42f1RGoXqiy4UlHEgswoUmXDeY89IUxoZgBmr4jLekTM0n2y\nIWT49ZA8wYhndEMHf6zj5ya+LWj67kd3nAY4R7YtfwTBnf5Y9Be80Jwo6ez66oKR\nDwU/I6PcF9sLzsl7MEiPxrH2xYmjiXw52Hp4GhIPLBfrt1jrNGdtHEq+pEu+ih6U\n32tyY2LHx7fDQ8PMOHtx/D8EMzYkT/bV3jAEikM93pjI/3pOh8Y4oWPahQARAQAB\ntLpnTGludXggUmFwdHVyZSBBdXRvbWF0aWMgU2lnbmluZyBLZXkgKC8vZGVwb3Qv\nZ29vZ2xlMy9wcm9kdWN0aW9uL2JvcmcvY2xvdWQtcmFwdHVyZS9rZXlzL2Nsb3Vk\nLXJhcHR1cmUtcHVia2V5cy9jbG91ZC1yYXB0dXJlLXNpZ25pbmcta2V5LTIwMjAt\nMTItMDMtMTZfMDhfMDUucHViKSA8Z2xpbnV4LXRlYW1AZ29vZ2xlLmNvbT6JASgE\nEwEIABwFAl/Jfl4JEItXxcKDb0vrAhsDBQkDwwqwAhkBAABBeggAmnpK6OmlCSXd\n5lba7SzjnsFfHrdY3qeXsJqTq3sP6Wo0VQXiG1dWsFZ9P/BHHpxXo5j+lhXHQlqL\ng1SEv0JkRUFfTemFzfD4sGpa0Vd20yhQR5MGtXBB+AGnwhqNHA7yW/DdyZzP0Zm9\nSkhiq+2V6ZpC7WFaq+h4M5frJ65R9F8LJea90sr6gYL0WE0CmaSqpgRHdbnYnlaC\n0hffPJCnjQ4xWvkNUo2Txlvl7pIBPJAVG0g8fGPKugrM4d1VWPuSVHqopkYCdgA2\nNv95RLQGTrZsHAZYWNHD1laoGteBO5ExkligulvejX8vSuy+GKafJ0zBK7rNfNWq\nsMDXzKp6Z7kBDQRfyX5eAQgAw0ofinQXjYyHJVVZ0SrdEE+efd8heFlWbf04Dbmh\nGebypJ6KFVSKvnCSH2P95VKqvE3uHRI6HbRcinuV7noKOqo87PE2BXQgB16V0aFK\nJU9eJvqpCfK4Uq6TdE8SI1iWyXZtzZa4E2puUSicN0ocqTVMcqJZx3pV8asigwpM\nQUg5kesXHX7d8HUJeSJCAMMXup8sJklLaZ3Ri0SXSa2iYmlhdiAYxTYN70xGI+Hq\nHoWXeF67xMi1azGymeZun9aOkFEbs0q1B/SU/4r2agpoT6aLApV119G24vStGf/r\nlcpOr++prNzudKyKtC9GHoTPBvvqphjuNtftKgi5HQ+f4wARAQABiQEfBBgBCAAT\nBQJfyX5eCRCLV8XCg29L6wIbDAAAGxoIAMO5YUlhJWaRldUiNm9itujwfd31SNbU\nGFd+1iBJQibGoxfv2Q3ySdnep3LkEpXh+VkXHHOIWXysMrAP3qaqwp8HO8irE6Ge\nLMPMbCRdVLUORDbZHQK1YgSR0uGNlWeQxFJq+RIIRrWRYfWumi6HjFTP562Qi7LQ\n1aDyhKS6JB7v4HmwsH0/5/VNXaJRSKL4OnigApecTsfq83AFae0eD+du4337nc93\nSjHS4T67LRtMOWG8nzz8FjDj6fpFBeOXmHUe5CipNPVayTZBBidCkEOopqkdU59J\nMruHL5H6pwlBdK65+wnQai0gr9UEYYK+kwoUH+8p1rD8+YBnVY4d7SM=\n=UVi6\n-----END PGP PUBLIC KEY BLOCK-----\n-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v1\n\nmQENBFWKtqgBCADmKQWYQF9YoPxLEQZ5XA6DFVg9ZHG4HIuehsSJETMPQ+W9K5c5\nUs5assCZBjG/k5i62SmWb09eHtWsbbEgexURBWJ7IxA8kM3kpTo7bx+LqySDsSC3\n/8JRkiyibVV0dDNv/EzRQsGDxmk5Xl8SbQJ/C2ECSUT2ok225f079m2VJsUGHG+5\nRpyHHgoMaRNedYP8ksYBPSD6sA3Xqpsh/0cF4sm8QtmsxkBmCCIjBa0B0LybDtdX\nXIq5kPJsIrC2zvERIPm1ez/9FyGmZKEFnBGeFC45z5U//pHdB1z03dYKGrKdDpID\n17kNbC5wl24k/IeYyTY9IutMXvuNbVSXaVtRABEBAAG0Okdvb2dsZSBDbG91ZCBQ\nYWNrYWdlcyBSUE0gU2lnbmluZyBLZXkgPGdjLXRlYW1AZ29vZ2xlLmNvbT6JATgE\nEwECACIFAlWKtqgCGy8GCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEPCcOUw+\nG6jV+QwH/0wRH+XovIwLGfkg6kYLEvNPvOIYNQWnrT6zZ+XcV47WkJ+i5SR+QpUI\nudMSWVf4nkv+XVHruxydafRIeocaXY0E8EuIHGBSB2KR3HxG6JbgUiWlCVRNt4Qd\n6udC6Ep7maKEIpO40M8UHRuKrp4iLGIhPm3ELGO6uc8rks8qOBMH4ozU+3PB9a0b\nGnPBEsZdOBI1phyftLyyuEvG8PeUYD+uzSx8jp9xbMg66gQRMP9XGzcCkD+b8w1o\n7v3J3juKKpgvx5Lqwvwv2ywqn/Wr5d5OBCHEw8KtU/tfxycz/oo6XUIshgEbS/+P\n6yKDuYhRp6qxrYXjmAszIT25cftb4d4=\n=/PbX\n-----END PGP PUBLIC KEY BLOCK-----",
    "check_gpg": true
},
{
    "name": "google-cloud-sdk",
    "baseurl": "https://packages.cloud.google.com/yum/repos/cloud-sdk-el8-x86_64",
    "gpgkey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQENBGA9EFkBCAC1ilzST0wns+uwZyEA5IVtYeyAuXTaQUEAd70SqIlQpDd4EyVi\nx3SCanQIu8dG9Zq3+x28WBb2OuXP9oc06ybOWdu2m7N5PY0BUT4COA36JV/YrxmN\ns+5/M+YnDHppv63jgRIOkzXzXNo6SwTsl2xG9fKB3TS0IMvBkWdw5PGrBM5GghRc\necgoSAAwRbWJXORHGKVwlV6tOxQZ/xqA08hPJneMfsMFPOXsitgGRHoXjlUWLVeJ\n70mmIYsC/pBglIwCzmdD8Ee39MrlSXbuXVQiz38iHfnvXYpLEmgNXKzI0DH9tKg8\n323kALzqaJlLFOLJm/uVJXRUEfKS3LhVZQMzABEBAAG0UVJhcHR1cmUgQXV0b21h\ndGljIFNpZ25pbmcgS2V5IChjbG91ZC1yYXB0dXJlLXNpZ25pbmcta2V5LTIwMjEt\nMDMtMDEtMDhfMDFfMDkucHViKYkBKAQTAQgAHAUCYD0QWQkQ/uqRaTB+oHECGwMF\nCQPDCrACGQEAAHtlCACxSWMp3yRcLmsHhxGDt59nhSNXhouWiNePSMe5vETQA/lh\nip9Zx/NPRCa4q5jpIDBlEYOg67YanztcjSWGSI35Xblq43H4uLSxh4PtKzZMo+Uj\n8n2VNHOZXBdGcsODcU3ynF64r7eTQevUe2aU0KN2o656O3HrE4itOVKYwnnkmNsk\nG45b9b7DJnsQ6WPszUc8lNhsa2gBI6vfLl68vjj7PlWw030BM/RoMEPpoOApohHo\nsfnNhxJmE1AxwBkMEzyo2kZhPZGh85LDnDbAvjSFKqYSPReKmRFjLlo3DPVHZ/de\nQn6noHbgUChLo21FefhlZO6tysrb283MWMIyY/YSuQENBGA9EFkBCADcdO/Aw1qu\ndZORZCNLz3vTiQSFcUFYyScfJJnwUsg8fy0kgg9olFY0GK5icT6n/shc1RlIpuqr\nOQYBZgtK3dSZfOAXE2N20HUvC+nrKKuXXX+jcM/X1kHxwX5tG6fB1fyNH0p/Qqsz\nEfYRHJu0Y4PonTYIslITnEzlN4hUN6/mx1+mWPl4P4R7/h6+p7Q2jtaClEtddF0e\neOf16Ma5S8fff80uZCLJoVu3lOXCT22oCf7qmH2XddmqGisUScqwmbmuv30tdQed\nn+8njKo2pfpVF1Oa67CWRXdKTknuZybxI9Ipcivy8CISL2Do0uzij7SR7keVf7G1\nQ3K3iJ0wn6mDABEBAAGJAR8EGAEIABMFAmA9EFkJEP7qkWkwfqBxAhsMAAA/3Af9\nFJ2hEp2144fzgtNWHOVFv27hsrO7wYFZwoic9lHSl4iEw8mJc/3kEXdg9Vf9m1zb\nG/kZ6slmzpfv7zDAdN3h3HT0B1yrb3xXzRX0zhOYAbQSUnc6DemhDZoDWt/wVceK\nfzvebB9VTDzRBUVzxCduvY6ij0p2APZpnTrznvCPoCHkfzBMC3Zyk1FueiPTPoP1\n9M0BProMy8qDVSkFr0uX3PM54hQN6mGRQg5HVVBxUNaMnn2yOQcxbQ/T/dKlojdp\nRmvpGyYjfrvyExE8owYn8L7ly2N76GcY6kiN1CmTnCgdrbU0SPacm7XbxTYlQHwJ\nCEa9Hf4/nuiBaxwXKuc/y5kBDQRfyX5eAQgA0z1F3ZDbtOe1/j90k1cQsyaVNjJ/\nrVGpinUnVWpmxnmBSDXKfxBsDRoXW9GtQWx7NUlmGW88IeHevqd5OAAc1TDvkaTL\nv2gcfROWjp+XPBsx42f1RGoXqiy4UlHEgswoUmXDeY89IUxoZgBmr4jLekTM0n2y\nIWT49ZA8wYhndEMHf6zj5ya+LWj67kd3nAY4R7YtfwTBnf5Y9Be80Jwo6ez66oKR\nDwU/I6PcF9sLzsl7MEiPxrH2xYmjiXw52Hp4GhIPLBfrt1jrNGdtHEq+pEu+ih6U\n32tyY2LHx7fDQ8PMOHtx/D8EMzYkT/bV3jAEikM93pjI/3pOh8Y4oWPahQARAQAB\ntLpnTGludXggUmFwdHVyZSBBdXRvbWF0aWMgU2lnbmluZyBLZXkgKC8vZGVwb3Qv\nZ29vZ2xlMy9wcm9kdWN0aW9uL2JvcmcvY2xvdWQtcmFwdHVyZS9rZXlzL2Nsb3Vk\nLXJhcHR1cmUtcHVia2V5cy9jbG91ZC1yYXB0dXJlLXNpZ25pbmcta2V5LTIwMjAt\nMTItMDMtMTZfMDhfMDUucHViKSA8Z2xpbnV4LXRlYW1AZ29vZ2xlLmNvbT6JASgE\nEwEIABwFAl/Jfl4JEItXxcKDb0vrAhsDBQkDwwqwAhkBAABBeggAmnpK6OmlCSXd\n5lba7SzjnsFfHrdY3qeXsJqTq3sP6Wo0VQXiG1dWsFZ9P/BHHpxXo5j+lhXHQlqL\ng1SEv0JkRUFfTemFzfD4sGpa0Vd20yhQR5MGtXBB+AGnwhqNHA7yW/DdyZzP0Zm9\nSkhiq+2V6ZpC7WFaq+h4M5frJ65R9F8LJea90sr6gYL0WE0CmaSqpgRHdbnYnlaC\n0hffPJCnjQ4xWvkNUo2Txlvl7pIBPJAVG0g8fGPKugrM4d1VWPuSVHqopkYCdgA2\nNv95RLQGTrZsHAZYWNHD1laoGteBO5ExkligulvejX8vSuy+GKafJ0zBK7rNfNWq\nsMDXzKp6Z7kBDQRfyX5eAQgAw0ofinQXjYyHJVVZ0SrdEE+efd8heFlWbf04Dbmh\nGebypJ6KFVSKvnCSH2P95VKqvE3uHRI6HbRcinuV7noKOqo87PE2BXQgB16V0aFK\nJU9eJvqpCfK4Uq6TdE8SI1iWyXZtzZa4E2puUSicN0ocqTVMcqJZx3pV8asigwpM\nQUg5kesXHX7d8HUJeSJCAMMXup8sJklLaZ3Ri0SXSa2iYmlhdiAYxTYN70xGI+Hq\nHoWXeF67xMi1azGymeZun9aOkFEbs0q1B/SU/4r2agpoT6aLApV119G24vStGf/r\nlcpOr++prNzudKyKtC9GHoTPBvvqphjuNtftKgi5HQ+f4wARAQABiQEfBBgBCAAT\nBQJfyX5eCRCLV8XCg29L6wIbDAAAGxoIAMO5YUlhJWaRldUiNm9itujwfd31SNbU\nGFd+1iBJQibGoxfv2Q3ySdnep3LkEpXh+VkXHHOIWXysMrAP3qaqwp8HO8irE6Ge\nLMPMbCRdVLUORDbZHQK1YgSR0uGNlWeQxFJq+RIIRrWRYfWumi6HjFTP562Qi7LQ\n1aDyhKS6JB7v4HmwsH0/5/VNXaJRSKL4OnigApecTsfq83AFae0eD+du4337nc93\nSjHS4T67LRtMOWG8nzz8FjDj6fpFBeOXmHUe5CipNPVayTZBBidCkEOopqkdU59J\nMruHL5H6pwlBdK65+wnQai0gr9UEYYK+kwoUH+8p1rD8+YBnVY4d7SM=\n=UVi6\n-----END PGP PUBLIC KEY BLOCK-----\n-----BEGIN PGP PUBLIC KEY BLOCK-----\nVersion: GnuPG v1\n\nmQENBFWKtqgBCADmKQWYQF9YoPxLEQZ5XA6DFVg9ZHG4HIuehsSJETMPQ+W9K5c5\nUs5assCZBjG/k5i62SmWb09eHtWsbbEgexURBWJ7IxA8kM3kpTo7bx+LqySDsSC3\n/8JRkiyibVV0dDNv/EzRQsGDxmk5Xl8SbQJ/C2ECSUT2ok225f079m2VJsUGHG+5\nRpyHHgoMaRNedYP8ksYBPSD6sA3Xqpsh/0cF4sm8QtmsxkBmCCIjBa0B0LybDtdX\nXIq5kPJsIrC2zvERIPm1ez/9FyGmZKEFnBGeFC45z5U//pHdB1z03dYKGrKdDpID\n17kNbC5wl24k/IeYyTY9IutMXvuNbVSXaVtRABEBAAG0Okdvb2dsZSBDbG91ZCBQ\nYWNrYWdlcyBSUE0gU2lnbmluZyBLZXkgPGdjLXRlYW1AZ29vZ2xlLmNvbT6JATgE\nEwECACIFAlWKtqgCGy8GCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEPCcOUw+\nG6jV+QwH/0wRH+XovIwLGfkg6kYLEvNPvOIYNQWnrT6zZ+XcV47WkJ+i5SR+QpUI\nudMSWVf4nkv+XVHruxydafRIeocaXY0E8EuIHGBSB2KR3HxG6JbgUiWlCVRNt4Qd\n6udC6Ep7maKEIpO40M8UHRuKrp4iLGIhPm3ELGO6uc8rks8qOBMH4ozU+3PB9a0b\nGnPBEsZdOBI1phyftLyyuEvG8PeUYD+uzSx8jp9xbMg66gQRMP9XGzcCkD+b8w1o\n7v3J3juKKpgvx5Lqwvwv2ywqn/Wr5d5OBCHEw8KtU/tfxycz/oo6XUIshgEbS/+P\n6yKDuYhRp6qxrYXjmAszIT25cftb4d4=\n=/PbX\n-----END PGP PUBLIC KEY BLOCK-----",
    "check_gpg": true
}

OpenShift template failed to apply

11:51:39 [2021-01-11 17:51:39] [ERROR] [saasherder.py:_process_template:328] - [https://github.com/osbuild/image-builder//templates/image-builder.yml:main] error fetching template: while parsing a block collection
11:51:39   in "<byte string>", line 170, column 11:
11:51:39               - image: "${COMPOSER_IMAGE}:${CO ... 
11:51:39               ^
11:51:39 expected <block end>, but found '?'
11:51:39   in "<byte string>", line 185, column 11:
11:51:39               volumeMounts:
11:51:39               ^

image builds stuck in prod

Image builds appear to be stuck in production. I did not yet dig deeper.

Do some research figuring out persistent data storage

Find a database with minimal dependencies

How do we get persistent storage on the target platform

Figure out authentication

How to verify that a specified identity actually has access to this service.

unit tests are too flaky

The tests have to be rerun often, especially on PRs.

Add tests which expect osbuild-composer(-cloud) running on the host

Let's wait with this until we have the cloudapi in osbuild-composer released to start testing this on fedora.

metrics: measure ratio of slow requests

Rather than measuring the average time, measure the ratio of slow (slower than the cut off) requests to fast ones.

I believe this follows the best practices of AppSRE better, where every event is success or failure and we measure the ratio between them.

Make minimally functioning concept

Add an endpoint which makes a (hardcoded) call to a running osbuild-composer service, which then returns an image.

Image-builder should use the dnf-json program from osbuild-composer

buildCustomizations() always returns nil for error

From server.go:

    custom, err := buildCustomizations(composeRequest.Customizations)
    if err != nil {
        return err
    }

but the current implementation of buildCustomizations() function always returns nil for error. I'm not sure if it is missing an error condition check or there's simply no way for customizations to error out at the moment.

Comment from @ondrejbudai:

hmm... I guess there might be more validation
I'm not sure how Sanne wants to validate the API requests.

Opening issue so we don't forget about this.

Let the integration test assume image-builder on :8086

Right now the integration tests start image-builder on :8086 and expects a container on :8087. And then runs the same test against both. This is a bit strange.

The integration tests should just expect image-builder on :8086, and then schutzbot should just start a container before running the tests.

Add more integration tests for corner cases

Test the endpoints with wrong parameters and see you still get the expected response.

Blocked on #119

Research implementing package search from `primary` data in repomd.xml

Find out if we can expect all repos to have a primary_db sqlite file.
Figure out if we can do this for repos that need authentication (rhel-cdn).

Azure test sometimes fails when downloading azure-cli

Failed job: https://gitlab.com/osbuild/ci/image-builder/-/jobs/1354688824

Installing:
 azure-cli         x86_64         2.25.0-1.el7          azure-cli          43 M
Transaction Summary
================================================================================
Install  1 Package
Total download size: 43 M
Installed size: 557 M
Downloading Packages:
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (35): SSL connect error for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to packages.microsoft.com:443 ]
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (28): Timeout was reached for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [Connection timed out after 30001 milliseconds]
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (28): Timeout was reached for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [Connection timed out after 30001 milliseconds]
[MIRROR] azure-cli-2.25.0-1.el7.x86_64.rpm: Curl error (35): SSL connect error for https://packages.microsoft.com/yumrepos/azure-cli/azure-cli-2.25.0-1.el7.x86_64.rpm [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to packages.microsoft.com:443 ]
[FAILED] azure-cli-2.25.0-1.el7.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Failed to set locale, defaulting to C.UTF-8
Error: Error downloading packages:
  Cannot download azure-cli-2.25.0-1.el7.x86_64.rpm: All mirrors were tried

I've seen this several times, when trying manually or in osbuild-composer repository and I think it's the same issue. The way it's being installed is provided in official documentation so that should be fine. Usually a restart fixes it, so I guess adding some retry before failing the test could solve this easily...

Implement livenessProbe and readinessProbe endpoints

This could be something under /status or /health.

I think both of these can point to the same endpoint, a sensible readiness/health check would be stuff like:

Is composer available? (maybe we need a /status endpoint there too, we just need to verify it's there and auth is good).
Are there distributions available ? For the container /app/distributions should contain stuff.

In future we can also check if the database is there.

📜 Multiple instances of image-builder fail to send logs to cloudwatch

We are seeing this error in OpenShift:

Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49612946959762469195597770664378024234801679062591571890
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "da491f7a-ae72-4317-b52b-5038030bfd19"
  },
  ExpectedSequenceToken: "49612946959762469195597770664378024234801679062591571890",
  Message_: "The given sequenceToken is invalid. The next expected sequenceToken is: 49612946959762469195597770664378024234801679062591571890"
}

The issue seems to be related to one of the pods sending a message and incrementing its log sequence token, but then the other pod sends a log with the old sequence token. More details are in this stack overflow post.

// TODO osbuild-composer-api isn't available in rhel8
// stage('EL8') {

however for clouddot we're expected to be deploying this on an EL8 system so IMO it makes sense to continue testing even if we don't ship any RPMs.

Are there any issues/problems preventing us from building and testing on EL8 ?

	"rhel-8.2" \| "rhel-8.3" \| "rhel-8.4")
	DISTRO="rhel-84"
	SSH_USER="cloud-user"