mawalu / hetzner-gitlab-runner Goto Github PK

Autoscale gitlab ci runners on hetzner cloud

Dockerfile 100.00%

hetzner hetzner-cloud gitlab-ci docker-image docker gitlab-runner docker-machine ci-jobs auto-scale

hetzner-gitlab-runner's Introduction

Auto-scale CI jobs with Gitlab Runner and the Hetzner Docker Machine driver

With this patched GitLab Runner image, you can auto-scale your CI jobs on Hetzner Cloud. The patch installs the Hetzner Docker Machine driver and is available on Docker Hub.

You can follow the official GitLab docs for configuring the GitLab Runner with docker machine, just use this image as a drop in replacement for gitlab/gitlab-runner.

See the example config.toml and docker-compose.yaml below for the hetzner specific docker machine options.

Usage

Use this image instead of the gitlab/gitlab-runner image and set MachineDriver to hetzner in your runner configuration.

Example config.toml:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker-machine"
  url = "https://gitlab.com"
  token = "your-token"
  executor = "docker+machine"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 536870912
  [runners.cache]
  [runners.machine]
    IdleCount = 0
    IdleTime = 1800
    MaxBuilds = 10
    MachineDriver = "hetzner"
    MachineName = "runner-%s"
    MachineOptions = [
      "hetzner-api-token=hetzner-api-token",
      "hetzner-image=ubuntu-18.04",
      "hetzner-server-type=cx31",
    ]

Example docker-compose.yaml:

version: "3.8"

services:
  hetzner-runner:
    image: mawalu/hetzner-gitlab-runner:latest
    volumes:
      - "./hetzner_config:/etc/gitlab-runner"

Docker Errors

If you face docker problems at runtime (such as the one below), you can specify a Docker version in the MachineOptions.

 MachineOptions = [
      "engine-install-url=https://releases.rancher.com/install-docker/19.03.9.sh",
    ]

Possible error that is caused by Docker:

ERROR: Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded  driver=hetzner name=runner-xxx-xxx operation=create

See this issue for more information.

Versions

Currently this image is build using the gitlab/gitlab-runner:latest image and the latest docker-machine hetzner plugin. If you need builds for another version feel free to open an PR.

License

MIT

hetzner-gitlab-runner's People

Stargazers

Watchers

Forkers

solidnerd amonforstmann gabrielke patschi l-u-c-k-y phaus janhelke jawabuu lukasleitsch inspired-consulting interfacewerk buzzwordy tkreuder msltwtf heyimmarc silviupanaite hojerst

hetzner-gitlab-runner's Issues

tls_verify = false in exmple config

Usually tls_verify = false is quite a bad thing to do. Providing this "insecure" parameter in a default example config ... questionable at best and causing insecure installations for most people.

Could you explain which communication is usually protected with this TLS check / parameter?

Missing Documentation

First nice work, I was looking for something like this, yet your Readme lacks a bit the actual instructions.

I assume that you need to have 1 host somewhere, preconfigured with your config.toml somewhere static. One has to register this one host as runner with gitlab-ci and this custom mawalu/hetzner-gitlab-runner image instead of the normal gitlab ci image.

And that runner, will use hetzner to spawn new VMs, which then will do the actual build and so on.

If you would like that other people use this image actually, then you have to provide some instructions about how to get your code running.

How frequent is a new image pushed?

First of all, thanks for the great work! The latest runner image that was pushed is using Gitlab Runner 14.1.0. Would it be possible to trigger a new build&push with the latest Gitlab RUnner image?

Thanks!

General Questions about autoscaling & downscaling

Hi martin,

First thanks for hetzner-gitlab-runner. I find it amazing!

Some questions:

I have everything now running, a hetzner machine is spawning and doing the task. Yet after the task its not automatically deleted. How is deletion controlled?
My pipeline has multiple jobs that can run in parallel, yet the expected autospawning only spawned 1 runner. I assume that I can set concurrent = 1 to 10 or something like that to allow more to spawn? Or would those jobs end up on the same runner?

Best regards
Sascha

Better fix for docker error / add possbility to use newest docker version

Hi,

I just wanted to leave some notes as I recently invested quite some time to get the new docker 20.10 running. (Ubuntu 22.04 image for example won't build on older docker versions)
The trick is to use gitlab's docker machine here: https://gitlab.com/gitlab-org/ci-cd/docker-machine
(The original docker machine is in maintenance mode and the guys from docker refuse to fix the issue for docker 20.10)

Then you can get rid of this MachineOptions engine-install-url parameter , or upgrade it to:

MachineOptions = [
     "engine-install-url=https://releases.rancher.com/install-docker/20.10.12.sh",
]

Failing with

It seem that the newly spawned worker can't connect... any idea what goes wrong there?

Is it also possible to log into the worker somehow? I can connect in the hetzner console, but I don't know the user and/or password...

hetzner-runner_1  | Copying certs to the remote machine...              driver=hetzner name=runner-ux2alzk4-runner-1607630955-16c49b06 operation=create
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | Setting Docker configuration on the remote daemon...  driver=hetzner name=runner-ux2alzk4-runner-1607630955-16c49b06 operation=create
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | WARNING: Failed to process runner                   builds=0 error=failed to update executor: no free machines that can process builds executor=docker+machine runner=uX2ALzK4
hetzner-runner_1  | ERROR: Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded  driver=hetzner name=runner-ux2alzk4-runner-1607630955-16c49b06 operation=create
hetzner-runner_1  | WARNING: Problem while reading command output       error=read |0: file already closed
hetzner-runner_1  | WARNING: Machine creation failed, trying to provision  error=exit status 1 name=runner-ux2alzk4-runner-1607630955-16c49b06

Which gitlab/gitlab-runner image to replace?

Hi @mawalu

Many thanks for this repo, it already helped me in setting up docker machine with Hetzner.

I have installed Docker Machine + GitLab Runner on the server, but I am confused with the following instructions:

just use this image as a drop in replacement for gitlab/gitlab-runner.

Which image am I supposed to replace? I have followed the GitLab docs (maybe I missed something).

Will open a PR with updated instructions if that's fine for you afterwards.

Thanks!

Machine Creation Loop of Death

Hey,
when creating a Job, a Machine gets created as supposed to, but after the machine "spun up" the Docker daemon just times out and a new Machine gets created without deletion of the old one.

hetzner-runner_1  | Creating machine...                                 driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8) Creating SSH key...  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8) SSH key not found in Hetzner. Uploading...  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8) Creating Hetzner server...  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8)  -> Creating server runner-l7mc7so7-1635714803-ed01fdb8[15658509] in create_server[299061970]  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8)  -> Server runner-l7mc7so7-1635714803-ed01fdb8[15658509]: Waiting to come up...  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8) Using public network ...  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8)  -> Server runner-l7mc7so7-1635714803-ed01fdb8[15658509] ready. Ip 65.21.181.221  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Waiting for machine to be running, this may take a few minutes...  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Detecting operating system of created instance...   driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Waiting for SSH to be available...                  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Detecting the provisioner...                        driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Provisioning with ubuntu(systemd)...                driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Installing Docker...                                driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Copying certs to the local machine directory...     driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Copying certs to the remote machine...              driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Setting Docker configuration on the remote daemon...  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | ERROR: Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded  driver=hetzner name=runner-l7mc7so7-1635714803-ed01fdb8 operation=create
hetzner-runner_1  | Skipping provision retry on failed machine          error=exit status 1 name=runner-l7mc7so7-1635714803-ed01fdb8
hetzner-runner_1  | ERROR: Machine creation failed                      error=exit status 1 name=runner-l7mc7so7-1635714803-ed01fdb8 time=2m25.185163291s
hetzner-runner_1  | WARNING: Requesting machine removal                 lifetime=2m25.185281801s name=runner-l7mc7so7-1635714803-ed01fdb8 now=2021-10-31 21:15:48.456298529 +0000 UTC m=+197.856950442 reason=Failed to create used=2m25.185282591s usedCount=0
hetzner-runner_1  | WARNING: Stopping machine                           lifetime=2m25.20150685s name=runner-l7mc7so7-1635714803-ed01fdb8 reason=Failed to create used=16.190793ms usedCount=0
hetzner-runner_1  | Stopping "runner-l7mc7so7-1635714803-ed01fdb8"...   name=runner-l7mc7so7-1635714803-ed01fdb8 operation=stop
hetzner-runner_1  | (runner-l7mc7so7-1635714803-ed01fdb8)  -> Shutting down server runner-l7mc7so7-1635714803-ed01fdb8[15658509] in shutdown_server[299063151]...  name=runner-l7mc7so7-1635714803-ed01fdb8 operation=stop
hetzner-runner_1  | Running pre-create checks...                        driver=hetzner name=runner-l7mc7so7-1635714949-f1598b9f operation=create
hetzner-runner_1  | Creating machine...

All local runners seem to work fine, only the Hetzner Runner has some issues.

ERROR: Runner https://gitlab.com is not healthy and will be disabled!

Hello,

i have steup the gitlab runner but i doesnt have a connection to gitlab to register runners.

Logs:
hetzner-runner_1 | ERROR: Checking for jobs... forbidden runner=J1vnM1Vx hetzner-runner_1 | ERROR: Checking for jobs... forbidden runner=J1vnM1Vx hetzner-runner_1 | ERROR: Checking for jobs... forbidden runner=J1vnM1Vx hetzner-runner_1 | ERROR: Runner https://git.domain.deTOKEN is not healthy and will be disabled!

Can anyone help me?