cattle-ops / terraform-aws-gitlab-runner Goto Github PK

If runner tags are passed in using the "tags" variable, and those runner tags contain ":" (which we're using for namespacing tags), the string passed into the gitlab runner's user_data is incorrect, splitting the name of the key into two (or more) strings.

Moonpig@f97efeb is an alternative (tf 12 only) strategy to processing the keys, which treats those keys correctly.

If you're okay with that implementation, I can add an example to runner-default(?) and PR this change.

Terraform has launched the version 0.12, are you @npalm planning move to terraform 0.12 ?
I can do a pull request with the changes if you want.

Create a test set for example with Terraform Kitchen sink / terratest to verify a deployment

The runners are using a local cache. Shared cache is not used.

currently we have no way to put a name different than the environment, and it's a bit confusing to have a instance named "dev"
that's especially true if one wants to have several ASG , like one with GPU instances , tagged with "has-gpu" (to run some specific task) and one named "low-cpu-instances" , both in the same environment

Hi @npalm and thank you for this great module, just starting with gitlab CI.

I wanted to enable ssh in the runner instance by using the variable:

enable_gitlab_runner_ssh_access=true

after running the plan the first time with the default value at "false"

It does create a new sg with port 22 (aws_security_group.runner-ssh) but the launch configuration still refers to the first created sg (aws_security_group.runner)

what am I missing ?
Thank you

Add an option to disable SSH (ingress) connection to the runner instance. The default could be disable access.

would it make sense to update to Amazon Linux 2018.03. Should I give it a try and create a MR?

https://aws.amazon.com/de/amazon-linux-ami/2018.03-release-notes/

I'm running this on Windows. Since the template tries to chmod keys that it generates, I overrode with a key I already had. However, it still tries to chmod.

On an apply of the runner-default example, this seemed to be the only blocker for Windows execution.

Some Possible Fix Implementations:

1. Make permissions commands natively compatible with Windows.
2. Assume that permissions on stipulated (non-generated) keys are correct and do not run permissions commands on them - let it fall through to ssh error or maybe warn that permissions were not verified.

I'm not sure it is a good idea to change permissions on existing keys in any case?

Windows 10, PowerShell 7.0.0-preview.3, Terraform 12.7

Hello,

it looks like since version 2.1.0 the option amazonec2-iam-instance-profile is missing.
Looks like this was removed by accident when resolving merge conflicts.

The diff is here:
2.0.0...2.1.0#diff-037af380e1e9a26cfbcaa481f820aacbL42

Could you add the missing option again?

Background

Currently, the tags for runners (not the bastion) are hard-coded to:

amazonec2-tags=environment,${environment}

(see current runner-config.tpl)

Proposal

• It'd be good if users could specify their own / extra tags for runners.
• Perhaps the current environment could just be a default key/value pair, meaning users could override existing ones too (for example, we usually use Environment (capitalised)).

Currently it is required to register the runner before running the terraform scripts. In branch https://github.com/npalm/terraform-aws-gitlab-runner/tree/feature/auto-register-runner I am preparing scripts so that the runner can register itself based on the registration token. Runner tokens will be kept in the parameter store.

I need to launch a1 machine type for the runners.
spot instances don't exist in my region for this type of ec2 machines.
In the context of amazonec2-request-spot-instance=false, when starting my gitlab ci pipeline, I can see in the runner logs:

gitlab-runner[2517]: #033[31;1mERROR: Error creating machine: Error in driver during machine creation: Error launching instance: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message:xxxxxxxxx

Hello,
Thanks for this awesome Module.

While trying it out I ran into a problem. The description of the variable enable_manage_gitlab_token is kind of misleading.

variable "enable_manage_gitlab_token" {
  description = "Boolean to enable the management of the GitLab token in SSM. If `true` the Gitlab token will be managed via terraform state. If `false` the token will still be stored in SSM however, it will not be managed via terraform."
  default     = true
}

The first sentence of the description states, that this will enable the management via SSM. But the following sentences are implying the opposite.

As I want to manage it in terraform I thought the default-value would be great, when reading the description (aside the first sentence) in conjunction with the variable_name (since we are in terraform and i would like to enable the management via terraform)

Looking at main.tf it gets clear, that the first sentence of the description is telling the truth.

Besides the semantical point of view the description should be updated anyways imho.

Best regards

user-data should not write sensitive information to stdout / logging. Currently the GitLab runner token is visible

I started seeing this error in the Gitlab CI jobs, after updating the gitlab runner instance:

Running with gitlab-runner 11.10.1 (1f513601)
  on aws-spot-gitlab-runner DbkeXXxX
ERROR: Preparation failed: fork/exec /usr/bin/docker-machine: exec format error
Will be retried in 3s ...
ERROR: Preparation failed: fork/exec /usr/bin/docker-machine: exec format error
Will be retried in 3s ...
ERROR: Preparation failed: fork/exec /usr/bin/docker-machine: exec format error
Will be retried in 3s ...
ERROR: Job failed (system failure): fork/exec /usr/bin/docker-machine: exec format error

After some troubleshooting, I found out that the docker-machine binary was an HTML file:

# docker-machine ps
/usr/bin/docker-machine: line 1: syntax error near unexpected token `newline'
/usr/bin/docker-machine: line 1: `<!DOCTYPE html>'

Checking the contents of the file, I found a Github error page, which contained messages like this (content cleaned up for succintness):

...
    <title>GitHub — Unexpected problem serving your request</title>
...
      <h1>There was an unexpected problem serving your request</h1>
      <p>Please try again and contact us if the problem persists including 35FE:25F22:C675652:12DBD87E:5CED2B45 in your message</p>
...

It's a bit unexpected because judging from the code here: https://github.com/npalm/terraform-aws-gitlab-runner/blob/develop/template/gitlab-runner.tpl#L19 -- I'd expect the script to fail if the curl execution failed, but maybe it isn't failing if it gets a response, even if it's an error message?

Hi,

when I use your module twice, so I can use different sizes of instances for different jobs, I see no possibility to use the same bucket as cache.

The reason for this, is the fact that the module will always try to create a bucket, but if this bucket already exists, terraform will fail.

Maybe it makes sense to remove the cache-bucket from this module and better inject a cache_bucket-name into the module as parameter to be able to get the required information of the bucket inside of the module.

I tried to work around this issue, by introducing a count-variable for the bucket in conjunction with an extra parameter, but that forced me to add count-variables to most of the other resources too, which enforces a lot of duplication.

• lint
• validate
• check modules

Currently, I need to provision a gitlab-runner only for a stage, would be great if the module support count property to provisioning if the stage is "some"

We want in some cases to debug l pipelines and log is key. Can you add this variable to the runners config input?

https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/advanced-configuration.md#the-runners-section

Just spent the better part of a day trying to setup the runner via this module without success. After some digging around I realized that there are some settings that are necessary but not provided in the documentation (but exist in the example folder):

  runners_off_peak_periods = "[\"* * 0-9,17-23 * * mon-fri *\", \"* * * * * sat,sun *\"]" 

If runners_off_peak_periods value is not set the setup script will fail due to empty variable. This either need to be a required variable or a default value needs to be set.

Thanks for a great module, apart from this little failure it works like a charm!

Creation of instances / docker machines should be done by using an aws instance profile instead of an IAM user.

Limit the access to the runner instance by CIDR.

Hello,

First, thank you for this module, it makes setting up a CI infrastructure a breeze ;-)

We are currently facing performances issues during docker push/pull operations due to IOPS limitations on EBS storage (see AWS documentation)
We successfully tried to mount /var/lib/docker to tmpfs, using the runner's tmpfs configurations.

Those configurations cannot be customized in the module so the suggestion is, for instance, to add runners_services_tmpfs and runners_tmpfs variables. What would you think ?

It would be nice if this module supports AWS Systems Manager Session Manager. We can access runners in a private subnet by the session manager instead of a bastion and SSH.

We need the following extra work:

• Add an input parameter like enable_gitlab_runner_ssm_access (boolean).
• Add the session manager permissions to the instance role of runners.
• Do not create a ssh key pair resource if ssh_public_key = "" and ssh_key_pair = "".

It would be cool if we could support session_server in the gitlab runner config for interactive webterminals

https://docs.gitlab.com/ee/ci/interactive_web_terminal/index.html
https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section

I continue without being able to exeute runners. Docker machine create spot instancesa but nos job is running

75 gitlab-runner[8742]: time="2018-08-24T10:41:33Z" level=info msg="Error getting SSH command to check if the daemon is up: ssh command error:" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 ope
ration=create
Aug 24 10:41:33 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:33Z" level=info msg="command : sudo docker version" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=create
Aug 24 10:41:33 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:33Z" level=info msg="err : exit status 1" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=create
Aug 24 10:41:33 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:33Z" level=info msg="output : sudo: docker: command not found" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=create
Aug 24 10:41:34 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:34Z" level=info msg="Error getting SSH command to check if the daemon is up: ssh command error:" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c ope
ration=create
Aug 24 10:41:34 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:34Z" level=info msg="command : sudo docker version" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:34 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:34Z" level=info msg="err : exit status 1" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:34 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:34Z" level=info msg="output : sudo: docker: command not found" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:36 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:36Z" level=info msg="Error getting SSH command to check if the daemon is up: ssh command error:" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 ope
ration=create
Aug 24 10:41:36 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:36Z" level=info msg="command : sudo docker version" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=create
Aug 24 10:41:36 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:36Z" level=info msg="err : exit status 1" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=create
Aug 24 10:41:36 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:36Z" level=info msg="output : sudo: docker: command not found" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=create
Aug 24 10:41:37 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:37Z" level=info msg="Error getting SSH command to check if the daemon is up: ssh command error:" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c ope
ration=create
Aug 24 10:41:37 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:37Z" level=info msg="command : sudo docker version" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:37 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:37Z" level=info msg="err : exit status 1" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:37 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:37Z" level=info msg="output : sudo: docker: command not found" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:40 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:40Z" level=info msg="Error getting SSH command to check if the daemon is up: ssh command error:" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c ope
ration=create
Aug 24 10:41:40 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:40Z" level=info msg="command : sudo docker version" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:40 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:40Z" level=info msg="err : exit status 1" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:40 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:40Z" level=info msg="output : sudo: docker: command not found" driver=amazonec2 name=runner-17ecdb74-runner-1535106043-bcbc362c operation=create
Aug 24 10:41:41 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:41Z" level=error msg="Error creating machine: Error running provisioning: Maximum number of retries (60) exceeded" driver=amazonec2 name=runner-17ecdb74-runner-15351
06043-bd5f07e0 operation=create
Aug 24 10:41:41 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:41Z" level=info msg="The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag." driver=amazonec2 name=runner-17
ecdb74-runner-1535106043-bd5f07e0 operation=create
Aug 24 10:41:42 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:42Z" level=warning msg="Machine creation failed, trying to provision" error="exit status 1" name=runner-17ecdb74-runner-1535106043-bd5f07e0
Aug 24 10:41:43 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:43Z" level=info msg="Waiting for SSH to be available..." name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=provision
Aug 24 10:41:43 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:43Z" level=info msg="Detecting the provisioner..." name=runner-17ecdb74-runner-1535106043-bd5f07e0 operation=provision
Aug 24 10:41:45 ip-10-0-1-75 gitlab-runner[8742]: time="2018-08-24T10:41:45Z" level=error msg="Error creating machine: Error running provisioning: Maximum number of retries (60) exceeded" driver=amazonec2 name=runner-17ecdb74-runner-15351
06043-bcbc362c operation=create

When the aws_ssm_parameter.runner_registration_token contains an underscore I'm getting the error:
Error creating machine: Invalid hostname specified. Allowed hostname chars are: 0-9a-zA-Z . -.
I solved it by recreating a runner to get a token without underscore. But maybe it makes sense to substitute underscore with a dash?

This is caused by the conditionals used in the security group to enable ssh for the runner instance

Make it possible to extend the default tags.

Logs are not popping up in cloudwatch. Looks like last changes in policies broke the logging.

I am trying to use this module in the public subnet. It seems the runner EC2 instance cannot reach the Internet because it does not have a public IP address due to the following config:

https://github.com/npalm/terraform-aws-gitlab-runner/blob/2f45f1d73ba0cffd310a859cf30f08e88b8a1b03/main.tf#L218

I changed the above to true and finally the runner has been registered successfully. Is this expected design? Some extra config needed?

Thank you for the great work.

In some cases when using spot, I'd like to be able to provide an EIP and reuse them in case the spot is terminated.

This is useful to prevent changing inbound rules that uses the runner's public IP as a source in their inbound rule.

This is a submodule used internally by //. Using this submodule on its own is not recommended.

Submodules without a README or README.md are considered to be internal-only by the Terraform Module Registry. If this submodule should not be considered internal, add a readme which describes what this submodule is for and how it should be used.

For more information about the expected structure of modules in the Registry, see the Standard Module Structure in the Terraform Module Registry documentation.

Hi there,

I am currently building a simple runner setup which shall be deployed to two different AWS regions, first eu-central-1, and second cn-northwest-1.

The european region is already running flawlessly by using this module (thanks for that)! However, today I started to work on the chinese region. There are multiple issues occurring there, for instance, that you do not have a "SecureString" type in parameter store. I have forked the repo and added support to parameterize that type to be a "regular" string. Probably thats not a good solution in particular but it is the only solution for the chinese regions currently (considering Parameter Store).

Additionally, there are also other issues with IAM roles which I will try to figure out in the next days.

Would you like me to create a PR with the changes or would you like to keep it separated because you do not want "support/compatibility" for the chinese AWS regions?

Best regards!

Hi. First of all, thanks for a great module (and blog post). Cool stuff.

Background

I've been setting it up here for a self-hosted, locked down Gitlab server. After getting the (organisation) region, VPC and subnet fixed for $WORK (but using data at least), the runner is green.

Problem

However it is failing with

Error creating machine: Error in driver during machine creation: Error request spot instance: 
InvalidParameterValue: Value (us-east-2a) for parameter availabilityZone is invalid. 
Subnet 'subnet-****************' is in the availability zone us-east-2c

Clearly I guess the subnet we've been given is in us-east-2c AZ but at no point can I seem to specify this, as the template seems not to have this MachineOption.

I guess the example always creates the subnet in zone a too, so maybe this is why it's never much of an issue.

Proposed improvement

It would be good to have another variable for aws_availability_zone or similar.

Workaround

Use docker_machine_options, I'm guessing?

I think it has now been replaced by m5.large, shouldn't it be the new default ?

Using mostly the default setup with v3.3.0 and the jobs keep getting a 403 forbidden when trying to access the shared s3 cache. I checked the docker-machine-role assuming it would give each runner permission to access the cache but it looks like there are no permissions attached to that role by default. I tested out explicitly adding a s3:* permissions to that role just to see if that would make things work and I'm still getting 403s on cache pull.

Any ideas?

Hi,

I have the following issue.

In my terraform configuration for GitLab runners I specify instance type equals to "t2.medium".
After terraform apply I can see the new Launch Configuration and its Instance Type param is also set to "t2.medium".
http://take.ms/bZer0
But when I launch a new pipeline in GitLab I see that a new spot instance for runner is created and its type is "t2.small" instead of "t2.medium".
http://take.ms/GFzAa

Here is my terraform config:

module "gitlab-runner" {
  source = "npalm/gitlab-runner/aws"

  aws_region     = "${var.region}"
  environment    = "${var.prefix}-ci-runners"
  ssh_public_key = "${file("generated/id_rsa_ci.pub")}"

  vpc_id                  = "${module.networking.vpc_id}"
  subnet_id_gitlab_runner = "${element(module.networking.public_subnet_ids, 0)}"
  subnet_id_runners       = "${element(module.networking.public_subnet_ids, 0)}"

  runners_name       = "${var.prefix}-ci-runner"
  runners_gitlab_url = "https://gitlab.com/"
  runners_token      = "${var.runner_token}"
  cache_user         = "${var.prefix}-new-cache_user"

  runners_off_peak_timezone   = "Europe/Amsterdam"
  runners_off_peak_idle_count = 0
  runners_off_peak_idle_time  = 60
  runners_off_peak_periods = "[\"* * 0-9,17-23 * * mon-fri *\", \"* * * * * sat,sun *\"]"
  gitlab_runner_version = "11.1.0-1"
  // https://aws.amazon.com/ec2/spot/pricing/
  docker_machine_spot_price_bid = "0.04"
  instance_type = "t2.medium"
  docker_machine_instance_type = "t2.small"
  runners_use_private_address = "false"
  allow_iam_service_linked_role_creation = "false"
}

Hi,
I'm not sure if it makes sens for the variable enable_manage_gitlab_token to be false, but anyway it makes the plan fails if it is.

Error: Error running plan: 1 error(s) occurred:

* module.runner.aws_iam_role_policy_attachment.ssm: 1 error(s) occurred:

* module.runner.aws_iam_role_policy_attachment.ssm: Resource 'aws_iam_policy.ssm' not found for variable 'aws_iam_policy.ssm.arn'

Background

When adding only one instance type for a spot request (in the wizard), it becomes clear this is affecting the availability (which brings down our gitlab runners sometimes with capacity-not-available)

See related #76

Proposal

• Deprecate instance_type in favour of instance_types, a list (e.g. instance_types = ['m4.large', 'm5.large'], and, err, somehow make this work downstream...

Notes

This probably isn't possible until Fleets / multiple instance types are supported in docker-machine, unless perhaps it could be done manually, or within Gitlab Runner itself somehow... 🤔

The example link throughout documentation returns 404. https://registry.terraform.io/modules/npalm/gitlab-runner/aws/3.2.0

Hi

How can i change a volume size of the instance created? there is no option!

Thanks in advance
Daniel

Sometimes the gitlab runner instance is not starting properly. After terminating the instance a new one will be created by the autolaunch group. This second one starts correctly.

System log:

user-data: + chmod +x /tmp/docker-machine
user-data: + cp /tmp/docker-machine /usr/local/bin/docker-machine
user-data: + ln -s /usr/local/bin/docker-machine /usr/bin/docker-machine
user-data: ++ aws ssm get-parameters --names runner-public-runner-token --with-decryption --region ap-south-1
user-data: ++ jq -r '.Parameters | .[0] | .Value'
user-data: /var/lib/cloud/instance/scripts/part-001: line 96: jq: command not found
user-data: + token=
Apr 03 06:55:56 cloud-init[2236]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [127]

Background

At $WORK, we have a pre-defined VPC, and one subnet within that (but want more to avoid any further capacity issues) for Gitlab & runners.

Having multiple AZs is entirely possible, as the spot request wizard has checkboxes for each:

Proposal

• Multiple availability zones (via aws_zone) can be specified as parameters, or
• Multiple subnets (via a new subnet_ids_runners perhaps?) in various AZs, with the AZ inferred automatically, so deprecating aws_zone perhaps?

Notes

I don't know if this necessarily makes sense, but am definitely keen to find ways to improve our capacity-not-available problems (not pricing), and this seems like one major cause

Can you help me? Is the spot price to low? No clue...

This is my config file
`concurrent = 10
check_interval = 0

[[runners]]
name = "iob-gitlab-runner"
url = "https://gitlab.com/"
token = "XXXXX"
executor = "docker+machine"
limit = 0
[runners.docker]
tls_verify = false
image = "docker:17.11.0-ce"
privileged = true
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
Type = "s3"
ServerAddress = "s3-eu-west-1.amazonaws.com"
AccessKey = "XXXX"
SecretKey = "XXXXX"
BucketName = "442729568030-gitlab-runner-cache"
Insecure = false
[runners.machine]
IdleCount = 0
IdleTime = 600
MachineDriver = "amazonec2"
MachineName = "runner-%s"
MachineOptions = ["amazonec2-access-key=XXXXX", "amazonec2-secret-key=XXX", "amazonec2-instance-type=m4.large", "amazonec2-region=eu-west-1", "amazonec2-vpc-id=vpc-018411147ac4eff0d", "amazonec2-subnet-id=subnet-02b62af9c4fc90770", "amazonec2-private-address-only=true", "amazonec2-request-spot-instance=true", "amazonec2-spot-price=0.04", "amazonec2-security-group=preproduction-docker-machine20180731072557855900000001", "amazonec2-tags=environment,preproduction"]
OffPeakTimezone = ""
OffPeakIdleCount = 0
OffPeakIdleTime = 0`

I've recently updated our modufle configuration to change the terraform ressource name

from

module "runner_c5_xlarge_eu_west_3_b" {                                                                                                                                                       
  source  = "npalm/gitlab-runner/aws"                                                                                                                                                        
  version = "4.1.0"                                                                                                                                                                          
                                                                                                                                                                                             
  aws_region = "eu-west-3"                                                                                                                                                                   
  aws_zone   = "b"                                                                                                                                                                           
                                                                                                                                                                                             
  # Here we set a different environment than the previous (and other runner modules)                                                                                                         
  # because every resource created by this module is indexed by this variable                                                                                                                
  # so in order not to have name conflicts, we need a different environment for                                                                                                              
  # each runner                                                                                                                                                                              
  environment = "Service-4"        
  ....
}

to

module "runner_c5_large_eu_west_3_b" {                                                                                                                                                       

(only the module name as changed)

when running the apply we got

Error: Error import KeyPair: InvalidKeyPair.Duplicate: The keypair 'Service-4-gitlab-runner' already exists.
	status code: 400, request id: f132a471-57ea-4106-97bb-5e8beb85a0a5

  on .terraform/modules/runner_c5_large_eu_west_3_b/npalm-terraform-aws-gitlab-runner-31b0bab/main.tf line 1, in resource "aws_key_pair" "key":
   1: resource "aws_key_pair" "key" {



Error: Error creating IAM Role Service-4-instance-role: EntityAlreadyExists: Role with name Service-4-instance-role already exists.
	status code: 409, request id: cbb8b28e-ca34-11e9-b038-abe555558e62

  on .terraform/modules/runner_c5_large_eu_west_3_b/npalm-terraform-aws-gitlab-runner-31b0bab/main.tf line 288, in resource "aws_iam_role" "instance":
 288: resource "aws_iam_role" "instance" {



Error: Error creating IAM policy Service-4-docker-machine: EntityAlreadyExists: A policy called Service-4-docker-machine already exists. Duplicate names are not allowed.
	status code: 409, request id: ca24f337-ca34-11e9-aaa5-3d97689ee397

  on .terraform/modules/runner_c5_large_eu_west_3_b/npalm-terraform-aws-gitlab-runner-31b0bab/main.tf line 302, in resource "aws_iam_policy" "instance_docker_machine_policy":
 302: resource "aws_iam_policy" "instance_docker_machine_policy" {



Error: Error creating IAM Role Service-4-docker-machine-role: EntityAlreadyExists: Role with name Service-4-docker-machine-role already exists.
	status code: 409, request id: cc06365f-ca34-11e9-ae4f-bb716bff761d

  on .terraform/modules/runner_c5_large_eu_west_3_b/npalm-terraform-aws-gitlab-runner-31b0bab/main.tf line 331, in resource "aws_iam_role" "docker_machine":
 331: resource "aws_iam_role" "docker_machine" {

I guess it happens because of a ordering problem in the order the resources are deleted / recreated ?