• docker
• docker-compose

./doit.sh start

1. (optional) Modify r10k config file located at custom/r10k/r10k.yaml

2. Run r10k to deploy environments

   docker-compose exec puppet /r10k

   Note: No output indicates success. Otherwise, errors will be listed and the relevant logfile noted.

docker-compose exec agent-centos puppet agent -t

After commiting changes on a "topic" branch of the repo:

1. Update enc for the node to use a topic branch

   docker-compose exec puppet server enc_adm --topic topic/aloftus/update_module_versions agent-centos-1.internal
   docker-compose exec puppet server enc_adm -l

2. Run puppet agent

   docker-compose exec agent-centos puppet agent -t

See: vagrant/README

• Reset the entire environment to start from scratch

  ./doit.sh reset

• Remove containers and images, leave local customizations in place

  ./doit.sh clean

• Start puppet server only

  ./doit.sh start puppet

• Stop all containers

  ./doit.sh stop

• Stop only a specific container

  ./doit.sh stop <container_name>

• Run puppet agent in dry run mode (don't make any changes, only list what would be done)

  docker-compose exec agent-centos puppet agent -t --noop

• Restart puppetserver (needed after, for instance, making config or cert changes)

  docker-compose exec puppet server pkill -HUP -u puppet java
  docker logs server #optional, to monitor server restart

• Exec a bash shell in the puppet master container

  docker-compose exec puppet bash

  OR

  docker-compose exec puppet server bash

• Add node agent-centos-3 to enc

  docker-compose exec puppet server enc_adm --add --fqdn agent-centos-3.internal

• Check enc contents

  docker-compose exec puppet server enc_adm -l

1. Get more ENC help

docker-compose exec puppet server enc_adm --help

R10K has a known issue RK-323 that causes issues with cached values or modules. This workaround deletes all caches and currently deployed modules. It is safe to use always, even if the issue isn't currently happening, however it causes the r10k run to take longer since all the repos and modules must be re-downloaded.

# workaround only - delete all currently deployed environments and r10k cache
docker-compose exec puppet bash -c 'rm -rf /etc/puppetlabs/code/* /var/cache/r10k'

# workaround , then re-run r10k deploy
docker-compose exec puppet bash -c 'rm -rf /etc/puppetlabs/code/* /var/cache/r10k; /r10k'

• Create an ssh key to use as a deploy key

  mkdir -p custom/r10k/ssh
  ssh-keygen -t ed25519 -f custom/r10k/ssh/private-hiera-deploy-key

• Install public portion of deploy key on the git server
  • Refer to your specific git server documentation
• Create necessary ssh config to tunnel through any bastion/proxy hosts

  vim -p custom/r10k/ssh/config

  # SAMPLE SSH CONFIG
  Host bastion
      Hostname bastion.fqdn
      User your_user_name
  Host proxy
      Hostname proxy.fq.dn
      User a_valid_username
      ProxyCommand ssh -W %h:%p bastion
  Host git-sec
      Hostname git-secure.f.q.d.n
      User git
      PreferredAuthentications publickey
      IdentityFile /etc/puppetlabs/r10k/ssh/private-hiera-deploy-key
      ForwardX11 no
      ProxyCommand ssh -W %h:%p proxy
  Host *
  ServerAliveInterval 60
  ServerAliveCountMax 3
  ControlMaster auto
  ControlPath ~/%l-%r@%h:%p
  ControlPersist 2d
  

• Link custom ssh directory to root's home inside the container

  docker-compose exec puppet ln -s /etc/puppetlabs/r10k/ssh /root/.ssh
  docker-compose exec puppet chown root:root /root/.ssh/config

• Initialize ssh connection from container to the secure git server

  # start a shell in the container
  docker-compose exec puppet bash
  # make initial connection to get-sec
  # ...will require manual login to bastion, proxy, etc.
  ssh -T git-sec
  # exit the container
  exit

• Verify non-interactive login (re-uses the authenticated channel created above)

  docker-compose exec puppet ssh -T git-sec

  Note: If password prompts continue, might have to login directly to each host in the path. Check for files (inside the container), should have one per host: /root/puppet.internal-<USER>@<HOST>:22=
• Ensure r10k.yaml uses ssh for access to private hiera
  • The "source" for private hiera should use the git@server:repo format, such as:

  sources:
    private-hiera:
      remote: git@git-sec:lsst-it/hiera-private.git

• Verify r10k access to all repos listed in r10k.yaml

  docker-compose exec puppet bash -c 'awk "\$1==\"remote:\"{print \$NF}" /etc/puppetlabs/r10k/r10k.yaml | xargs -n1 git ls-remote'

  NOTE: from within container, just run:

  awk '$1=="remote:"{print $NF}' /etc/puppetlabs/r10k/r10k.yaml | xargs -n1 git ls-remote

• From now on, while puppetserver container is up, run r10k as usual...

  docker-compose exec puppet /r10k