tozny / rancher-lets-encrypt Goto Github PK

View Code? Open in Web Editor NEW

91.0 16.0 26.0 659 KB

Automatically create and manage certificates in Rancher using Let's Encrypt webroot verification via a minimal service

License: Other

Python 97.15% Dockerfile 1.24% Shell 1.62%

rancher rancher-api rancher-service webroot python cert letsencrypt ssl-certificates tls-certificate

rancher-lets-encrypt's Introduction

Rancher Let's Encrypt Service

Let's Encrypt verification

Let's Encrypt has two methods of verifying ownership of domains. The first is through the addition of a custom DNS record (say acme-12321313.subdomain.domain.com). This is what https://github.com/janeczku/rancher-letsencrypt does. That service creates Let's Encrypt challenges via DNS resolution. The other way of proving ownership of domains is through a webserver webroot over HTTP.

Update: The janeczku/rancher-letsencrypt project now supports HTTP webroot verification. The Tozny project was created many months before this feature was added.

Our Service

With our environment, we wanted to do webroot verification for Let's Encrypt and Rancher. We wanted a service that would manage TLS certificates automatically, and renew them as needed. We also wanted this tightly integrated with Rancher for complete automation. This way load balancers (and other services) could automatically pick up certs through the Rancher API. Also, when we update a cert in Rancher, the load balancers will receive the updated cert with zero downtime. We also did not want to give keys for updating DNS records for our entire domain to every rancher environment for security purposes (isolation is best!)

Tozny has been using this service in production for over a year now, and has been battle tested. We renew over 40 subdomains regularly without issue.

How it Works

The service launches two containers:

letsencrypt-nginx
letsencrypt-python

The letsencrypt-nginx container is stock nginx, but shares the webroot with the letsencrypt-python service container. This way the letsencrypt-python container can add ACME challenges to the <host>/.well-known/acme-challenge/ directory on the webserver for verification. The python container is a sidekick of the nginx container. The containers are launched as a Rancher Service Account, so special environment variables containing the Rancher server API url, and access keys are passed into the container at runtime.

Example Rancher Load Balancer (HAProxy) GUI Config

^{(Based on Rancher GUI v1.3.3)}

Use the "Add Service" dropdown to select "Add Load Balancer" or edit an existing Load Balancer
If empty, fill in the Name
Enter the following into the Port Rules section for each server for which you are requesting a certificate:

Access	Protocol	Request Host	Port	Path	Target	Port
Public	HTTP	yourserver.name.com	80	/.well-known/	letsencrypt-nginx	80
^{Example "Target" is based on the default container name letsencrypt-nginx used by this project}

Note: If you are using custom haproxy.cfg settings to redirect http traffic to https (or wish to do so now), make sure to exclude the /.well-known/ directory using !{ url_dir /.well-known/ } as in:

frontend 80
  redirect scheme https code 301 if !{ url_dir /.well-known/ } !{ ssl_fc }

^{This example custom haproxy.cfg will merge the redirect setting with the default Rancher haproxy.cfg frontend definition and set up permanent ("301") redirects to HTTPS for all other HTTP traffic.}

Requirements

DNS control of domain names (ability to create host.subdomain.domain.com records to point to Rancher IP)
Front-end load balancer exposing a privileged port (less than 1024) to the internet for Let's Encrypt verification
This Rancher service
Rancher Cattle as Container Scheduler/Orchestrator
Rancher v1.1.4 - v1.4.2 (versions tested with this service)

How to use

Create a front end load balancer (or use the one in traffic-manager directory). If you are making one, you need to make sure it is a L7 HTTP load balancer on your chosen privileged port. This way the load balancer can redirect /.well-known/* traffic to the letsencrypt-nginx container for verification. You can then route all other traffic to your normal HTTP services. This way only during verification does traffic get directed to the letsencrypt-nginx container.

Rancher Compose

Use rancher-compose up to launch the stack in rancher. In order to get a Let's Encrypt Production certificate, you must set the environment variable STAGING=False. This will then tell the service to use the production Let's Encrypt api instead of the staging api. To use the environment file, you need to pass the path using the --env-file or -e option.

Rancher Catalog (UI)

Add this repository as a catalog to your rancher instance:

Open Rancher
Select Admin in the navigation
Select Settings

In the Catalog section you can add this catalog by entering a name (e.g. rancher-lets-encrypt), the URL to this repository and a branch.

Afterwards you will be able to select the new catalog from the Catalog menu item in the navigation. There you will find the Rancher Let's Encrypt Service. By clicking View Details you can configure the service to your needs and then launch it.

Certificate Workflows

"staging" refers to Let's Encrypt staging API. "production" refers to Let's Encrypt production API.

This flowchart/execution diagram shows all the cases the service deals with, and how it responds to different stages.

get certs from rancher API
- local copy of cert
  - cert in rancher
    - upgrade staging cert to production
      - create cert
      - push to rancher
    - upgrade self signed cert to production
      - create cert
      - push to rancher
    - rancher cert expired
      - local cert expired
        
        create cert (renew)
        
        push to rancher
      - local cert not expired
        
        push to rancher
  - cert not in rancher
    - local cert expired
      - create cert
      - push to rancher
    - local cert not expired
      - push to rancher
- no local copy of cert
  - create cert
  - push to rancher

rancher-lets-encrypt's People

Contributors

Stargazers

Watchers

rancher-lets-encrypt's Issues

Certbot and dependencies versions

Need to upgrade versions for certbot and acme.

Accidental Overwriting certificates that are not from LetsEncrypt

Steps to reproduce:

Add cert from other Certificate Authority (Godaddy, Verisign)
Name cert in Rancher UI the same as domain used in Rancher-lets-encrypt (test.example.com)
Start Rancher-lets-encrypt service
RLE looks for certificate in Rancher API that matches hostname (test.example.com)
RLE finds cert, but it is not signed by LetsEncrypt CA, so it deletes the existing (test.example.com) certificate
RLE then provisions a Lets-encrypt certificate matching (test.example.com) and uploads it through the Rancher API.

Outcome:
Old certificate from CA other than Lets-Encrypt is overwritten.

Desired:
Any certificate that is named the same as hostname, but not signed by either "Staging" or "Production" Lets-encrypt CA should be ignored.

[Logging] Logging should be easier to understand

We should come up with a standard format for logging, so if someone wants to parse it, or just read it, it is easier to understand. Right now there are lots of print statements that dont have a particular format.

Run certbot from with --non-interactive

8/28/2017 12:16:21 PM Skipped user interaction because Certbot doesn't appear to be running in a terminal. You should probably include --non-interactive or --force-interactive on the command line.

This is just a warning generated from certbot. It should be fixed to force certbot to know it is not expecting user input due to the automated nature of the service.

Automation

Hello,

Is it suppose to automatically create a new certificate when a new path is add in the load balancer ?
(Almost the same way JrCs/docker-letsencrypt-nginx-proxy-companion do ? )
Or Am I wrong ? and domain must be add to environment variable in letsencrypt-nginx and restarted ? )

Regards,

Update certbot and requirements version

The actual image certbot version is 0.13.0 and the current certbot version is 0.15.0. The certificates are not being issued because of this difference in versions, since acme protocol was changed from one version to the other.

Prepare for rancher web ui usage / add to catalog

Hi,

the service sounds great, but I can't get ist work easily with rancher web ui because of the env file. Would be great to see rancher-lets-encrypt at catalog or compatible for rancher ui usage.

Kubernetes support

Maybe it's worth to add support to Kubernetes, becouse Rancher 2 is on the horizont?

Rancher-lets-encrypt thinks it is a self-signed certificate and tries to create the cert everytime

I've been using this in rancher for many years now without issue so I don't check it too often 😄 but I recently added a new domain to let's encrypt and noticed an odd error in my logs that could have been going on for a while now.

"Error creating new order :: too many certificates already issued for exact set of domains". Looks like I've indeed hit my limit of 5 duplicate certificates for each domain, strange?

Taking a quick peek at the logs in /var/log/letsencrypt I noticed that the following block must be getting triggered based on the output which seems to create certs without checking their expiry.

      elif "X3" not in server_cert_issuer and not STAGING:
          # we have a self-signed certificate we should replace with a prod certificate.
          # this should only happen once on initial rancher install.
          print("INFO: Replacing self-signed certificate: {0}, "
                "{1} with production LE cert".format(server, server_cert_issuer))
          self.create_cert(server)
          self.post_cert(server)

Looking at the output

Replacing self-signed certificate: *******.com, CN=R3,O=Let's Encrypt,C=US

Looking at https://letsencrypt.org/certificates/ that seems to make sense as X3 is retired and the default is now R3.

I'm guessing hardcoding the issuer here is a bad idea anyways? R4 / E2 are backups and this is subject to change anyways.

[Testing] Automated integration testing?

Currently workflow for testing is:

Build image on internal CI system
Push image to private registry
Pull image on testing rancher instance
Manually poking and prodding
Once good, submit PR and merge to master
Docker hub build triggers
Image available publicly

This is mostly a placeholder issue while I come up with better testing system. This does not include actual code testing, but more for integration between rancher-lets-encrypt and the Rancher API.

Rancher 2.0 Beta compatiblity

Is rancher-lets-encrypt compatible to rancher 2.0 beta yet?

I'm not sure if I should test this out. Propably, I will kill my productive environment...

[Git Feature] Tagged Docker Hub image builds

It would be nice to have a v1.0 and v1.1 builds available rather than always having to use latest for Docker image builds.

[Feature request] Multiple domain names (SAN) per certificate

Hi,
I looked through the code, but it seems like it will create a single certificate for every domain specified in DOMAINS in the letsencrypt.env file. Since lets-encrypt supports to use multiple domains within a single certificate, it would be awesome, if this could be added to rancher-lets-encrypt.

The use-case would be to have a single certificate for example.com and www.example.com, or example.com and login.example.com.

A possible solution would be to split at ; per certificate and at , per domain:

DOMAINS=example.com,www.example.com;example-service.com,api.example-service.com

This would create two certificates:

example.com with www.example.com as subject alternative name
example-service.com with api.example-service.com as subject alternative name

When the repository supports rancher catalogs (I'll create a PR later today) The split could possibly be achieved by using a multiline question with one certificate per line and each line can list multiple certificates split by a ,

Edit: Here is the PR I mentioned above: #11

NameError: global name 'HOST_CHECK_LOOP_TIME' is not defined

Hi there,

When I try to start the Docker containers, I get the following error:

[root@srv rancher-lets-encrypt]# docker-compose up
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use the bundle feature of the Docker experimental build.

More info:
https://docs.docker.com/compose/bundles

Recreating rancherletsencrypt_rancher-lets-encrypt_1
Starting rancherletsencrypt_letsencrypt-nginx_1
Attaching to rancherletsencrypt_letsencrypt-nginx_1, rancherletsencrypt_rancher-lets-encrypt_1
rancher-lets-encrypt_1  | t find an Environment variable set.
rancher-lets-encrypt_1  | 'CATTLE_URL'
rancher-lets-encrypt_1  | Traceback (most recent call last):
rancher-lets-encrypt_1  |   File "/python/rancher.py", line 393, in <module>
rancher-lets-encrypt_1  |     service.check_hostnames_and_ports()
rancher-lets-encrypt_1  |   File "/python/rancher.py", line 362, in check_hostnames_and_ports
rancher-lets-encrypt_1  |     print "Sleeping during host lookups for {0} seconds".format(HOST_CHECK_LOOP_TIME)
rancher-lets-encrypt_1  | NameError: global name 'HOST_CHECK_LOOP_TIME' is not defined

My letsencrypt.env:

[root@srv rancher-lets-encrypt]# cat letsencrypt.env 
DOMAINS=srv.example.com
CERTBOT_WEBROOT=/var/www
[email protected]
RENEW_BEFORE_DAYS=14
LOOP_TIME=300
# Staging = True will get you a cert with the CA Fake, while Staging = False will use production Let's Encrypt to get the Let's Encrypt V3 CA.
STAGING=False
HOST_CHECK_PORT=80
HOST_CHECK_LOOP_TIME=10

As you can see, the HOST_CHECK_LOOP_TIME variable is set. Do you have any ideas why this error happens?

Thanks!

Ragards,
Philip

How to generate new certs and Native LB from Rancher ???

Found the info via closed ticket here: #5
and trail/error :)

I guess documentation will come soon for better understanding.

Thx

Feature request: Auto generated list of domains?

Is ist possible to generate the list of domains (via rancher API, from connected loadbalancer)?

[Bug] Service dies if Rancher server port dies

If the rancher server port 8080 falls over/stops, then the Rancher Lets encrypt service fails. Rather than trying to reconnect if we fail, we should wait a configured time and try to connect later.

3/16/2017 9:22:14 AMSleeping: 300 seconds...
3/16/2017 9:27:14 AMTraceback (most recent call last):
3/16/2017 9:27:14 AM  File "/python/rancher.py", line 409, in <module>
3/16/2017 9:27:14 AM    service.loop()
3/16/2017 9:27:14 AM  File "/python/rancher.py", line 152, in loop
3/16/2017 9:27:14 AM    self.cert_manager()
3/16/2017 9:27:14 AM  File "/python/rancher.py", line 166, in cert_manager
3/16/2017 9:27:14 AM    rancher_cert_servers = self.get_rancher_certificate_servers()
3/16/2017 9:27:14 AM  File "/python/rancher.py", line 353, in get_rancher_certificate_servers
3/16/2017 9:27:14 AM    returned_json = self.get_certificate()
3/16/2017 9:27:14 AM  File "/python/rancher.py", line 47, in get_certificate
3/16/2017 9:27:14 AM    r = requests.get(url=url, auth=self.auth())
3/16/2017 9:27:14 AM  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
3/16/2017 9:27:14 AM    return request('get', url, **kwargs)
3/16/2017 9:27:14 AM  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
3/16/2017 9:27:14 AM    return session.request(method=method, url=url, **kwargs)
3/16/2017 9:27:14 AM  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
3/16/2017 9:27:14 AM    resp = self.send(prep, **send_kwargs)
3/16/2017 9:27:14 AM  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
3/16/2017 9:27:14 AM    r = adapter.send(request, **kwargs)
3/16/2017 9:27:14 AM  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
3/16/2017 9:27:14 AM    raise ConnectionError(e)
3/16/2017 9:27:14 AMrequests.exceptions.ConnectionError: HTTPConnectionPool(host='10.1.4.20', port=8080): Max retries exceeded with url: /v1/certificate (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

Service stopped working because of certbot and acme versions

There were a few upgrades on certbot and acme modules, that need to be updated in order to have the service working again.

Updating certbot and requirements versions

I've updated the requirements and certbot version, since it was failing the certificates renew.

Please take note that I've also updated the rancher.py script to Python 3.7 and did the proper modifications to the Dockerfile.

As I don't have plans on moving my environment to kubernets so soon, I'm still using this. 😃