This project defines a set of utility functions for the Dehydrated ACMEv2 client, supporting direct integration with F5 BIG-IP, and including additional advanced features:

• Simple installation, configuration, and scheduling
• Supports renewal with existing private keys to enable certificate automation in HSM/FIPS environments
• Supports per-domain configurations, and multiple ACME providers
• Supports OCSP and periodic revocation testing
• Supports External Account Binding (EAB)
• Supports SAN certificate renewal
• Supports explicit proxy egress
• Supports granular scheduling
• Supports SMTP Reporting
• Supports high availability
• Supports debug logging

Why Kojot? Often pronounce "koyot", this is a word for "coyote" that has origins in Czech, Hungarian, Polish, and Serbo-Croatian languages. But most important, a famous coyote you may know, Wile E. Coyote, was a great consumer of ACME services, so the name checks out. ;)

${\LARGE{\textnormal{\textbf{\color{blue}Installation\ and\ Configuration}}}}$

Installation to the BIG-IP is simple. The only constraint is that the certificate objects installed on the BIG-IP must be named after the certificate subject name. For example, if the certificate subject name is www.foo.com, then the installed certificate and key must also be named www.foo.com. Certificate automation is predicated on this naming construct. To install the utility functions to the BIG-IP:

• ${\normalsize{\textbf{\color{red}Step\ 1}}}$ (Install): SSH to the BIG-IP shell and run the following command. This will install all required components to the /shared/acme folder on the BIG-IP. In an HA environment, perform this action on both BIG-IP instances.

  curl -s https://raw.githubusercontent.com/f5devcentral/kojot-acme/main/install.sh | bash
  

• ${\normalsize{\textbf{\color{red}Step\ 2}}}$ (Global Configuration): Update the new dg_acme_config data group and add entries for each managed domain (certificate subject). You must minimally include the subject/domain (key) and a corresponding --ca value. In an HA environment, this data group is synced between the peers. See the Global Configuration Options section below for additional details. Examples:

  www.foo.com := --ca https://acme-v02.api.letsencrypt.org/directory
  www.bar.com := --ca https://acme.zerossl.com/v2/DV90 --config /shared/acme/config_www_example_com
  www.baz.com := --ca https://acme.locallab.com:9000/directory -a rsa
  

• ${\normalsize{\textbf{\color{red}Step\ 3}}}$ (Client Configuration): Adjust the client configuration config file in the /shared/acme folder as needed for your environment. In most cases you will only need a single client config file, but this utility allows for per-domain client configurations. For example, you can define separate config files when EAB is needed for some provider(s), but not others. In an HA environment, the utility ensures these config files are available to the peer. See the ACME Client Configuration Options section below for additional details.

• ${\normalsize{\textbf{\color{red}Step\ 4}}}$ (HTTP Virtual Servers): Minimally ensure that an HTTP virtual server exists on the BIG-IP that matches the DNS resolution of each target domain (certificate subject). As a function of the ACMEv2 http-01 challenge process, the ACME server will attempt to contact the requested domain IP address on port 80 (HTTP). Attach the acme_handler_rule iRule to each HTTP virtual server.

• ${\normalsize{\textbf{\color{red}Step\ 5}}}$ (Initial Fetch): Initiate an ACMEv2 fetch. This command will loop through the dg_acme_config global configuration data group and perform required ACMEv2 certificate renewal operations for each configured domain. By default, if no certificate and key exists for a domain, ACMEv2 renewal will generate a new certificate and key. If a private key exists, a CSR is generated from the existing key to renew the certificate only. This it to support HSM/FIPS environments, but can be disabled to always generate a new private key. See the Utility Command Line Options and ACME Client Configuration Options sections below for additional details.

  cd /shared/acme
  ./f5acmehandler.sh --verbose
  

• ${\normalsize{\textbf{\color{red}Step\ 6}}}$ (Schedule): Once all configuration updates have been made and the utility function is working as desired, define scheduling to automate the process. By default, each domain (certificate) is checked against the defined threshold (default: 30 days) and only continues if the threshold is exceeded. In an HA environment, perform this action on both BIG-IP instances. See the Scheduling section below for additional details. For example, to set a weekly schedule, to initiate an update check every Monday at 4am:

  cd /shared/acme
  ./f5acmehandler.sh --schedule "00 04 * * 1"
  

• ${\normalsize{\textbf{\color{red}Step\ 7}}}$ (Client SSL Profile): The f5acmehandler.sh utility maintains the freshness of the certificates (and private keys) installed on the BIG-IP. Ultimately, these certificates and keys will then need to be applied to SSL profiles, and the SSL profiles applied to application virtual servers. Creating the SSL profiles and virtual servers is outside the scope of this utility, but optionally you can set the CREATEPROFILE option in the client config file to 'true' to have the utility create a client SSL profile if missing, and attach the certificate and key to that profile.

${\LARGE{\textnormal{\textbf{\color{blue}Configuration\ Details}}}}$

Configuration options for this utility are found in the following locations:

www.foo.com := --ca https://acme-v02.api.letsencrypt.org/directory
www.bar.com := --ca https://acme.zerossl.com/v2/DV90 --config /shared/acme/config_www_example_com
www.baz.com := --ca https://acme.locallab.com:9000/directory -a rsa

1. minutes (0-59)
2. hour (0-23)
3. day of the month (1-31)
4. month (1-12)
5. day of the week (0-6)

./f5acmehandler.sh --schedule "00 04 * * 1"

${\LARGE{\textnormal{\textbf{\color{blue}ACME\ Protocol\ Flow}}}}$

Provided below are detailed descriptions of the control flows. The ACME Utility Architecture section describes the files and folders in use. The ACME Functional Flow on BIG-IP section describes the interaction of f5acmehandler and ACME client processes. The ACME Protocol Flow Reference details the general ACMEv2 protocol flow per RFC8555.

${\LARGE{\textnormal{\textbf{\color{blue}Additional\ Configuration\ Options}}}}$

Below are descriptions of additional features and environment options.

# Extended Account Binding (EAB) support
EAB_KID=kid-1
EAB_HMAC_KEY=zWNDZM6eQGHWpSRTPal5eIUYFTu7EajVIoguysqZ9wG44nMEtx3MUAsUDkMTQ12W

cd /shared/acme
./f5acmehandler.sh --testrevocation foo.f5labs.com

./f5acmehandler.sh --save

echo -e "From: <sender email address>\nSubject: Test Email\n\nTest Email" | ssmtp -v -C /shared/acme/config_reporting <receiver email address>

curl -s https://raw.githubusercontent.com/f5devcentral/kojot-acme/main/uninstall.sh | bash

${\LARGE{\textnormal{\textbf{\color{blue}Troubleshooting}}}}$

${\LARGE{\textnormal{\textbf{\color{blue}Testing}}}}$

There are a number of ways to test the f5acmehandler utility, including validation against local ACME services. The acme-servers folder contains Docker-Compose options for spinning up local Smallstep Step-CA and Pebble ACME servers. The following describes a very simple testing scenario using one of these tools.

• On the BIG-IP, install the f5acmehandler utility components on the BIG-IP instance. SSH to the BIG-IP shell and run the following command:

  curl -s https://raw.githubusercontent.com/f5devcentral/kojot-acme/main/install.sh | bash

• Install the Smallstep Step-CA ACME server instance on a local Linux machine. Adjust the local /etc/hosts DNS entries at the bottom of the docker-compose YAML file accordingly to allow the ACME server to locally resolve your ACME client instance (the set of BIG-IP HTTP virtual servers). This command will create an ACME service listening on HTTPS port 9000.

  git clone https://github.com/f5devcentral/kojot-acme.git
  cd kojot-acme/acme-servers/
  docker-compose -f docker-compose-smallstep-ca.yaml up -d

• On the BIG-IP, for each of the above /etc/hosts entries, ensure that a matching HTTP virtual server exists on the BIG-IP. Define the destination IP (same as /etc/hosts entry), port 80, a generic http profile, the proper listening VLAN, and attach the acme_handler_rule iRule.

• On the BIG-IP, update the dg_acme_config data group and add an entry for each domain (certificate). This should match each /etc/hosts domain entry specified in the docker-compose file.

  www.foo.com := --ca https://<acme-server-ip>:9000/acme/acme/directory
  www.bar.com := --ca https://<acme-server-ip>:9000/acme/acme/directory -a rsa

• To view DEBUG logs for the f5acmehandler processing, ensure that the DEBUGLOG entry in the config file is set to true. Then in a separate SSH window to the BIG-IP, tail the acmehandler log file:

  tail -f /var/log/acmehandler

  or use the --verbose option with the f5acmehandler.sh script:

  ./f5acmehandler.sh --verbose

• Trigger an initial ACMEv2 certificate fetch. This will loop through the dg_acme_config data group and process ACME certificate renewal for each domain. In this case, it will create both the certificate and private key and install these to the BIG-IP. You can then use these in client SSL profiles that get attached to HTTPS virtual servers. In the BIG-IP, under System - Certificate Management - Traffic Certificate Management - SSL Certificate List, observe the installed certificate(s) and key(s).

• Trigger a subsequent ACME certificate fetch, specifying a single domain and forcing renewal. Before launching the following command, open the properties of one of the certificates in the BIG-IP UI. After the command completes, refresh the certificate properties and observe the updated Serial Number and Fingerprint values.

  ./f5acmehandler.sh --domain www.foo.com --force

${\LARGE{\textnormal{\textbf{\color{blue}Credits}}}}$

Special thanks to:

• @f5-rahm and his lets-encrypt-python project for inspiration, and for coming up with the cool project name. ;)
• @Lukas2511 for the dehydrated ACME client utility