ETA for completion of this epic: 2018-09-21
Confidence level of ETA accuracy: High

Context

HashiCorp's Terraform is a widely-used tool used to manage infrastructure as code. Terraform manifests are written in HCL.

Terraform is extendable with providers, which can declare their own resources. A Conjur provider does not yet exist. Secrets are often needed in Terraform manifests, so being able to fetch them from Conjur improves the security stance of Terraform.

Current Capabilities

It is currently possible to fetch secrets from Conjur within Terraform by injecting them via environment variables. This requires the use of summon and specially named environment variables. While this is possible, the goal of creating a Conjur provider is to provide a better experience.

In order to use summon + environment variables, the user running terraform must already be authenticated with Conjur.

Terraform's TF_VAR_name syntax allows a user to set Terraform variables via environment variables. To use Terraform with Summon, prefix the environment variable names in secrets.yml with TF_VAR_.

secrets.yml

TF_VAR_access_key: !var aws/dev/sys_powerful/access_key_id
TF_VAR_secret_key: !var aws/dev/sys_powerful/secret_access_key

variables.tf

variable "access_key" {} /* Variable pulled from TF_VAR_access_key environment variable */
variable "secret_key" {} /* Variable pulled from TF_VAR_secret_key environment variable */

Run Terraform with Summon:

summon terraform apply

Objective

Provide a Terraform conjur provider with a conjur_secret resource that can be used to fetch secrets from a configured Conjur v5 instance.

We should also instruct customers on how to use rotation (and possible encryption) to handle exposed secrets in plan files.

Original Epic Goal & Resources

Initial support

When using Terraform, credentials will be able to be pulled out of Conjur.

This should mirror what HC currently implements as part of their Vault provider. Our provider will leverage rotation to remove credentials exposed.

• hashicorp/terraform#516
• https://www.terraform.io/docs/extend/writing-custom-providers.html
• https://github.com/terraform-providers/terraform-provider-vault

Terraform Provider Configuration

Prerequisite is a setup and configured Conjur instance. In order to configure the Terraform provider, configuration values can be included in the Terraform manifest, or can be loaded using the standard Conjur client configuration mechanisms (e.g. ~/.conjurrc, environment variables, etc). Values specified in the configuration file take precedence over other values.

Policy For Examples

Terraform Manifest Provider Configuration

provider "conjur" {
  appliance_url = "http://localhost:8080" /* The Conjur appliance URL */
  account = "quick-start" /* The Conjur account */
  login = "admin" /* The Conjur authn login */
  api_key = "3ahcddy39rcxzh3ggac4cwk3j2r8pqwdg33059y835ys2rh2kzs2a" /* The Conjur authn API key */
  ssl_cert_path = "/etc/conjur.pem" /* The Conjur certificate file */
}

Environment Variable Provider Configuration

export CONJUR_APPLIANCE_URL="https://localhost:8080"
export CONJUR_ACCOUNT="quick-start"
export CONJUR_AUTHN_LOGIN="admin"
export CONJUR_AUTHN_API_KEY="3ahcddy39rcxzh3ggac4cwk3j2r8pqwdg33059y835ys2rh2kzs2a"
export CONJUR_CERT_FILE="/etc/conjur.pem"

And then within the Terraform manifest:

provider "conjur" {}

Fetching Secrets Within Terraform

Secrets are dynamically fetched from the Conjur service and injected as variables into the Terraform manifest when used. They need to be defined within the Terraform manifest prior to use.

Declaring Conjur Sourced Variables

The variables need to be declared prior to use. They are declared like any other data variable with Terraform, using the path within Conjur for the name.

data "conjur_secret" "dbpass" {
  name = "path/to/secret" /* The Conjur secret to fetch */
}

Using Variables

The data for the variable is extracted into data.<declared_path>.<declared_name>.value. This can be then tied to a Terraform output in order to be used. The 'sensitive' declaration of true will prevent the value from being printed to the console, but will not prevent it from showing up in .plan files or logs.

output "dbpass_output" {
  value = "${data.conjur_secret.dbpass.value}" /* Extract the value from the declared variable */
  sensitive = true  /* Whether to log the value to the console when running */
}

Full Terraform Manifest example

main.tf

provider "conjur" {
  appliance_url = "http://localhost:8080" /* The Conjur appliance URL */
  account = "quick-start" /* The Conjur account */
  login = "admin" /* The Conjur authn login */
  api_key = "3ahcddy39rcxzh3ggac4cwk3j2r8pqwdg33059y835ys2rh2kzs2a" /* The Conjur authn API key */
  ssl_cert_path = "/etc/conjur.pem" /* The Conjur certificate file */
}

data "conjur_secret" "dbpass" {
  name = "path/to/secret" /* The Conjur secret to fetch */
}

output "dbpass_output" {
  value = "${data.conjur_secret.dbpass.value}" /* Extract the value from the declared variable */
  sensitive = true  /* Whether to log the value to the console when running */
}

Additional Notes

• If any of the provided credentials change and are needed in any resources, terraform apply needs to be run so that the secrets are available.

• Test Conjur v5 Enterprise w/ Jenkins pipeline - parent of #2
• Add homebrew release - parent of #3
• Document summon + terraform flow - parent of #4
• Use a better, more interesting test policy - parent of #5
• Is there a way to avoid the secrets ending up in state files? - parent of #6 (closed)
• Document how to install this provider - parent of #11
• The provider uses the API to load configuration - parent of #12
• A demo shows that the provider can be part of creating a Rails app - parent of #13
• Provider should be documented on conjur.org - parent of #19
• README is missing Authentication and Policy sections - parent of #20

Copied from original issue: conjurinc/appliance#262

AC:
The following templates have been added to the repo, with appropriate component labels:

• Bug template
• Feature request template

Summary

Provider is not up to date and does not support CONJUR_AUTHN_TOKEN that is included in conjur-api-go

Currently, this repo only has integration tests. Some level of unit testing would improve cycle times and ensure that we cover even more code with testing that we don't do with our integration tests.

AC:

• Provider code logic paths are unit tested

output "dbpass_output" {
  value = "${data.conjur_secret.dbpass.value}"
  sensitive = true  # toggle this off to view value
}

When used in a remote state environment this will write your secret out to a state file in plaintext. In the documentation there should at least be a disclaimer around this warning people not to do this.

The provider doesn't work with Terraform v0.12.0
Error message after "terraform apply":

Error: Failed to instantiate provider "conjur" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

It's just a single variable right now.

Instead, we should have a policy with a layer of web applications authorized to fetch a certain secret. Both the demo and tests should use this. Reuse an existing demo app if possible.

This is related to #13 and probably will be completed before it. If so we'll close that issue.

Summary

Manifest example in provider's documentation is wrong.

Expected Results

The provider name should be "cyberark/conjur".

Actual Results (including error logs, if applicable)

The provider name is "conjur" and terraform init fails with the below:

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...

Provider "conjur" not available for installation.

A provider named "conjur" could not be found in the Terraform Registry.

This may result from mistyping the provider name, or the given provider may
be a third-party provider that cannot be installed automatically.

In the latter case, the plugin must be installed manually by locating and
downloading a suitable distribution package and placing the plugin's executable
file in the following directory:
    terraform.d/plugins/linux_amd64

Terraform detects necessary plugins by inspecting the configuration and state.
To view the provider versions requested by each module, run
"terraform providers".


Error: no provider exists with the given name


ERROR: 1

Reproducible

• Always
• Sometimes
• Non-Reproducible

A new release of Terraform came out, but our current plugin doesn’t work with it.

AC:

• Recompile our provider, and verify that it works with the latest release of Terraform
• If the previous step is complete, add a commit with an updated CHANGELOG and version number for our provider to facilitate a new release

Any additional easy work to improve this project, including improved test coverage or improved documentation, would also be appreciated.

If the repo has a changelog that doesn't meet the standard, do try to change earlier entries to match the standard.
If the repo doesn't have a changelog use this as a starter:

# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [Unreleased]

Acceptance criteria

• CHANGELOG.md exists and follows the KeepAChangeLog format. For reference consult https://github.com/cyberark/conjur/blob/master/CHANGELOG.md.
• Add parse-changelog validation step to Jenkinsfile. For reference consult https://github.com/cyberark/conjur/blob/master/Jenkinsfile#L27.

We should doc in README how to install this provider. Can you point terraform to a github tarball release?

Once #3 is done, we can show homebrew install instructions too.

This will include Apple Silicon support.

Currently this repo uses the cyberark/goreleaser Docker image, but since the official goreleaser/goreleaser image was recently updated, it should work for our use cases. We should update to use the official goreleaser image in lieu of our custom one so that our custom image can be deprecated.

Is your feature request related to a problem? Please describe.

I would like to authenticate to the provider with access token along with authenticating with API because I want to run terraform in K8S cluster where the conjur k8s authenticator provides me an access token.

Describe the solution you would like

Add variable to the provider of access_token and make it alternative to the api_key

Currently the README is severely out of date and needs a bit of cleanup

AC:

• There is no outdated information in README

Summary

When using the provider with Conjur Cloud, requires /api to be included in the appliance url. Since this is supported between self-hosted and Conjur Cloud, hardcode of /api should be included.

Steps to Reproduce

1. Get Conjur Cloud tenant.
2. Setup provider without /api on end of appliance url.
3. Witness failure.

Expected Results

/api should be hardcoded into the request URL to not require it to be provided.

Actual Results

Results in error with /api is not provided as part of appliance url.

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

• Affects every version

Environment setup

• It doesn't matter. Affects every environment.

Additional Information

n/a

The test automation is failing in GH actions and Jenkins with the release of v0.15

The broken terraform provider build here needs to be fixed.

Issue description

Steps to reproduce the issue

Assumptions

• Conjur master is setup
• Valid credentials are available to the Terraform script as variables

1. Given the following TF template:

   # main.tf
   provider "conjur" {
     account = "${var.conjur_account}"
     appliance_url = "${var.conjur_appliance_url}"
     api_key = "${var.conjur_api_key}"
     login = "${var.conjur_login}"
     ssl_cert = "${var.conjur_ssl_cert}"
     ssl_cert_path = "${var.conjur_ssl_path}"
   }
   

2. Run terraform apply

What's the expected result?

• The values for account, appliance_url, ssl_cert, and ssl_cert_path are used to connect to Conjur

What's the actual result?

• Only the values for api_key and login are used to connect. The rest of the values must be set using environment variables.

Dev Notes

In tracking this bug down, I noticed the environment variables mentioned in the readme (ex. export CONJUR_APPLIANCE_URL="https://localhost:8443") are the environment variables the Conjur GoLang library uses. It looks like the convention for setting configuration with environment variables is more like:

Schema: map[string]*schema.Schema{
  "appliance_url": &schema.Schema{
    Type: schema.TypeString,
    Required: true,
    DefaultFunc: schema.EnvDefaultFunc("CONJUR_APPLIANCE_URL", nil),
    Description: "URL of the Conjur Appliance",
  },
  "account": &schema.Schema{
    Type: schema.TypeString,
    Required: true,
    DefaultFunc: schema.EnvDefaultFunc("CONJUR_ACCOUNT", nil),
    Description: "Conjur Account",
  },
  ...
}

which creates a more direct mapping between the environment variables and the provider attributes. Does it make sense to update this technical debt as part of this effort?

Summary

When using goreleaser, brew recipe created does not create proper URLs:

• linux URL is for darwin
• darwin URL doesn't contain the version number in link

Currently there are no good instructions for installing and running this plugin on Windows.

AC:

• README contains instructions on how to install and run this plugin on Windows

From @ismarc on August 2, 2018 15:18

Context

HashiCorp's Terraform is a widely-used tool used to manage infrastructure as code. Terraform manifests are written in HCL.

Terraform is extendable with providers, which can declare their own resources. A Conjur provider does not yet exist. Secrets are often needed in Terraform manifests, so being able to fetch them from Conjur improves the security stance of Terraform.

Objective

Provide a Terraform conjur provider with a conjur_secret resource that can be used to fetch secrets from a configured Conjur v5 instance.

We should also instruct customers on how to use rotation (and possible encryption) to handle exposed secrets in plan files.

Original Epic Goal & Resources

Initial support: pull credentials out of Conjur when using Terraform.

Initial will would mirror what HC currently implements as part of their Vault provider. Our provider will leverage rotation to remove credentials exposed.

• hashicorp/terraform#516
• https://www.terraform.io/docs/extend/writing-custom-providers.html
• https://github.com/terraform-providers/terraform-provider-vault

Copied from original issue: conjurinc/appliance#262

Is your feature request related to a problem? Please describe.

Ability to update the Conjur secret value

Describe the solution you would like

Using the AddSecret function form conjur-api-go

Describe alternatives you have considered

No alternatives

Additional context

Already created PR with test case for this feature
#133

Since HashiCorp doesn't provide an easy way to install community Terraform providers, we should provide a Homebrew formula, similar to https://github.com/beamly/terraform-provider-gocd#installation.

The project is already integrated with goreleaser, we just need to set up the homebrew parts. Since this is a binary that needs to be installed in a non-standard place, use summon-conjur as a guide:
https://github.com/cyberark/summon-conjur/blob/master/.goreleaser.yml#L30.

Users should be able to run these commands to install the provider:

brew tap cyberark/tools
brew install terraform-provider-conjur

Note that we need to release a 0.1.0 initial version to get this working. This task is a followup from #15 anyways.

Is your feature request related to a problem? Please describe.

Terraform providers are now easily downlodable through official Terraform registry

It could allow simple installation as follow :

terraform {
  required_providers {
    conjur = {
      source  = "cyberark/conjur"
      version = "~> 0.4"
    }
  }
}

Describe the solution you would like

Publicsh the provider following the Terraform documentation about publishing provider to the Terraform registry.

Describe alternatives you have considered

Manually download the provider and put it in the right directory before execution. But it's a pain to maintain and required additional dependencies like curl/wget/tar

Additional context

N/A

Review this repo:

• Determine whether improvements can be made to update/modernize the release process
• Review README and CONTRIBUTING to determine if either documentation set is missing important information
• Determine if automated test suite should be augmented to alert maintainer team to possible breakages earlier
• Determine what sorts of standard maintenance tasks this repo requires. Does it need regular review to update the Terraform version used in the integration tests? Are there other dependencies (eg Golang?) that should be updated at regular intervals?
• Determine whether any additional updates should be slated for work soon, including potential feature requests logged in GitHub issues (for small changes, we can work into backlog - for larger changes that take greater than one standard 2-3 day PR cycle, we will raise with PM)

The output of this spike will be a list of items for each category above listed in a comment on this issue, and linking to the GitHub issues for each item.

Summary

Installing conjur provider on apple silicon fails

Steps to Reproduce

required_providers {
    conjur = {
      source = "cyberark/conjur"
      version = "0.6.3"
    }
}

terraform init

Expected Results

Successful init

Actual Results

│ Error: Failed to install provider
│
│ Error while installing cyberark/conjur v0.6.3: provider binary not found: could not find executable file starting with terraform-provider-conjur

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

latest

Environment setup

darwin arm64

Additional Information

None

I used this code to fetch the secret from Conjur.
But I can see two entries of audit log in the Conjur UI that the user is actually fetching the secret twice
->First fetch is done during the Terraform plan
->Second fetch is done during the Terraform apply

How to prevent the fetch done during the Terraform plan?

@Andrew.Copeland has made some changes to our terraform provider in order to support the new version of terraform that came out.

The objective of this card is :

1. To ensure all changes are in
2. To release the binaries of the provider

Also see
#26

We can automate a lot of tasks that are currently manual with GitHub actions and move away from gitlab/Jenkins.

AC:

• Testing is done on all opened PRs
• Tag creation creates a draft release with attached artifacts of build
• Any other tasks that make sense automated w/ GH actions should be moved there too

We test against Conjur 5 OSS, we should test against Conjur 5 Enterprise too. These tests can run in Jenkins.

To show how the Conjur Terraform provider can be used, we'll build a demo that brings up Docker containers running a simple Rails app. The app will use Postgres for its persistence. The database password will be stored in Conjur.

At current the build is failing. It needs to be looked into and resolved.

Context: hashicorp/terraform#516

Is your feature request related to a problem? Please describe.

I would like to be able to create secrets with Terraform.

Describe the solution you would like

Create a new resource to create secrets with a policy via Terraform.

Describe alternatives you have considered

Currently we are hacking a similar solution based on a shell script used within a null_resource local-exec resource which feels pretty wrong.

Additional context

None

Currently there is no easy way for us to notice if there is a change in upstream dependencies that breaks this plugin. We need to research and apply a way to do this.

AC:

• We have automation that can notify us when upstream dependencies break this plugin

Is your feature request related to a problem? Please describe.

In #77 this plugin was updated to use the v1 plugin SDK, but v2 has been released for some time.

Describe the solution you would like

In this issue, we should update to the v2 plugin SDK. There is a migration guide here.

During this process, we should investigate our test infrastructure for this project to identify any potential improvements.

Summary

When I tried to install the provider by using the instructions, I wasn't able to get that up and running. I had to change the folder structure and had to tell terraform where to find the provider:

It has to be at a different place now, e.g.
.terraform.d/plugins/cyberark.com/edu/conjur/0.4.0/linux_amd64
and has to be named as:
terraform-provider-conjur

and requires an entry in the main.tf
terraform {
required_providers {
conjur = {
version = "0.4.0"
source = "cyberark.com/edu/conjur"
}
}
}

Rather than duplicate support for loading configuration from various sources (including the CONJUR_* environment variables), the provider should just have conjur-api-go load the configuration.

After the configuration is loaded, any values specified in the manifest can be applied to the configuration.

We should at least mention the workflow we recommend, including how Conjur authentication and policy work. Link out to conjur.org liberally.

The release should include the changes from #76

Issue description

This provider does not include any version information. This makes validating which version you have very problematic.

Steps to reproduce the issue

1. View Terraform version:

   $ terraform version
   

What's the expected result?

Output would look something like:

Terraform v0.12.1
+ provider.conjur v0.2

What's the actual result?

Terraform v0.12.1
+ provider.conjur (unversioned)

Summary

Unable to use Conjur provider for Terraform.

Steps to Reproduce

Steps to reproduce the behavior:

1. Using this simple main.tf

provider "aws" {
  version = "~> 2.0"
  region = "eu-west-3"
}


provider "cyberark/conjur" {
  version = "0.6.2"
}

2. And this docker image: hashicorp/terraform:latest (2021/09/08)

3. Terraform init fails with the below error:

Creating conjur_playground_terraform_run ... done

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...

Error verifying GPG signature for provider "cyberark/conjur"
Terraform was unable to verify the GPG signature of the downloaded provider
files using the keys downloaded from the Terraform Registry. This may mean that
the publisher of the provider removed the key it was signed with, or that the
distributed files were changed after this version was released.


Error: unable to verify signature


ERROR: 1

Expected Results

Terraform init succeeds

Actual Results (including error logs, if applicable)

Terraform init fails with GPG error (see above)

Reproducible

• [X ] Always
• Sometimes
• Non-Reproducible

Version/Tag number

latest (0.6.2)

The documentation (Readme.md), does not mention that users will need to run terraform init to initialize their Terraform project.

Expected Behavior
All instructions to initialize and execute a Terraform run are present in the documentation, so readers do not become confused.

We should probably just explain it briefly and link to project README.

We use:

• Cuke-master image for enterprise tests
• PG 9.3 image for OSS builds
• Goreleaser is run for some reason in compose

We should clean this up as much as we can.

AC:

• Enterprise tests should use the appliance image
• OSS tests should use helm-like or container DAP images/setup
• Any other items that can be improved in these files should be done here too

Is your feature request related to a problem? Please describe.

As reported in Homebrew tools
Apple silicon is not supported.

Describe the solution you would like

GoReleaser should build an Apple M1

Describe alternatives you have considered

None

Additional context

Add any other context information about the feature request here.

This is just a reduction of tech debt that has accumulated over time

AC:

• go.mod contains only the newest dependency versions
• go.sum is cleaned up