devops-rob / terraform-provider-terracurl Goto Github PK

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

1.3.4

Affected Resource(s)

• all

Terraform Configuration Files

resource "terracurl_request" "vault_init" {
  name = "vault init"

  lifecycle {
    ignore_changes = [
      headers,
    ]
  }

  method         = "POST"
  url            = "${google_cloud_run_service.vault.status.0.url}/v1/sys/init"
  request_body   = file("${path.module}/init.json")
  response_codes = [200]

  # Vault might no be immediately able to serve the api request, retry this 
  max_retry      = 5
  retry_interval = 30

  headers = {
    "Authorization" = "Bearer ${data.google_service_account_id_token.oidc.id_token}"
  }
}

Expected Behavior

When a request fails for the first time, the terracurl_request should retry the operation by passing the same HTTP request body.

Actual Behavior

The request is retried; however, since the http.Request is not recreated, the Request.Body was closed and therefore not sent with the retry. This resulted in no body being sent with the request and a bad request error from the server.

Hi @devops-rob,

Are you scheduling a release of the provider to include #47 ?

Thanks Rob.

When executing against as system with a self-signed certificate it would be great to be able to pass an insecure flag. Today if the self-signed certificate is not known the plan fails and you get the following: x509: certificate signed by unknown authority

Terraform Version

Terraform v1.3.0
on windows_amd64

• provider registry.terraform.io/devops-rob/terracurl v0.1.1

Affected Resource(s)

Please list the resources as a list, for example:

• terracurl_request

First, great project. This will solve so many problem when using APIs not natively supported by provider published resource. Super nice! Thank you.

To the point; is it possible to rebase this project on the currently supported https://github.com/hashicorp/terraform-provider-scaffolding-framework ? https://github.com/hashicorp/terraform-provider-scaffolding has been archived by Hashicorp and there is concern from my decision makers concerning the upstream project

Terraform Version

n/a

Affected Resource(s)

Project context.

Terraform Configuration Files

n/a

Debug Output

n/a

Panic Output

n/a

Expected Behavior

n/a

Actual Behavior

n/a

Steps to Reproduce

n/a

Important Factoids

n/a

References

n/a

Describe the issue
Setting a dynamic token in X-Vault-Token leads to authentication failure when running Terraform apply.

To Reproduce

1. Use the vault token create command to create a token with a validity of 1 hour.
2. Set this token in X-Vault-Token of TerraCurl.
3. Run Terraform apply and resources are created successfully.
4. After 1 hour, the previous token expires. Create a new vault token and set it in X-Vault-Token.
5. Run Terraform plan, the result shows that the value of X-Vault-Token will be updated and the resource(aws engine) will be re-created.

  # terracurl_request.aws_engine must be replaced
-/+ resource "terracurl_request" "aws_engine" {
      ~ destroy_headers        = {
          ~ "X-Vault-Token" = "hvs.XXXXXXXXXX" -> "hvs.YYYYYYYYYY"
        }
      ~ headers                = { # forces replacement
          ~ "X-Vault-Token" = "hvs.XXXXXXXXXX" -> "hvs.YYYYYYYYYY"
        }
      ~ id                     = "aws" -> (known after apply)
        name                   = "aws"
      ~ request_url_string     = "http://172.XXX.XXX.XXX:8201/v1/sys/mounts/aws" -> (known after apply)
      + response               = (known after apply)
      ~ status_code            = "204" -> (known after apply)
        # (9 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

6. When running Terraform apply, a request error occurred.

terracurl_request.aws_engine: Destroying... [id=aws]
terracurl_request.aws_engine: Still destroying... [id=aws, 10s elapsed]
?
x Error: unable to make request: request failed, retries exceeded: %!s(<nil>)

7. After manually updating the token value of X-Vault-Token in the terraform-state file, Terraform executed successfully.

Expected behavior

1. The new token set in X-Vault-Token should authenticate with Vault successfully.
2. The resource should be updated without being deleted and recreated.

Environment

• Vault Server 1.12.4
• TerraForm 1.4.6
• TerraCurl 1.1.0

Additional context
After the first run of terraCurl, the X-Vault-Token value is recorded in Terraform's state file and will be used for authentication with Vault the next time.
When the token on the Vault side expires, the X-Vault-Token value fails to authenticate with Vault and cannot be updated any more.

Blocking the release of #57

Tests appear to be working locally, as seen in #57 (comment)

But failing in GHA, as seen in https://github.com/devops-rob/terraform-provider-terracurl/actions/runs/6149483782/job/16685497278

Terraform Version

TF: 1.3.9

- Installed devops-rob/terracurl v1.0.2 (self-signed, key ID C04C4BF84C9276F1)

Affected Resource(s)

I've seen the issue on the resource, but I expect the datasource would have the same problem

Terraform Configuration Files

locals {
  webserver_url = "http://${aws_instance.web.public_ip}"
}

resource "terracurl_request" "test" {
  name   = "smoke test webserver"
  url    = local.webserver_url
  method = "GET"

  response_codes = [
    200
  ]

  // Retry for up to 60s
  max_retry      = 
  retry_interval = 15

  lifecycle {
    // Any change to the EC2 instance, do another TerraCurl check
    replace_triggered_by = [aws_instance.web]

    // And check that we have an HTTP 200
    postcondition {
      condition     = self.status_code == "200"
      error_message = "Webserver did not respond with an HTTP 200"
    }
  }
}

To reproduce the issue, I updated the security group on my EC2 instance to remove inbound port 80.

Expected Behavior

Request should timeout, and TF should retry

Actual Behavior

Request does not timeout

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

Looks like it's a relatively simple addition:
https://github.com/devops-rob/terraform-provider-terracurl/blob/main/internal/provider/resource_curl.go#LL328C54-L328C58

context.TODO() replace with context.WithTimeout()
https://pkg.go.dev/context#WithTimeout

• relevant parameters to the resource/datasource to configure that with sensible defaults

I'm making requests to URLs with Vault-generated certs. As such, I need to trust Vault's CA cert.
Doesn't look like the provider supports that yet.

Hello,

Does Terracurl support Digest Authentication? Thanks.

It would be great to have pre-post condition that can run a command or a resource
for those isolated vault environments.. it'll save me from having to use a null resource local-provisioner for kubectl port-forward.

Thanks!

the API i am using can creates (POST) an object, the result of the creation is a UUID returned in a JSON response. The suppression of the same objet (DELETE) is done using the object UUID referenced in the delete_url request.

for example

resource "terracurl_request" "myobject" {
    ...
    url = "https://10.1.11.111/api/v1/objects"
    ...
}
output "id" {
    value = jsondecode(terracurl_request.myobject.response.id)
}

the "id" is the UUID of the object which is used as the reference for the object suppression.

the destroy_url is then

destroy_url = "https://10.1.11.111/api/v1/objects/<UUIDv4_value>"
destroy_method = "DELETE"

could you help me understand is there are any way moving forward?

Hi there,

Would you be open to including support for request_parameters ? I've made a change here that allows optional request parameters for both create & destroy.

My use case is so that I can interact with the AWS APIs without needing any external environment setup. I've created a lambda function that I can invoke via terraform which returns a generated SIGV4 signature. I can then use this within the terracurl request resource. However I need to supply the request params to the API endpoint i.e. ?Action=CreateCluster

So if the AWS terraform provider doesn't support a feature or service, you can still use native terraform to interact directly with the AWS APIs.

Affected Resource(s)

Please list the resources as a list, for example:

• terracurl_request

Terraform Configuration Files

data "aws_lambda_invocation" "create" {
  function_name = "sigv4-with-layer"
  input         = <<JSON
{
  "service": "ecs",
  "region": "eu-west-1",
  "method": "POST",
  "url": "https://ecs.eu-west-1.amazonaws.com",
  "headers": {
        "Accept-Encoding": "identity",
        "Content-Type": "application/x-amz-json-1.1",
        "X-Amz-Target": "AmazonEC2ContainerServiceV20141113.CreateCluster"
  },
  "data": {
    "clusterName": "MyCluster"
  },
  "params": {
    "Action": "CreateCluster"
  }
}
JSON
}

data "aws_lambda_invocation" "destroy" {
  function_name = "sigv4-with-layer"
  input         = <<JSON
{
  "service": "ecs",
  "region": "eu-west-1",
  "method": "POST",
  "url": "https://ecs.eu-west-1.amazonaws.com",
  "headers": {
        "Accept-Encoding": "identity",
        "Content-Type": "application/x-amz-json-1.1",
        "X-Amz-Target": "AmazonEC2ContainerServiceV20141113.DeleteCluster"
  },
  "data": {
    "cluster": "MyCluster"
  },
  "params": {
    "Action": "DeleteCluster"
  }
}
JSON
}

resource "terracurl_request" "name" {
  name   = "AWS"
  url    = "https://ecs.eu-west-1.amazonaws.com"
  method = "POST"

  headers        = jsondecode(data.aws_lambda_invocation.create.result)["headers"]
  response_codes = [200, 400, 403]

  request_body       = jsondecode(data.aws_lambda_invocation.create.result)["data"]
  request_parameters = jsondecode(data.aws_lambda_invocation.create.result)["request_params"]

  destroy_url          = "https://ecs.eu-west-1.amazonaws.com"
  destroy_method       = "POST"
  destroy_headers      = jsondecode(data.aws_lambda_invocation.destroy.result)["headers"]
  destroy_request_body = jsondecode(data.aws_lambda_invocation.destroy.result)["data"]
  destroy_parameters   = jsondecode(data.aws_lambda_invocation.destroy.result)["request_params"]

  destroy_response_codes = [200, 400, 403]
}

Expected Behaviour

Provide request parameters to allow direct calls to AWS APIs

Sorry, more of a usage question/discussion than an issue.

I'm trying to understand if this provider is a good match for my use-case for which I was considering using a null_resource.

It seems straightforward to use terracurl to create a resource. In my case, once the resource is created, if I call the create resource url again I'll get an error that the resource already exists - i.e. calls to the endpoint aren't idempotent, I have to first check for the existence of the resource before creating it if it doesn't already exist.

I can easily check for the existence of the resource by hitting another endpoint.

Is there a way to ensure the create request is a no-op conditional on the result of another request so that repeated applies don't fail with Already Exists errors?

Does that make sense?!?

Terraform Version

$ terraform -v
Terraform v1.5.7
on darwin_arm64

When I'm using the TF_LOG=debug with the terracurl provider I can not see the exact urls + data which is it sending/receiving.

The debug output is not detailed enough to allow me to troubleshoot API issues (if needed).

The following lines (captured from the output) are not detailed enough:

Error: -20T09:10:20.536Z [ERROR] provider.terraform-provider-terracurl_v1.2.1: Response contains error diagnostic: diagnostic_detail= diagnostic_summary="unable to make request: request failed, retries exceeded: %!s(<nil>)" tf_provider_addr=registry.terraform.io/devops-rob/terracurl tf_resource_type=terracurl_request @caller=/Users/rbarnes/go/pkg/mod/github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_severity=ERROR tf_proto_version=5.4 tf_req_id=c0e0957e-40f0-783e-1133-0c1cb7b10a76 tf_rpc=ApplyResourceChange timestamp=2024-05-20T09:10:20.536Z

Error: -20T09:10:20.537Z [ERROR] vertex "terracurl_request.pmck_status (destroy)" error: unable to make request: request failed, retries exceeded: %!s(<nil>)

Is there a way how to really see the HTTP details?

Thank you...

Would like be good if we support custom CA and Certificate, like ca_file, cert_file, key_file,
useful when we enforced client to include it like, enforce mTLS in Vault / Consul / Nomad

Terraform Version

Terraform v1.7.0

Terracurl does not provide an option to override the Host using Host header, similar to curl.

Terraform Configuration Files

data "terracurl_request" "home" {
    name   = "home"
    url    = "https://d2mxderzerczer.cloudfront.net"
    method = "GET"

    headers = {
      "Host"               = "www.example.com"
    }

    response_codes = [
      200,
    ]
    max_retry      = 1
    retry_interval = 3
  }

Origin has host header checks and return 400 Bad request if wrong host is provided.

Expected Behavior

Since the header overrides Host with correct value, should return 200.

Actual Behavior

Returns 400.

Use case

My use case is, I'm adding an end to end test using terraform checks to validate the response codes and headers.

check "home_page" {
  data "terracurl_request" "home" {
    name   = "home"
    url    = "https://d2mxderzerczer.cloudfront.net"
    method = "GET"

    headers = {
      "Host"               = "www.example.com"
    }

    response_codes = [
      200,
    ]
    max_retry      = 1
    retry_interval = 3
  }

  assert {
    condition     = data.terracurl_request.home.status_code == 200
    error_message = "${data.terracurl_request.home.url} returned an unhealthy status code"
  }
}

References

This issue is similar to the http provider issue. hashicorp/terraform-provider-http#311

Hi there,

skip_tls_verify and destroy_skip_tls_verify aren't not working as expected when the endpoint doesn't have a valid certificate

Terraform Version

1.3.9

Affected Resource(s)

• terracurl_request

Terraform Configuration Files

terraform {
  required_version = "~> 1.3"
  required_providers {
    terracurl = {
      source  = "devops-rob/terracurl"
      version = "~> 1.0"
    }
  }
}

provider "terracurl" {}

resource "terracurl_request" "this" {
  name = "this"

  url             = "https://expired.badssl.com/?action=create"
  method          = "GET"
  response_codes  = [200]
  skip_tls_verify = true


  destroy_url             = "https://expired.badssl.com/?action=destroy"
  destroy_method          = "GET"
  destroy_response_codes  = [200]
  destroy_skip_tls_verify = true
}

Debug Output

terracurl_request.this: Creating...
terracurl_request.this: Still creating... [10s elapsed]
╷
│ Error: unable to make request: request failed, retries exceeded: Get "https://expired.badssl.com/?action=create": x509: “*.badssl.com” certificate is expired
│
│   with terracurl_request.this,
│   on main.tf line 12, in resource "terracurl_request" "this":
│   12: resource "terracurl_request" "this" {
│

Panic Output

Expected Behavior

The provider should perform the HTTP requests skipping the TLS errors, e.g. expired certificate

Actual Behavior

Terraform cannot be applied due to a bad certificate

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

References

• n/a

Terraform Version

v1.2.8

Affected Resource(s) or Data Source(s)

• d/terracurl_request
• r/terracurl_request

Terraform Configuration Files

Not applicable.

Debug Output

internal/provider/data_source_curl_request.go:112:7: Error return value of `d.Set` is not checked (errcheck)
        d.Set("response", string(body))
             ^
internal/provider/resource_curl.go:198:7: Error return value of `d.Set` is not checked (errcheck)
        d.Set("response", string(body))

SA1019: "io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details.

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

From the source directory, run:

golangci-lint run

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• SA1019: golang/go#40025

Please document the defaults for max_retry and destroy_max_retry. I believe from the source they are both zero, which means no retries by default. I expected to find this in the docs but had to go to the source code.

It may also be worth setting up a non-zero default so that network errors don't necessary cause terraform apply failures, despite the surprisingness of changing defaults. But I think in general terraform providers are expected to make a reasonable effort to keep going to reach the target state?

I see that the data block provides way to read response of an HTTP url but can this be extended to download binary files in base64 encoding as well?

Usecases: hashicorp/terraform-provider-http#158

Currently the only way to have async calls is to use a null_resource and a script. This works, but it doesn't track the state of the async call for purposes of another run in terraform plan and apply.

terracurl fixes this issue for sync calls but not async. If we could add an async poll api endpoint and expected return code then the provider could poll the endpoint until a 200 was returned. This would allow for async calls throughout a system to be made.

If nothing changed in the call on another plan and apply it would simply be skipped, but if the payload changed because it is an update to the resource, then terraform would run the async call again.

I could use this today in a system I am building for a client. I may try to fork and do it on my own, but I am not familiar with writing providers.

Thanks

Looks like the provider binary ain’t working anymore:

│ Error: Failed to install provider
│ 
│ Error while installing devops-rob/terracurl v1.1.0: error checking
│ signature: openpgp: key expired
╵

Hi!

Recently I have been trying to bootstrap some Azure webapps, part of this work includes uploading a ZIP file using the Kudu API via a PUT request.

Problem I have is specifying my .zip file in the request_body.

Would be super nice if TerraCurl could support file payloads! :)

Thanks!
Mike

I've got a weird use-case, where I want to trigger a curl to create a thing, but I don't need to destroy it when doing a Terraform Destroy.

My workaround for now is:

  destroy_url            = "http://example.com"
  destroy_method         = "DELETE"
  destroy_response_codes = [405]

But if we could either make those parameters optional, that could be helpful.