API key creation for dedicated clusters fails if terraform can't access cluster about terraform-provider-confluent HOT 14 CLOSED

👋 @west-david thanks for opening an issue!

I think your concern totally makes sense and we'll implement your 2nd suggestion in one of our future releases!

from terraform-provider-confluent.

@linouk23 ahhhhhh good point... 🤦 definitely overlooked that. As awesome as it would be to have the ability to terraform everything end to end over the public cloud API endpoint I definitely acknowledge that there are some limitations. I suppose we will have to re-work some of the CI to run "inside" the cloud environments that have access to the clusters.

from terraform-provider-confluent.

As a future enhancement, would it be possible to add the DNS Domain as an output variable for the confluent_network resource?

That's a great idea, #40

from terraform-provider-confluent.

@ericdalling we're happy to let you know we've just released 0.9.0 version of TF Provider that includes dns_domain and zonal_subdomains computed attributes for confluent_network resource so we closed #40.

from terraform-provider-confluent.

@west-david @ericdalling @maheshbhole check out our latest 0.13.0 release where we

Added disable_wait_for_ready attribute to disable readiness check for confluent_api_key resource (#25, #51).

from terraform-provider-confluent.

thanks @linouk23 🙌

from terraform-provider-confluent.

On a somewhat related note, could you share with us how you are going to create topic / ACLs using created Kafka API Key given the fact the Kafka cluster won't be reachable from the CI pipeline? cc @west-david

from terraform-provider-confluent.

I'm running into the same error when creating a confluent_api_key with a dedicated cluster over AWS Private Link. We have added the private link to the VPC of our CI pipeline but still get the same error. I'm wondering if Terraform is using the wrong URL to communicate with the cluster, when using private link. It looks like it is using the REST endpoint vs the private link's DNS domain. Is that the right behavior?

from terraform-provider-confluent.

@ericdalling that's very surprising, are you using one of our example PL configurations by any chance?

from terraform-provider-confluent.

@linouk23 thanks for sharing that example, I hadn't seen that before. Most of what we are doing looks the same, except for the Route53 resources. So will this only work if we create the Route53 private hosted zone to override the URLs for the REST endpoints? I'm not sure that is the best approach. It seems like it would be better to have an optional variable at the resources that need to connect to the cluster to override the cluster's URL. This way we could pass in the private links URL and not need to use Route53. I'm not sure we will get approved to setup Route53 in our build team's AWS account.

from terraform-provider-confluent.

@ericdalling the example I sent corresponds to our tutorial:

Any DNS provider can be used - AWS Route53 (used in this example) is not required. Any DNS provider that can ensure DNS is routed as follows is acceptable.

It seems like it would be better to have an optional variable at the resources that need to connect to the cluster to override the cluster's URL. This way we could pass in the private links URL and not need to use Route53. I'm not sure we will get approved to setup Route53 in our build team's AWS account.

That sounds interesting, it'd be great if could share this config in a PR or something.

Did you include depends_on block for your api_key resource like this?

# The goal is to ensure that
  # 1. confluent_role_binding.app-manager-kafka-cluster-admin is created before
  # confluent_api_key.app-manager-kafka-api-key is used to create instances of
  # confluent_kafka_topic resource.
  # 2. Kafka connectivity through AWS PrivateLink is setup.
  depends_on = [
    confluent_role_binding.app-manager-kafka-cluster-admin,

    confluent_private_link_access.aws,
    aws_vpc_endpoint.privatelink,
    aws_route53_record.privatelink,
    aws_route53_record.privatelink-zonal,
  ]

from terraform-provider-confluent.

@linouk23 in the tutorial that you shared, it says

Paste Confluent Cloud DNS into Domain Name. This can be found in the Confluent Cloud Console.

Is that referring the the Private Link's DNS Domain? If so, it doesn't appear that the DNS Domain is available as an output variable on the confluent_network resource. As a future enhancement, would it be possible to add the DNS Domain as an output variable for the confluent_network resource?

from terraform-provider-confluent.

Hello,
I am still getting above error.
Is this check of validation of API key against actual clsuter disabled ?
I tried another work around of giving wait time but that also does not work?

from terraform-provider-confluent.

@maheshbhole see #51 (comment)

from terraform-provider-confluent.