support for asynchronous endpoints about terraform-provider-restapi HOT 15 OPEN

The Terraform aws_instance resource has a default timeout of 10 minutes https://www.terraform.io/docs/providers/aws/r/instance.html
The provider calls run_instances and then polls the describe_instances API to figure out when the instance is ready.
That is to say, the situation described here isn't unique.

from terraform-provider-restapi.

I agree that this would be a neat feature, but I am concerned that this falls into the territory of being too API-specific for this general-purpose provider. Is there a "standard" (for lack of better terms) for how the object ID would be returned? In a few cases, I've seen ref returned in the Location header, but I've also seen this being the location of the new resource created.

from terraform-provider-restapi.

There is no real standard in REST, but it seems to be common practice to immediately accept a request and spin up a workflow in the background. Isn't it possible to add a REGEX alike variable where you can define how the provider should catch the ref/job_id from the return value? I would also add a variable where you define the time interval between the GET poll requests

from terraform-provider-restapi.

Right - totally agreed that REST doesn't exactly have a standard. I just mean to ask if there's a general convention that would be expected. My inclination is that this would be very API-specific so a potentially bad fit for a generic provider like this one.

Pointers welcome

from terraform-provider-restapi.

Isn't it HTTP standard that what's in the Location header?
https://tools.ietf.org/html/rfc7231#section-7.1.2

from terraform-provider-restapi.

(Or maybe I don't get what exactly you mean by ref, maybe the id?)

from terraform-provider-restapi.

Sorry for the super slow reply, @BernhardBln. I mean to say that it's possible that the object location can be returned in the Location header, but is that what is commonly expected with such asynchronous APIs?

Under the covers of the provider, enabling something like this would require a custom redirect handler (if the POST returns a Location header to the "real" resource) that would follow the redirect and retry a number of times. I think that's doable, but am not sure it's a great pattern. However, a major issue remains: how does that "real" path get fixed in the .tf file? The object being manage is expected to be provided in path, so if a redirect occurs that points elsewhere, there's a conflict the provider wouldn't be able to rectify...

from terraform-provider-restapi.

I'd say that is what is commonly expected: http://restcookbook.com/Resources/asynchroneous-operations/

Maybe for that case you would not specify the object that is expected? Especially in case you can't know, e.g. when you are not allowed to provide e.g. a random UUID as id for the to-be-created resource?

I guess then you would not initially GET the object to see if it already exists, but rather check in the state if you have a URL there (and if not, you need to create the object).

So I'd see this workflow:

POST /workflows/{workflow_uuid}/jobs     -> creates object and returns status URL

Save the status URL in the TF state to indicate that object creation was triggered

Now keep checking if the resource was created. Since it's in the TF state, you can even do this if TF crashes and tries again:

GET {status url}  -> query until you receive 303, 
                     then get the URL of the created resource from Location Header

Then remove the status URL from the TF state, and save the URL of the created resource and make available as output.

That also means, here you do not think in terms of IDs and URLs with path params any more, but in terms of URLs that could theoretically point wherever...

from terraform-provider-restapi.

Thanks, @BernhardBln, I think I understand the idea and agree that this is a standard-enough mechanism. It makes sense to add to the provider (as time becomes available - adding the helpwanted tag).

I think this would require a fairly delicate orchestration of behavior, but it seems like the right flow of thought. I suppose there could be a number of internal bits that can be used to follow the request flow and a custom handler would need to be created so Go doesn't try to follow the 3xx with no understanding.

One other thing to consider: The provider should "block" until the object is created since other resources may rely on the resource being created. This could wreak havoc with terraform's timeouts. If a timeout occurs, but the server deletes the temporary resource (returns a 410 - Gone), I think we'd basically be lost and a thing would exist on the API server that we don't know about. As we chat design, I'd love to hear how you think we might address that.

from terraform-provider-restapi.

Yeah, we'd run into a timeout, but another run would theoretically fix that.

I'd assume you would not do heavy crunching or trigger huge backups in a provisioning pipeline, but if you do, you need to make sure the pipeline is more robust then.

Have to admit I haven't thought about the 410 GONE situation, good catch, and let's be realistic, in most cases it'll be 404 anyways... I'd suggest to simply drop the status URL from the state then and try again? That must be safe to do anyways. Maybe make configurable if a retry is desired and how many are allowed?

Feels like this should be driven by actual needs, unfortunately I don't have one any more at the moment 😉

from terraform-provider-restapi.

Any Chance this can get merged?

from terraform-provider-restapi.

Any Chance this can get merged?

from terraform-provider-restapi.

Any plan to get this feature into master branch?

from terraform-provider-restapi.

+1 for this feature. Will be ready to help on this do we need a design discussion to finalise?

from terraform-provider-restapi.

Any chances to finalize it?

from terraform-provider-restapi.