Allow filtering of resources about terraforming HOT 21 OPEN

@kbroughton I pushed up my implementation to master...soapy1:pre-filtering-resources

You can use it like this:

bin/terraforming ec2 --filters '[{"name":"instance-id", "values":["i-0bdd70830cd1a3ac0"]}]'

so far, it's only able to filter ec2 instances.

I haven't written any tests for it yet. If people like the implementation then I will flush it out a bit more, add tests and make a pr.

from terraforming.

This was an incredibly naive implementation but it worked for my purposes. Our internal priorities changed so I haven't had much time to try to turn it into a suitable PR, but hopefully it'll be helpful for someone: master...iragsdale:master

from terraforming.

This would also really help me. +1

from terraforming.

Like this?

$ terraforming s3 --resource hoge
resource "aws_s3_bucket" "hoge" {
    bucket = "hoge"
    acl    = "private"
}

It sounds good and useful if it exists.
@iragsdale Could you send a pull request if you already have an idea of implementation?

from terraforming.

Something like that, yeah. I'll see if I can take a crack at it soon.

from terraforming.

So this isn't quite ready for a pull request, but it's functional for EC2 resources and illustrates a relatively simple approach.

iragsdale@d82159c

It lets you specify multiple patterns like:

terraforming ec2 --limit instance_type=m3.medium tags=~whatever placement.availability_zone=us-east-1b

I ran into some trouble with the existing specs, this blows up around 25 of them for some reason and I'm a bit too tired to figure out why at the moment. (Plus my ruby is a bit rusty).

Anyway, if the approach makes sense I don't think it should be too difficult to implement the basic matching for the remaining resource types.

It's also a bit confusing because we're matching on the objects being returned by the SDK but printing out the values expected by Terraform, so the output shows availability_zone but the matcher requires placement.availability_zone. There are some other minor matching issues like that where some cleverness could make it way easier to use (tags are one spot).

Anyway, if you could give this a once over and maybe point me in the right direction on the specs, I could maybe clean it up enough to submit a pull request.

from terraforming.

Sorry for the late reply ... 🙇 🙇 🙇
It seems good creating Matcher class 😄

I understood your problem. placement.availability_zone= is not a good option, because user should know about AWS API.

I think maybe constructing the struct which has the same fields as Terraform resource before executing filtering is very helpful...
e.g. EC2) SDK returns Aws::EC2::Types::Instance instance, then convert it to the struct like Terraform's instance resource. After that Terraforming executes filtering.

Because this approach may be so heavy, however, how about implement a simple filter to select by resource name?

from terraforming.

We are using Terraforming on our project as well, and have to reverse engineer a giant pile of EC2 resources. Filtering, especially by things like instance tag, would be incredibly helpful. What is the status of this work in progress? Can it be turned into a pull request within a reasonably speedy timeframe? ;)

from terraforming.

I think filtering by tag would be really helpful. Currently I am trying to transform multiple CloudFormation stacks into terraform environments.

from terraforming.

+1

from terraforming.

Thanks,
I pulled your branch master.

I was able to get filtering with
terraforming ec2 --limit placement.availability_zone=us-west-2a

The options suggest you can filter by tag, but I'm not sure how to
Options:
[--merge=MERGE] # tfstate file to merge
[--tfstate], [--no-tfstate] # Generate tfstate
[--limit=image_id=ami-xxxx placement.availability_zone=us-east-1b tags=name] # List matching resources. Arguments using = match using regular expressions.

I would expect --limit tags=<tag_key>=<tag_value> or --limit tags="<tag_key>,tag_value" so that you can specify the filtering to only match when supplied key,value is a match.

Could you supply the correct syntax?

from terraforming.

@kbroughton I'm sorry, I don't think I'm gonna be able to provide much in the way of support right now. The syntax sort of depends on the structure of the objects being matched, and I'm a bit too busy at work to get set up to figure that out right now. :(

from terraforming.

I ran into a similar problem at work, where I was trying to use terraforming to describe only a small subset of aws resources. I approached the problem by using leveraging the aws sdk opposed to matching. I was wondering what people thought of this approach and if it would be useful to anybody if I submitted a pr with this feature?

from terraforming.

That's a great approach. Is there a way to factor it out so you don't have to touch every single file? It seems the current architecture makes it difficult to get the hook into the klass on execute.

I would also be fine for now with just being able to filter after, but i'm so new to ruby, i don't know how to access the dict items from output like
resource "aws_instance" "control" {
blah: blah
tags: { dict i want access to }
}

Could you suggest a code snippet to get at tags for me?

from terraforming.

@soapy1 if you have a beta version of your PR i am working on this issue for the next day or two.

from terraforming.

Thanks a lot soapy! I'm using it now. Nice implementation.
I've written some automation to add the snippet equivalents for each resource that is taggable.

          attributes.merge!(tags_attributes_of(instance))

and

      def tags_attributes_of(instance)
        tags = instance.tags
        attributes = { "tags.#" => tags.length.to_s }
        tags.each { |tag| attributes["tags.#{tag.key}"] = tag.value }
        attributes
      end

Now i'm working on spreading the filtering you did for ec2.rb across all resources.

from terraforming.

Hi soapy1,

I got a little further extending your work to other modules and adding better tag support.
#215

from terraforming.

@kbroughton any chance we can merge this? I would really benefit from the --filter flag.

from terraforming.

I'm afraid I haven't touched this for years, and won't have time to work on it this month. Happy to have someone fork it and re-submit.

from terraforming.

@kbroughton I pushed up my implementation to master...soapy1:pre-filtering-resources

You can use it like this:

bin/terraforming ec2 --filters '[{"name":"instance-id", "values":["i-0bdd70830cd1a3ac0"]}]'

so far, it's only able to filter ec2 instances.

I haven't written any tests for it yet. If people like the implementation then I will flush it out a bit more, add tests and make a pr.

Hi Soapy1, Not sure if the filtering was implemented but I really need this feature as I only want to do terraforming with single vpc (prod). pls let me know the resolution on this, thanks a lot.

from terraforming.

@skambiente, I'm in a similar boat at kbroughton here. I haven't touched this in a long time, and I'm not involved in the project, so I don't know. It seems like the PR that add this feature is still open #215. You might be able to use a check out of that branch to use the feature.

from terraforming.