Using KMS encrypted secret key about aws-fluent-plugin-kinesis HOT 8 CLOSED

@jtblin Thanks for your comment. Let me think about this.

from aws-fluent-plugin-kinesis.

@jtblin Thank you for waiting.

I have thought about the idea you suggested. Even if we use kms_aws_sec_key, we have to decrypt it with properly delegated IAM user/role, and stored the decrypted plain secret key in the process. It means secret key can be leaked if we put some debug method into the code.

Another idea is to use AssumeRole. This is a clean solution, however we need to run token vending machine in the account of Kinesis Stream holder. ( and of course we need some tweak to this plugin ).

Thus, how about create a dedicated IAM user for Kinesis data ingestion which can only do data ingestion to Kinesis, then distribute the AK/SK to producers? It might be what you are doing currently though, it looks reasonable solution to me.

from aws-fluent-plugin-kinesis.

@imaifactory yes you need to decrypt the keys but that can be done using an IAM role as it can be encrypted in the same account as the client. The secret key could still be leaked if someone modify the code to display it so it is not entirely bullet proof but it's OK in my opinion. Our use case is that these keys are deployed on hundreds of instances for internal services and I'm not looking for a perfect solution but protect against basic leakage of the credentials which is the case as of now.

AssumeRole seems like a great option, but it only returns credentials valid for one hour. I guess you could make it so that it gets new credentials before they expire?

Re: dedicated users, as mentioned above we have hundreds of services so creating one user per service is not a valid option in our case, and rotating the credentials would be hard to manage.

Note that this issue is mostly due to kinesis not supporting passing the full ARN instead of the stream name, so if this is fixed soon then the issue would disappear for us, but at the moment it is a serious problem for us.

Happy to look into the AssumeRole more, or is it something that you'd be likely to do? If not do you have any particular design guideline in mind?

from aws-fluent-plugin-kinesis.

@jtblin

AssumeRole seems like a great option, but it only returns credentials valid for one hour. I guess you could make it so that it gets new credentials before they expire?

Yes. AssumeRole support to provide credential valid for 1 hour at maximum. So we need to refresh credentials continuously in the plugin. But I think this is reasonable and right way.

In this case, the configuration would be like this

<match test.source>
    type kinesis
    stream_name STREAM
    region us-east-1
    flush_interval 1
    random_partition_key

    role_arn YOUR_ROLE_TO_BE_ASSUMED

</match>

I have a question for you.
http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-walkthrough-crossacct.html
We can find the steps in the web page above, you need to configure trust policy from kinesis stream holder account to each client accounts. Is this reasonable for you?

from aws-fluent-plugin-kinesis.

Yes that would be totally reasonable. And it looks like the ruby aws-sdk already handles auto refreshing the token so should be fairly simple: aws/aws-sdk-ruby#520.

from aws-fluent-plugin-kinesis.

I will leave this open until RubyGems will be released.

from aws-fluent-plugin-kinesis.

Thank you for waiting. We have released.
https://rubygems.org/gems/fluent-plugin-kinesis/versions/0.3.6

from aws-fluent-plugin-kinesis.

Thanks @imaifactory 👍

from aws-fluent-plugin-kinesis.