cloudyr / aws.iam Goto Github PK

To assist with bootstrapping credentials in different environments, it is helpful to be able to explicitly load temporary credentials from AWS STS into R environment variables.

This lets one control for explicitly which role to get credentials for. The problem with the calls to aws.signature::locate_credentials within various other functions is that one cannot pass a role.

Also, storing the temporary credentials within environment variables leads to better performance when doing many API requests that require credentials.

suggested usage:

res <- aws.iam::set_temp_role_creds_in_env('$myrole$')
# res == TRUE if successful
# writes to AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN

# then one can do:
aws.s3::s3SaveRDS(mtcars, object = 'test.Rds', bucket = '$mybucket$')
# and know that the specific role is being used

Need to add to docs. Thanks for the package. I'm just having trouble finding out how to get a session token, and I don't see any docs about it.

* checking Rd \usage sections ... WARNING
Undocumented arguments in documentation object 'get_session_token'
  ‘externalid’
Duplicated \argument entries in documentation object 'get_session_token':
  ‘policy’

Duplicated \argument entries in documentation object 'get_account':
  ‘...’

Undocumented arguments in documentation object 'create_group'
  ‘user’ ‘prefix’

Undocumented arguments in documentation object 'iamHTTP'
  ‘...’
Documented arguments not in \usage in documentation object 'iamHTTP':
  ‘dots’

Undocumented arguments in documentation object 'create_profile'
  ‘profile’ ‘path’ ‘...’ ‘role’ ‘n’ ‘marker’ ‘prefix’

Undocumented arguments in documentation object 'create_login'
  ‘user’ ‘password’ ‘reset’ ‘...’

Undocumented arguments in documentation object 'change_pwd'
  ‘requirements’
Documented arguments not in \usage in documentation object 'change_pwd':
  ‘require’

Documented arguments not in \usage in documentation object 'add_policy':
  ‘path’

Undocumented arguments in documentation object 'create_role'
  ‘profile’

Functions with \usage entries need to have the appropriate \alias
entries, and all their arguments documented.
The \usage entries must correspond to syntactically valid R code.
See chapter ‘Writing R documentation files’ in the ‘Writing R
Extensions’ manual.

It is easy enough to add documentation where it is missing and remove it where it is duplicative. However, doing it accurately may be a bit more rough without actually using the package a bit. Is anybody working on this right now or should I think about giving it a stab?

This package is not being actively maintained. If you're interested in contributing or taking over, please express your interest here.

I would like to know how to access s3 bucket via role in R. I understand for a user who has access to their aws access/secret key can store in sysgetenv or .Renviron and access .

how to do the same with iam role which has that s3 permissions:

aws.iam::assume_role(role = "arn:aws:iam::acc-no:role/rolename-Assume-Role", session = "mySession", use = TRUE)
Error in locate_credentials(key = key, secret = secret, session_token = session_token, :
could not find function "locate_credentials"

Put your code here:

## load package
library("aws.iam")

## code goes here


## session info for your system
sessionInfo()


sessionInfo() R version 3.5.3 (2019-03-11) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 18.04.3 LTS  Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1  locale:  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8      [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                   [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C         attached base packages: [1] stats     graphics  grDevices utils     datasets  methods   base       other attached packages: [1] aws.s3_0.3.20   asreml_3.0      lattice_0.20-38 devtools_2.2.1  usethis_1.5.1   shiny_1.4.0      loaded via a namespace (and not attached):  [1] Rcpp_1.0.3               compiler_3.5.3           later_1.0.0              base64enc_0.1-3           [5] prettyunits_1.1.0        remotes_2.1.0            tools_3.5.3              packrat_0.5.0             [9] testthat_2.3.1           digest_0.6.23            pkgbuild_1.0.6           pkgload_1.0.2            [13] jsonlite_1.6             memoise_1.1.0            rlang_0.4.2              cli_2.0.1                [17] rstudioapi_0.10          curl_4.3                 aws.iam_0.1.7            fastmap_1.0.1            [21] xml2_1.2.2               httr_1.4.1               withr_2.1.2              desc_1.2.0               [25] fs_1.3.1                 grid_3.5.3               rprojroot_1.3-2          glue_1.3.1               [29] R6_2.4.1                 processx_3.4.1           fansi_0.4.1              sessioninfo_1.1.1        [33] callr_3.4.0              magrittr_1.5             backports_1.1.5          promises_1.1.0           [37] ps_1.3.0                 ellipsis_0.3.0           htmltools_0.4.0          assertthat_0.2.1         [41] aws.signature_0.5.2.9000 mime_0.8                 xtable_1.8-4             httpuv_1.5.2             [45] crayon_1.3.4
--
 
> | >
>

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

We do have jupyterhub running in EKS. Each jupyter pods uses iam role for service account in order to have access to s3, etc. That works fine with python boto3 and awscli.

However, in R, the only thing i see is the role attached to the worker node, which is not the correct role.

Is there a way to use the same authentication mechanism than awscli or boto3 ? If not, is there a way to use assume role with webidentity ?

Put your code here:

## load package
library("aws.s3")
library("aws.signature")
library("aws.iam")

## code goes here

> get_caller_identity()

$Account  '012345678910'
$Arn  'arn:aws:sts::012345678910:assumed-role/eks-node-group-dev040/i-01adcd0b33f520bf9'
$UserId  'AROAQT5SGVC3YD227SYGI:i-01adcd0b33f520bf9'

> assume_role("arn:aws:iam::01234567810:role/terraform/jupyterhub/dev040-jupyterhub", "test", use=TRUE, key=NULL, secret=NULL, session_token="eyJhbGciOiJSUzI....")

Warning message in stsHTTP(query = query, ...):
“Forbidden (HTTP 403).”
List of 3
 $ Code   : chr "AccessDenied"
 $ Message: chr "User: arn:aws:sts::01234567810:assumed-role/eks-node-group-dev040/i-01adcd0b33f520bf9 is not authorized to per"| __truncated__


## session info for your system
sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Matrix products: default
BLAS/LAPACK: /opt/conda/lib/libopenblasp-r0.3.12.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] aws.iam_0.1.8       aws.signature_0.6.0 aws.s3_0.3.21      

loaded via a namespace (and not attached):
 [1] aws.ec2metadata_0.2.0 digest_0.6.27         crayon_1.3.4         
 [4] IRdisplay_0.7.0       R6_2.5.0              repr_1.1.0           
 [7] lifecycle_0.2.0       jsonlite_1.7.1        evaluate_0.14        
[10] httr_1.4.2            pillar_1.4.7          rlang_0.4.9          
[13] curl_4.3              uuid_0.1-4            xml2_1.3.2           
[16] ellipsis_0.3.1        IRkernel_1.1.1        tools_4.0.3          
[19] compiler_4.0.3        base64enc_0.1-3       htmltools_0.5.0      
[22] pbdZMQ_0.3-3.1

Hey, is there a way to assume role with SAML via this package?

So, given RoleArn, PrincipalArn, SAMLAssertion, and duration:
R code w/ botor package would be like this, but I'm trying to avoid the python requirement:

stsclient <- botor::botor()$client("sts")
token <- stsclient$assume_role_with_saml(RoleArn=role_arn,
                                         PrincipalArn=principal_arn,
                                         SAMLAssertion=assertion,
                                         DurationSeconds=43200L)

Thank you for your effort.

Cory

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

The session_token in stsHTTP is not being read from locate_credentials or environment like it should, and default arg value is NULL which is causing temporary credentials to not work.

More details at cloudyr/aws.signature#19 (comment)

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

Hi, thanks for the great package.

When I tried the list_roles function, my default region was "ap-southeast-1" and I immediately got this error

> list_roles()
$Code
[1] "SignatureDoesNotMatch"

$Message
[1] "Credential should be scoped to a valid region, not 'ap-southeast-1'. "
...

It turns out that the IAM does not require a region for it is a global setting. We can confirm it by going to the IAM setting on the web console, the region at the top right corner should show "global". The default region us-east-1 should be used in the HTTP request. A temporary workaround for me is to pass the region as an argument

> list_roles(region = "us-east-1")
[[1]]
RoleName:   /aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS 
...

I'm not sure how many functions will have the same issue but I guess list_roles is not the only one. It would be better to not follow the region setting returned by aws.signature::locate_credentials and just set it to "us-east-1". Then the user's default region will not mess up the IAM HTTP request.

Best,
Jiefei

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

iam API calls fail in sessions created with get_session_token()

> get_session_token(use=TRUE)
> str(list_roles())
List of 3
 $ Code   : chr "InvalidClientTokenId"
 $ Message: chr "The security token included in the request is invalid"
[...]
> str(get_caller_identity())
List of 3
 $ Account: chr "111111111111"
 $ Arn    : chr "arn:aws:iam::111111111111:user/xxxxx"
 $ UserId : chr "AIDAXXXXXXXXXX"
> aws.s3::bucketlist()
                       Bucket             CreationDate
[...]

Given that the error is "InvalidClientTokenId" and not "AccessDenied" this looks like some possible issue in handling the request. Note, however, that it is unique to iam and does not happen with sts (or S3).

For authentication, is it possible to use the instance metadata service on an Amazon EC2 instance that has an IAM role configured? It would be nice to leverage the facility in Amazon for managing credentials, rather than having to manually configure the environment/code.

Per discussion at cloudyr/aws.s3#65:

• Add a function assume_role(), which assumes a named role, and replaces environment variables
• Add another function to revert to default credentials, maybe release_role() or resume_role(), by retrieving credentials stored in a package environment and resetting environment variables accordingly
• given this is actually part of the STS API, should update docs to reflect that STS is being incorporated here.

Please specify whether your issue is about:

• a possible bug

When running the following code snippet:

library(aws.s3)
library(aws.iam)
library(aws.signature)

delete_saved_credentials()
Sys.unsetenv("AWS_SESSION_TOKEN")
Sys.unsetenv("AWS_ACCESS_KEY_ID")
Sys.unsetenv("AWS_SECRET_ACCESS_KEY")

aws.signature::use_credentials(profile='fastq_backup')
aws.iam::assume_role(role="arn:aws:iam::<ACCOUNT>:role/testRole", session='test', verbose=TRUE, use=TRUE)

get_bucket(bucket = 'testbucket', verbose=TRUE, region='ap-southeast-2')

Without the following four lines, STS tokens are not reused consistently over re-runs:

delete_saved_credentials()
Sys.unsetenv("AWS_SESSION_TOKEN")
Sys.unsetenv("AWS_ACCESS_KEY_ID")
Sys.unsetenv("AWS_SECRET_ACCESS_KEY")

The aws.iam::assume_role fails and the traceback looks like this (could be malformed token or other errors, depending on the ENV state):

> aws.iam::assume_role(role="arn:aws:iam::<ACCOUNT>:role/testRole", session='test', verbose=TRUE, use=TRUE)
List of 3
 $ Code   : chr "InvalidClientTokenId"
 $ Message: chr "The security token included in the request is invalid."
 $ Type   : chr "Sender"
 - attr(*, "headers")=List of 4
  ..$ x-amzn-requestid: chr "dda18a50-79d0-11e8-8927-310c4070d94f"
  ..$ content-type    : chr "application/json"
  ..$ content-length  : chr "175"
  ..$ date            : chr "Wed, 27 Jun 2018 06:11:01 GMT"
  ..- attr(*, "class")= chr [1:2] "insensitive" "list"
 - attr(*, "class")= chr "aws_error"
 - attr(*, "request_canonical")= chr "GET\n/\nAction=AssumeRole&DurationSeconds=900&RoleArn=arn%3Aaws%3Aiam%3A%3A472057503814%3Arole%2FtestRolel"| __truncated__
 - attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n20180627T061101Z\n20180627/us-east-1/sts/aws4_request\ne5811"| __truncated__
 - attr(*, "request_signature")= chr "AWS4-HMAC-SHA256 Credential=ASIAJRSAYSUUWOYOKDYA/20180627/us-east-1/sts/aws4_request, SignedHeaders=host;x-amz-"| __truncated__
NULL
Warning message:
In stsHTTP(query = query, ...) : Forbidden (HTTP 403).

When use: parameter is FALSE, no STS token seems to be ever issued :/

Here is some system information:

R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin17.5.0 (64-bit)
Running under: macOS High Sierra 10.13.5

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] aws.signature_0.4.3 aws.iam_0.1.7       aws.s3_0.3.13       here_0.1           

loaded via a namespace (and not attached):
 [1] httr_1.3.1      compiler_3.5.0  backports_1.1.2 R6_2.2.2        rprojroot_1.3-2 tools_3.5.0     base64enc_0.1-3 curl_3.2        Rcpp_0.12.17   
[10] xml2_1.2.0      jsonlite_1.5    digest_0.6.15  

/cc @reisingerf

locate_credentials are used, but not imported from aws.signature. If the latter is not explicitly attached, most operations will fail with

Error in locate_credentials(key = key, secret = secret, session_token = session_token, :
could not find function "locate_credentials"

Not sure if this belongs here or under aws.signature.

Following up on this Twitter thread - https://twitter.com/dseverski/status/1233162504076976129, for consideration if someone is able to adopt the cloudyr aws suite. It would be wonderful to create support for the Assume Role Provider functionality present in boto3/botocore and documented at https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html. Specifically, I'm looking to find support to read and honor the source_profile, mfa_serial, and credential_process parameters. With these three, cloudyr/aws could be used to transparently assume a role where credentials are not stored on disk in plaintext, but instead are retrieved via secure storage with a program such as aws-vault, assuming a role with temporary credentials.

This is a complicated request, but is currently a missing feature in all of the R AWS packages and would be a tremendous add.