cloudyr / aws.iam Goto Github PK
View Code? Open in Web Editor NEWAWS IAM Client Package
Home Page: https://cran.r-project.org/package=aws.iam
AWS IAM Client Package
Home Page: https://cran.r-project.org/package=aws.iam
To assist with bootstrapping credentials in different environments, it is helpful to be able to explicitly load temporary credentials from AWS STS into R environment variables.
This lets one control for explicitly which role to get credentials for. The problem with the calls to aws.signature::locate_credentials
within various other functions is that one cannot pass a role.
Also, storing the temporary credentials within environment variables leads to better performance when doing many API requests that require credentials.
suggested usage:
res <- aws.iam::set_temp_role_creds_in_env('$myrole$')
# res == TRUE if successful
# writes to AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
# then one can do:
aws.s3::s3SaveRDS(mtcars, object = 'test.Rds', bucket = '$mybucket$')
# and know that the specific role is being used
Need to add to docs. Thanks for the package. I'm just having trouble finding out how to get a session token, and I don't see any docs about it.
* checking Rd \usage sections ... WARNING
Undocumented arguments in documentation object 'get_session_token'
‘externalid’
Duplicated \argument entries in documentation object 'get_session_token':
‘policy’
Duplicated \argument entries in documentation object 'get_account':
‘...’
Undocumented arguments in documentation object 'create_group'
‘user’ ‘prefix’
Undocumented arguments in documentation object 'iamHTTP'
‘...’
Documented arguments not in \usage in documentation object 'iamHTTP':
‘dots’
Undocumented arguments in documentation object 'create_profile'
‘profile’ ‘path’ ‘...’ ‘role’ ‘n’ ‘marker’ ‘prefix’
Undocumented arguments in documentation object 'create_login'
‘user’ ‘password’ ‘reset’ ‘...’
Undocumented arguments in documentation object 'change_pwd'
‘requirements’
Documented arguments not in \usage in documentation object 'change_pwd':
‘require’
Documented arguments not in \usage in documentation object 'add_policy':
‘path’
Undocumented arguments in documentation object 'create_role'
‘profile’
Functions with \usage entries need to have the appropriate \alias
entries, and all their arguments documented.
The \usage entries must correspond to syntactically valid R code.
See chapter ‘Writing R documentation files’ in the ‘Writing R
Extensions’ manual.
It is easy enough to add documentation where it is missing and remove it where it is duplicative. However, doing it accurately may be a bit more rough without actually using the package a bit. Is anybody working on this right now or should I think about giving it a stab?
This package is not being actively maintained. If you're interested in contributing or taking over, please express your interest here.
I would like to know how to access s3 bucket via role in R. I understand for a user who has access to their aws access/secret key can store in sysgetenv or .Renviron and access .
how to do the same with iam role which has that s3 permissions:
aws.iam::assume_role(role = "arn:aws:iam::acc-no:role/rolename-Assume-Role", session = "mySession", use = TRUE)
Error in locate_credentials(key = key, secret = secret, session_token = session_token, :
could not find function "locate_credentials"
Put your code here:
## load package
library("aws.iam")
## code goes here
## session info for your system
sessionInfo()
sessionInfo() R version 3.5.3 (2019-03-11) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 18.04.3 LTS Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] aws.s3_0.3.20 asreml_3.0 lattice_0.20-38 devtools_2.2.1 usethis_1.5.1 shiny_1.4.0 loaded via a namespace (and not attached): [1] Rcpp_1.0.3 compiler_3.5.3 later_1.0.0 base64enc_0.1-3 [5] prettyunits_1.1.0 remotes_2.1.0 tools_3.5.3 packrat_0.5.0 [9] testthat_2.3.1 digest_0.6.23 pkgbuild_1.0.6 pkgload_1.0.2 [13] jsonlite_1.6 memoise_1.1.0 rlang_0.4.2 cli_2.0.1 [17] rstudioapi_0.10 curl_4.3 aws.iam_0.1.7 fastmap_1.0.1 [21] xml2_1.2.2 httr_1.4.1 withr_2.1.2 desc_1.2.0 [25] fs_1.3.1 grid_3.5.3 rprojroot_1.3-2 glue_1.3.1 [29] R6_2.4.1 processx_3.4.1 fansi_0.4.1 sessioninfo_1.1.1 [33] callr_3.4.0 magrittr_1.5 backports_1.1.5 promises_1.1.0 [37] ps_1.3.0 ellipsis_0.3.0 htmltools_0.4.0 assertthat_0.2.1 [41] aws.signature_0.5.2.9000 mime_0.8 xtable_1.8-4 httpuv_1.5.2 [45] crayon_1.3.4
--
> | >
>
Please specify whether your issue is about:
We do have jupyterhub running in EKS. Each jupyter pods uses iam role for service account in order to have access to s3, etc. That works fine with python boto3 and awscli.
However, in R, the only thing i see is the role attached to the worker node, which is not the correct role.
Is there a way to use the same authentication mechanism than awscli or boto3 ? If not, is there a way to use assume role with webidentity ?
Put your code here:
## load package
library("aws.s3")
library("aws.signature")
library("aws.iam")
## code goes here
> get_caller_identity()
$Account '012345678910'
$Arn 'arn:aws:sts::012345678910:assumed-role/eks-node-group-dev040/i-01adcd0b33f520bf9'
$UserId 'AROAQT5SGVC3YD227SYGI:i-01adcd0b33f520bf9'
> assume_role("arn:aws:iam::01234567810:role/terraform/jupyterhub/dev040-jupyterhub", "test", use=TRUE, key=NULL, secret=NULL, session_token="eyJhbGciOiJSUzI....")
Warning message in stsHTTP(query = query, ...):
“Forbidden (HTTP 403).”
List of 3
$ Code : chr "AccessDenied"
$ Message: chr "User: arn:aws:sts::01234567810:assumed-role/eks-node-group-dev040/i-01adcd0b33f520bf9 is not authorized to per"| __truncated__
## session info for your system
sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS
Matrix products: default
BLAS/LAPACK: /opt/conda/lib/libopenblasp-r0.3.12.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] aws.iam_0.1.8 aws.signature_0.6.0 aws.s3_0.3.21
loaded via a namespace (and not attached):
[1] aws.ec2metadata_0.2.0 digest_0.6.27 crayon_1.3.4
[4] IRdisplay_0.7.0 R6_2.5.0 repr_1.1.0
[7] lifecycle_0.2.0 jsonlite_1.7.1 evaluate_0.14
[10] httr_1.4.2 pillar_1.4.7 rlang_0.4.9
[13] curl_4.3 uuid_0.1-4 xml2_1.3.2
[16] ellipsis_0.3.1 IRkernel_1.1.1 tools_4.0.3
[19] compiler_4.0.3 base64enc_0.1-3 htmltools_0.5.0
[22] pbdZMQ_0.3-3.1
Hey, is there a way to assume role with SAML via this package?
So, given RoleArn, PrincipalArn, SAMLAssertion, and duration:
R code w/ botor package would be like this, but I'm trying to avoid the python requirement:
stsclient <- botor::botor()$client("sts")
token <- stsclient$assume_role_with_saml(RoleArn=role_arn,
PrincipalArn=principal_arn,
SAMLAssertion=assertion,
DurationSeconds=43200L)
Thank you for your effort.
Cory
Please specify whether your issue is about:
The session_token
in stsHTTP
is not being read from locate_credentials
or environment like it should, and default arg value is NULL
which is causing temporary credentials to not work.
More details at cloudyr/aws.signature#19 (comment)
Please specify whether your issue is about:
Hi, thanks for the great package.
When I tried the list_roles
function, my default region was "ap-southeast-1" and I immediately got this error
> list_roles()
$Code
[1] "SignatureDoesNotMatch"
$Message
[1] "Credential should be scoped to a valid region, not 'ap-southeast-1'. "
...
It turns out that the IAM does not require a region for it is a global setting. We can confirm it by going to the IAM setting on the web console, the region at the top right corner should show "global". The default region us-east-1
should be used in the HTTP request. A temporary workaround for me is to pass the region as an argument
> list_roles(region = "us-east-1")
[[1]]
RoleName: /aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS
...
I'm not sure how many functions will have the same issue but I guess list_roles
is not the only one. It would be better to not follow the region setting returned by aws.signature::locate_credentials
and just set it to "us-east-1"
. Then the user's default region will not mess up the IAM HTTP request.
Best,
Jiefei
Please specify whether your issue is about:
iam
API calls fail in sessions created with get_session_token()
> get_session_token(use=TRUE)
> str(list_roles())
List of 3
$ Code : chr "InvalidClientTokenId"
$ Message: chr "The security token included in the request is invalid"
[...]
> str(get_caller_identity())
List of 3
$ Account: chr "111111111111"
$ Arn : chr "arn:aws:iam::111111111111:user/xxxxx"
$ UserId : chr "AIDAXXXXXXXXXX"
> aws.s3::bucketlist()
Bucket CreationDate
[...]
Given that the error is "InvalidClientTokenId" and not "AccessDenied" this looks like some possible issue in handling the request. Note, however, that it is unique to iam
and does not happen with sts
(or S3
).
For authentication, is it possible to use the instance metadata service on an Amazon EC2 instance that has an IAM role configured? It would be nice to leverage the facility in Amazon for managing credentials, rather than having to manually configure the environment/code.
Per discussion at cloudyr/aws.s3#65:
assume_role()
, which assumes a named role, and replaces environment variablesrelease_role()
or resume_role()
, by retrieving credentials stored in a package environment and resetting environment variables accordinglyPlease specify whether your issue is about:
When running the following code snippet:
library(aws.s3)
library(aws.iam)
library(aws.signature)
delete_saved_credentials()
Sys.unsetenv("AWS_SESSION_TOKEN")
Sys.unsetenv("AWS_ACCESS_KEY_ID")
Sys.unsetenv("AWS_SECRET_ACCESS_KEY")
aws.signature::use_credentials(profile='fastq_backup')
aws.iam::assume_role(role="arn:aws:iam::<ACCOUNT>:role/testRole", session='test', verbose=TRUE, use=TRUE)
get_bucket(bucket = 'testbucket', verbose=TRUE, region='ap-southeast-2')
Without the following four lines, STS tokens are not reused consistently over re-runs:
delete_saved_credentials()
Sys.unsetenv("AWS_SESSION_TOKEN")
Sys.unsetenv("AWS_ACCESS_KEY_ID")
Sys.unsetenv("AWS_SECRET_ACCESS_KEY")
The aws.iam::assume_role
fails and the traceback looks like this (could be malformed token or other errors, depending on the ENV state):
> aws.iam::assume_role(role="arn:aws:iam::<ACCOUNT>:role/testRole", session='test', verbose=TRUE, use=TRUE)
List of 3
$ Code : chr "InvalidClientTokenId"
$ Message: chr "The security token included in the request is invalid."
$ Type : chr "Sender"
- attr(*, "headers")=List of 4
..$ x-amzn-requestid: chr "dda18a50-79d0-11e8-8927-310c4070d94f"
..$ content-type : chr "application/json"
..$ content-length : chr "175"
..$ date : chr "Wed, 27 Jun 2018 06:11:01 GMT"
..- attr(*, "class")= chr [1:2] "insensitive" "list"
- attr(*, "class")= chr "aws_error"
- attr(*, "request_canonical")= chr "GET\n/\nAction=AssumeRole&DurationSeconds=900&RoleArn=arn%3Aaws%3Aiam%3A%3A472057503814%3Arole%2FtestRolel"| __truncated__
- attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n20180627T061101Z\n20180627/us-east-1/sts/aws4_request\ne5811"| __truncated__
- attr(*, "request_signature")= chr "AWS4-HMAC-SHA256 Credential=ASIAJRSAYSUUWOYOKDYA/20180627/us-east-1/sts/aws4_request, SignedHeaders=host;x-amz-"| __truncated__
NULL
Warning message:
In stsHTTP(query = query, ...) : Forbidden (HTTP 403).
When use:
parameter is FALSE
, no STS token seems to be ever issued :/
Here is some system information:
R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin17.5.0 (64-bit)
Running under: macOS High Sierra 10.13.5
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
locale:
[1] C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] aws.signature_0.4.3 aws.iam_0.1.7 aws.s3_0.3.13 here_0.1
loaded via a namespace (and not attached):
[1] httr_1.3.1 compiler_3.5.0 backports_1.1.2 R6_2.2.2 rprojroot_1.3-2 tools_3.5.0 base64enc_0.1-3 curl_3.2 Rcpp_0.12.17
[10] xml2_1.2.0 jsonlite_1.5 digest_0.6.15
/cc @reisingerf
locate_credentials
are used, but not imported from aws.signature
. If the latter is not explicitly attached, most operations will fail with
Error in locate_credentials(key = key, secret = secret, session_token = session_token, :
could not find function "locate_credentials"
Not sure if this belongs here or under aws.signature
.
Following up on this Twitter thread - https://twitter.com/dseverski/status/1233162504076976129, for consideration if someone is able to adopt the cloudyr aws suite. It would be wonderful to create support for the Assume Role Provider functionality present in boto3/botocore and documented at https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html. Specifically, I'm looking to find support to read and honor the source_profile
, mfa_serial
, and credential_process
parameters. With these three, cloudyr/aws
could be used to transparently assume a role where credentials are not stored on disk in plaintext, but instead are retrieved via secure storage with a program such as aws-vault, assuming a role with temporary credentials.
This is a complicated request, but is currently a missing feature in all of the R AWS packages and would be a tremendous add.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.