[AWS] Implement retry/backoff when being rate limited by a resource API about cloudquery HOT 9 CLOSED

2021/03/31 14:06:35 rpc error: code = Unknown desc = operation error IAM: GenerateCredentialReport, request canceled, context deadline exceeded
ERROR: 1

Error: rpc error: code = Unknown desc = operation error S3: GetBucketCors, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded

Error: rpc error: code = Unknown desc = operation error IAM: GetAccountAuthorizationDetails, https response error StatusCode: 0, RequestID: , canceled, context deadline exceeded

aws_debug: true
max_retries: 5
max_backoff: 30
timeout: 30

providers:
  - name: aws
    version: latest
    accounts:     
      - id: production
      - id: sharedstorage
      - id: test-us
      - id: test-eu
      - id: test-rds
      - id: prod-kms
      - id: uballot
      - id: mmkpm
#    regions: # Optional. if commented out assumes all regions
#      - us-east-1
#      - us-west-2
    log_level: debug # Optional. if commented out will enable AWS SDK debug logging. possible values: debug, debug_with_signing, debug_with_http_body, debug_with_request_retries, debug_with_request_error, debug_with_event_stream_body
    aws_debug: true
    max_retries: 5
    max_backoff: 30
    timeout: 300
    resources: # You can comment resources your are not interested in for faster fetching.
    *

11:21AM INF Fetched resources @module=aws account_id=12345678912 count=0 region=sa-east-1 resource=fsx.backups timestamp=2021-04-01T11:21:06.750Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=ec2.security_groups timestamp=2021-04-01T11:21:06.751Z
SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/2.0 200 OK
Content-Length: 14
Content-Type: application/x-amz-json-1.1
Date: Thu, 01 Apr 2021 11:21:06 GMT
X-Amzn-Requestid: aa6764e5-de84-4321-87ed-dec999a19428

SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 210
Content-Type: application/x-amz-json-1.1
Date: Thu, 01 Apr 2021 11:21:06 GMT
X-Amzn-Requestid: 647c8634-da01-4ca4-bf66-2ddc95df1233

11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=ec2.network_acls timestamp=2021-04-01T11:21:06.768Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=cloudtrail.trails timestamp=2021-04-01T11:21:06.776Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=ec2.vpcs timestamp=2021-04-01T11:21:06.783Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=0 region=sa-east-1 resource=sns.subscriptions timestamp=2021-04-01T11:21:06.841Z
SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 291
Content-Type: text/xml
Date: Thu, 01 Apr 2021 11:21:05 GMT
X-Amzn-Requestid: 53c9f70e-63ad-55e9-8a85-afd1cd4fa10a

Error: rpc error: code = Unknown desc = operation error IAM: ListRoleTags, https response error StatusCode: 0, RequestID: , canceled, context deadline exceeded
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=0 region=sa-east-1 resource=ecs.cluster timestamp=2021-04-01T11:21:06.901Z
SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/1.1 200 
Content-Length: 108
Content-Type: application/x-amz-json-1.1
Date: Thu, 01 Apr 2021 11:21:06 GMT
X-Amzn-Requestid: 042f8c02-e5e8-4546-8952-c225be8da1b1

Usage:
  cloudquery fetch [flags]

Flags:
      --driver string   database driver postgresql/neo4j (env: CQ_DRIVER) (default "postgresql")
      --dsn string      database connection string (env: CQ_DSN) (example: 'host=localhost user=postgres password=pass DB.name=postgres port=5432')
  -h, --help            help for fetch
      --path string     path to configuration file. can be generated with 'gen config' command (env: CQ_CONFIG_PATH) (default "./config.yml")
      --version         version for fetch

Global Flags:
      --enableConsoleLog      Enable console logging (default true)
      --enableFileLogging     enableFileLogging makes the framework logging to a file (default true)
      --encodeLogsAsJson      EncodeLogsAsJson makes the logging framework logging JSON
      --logDirectory string   Directory to logging to to when file logging is enabled (default ".")
      --logFile string        Filename is the name of the logfile which will be placed inside the directory (default "cloudquery.log")
      --maxAge int            MaxAge the max age in days to keep a logfile (default 3)
      --maxBackups int        MaxBackups the max number of rolled files to keep (default 3)
      --maxSize int           MaxSize the max size in MB of the logfile before it's rolled (default 30)
      --plugin-dir string     Directory to save and load Cloudquery plugins from (env: CQ_PLUGIN_DIR) (default "/root")
  -v, --verbose               Enable Verbose logging

2021/04/01 11:21:06 rpc error: code = Unknown desc = operation error IAM: ListRoleTags, https response error StatusCode: 0, RequestID: , canceled, context deadline exceeded
root@59f586c902df:~# 

11:20AM INF No regions specified in config.yml. Assuming all 22 regions @module=aws timestamp=2021-04-01T11:20:00.526Z
11:20AM INF Configuring SDK retryer @module=aws max_backoff=30 retry_attempts=5 timestamp=2021-04-01T11:20:00.526Z
SDK 2021/04/01 11:20:00 DEBUG Request