Code Monkey home page Code Monkey logo

smoke-aws's Introduction

Build - Main Branch Swift 5.7, 5.8 and 5.9 Tested Ubuntu 20.04 and 22.04 Tested Join the Smoke Server Side community on gitter Apache 2

SmokeAWS

The SmokeAWS package is a library for communicating with AWS services written in the Swift programming language. Using SwiftNIO for its networking layer, the library provides the ability to call API methods for AWS services either synchronously or asynchronously.

Support Policy

The support policy for this package is described here.

Conceptual overview

Each AWS service has two libraries and corresponding targets in this package-

  • a model library that provides the structure and types that express in Swift the service's API model
  • a client library that provides a number of clients to contact the service or to mock the service for testing-
    • a protocol that defines synchronous and asynchronous variants for all service API methods.
    • an AWS client that can be used to contact the actual AWS service.
      • Support for expontential backoff retries.
      • Logging and emittion of invocation metrics, with API-level support for enabling/disabling emitting metrics
    • a mock client that by default returns a default instance of the return type for each API method.
    • a throwing client that by default throws an error of each API method.

Getting Started

Step 1: Add the SmokeAWS dependency

SmokeAWS uses the Swift Package Manager. To use the framework, add the following dependency to your Package.swift-

dependencies: [
    .package(url: "https://github.com/amzn/smoke-aws.git", from: "2.0.0")
]

Step 2: Depend on the SmokeAWS client libraries you need to use

Once you have specified the SmokeAWS package as a dependency, you can specify the targets from this package that your application needs to depend on. Swift Package Manager will compile these targets as part of your application for you to use. It will not compile the targets in the SmokeAWS package that you don't depend on.

For swift-tools version 5.2 and greater-

    targets: [
        .target(
            name: "SampleServiceOperations", dependencies: [
                .product(name: "ElasticComputeCloudClient", package: "smoke-aws"),
            ]),
        .testTarget(
            name: "SampleServiceOperationsTests", dependencies: [
                .target(name: "SampleServiceOperations"),
            ]),
    ]

For swift-tools version 5.1 and prior-

    targets: [
        .target(
            name: "SampleServiceOperations",
            dependencies: ["ElasticComputeCloudClient"]),
        .testTarget(
            name: "SampleServiceOperationsTests",
            dependencies: ["SampleServiceOperations"]),
    ]

Step 3: Use the client protocol

While it is possible to use the AWS clients directly, in most cases you will want to use the corresponding protocol so you can unit test your code without contacting the AWS service, rather using one of the mock clients to handle service calls locally.

import ElasticComputeCloudClient

public struct SampleServiceOperationsContext {
    public let ec2Client: ElasticComputeCloudClientProtocol

    public init(ec2Client: ElasticComputeCloudClientProtocol) {
        self.ec2Client = ec2Client
    }
}

Using this protocol you can call service API methods and get results-

import ElasticComputeCloudModel
...

    let request = RunInstancesRequest(
        clientToken: nil,
        launchTemplate: instanceLaunchTemplate,
        maxCount: maxCount,
        minCount: minCount,
        subnetId: subnetId)
    let response = try context.ec2Client.runInstancesSync(input: request)
    
    try response.instances?.forEach { instance in ... }

Step 4: Instantiate the AWS client for production

When starting your application in production, you can instantiate an instance of the AWS client and pass it in the place of the protocol to contact the actual AWS service.

Each AWS service provides a Generator type that can be globally instatiated for the application and used to produce a request-specific client.

At application startup-

import ElasticComputeCloudClient
import SmokeAWSCredentials
...

    guard let credentialsProvider = AwsContainerRotatingCredentials.getCredentials(fromEnvironment: environment) else {
        return Log.error("Unable to obtain credentials from the container environment.")
    }
    
    // optional: for the EC2 clients, only emit the retry count metric
    // only report 5XX error counts for DescribeInstances (even if additional operations are added in the future)
    // only report 4XX error counts for operations other than DescribeInstances (including if they are added in the future)
    let reportingConfiguration = SmokeAWSClientReportingConfiguration<ElasticComputeCloudModelOperations>(
        successCounterMatchingOperations: .none,
        failure5XXCounterMatchingRequests: .onlyForOperations([.describeInstances]),
        failure4XXCounterMatchingRequests: .exceptForOperations([.describeInstances]),
        retryCountRecorderMatchingOperations: .all,
        latencyTimerMatchingOperations: .none)

    self.ec2ClientGenerator = AWSElasticComputeCloudClientGenerator(
        credentialsProvider: credentialsProvider,
        awsRegion: region,
        endpointHostName: ec2EndpointHostName,
        connectionTimeoutSeconds: connectionTimeoutSeconds, // optional
        retryConfiguration: retryConfiguration,             // optional
        eventLoopProvider: .createNew,                      // optional
        reportingConfiguration: reportingConfiguration)     // optional

The inputs to this constructor are-

  1. credentialsProvider: The provider of credentials to use for this client.
  • Here we use the SmokeAWSCredentials package to obtain rotating credentials from an AWS runtime such as ECS.
  1. awsRegion: The AWS region to use for this client
  2. endpointHostName: The hostname to contact for invocations made by this client. Doesn't include the scheme or port.
  • For example dynamodb.us-west-2.amazonaws.com.
  1. connectionTimeoutSeconds: The timeout in seconds for requests made by this client.
  2. retryConfiguration: An instance of type HTTPClientRetryConfiguration to indicate how the client should handle automatic retries on failure. Default to a configuration with 5 retries starting at a 500 ms interval.
  3. eventLoopProvider: The provider of the event loop for this client. Defaults to creating a new event loop.
  4. reportingConfiguration: An instance of SmokeAWSClientReportingConfiguration that indicates what metrics to emit for the client. Defaults to a configuration where all metrics for all APIs are emitted.

Within a request-

    let ec2Client = self.ec2ClientGenerator.with(logger: logger)

Recording metrics from the AWS clients will require an metrics implementation to be instatiated for the application for swift-metrics. Currently SmokeAWS doesn't provide a default implementation for Cloudwatch.

The metrics emitted by the AWS clients are-

  1. success: The count of successful invocations.
  2. failure5XX: The count of unsuccessful invocations of the client that return with a 5xx response code.
  3. failure4XX: The count of unsuccessful invocations of the client that return with a 4xx response code.
  4. retryCount: The retry count for invocations of the client.
  5. latency: The latency of invocations from the client.

Step 5: Instantiate a mock client for testing

In unit tests, you can instantiate an instance of the mock or throwing client and pass it in the place of the protocol to verify your code acts as expected. Both mock clients allow you to optionally pass closures to override the default behavior for particular API methods, allowing you to provide custom mock behavior for some but not all API methods.

    var instances: [(instanceId: String, subnetId: String)] = []
    var terminatedInstanceIds: [String] = []
    
    func runInstancesSync(_ input: ElasticComputeCloudModel.RunInstancesRequest)
        throws -> ElasticComputeCloudModel.Reservation {
            var instanceList: InstanceList = []
            
            for _ in 0..<input.maxCount {
                let instanceId = "instance_\(UUID().uuidString)"
                let instance = ElasticComputeCloudModel.Instance(instanceId: instanceId)
                instanceList.append(instance)
                instances.append((instanceId: instanceId, subnetId: input.subnetId!))
            }
            
            return ElasticComputeCloudModel.Reservation(instances: instanceList)
    }
    
    func terminateInstancesSync(input: ElasticComputeCloudModel.TerminateInstancesRequest) throws 
        -> ElasticComputeCloudModel.TerminateInstancesResult {
            terminatedInstanceIds.append(contentsOf: input.instanceIds)
            return ElasticComputeCloudModel.TerminateInstancesResult()
    }
    
    let ec2Client = MockElasticComputeCloudClient(runInstancesSync: runInstancesSync,
                                                  terminateInstancesSync: terminateInstancesSync)
    let context = SampleServiceOperationsContext(ec2Client: ec2Client)

Further Concepts

Package generation

The majority of this package is code generated using SmokeAWSGenerate.

License

This library is licensed under the Apache 2.0 License.

smoke-aws's People

Contributors

haotianzhu avatar jaechoi2 avatar janujan avatar jpeddicord avatar jwenzhon avatar knovichikhin avatar luiabrah avatar maxdesiatov avatar pbthif avatar sushichop avatar tachyonics avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

smoke-aws's Issues

Pre-release identifiers

I'm running into some strange SPM errors when using .branch("2.25.35.alpha.1"). These go away if I specify a specific revision. In this case, .revision("43b6b5..."). I think this is due to how SPM treats branches differently than revisions or tagged versions.

If this package used standardized semver pre-release identifiers, then people be able to depend on alpha versions without using .branch. For example, if the release were tagged as 2.0.0-alpha.1, one could do:

.package(url: "https://github.com/amzn/smoke-aws.git", from: "2.0.0-alpha")

This has the added benefit of automatically opting into later alphas, betas, and the final release when one does swift package update.

Fails to compile with Swift 5.1 on Linux

smoke-aws does not compile using Swift 5.1 compiler on Linux.

Repro Steps

  1. docker run -ti swift:latest /bin/bash
  2. apt-get update && apt-get install -y libssl-dev
  3. git clone https://github.com/amzn/smoke-aws.git
  4. cd smoke-aws
  5. swift build | grep -v warning

Expected

Successful compile

Actual

/smoke-aws/.build/checkouts/XMLCoding/Sources/XMLCoding/XMLStackParser.swift:292:43: error: 'XMLParser' is unavailable: This type has moved to the FoundationXML module. Import that module to use it.
func parserDidStartDocument(_ parser: XMLParser)

(and many more errors)

AWS API fails because lower case codingkeys

When launching an instance, a lot of nested keys fails. There are some examples here:

deleteOnTermination should have `= "DeleteOnTermination"
https://github.com/amzn/smoke-aws/blob/0.16.9/Sources/ElasticComputeCloudModel/ElasticComputeCloudModelStructures.swift#L11575

deviceName should have = "DeviceName"
https://github.com/amzn/smoke-aws/blob/0.16.9/Sources/ElasticComputeCloudModel/ElasticComputeCloudModelStructures.swift#L1394

There are more, however here are a couple of examples.

S3 client unable to specify ContentType for PutObject

Using this code:

let putRequest:PutObjectRequest = .init( body:objectData, bucket:"bucket-name", contentType:"image/jpeg", key:objectKey )

I get a SignatureDoesNotMatch error.

Using this code where I omit the contentType parameter:
let putRequest:PutObjectRequest = .init( body:objectData, bucket:"bucket-name", key:objectKey )

The request succeeds, but the object put into the bucket as a ContentType of application/x-amz-rest-xml

I looked into the underlying code, and I suspect this is due to the Smoke HTTP client adding a header for both the content type supplied by the caller, and the one added by the framework, which seems invalid.

Consider depending on Swift Metrics 2.x (whilst still allowing 1.x)

Swift Metrics 2.0 just got released. It is almost completely compatible with Swift Metrics 1.x unless you exhaustively switch on the TimeUnit enum. I would highly recommend depending on swift-metrics like the following:

// swift-metrics 1.x and 2.x are almost API compatible, so most clients should use
.package(url: "https://github.com/apple/swift-metrics.git", "1.0.0" ..< "3.0.0"),

alternatively, from: "2.0.0" will also work but may in the interim cause compatibility issues with packages that specify from: "1.0.0".

If at all possible, this should be done before tagging the next release here. If using "1.0.0" ..< "3.0.0" this isn't even a SemVer major change because 1.x versions are still accepted.

FatalError: Precondition failed: BUG DETECTED: wait() must not be called when on an EventLoop

Cross-posted: amzn/smoke-http#25

I am using DynamoDBClient from inside a Vapor app.

Inside a route I call AWSDynamoDBClient.queryAsync(input:completion:)

this will eventually call HTTPClient.executeAsync<A>()

in there we have:

try bootstrap.connect(host: endpointHostName, port: endpointPort).wait()

this raises a fatal error:

NIO-ELT-#1 (13): Precondition failed: BUG DETECTED: wait() must not be called when on an EventLoop.
Calling wait() on any EventLoop can lead to
- deadlocks
- stalling processing of other connections (Channels) that are handled on the EventLoop that wait was called on

Further information:
- current eventLoop: Optional(SelectableEventLoop { selector = Selector { descriptor = 17 }, scheduledTasks = PriorityQueue(count: 0): [] })
- event loop associated to future: SelectableEventLoop { selector = Selector { descriptor = 7 }, 

Probably in a method called Async you should never .wait() ... ?

Unauthorized on RunInstances

Hi, i am trying to setup an instance through this package. I have tried various other calls Create key, List keys, describe regions, describe az etc. Without problems.

However when i try to create an instance, i get:

[2019-02-15T11:28:22.650+01:00] [VERBOSE] [HTTPClientChannelInboundHandler.swift:175 handleCompleteResponse(context:)] Got response from endpoint: https://ec2.eu-west-1.amazonaws.com:443/?Action=RunInstances&BlockDeviceMapping.1.deviceName=test&BlockDeviceMapping.1.ebs.deleteOnTermination=true&BlockDeviceMapping.1.ebs.volumeSize=20&dryRun=true&ImageId=ami-00035f41c82244dab&InstanceType=t2.small&Ipv6AddressCount=1&KeyName=jonas-test&MaxCount=1&MinCount=1&SecurityGroup.1=default&SubnetId=subnet-0641b27277697f019&TagSpecification.1.Tag.1.key=Name&TagSpecification.1.Tag.1.value=my-instance&Version=2016-11-15 and path: /?Action=RunInstances&BlockDeviceMapping.1.deviceName=test&BlockDeviceMapping.1.ebs.deleteOnTermination=true&BlockDeviceMapping.1.ebs.volumeSize=20&dryRun=true&ImageId=ami-00035f41c82244dab&InstanceType=t2.small&Ipv6AddressCount=1&KeyName=jonas-test&MaxCount=1&MinCount=1&SecurityGroup.1=default&SubnetId=subnet-0641b27277697f019&TagSpecification.1.Tag.1.key=Name&TagSpecification.1.Tag.1.value=my-instance&Version=2016-11-15 with headers: HTTPResponseHead { version: HTTP/1.1, status: unauthorized, headers: [("Transfer-Encoding", "chunked"), ("Date", "Fri, 15 Feb 2019 10:28:23 GMT"), ("Server", "AmazonEC2")] } and body: 254 bytes
[2019-02-15T11:28:22.650+01:00] [DEBUG] [XMLAWSHttpClientDelegate.swift:120 getResponseError(responseHead:responseComponents:)] Attempting to decode error data into XML: <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>68379e92-59b8-400a-a251-cc72debffa48</RequestID></Response>

I use the following code:

let context = AWSContext(
    region: .eu_west_1,
    accessKey: Environment.get("AWS_KEY") ?? "",
    secretKey: Environment.get("AWS_SECRET") ?? ""
)

// ...

try createInstance(context)

// ...

func createInstance(_ context: AWSContext) throws {
    let instance = RunInstancesRequest(
        blockDeviceMappings: BlockDeviceMappingRequestList([BlockDeviceMapping(
            deviceName: "myEbs",
            ebs: EbsBlockDevice(
                deleteOnTermination: true,
                volumeSize: 20
            )
        )]),
        clientToken: nil,
        dryRun: true,
        imageId: "ami-00035f41c82244dab",
        instanceType: .t2Small,
        ipv6AddressCount: 1,
        keyName: "myKey",
        maxCount: 1,
        minCount: 1,
        securityGroups: SecurityGroupStringList(["default"]),
        subnetId: "subnet-0641b27277697f019",
        tagSpecifications: TagSpecificationList([
            TagSpecification(
                tags: TagList([
                    Tag(key: "Name", value: "my-instance")
                ])
            )
        ])
    )

    let runningInstance = try context.ec2Client.runInstancesSync(input: instance)
}

// ...

public struct AWSContext {
    public var ec2Client: ElasticComputeCloudClientProtocol
    
    public init(region: AWSRegion, accessKey: String, secretKey: String) {
        let credentials = StaticCredentials(
            accessKeyId: accessKey,
            secretAccessKey: secretKey,
            sessionToken: nil
        )

        self.ec2Client = AWSElasticComputeCloudClient(
            credentialsProvider: credentials,
            awsRegion: region,
            endpointHostName: "ec2.\(region.rawValue).amazonaws.com"
        )
    }
}

I have tried with the following credentials:

  • Root credentials
  • IAM credentials with policy AmazonEC2FullAccess

Both credential set give me the same error. Are there something i am missing?

SmokeAWS version: 0.16.9

Add a full S3 integration

Description

Add a full S3 integration code generated from the c2j model.

Blocked on: amzn/smoke-http#6

Exit Criteria

  • Add code generated model types and clients for S3
  • Manually verify the ability to connect to S3 using the generated client.

SecretsManager Integration

I'm trying to rewrite a service that makes pretty regular use of SecretsManager in Swift - could this be added to the roadmap? :)

SmokeAWSCredentials in README.md

import SmokeAWSCredentials
...

    guard let credentialsProvider = AwsContainerRotatingCredentials.getCredentials(fromEnvironment: environment) else {
        return Log.error("Unable to obtain credentials from the container environment.")
    }

The README.md has a section about ** Step 4: Instantiate the AWS client for production** as seen above. I can't find bespoken library neither in the master nor in the swift_5_nio_2 branch. Also I can't find AwsContainerRotatingCredentials. Is StaticCredentials from SmokeAWSCore therefore the way to go? If so give me a thumbs up, I will open a PR to update the documentation.

Sync APIs throw HTTPClientError rather than cause.

Where in SmokeAWS 1.x DynamoDBError.conditionalCheckFailed had been thrown, now HTTPClientError(responseCode: 400, cause: DynamoDBModel.DynamoDBError.conditionalCheckFailed) is being thrown.

This only affects the *Sync APIs. The *Async APIs correctly return the cause error.

Add RDS API

Allowing for managing RDS through this package would be a very nice addition :)

Update Allowed Versions of https://github.com/apple/swift-crypto

Right now only versions less than 2 are allowed. I have other dependencies that require newer versions of the crypto library. The crypt library is now up to version 3.

I updated the dependency and it does compile and pass all the tests. I have not done any testing more strenuous than that.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.