seratch / awscala Goto Github PK

Hi,

We're currently using 0.5.2. We were using an older version (0.3.x) before, and this code worked, so something somewhere has changed!

Here is the code:

implicit val sqs = SQS.apply(, )(Region.Ireland)

This is the error:
java.lang.NoSuchFieldError: EU_CENTRAL_1
awscala.Region0$.(Region0.scala:27)
awscala.Region0$.(Region0.scala)
awscala.package$.(package.scala:3)
awscala.package$.(package.scala)

Any clues?

I'm trying to upload a file and set a expiration time to it.

    val metadata: ObjectMetadata = new ObjectMetadata()
    metadata.setContentLength(file.length)
    metadata.setExpirationTime(DateTime.now().plusMinutes(10).toDate)

    bucket.get.putObjectAsPublicRead(filename, file, metadata)

But when I check the file in the s3 bucket it doesn't have expiry date. It's some bug in AWScala or am I using it wrong?

Hi. Wanted to point out this issue I have noticed: If you first create a table, then you delete it, and wait for the deletion to take place, calling

dynamoDB.table(tableName)

throws a NullPointerException rather than returning None. Calling

dynamoDB.table("thisisarandomnameneverusedbefore") returns None, as by expected behavior.

This is the code to replicate the deletion error:
//Table created here...//

val table = dynamoDB.table(tableName).get
dynamoDB.delete(table)

//Wait for deletion
println("Waiting for deletion")
Thread.sleep(timeout.toMillis)

dynamoDB.table(tableName) should be (None) //<-- NPE here

and here is the stacktrace, it appears to be a TableMeta problem:

java.lang.NullPointerException was thrown.
java.lang.NullPointerException
at scala.collection.convert.Wrappers$JListWrapper.iterator(Wrappers.scala:87)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at awscala.dynamodbv2.TableMeta$.apply(TableMeta.scala:13)
at awscala.dynamodbv2.DynamoDB$class.describe(DynamoDB.scala:49)
at awscala.dynamodbv2.DynamoDBClient.describe(DynamoDB.scala:333)
at awscala.dynamodbv2.DynamoDB$class.table(DynamoDB.scala:52)
at awscala.dynamodbv2.DynamoDBClient.table(DynamoDB.scala:333)

Hope it helps!

D.

In some circumstances, leveraging the default constructor for the java S3 client is more appealing than the default constructor provided in AWScala.

Both default constructors look for credentials stored in system properties. However, the scala API fails if the parameters aren't present. The java API won't fail. And if you're running the code on an EC2 instance with an IAM role that grants permissions to S3 buckets, you'll be in the clear. See http://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2

At the moment, this is how I initialize my S3 client.

val s3 = new AmazonS3Client with S3 {
  override def createBucket(name: String): Bucket = super.createBucket(name)
}

Can we add more direct support for the default constructors of the underlying java APIs?

If I use the following code:

import awscala._
import s3._

object AwsTest {
  def main(args: Array[String]): Unit = {
    implicit val s3 = S3.at(Region.US_EAST_1)
    val bucket = s3.bucket("foo").get
    println(bucket.totalSize)
  }
}

I get the error:

[error] (run-main-c) java.lang.IllegalArgumentException: Cannot create enum from us-east-1 value!
java.lang.IllegalArgumentException: Cannot create enum from us-east-1 value!
    at com.amazonaws.services.s3.model.Region.fromValue(Region.java:224)
    at awscala.s3.S3$class.at(S3.scala:28)
    at awscala.s3.S3Client.at(S3.scala:239)
    at awscala.s3.S3$.apply(S3.scala:11)
    at awscala.s3.S3$.at(S3.scala:15)

Since I created my buckets in the us-east-1 region, and AWScala uses US_WEST_1 as the default region (unlike the AWS SDK for Java, which uses us-east-1 as the default region), while I can list buckets, I can do no operations on specific buckets because the Region defaults to US_WEST_1 and I'm unable to override this as you can see above.

If I change the definition of the s3 var to just S3(), then I get the following error from the AWS SDK:

[error] (run-main-e) com.amazonaws.services.s3.model.AmazonS3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: 35046B17DA2581B0), S3 Extended Request ID: iOteNJlE7qEkrQ6F+aMfpBeXgKjdZV0aJtB2BJCPgAQkDqTjPIgMoyvftB9kHGIdeAENGrrA0cQ=

All DynamoDB get operations (get, query, scan, etc...) return Item objects. While the AWS API returns a Map, the Item object turns it into a simple Seq(name, AttributeValue). That means that to get the value of a specific attribute (which I assume is a common case you have to either turn it back into a Map or iterate. It would be much more useful to retain the attribute names and be able to do, for example, item("attribute").

Good Morning,

i have an issue working with this library while using an user without the right to list all existing bucket. If I got it right, every time I try to retrieve a Bucket, the library first tries to list all the existing buckets and then picks the one requested by the user, causing a 403 Forbidden error when using a "limited access" user.

Looking at how the AWS Java client deal with this situation by providing a direct method to retrieve an object by providing a bucket name and file path, i think it could be enough to add a similar method to get around the problem. For example, looking at getObject, i think adding a new

def getObject(bucketName: String, key: String)

could go.

Unfortunately I am not an S3 expert to evaluate all the possible implication, i just hope this to be useful.

What do you think?

Thanks for your work

Has the repo in the instructions been moved?

warn] ==== piotrga: tried
[warn] https://raw.github.com/piotrga/piotrga.github.com/master/maven-repo/asyncdynamo/async-dynamo/1.6.0/async-dynamo-1.6.0.pominfo] Resolving com.typesafe.trace#trace-sigar-libs;0.1.3 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: asyncdynamo#async-dynamo;1.6.0: not found

The listDomains method returns a nextToken when more results are available. This token is ignored.

Hi there,

Checking if a table exists :

dynamoDB.table("mytable").nonEmpty

I expect to get either None or Table. Instead, it results in:

java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:347)
    at scala.None$.get(Option.scala:345)
    at awscala.dynamodbv2.TableMeta.table(TableMeta.scala:36)
    at awscala.dynamodbv2.DynamoDB$$anonfun$table$1.apply(DynamoDB.scala:53)
    at awscala.dynamodbv2.DynamoDB$$anonfun$table$1.apply(DynamoDB.scala:53)
    at scala.Option.map(Option.scala:146)
    at awscala.dynamodbv2.DynamoDB$class.table(DynamoDB.scala:53)
    at awscala.dynamodbv2.DynamoDBClient.table(DynamoDB.scala:338)

Reason for this is that table function will call get on an option type:

hashPK = keySchema.find(_.keyType == aws.model.KeyType.HASH).get.attributeName,

while the keySchema may not have a hash key, for example when a table doesn't exist at all.

Hello, we are using awscala for an intern tool at our university and we need to use the cloudFormation service from Amazon and I was wondering if you are interested to create a new service in your library to support this. Waiting for your reply. Have a nice day :)

I have a table that has more than 1000 rows. and I need to paginate through the result of the scan operation, in the JAVA SDK I used getLastEvaluatedKey() and then start with that for my next iteration , is there something similar in this awsscala ?

for example,

val googlers: Seq[Item] = table.scan(Seq("Company" -> Condition.gt("Google"))) // will filter against the 1st 1000 rows only.

thanks in advance

I am having some problems using a limit in a select with AWScala. It seem to fetch the records in batches of the limit size up until the last one. In my query

select * from logs where liveFrom < '1421192471363' and liveTo > '1421192471363' and rank > '0' order by rank limit 10

I get back 320 records out of a total of 323 matching records without the limit.

I have a feeling this might be related to the Stream based next token implementation but didn't have a chance to investigate.

The Java API seems to work correctly and I switched to Java just for this.

Thanks

Is there any way to use SQS with a stream so I don't have to set up a timer to check and I can provide a function to handle any new messages?

The default scan method assumes that the returned results do not exceed 1MB. However, if the result returned by the scan operation exceeds 1MB this method requires keeping track of the ExclusiveStartKey and then iterating over the result based on that. please refer to the following code snippet for example:

def scan2(table: Table,
    filter: Seq[(String, aws.model.Condition)],
    limit: Int = 1000,
    segment: Int = 0,
    totalSegments: Int = 1,
    select: Select = aws.model.Select.ALL_ATTRIBUTES,
    attributesToGet: Seq[String] = Nil): Seq[Item] = try {

    val req = new aws.model.ScanRequest()
      .withTableName(table.name)
      .withScanFilter(filter.toMap.asJava)
      .withSelect(select)
      .withLimit(limit)
      .withSegment(segment)
      .withTotalSegments(totalSegments)
    if (!attributesToGet.isEmpty) {
      req.setAttributesToGet(attributesToGet.asJava)
    }


     implicit val dynamoDB = DynamoDB(credentialsawsscala)
     val scanResult = dynamoDB.scan(req)


     var lastEvaluatedKey = scanResult.getLastEvaluatedKey()

     var all_data = scala.collection.mutable.Seq[Item]()

     val single_iteration = scanResult.getItems.asScala.map(i => Item(table, i)).toSeq

     all_data = all_data ++ single_iteration
     while (lastEvaluatedKey!=null)
     {

      val req: com.amazonaws.services.dynamodbv2.model.ScanRequest = new aws.model.ScanRequest()
      .withTableName(table.name)
      .withScanFilter(filter.toMap.asJava)
      .withSelect(select)
      .withLimit(limit)
      .withSegment(segment)
      .withTotalSegments(totalSegments)
      .withExclusiveStartKey(lastEvaluatedKey)

      val scanResult = dynamoDB.scan(req)
      val single_iteration = scanResult.getItems.asScala.map(i => Item(table, i)).toSeq
      all_data = all_data ++ single_iteration
      lastEvaluatedKey = scanResult.getLastEvaluatedKey()
     }
     all_data 

  } catch { case e: aws.model.ResourceNotFoundException => Nil }

Please let me know how do you want to address this issue, I will be happy to send a pull request .

When calling runAndAwait, the following exception is throw, even though the instance is created.

com.amazonaws.AmazonServiceException: The instance ID 'i-35ebdfce' does not exist (Service: AmazonEC2; Status Code: 400
  at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)
  at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)
  at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
  at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9393)
  at com.amazonaws.services.ec2.AmazonEC2Client.describeImages(AmazonEC2Client.java:4762)

Sbt version in ./sbt is 0.13.2, while build.properties has 0.13.8.

Support an SNS wrapper, similar to the SQS wrapper.

This means that either you need to provide a mapping for EVERY aws instance call, or provide access to the underlying instance class.

For example, this means there is currently no way to set the tags on an Instance.

def deleteObjects(objs: Seq[S3Object]): Unit = objs.headOption.map {
obj =>
val req = new aws.model.DeleteObjectsRequest(obj.bucket.name)
req.setKeys(objs.map(obj => new aws.model.DeleteObjectsRequest.KeyVersion(obj.key, obj.versionId)).asJava)
deleteObjects(req)
}

First, thanks for this project, it's really helpful.

I got a major bug in the scala 2.11.5 Repl: https://issues.scala-lang.org/browse/SI-9165

as pointed out in the comment, it may be because of the confusion between the type alias Region and the object Region. Is there away we can remove this ambuiguity ?

DynamoDB imposes a limit of the number (100) and result size from a batch get request. The AWS API BatchGetResult does list the items that were NOT returned. The Scala version should either make those items available OR return a Stream that provides ALL the requested items.

Trying to follow the README section on dynamo

scala> import awscala._, dynamodbv2._
import awscala._
import dynamodbv2._

scala> DynamoDB.at(Region.Tokyo)
<console>:14: error: value Tokyo is not a member of object awscala.Region
              DynamoDB.at(Region.Tokyo)
                                 ^

Hello! Your library is really useful and saving me a lot of time, but something tripped me up today.
I was defining a Policy that had conditions, Something like this:

val policy = Policy(
        Seq(
            Statement(Effect.Allow, 
                Seq(
                    Action("s3:ListAllMyBuckets"), 
                    Action("s3:GetBucketLocation")
                ),
                Seq(
                    Resource("arn:aws:s3:::*"),
                    Resource(s"arn:aws:s3:::${bucketName}")
                )
            ),
            Statement(Effect.Allow,
                Seq(
                    Action("s3:ListBucket")
                ),
                Seq(
                    Resource(s"arn:aws:s3:::${bucketName}")
                ),
                conditions = Seq(
                    new awscala.Condition(
                        "StringLike",
                        "s3:prefix",
                        Seq("", s"${keyToAllow}")
                    )
                )
            )
        )
    )

Something that really threw me off was that Resource, Statement and Action are all case classes, so they have the companion object's apply method to Simply state Action(...) instead of new Action(...). While it's not a huge deal, you'll notice that Condition is not a case class and therefore needs to be instantiated with new.

This puzzled me for a while because I was doing the import awscala.Condition fine, but was getting the error not found: value Condition when trying to make my policy. I figure'd it out as you can tell from the above. But is there a reason why Condition is not a case class?

Looks like the current version of AWScala doesn't allow me to specify a proxy configuration (I'm using this for S3). AWS's Java SDK supports proxy through a ClientConfiguration object which can be passed into the S3 constructor.

Any plans to support this in the future? I am playing around with it myself, we'll see if I can add it, but let me know if you have pointers or are planning on adding this.

The awscala.ec2.InstanceType object proxies the aws.model.InstanceType class from the Java SDK. However, it misses some of the new instance types. In my particular case, I'm missing the t2.micro type.

Hello

When i try to create table on local DynamoDB instance, i am getting this error

I'm working on adding CloudFront support in a fork. I've hit a problem that puzzles me and I posted it on StackOverflow. Anyone here care to take a look?

It's available in the sdk: https://github.com/aws/aws-sdk-java/blob/1.8.0/src/main/java/com/amazonaws/services/s3/AmazonS3.java#L2187
but not in this project: https://github.com/seratch/AWScala/blob/master/src/main/scala/awscala/s3/S3.scala#L208

Having this support in the scala client would make a lot of projects easier for me and a lot of people.
The feature is explained in here for Java:
https://aws.amazon.com/blogs/aws/dynamodb-update-json-and-more/

Have I missed it? I couldn't find a similar feature (without implementing my own Json to Map[String, _] conversion).

Hi ,

I'm searching to know how to specify StorageClass option when I put a new object and also to update an existing object (all my objects are stored as STANDARD instead of RR).

KR

/François

I'm updating a project to Scala 2.11.1 and there seems to be no AWScala artifact.

Hi,
do you plan to support async aws clients (i.e AmazonDynamoDBAsyncClient etc) ?
Thanks
Jakub

Fields defined in awscala.ec2.SecurityGroup are hidden by appropriated fields( with the same name) in base class in aws.model.SecurityGroup.

Hello, my objects are created with the wrong content-type and I'd like to change that. The code runs fine and gives no errors, still the metadata is not changed. Here's my code, note that setAsPublicRead works and keys are indeed publicly accessible.

case Some(key) =>
          key.setAsPublicRead()

          key.metadata.setContentType("text/plain")

          val meta = key.getObjectMetadata
          meta.setContentType("text/plain")

          val userMeta = new java.util.HashMap[String, String]
          userMeta.put("test-key", "test-value")
          meta.setUserMetadata(userMeta)

          key.setObjectMetadata(meta)

Issue #89 might be related

Would you consider breaking out the dependencies into more modular pieces? Having to depend on all of aws-java-sdk when only using S3 is really unnecessary. aws-java-sdk is already broken up into pieces, so it would just take breaking the build up into sub-projects and having those depend on each other as needed.

This could potentially increase library adoption. I for one, wouldn't use the library as it is since it's adding something like 85mb of dependencies that I won't be using.

The README shows pretty well how to use the library features but I couldn't find how to configure my AWS credentials(key and secret). How can I do it?

I believe we could add it to README also.

It would be nice to support copyObject method: http://docs.aws.amazon.com/AmazonS3/latest/dev/CopyingObjectUsingJava.html

It's already available in Java SDK

I coded against your library which depends on a very old httpclient and apache common codec. Now I have a conflict that prevents me to continue my development.

I noticed that upgrading to the latest version of aws-java would solve the problem.

Please update the maven repo with a new version ASAP because I'm completely stuck now!

bucket.get.putObject(filename, file, metadata)
How can I configure the ACL?

I just ran sbt test on the develop branch, and got:

[info] Run completed in 54 seconds, 301 milliseconds.
[info] Total number of tests run: 9
[info] Suites: completed 9, aborted 0
[info] Tests: succeeded 6, failed 3, canceled 0, ignored 0, pending 0
[info] *** 3 TESTS FAILED ***
[error] Failed tests:
[error]     awscala.EC2Spec
[error]     awscala.DynamoDBV2Spec
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 200 s, completed May 1, 2014 10:49:17 PM

Is anything else supposed to be running in order for these tests to work, or are there configurations, permissions or configurations required... or are these actually broken tests?

The update to use SDK 1.7.1 has broken SQSSpec, presumably because of the renaming of awscala.sqs.SQS$createQueue to awscala.sqs.SQS$createQueueAndReturnQueueName.

I don't want to fix this myself because I'm unsure of the reasoning behind the name change.

I'm assuming you are aware that the S3 objectSummaries which uses the list objects truncates at 1000 objects and needs to support the isTruncated() and listNextBatchOfObjects() methods/requests

is this on your near term roadmap, is there a workaround (not obvious from reading code) or is it a punt for the foreseeable future?

The IpPermissions companion object's apply method attempts to dereference fields that may be null.