d2a4u / meteor Goto Github PK

I noticed that the underlying Dynamo client is a member of the table definitions. I am wondering why this is the case?

It feels like it goes against my intuition of seeing table definitions as pure data structures. I wonder if their inclusion undermines the table/indexes classes as pure/case

Also, unwinding this decision would be huge for the API so maybe something to consider for a v2 or overhaul

Currently SecondarySimpleIndex retrieve method returns F[Option[T]]. But if you have a simple Global Secondary Index, you can have any partition key (not guaranteed unique value), so retrieve should return Stream[F, T]. (While a LocalSecondaryIndex would have the same partition key as the table, it must have a sort key, and so wouldn't be used with a SecondarySimpleIndex).

Currently having to use a workaround like:

val byAuthorIndexFixed = SecondaryCompositeIndex[IO, String, String](
    "book_table",
    "book_by_author_GSI",
    KeyDef[String]("author", DynamoDbType.S),
    KeyDef[String]("dummy", DynamoDbType.S),
    dynamoDbAsyncClient
  )

looks like only Scala 2.12 was published?

https://mvnrepository.com/artifact/io.github.d2a4u/meteor-dynosaur

Description
The default Encoder[Instant] instance fails with a java.lang.ArithmeticException: long overflow error when trying to encode dates that are very far in the future.

The cause of the bug is that the default Encoder represents time values as Long millisecond timestamps, although Instant is capable of storing much larger values than what a Long can.

Example

import meteor.syntax._
import java.time.Instant

val t = Instant.ofEpochMilli(Long.MaxValue).plusMillis(1)
t.asAttributeValue // throws ArithmeticException

Possible solution
Changing Encoder[Instant] to represent values as ISO strings.
This would also have the added benefit of not truncating sub-millisecond values.

implicit val InstantCodec: Codec[Instant] =
  new Codec[Instant] {
    def read(av: AttributeValue): Either[DecoderError, Instant] =
      av.as[String].flatMap { str =>
        Either
          .catchNonFatal(Instant.parse(str))
          .left
          .map(err => DecoderError(err.getMessage(), err.getCause().some))
      }

    def write(a: Instant): AttributeValue = a.toString().asAttributeValue

  }

Limit is an optional parameter to the Query API: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-Limit

When not sent, AWS automatically limits the data to around 1MB. Being forced to pick a value to exceed that amount adds friction to using the API. Since Meteor automatically pages through using lastEvaluatedKey, setting a limit doesn't seem to be of value, as it doesn't limit how many records Meteor returns, just the size of a batch.

We've run into a problem when using Meteor and Dynosaur. When there is a conversion error, the exception message is not set in the DecodeError in this line conversions.scala#L23.

When debugging, I observed that the Throwable.detailedMessage field doesn't contain a value, but for some reason, the object contains a message field that includes the error information.

Losing this information makes it very hard to troubleshoot problems with the data and/or the application.,

I will try to set up a simple test case and update this ticket.

I realise this would be a lot of work but thought I'd mention it anyway, it would be great to get scalajs support, this could be achieved either by using the aws JS SDK on by smithy4s

table.update(
  partitionKey = "some-partition-key",
  update = Expression(
    "REMOVE #x1",
    Map("#x1" -> "someAttribute"),
    Map.empty
  )
)

This will result in the following error:

software.amazon.awssdk.services.dynamodb.model.DynamoDbException: ExpressionAttributeValues must not be empty

If I understand correctly you shouldn't set the expressionAttributeValues to an empty collection, but rather not set it at all in that case. I guess that probably the same issue can occur for expressionAttributeNames.
Essentially it's a bug in the aws SDK imho.

A workaround I found for now is adding a condition that should always be true, because the names and values will be merged with those of the update:

condition = Expression(
  "attribute_not_exists(#idontexist) OR #idontexist = :dummy",
  Map("#idontexist" -> "idontexist"),
  Map(":dummy" -> "dummy".asAttributeValue)
)

AWS Docs: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-sort-keys.html

Context:

Suppose the following type represents an entry in a Dynamo table:

case class Entry(
    partitionKey: String,
    compositeSortKey: Rich,
    date: LocalDate
)

object Entry {
  implicit val entryDynosaurSchema: DynosaurSchema[Entry] =
    DynosaurSchema.record { field =>
      (
        field("partitionKey", _.partitionKey)(DynosaurSchema[String]),
        field("compositeSortKey", _.compositeSortKey)(DynosaurSchema[Rich]),
        field("date", _.date)(
          DynosaurSchema[LocalDate]
        )
      ).mapN(Entry.apply)
    }

  implicit val entryMeteorCodec: MeteorCodec[Entry] =
    schemaToCodec[Entry](entryDynosaurSchema)
}

We define a composite sort key by:

case class Rich(field1: String, field2: LocalDate)

object Rich {
  private def parseField1(
      field1: String
  ): Either[ReadError, String] = ...

  private def parseField2(
      field2: String
  ): Either[ReadError, LocalDate] = ...

  implicit val richSchema: DynosaurSchema[Rich] =
    DynosaurSchema[String].imapErr {
      case s"$field1#$field2" =>
        (parseField1(field1), parseField2(field2)).mapN(
          Rich.apply
        )
      case v => ReadError(s"Bad composite: $v").asLeft
    }(r => show"${r.field1}#${r.field2.toEpochDay}")

  implicit val richMeteorCodec: MeteorCodec[Rich] = schemaToCodec(
    richSchema
  )
}

(It is hinted at above, but for the sake of clarity I'll note that I've written machinery to normalize LocalDate to a Long for comfortable reading and writing. It's not relevant to the issue at hand, but I want the behavior described to make sense)

So, when I go to write entries, I can write an Entry("someId", Rich("low-cardinality", LocalDate.now()), LocalDate.now()) and get a row in Dynamo like

This is all lovely, but I'd like to be able to use a SortKeyQuery to target my queries more specifically. In particular, I'd like to be able to say BeginsWith("low-cardinality"), but I can't since the S has to be Rich. E.g.,

val byCompositeKey: CompositeTable[IO, String, Rich] =
  CompositeTable(
    "example-table",
    KeyDef[String]("partitionKey", DynamoDbType.S),
    KeyDef[Rich]("sortKey", DynamoDbType.S),
    ddb
  )

val byCompositeKeyUnsafe: CompositeTable[IO, String, String] =
  CompositeTable(
    "example-table",
    KeyDef[String]("partitionKey", DynamoDbType.S),
    KeyDef[String]("sortKey", DynamoDbType.S),
    ddb
  )

The first only allows me to construct SortKeyQuery[Rich], while the second lets me do whatever with the string rep at the cost of type safety. I'd really love to be able to represent a more principled query, but I don't know if the tools exist in the library to do so.

This feels like something that could be solvable with optics, but I'm not quite sure how to go about it. It seems like I ought to be able to construct a Prism[Rich, String] (the real code uses newtypes pervasively, so it wouldn't actually be String) and then use that to construct a SortKeyQuery specialized to one or more parts of the sort key.

I'm on vacation for the next couple of weeks, so I may see if I can grok enough of the code that's already here to prove out that idea, but I figured I'd throw the idea out in case it gave you an AHA! moment.

Implement BatchWriteItem per API doc: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html

The project is currently cross published for scala 2.12 and 2.13. Are there any plans to start supporting scala3?
Importing meteor in scala3 projects using cross(CrossVersion.for3Use2_13) is not an option, because dependencies have different incompatible artefacts for scala2 and scala3

I'm not sure if this project currently publishes scaladoc for its API? If it does then I am struggling to find a link to it, perhaps it could be made more prominent?

Underlying Dynamo queries can consume the indexes they hit in an ascending or descending order. The default value is ascending

It is possible to pass a ConditionExpression to DeleteItem request so that the deletion is only perform under certain condition. https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html#API_DeleteItem_RequestSyntax