d2a4u / meteor Goto Github PK
View Code? Open in Web Editor NEWhttps://d2a4u.github.io/meteor/
License: MIT License
https://d2a4u.github.io/meteor/
License: MIT License
Description
The default Encoder[Instant]
instance fails with a java.lang.ArithmeticException: long overflow
error when trying to encode dates that are very far in the future.
The cause of the bug is that the default Encoder
represents time values as Long
millisecond timestamps, although Instant
is capable of storing much larger values than what a Long
can.
Example
import meteor.syntax._
import java.time.Instant
val t = Instant.ofEpochMilli(Long.MaxValue).plusMillis(1)
t.asAttributeValue // throws ArithmeticException
Possible solution
Changing Encoder[Instant]
to represent values as ISO strings.
This would also have the added benefit of not truncating sub-millisecond values.
implicit val InstantCodec: Codec[Instant] =
new Codec[Instant] {
def read(av: AttributeValue): Either[DecoderError, Instant] =
av.as[String].flatMap { str =>
Either
.catchNonFatal(Instant.parse(str))
.left
.map(err => DecoderError(err.getMessage(), err.getCause().some))
}
def write(a: Instant): AttributeValue = a.toString().asAttributeValue
}
The project is currently cross published for scala 2.12 and 2.13. Are there any plans to start supporting scala3?
Importing meteor in scala3 projects using cross(CrossVersion.for3Use2_13)
is not an option, because dependencies have different incompatible artefacts for scala2 and scala3
We've run into a problem when using Meteor and Dynosaur. When there is a conversion error, the exception message is not set in the DecodeError
in this line conversions.scala#L23.
When debugging, I observed that the Throwable.detailedMessage
field doesn't contain a value, but for some reason, the object contains a message
field that includes the error information.
Losing this information makes it very hard to troubleshoot problems with the data and/or the application.,
I will try to set up a simple test case and update this ticket.
I'm not sure if this project currently publishes scaladoc for its API? If it does then I am struggling to find a link to it, perhaps it could be made more prominent?
Currently SecondarySimpleIndex
retrieve
method returns F[Option[T]]
. But if you have a simple Global Secondary Index, you can have any partition key (not guaranteed unique value), so retrieve
should return Stream[F, T]
. (While a LocalSecondaryIndex would have the same partition key as the table, it must have a sort key, and so wouldn't be used with a SecondarySimpleIndex
).
Currently having to use a workaround like:
val byAuthorIndexFixed = SecondaryCompositeIndex[IO, String, String](
"book_table",
"book_by_author_GSI",
KeyDef[String]("author", DynamoDbType.S),
KeyDef[String]("dummy", DynamoDbType.S),
dynamoDbAsyncClient
)
AWS Docs: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-sort-keys.html
Context:
Suppose the following type represents an entry in a Dynamo table:
case class Entry(
partitionKey: String,
compositeSortKey: Rich,
date: LocalDate
)
object Entry {
implicit val entryDynosaurSchema: DynosaurSchema[Entry] =
DynosaurSchema.record { field =>
(
field("partitionKey", _.partitionKey)(DynosaurSchema[String]),
field("compositeSortKey", _.compositeSortKey)(DynosaurSchema[Rich]),
field("date", _.date)(
DynosaurSchema[LocalDate]
)
).mapN(Entry.apply)
}
implicit val entryMeteorCodec: MeteorCodec[Entry] =
schemaToCodec[Entry](entryDynosaurSchema)
}
We define a composite sort key by:
case class Rich(field1: String, field2: LocalDate)
object Rich {
private def parseField1(
field1: String
): Either[ReadError, String] = ...
private def parseField2(
field2: String
): Either[ReadError, LocalDate] = ...
implicit val richSchema: DynosaurSchema[Rich] =
DynosaurSchema[String].imapErr {
case s"$field1#$field2" =>
(parseField1(field1), parseField2(field2)).mapN(
Rich.apply
)
case v => ReadError(s"Bad composite: $v").asLeft
}(r => show"${r.field1}#${r.field2.toEpochDay}")
implicit val richMeteorCodec: MeteorCodec[Rich] = schemaToCodec(
richSchema
)
}
(It is hinted at above, but for the sake of clarity I'll note that I've written machinery to normalize LocalDate
to a Long
for comfortable reading and writing. It's not relevant to the issue at hand, but I want the behavior described to make sense)
So, when I go to write entries, I can write an Entry("someId", Rich("low-cardinality", LocalDate.now()), LocalDate.now())
and get a row in Dynamo like
partitionKey |
sortKey |
date |
---|---|---|
someId |
low-cardinality#12345 |
12345 |
This is all lovely, but I'd like to be able to use a SortKeyQuery
to target my queries more specifically. In particular, I'd like to be able to say BeginsWith("low-cardinality")
, but I can't since the S
has to be Rich
. E.g.,
val byCompositeKey: CompositeTable[IO, String, Rich] =
CompositeTable(
"example-table",
KeyDef[String]("partitionKey", DynamoDbType.S),
KeyDef[Rich]("sortKey", DynamoDbType.S),
ddb
)
val byCompositeKeyUnsafe: CompositeTable[IO, String, String] =
CompositeTable(
"example-table",
KeyDef[String]("partitionKey", DynamoDbType.S),
KeyDef[String]("sortKey", DynamoDbType.S),
ddb
)
The first only allows me to construct SortKeyQuery[Rich]
, while the second lets me do whatever with the string rep at the cost of type safety. I'd really love to be able to represent a more principled query, but I don't know if the tools exist in the library to do so.
This feels like something that could be solvable with optics, but I'm not quite sure how to go about it. It seems like I ought to be able to construct a Prism[Rich, String]
(the real code uses newtypes pervasively, so it wouldn't actually be String
) and then use that to construct a SortKeyQuery
specialized to one or more parts of the sort key.
I'm on vacation for the next couple of weeks, so I may see if I can grok enough of the code that's already here to prove out that idea, but I figured I'd throw the idea out in case it gave you an AHA! moment.
Underlying Dynamo queries can consume the indexes they hit in an ascending or descending order. The default value is ascending
looks like only Scala 2.12 was published?
https://mvnrepository.com/artifact/io.github.d2a4u/meteor-dynosaur
I noticed that the underlying Dynamo client is a member of the table definitions. I am wondering why this is the case?
It feels like it goes against my intuition of seeing table definitions as pure data structures. I wonder if their inclusion undermines the table/indexes classes as pure/case
Also, unwinding this decision would be huge for the API so maybe something to consider for a v2 or overhaul
Limit is an optional parameter to the Query API: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#DDB-Query-request-Limit
When not sent, AWS automatically limits the data to around 1MB. Being forced to pick a value to exceed that amount adds friction to using the API. Since Meteor automatically pages through using lastEvaluatedKey
, setting a limit doesn't seem to be of value, as it doesn't limit how many records Meteor returns, just the size of a batch.
table.update(
partitionKey = "some-partition-key",
update = Expression(
"REMOVE #x1",
Map("#x1" -> "someAttribute"),
Map.empty
)
)
This will result in the following error:
software.amazon.awssdk.services.dynamodb.model.DynamoDbException: ExpressionAttributeValues must not be empty
If I understand correctly you shouldn't set the expressionAttributeValues
to an empty collection, but rather not set it at all in that case. I guess that probably the same issue can occur for expressionAttributeNames
.
Essentially it's a bug in the aws SDK imho.
A workaround I found for now is adding a condition
that should always be true, because the names and values will be merged with those of the update
:
condition = Expression(
"attribute_not_exists(#idontexist) OR #idontexist = :dummy",
Map("#idontexist" -> "idontexist"),
Map(":dummy" -> "dummy".asAttributeValue)
)
It is possible to pass a ConditionExpression
to DeleteItem
request so that the deletion is only perform under certain condition. https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html#API_DeleteItem_RequestSyntax
Implement BatchWriteItem
per API doc: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html
I realise this would be a lot of work but thought I'd mention it anyway, it would be great to get scalajs support, this could be achieved either by using the aws JS SDK on by smithy4s
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.