n3twork / dynamap Goto Github PK

View Code? Open in Web Editor NEW

5.0 12.0 4.0 653 KB

Dynamap is Java object model and mapping library for Amazon's DynamoDB database.

Home Page: https://dynamap.n3twork.com

License: Apache License 2.0

Java 87.38% FreeMarker 12.62%

dynamodb object-mapping object-mapping-to-db java

dynamap's Introduction

Dynamap

Dynamap is Java object model and mapping library for Amazon's DynamoDB database.

It generates strongly typed Java classes that represent your schema and indexes, and provides methods for saving, querying and updating the state.

Creating and persisting an object is as simple as:

 UserBean user = new UserBean("mark").setCurrencyBalancesAmount("gold",10);
 dynamap.save(new SaveParams(user));

Reading an object:

UserBean user = dynamap.getObject(new GetObjectParams(new GetObjectRequest<>(UserBean.class).withHashKeyValue("mark")));

Updating an object:

UserBeanUpdates updates = user.createUpdates()
updates.incrementCurrencyBalancesAmount(2).setStatus("away");
dynamap.update(new UpdateParams(updates));

Benefits:

Define your schema and attribute behavior using JSON.
Strongly typed classes are automatically generated.
Automatically creates collections and indexes in DynamoDB.
Update objects hide the complexity of DynamoDB update expressions and provide the ability to execute fine grained, concurrently safe updates.
Updates track changes to the state of the object, including delta amounts for numeric types. Original state can be retrieved if necessary.
Supports blind updates - the ability to modify persisted state without having to read it first.
Provides simple methods that hide the complexity of building conditional expressions.
Significantly reduces the amount of code you have to write and makes your code easy to read and comprehend.
Provides a mechanism for rate limiting reads and writes.
Additional custom generated types can be defined and nested in the top level document.
Easy to make simple modifications to your schema such as adding new fields. Just make the change and regenerate the code
Provides a mechanism for schema migrations for more complex changes

Quick Start

Check out the quick start guide

or consult the full documentation on the official site: https://dynamap.n3twork.com

Available on Maven Central

Dynamap is published on Maven central

<dependency>
    <groupId>com.n3twork.dynamap</groupId>
    <artifactId>dynamap</artifactId>
    <version>0.9.74</version>
</dependency>

dynamap's People

Contributors

Stargazers

Watchers

Forkers

killergerbah bm-w vazor-n3twork perblue theenigmathatisme

dynamap's Issues

Generated Dynamap Interface Fields Should Be All Constants

Generated interface fields are in the format without visibility, scope and final modifiers, i.e. String ID_FIELD = "Id";

The variables should have the public static final to be true constants, i.e.

public static final String ID_FIELD = "Id";

Generated Bean Class JSONProperty Requires Constant

Version: 0.9.56

After the generation of the Dynmap class files, Eclipse reports errors for each interface constant used in the @JsonProperty annotation used in the bean class.

Referencing the interface in the annotation resolves this, e.g. @JsonProperty(Application.ID_FIELD)

{  
  "tables": [
    {
      "table": "Application",
      "package": "com.example.application",
      "type": "Application",
      "version": 1,
      "hashKey": "id",
      "types": [
        {
          "name": "Application",
          "description": "New application",
          "fields": [
            {
              "name": "id",
              "description": "Primary key",
              "dynamoName": "Id",
              "type": "String"
            },
            {
              "name": "profile",
              "description": "Profile name",
              "dynamoName": "profile",
              "type": "String"
            },
            {
              "name": "lastUpdatedAt",
              "description": "Updated date/time",
              "dynamoName": "LastUpdatedAt",
              "type": "String"
            },
            {
              "name": "applicationName",
              "description": "The application name",
              "dynamoName": "ApplicationName",
              "type": "String"
            }
          ]
        }
      ]
    }
  ]
}

Schema in Quickstart Guide Requires Update

Reference Quickstart guide, Step 2. Define a schema.

The following updates are required:

Missing opening curly brackets
Property hashkey should be hashKey

{
  "tables": [
    {
      "table": "User",
      "package": "com.n3twork.dynamap.example",
      "type": "User",
      "version": 1,
      "hashKey": "id",
   ...
  }]
}

Compilation failure when optimistic locking enabled and a custom "revision' field

Bad schema:

{
  "tables": [
    {
      "table": "Test",
      "package": "com.n3twork.wom.test",
      "schemaVersionField": "schemaVersion",
      "type": "Test",
      "version": 1,
      "hashKey": "id",
      "optimisticLocking": true,
      "types": [
        {
          "name": "Test",
          "fields": [
            {
              "name": "id",
              "dynamoName": "id",
              "type": "String"
            },
            {
              "name": "revision",
              "dynamoName": "revision",
              "type": "String"
            }
          ]
        }
      ]
    }
  ]
}

scanspec.withExclusiveStartKey is being initialized with wrong arguments

dynamap/src/main/java/com/n3twork/dynamap/Dynamap.java

Line 381 in 8ca9085

    
           scanspec.withExclusiveStartKey(scanRequest.getStartExclusiveHashKey(), scanRequest.getStartExclusiveRangeKey());

API is:
With only HasKey
scanspec.withExclusiveStartKey(String hashKeyName, Object hashKeyValue)

With HashKey and RangeKey
scanspec.withExclusiveStartKey(String hashKeyName, Object hashKeyValue, String rangeKeyName, Object rangeKeyValue)

Fields with persisted == false are annotated with @JsonProperty in the corresponding bean class

This can cause Jackson to attempt to deserialize such fields out of the field loaded from Dynamo DB which may result in exceptions like the following:

! java.lang.IllegalArgumentException: Conflicting setter definitions for property "consumedCapacity": com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(java.util.List) vs com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(com.amazonaws.services.dynamodbv2.model.ConsumedCapacity)
! at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getSetter(POJOPropertyBuilder.java:512)
! at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getPrimaryType(POJOPropertyBuilder.java:353)
! at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getRawPrimaryType(POJOPropertyBuilder.java:373)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.filterBeanProps(BeanDeserializerFactory.java:709)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.addBeanProps(BeanDeserializerFactory.java:559)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:277)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:150)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:414)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
! ... 84 common frames omitted
! Causing: com.fasterxml.jackson.databind.JsonMappingException: Conflicting setter definitions for property "consumedCapacity": com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(java.util.List) vs com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(com.amazonaws.services.dynamodbv2.model.ConsumedCapacity)
!  at [Source: UNKNOWN; line: -1, column: -1]
! at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:302)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:268)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
! at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
! at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:581)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
! at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
! at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:558)
! at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:188)
! at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:28)
! at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:765)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:550)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
! at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
! at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
! at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:591)
! at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4733)
! at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4307)
! ... 66 common frames omitted
! Causing: java.lang.IllegalArgumentException: Conflicting setter definitions for property "consumedCapacity": com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(java.util.List) vs com.n3twork.dynamap.DynamoRateLimiter#setConsumedCapacity(com.amazonaws.services.dynamodbv2.model.ConsumedCapacity)
!  at [Source: UNKNOWN; line: -1, column: -1]
! at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4314)
! at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4245)
! at com.n3twork.dynamap.DynamapBeanFactory.asDynamapBean(DynamapBeanFactory.java:33)
! at com.n3twork.dynamap.DynamapLoadService.loadItem(DynamapLoadService.java:92)
! at com.n3twork.dynamap.Dynamap.batchGetObject(Dynamap.java:387)```

Invalid code generation when using collections of generated types

Take for example this (partial) schema, where the root type has a field that’s a list of element of another generated type:

{ …, "types": [
  {
    "name": "Root",
    "description": "The table’s root type",
    "fields": [
      { "name": "id", "dynamoName": "id", "description": "The hash key", "type": "String" },
      { "name": "listOfGenerated", "dynamoName": "logen", "type": "List", "elementType": "GeneratedListElement" }
    ]
  },
  {
    "name": "GeneratedListElement",
    "fields": [
      { "name": "foo", "dynamoName": "foo", "type": "String" }
    ]
  }
] }

The generated RootBean class will have the following annotated member variable, where the argument to the annotation contains invalid syntax:

@JsonDeserialize(as=List<GeneratedListElement>Bean.class)
private List< GeneratedListElement> listOfGenerated;

This invalid code is generated by line 42 of bean.ftl (as of current master, 952ed0e):

@JsonDeserialize(as=<@field_type field=field />Bean.class)

Remove need to check for null on batchGetObjectSingleCollection

//FIXME:dynamap should ensure we always get a list. On version 0.9.25 it may return null if none was in storage
List list = dynamap.batchGetObjectSingleCollection(new BatchGetObjectRequest()
.withGetObjectRequests(beanRequests));
return list != null ? list : Collections.emptyList();

Validation: attribute missing "dynamoName" gives unhelpful error message

Ran into this refactoring a schema that was previously persist: false, so it didn't need a dynamoName. It will now be persisted and I failed to provide a dynamoName. The error I get is pretty unhelpful. I think we can add this to a more general "validation" task for schemas.

Exception in thread "main" java.lang.RuntimeException: Error processing ./schemas/guild.json
	at com.n3twork.dynamap.CodeGenerator.main(CodeGenerator.java:64)
Caused by: java.lang.IllegalArgumentException: Type: Guild has a duplicated field dynamo name: null. Field dynamo name must be unique.
	at com.n3twork.dynamap.CodeGenerator.generateSchemaClasses(CodeGenerator.java:115)
	at com.n3twork.dynamap.CodeGenerator.generateCode(CodeGenerator.java:80)
	at com.n3twork.dynamap.CodeGenerator.main(CodeGenerator.java:62)

Implement the ability to execute parallel scans

Incorrect symbol used for NOT_EQUALS comparison operator

Conditional updates that use the NOT_EQUALS operato fails with an exception like:

com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Invalid ConditionExpression: Syntax error; token: "!", near: "#t1a0 !=" (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: b1d63797-a48d-4c35-8a7c-45c92c508b08)

This is "<>" should be used instead of "!=" to represent "not-equals" in the update condition according to:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html

Dynamap attempts to migrate query results from GSIs that aren't configured with ALL projection type

Dynamap attempts to migrate query results for GSIs that are not configured with ALL projection type, which may result in the log message: "Schema version field does not exist for {} on item with hash key {}. Migrating item to current version"

In this case the message is triggered because the schema version field is not being projected into the index.

Provide a standalone version of the Dynamap code generator.

For various reasons, we want a workflow where we can run the Dynamap code generator from the commandline or called from a driver script. Currently, this is difficult because we need to build a classpath for the Dynamap code generator and that involves tracking down many jar files.

Ideally, the mvn artifact for Dynamap would include an application jar version of the code generator so we could simply:

java -jar dynamapcodegen.jar <args>

This would make it easy to integrate into the workflow we're adopting.
thanks!

Implement the ability to replace nested generated types without updates object for those with special field handling.

Currently it is not possible to replace an entire nested generated type object in a parent updates object instead of using it's associated updates object if the nested type contains compressed or serialize as list fields.

Note, however, in most situations it should not be necessary to replace a nested generated object like this. The whole point of using generated types is that you can work with an updates object. This should even work for creating brand new objects.

Implement partitioning on batchGetObject to handle the 100 items limit on DynamoDB

Potential issue with addCheckAttributeInMapNotExistsCondition

The "map" that I was adding addCheckAttributeInMapNotExistsCondition and addCheckAttributeInMapExistsCondition on was defined like this:

"compressCollection": "gzip"

So obviously that isn't going to work! The scary thing is that a addCheckAttributeInMapNotExistsCondition on a gzipped map passes. The addCheckAttributeInMapExistsCondition was failing. Dynamap boils these down to simple attribute_exists() statements in DynamoDB... so it makes sense that this is the behavior we get.

This could be dangerous though for uses of addCheckAttributeInMapNotExistsCondition. I just checked the few usages on our game and they look okay but I can easily imagine such a bug being introduced when we scramble to save document space and migrate to gzipped maps without noticing that we neutered one of these conditional checks.

I don't think there's a straightforward fix in Dynamap... we would have to find a way to vary the DynamapExpressionBuilder based on the Bean type. Not impossible... but probably something that needs careful consideration.

Add List<T> batchQueryOneCollection

Unable to use projection type KEYS_ONLY for GSIs

When a table is configured with a GSI with projection type KEYS_ONLY, the GSI is created with projection type ALL instead.

Compilation error when two fields with the same dynamo name defined

The schema should fail validation.

equalsFields are not checked against fields

Specifying values in equalsFields in a Dynamap schema that are not real fields defined in the schema should be a codegen error... but currently Dynamap will include the fields in the generated equals method which results in broken code.

set* methods for fields of type List do not cause the specified value to be persisted unless `deltas` is false

Given a field with this configuration:

            {
              "name": "stringList",
              "dynamoName": "stringList",
              "type": "List",
              "elementType": "String"
            }

The generated setStringList method will set the in-memory state of the *Updates object, but the list is not actually persisted since only the list adds are used when generating the Dynamo DB update expression. On the other hand, setting deltas: false will cause values passed to setStringList to be persisted. It may be less confusing if setStringList is not generated at all in the case of deltas: true.

Invalid code generation when `equalsFields` includes erroneous fields

Ran into this just now: have a schema that lists specific equalsFields and I dropped one of those fields from the core schema.

Expected behavior: Schema validation error, invalid equalsFields.
Actual behavior: Code generation succeeds and uses the invalid fields listed in equalsFields leading to invalid Java code.

First read fails after migration when using optimistic locking

When using optimistic locking, the bean return by the migration does not include the updated revision number.
If you then try to do an update using an updates object created from this bean then it will fail due to condition check.

Create tables with on demand provisioning

Dynamap will create tables in DynamoDB if they don't yet exist. It creates them with explicit provisioning and 1 read + write IOP. Better to just create the table with on demand provisioning... that's likely to be how the table stays for dev and live environments.