awspilot / dynamodb-oop Goto Github PK

Hi,

I am my config.json contains region. I am using

// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
// Load credentials and set region from JSON file
AWS.config.loadFromPath('config/config.json');

//AWS.config.update({region: 'us-east-1'});

var $db = new AWS.DynamoDB();
var DynamoDB = require('@awspilot/dynamodb')($db);

But I still get
ConfigError: Missing region in config error .

Config.json looks like
{ "accessKeyId": "XXXXXXXXXX",
"secretAccessKey": "##############",
"region": "us-east-1"
}

Instead of the entire aws-sdk library, we should be able to...

const { DynamoDB } = require("@aws-sdk/client-dynamodb")
const $db = new DynamoDB({ region: 'us-west-2' })
const DynamodbFactory = require('@awspilot/dynamodb')
const dynamodb = new DynamodbFactory($db)

Trying to get an item, it's missing the string set field.

I'm trying to perform a recursive scan, following the documentation at https://awspilot.github.io/dynamodb-oop/pages/scan/ (tab N°2)

	// continous scan until end of table
	(function recursive_call( $lastKey ) {
	    DynamoDB
	        .table('messages')
	        .resume($lastKey)
	        .scan(function( err, data ) {
	            // handle error, process data ...

	            if (this.LastEvaluatedKey === null) {
	                // reached end, do a callback() maybe
	                return;
	            }

	            var $this = this
	            setTimeout(function() {
	                recursive_call($this.LastEvaluatedKey);
	            },1000);

	        })
	})(null);

Why is there a 1sec timeout? That's huge, what does it do really?
How does this example returns the content of the table? Calls are recursive but I don't see any variable holding the dataset nor updating it. I don't understand how it works.

Could the doc provide a better example with an actual use-case? Like recursive scan a table then display the results, using a reusable function, for instance?

Firstly, well done, nice module.

This is not really an issue, just a doco comment that you might consider adding to save someone a bit of time that is new to AWS DynamoDB (maybe why they are using your module).

Basically I was trying to use a value in the data object returned from insert_or_update to do another operation and it took me a while to figure out that the insert_or_replace returned the old changed data!

After reading the AWS doco section on return values I finally understood that DynamoDB returns the old values if attributes change.

http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html
ALL_OLD - If PutItem overwrote an attribute name-value pair, then the content of the old item is returned.

Anyway, I was thinking a comment, perhaps against the console.log message saying something along the lines of, the data object contains the old data if attributes change.

E.g.

DynamoDB
    .table('users')
    ...
    .insert_or_update({
    ...
    }, function( err, data ) {
        console.log( err, data ); // data object contains the old data if attributes change.
    })

Again, cheers for the module.

Do your lib support these functions for set ?

For example: How to pass params to function for these ? I tried to use DynamoDB.SS(['a', 'b']) but get Exception:

[ValidationException: One or more parameter values were invalid: Invalid number of argument(s) for the IN ComparisonOperator]

I have a table feedback with a rating field, which is a number.

I'd like to get the average value of this field, is there any simple way of doing it? So far I only see a query which gets all the rows and do the average calculation once they've all been retrieved.

Hi
I am trying to use the batch_insert query using the lip but it doesn't seem to be working.
-Randeep

If I use the boolean property "false", the property is presented as "undefined" in the database.
What I have done wrong, or is this a bug?

Thanks :-)

I get this weird error:

../node_modules/aws-sdk/lib/request.js:31
            throw err;
            ^

TypeError: Cannot read property 'hasOwnProperty' of undefined
    at Object.DynamoUtil.normalizeItem (../node_modules/aws-dynamodb/lib/util.js:103:18)
    at Request.<anonymous> (../node_modules/aws-dynamodb/lib/dynamodb.js:477:60)
    at Request.<anonymous> (../node_modules/aws-dynamodb/lib/dynamodb.js:279:20)
    at Response.<anonymous> (../node_modules/aws-dynamodb/lib/dynamodb.js:274:13)
    at Request.<anonymous> (../node_modules/aws-sdk/lib/request.js:355:18)
    at Request.callListeners (../node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (../node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (../node_modules/aws-sdk/lib/request.js:668:14)
    at Request.transition (../node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (../node_modules/aws-sdk/lib/state_machine.js:14:12)

Code where it happens:

let token = 'x';
let amount = 1;
dynamodb.table('project').where('token').eq(token).index('token').update({
	amount: dynamodb.add(amount)
}

No idea why this happens, even updating another column gives this error.

Could you please implement backoff and retry capbabilites per http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ErrorHandling.html#APIRetries

If it already exists, could you information about it to the documentation?

The ASIN is also the main key of the table.

update or insert data:
{"ASIN":"B00CIDQ470","date_set_at":"2016-11-23T22:03:53.100Z","lowestPrice":51.59}

object after update:

{ date_set_at: '2016-11-23T22:03:53.100Z', lowestPrice: 51.59 }

I've solved it by cloning the data object and using the clone instead, but I dont think it should be modifying my object.

function update_or_insert_item(key, value, data, callback) {
  var tmp_data = _.clone(data) // it's removing ASIN if i dont clone it
  console.log('update or insert data: %j', tmp_data)
  DynamoDB
    .table(dynamoTable)
    .where(key).eq(value)
    .return(DynamoDB.NONE)
    .insert_or_update(tmp_data, function (err, res) {
      if (err) {
        console.log(err, res)
      }
      //console.log( 'update result:', err, res )
      //console.log(this.ConsumedCapacity)
      return callback(err, res)
    })
}

I've tried using the if function to implement a conditional write on a version property, but it doesn't seem to be working. I see from the implementation you're using the deprecated Expected attribute, are planning to move to the ConditionExpression instead?

How does this work? I couldn't find an example how to get the consumed capacity to return.

Is there any documentation on how to satisfy/turn off the new schema warning present in latest updates?

⚠️ [AWSPILOT] missing schema for table ...

Thanks!

Hi I believe its part of a the recent update, of all my dynamodB requests, I see one last object to be completely empty.
It's happening with the scan operation.

DynamoDB.table('TableName')
.scan (function (err, data) {
if (err) {
cb(err, null);
}
cb(null, data);
});

I use Lambda Layer arn:aws:lambda:ap-northeast-1:452980636694:layer:awspilot-dynamodb-2_0_0-beta:1.

I have problem while transaction on the layer.
Example followings:

const DynamoDB = require('@awspilot/dynamodb');

exports.handler = async (event) => {
    const db = DynamoDB({});
    db.schema({
      TableName: "my-table",
      KeySchema: [{ KeyType: 'HASH', AttributeName: 'id' }],
    });
    const tran = db.transact();
    const table = tran.table("my-table");
    console.log(table);
    console.log(table.insert);

    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

The outputs were followings:

INFO	Transact {
  pending: { tableName: 'my-table' },
  TransactItems: [],
  // Omited....
}
INFO	undefined

insert method undefineded.

I have unittest code on local.
The local environment using npm package.
I added logging to unittest code samely.

The local environment outputs were folliowings:

  console.log
    Transact {
      pending: { tableName: 'my-table' },
      TransactItems: [],
      // Omited....
    }

  console.log
    insert [Function]

In this case insert is function and callable.

Can I use transact.insert on Lambda Layer ?

===

• Node version: 12

When I stringify my data and pass it into the insert function, the data is inserted with \\ escapes everywhere. Here is my call:

FYI This is fake data.

data = {"data":{"records":[{"id":"caa6b906e3535a5b29d3db2cb5cce6","first_name":"Myhanh","first_pronounced":"","middle_name":"L","last_name":"May","name":"Myhanh L May","language":"English","phone":[{"number":"8123691112","resource_uri":"/v2/records/8123691112/"}],"address":[{"city":"Topeka","extended_zip":"1918","country":"USA","line1":"2510 Se Burton St","line2":"","state":"KS","zip_code":"66605"}],"relatives":[],"email":"[email protected]","line_type":"Landline","carrier":"At&t","linked_emails":["[email protected]","[email protected]"],"social_links":[],"dob":"","age":"45-54","education":"Completed College","gender":"Female","high_net_worth":"","home_owner_status":"Own","household_income":"100k-125k","length_of_residence":"11-15 years","marital_status":"Married","market_value":"150k-200k","occupation":"","presence_of_children":"No","department":"not specified","telco_zip":"66607","telco_zip_4":"1734","resource_uri":"/v2/users/caa6b906e3535a5b29d3db2cb5cce6/"}]}}

 DynamoDB
                    .table('numbers')
                    .insert({
                        phoneNumber: number,
                        data: cleanData    //Data gets inserted with tons of escapes

                    }, function(err,data) {
                        console.log( err, data );
                        if(err){
                            console.log("Error on DB Save");

                        }

                        res.statusCode = 200;
                        res.setHeader('Content-Type', 'application/json');
                        res.send(response);
                        next();

                    });

Here is what a table entry looks like:

{\"data\":{\"records\":[{\"id\":\"e91ed2a024d779ea6b05c2fc54740e\",\"first_name\":\"Dawn\",\"first_pronounced\":\"DAWN\",\"middle_name\":\"\",\"last_name\":\"Rodenbaugh\",\"name\":\"Dawn Rodenbaugh\",\"language\":\"English\",\"phone\":[{\"number\":\"8123691111\",\"resource_uri\":\"/v2/records/8123691111/\"}],\"address\":[{\"city\":\"Holiday\",\"extended_zip\":\"\",\"country\":\"USA\",\"line1\":\"2642 Ochatilla Rd\",\"line2\":\"\",\"state\":\"FL\",\"zip_code\":\"34691\"}],\"relatives\":[],\"email\":\"[email protected]\",\"line_type\":\"Landline\",\"carrier\":\"Verizon\",\"linked_emails\":[\"[email protected]\"],\"social_links\":[],\"dob\":\"\",\"age\":\"\",\"education\":\"\",\"gender\":\"Female\",\"high_net_worth\":\"\",\"home_owner_status\":\"\",\"household_income\":\"\",\"length_of_residence\":\"\",\"marital_status\":\"\",\"market_value\":\"\",\"occupation\":\"\",\"presence_of_children\":\"\",\"department\":\"not specified\",\"telco_zip\":\"33602\",\"telco_zip_4\":\"1702\",\"resource_uri\":\"/v2/users/e91ed2a024d779ea6b05c2fc54740e/\"}]}}

I wonder if this lib can perform operation such as UpdateTable: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#updateTable-property

I couldn't find such method in the doc http://awspilot.github.io/dynamodb-oop/

Hello,

I am able to connect to dynamoDB, get list of ducments from database. I could update string values in documents as well. If details is string it is working fine.

          DynamoDB
                    .table('user')
                    .where('id').eq(customer.id)
                    .update({
                        "EmailAddress": customer.EmailAddress,
                        "data": details
                    }, function( err, data ) {
                        console.log("555 "+ err )
                        console.log( data )
                });

But How can I update nested JSON, using this "aws-dynamodb" npm module, when details is nested json.

Please let me know.

Thanks,
Mayuri Zingade

How would I do it? Does this support that?

any plans to create wrapper for DynamoDB.DocumentClient?

Hi
When I try to perform the query by using .select(DynamoDB.ALL).
It raises exception
{ [InvalidParameterType: Expected params.AttributesToGet[0] to be a string]
message: 'Expected params.AttributesToGet[0] to be a string',
code: 'InvalidParameterType',
time: Tue Jun 07 2016 23:30:16 GMT-0700 (PDT) }

This would be a big help to my app as Dynamo Transactions opens up a lot of use cases.

... that this is by far THE BEST EVER DynamoDB wrapper I've worked with. Thanks a million for your efforts.
I know there is still a lot of work to be done on the same, but, the API coverage and the ease of development after using this should be appreciated.

Thanks...

I think right now, this module only supports arrays, but I'd like to add unique values only, I think using a String Set is the way that is done.

Can you consider adding that feature.

For now, I modified the add prototype for my testing.

    DynamoDB.prototype.addSS = function(data) {
        if (Array.isArray(data)) {
            var to_ret = {'SS': [] }
            for (var i in data) {
                to_ret.SS[i] = data[i]
            }
            return new DynamoDB.Raw({
                Action: 'ADD',
                Value: to_ret
            })
        }
    }

As per documentation, _the recommended use-case for npm-shrinkwrap.json is applications deployed through the publishing process on the registry: for example, daemons and command-line tools intended as global installs or devDependencies.

It's strongly discouraged for library authors to publish this file, since that would prevent end users from having control over transitive dependency updates._

Please remove the file from npm package to prevent such issues in e.g. webpack projects.

Hello
Could you please add this feature
dynamoDb .table("table") .select( ["attribute1", "attribute2", ...] )

Thanks

async function ScanDynamodbTable(lastKey) {
	return await DynamoDB.table('mytable')
		.resume(lastKey)
		.limit(1)
		.scan()
}

Returns a single array object. But if I wanted to resume scanning, it comes from the this.LastEvaluatedKey which can only be obtained from within scan.

Basically, i want to create a do while loop with async/await, because recursion with promises has a memory leak, and I can't seem to get the LastEvaluatedKey.

Having a look through the source for routeCall it looks like this package can't be used with promises (by not passing a callback and calling .promise() on the AWS SDK service call result). Which is a shame given I love the resulting 'language' this package creates on top of DynamoDB. That callbacks are used internally in dynamodb.js might make supporting promises and callbacks a bit of a refactor. Are there any other reasons promises aren't supported?

Docs scroll to top when I highlight something in Ace editor and click. New version of Ace fix this issue. I test with Mac+Chrome.

It seems that only "AND" is supported right now. Is it the case?

Hi,

I'm afraid that new version of your library api is not backward compatible with 0.1.33 version.
For instance in following code

var command = dynamodb.table(tablePrefix + table);
    for (var key in query) {
        command = command.where(key).eq(query[key]);
    }
    command.get(callback);

in callback function I used following:

function (result) {
        return result[0];
    }

after update to version 0.1.44 I had to use return result[0][0] because additional layer in result array appeared.

Your code looks great.

I'm not sure if you've seen the AWS API Gateway, Lambda implementations, using DynamoDB and node.js.

The way they're doing the update process is painful. My sense is they could use some of your insight.

Can I add your library to Lambda?

Let me know.

Thanks

Joe (Creatively.Me)

I using the library and working good. In this point I found this error. I resolved this error using the native aws-sdk and works well.

##Error

{
  "message": "Value provided in ExpressionAttributeValues unused in expressions: keys: {:create_at_2_v1, :create_at_1_v1}",
  "code": "ValidationException",
  "time": "2016-11-23T18:06:22.315Z",
  "requestId": "CVAO4IP0SKPNVCULVHTTS7DC8RVV4KQNSO5AEMVJF66Q9ASUAAJG",
  "statusCode": 400,
  "retryable": false,
  "retryDelay": 0
}

##Example

//...
    function (init, end) {
        const def = new Defer();
        const initDate = new Date(init);
        const endDate = new Date(end);

        console.log(+initDate, +endDate); // log for test
        
        dynamodb
          .table('table')
          .having('key1').null()
          .having('create_at').between(+initDate, +endDate)
          .scan(function(err, data) {
              if (err) return def.reject(err);
              def.resolve(data);
          });

        return def.promise;
    }
//...