aws-samples / aws-lambda-fanout Goto Github PK

Thank you for making this very useful feature. Could include all of the available commands--hook, unhook, pause, resume--in "The CLI" section?

TL;DR: When I was setting it up, I scratched my head for a while about why nothing was happening and, after some investigation, it turned out that, although I had registered a stream to stream mapping, I had not hooked the source stream up to the fanout lambda. The reason I hadn't was a failure to scroll to the very end of the readme, where the hook command was described for the first time. It would have helped me to have the need to hook the stream up mentioned somewhere else nearer the top of the readme, e.g., if it were included in the list of "Available commands" in "The CLI" section. Thanks again for the feature.

I'm trying to setup a mapping that works with cross-account support. When I register the mapping and try to use the --destination-role-arn, the response is the following:

Invalid ARN 'arn:aws:iam::012345678910:role/fanoutCrossAccountRole', must be a fully qualified AWS IAM Role ARN

The format I'm using for the register action is as follows:

./fanout register kinesis --function fanout --source-type kinesis --source source-stream --id target1 --destination arn:aws:kinesis:us-west-2:012345678910:stream/target-stream --destination-region us-west-2 --destination-role-arn arn:aws:iam::012345678910:role/fanoutCrossAccountRole --active true --profile redacted --region us-west-2

However, the following does work including the --destination ARN, but it does not provide cross-account support because it's missing the --destination-role-arn, and is really just to prove that the ARN format is correct:

./fanout register kinesis --function fanout --source-type kinesis --source source-stream --id target2 --destination arn:aws:kinesis:us-west-2:012345678910:stream/target-stream --destination-region us-west-2 --active true --profile redacted --region us-west-2

Is the --destination-role-arn feature for cross-account support functional? If so, what am I missing?

When fetching from a Kinesis stream and pushing to Redis (AWS Elasticache), I get the following error in CloudWatch:

Error occured while preparing Redis command: [TypeError: list argument must be an Array of Buffers.]

Not being familiar with Redis does some preparation need to be done before writing data?

Hi All,

I am getting below error while using the hook command to register the event source mapping for cross Aws Account:

An error occurred (InvalidParameterValueException) when calling the CreateEventSourceMapping operation: Role and event source must be in the same account as the cloud function.

I have a kinesis stream in account A and I have a Lambda function in account B. I am trying to register the kinesis stream in account A to Lambda function in account B.

Your help will be really appreciated.

Thanks

Hi,

When configuring a mapping for a DynamoDB Stream, the lambda generates the following error:

Invalid configuration for target 'target1' of type 'kinesis', 'sourceArn' property must be a valid ARN of an Amazon Kinesis Stream or Amazon DynamoDB Stream. Ignoring target

I think that the problem is related to an erroneous regex in configuration.js file. Current value of regex is:

var sourceRegex = /^arn:aws:((kinesis:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:stream/([a-zA-Z0-9-]{1,128}))|(kinesis:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:table/[A-Za-z0-9_.-]/stream/[0-9A-Z_.:-]+))$/;_

I've replaced it with the following regex and it has worked:

var sourceRegex = /^arn:aws:((kinesis:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:stream/([a-zA-Z0-9-]{1,128}))|(dynamodb:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:table/[A-Za-z0-9_.-]+/stream/[0-9A-Z_.:-]+))$/;_

KR,

Juan Carlos González

Should be replaced with nodejs10.x

Hi,

I'm seeing the following error in my kinesis fanout handler deployment:

'Record too large to be pushed to target 'reconnectToDEV' of type 'kinesis''

The system is setup to take a single record from a kinesis stream (batch size == 1) and push it to a different kinesis stream.

So the question is, how is a single record from a kinesis stream too big to put into a different kinesis stream?

I suspect there is an issue with how the fanout handler is calculating the max allowed size from this line:
https://github.com/aws-samples/aws-lambda-fanout/blob/master/fanout.js#L63

AWS Support suggested that I open up a github issue, ticket #6553563951

Thank you,

Joe Bowen

I am getting the below deprecation warning/error while running the code from AWS Lambda with Node.js 10.x runtime. Much appreciated for suggestions

ERROR (node:8) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

Hi,

I've configured fanout Lambda for replicating DynamoDB Stream events to a Kinesis Stream. When I make changes to DynamoDb table, events appear in Kinesis Stream. However it looks like the events are not complete as some attributes are missing: eventName, sequenceNumber, etc. It looks like the only attributes available are those contained in NewImage attribute.

KR,

Juan Carlos González

Very nice feature!

I just want to let you know that I think you should add the --table parameter (as optional) under the Destroy in README. because you need to specify this if you have specified a custom --table in Deploy.

Update: Should probably solve the "--table is not supported " issue first.

• #23

I have a Kinesis Stream in one region connected to a Kinesis Stream in another region via this project. It appears that the records in the destination stream have been concatenated together, resulting in records that are too large for my final processing to work with. I can't figure out how this is happening though or how to stop it. Any ideas?

It looks like the current sourceRegex does not support periods in Kinesis Stream names like arn:aws:kinesis:us-east-1:123456789012:stream/name.with.periods. The Kinesis Streams Service API Reference suggests the pattern [a-zA-Z0-9_.-]+ for StreamName.

Adding the missing period to sourceRegex seems to fix the problem.

When creating a lambda with a specific table name, then issuing all needed command with --table-arn leads to a non-working lambda looking for table <function name>Targets instead of the specified table arn/name. The specified table name is created in Dynamo, but lambda will look for <function name>Targets.
Unregistering such a lambda gives an error message saying that <function name>Targets table doesn't exist, while a valid table arn was issued.

Looking at code, it looks like table name and table arn are used by shell scripts to issue the right aws-cli commands, so we have the right names for created lambda, created dynamo table, source, etc. The shell scripts sets TABLE_NAME and TABLE_ARN variables correctly, but no trace of them in lambda js code.

Hi,

we're using the fanout function to push Kinesis Streams' data from one AWS account to another. Looking at the Streams' data throughput in both accounts, it seems to be working correctly. However, when we enabled a Cloud Watch alarm to capture the function's invocation errors, here's what we get periodically:

This happens several times a day and seems as a temporary glitch, but we haven't yet had the opportunity to validate if there is any data loss occurring in the process.

Any information you could provide would be appreciated.

Thanks,

Hi,

First of all, thanks for this great work. I've successfully executed the deploy command with latest aws cli release. However, when I execute the following command:

./fanout register kinesis --function fanout --source-type dynamodb --source zeus-auth --id target1 --destination fanoutTest --active true

the following error is shown:

Error parsing parameter '--item': Invalid JSON: No JSON object could be decoded
JSON received: {"sourceArn": {"S":"arn:aws:dynamodb:us-east-1:292326452532:table/zeus-auth/stream/2016-09-14T15:57:43.105"}, "id": {"S":"target1"}, "type": {"S":"kinesis"}, "destination": {"S":"fanoutTest"}, "active": {"BOOL":none}}

The full stacktrace, obtained enabling debug, is:

2016-09-14 18:08:43,284 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dynamodb.put-item.table-name: calling handler <function uri_param at 0x102cb0e60>
2016-09-14 18:08:43,285 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.dynamodb.put-item: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10341cc10>
2016-09-14 18:08:43,285 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'fanoutTargets' for parameter "table_name": u'fanoutTargets'
2016-09-14 18:08:43,285 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dynamodb.put-item.item: calling handler <function uri_param at 0x102cb0e60>
2016-09-14 18:08:43,285 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.dynamodb.put-item: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10341cc10>
2016-09-14 18:08:43,285 - MainThread - awscli.argprocess - DEBUG - Param item looks like JSON, not considered for param shorthand.
2016-09-14 18:08:43,285 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/clidriver.py", line 186, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/clidriver.py", line 381, in call
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/clidriver.py", line 522, in call
parsed_args, self.arg_table)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/clidriver.py", line 578, in _build_call_parameters
arg_object.add_to_params(service_params, value)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/arguments.py", line 453, in add_to_params
unpacked = self._unpack_argument(value)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/arguments.py", line 470, in _unpack_argument
return unpack_cli_arg(self, value)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/argprocess.py", line 166, in unpack_cli_arg
cli_argument.cli_name)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/argprocess.py", line 175, in _unpack_cli_arg
argument_model, value, cli_name)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/argprocess.py", line 189, in _unpack_complex_cli_arg
% (e, value))
ParamError: Error parsing parameter '--item': Invalid JSON: No JSON object could be decoded
JSON received: {"sourceArn": {"S":"arn:aws:dynamodb:us-east-1:292326452532:table/zeus-auth/stream/2016-09-14T15:57:43.105"}, "id": {"S":"zeus-auth-target"}, "type": {"S":"kinesis"}, "destination": {"S":"fanoutTest"}, "active": {"BOOL":none}}
2016-09-14 18:08:43,329 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Please, could you give me any hint?.

KR,
Juan Carlos González

If you're fanning out to a kinesis stream for example and you get back a FailedRecordCount > 0, no error is raised up from the lambda, so only a partial batch is successfully fanned out and the rest are lost. The data element should be being looked at in the callback and this should be handled in some way

The number of consuming streams (fanout/destination) streams seems to be limited by the parallelTargets parameter defined here: https://github.com/aws-samples/aws-lambda-fanout/blob/master/fanout.js#L34

If this parameter is 2, the data flows into two of the destination streams, if it is increased to 5 we observed that the data will flow into 5 destination streams.

Is this expected behavior? If yes, should this be documented/parameterized?

Javascript error when pushing data to Memcached.

Hi,

The sourceRegex pattern in configuration.js does not work for source dynamodb table with name length greater than 1. This works fine when i change the pattern from:
var sourceRegex = /^arn:aws:((kinesis:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:stream\/([a-zA-Z0-9_-]{1,128}))|(dynamodb:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:table\/[A-Za-z0-9_.-]\/stream\/[0-9A-Z_.:-]+))$/;
to:
var sourceRegex = /^arn:aws:((kinesis:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:stream\/([a-zA-Z0-9_-]{1,128}))|(dynamodb:[a-z]+-[a-z]+-[0-9]:[0-9]{12}:table\/[A-Za-z0-9_.-]+\/stream\/[0-9A-Z_.:-]+))$/;

Thanks,
Abhishek

What is the right way to invoke the lambda? Should it be a trigger from each input stream?