mapbox / s3touch Goto Github PK

Trigger S3 event notifications for objects in an s3 bucket

License: ISC License

JavaScript 100.00%

s3touch's Introduction

🌇 As of January 2020, this module is deprecated. It is provided as-is, with no warranty. We are not accepting new bugfixes or feature requests at this time.

s3touch

Utility to simulate S3 events without actually re-PUTing objects to S3.

Usage: s3touch <s3 path> [--topic <arn string>]

# touch a single object
s3touch s3://my-bucket/some-object

# touch all objects under a prefix
s3touch --recursive s3://my-bucket/some-prefix/

# touch a single object with custom SNS topic
s3touch s3://my-bucket/some-object --topic arn:aws:sns:us-east-1:1234:this-bucket-s3-events

Looks up SNS topic that is subscribed to the given S3 bucket,
Looks up information about some-object (size, etag),
and sends a simulated S3 event message to the SNS topic already set within the bucket or to the SNS topic explictly sent via the --topic flag

Install

npm install -g s3touch

s3touch's People

Contributors

Stargazers

Watchers

Forkers

jpalach bbc 3b00d iflix belalh malekmakan testbigorg isabella232 mapclone

s3touch's Issues

add documentation on how to set up SNS topics?

seems like something that has to happen before this tool can be used?

Tests don't run - access denied

Currently getting access denied (403) errors when I try to run the tests. Is there a way we can run tests locally?

Does not work in eu-west-2

When running s3touch in the new London region I get the following error

InvalidRequest: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

This appears to be caused by using a legacy authentication method which is only supported in S3 regions that were launched pre 2014.

http://stackoverflow.com/questions/26533245/the-authorization-mechanism-you-have-provided-is-not-supported-please-use-aws4

Allow subject to be passed in.

Currently s3Touch allows you to simulate an s3 object event through SNS, but it doesn't use a lot of the features of SNS. I imagine this is because you want to simulate an S3 event rather than an SNS event. However, I'd like to be able to pass a lambda command string in via the subject line. Would it be okay to add some parameters in the createMessage/publishEvent functions to add a subject line to the SNS event? Default could be "s3touch".

enable bucket region lookup in createMessage()

I've run into the problem of wanting to s3touch an item in an s3bucket located in us-east-1 & send work message to an SNS topic that is in us-west-2. I've set env var AWS_DEFAULT_REGION=us-west-2 to override default region var in s3touch so that we have proper access to the SNS topic in us-west-2.

However, this region var also gets used in the createMessage() function to relay bucket & object information in the headObject request. This means, if default region is set to us-west-2, we are unable to access objects in us-east-1 and receive Error: Could not HEAD object ("301")

In order to change this behavior, I'm going to work on updating the createMessage() function to lookup region for object request based on the region its parent bucket is housed in.

This should be able to be accomplished with an getBucketLocation call

cc @yhahn

Supporting multiple notification rules and prefix/suffix filters

From about 5 minutes of digging it's not clear how s3touch could read up the notification rules and apply prefixes/suffixes.

Will need some more investigation.

cc @willwhite @GretaCB

Question: Can you s3touch an entire bucket

Hi,
Thanks for a great program, really useful.

I wanted to ask if there is an easy way to use it for every object in a bucket and not have to use a prefix?

Thanks
Dave

Error: Cannot read property 'Topic' of undefined

I touch a single file in S3 bucket

s3touch s3://bucket/folder/file.jpg

and get this error:

/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/request.js:30
            throw err;
            ^

TypeError: Cannot read property 'Topic' of undefined
    at Response.<anonymous> (/usr/local/lib/node_modules/s3touch/index.js:34:56)
    at Request.<anonymous> (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/request.js:353:18)
    at Request.callListeners (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/request.js:595:14)
    at Request.transition (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/request.js:21:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/request.js:37:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/s3touch/node_modules/aws-sdk/lib/request.js:597:12)

Thanks in advance.

Doesn't work when there is whitespace in the key

Error: Could not HEAD object ("404")

however aws s3 cli is able to list the file.