minio / thumbnailer Goto Github PK

A thumbnail generator example using Minio's listenBucketNotification API

License: Apache License 2.0

JavaScript 100.00%

minio lambda s3-bucket lambda-functions image-processing image-manipulation

thumbnailer's Introduction

Thumbnailer

A thumbnail generator example using Minio's listenBucketNotification API. This is a simple example of how one can build a AWS Lambda like functionality on a laptop.

Dependencies

Dependencies are managed by npm use npm install.

npm install

Configure

Please edit config/development.json with your local parameters, currently the example points to https://play.minio.io:9000

This example works only with Minio server using an extended API

Run

Once configured proceed to run.

node thumbnail.js
Listening for events on "images"

Now upload an image using mc

mc cp ./toposort/graph.jpg play/images/
./toposort/graph.jpg:  34.29 KB / 34.29 KB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 100.00% 386.91 KB/s 0s

You will see thumbnail being generated after uploading the image using mc.

node thumbnail.js
Listening for events on "images"
Uploading new thumbail to "images-processed"
Successfully generated "graph-thumbnail.jpg" with md5sum "ca78ee1cc48358b4dbd883a589523e54"

To validate if the thumbnail was created at destination bucket use mc.

mc ls play/images-processed
[2017-01-22 23:44:51 PST]   629B graph-thumbnail.jpg

thumbnailer's People

Contributors

Stargazers

Watchers

Forkers

brunni bhanditz kanagarajkm poornas kannappanr sebalvarez97 h2atecnologia kiina ranger-x isabella232 champomix emg110 knightley sree1026 giuseppem92 saber13812002

thumbnailer's Issues

Does webhook waiting for thumbnail to complete ?

I know webhook is triggered after file uploaded. But i just wonder does upload response waiting webhook for thumbnail has resized or just responds initially when upload completed.

S3Error: The specified key does not exist is thrown for objects with prefix

For example,
When a new object is uploaded to mybucket/prefix/test.jpg, getObject is failing with S3Error: The specified key does not exist. This is because the object name is url encoded in the notification event. It should be decoded before doing getObject.

As per the aws documentation (http://docs.amazonaws.cn/en_us/AmazonS3/latest/dev/notification-content-structure.html), object keyname should be url encoded. So that, prefix/test.jpg becomes prefix%2Ftest.jpg and test 1.jpg becomes test+1.jpg

listenBucketNotification wrong key encoding

The event from the listBucketNotification contains the key of the changed object.
In this case the key is test12/DG0890EY2.jpg but provided as test12%2FDG0890EY2.jpg

This leads to wrong behavior also with some example code, where the key is directly used to call "getObject" method.

e.g. in the Thumbnailer Example:
https://github.com/minio/thumbnailer/blob/master/thumbnail.js
where a slash leads to " S3Error: The specified key does not exist."

Event provided from notification:

  eventSource: 'minio:s3',
  awsRegion: '',
  eventTime: '2018-10-26T11:18:32Z',
  eventName: 's3:ObjectCreated:Put',
  userIdentity: { principalId: 'SV8EUTD78ZV13T1PP8FM' },
  requestParameters: { sourceIPAddress: '127.0.0.1' },
  responseElements:
   { 'x-amz-request-id': '15612487A9DF732C',
     'x-minio-origin-endpoint': 'https://10.0.75.1:9000' },
  s3:
   { s3SchemaVersion: '1.0',
     configurationId: 'Config',
     bucket:
      { name: 'images',
        ownerIdentity: [Object],
        arn: 'arn:aws:s3:::images' },
     object:
      { key: 'test12%2FDG0890EY2.jpg',
        size: 629648,
        eTag: 'b7ce786cd89b182bfd5ae765557bbc37',
        contentType: 'image/jpeg',
        userMetadata: [Object],
        versionId: '1',
        sequencer: '15612487A9DF732C' } },
  source:
   { host: '',
     port: '',
     userAgent: 'aws-sdk-dotnet-45/3.3.25.0 aws-sdk-dotnet-core/3.3.27.1 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_
Windows_NT_10.0.15063.0 ClientAsync' } }

Unexpected token R

Got this from thumbnailer-webhook.js when trying to start minio server with webhook enabled:

SyntaxError: Unexpected token R
    at parse (/path/to/thumbnailer/node_modules/body-parser/lib/types/json.js:83:15)
    at /path/to/thumbnailer/node_modules/body-parser/lib/read.js:116:18
    at invokeCallback (/path/to/thumbnailer/node_modules/raw-body/index.js:262:16)
    at done (/path/to/thumbnailer/node_modules/raw-body/index.js:251:7)
    at IncomingMessage.onEnd (/path/to/thumbnailer/node_modules/raw-body/index.js:307:7)
    at emitNone (events.js:105:13)
    at IncomingMessage.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1059:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Minio server config is no longer at ~/.minio/config.json

After running NODE_ENV=webhook node thumbnail-webhook.js per this tutorial, the output says:

Please update minio server config ~/.minio/config.json to enable webhook notification target.

However, the Minio server configuration file is no longer stored at ~/.minio/config, as explained here.

Could the instructions in the output be modified to account for the newer way that configuration is handled in Minio?

ERR_STREAM_WRITE_AFTER_END

When I submit a second jpg file I get the following error :

  $ node thumbnail.js
  Listening for events on "images"
  Uploading new thumbail to "images-processed"
  Successfully uploaded "16203966045443147065447901540390-thumbnail.jpg" w                                                                                                                                                                                                                                                     ith md5sum "bed3873882f1697677c8fcc4176aafa4"
  Uploading new thumbail to "images-processed"
  Error [ERR_STREAM_WRITE_AFTER_END] [ERR_STREAM_WRITE_AFTER_END]: write a                                                                                                                                                                                                                                                     fter end
      at writeAfterEnd (_stream_writable.js:266:14)
      at Sharp.Writable.write (_stream_writable.js:315:5)
      at IncomingMessage.ondata (_stream_readable.js:718:22)
      at IncomingMessage.emit (events.js:314:20)
      at IncomingMessage.Readable.read (_stream_readable.js:507:10)
      at flow (_stream_readable.js:1007:34)
      at resume_ (_stream_readable.js:988:3)
      at processTicksAndRejections (internal/process/task_queues.js:84:21)

npm depedencies :

├─┬ [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ └─┬ [email protected]
│   ├── [email protected]
│   └── [email protected] deduped
├─┬ [email protected]
│ └─┬ [email protected]
│   └── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ └── [email protected]
├─┬ [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │   ├── [email protected]
│ │   ├── [email protected] deduped
│ │   ├── [email protected]
│ │   ├── [email protected]
│ │   ├── [email protected] deduped
│ │   ├─┬ [email protected]
│ │   │ └── [email protected] deduped
│ │   └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ └── [email protected]
│ │ └── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ └─┬ [email protected]
│   ├── [email protected]
│   └── [email protected]
├── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ └─┬ [email protected]
│ │   ├── [email protected] deduped
│ │   └─┬ [email protected]
│ │     └── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ └── [email protected] deduped
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected]
│ │ │ │ ├─┬ [email protected]
│ │ │ │ │ └── [email protected]
│ │ │ │ └── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected]
│ │ │ └─┬ [email protected]
│ │ │   └── [email protected] deduped
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │   ├─┬ [email protected]
│ │ │   │ ├─┬ [email protected]
│ │ │   │ │ ├── [email protected]
│ │ │   │ │ └── [email protected]
│ │ │   │ ├── [email protected]
│ │ │   │ └─┬ [email protected]
│ │ │   │   ├── [email protected] deduped
│ │ │   │   ├─┬ [email protected]
│ │ │   │   │ └── [email protected]
│ │ │   │   └── [email protected] deduped
│ │ │   ├── [email protected] deduped
│ │ │   ├── [email protected]
│ │ │   ├── [email protected] deduped
│ │ │   └─┬ [email protected]
│ │ │     ├── [email protected] deduped
│ │ │     ├─┬ [email protected]
│ │ │     │ └── [email protected]
│ │ │     └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ └── [email protected]
│ └─┬ [email protected]
│   └── [email protected]
└── [email protected]

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.