Code Monkey home page Code Monkey logo

serverless-s3-deploy's Introduction

serverless-s3-deploy

Plugin for serverless to deploy files to a variety of S3 Buckets

Note: This project is currently not maintained.

Installation

 npm install --save-dev serverless-s3-deploy

Usage

Add to your serverless.yml:

  plugins:
    - serverless-s3-deploy

  custom:
    assets:
      targets:
       - bucket: my-bucket
         files:
          - source: ../assets/
            globs: '**/*.css'
          - source: ../app/
            globs:
              - '**/*.js'
              - '**/*.map'
       - bucket: my-other-bucket
         empty: true
         prefix: subdir
         files:
          - source: ../email-templates/
            globs: '**/*.html'

You can specify any number of targets that you want. Each target has a bucket and a prefix.

bucket is either the name of your S3 bucket or a reference to a CloudFormation resources created in the same serverless configuration file. See below for additional details.

You can specify source relative to the current directory.

Each source has its own list of globs, which can be either a single glob, or a list of globs.

Setting empty to true will delete all files inside the bucket before uploading the new content to S3 bucket. The prefix value is respected and files outside will not be deleted.

Now you can upload all of these assets to your bucket by running:

$ sls s3deploy

If you have defined multiple buckets, you can limit your deployment to a single bucket with the --bucket option:

$ sls s3deploy --bucket my-bucket

ACL

You can optionally specificy an ACL for the files uploaded on a per target basis:

  custom:
    assets:
      targets:
        - bucket: my-bucket
          acl: private
          files:

The default value is private. Options are defined here.

Content Type

The appropriate Content Type for each file will attempt to be determined using mime-types. If one can't be determined, a default fallback of 'application/octet-stream' will be used.

You can override this fallback per-source by setting defaultContentType.

  custom:
    assets:
      targets:
        - bucket: my-bucket
          files:
            - source: html/
              defaultContentType: text/html
              ...

Other Headers

Additional headers can be included per target by providing a headers object.

See http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html for more details.

  custom:
    assets:
      targets:
        - bucket: my-bucket
          files:
            - source: html/
              headers:
                CacheControl: max-age=31104000 # 1 year

Resolving References

A common use case is to create the S3 buckets in the resources section of your serverless configuration and then reference it in your S3 plugin settings:

  custom:
    assets:
      targets:
        - bucket:
            Ref: MyBucket
          files:
            - source: html/

  resources:
    # AWS CloudFormation Template
    Resources:
      MyBucket:
        Type: AWS::S3::Bucket
        Properties:
          AccessControl: PublicRead
          WebsiteConfiguration:
            IndexDocument: index.html
            ErrorDocument: index.html

You can disable the resolving with the following flag:

  custom:
    assets:
      resolveReferences: false

Auto-deploy

If you want s3deploy to run automatically after a deploy, set the auto flag:

  custom:
    assets:
      auto: true

IAM Configuration

You're going to need an IAM policy that supports this deployment. This might be a good starting point:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${bucket}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::${bucket}/*"
            ]
        }
    ]
}

Upload concurrency

If you want to tweak the upload concurrency, change uploadConcurrency config:

config:
  assets:
    # defaults to 3
    uploadConcurrency: 1

Verbosity

Verbosity cloud be enabled using either of these methods:

Configuration:

  custom:
    assets:
      verbose: true

Cli:

  sls s3deploy -v

serverless-s3-deploy's People

Contributors

alexandrusavin avatar aljohri avatar arabold avatar claabs avatar dancrumb avatar evgsil avatar frozenbonito avatar funkybob avatar haydnhkim avatar jonasho avatar jsyrjala avatar marcoreni avatar mikestaub avatar pistazie avatar romeara avatar smiklos avatar trend-kevin-c-chen avatar tsdorsey avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

serverless-s3-deploy's Issues

Document ACL property

There's support for setting the ACL property of files being deployed.

Update the documentation to expose this

Does not work on Windows, backslash vs forward slash problem

When uploading resources, it should have created subdirectories, but it just puts the files in the bucket with the folder in the name. This is probably related to backslash on windows vs forward slash on linux.

This seems to have been fixed on the repository. However, it has not been published yet as a NPM package update. I would need that fix right away. Could someone publish it? Thanks.

globs issue

Using your example configuration I get the following error:

YAMLException: unidentified alias "*/*.css" in "C:\tkbt\webapi\serverless.yml" at line 97, column 28:
                globs: **/*.css
                               ^
    at generateError (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:162:10)
    at throwError (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:168:9)
    at readAlias (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1244:5)
    at composeNode (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1336:20)
    at readBlockMapping (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1057:11)
    at composeNode (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1327:12)
    at readBlockSequence (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:925:5)
    at composeNode (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1326:12)
    at readBlockMapping (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1057:11)
    at composeNode (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1327:12)
    at readBlockSequence (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:925:5)
    at composeNode (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1326:12)
    at readBlockMapping (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1057:11)
    at composeNode (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1327:12)
    at readBlockMapping (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1057:11)
    at composeNode (C:\Users\gemme\AppData\Roaming\npm\node_modules\serverless\node_modules\js-yaml\lib\js-yaml\loader.js:1327:12)

My configuration looks like:

...
plugins:
  - serverless-offline
  - serverless-s3-deploy
  
custom:
    assets:    
      targets:
       - bucket: mybucket
         files:
          - source: ../assets/
            globs: **/*.css

Add eslint support

What do you think about adding eslint support? I think it would have caught that new Promise.resolve() issue... Maybe ๐Ÿ˜ฌ

I'm happy to make a PR for it

0.5.0 on NPM doesn't include the promised updates

Please tell me it's just me and some weird configuration I have locally, but if I check out the latest and greatest 0.5.0 of this repo, it doesn't include my pull request from yesterday. It's should be in there but it seems that the version that was published to NPM is missing the code changes. It only updated the package.json ๐Ÿ˜•

Ref to bucket ignore stage

reference to bucket across stages does not work, it keep reference from first stage.

targets:
  - bucket:
      Ref: DistBucket

not uploading files for serverless versions newer than 1.32.0

I've tried to upgrade from serverless 1.32.0 to the following versions but the assets aren't being uploaded to the S3 bucket:
1.33.2
1.34.1
1.35.1
1.36.3
1.37.1
1.38.0

So 1.32.0 seems to be the latest version of serverless with which this plugin works.
Can anyone confirm that it has been using a version older than serverless 1.32.0 with serverless-s3-deploy successfully?

serverless.yml config

      "assets": {
            "auto": true,
            "targets": [{
                "bucket": "${self:custom.bundlesS3BucketName}",
                "files": [{
                    "source": "./bundles-${self:provider.stage}",
                    "globs": ["**/*.css", "**/*.css.map", "**/*.js", "**/*.js.map", "**/*.png"]
                }]
            }]
        },```

this.provider.request(...).then(...).return is not a function

TypeError: this.provider.request(...).then(...).return is not a function
      at Assets.listStackResources (..../node_modules/serverless-s3-deploy/index.js:86:12)
      at Assets.deployS3 (..../node_modules/serverless-s3-deploy/index.js:143:17)
      at Assets.afterDeploy (..../node_modules/serverless-s3-deploy/index.js:69:19)

With assets: auto: true

Serverless version: 2.25.1

Dependency on 'serverless/lib/plugins/aws/lib/validate' is not in package.json

Version 0.2.1 has a dependency on the .../validate module, but it's not listed in package.json.

As a result, I'm getting:

Error --------------------------------------------------
 
     Cannot find module 'serverless/lib/plugins/aws/lib/validate'
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
 
     Please report this error. We think it might be a bug.
 
  Your Environment Information -----------------------------
     OS:                 darwin
     Node Version:       6.9.1
     Serverless Version: 1.8.0

s3 auto deploy should be done before cloudformation deployment

The auto deploy of this plugin uploads the assets to the S3 bucket AFTER the cloudformation stack has been deployed.
In my use case (and possibly everyone's use case) the upload of the assets should happen BEFORE the cloudformation stack has been deployed.

SOLUTION
Change the hook event used from after:deploy:finalize to another hook event happening before the deployment like for example before:deploy:deploy

MY USE CASE (probably everyone's)
I'm using serverless-s3-deploy to upload some bundles for a SPA.
In my case I'm uploading 30+ bundles and it takes sometimes.
The auto deploy of this plugin happens after the deployment stage of the cloudformation stack. At that point Api Gateway and AWS Lambda are already answering with an index.html file that is pointing to a bundle that is still being uploaded by the serverless-s3-deploy.

LIKELY FIX #19 AS WELL
This would likely fix also this issue #19 since the order of the bundles wouldn't matter anymore since they are deployed all before the cloudformation stack. What do you think @jrencz?

Deployment Error: Access Denied

I am following the guidelines at https://www.serverless.com/blog/react-js-on-aws-lambda. npx sls deploy command creates the Cloudformation stack successfully but the command returns Access Denied error.

(base) bcd07414e341:ServerlessReact mustafa$ npx sls deploy

Deploying serverless-react-boilerplate to stage dev (us-east-1)

> [email protected] build:browser
> webpack --config webpack.browser.config.js --mode production

assets by info 291 KiB [immutable]
  assets by chunk 4.87 KiB (name: main)
    asset main.f73cf1e4.js 4.38 KiB [emitted] [immutable] [minimized] (name: main) 1 related asset
    asset main.4a854c81.css 499 bytes [emitted] [immutable] (name: main) 1 related asset
  asset vendor.ae97a551.js 284 KiB [emitted] [immutable] [minimized] [big] (name: vendor) (id hint: vendor) 2 related assets
  asset runtime.3553f038.js 1.22 KiB [emitted] [immutable] [minimized] (name: runtime) 1 related asset
  asset components.bc92458d.js 424 bytes [emitted] [immutable] [minimized] (name: components) (id hint: components) 1 related asset
assets by path *.json 457 bytes
  asset manifest.json 286 bytes [emitted] [from: public/manifest.json] [copied]
  asset stats.json 171 bytes [emitted]
asset favicon.ico 3.78 KiB [emitted] [from: public/favicon.ico] [copied]
Entrypoint main [big] 291 KiB (488 KiB) = runtime.3553f038.js 1.22 KiB vendor.ae97a551.js 284 KiB components.bc92458d.js 424 bytes main.4a854c81.css 499 bytes main.f73cf1e4.js 4.38 KiB 5 auxiliary assets
orphan modules 9.06 KiB (javascript) 1.83 KiB (runtime) [orphan] 16 modules
runtime modules 3.02 KiB 5 modules
modules by path ./node_modules/core-js/ 503 KiB 479 modules
modules by path ./src/ 5.52 KiB (javascript) 452 bytes (css/mini-extract) 5 modules
modules by path ./node_modules/react/ 6.48 KiB
  ./node_modules/react/index.js 190 bytes [built] [code generated]
  ./node_modules/react/cjs/react.production.min.js 6.3 KiB [built] [code generated]
modules by path ./node_modules/react-dom/ 119 KiB
  ./node_modules/react-dom/index.js 1.33 KiB [built] [code generated]
  ./node_modules/react-dom/cjs/react-dom.production.min.js 118 KiB [built] [code generated]
modules by path ./node_modules/scheduler/ 4.91 KiB
  ./node_modules/scheduler/index.js 198 bytes [built] [code generated]
  ./node_modules/scheduler/cjs/scheduler.production.min.js 4.72 KiB [built] [code generated]
./node_modules/regenerator-runtime/runtime.js 24.3 KiB [built] [code generated]
./node_modules/object-assign/index.js 2.06 KiB [built] [code generated]
webpack 5.72.0 compiled successfully in 2921 ms
Environment: darwin, node 16.15.0, framework 3.14.0 (local), plugin 6.2.1, SDK 4.3.2
Credentials: Local, "default" profile
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Access Denied

2 deprecations found: run 'serverless doctor' for more details

โœ– Stack serverless-react-boilerplate-dev failed to deploy (145s)
Environment: darwin, node 16.15.0, framework 3.14.0 (local), plugin 6.2.1, SDK 4.3.2
Credentials: Local, "default" profile
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Error: Command failed: npx sls s3deploy --stage dev
    at checkExecSyncError (node:child_process:828:11)
    at execSync (node:child_process:899:15)
    at /Users/mustafa/workspace/ServerlessReact/node_modules/cli-progress-footer/lib/private/cli-progress-footer/disable-props.js:86:41
    at handleSubProcessSync (/Users/mustafa/workspace/ServerlessReact/node_modules/cli-progress-footer/lib/private/cli-progress-footer/disable-props.js:32:15)
    at childProcess.execSync (/Users/mustafa/workspace/ServerlessReact/node_modules/cli-progress-footer/lib/private/cli-progress-footer/disable-props.js:85:11)
    at Scripts.execute (/Users/mustafa/workspace/ServerlessReact/node_modules/serverless-plugin-scripts/lib/index.js:113:35)
    at Scripts.runHook (/Users/mustafa/workspace/ServerlessReact/node_modules/serverless-plugin-scripts/lib/index.js:108:12)
    at PluginManager.runHooks (/Users/mustafa/workspace/ServerlessReact/node_modules/serverless/lib/classes/plugin-manager.js:530:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async PluginManager.invoke (/Users/mustafa/workspace/ServerlessReact/node_modules/serverless/lib/classes/plugin-manager.js:564:9)
    at async PluginManager.run (/Users/mustafa/workspace/ServerlessReact/node_modules/serverless/lib/classes/plugin-manager.js:604:7)
    at async Serverless.run (/Users/mustafa/workspace/ServerlessReact/node_modules/serverless/lib/serverless.js:174:5)
    at async /Users/mustafa/workspace/ServerlessReact/node_modules/serverless/scripts/serverless.js:765:9

2 deprecations found: run 'serverless doctor' for more details

Fn::ImportValue does not work

I am trying to Import bucket name:

custom:
  assets:
    auto: true # run automatically after deploy command
    targets:
      - bucket:
          Fn::ImportValue: persistent-provenance-${self:provider.stage}-MachineLearningResourcesBucketName
...

Or use variable that imports bucket name:

custom:
  mlBucketName:
    Fn::ImportValue: persistent-provenance-${self:provider.stage}-MachineLearningResourcesBucketName
  assets:
    auto: true # run automatically after deploy command
    targets:
      - bucket: ${self:custom.mlBucketName}
...

None of the options work. I get the error:

  Error: Invalid bucket name [object Object]
      at Assets.resolveBucket (/.../node_modules/serverless-s3-deploy/index.js:107:31)
      at BbPromise.map.concurrency (/.../node_modules/serverless-s3-deploy/index.js:149:23)

Feature request: Delete bucket files on sls stack remove

If you try to remove serverless stack (cmd: sls remove) with bucket resource and bucket is not empty you will get the error:

An error occurred: YOUR_Bucket_NAME - The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty;

A way to fix it is serverless-plugin-scripts plugin:

  # serverless-plugin-scripts
  scripts:
    hooks:
      'before:remove:remove': aws s3 rm s3://${self:custom.YOUR_Bucket_NAME} --recursive || true

But it would be great if you will add the option that allows to do that.

Thanks.

Feature Request: add parallel/sequence control

Right now the plugin is processing its configuration and it adds each file to the queue handled by serverless AWS provider.
This gives no control over the order of uploaded files

My use case:
I have a project that consists of source code of a single page app. The code is bundled into several css js and font files and one html file.
It is important for the deployment to be consistent to have a way to upload everything but the html file first and only if it succeeded then upload html file.

I see no way to do that just by using configuration of the plugin

I analysed the code and I see that the promise returned by the awsProvider@request is discarded:

this.provider.request('S3', 'putObject', details, this.options.stage, this.options.region);

File uploads could possibly be sequentialised on 2 levels:

  • assets
  • files in each asset

with current syntax. Right now everything runs in parallel (or: as parallel as awsProvider allows it to be)

I'd like to open a discussion about:

  1. whether the plugin should allow sequences

  2. should it

    • process assets and files in sequence
    • process assets in parallel but files should be uploaded sequentially
    • process assets sequentially but files should be uploaded in parallel
    • introduce new syntax to express sequences and keep current syntax for maximum parallelisation

The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

When I tried to deploy the service in af-south-1, ap-east-1, eu-south-1, and me-south-1, there was serverless error "The unspecified location constraint is incompatible for the region specific endpoint this request was sent to" which is coming after the stack is successfully deployed. Is the serverless-s3-deploy plugin has been configure in all of these regions ?

Because it is working fine for all other regions. when I put "- serverless-s3-deploy" plugin in comment, this error is not coming.

Feature Request: Run after a regular deploy

First, let me thank you for this plugin. It's actually exactly what I'm looking for.

However, is there a reason why you don't run automatically after every sls deploy? When deploying a mixed backend/frontend web application I don't want to "forget" deploying my scripts to S3, too. So having it in a single command would be great. Any thoughts on this?

Propose you publish 0.3.0

So, maybe it's worth publishing 0.3.0

I suggest a minor bump since it's a large piece of new function, but it's a non-breaking change (and generally a major bump, to 1.0.0, implies a breaking change).

What do you think?

List stack resources function does not work

Looks like serverless doesn't use bluebird promises anymore, so that plugin fails here:
https://github.com/funkybob/serverless-s3-deploy/blob/v0.9.0/index.js#L86

  TypeError: this.provider.request(...).then(...).return is not a function
      at Assets.listStackResources (/.../node_modules/serverless-s3-deploy/index.js:87:12)
      at Assets.deployS3 (/.../node_modules/serverless-s3-deploy/index.js:144:17)
      at Assets.afterDeploy (/.../node_modules/serverless-s3-deploy/index.js:69:19)

Unable to deploy. Always prompting Access Denied

sls s3deploy

Serverless Error ---------------------------------------

Access Denied

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
OS: win32
Node Version: 8.10.0
Serverless Version: 1.40.0

I tried adding the recommended IAM policy but still I have the isue. :(

Does not like running on windows

Uploaded, but where it should have created subdirectories it's just banged the file up there with the folder in the name. I assume it's due to the backslash on windows vs forward slash on linux.

e.g. a using this plugin it's put the resources up there but with the key font-awesome\css\font-awesome.min.css (instead of creating subfolders).

Serverless 3.0 option type

Currently the sls shows this warning:

Serverless: Deprecation warning: CLI options definitions were upgraded with "type" property (which could be one of "string", "boolean", "multiple"). Below listed plugins do not predefine type for introduced options:
             - Assets for "verbose", "bucket"
            Please report this issue in plugin issue tracker.
            Starting with next major release, this will be communicated with a thrown error.
            More Info: https://www.serverless.com/framework/docs/deprecations/#CLI_OPTIONS_SCHEMA

This needs to be addressed in order for the plugin to work with Serverless 3.

More info:

Feature request: disable resource referencing

Hello,

I'm trying to use this plugin to deploy a packaged artifact to S3 without executing any deploy.

I'm encountering an issue where the plugin tries to list the resources of my Cloudformation Stack, but I don't actually have a CloudFormation stack.

Can this be disabled via an option value?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.