Code Monkey home page Code Monkey logo

mongoose-crate's Introduction

mongoose-crate

Dependency Status devDependency Status Build Status Coverage Status

mongoose-crate is a plugin for Mongoose for attaching files to documents.

File meta data is stored in MongoDB, whereas the actual file itself is stored on the local filesystem, Amazon S3 or Google Cloud Storage. For others pull requests are gratefully accepted.

Uploaded images can optionally be passed through ImageMagick to generate one or more images (e.g. thumbnails, full size, original image, etc) before saving.

The architecture is nominally based on mongoose-attachments but that project hasn't seen updates in a while.

## Usage

The following example extends the 'Post' model to use attachments with a property called 'attachment'.

const mongoose = require('mongoose')
const crate = require('mongoose-crate')
const LocalFS = require('mongoose-crate-localfs')

const PostSchema = new mongoose.Schema({
  title: String
})

PostSchema.plugin(crate, {
  storage: new LocalFS({
    directory: '/path/to/storage/directory'
  }),
  fields: {
    attachment: {}
  }
})

const Post = mongoose.model('Post', PostSchema)

.. then later:

const post = new Post()
post.attach('attachment', {path: '/path/to/file'}, (error) => {
	// attachment is now attached and post.attachment is populated e.g.:
	// post.attachment.url

	// don't forget to save it..
	post.save((error) => {
		// post is now persisted
	})
})

.. or using promises:

const post = new Post()
post
  .attach('attachment', {path: '/path/to/file'})
  .then(() => post.save())

Arrays

Files can be stored in arrays as well as individual properties. Just specify the array property to the field definition:

const mongoose = require('mongoose')
const crate = require('mongoose-crate')
const LocalFS = require('mongoose-crate-localfs')

const PostSchema = new mongoose.Schema({
  title: String
})

PostSchema.plugin(crate, {
  storage: new LocalFS({
    directory: '/path/to/storage/directory'
  }),
  fields: {
    attachments: {
      array: true
    }
  }
})

const Post = mongoose.model('Post', PostSchema)

.. then later:

const post = new Post()
post.attach('attachments', {path: '/path/to/file'}, (error) => {
  // post.attachments.length == 1

  post.attach('attachments', {path: '/path/to/another/file'}, (error) => {
    // post.attachments.length == 2
  })
})

.. or using promises:

const post = new Post()
post.attach('attachments', {path: '/path/to/file'})
  .then(() => post.attach('attachments', {path: '/path/to/another/file'}))

Images

See mongoose-crate-gm.

mongoose-crate-imagemagick is also available but should be considered deprecated because the underlying dependencies are no longer maintained.

Using with Express.js uploads

Assuming that the HTML form sent a file in a field called 'image':

app.post('/upload', (req, res, next) => {
  const post = new mongoose.model('Post')()
  post.title = req.body.title
  post.description = req.body.description
  post.attach('image', req.files.image)
    .then(() => post.save())
    .then(() => res.send('Post has been saved with file!'))
    .catch(err => next(err))
})

Metadata

Basic meta data is captured about uploaded files.

Example:

{
  "name" : "dragon.png",
  "size" : 26887,
  "type": "image/png",
  "url" : "http://my_bucket.s3.amazonaws.com/folder/4fbaaa31db8cec0923000019-medium.png"
}

Plugins can add extra meta data. E.g. mongoose-crate-imagemagick adds width, height, etc.

## Deletes and updates

If you delete a model, any attached files will be removed along with it (with one caveat, see Schema methods vs Queries below). Similarly, if you attach a file to a field that already has an attachment, the old file will be deleted before the new one is added.

For attachment arrays, when the model is saved, any attachments that are no longer in the array will have their files removed.

Schema methods vs Queries

Removal of files happens via middleware - if you use findById, findOne or anything else that returns a Query and call methods on that query, middleware is not executed. See the Mongoose middleware docs for more.

In short, do this sort of thing:

MySchema.remove({...}, callback)

or this:

MySchema.findOne({...}, (err, doc) => {
  doc.remove(callback)
})

..but not this:

MySchema.findOne({...}).remove(callback)

Array attachment deletion

const mongoose = require('mongoose')
const crate = require('mongoose-crate')
const LocalFS = require('mongoose-crate-localfs')

const MySchema = new mongoose.Schema({
  name: {
    type: String,
    required: true
  }
})

MySchema.plugin(crate, {
  storage: new LocalFS({
    directory: '/path/to/storage/directory'
  }),
  fields: {
    files: {
      array: true
    }
  }
})

// ...

const model = new MySchema()
model.name = 'hello'
model.attach('files', {
    path: file
}, callback)

// some time later remove one of the array entries

model.files.pop()
model.save()

Non array attachment deletion

const mongoose = require('mongoose')
const crate = require('mongoose-crate')
const LocalFS = require('mongoose-crate-localfs')

const MySchema = new mongoose.Schema({
  name: {
    type: String,
    required: true
  }
})

MySchema.plugin(crate, {
  storage: new LocalFS({
    directory: '/path/to/storage/directory'
  }),
  fields: {
    file: {}
  }
})

// ...

const model = new MySchema()
model.name = 'hello'
model.attach('file', {
    path: file
}, callback)

// some time later delete the file

model.file = null
model.save()

mongoose-crate's People

Contributors

achingbrain avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

mongoose-crate's Issues

Delete is not removing files from s3 bucket.

Hi, my image files are not being deleted along with the model.
my model and crate config look like this.

var path = require('path');
var mongoose = require('mongoose');
var timestamps = require('mongoose-times');
var URLSlugs = require('mongoose-url-slugs');
var crate = require('mongoose-crate');
var S3 = require('mongoose-crate-s3');
var ImageMagick = require('mongoose-crate-imagemagick');
var Schema = mongoose.Schema;

var aws = require('../config/env').config().aws;

var crateConfig = {
  storage: new S3({
    key: aws.accessKeyId,
    secret: aws.secretAccessKey,
    bucket: aws.bucket,
    acl: 'public-read',
    region: aws.region,
    path: function(image) {
      return '/images/' + path.basename(image.path);
    }
  }),
  fields: {
    image: {
      processor: new ImageMagick({
        tmpDir: '/tmp/ImageMagick',
        formats: ['JPEG', 'GIF', 'PNG'],
        transforms: {
          original: {
            format: '.jpg'
          },
          medium: {
            resize: '640x480',
            format: '.jpg'
          },
          small: {
            resize: '320x240',
            format: '.jpg'
          }
        }
      })
    }
  }
};

var postsSchema = new Schema({
  title: { type:String, required:true },
  body: { type:String, required:true },
  author: { type:String, required:true },
  published: { type:Boolean, required:true, default:false }
});

postsSchema.plugin(timestamps);
postsSchema.plugin(URLSlugs('title', {update: true}));
postsSchema.plugin(crate, crateConfig);

mongoose.model('posts', postsSchema);

File size limit

Is it possible to limit file size?
e.g.

attachment: {
 limit: 512k
},

attachments: {
  limit: 2m,
  array: true
}

Upload to S3 returns successfully with no error but files not uploaded

I have a mongoose schema where I'm just trying to attach an image with no extra processing and have that image upload to one of my S3 buckets.

userSchema.plugin(crate, {
  storage: new S3({
    key: '',
    secret: '',
    bucket: 'images',
    acl: 'public-read', 
    region: 'us-standard', 
    path: function(attachment) { 
         return '/user/' + path.basename(attachment.path)
    }
  }),
  fields: {
    image: {}
  }
})

Calling .attach on any user object returns successfully with a populated URL and file size but my S3 console shows no files in any of the buckets. I have no idea what's going on since there's no error logging since the call is successful. Could this have something do with with IAM roles and permissions on the bucket?

this.toObject is not defined

I am using crate with s3. When I do a Upload.findById(this.params.id) on the model I'm getting the error

node_modules\mongoose-crate\lib\Crate.js:176
var model = this.toObject()
                 ^
TypeError: undefined is not a function
    at EventEmitter.<anonymous> (/home/project/node_modules\mongoose-crate\lib\Crate.js:176:22)

At this point in time, this seems to be the global object, it shows on the console as

{ domain: null, _events: { init: [Function] }, _maxListeners: 0 }

My model is defined as follows:

var uploadSchema = new Schema({
    user: { type: Schema.Types.ObjectId, ref: 'User' }
});

uploadSchema.plugin(Crate, {
    storage: new S3({
        ....
    }),
    fields: {
        file: {}
    }
});

Pre save hook fails while saving model that doesn't have a field selected..

Model:

var TeamSchema = new Schema({
name: String // with select: false
});

Plugin

TeamSchema.plugin(crate, {
    storage: new S3({`
     ........
    }),
    fields: {
        gallery: {
            array: true,
        }
    }
});

Save: -- Fails..

Team.findOne({
        _id:'sfsfg435345345t'
    }).select('name')
   .exec(function (err, team) {
        _team.save(function (err) {  
        });
    });

Error:

error: uncaughtException: Cannot call method 'forEach' of undefined

Line 198

https://github.com/achingbrain/mongoose-crate/blob/master/lib/Crate.js#L198

model.__cached_attachments[field] is undefined which is expected because that wasn't selected as part of query and hence not available..

A simple undefined check on it would fix the issue... Would you like me to create a pull request to do a null check there.. ?

dependencies

Dependencies are out of date on all related projects.

Also, does this work with latest Mongoose + Express 4?

Question: How to get stored file

So, I have stored the file. Now how can I get it from the database?

As I understood there is a URL field in the model, and the file should be renamed using name field before getting it back

delete with non-array usage

Firstly, thanks. All works as advertised.

Under 'Deletes and updates' in the readme, there doesn't seem to be any indication of how to delete/remove a file with the non-array usage. I have had looked at the files to see how deletes and updates remove files in the situations covered, yet I am not sure how a simple remove should be implemented. I have this working for myself (with ugly code but working), so I am not in a rush. I don't know if you have plans for an 'unAttach' method , or .pre("save") check for the non-array usage. I need to stare at the code a bit longer to understand everything, but if you already have a plan for this, please let me know.

no method `.attach`

I get the following error when trying to update an existing object.

User object has no method 'attach'

UserSchema.plugin(crate, {
    storage: new LocalFS({
        directory: cfg.data + '/uploads'
    }),
    fields: {
        profileImage: {}
    }
});




User.findById(me).exec(function(err, user){
    user.attach("profileImage", { path: file.path }, function(err) {
        // attachment is now attached and post.attachment is populated e.g.:
        // post.attachment.url

        // don't forget to save it..
        user.save(function(err) {
            // post is now persisted
            res.json(200, user.profileImage.url);
        });
    });
}); 

Custom File Names

Hey, would you accept a pull request to allow filename processing? In short a way to configure the file name to be used for the storage strategies?

Specifically, I need a way to put the modelName, fieldName, model's ID, & a time stamp into the file name. The first 3 are for organization & the timestamp is for cache busting.

something like:
/public/attachments/:modelName/:fieldName/:id-:updated_at-:guid.ext

Retriving file fucntionality

I'd like to write this functionality for the plugin. When it will be done I will make a pull request.

My question is - would you mind if I use ES6, promises, and generators. It will get us more clean and flat code.

For example, instead of this

tasks.push(function(callback) {
  fs.exists(model[field].url, function(result) {
    callback(!result ? new Error('No file exists at ' + model[field].url) : undefined)
  })
})

it will be something like this

let result = yield fs.exists(model[field].url);
if (!result) { // and so on... }

And I'm planning to make a method which returns a promise, for being more eligible for koa apps and promise-based apps

"Error: Attachment has no path property"

hi, i try to attach an image directly from an express request according to the example in the Readme.md, but it seems not to work since the .attach() method gives me the following error:

Error: Attachment has no path property!

is this usage obsolete or is this a bug or am i doing something wrong? im not really familiar with express file uploads so it might be an issue on my side.

thanks in advance

remove method of FileProcessor blocking

The remove method of FileProcessor is blocking:

FileProcessor.prototype.remove = function(storageProvider, model, callback) {
  if(!model.url) {
    return
  }

  storageProvider.remove(model, callback)
}

if model has no url field then callback must be called:

FileProcessor.prototype.remove = function(storageProvider, model, callback) {
  if(!model.url) {
    return callback();
  }

  storageProvider.remove(model, callback)
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.