adamreisnz / mongoose-upsert-many Goto Github PK

View Code? Open in Web Editor NEW

20.0 20.0 5.0 21 KB

A plugin that adds an upsertMany bulk op to Mongoose schemas

JavaScript 100.00%

mongoose-upsert-many's People

Contributors

Stargazers

Watchers

Forkers

bump44 timweri raydaim zaidkazi cgtarmenta

mongoose-upsert-many's Issues

modified count is higher compared to bulkWrite with unmodified items

I've noticed that nModified shows all items were modified even if I send in the exact same data twice in a row on an uninitialised collection. When I do the same with a bulkWrite, this doesn't seem to be the case. See the example and output below.

Isn't this a sign that much more database activity occurred than it should have for unmodified data? In my scenario, I intend to call this with a few hundred or even thousand items, so I'm not sure if the lengthier bulkWrite code isn't more performant.

This is using Mongoose 5.9.5 and 1.5.2 of your plugin.

How many round trips to the database does this make BTW? Will this make one call per item in the array passed in?

import mongoose from 'mongoose';
import upsertMany from '@meanie/mongoose-upsert-many';

mongoose.plugin(upsertMany);

const Schema = mongoose.Schema;

const Thing = mongoose.model('Thing', new Schema({
  name: String,
  count: Number
}, {
  versionKey: false,
  upsertMatchFields: ['name']
}));

const things = [
  { name: 'couch', count: 1 },
  { name: 'table', count: 2 },
  { name: 'chair', count: 4 }
];

(async () => {
  var db = await mongoose.connect(process.env.MONGODB_URL, {
    useNewUrlParser: true,
    useUnifiedTopology: true });

  var results = await Thing.upsertMany(things);
  console.log(`Initial write: ${JSON.stringify(results, null, 2)}`);

  results = await Thing.upsertMany(things);
  console.log(`Update with upsertMany: ${JSON.stringify(results, null, 2)}`);

  results = await Thing.bulkWrite(things.map((thing)  => ({
    updateOne: {
      filter: { name: thing.name },
      update: thing,
      upsert: true
    }
  })));
  console.log(`Update with bulkWrite: ${JSON.stringify(results, null, 2)}`);

  mongoose.disconnect();
})();

Initial write: {
  "ok": 1,
  "writeErrors": [],
  "writeConcernErrors": [],
  "insertedIds": [],
  "nInserted": 0,
  "nUpserted": 3,
  "nMatched": 0,
  "nModified": 0,
  "nRemoved": 0,
  "upserted": [
    {
      "index": 0,
      "_id": "5e8efaacdfdfb7d5cd092076"
    },
    {
      "index": 1,
      "_id": "5e8efaacdfdfb7d5cd092077"
    },
    {
      "index": 2,
      "_id": "5e8efaacdfdfb7d5cd092078"
    }
  ]
}
Update with upsertMany: {
  "ok": 1,
  "writeErrors": [],
  "writeConcernErrors": [],
  "insertedIds": [],
  "nInserted": 0,
  "nUpserted": 0,
  "nMatched": 3,
  "nModified": 3,
  "nRemoved": 0,
  "upserted": []
}
Update with bulkWrite: {
  "ok": 1,
  "writeErrors": [],
  "writeConcernErrors": [],
  "insertedIds": [],
  "nInserted": 0,
  "nUpserted": 0,
  "nMatched": 3,
  "nModified": 0,
  "nRemoved": 0,
  "upserted": []
}

'upsertMany is not a function'

I might be setting this up incorrectly, but after trying to add this as a plugin, I'm getting an error that 'upsertMany' is not a function when called. Have tried both mongoose.plugin(upsertMany) and adding it to the Schema.

My code is below:

const mongoose = require("mongoose");
const dateRange = require("./fetch/dateRange");
const inventoryFetch = require("./fetch/inventoryFetch");
const Brand = require("./models/brand");
const upsertMany = require("@meanie/mongoose-upsert-many");

const startingDate = new Date("1/1/2015");
const today = new Date();

const matchFields = ["ncid", "inventoryHistory.date"];
mongoose.plugin(upsertMany);

mongoose
  .connect(
    ...
  )
  .then(async () => {
    console.log("mongo connection established");
    let date = new Date(startingDate);

    await inventoryFetch(date)
      .then(dailyInventory => {
        return Brand.upsertMany(dailyInventory, matchFields);
      })
      .then(result => {
        console.log(result);
      })
      .catch(err => {
        console.log(err);
      });
  })
  .catch(err => console.log(err));

only upserting the last document of the array.

schema:

const mongoose = require("mongoose");
const upsertMany = require("@meanie/mongoose-upsert-many");

const IssueSchema = new mongoose.Schema(
  {
    repo: {
      type: mongoose.Schema.Types.ObjectId,
      ref: "Repos",
      required: true
    },
    number: {
      type: Number
    },
    opendAt: {
      type: Date 
    }
  },
  {
    timestamps: true
  }
);
IssueSchema.plugin(upsertMany);
const Issues = mongoose.model("Issues", IssueSchema);
module.exports = Issues;

trying like this:

const matchFields = ["repo"];

  //Perform bulk operation
  await Issues.upsertMany(pureIssues, matchFields);

but it only upserts or inserts or updates the last doc of the pureIssues

callback functions after upsertMany is done

How to call callback function from below code.
//Add new Product
module.exports.addBulkProducts = function(products, callback){
const matchFields = ['itemno'];
Products.upsertMany(products, matchFields);
}

await SyntaxError: Unexpected identifier

var result = await Products.upsertMany(products, matchFields);
^^^^^^^^
SyntaxError: Unexpected identifier

callback calls

Add the option to use $set

I have a use case where I need to upsert just some fields of the document the solution would be to add $set operator to the replaceOne

bulk .find(match) .upsert() .replaceOne({$set:item});

Not a true upsert?

I have a model that has default values.

employeeId: {
        type: String,
        required: true
    },
    isInvited: {
        type: Boolean,
        default: false
    },
    isNominee: {
        type: Boolean,
        default: false
    },
    isAttendee: {
        type: Boolean,
        default: false
    }

Upon initial upsertMany it creates a document with

{
 employeeId: 1,
 isInvited: true
}

If i want to upsertMany again only make the isNominee true,

{
 employeeId: 1,
isNominee: true
}

The modelToObject makes an object like

{
 employeeId:1,
 isInvited: false,
 isNominee:  true,
 isAttendee: false
}

This is incorrect behavior because I want to maintain the isInvited being true from the initial create. Any idea how to resolve this issue? The same thing happens for extra properties that may exist on initial create but are omitted from the upsert. If the object passed in does not contain the property then the upsert removes that property when it creates the new model from the modelToObject.
ex...
Initial Object in DB:

{
 shape: 'square',
color: 'Yellow',
text: 'Here'
}

OBJ to be updated:

{
  shape: 'square',
 color: 'Blue'
}

When your function is called the replaced document did not keep the text property from the original document

Is it possible to skip particular field to update

If I have specific field which I don't want to update if document already exists.
Can I do that with current code and how?

Lets Items say schema has following fields:
item_id:
name: string
price: Number
tax: Number
item_code: String

const items = [{item_id: "item_1234_9878", name: "item 1", price: 2000, tax: 18, item_code: "code_CSVN"},{item_id: "item_1235_9878", name: "item 2", price: 3000, tax: 18, item_code: "code_MXPL"}]

const matchFields = ['itemno'];

//Now Item 1 already exists matched by itemno and I want to update all other fields but not the item_id
//{item_id: "item_1111_9878", name: "item 1", price: 2000, tax: 18, item_code: "code_CSVN"}

const result = await Items.upsertMany(items, matchFields);

Await throws error 'await is only valid in async function'

Hello, is it possible to integrate traditional callback support forf this plugin?

Await does not work for me for some reason:

SyntaxError: await is only valid in async function

    const mongoose = require('mongoose');

    //Upsert many mongoose plugin
    const upsertMany = require('@meanie/mongoose-upsert-many');
    mongoose.plugin(upsertMany);
    
    var localPartnumber = require(__dirname + '/models/localPartNumber')
    var data = [...] //retrieved from FTP
    console.log('Processing ' + data.length + ' lines to database')
    const upsertConfig = {matchFields: ['code', 'shop']};

    //not async for some reason
    var processedPartNos = await localPartnumber.upsertMany(data, upsertConfig)

Promise resolution never returned

At the end of index.js the promise resolution is never returned, therefore is undefined when handling it outside of the upsertMany function.

I managed to get the result by changing the code the following way:

   return new Promise((resolve, reject) => {
      bulk.execute((error, result) => {
        if (error) {
          return reject(error);
        }
        return resolve(result);
      });
    });

I hope this helps. Cheers!

Not running Mongoose validations?

Hi, thanks for a very useful plugin. Could you confirm though whether this is bypassing the usual Mongoose validations (eg. for required fields) by using bulkWrite?

Question: how does matchFields work exactly?

//Large amount of items
const items = [
  ...
];

//Fields to match on for upsert condition
const matchFields = ['foo', 'bar.nested'];

//Perform bulk operation
const result = await MyModel.upsertMany(items, matchFields);

So if there's a doc in the db that has foo: 1 and bar.nested: 2, and an item with this data also exists in the list, then this doc will update the previous one, otherwise it will update it?

Thanks

Is there a way to retrieve the results of the operation?

I don't see this documented, but the object that is returned just contains the number of objects affected (i.e. updates or inserts) and doesn't return the new versions of the documents that were affected. Is this by design or is it something we should be expecting in a new version of this plugin?