jsebfranck / contentful-publication Goto Github PK

When syncing spaces with a large number of entries (in my case, 700+), it's possible to exceed the Contentful Manage API rate limit. This results in an ECONNRESET error. Here's an example (in this case, during publish):

2015-07-17T18:45:17.093Z - error: Can not publish entry 1HKZmiQGpK6Ia8Ko0sE8mu for reason {
    "stack": "Error: read ECONNRESET\n    at exports._errnoException (util.js:682:11)\n    at TCP.onread (net.js:530:26)",
    "code": "ECONNRESET",
    "errno": "ECONNRESET",
    "syscall": "read"
}

Here's the information I've gathered from Contentful regarding the rate limit:

On the Content Management API the per second limit is of approximately 10 requests, this limit is re-evaluated every second.

The hourly limit is triggered when the total number of requests in the last 60 minutes exceeds a number that is the equivalent of breaking the per second limit for most seconds of the past hour. The 60 minutes limit is re-evaluated every 15 minutes for the past 60 minutes.

Clients need to back off on HTTP code 429, encountering the hourly rate limit happens only by disregarding return codes.

So, I think a rate throttle should be introduced so that sync can safely execute. Thoughts?

hey @jsebfranck, I came here from this post - https://www.contentful.com/blog/2015/02/11/how-to-sync-spaces-in-contentful/ and I had a question about your workflow. What I'm most curious about is what prompts the promotion from "editing space" to "production space" -- is this manually done by a developer or do you use contentful-publication in some other abstracted way so that folks from Xebia can promote content?

Also, i'm happy to take this to email if you think it's more appropriate: kyle [at] carrotcreative.com

The log level doesn't seem to be configurable. I'm not a big fan of using a global logger like that, would much prefer it was created in bin/fullSync.js and passed in to the {Content,Model}Sync constructors, as that way programmatic users can supply their own logger.

node version : 0.10
system: ubuntu 14
sync models between 2 Contentful spaces works without error but syncing content fails with error:

2015-06-12T15:22:17.715Z - debug: Processing sync response Unhandled rejection OperationalError: { "sys": { "type": "Error", "id": "NotFound" }, "message": "The resource could not be found.", "details": { "sys": { "type": "Space" } }, "requestId": "9f3-2135316989" }

at BufferList.<anonymous> (/home/vagrant/.nvm/v0.10.38/lib/node_modules/contentful-publication/node_modules/questor/index.js:40:25)
at BufferList.end (/home/vagrant/.nvm/v0.10.38/lib/node_modules/contentful-publication/node_modules/questor/node_modules/bl/bl.js:75:10)
at IncomingMessage.onend (_stream_readable.js:502:10)
at IncomingMessage.g (events.js:180:16)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:944:16
at process._tickCallback (node.js:448:13)

Any ideas how to digest the problem?

It might be good to save the sync tokens as the sync is progressing, as that way there will be less to recover if something fails partway through.

Hi! We're hoping to integrate a contentful sync operation with our Hubot instance, and it's a bit cumbersome for us to have to store the configuration as a JSON file on said client. We'd rather keep the static configuration pieces in environment variables (and allow other non-static ones to be persisted in Redis or passed in as arguments).

Would you be open to a PR to support:

1. Allowing for usage without the CLI as the primary entry point
2. Providing a hook to do whatever the user wants/needs to do with the sync token once the operation has completed:

var ContentfulSync = require('something')
function getConfig() { /* ... */ }

var config = Object.assign({}, getConfig(), {syncToken: robot.brain.get('syncToken')})

ContentfulSync(config)
  .then(function(response) {
    robot.brain.set('syncToken', response.syncToken)
  })
  .catch(function(err) {
    robot.slack.chat.sendMessage({
      text: 'There\'s evil afoot!'
    })
  })

I'm happy to provide more information! I could also be missing a really simple way to accomplish this otherwise :D