Code Monkey home page Code Monkey logo

elasticsearch-writable-stream's People

Contributors

ganarajpr avatar marshall007 avatar rjferguson21 avatar voldern avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

elasticsearch-writable-stream's Issues

Example recovering on error

It's too often that elasticsearch is too busy and can't handle the request load.
Three questions:

  1. Will writable stream pause its source stream so that memory usage doesn't grow?
  2. How to continue/retry piping if elastic search goes down or otherwise timeouts?
  3. Are there other ways to delay/throttle piping?

Indexing error messages are swallowed during bulkWrite

When the data returned by client.bulk here contains errors, the error is propagated with an empty message.

The reason is this line.
The errors returned by bulk in this case are in item.index.error as an object.

Items with errors in the following format should not lose their error messages:

{
    "index": {
        "_index": "indexName",
        "_id": "recordId",
        "_type": "recordType",
        "status": 400,
        "error": {
          "type": "errorType",
          "reason": "errorReason",
          "caused_by": {
            "type": "causedType",
            "reason": "causedReason"
      }
   }
}

assertion-failed:index is required

Hi,

where do i modify my code in order to get below required format any other npm package which converts pgstreams directly to elastic search with some basic input params !

{
  index: 'name-of-index',
  type: 'recordType',
  id: 'recordId',
  parent: 'parentRecordType', // optional
  action: 'update', // optional (default: 'index')
  body: {
    name: 'Foo Bar'
  }
}

var ElasticsearchWritableStream = require('elasticsearch-writable-stream');

var stream = new ElasticsearchWritableStream(elasticsearchClient, {
  highWaterMark: 256,
  flushTimeout: 500
});

pg.connect(connectionString,function(err, client, done) {
--
  | if(err) throw err;
  | var query = new QueryStream('SELECT * FROM srt')
  | var streams = client.query(query)
  | //release the client when the stream is finished
  | streams.on('end', done)
  | streams.pipe(JSONStream.stringify()).pipe(stream)
  | })

localhost:92000/gov gives

{
  "gov": {
    "aliases": {},
    "mappings": {},
    "settings": {
      "index": {
        "creation_date": "1499854061799",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "DwHKUuBIRdiZqCyORwfCDA",
        "version": {
          "created": "5050099"
        },
        "provided_name": "gov"
      }
    }
  }
}

Individual errors

Hello, in elasticsearch js a response includes an error attribute for each record, along with a top level errors attribute (true/false).

When elasticsearch-writable-stream errors, the error object just contains the original records, there doesn't seem to be a way to determine which record failed.

I want to be able to determine which records failed, along with a count of total errors. Is there a way to do this?

thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.