kibana question about cf-nodejs-logging-support HOT 7 CLOSED

We currently do not track request context for custom logs, therefore we cannot provide correlation_id forwarding.

from cf-nodejs-logging-support.

After some investigation, we have implemented request context bound logging for custom messages. This will keep track of request_id and correlation_id. We documented the correct usage of this feature in the ReadMe file.

Kind regards,
Christian

from cf-nodejs-logging-support.

Hi Christian,

Thanks for getting back to me. Sorry for the vague questions, I checked the code out and saw that it should be showing me the correlation_id in kibana, but it wasn't. I'm working on a python version of the cf_logger. I noticed that if you change the type field to 'log' you start to see correlation_id in the 'Application Logs' section of 'Requests and Logs'. Do you want me to branch, confirm, and create a pull request?

Cheers,

Per

from cf-nodejs-logging-support.

Hi Per,

we could not reproduce this behaviour. In custom log messages, the 'type' field is already set to 'log' and correlation_id is forwarded in any case. We checked our logs with the kibana Discover tab, so we could confirm the correct parsed json data was put through to kibana. It would be nice if you could confirm that the correlation id is missing for you there.

Kind regards,
Christian

from cf-nodejs-logging-support.

Hey Christian,

I think I see the problem. I'm not using the winstonTransport. I will update the code and try again.
I've been basically doing:

var express = require('express');
var log = require('cf-nodejs-logging-support');
var busboy = require('connect-busboy');
var AWS = require('aws-sdk');
var randomstring = require('randomstring');
var proxy = require('proxy-agent');
var crypto = require('crypto');
var app = express();

// setup server and logging
log.setLoggingLevel('info');
app.use(log.logNetwork);
app.use(busboy());

// setup aws
AWS.config.update({
httpOptions: {
agent: proxy('http://proxy.wdf.sap.corp:8080')
}
});
AWS.config.region = 'eu-central-1';
AWS.config.sslEnabled = true;
AWS.config.logger = log;

// create cypher
var encryptionAlgorithm = 'aes-256-ctr';
var encryptionKey = '764efa883dda1e11db47671c4a3bbd9e';
var encrypt = crypto.createCipher(encryptionAlgorithm, encryptionKey);

var handlePost = function(streamFn) {
// use identify function as default
streamFn = streamFn || function(stream) { return stream; }

return function(req, res) {
    if(req.busboy) {
        log.logMessage('info', 'Got request');
        req.busboy.on('file', function (fieldName, fileStream, fileName) {
            uploadToS3(fileName, streamFn(fileStream), function (error, data) {
                res.send(error || data);
            });
        });

        return req.pipe(req.busboy);
    }
}

}

var uploadToS3 = function(name, stream, callback) {
log.logMessage('info', 'Uploading file: ' + name);

var bucket = {
    params: {
        Bucket: 'd054086',
        Key: randomstring.generate() + '/' + name
    }
};

var s3Upload = (new AWS.S3(bucket)).upload({
    Body: stream
});

s3Upload.on('httpUploadProgress', function(event) {
    log.logMessage('info', event);
});

s3Upload.send(function(error, data) {
    stream.end();
    callback(error, data);
});

}

// add routes
app.post('/', handlePost());
app.post('/encrypted', handlePost(function(stream) { return stream.pipe(encrypt); }));

// start server
var port = process.env['PORT'] || 3000;
app.listen(port, function() {
log.logMessage('info', 'server is started on port %d', port)
});

from cf-nodejs-logging-support.

If you want to get the correlation_id from logMessage, you need to call it like this:
req.logMessage(...)

We can't provide correlation_id tracking outside the context of a request, because we cannot be sure which request to correlate to, if there are multiple pending requests.

from cf-nodejs-logging-support.

ahhhh - okie - all clear now. Thank you!! sorry if I caused trouble...

from cf-nodejs-logging-support.