Code Monkey home page Code Monkey logo

logzio-nodejs's Introduction

Build Status

logzio-nodejs

NodeJS logger for Logz.io. The logger stashes the log messages you send into an array which is sent as a bulk once it reaches its size limit (100 messages) or time limit (10 sec) in an async fashion. It contains a simple retry mechanism which upon connection reset (server side) or client timeout, wait a bit (default interval of 2 seconds), and try this bulk again. It does not block other messages from being accumulated and sent (async). The interval increases by a factor of 2 between each retry until it reaches the maximum allowed attempts (3).

By default, any error is logged to the console. This can be changed by supplying a callback function.

Before you begin you will need:

  • Nodejs with version 14.x or above

Sample usage

var logger = require('logzio-nodejs').createLogger({
    token: '__YOUR_ACCOUNT_TOKEN__',
    type: 'YourLogType'     // OPTIONAL (If none is set, it will be 'nodejs')
});


// sending text
logger.log('This is a log message');

// sending an object
var obj = {
    message: 'Some log message',
    param1: 'val1',
    param2: 'val2'
};
logger.log(obj);

Note: If logzio-js is used as part of a serverless service (AWS Lambda, Azure Functions, Google Cloud Functions, etc.), add logger.sendAndClose() at the end of the run. For example sync Lambda and async Lambda

Options

  • token Mandatory. Your account token. Look it up in the Device Config tab in Logz.io
  • type - Log type. Help classify logs into different classifications
  • protocol - http, https or udp. Default: http
  • host - Destination host name. Default: listener.logz.io
  • port - Destination port. Default port depends on protocol. For udp default port is 5050, for http is 8070 and 8071 is for https
  • sendIntervalMs - Time in milliseconds to wait between retry attempts. Default: 2000 (2 sec)
  • bufferSize - The maximum number of messages the logger will accumulate before sending them all as a bulk. Default: 100.
  • numberOfRetries - The maximum number of retry attempts. Default: 3
  • debug - Should the logger print debug messages to the console? Default: false
  • callback - A callback function called when an unrecoverable error has occured in the logger. The function API is: function(err) - err being the Error object.
  • timeout - The read/write/connection timeout in milliseconds.
  • addTimestampWithNanoSecs - Add a timestamp with nano seconds granularity. This is needed when many logs are sent in the same millisecond, so you can properly order the logs in kibana. The added timestamp field will be @timestamp_nano Default: false
  • compress - If true the the logs are compressed in gzip format. Default: false
  • internalLogger - set internal logger that supports the function log. Default: console.
  • extraFields - Adds your own custom fields to each log. Add in JSON Format, for example: extraFields : { field_1: "val_1", field_2: "val_2" , ... }.

Using UDP

A few notes are worth mentioning regarding the use of the UDP protocol:

  • UDP has some limitations, and therefore it is not the recommended protocol:
    • There is no guarantee that the logs have been received.
    • UDP can't take advantage of the bulk API, so performance is sub-optimal.
  • When using UDP, each message is sent separately, and not using the bulk API. This means that the meaning of bufferSize is slightly different in this case. The messages will still be sent separately, but the logger will wait for the buffer to reach the size specified before sending out all the messages. If you want each message to be sent out immediately, then set bufferSize = 1.

Update log

2.1.8

  • Make User-Agent not optional and add the version to it.

2.1.7

  • upgrade axios to v1.6.4 (contributed by @gcagle3)

2.1.6

  • Test node versions 14-20
  • upgrade axios to v1.6.0 (contributed by @gcagle3)

2.1.5

  • Add sourceIP as a new field to each log

2.1.4

  • Replace from request to axios

2.0.4

  • Add parameter to manage User-agent

2.0.3

  • Add verbose logging to use in Azure serverless function

2.0.2

  • Updated required fields for typescript

2.0.1

  • Fixed sorting by nanosec-timestamp
  • Added option to log string with an object
  • Updated Typescript declaration for optional dependencies

2.0.0

  • Added support for TypeScript
  • End of support for node 6
  • Upgrade dependencies due to security vulnerabilities
Expand to check old versions

1.0.4 - 1.0.6

  • Upgrade dependencies due to security vulnerabilities

1.0.3

  • Added the bulk to the callback in case the send failed

1.0.2

  • Handle no Error code on bad requests

1.0.1

  • ES6
  • Support node greater than node 6
  • Added gzip compress option
  • Added internal logger option

0.4.14

  • UDP callback bug fix + tests
  • UDP close connection bug fix + tests
  • ESLint

0.4.12

  • Updated ability to add custom port

0.4.6

  • Updated moment (v2.19.3) and request (v2.81.0) packages

0.4.4

  • @timestamp and @timestamp_nano will no longer be overriden given a custom value by the user.

0.4.3

  • Add the @timestamp field to the logs on the client's machine (and not when it reaches the server)

0.4.1

  • Updated request dependency to 2.75.0

0.4.0

  • Fixed issue #12 - added support for UDP
  • Minor refactorings

0.3.10

  • Fixed issue #17 - sendAndClose() wasn't actually closing the timer

0.3.9

  • Added option to add a timestamp with nano second granularity

0.3.8

  • Updated listener url
  • Added sendAndClose() method which immediately sends the queued messages and clears the global timer
  • Added option to supress error messages

0.3.6

  • Fixed URL for github repository in package.json

0.3.5

  • Bug fix : upon retry (in case of network error), the message gets sent forever

0.3.4

  • Bug fix : jsonToString() was throwing an error in the catch()block

0.3.2

  • Enhancement : Added option to attach extra fields to each log in a specific instance of the logger.

0.3.1

  • Bug fix : When calling log with a string parameter, the object isn't constructed properly.

Scripts

  • run npm install to install required dependencies
  • run npm test to run unit tests

logzio-nodejs's People

Contributors

8naama avatar adrukh avatar alonmiz avatar asafm avatar barakhaim avatar boofinka avatar dependabot[bot] avatar doron-bargo avatar gcagle3 avatar gillyb avatar hagarut avatar idohalevi avatar imnotashrimp avatar kleinron avatar mend-for-github-com[bot] avatar omerlh avatar resdenia avatar ronish31 avatar schwin007 avatar seetrai avatar sempasha avatar sheandemontigny avatar talhibner avatar travisbuddy avatar twash avatar ukmadlz avatar yotamloe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

logzio-nodejs's Issues

Not working well with AWS Lambda

It seems that running an AWS Lambda (with the winston-logzio-node transport), causes the Lambda function to always be timed-out.

Was anyone able to make that work ?

Thanks!

CVE-2020-7598 (High) detected in minimist-0.0.8.tgz

CVE-2020-7598 - High Severity Vulnerability

Vulnerable Library - minimist-0.0.8.tgz

parse argument options

Library home page: https://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz

Path to dependency file: /tmp/ws-scm/logzio-nodejs/package.json

Path to vulnerable library: /tmp/ws-scm/logzio-nodejs/node_modules/mocha/node_modules/minimist/package.json

Dependency Hierarchy:

  • mocha-5.2.0.tgz (Root Library)
    • mkdirp-0.5.1.tgz
      • minimist-0.0.8.tgz (Vulnerable Library)

Found in HEAD commit: c8f318300c915c177b52af0bf40acb13741a5427

Vulnerability Details

minimist before 1.2.2 could be tricked into adding or modifying properties of Object.prototype using a "constructor" or "proto" payload.

Publish Date: 2020-03-11

URL: CVE-2020-7598

CVSS 3 Score Details (9.8)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: High
    • Integrity Impact: High
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://github.com/substack/minimist/commit/63e7ed05aa4b1889ec2f3b196426db4500cbda94

Release Date: 2020-03-11

Fix Resolution: minimist - 0.2.1,1.2.2

Sending both string and object in one log statement

Hi, I was trying to send log by using a statement like;
logger.log('Response: ', response);
However, it did not work. It would be nice if it can work like that or maybe a warning should be written in your docs. I know your docs say it expects string or an object.

However, as Javascript has a usage like console.log("statement", object); it feels natural to use such statements in logging.
Have a nice day.

sendAndClose Method Had Stopped Working

Hi,

The sendAndClose Method is not working any more.
When calling it the data is logged but the process doesn't end...
I had to revert back to my branch.
It seems that one of your recent changes broke this functionality.

--Barak

Suppport CORS handling in sending http request

When logging from my angular app, I get a 'no Access-control-allow-origin' message. (Even though the message is sent to my logzio account).

erroroncors2

It would be nice if setting headers could be configured when creating the logger.

Tags

hi,

Is it a way to send a tag relative to a log?

Thanks

@timestamp_nano multiple issues

  1. @timestamp_nano can't be used to compare timestamps of two different shippers, because hrtime(), used by @timestamp_nano, is relative to a process start time. So the order of log messages, sorted by @timestamp_nano, will be wrong when several shippers are involved.

  2. When converting hrtime() timestamp to a @timestamp_nano string, the integers are not left-padded with zeroes, this leads to errors when comparing resulting strings alphabetically (e.g. 1234 will be less than 567 when compared alphabetically). This will cause sorting errors.

Error does not always have a cause in catch block. Leading to another exception.

I'm running the latest version of the library and periodically get errors like this:

3|neo4j-ap | TypeError: Cannot read property 'code' of undefined
3|neo4j-ap |     at request.post.then.catch (/usr/src/app/node_modules/logzio-nodejs/lib/logzio-nodejs.js:287:45)
3|neo4j-ap |     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
3|neo4j-ap |     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
3|neo4j-ap |     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
3|neo4j-ap |     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
3|neo4j-ap |     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
3|neo4j-ap |     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
3|neo4j-ap |     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
3|neo4j-ap |     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
3|neo4j-ap |     at Immediate.Async.drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
3|neo4j-ap |     at Immediate.args.(anonymous function) (/usr/src/app/node_modules/event-loop-inspector/index.js:138:29)
3|neo4j-ap |     at runCallback (timers.js:810:20)
3|neo4j-ap |     at tryOnImmediate (timers.js:768:5)
3|neo4j-ap |     at processImmediate [as _immediateCallback] (timers.js:745:5)

It seems that it's not safe to assume the err var has a cause or code.

_defaultCallback is called unbound

Package Version: 0.4.16

We've been working until recently with v0.4.2, since the last auto-update (we do that for patch version) in some scenarios (simples one is no token) the following error will be thrown.

'TypeError: Cannot read property \'log\' of undefined',
     '    at LogzioLogger._defaultCallback (/Users/matan/git/bookmd/client/node_modules/logzio-nodejs/lib/logzio-nodejs.js:102:28)',
     '    at Request._callback (/Users/matan/git/bookmd/client/node_modules/logzio-nodejs/lib/logzio-nodejs.js:315:21)',
     '    at Request.self.callback (/Users/matan/git/bookmd/client/node_modules/request/request.js:185:22)',
     '    at emitTwo (events.js:106:13)',
     '    at Request.emit (events.js:191:7)',
     '    at Request.<anonymous> (/Users/matan/git/bookmd/client/node_modules/request/request.js:1161:10)',
     '    at emitOne (events.js:96:13)',
     '    at Request.emit (events.js:188:7)',
     '    at IncomingMessage.<anonymous> (/Users/matan/git/bookmd/client/node_modules/request/request.js:1083:12)',
     '    at IncomingMessage.g (events.js:291:16)',
     '    at emitNone (events.js:91:20)',
     '    at IncomingMessage.emit (events.js:185:7)',
     '    at endReadableNT (_stream_readable.js:974:12)',
     '    at _combinedTickCallback (internal/process/next_tick.js:74:11)',
     '    at process._tickDomainCallback (internal/process/next_tick.js:122:9)'

Multiple places in the package perform the following action:
var callback = this.callback or this.callback = this._defaultCallback which causes the method to become unbound and when this.internalLogger.log('logzio-logger error: ' + err, err); is called an error is thrown.

This did not happen in the older version since it used console.log directly. The only usage of this in the older context was if (err && !this.supressErrors) so I assume the supressErrors check never had an effect (when called from unbound context).

`npm test` blocks forever

$ npm test produces the following output:

> [email protected] test ........./winston-logzio
> mocha -w




  winston-logzio
    send string as log message
warn: Just a test message
      ✓ builds the log object properly


  1 passing (14ms)

and hangs indefinitely.
node v4.2.4, npm 2.14.12 on OSX 10.11.2

My gut feeling is that this has to do with winston transport socket left open and occupying the event loop, looking for more guidance here as I'm testing logzio implementation in my project.

Ability for events in the same millisecond to be sorted

Currently I can sort by timestamp, however for events that are in the same millisecond, the sort order is usually wrong (reversed?). According to logz.io support, the way to fix this is to add an incrementing sequence number that can be used when defining the sort order in the kibana dashboard (presumably as the secondary sort attribute).

unable to close server in mocha tests

In api test using mocha, last step after tests have run is to shutdown the nodejs server with:

 after(function (done) {
      server.close(done);
 })

however, with any script that has require('logzio-nodejs'); this does not shutdown the server as presumably some resources eg; sockets must still be in use.

Ability for events in the same millisecond to be sorted

Currently I can sort by timestamp, however for events that are in the same millisecond, the sort order is usually wrong (reversed?). According to logz.io support, the way to fix this is to add an incrementing sequence number that can be used when defining the sort order in the kibana dashboard (presumably as the secondary sort attribute).

Cannot override the type field

I am declaring the winston logger in this way:

const logzioWinstonTransport = new LogzioWinstonTransport({
  level: 'info',
  name: 'winston_logzio',
  token: '__YOUR_API_TOKEN__',
});

But it doesn't make what I put on the name variable, I am always receiving the type as nodejs:
image

Any idea how to configure this? It's important because in all the other services (using other loggers) this is working properly, and we use this field to filter out.

Thanks a lot!

PS: I first opened the issue here, but then I realized that the problem was in this repository.

Typing is incorrect for log method

The type of the log method is given as

    log(msg: string, obj?: object): void;

In the docs it is recommended to send an object as the first parameter, and this is supported in the Javascript code.

Note that the second parameter is stringified and appended to the first parameter, so does not appear to be useful for sending JSON.

Ability for events in the same millisecond to be sorted

Currently I can sort by timestamp, however for events that are in the same millisecond, the sort order is usually wrong (reversed?). According to logz.io support, the way to fix this is to add an incrementing sequence number that can be used when defining the sort order in the kibana dashboard (presumably as the secondary sort attribute).

WS-2019-0019 Medium Severity Vulnerability detected by WhiteSource

WS-2019-0019 - Medium Severity Vulnerability

Vulnerable Library - braces-1.8.5.tgz

Fastest brace expansion for node.js, with the most complete support for the Bash 4.3 braces specification.

path: /tmp/git/logzio-nodejs/node_modules/braces/package.json

Library home page: https://registry.npmjs.org/braces/-/braces-1.8.5.tgz

Dependency Hierarchy:

  • jest-23.6.0.tgz (Root Library)
    • jest-cli-23.6.0.tgz
      • micromatch-2.3.11.tgz
        • braces-1.8.5.tgz (Vulnerable Library)

Vulnerability Details

Version of braces prior to 2.3.1 are vulnerable to Regular Expression Denial of Service (ReDoS). Untrusted input may cause catastrophic backtracking while matching regular expressions. This can cause the application to be unresponsive leading to Denial of Service.

Publish Date: 2019-03-25

URL: WS-2019-0019

CVSS 2 Score Details (5.0)

Base Score Metrics not available

Suggested Fix

Type: Upgrade version

Origin: https://www.npmjs.com/advisories/786

Release Date: 2019-02-21

Fix Resolution: 2.3.1


Step up your Open Source Security Game with WhiteSource here

BUG: objects are lost

If you try to log a message containing objects then logz.io doesn't show these logs in Kibana.

How to reproduce it

var logger = require('logzio-nodejs').createLogger({
  token: 'process.env.LOGZ_IO_TOKEN',
  type: 'YourLogType',   
  debug: true
});

logger.log({message: 'string'}); // ok
logger.log({message: {a: 'object'}}); // LOST
logger.log({message: {a: 'object', b: {c: 2}}}); // LOST
logger.log({message: ['array']}); // ok
logger.log({message: ['number', 1]}); // ok
logger.log({message: ['number', 1, {a: 'object', b: {c: 2}}]}); // LOST
logger.log({message: "['the whole log is a string', 1, {a: 'object', b: {c: 2}}]"}); // ok

I verified that this lib actually sends the parsed objects. It seems like Logz.io servers can't parse the incoming requests. The lib logzio-nodejssends the requests usingrequest.post(options, function (err, res, body) {fromLogzioLogger.prototype._send. The options.body` contains all data:

{"message":"This is a log message","type":"YourLogType","@timestamp":"2018-03-21T12:19:15.440Z"}
{"message":"string","type":"YourLogType","@timestamp":"2018-03-21T12:19:15.440Z"}
{"message":{"a":"object"},"type":"YourLogType","@timestamp":"2018-03-21T12:19:15.440Z"}
{"message":{"a":"object","b":{"c":2}},"type":"YourLogType","@timestamp":"2018-03-21T12:19:15.440Z"}
{"message":["array"],"type":"YourLogType","@timestamp":"2018-03-21T12:19:15.440Z"}
{"message":["number",1],"type":"YourLogType","@timestamp":"2018-03-21T12:19:15.440Z"}
{"message":["number",1,{"a":"object","b":{"c":2}}],"type":"YourLogType","@timestamp":"2018-03-21T12:19:15.440Z"}
{"message":"['the whole log is a string', 1, {a: 'object', b: {c: 2}}]","type":"YourLogType","@timestamp":"2018-03-21T12:27:21.962Z"}

Actual result

This is a screen shot from kibana. All logs containing objects are not logged.
image

Expected result

Kibana should display also the logs for objects from the code snippet:

  • logger.log({message: {a: 'object'}}); // LOST
  • logger.log({message: {a: 'object', b: {c: 2}}}); // LOST
  • logger.log({message: ['number', 1, {a: 'object', b: {c: 2}}]}); // LOST

Ability to specify destination port

For those users who are behind firewall it will be useful to get an ability to send logs messages on common http ports, like 80 for plain HTTP and 443 for HTTPS. Seems that listener.logz.io ready to accept bulk logs via HTTPS on port 443.

Lets say it will port option of createLogger method, what you think about?

Add an option to disable proxy

You are using request package which itself uses HTTP_RPOXY environment variable to configure HTTP proxy to use. In the particular case of proxies, this is bad – I don't want logs to flood our internal proxies.

Add an option to explicitly disable use of any proxy for the logging transport.

WS-2019-0032 Medium Severity Vulnerability detected by WhiteSource

WS-2019-0032 - Medium Severity Vulnerability

Vulnerable Library - js-yaml-3.12.0.tgz

YAML 1.2 parser and serializer

path: /tmp/git/logzio-nodejs/node_modules/js-yaml/package.json

Library home page: https://registry.npmjs.org/js-yaml/-/js-yaml-3.12.0.tgz

Dependency Hierarchy:

  • eslint-5.9.0.tgz (Root Library)
    • js-yaml-3.12.0.tgz (Vulnerable Library)

Vulnerability Details

Versions js-yaml prior to 3.13.0 are vulnerable to Denial of Service. By parsing a carefully-crafted YAML file, the node process stalls and may exhaust system resources leading to a Denial of Service.

Publish Date: 2019-03-26

URL: WS-2019-0032

CVSS 2 Score Details (5.0)

Base Score Metrics not available

Suggested Fix

Type: Upgrade version

Origin: https://www.npmjs.com/advisories/788/versions

Release Date: 2019-03-26

Fix Resolution: 3.13.0


Step up your Open Source Security Game with WhiteSource here

Winston Transport?

We are extensively using winston in our application code. I suspect a lot of people are.

Is there a winston-transport wrapper for Logz.io? Any plans to make one in the near future?

Bunyan transporter?

We are using Bunyan - do you have a transporter for Bunyan as well? (Did see the one regarding Winston..)

setting log level to error

From reading the winston-logzio, there is an object part that I didn't found on here

const logObject = Object.assign({},
      info,
      msg, {
        level: info[LEVEL] || this.level,
        name: this.name,
      });

    this.logzioLogger.log(logObject);
    callback(null, true);
  }

Close not working as expected in a lambda

Describe the bug
When running logzio in an AWS lambda I did the logger.close() at the end of the execution as I was instructed.

However, if you close the logger and the lambda gets reused, the logger is closed and won't ship any more logs and will error with Attempt to write logs with no transports, which can increase memory usage

To Reproduce
Steps to reproduce the behavior:

  1. Run a lambda and initialize the logger
  2. Run close() at the end of the execution as instructed per docs
  3. Next warm execution of the lambda will fail logging

Expected behavior
Replace close with a flush that will instruct flushing the current batch of logs

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Node version support

Within the Travis build we still test against version 8, I believe we should stop active support and switch to the following support pattern:

  • Current LTS (currently that is both v14 & v12)
  • Long term maintenance version (v10)

We could consider the latest build (v16) as a future-proofing step to forewarn of possible breaking changes in the core node APIs, but probably shouldn't be a breaking test.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.