Can you add a way to capture subsegment in a promise friendly way?

PS: Some kind of how https://www.npmjs.com/package/aws-xray-lambda-promise-subsegment is doing

Hello,

Local Sampling Rules configured with json version 1 do not seem to work anymore.

const rules = {
  "rules": [{
    "description": "test.",
    "service_name": "*",
    "http_method": "*",
    "url_path": "*",
    "fixed_target": 0,
    "rate": 1
  }],
  "default": {
    "fixed_target": 1,
    "rate": 0.1
  },
  "version": 1
  }

AWSXRay.middleware.setSamplingRules(rules);

Please checkout small example that rules configured with version 1 and withot version 2 are precessed in defferent ways:

https://github.com/kvzaytsev/xray-sampling

I guess that's caused by

https://github.com/aws/aws-xray-sdk-node/blob/master/packages/core/lib/middleware/sampling/local_sampler.js#L131

if (config.version === 2)
  params[key] = value;
if (config.version === 1 && key === 'service_name')
  params['host'] = value;

if config version is 1, only host field is added to the params object so httpMethod and urlPath are not matched using Utils.wildcardMatch.

Thanks!

When trying to use restify and the dynamodb document client I get the following sub/segment error

Error: Failed to get the current sub/segment from the context.
    at Object.contextMissingRuntimeError [as contextMissing] (/Users/dominicscandinaro/Projects/api/node_modules/aws-xray-sdk-core/lib/context_utils.js:21:15)
    at Object.getSegment (/Users/dominicscandinaro/Projects/api/node_modules/aws-xray-sdk-core/lib/context_utils.js:92:45)
    at Object.resolveSegment (/Users/dominicscandinaro/Projects/api/node_modules/aws-xray-sdk-core/lib/context_utils.js:73:19)
    at features.constructor.captureAWSRequest [as customRequestHandler] (/Users/dominicscandinaro/Projects/api/node_modules/aws-xray-sdk-core/lib/patchers/aws_p.js:60:29)
    at features.constructor.addAllRequestListeners (/Users/dominicscandinaro/Projects/api/node_modules/aws-sdk/lib/service.js:283:12)
    at features.constructor.makeRequest (/Users/dominicscandinaro/Projects/api/node_modules/aws-sdk/lib/service.js:203:10)
    at features.constructor.svc.(anonymous function) [as getItem] (/Users/dominicscandinaro/Projects/api/node_modules/aws-sdk/lib/service.js:642:23)
    at DocumentClient.get (/Users/dominicscandinaro/Projects/api/node_modules/aws-sdk/lib/dynamodb/document_client.js:265:32)
    at DocumentClient.dynamoDb.get (/Users/dominicscandinaro/Projects/api/lgs-lib/lib/dynamoDb.js:22:19)
    at Object.get (/Users/dominicscandinaro/Projects/api/lgs-lib/lib/cart.js:9:37)
    at Server.Cart.server.get (/Users/dominicscandinaro/Projects/api/routes/cart.js:11:38)
    at next (/Users/dominicscandinaro/Projects/api/node_modules/restify/lib/server.js:912:30)
    at f (/Users/dominicscandinaro/Projects/api/node_modules/restify/node_modules/once/once.js:25:25)
    at Server.Auth (/Users/dominicscandinaro/Projects/api/lib/auth.js:261:21)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
    at process.fallback (/Users/dominicscandinaro/Projects/api/node_modules/async-listener/index.js:565:15)
Failed to get the current sub/segment from the context. RequestError: Failed to get the current sub/segment from the context.
    at Object.get (/Users/dominicscandinaro/Projects/api/lgs-lib/lib/favorite.js:24:15)
    at getFavorite (/Users/dominicscandinaro/Projects/api/routes/favorites.js:7:38)
    at Server.Favorites.server.get (/Users/dominicscandinaro/Projects/api/routes/favorites.js:59:15)
    at next (/Users/dominicscandinaro/Projects/api/node_modules/restify/lib/server.js:912:30)
    at f (/Users/dominicscandinaro/Projects/api/node_modules/restify/node_modules/once/once.js:25:25)
    at Server.Auth (/Users/dominicscandinaro/Projects/api/lib/auth.js:261:21)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
    at process.fallback (/Users/dominicscandinaro/Projects/api/node_modules/async-listener/index.js:565:15)

My restify app.js code:

const env     = process.env.NODE_ENV || 'local';
const config  = require('./env')
const restify = require('restify');
const xray    = require('aws-xray-sdk-restify');
const routes  = require('./routes');
const server  = restify.createServer({ name: 'api' });

// Enable x-ray
xray.enable(server, `api-${env}`);

My dynamodb module:

const AWS           = require('./aws');
const AWSXray       = require('aws-xray-sdk');
const dynamoOptions = {
    httpOptions: {
        timeout: 2000
    },
    maxRetries: 0
};

const ddbClient     = AWSXray.captureAWSClient(new AWS.DynamoDB(dynamoOptions));
const dynamoDb      = new AWS.DynamoDB.DocumentClient(dynamoOptions);
dynamoDb.service    = ddbClient;

Example

I use the following typescript code in our nodejs backend server:

import * as AWSWithoutXRayWrapper from "aws-sdk";
import * as AWSXRay from "aws-xray-sdk";
const AWS = AWSXRay.captureAWS(AWSWithoutXRayWrapper);
[...]
return new Promise((resolve, reject) => {
  [...]
  this.client.upload(settings as any, (err, data) => {
    if (err) {
        reject(ApiError.AwsUploadError(err));
    } else {
        const expirationSeconds = Math.floor((expirationDate.getTime() - new Date().getTime()) / 1000);
        const params = { Bucket: data.Bucket, Key: data.Key, Expires: expirationSeconds };

        // we must wrap calls to aws api which depend on another async function
        AWSXRay.captureAsyncFunc("SignS3Url", (subsegment) => {
            this.client.getSignedUrl("getObject", params, (error, url) => {
                if (err) {
                    reject(ApiError.AwsUploadError(err));
                 } else {
                    resolve(url);
                 }

                 if (subsegment) {
                    subsegment.close();
                 }
            });
        });
    }
 });
});

Issue

This code prevents flush() to be called on the segment which was created by xray-express.
I debugged the file aws_p.js and found out that the S3 client calls build when making the presign request, but never calls complete on that request.
Thus, the subsegment is never closed and the corresponding segment never flushed.

Could someone please look into this and tell me what is happening here? =)

We've had x-ray running in production for a couple of months now. We started before AWS introduced the ability to set sampling rules via the AWS console.

Just wanted to check what the expected behaviour is now with this new feature (Centralised Sampling Rules) added.

We've configured the sampling rules via the setSamplingRules method. We use a local json file to configure our sampling rates. For example, here we have this file which sets the sampling rate to 1%.

{
  "version": 1,
  ....
  "default": {
    "fixed_target": 1,
    "rate": 0.01
  }
}

The docs suggest that, with introduction of centralised sampling routes, by default the remote config will be picked up. And one has to explicitly indicate to the client for local config to be picked up via disableCentralizedSampling method.

If I understand that correctly, by not doing anything, our sampling rate would have gone up to whatever the default centralised sampling rules are. Which I believe is 5%.

So, two questions:

1. Is that the expected behaviour?
2. If this is the expected behaviour, isn't that a breaking change? For example, if you have less than 5 instances running, you are suddenly sampling more traces and as a result paying more?

I am getting the following exception after upgrading to v1.1.5

Unable to import module 'index': Error
at Function.Module._resolveFilename (module.js:469:15)
at Function.Module._load (module.js:417:25)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (/var/task/node_modules/aws-xray-sdk-core/lib/context_utils.js:5:11)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)

Since this library uses CLS it appears it does not support Promises and Async/Await in v8 node.

What is the right way to support that? Manual mode? And can someone send an example using manual mode?

Thanks!

node.js HTTP documentation shows that http.request (or http.get, but there really isn't a difference) can accept an object, string or somewhat more recently, an URL instance.

For the latter two, they are converted to the regular option object, and GET is assumed.

options can be an object, a string, or a URL object. If options is a string, it is automatically parsed with url.parse(). If it is a URL object, it will be automatically converted to an ordinary options object.

HTTP implementation showing the parsing: https://github.com/nodejs/node/blob/5164a12618cf60150db621aebe1f3d89efb9b9aa/lib/_http_client.js#L57

Currently, making an outgoing http request with a string argument results in this error.

TypeError: Cannot set property 'X-Amzn-Trace-Id' of undefined
    at captureOutgoingHTTPs (/home/ec2-user/examples/1/node_modules/aws-xray-sdk-core/lib/patchers/http_p.js:88:40)
    at Object.captureHTTPsGet [as get] (/home/ec2-user/examples/1/node_modules/aws-xray-sdk-core/lib/patchers/http_p.js:148:12)

I want to trace the ElasticCache performance from my lambda function, could not find a capture reference for redis in the documentation. Is it supported ?

Since MongoDB is probably the most popular DB used with Node.js apps it makes sense to support it in this SDK.

The Segment-constructor should parse or throw if an bad formated root or parent ID is passed in.

It would further be very convenient to pass in the information straight from the header X-Amzn-Trace-Id?

Hi,

Are there anywhere typescript definitions for this project?
If not, do you have any plans to provide typescript definitions in the future?

thanks,

Guy

I tried to use x-ray with pg in a lambda function. I use pg with promise. I tried different thinks, like add AWSXRay.capturePromise(), but I always get the following error:

TypeError: Cannot read property 'name' of undefined at createSqlData (/var/task/node_modules/aws-xray-sdk-postgres/lib/postgres_p.js:90:25) at Client.captureQuery [as query] (/var/task/node_modules/aws-xray-sdk-postgres/lib/postgres_p.js:44:27) at connect (/var/task/node_modules/pg-pool/index.js:256:14) at client.connect (/var/task/node_modules/pg-pool/index.js:227:11) at Connection.<anonymous> (/var/task/node_modules/pg/lib/client.js:150:7) at Object.onceWrapper (events.js:315:30) at emitOne (events.js:121:20) at Connection.emit (events.js:211:7) at Socket.<anonymous> (/var/task/node_modules/pg/lib/connection.js:123:12) at emitOne (events.js:116:13) at Socket.emit (events.js:211:7) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) at Socket.Readable.push (_stream_readable.js:208:10) at TCP.onread [as _originalOnread] (net.js:607:20) at TCP.onread (/var/task/node_modules/async-listener/glue.js:188:31)

As I understand it the capturePostgres never capture the query that I send.
I think I have followed the documentation. I suppose it's a bug, but not sure of the where ?
You can acces my code here : https://github.com/Back9digital/node_lib/blob/x-ray/lib/db/postgresql.js

Thanks

I've found super annoying issue after migrating to nodejs8.10 runtime with async handler.

Short Reproducible code

"use strict";

module.exports.foo = async (event) => {
  console.log("got event: %j", event);

  require("aws-xray-sdk-core");

  return { data: "oh my!" };
};

Expected Result

Lambda Execution result should be { data: "oh my!" }.

Actual Result

On lambda nodejs8.10 runtime: returns null.
On native node 8.10 docker image: returns { data: "oh my!" }

If comment require("aws-xray-sdk-core") line, lambda nodejs8.10 runtime returns expected { data: "oh my!" }.

I wasted about 8 hours to investigate the root cause of this problem. I could not find any documents.
At least please write compatibility note about node.js runtime on documentation for customers.

Is there any possibility that support will be added for the module mysql2?

Make it easier to support MYSQL and Elasticsearch 6.x with async await.

In the case captureAsyncFunc is given an invalid segment (which can be supported with some error logs in other capture solutions) the execution function is called without any support for subsegment.close().
I might make sense to pass a dummy object with a close function to not crash the client code, and thus avoiding to systematically call subsegment && subsegment.close()

Hi,

I am using captureHTTPsGlobal for outgoing https requests and it works fine. It traces every http requests. It shows response status code and HTTP method correctly, but the response itself is empty and content length of the response is 0, even though I know it returns something.

Do I need any extra setup to show responses in the trace log?

I've just noticed that v1.2.0 is out- is there anywhere can I find out what bumped the minor version?

We are experiencing an issue with node_modules/aws-xray-sdk-core/lib/daemon_config.js

stack trace:

Unhandled rejection ReferenceError: processAddress is not defined
    at Object.<anonymous> (C:\****\node_modules\aws-xray-sdk-core\lib/daemon_config.js:24:17)
    at Module._compile (module.js:652:30)
    at Module._compile (C:\****\node_modules\pirates\lib\index.js:83:24)
    at Module._extensions..js (module.js:663:10)
    at Object.newLoader [as .js] (C:\****\node_modules\pirates\lib\index.js:88:7)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

OS: win
node: 8.11

I'm on the cusp of using aws-xray but I have read this caveat for var AWS = captureAWS(require('aws-sdk')); about 10 times, and I’m just not getting it:

Create new clients as usual. Be sure any outgoing calls that are dependent on another async function are wrapped, or duplicate segments might leak.

• If I’m in a lambda function handler and all I do is call into AWS APIs, awaiting the .promise() version, do I have to do anything else but wrap the module in captureAWS?
• Am I right in understanding client to mean something created using new AWS.s3() etc?
• Isn’t anything I do with a client an outgoing call (e.g. s3.getObject())? Or does “outgoing calls” in this context mean asynchronous actions not using said clients (e.g. rp.post()).
• By ‘dependent on’, does it mean that anything in .then() (or a babel transpiled await)?

For the sake of people like me, can someone please explain this to me like I'm five (perhaps with an example of when you do, and don't, need to start wrapping things in captureAsyncFunc?).

Assuming I get it, I'm happy to digest it all back into a PR for the documentation to save some confusion for the next person like me!

We make requests to S3 outside of a normal request, e.g. on service start-up and then periodically. This causes errors as X-Ray tries to retrieve the current segment from the non-existing request context.

I found some guidance for solving this with the Java SDK, but I can't find any for Node.js.

Is there guidance on how to instrument these out-of-request activities?

Hi there. Thank you for your work on this project. X-Ray is great!

I have some concerns about some of the dependency decisions made in the project. I'd like to use the aws-xray-sdk-core module in a Lambda handler, however simply including this library adds around 900 KB of code. That's crazy!

There must be something we can do to reduce the code size. For example, does the package really need to depend on Moment.js? Also, is it necessary to inline polyfills? Is winston really necessary?

Let me know your thoughts. I'm willing to raise a PR with some improvements (like removing Moment, for example.)

Hello,

I was under assumption that if I use "AWSXRay.captureHTTPsGlobal", it would work with all 3rd party http client such as node-fetch. But it seems that it's not the case. Is there a workaround?
It does show the subsegment for the outgoing http call in the xray console, but within the application it's sort of breaking the application since it doesn't return the fetch data.

Error: Failed to get the current sub/segment from the context.
at Object.contextMissingRuntimeError [as contextMissing] (/app/node_modules/aws-xray-sdk-core/lib/context_utils.js:21:15)
at Object.getSegment (/app/node_modules/aws-xray-sdk-core/lib/context_utils.js:92:45)
at Object.resolveSegment (/app/node_modules/aws-xray-sdk-core/lib/context_utils.js:73:19)
at captureOutgoingHTTPs (/app/node_modules/aws-xray-sdk-core/lib/patchers/http_p.js:67:31)
at captureHTTPsRequest (/app/node_modules/aws-xray-sdk-core/lib/patchers/http_p.js:152:12)
at /app/node_modules/node-fetch/lib/index.js:1327:15
at new WrappedPromise (/app/node_modules/async-listener/es6-wrapped-promise.js:13:18)
at fetch (/app/node_modules/node-fetch/lib/index.js:1319:9)

Ran into this while testing the examples I was providing in #11. Not sure how much real-world impact this has, but the http patcher adds a response listener to outgoing http requests, and then sets up a listener for res.on('end', ...) in order to close the subsegment. The problem is, once you do add a response listener, then the response must be consumed.

From the http documentation:

If no 'response' handler is added, then the response will be entirely discarded. However, if a 'response' event handler is added, then the data from the response object must be consumed, either by calling response.read() whenever there is a 'readable' event, or by adding a 'data' handler, or by calling the .resume() method. Until the data is consumed, the 'end' event will not fire. Also, until the data is read it will consume memory that can eventually lead to a 'process out of memory' error.

Currently if the developer didn't add a listener of their own, the response is never consumed, meaning the
response data is leaked, and since end is never fired, the subsegment will never close meaning it and its parent (sub)segment will never be flushed and sent to the daemon.

Reason I'm not sure how much real-world impact this would have, is I'm not sure how often you would simply not care about the response, but I'm sure there's some sort of fire-and-forget systems out there...?

The following should fix it, but I've not developed any tests for it. It consumes the body if a callback wasn't provided, and if x-ray is the only response listener. Meaning neither of the two scenarios below occurred:

http.request('http://example.com', res => res.resume()).end();
http.request('http://example.com').on('response', res => res.resume()).end();

diff --git a/packages/core/lib/patchers/http_p.js b/packages/core/lib/patchers/http_p.js
index 5c0cb68..f5311ef 100644
--- a/packages/core/lib/patchers/http_p.js
+++ b/packages/core/lib/patchers/http_p.js
@@ -137,6 +137,8 @@ function enableCapture(module, downstreamXRayEnabled) {
         } else {
           callback(res);
         }
+      } else if (req.listenerCount('response') === 1) {
+        res.resume();
       }
     }).on('error', errorCapturer);

Of course, if the developer does add their own listener and neglects to consume the response, then you still run into the segment never closing, but that's a different issue, with developer error mixed in.

Goal

The goal of this issue is to track all tracing context preservation related problems, dive deep on why some of them work and some of them don't, discuss and gather feedback for a way going forward to provide much better customer experience with X-Ray node tracing and further contribute/shape the node context propagation in general.

Problem Statement

The Node.js event loop model has its downsides. One of those is the context preservation through the call chain within a single http request. It is hard to bind arbitrary data (user name, transaction id, tracing context etc) to a full request/response cycle and all relevent context stops at event loop. It's hard to know what is the context for the next callback being executed by event loop.

There is a library continuation-local-storage (CLS for short) that solved most of the problem by binding request context to callbacks. However, this library has two limitations:

• It's both a bug and a feature that CLS doesn't work with all node modules. The full reasoning from CLS author is here. In short it's difficult for CLS to know whether it should bind the context or not all the time.
• Node community evolves fast and with the major adoption of async/await syntax from Node 8 CLS doesn't fit some new use cases anymore.

Challenges for X-Ray SDK for Node.js

With all the context preservation issues listed above, they are solvable for most of the node application owners who need to preserve contextual data within requests.

• For the first issue, CLS provides various of wrappers with name like cls-* for explicitly binding context to callbacks in a particular module. Here is an example fork for node-redis that patches Redis driver for context preservation.
• For the second issue (async/await), cls-hooked are reported to be worked well with Node 8 so far by several SDK users. Zone.js is also a potential candidate for the new dependency of the SDK for context binding.

These workarounds are not ideal but work for each service owner as each service only has limited dependency and the owner knows what dependencies the service depends on. It's inevitable and necessary work for service owners to understand this problem space and make necessary code changes since it is business data (user name, transaction id etc) through a request/response cycle.

But for X-Ray SDK one of the core functionalities is to preserve the tracing context through the async call chain and the SDK should abstract this technical challenge away from users who just want all traced functions work once they import the SDK. This issue #28 has some detailed discussion regarding this concern. There are also limitations and known issues on cls-hooked one of which is it doesn't work with some certain libraries (example) while the X-Ray SDK should try not to have the SDK users to deal with these caveats.

Going Forward

(This section will likely be updated more frequently based on both internal and external discussions and feedback.)

The SDK should re-evaluate CLS as its core dependency and leverage the new tools (cls-hooked or zone.js) for preserving context for most of the major use cases today as well as being flexible enough to be forward compatible with new coming features.

A short-term solution could be to start with an experimental branch that replaces CLS with cls-hooked or Zone.js and thorough testing with existing supported libraries and new use cases we intend to cover. Or an alternative approach is to have something in user code with

var cls = process.env.AWS_XRAY_EXPERIMENTAL_ASYNC_AWAIT ? require('cls-hooked') : require('continuation-local-storage');

as suggested by @nakedible-p but the issue is that it makes the bundle size more bloated when the SDK is a dependency. Additionally, these libraries are not owned by AWS X-Ray and some critical fixes/supports for the library from X-Ray's prospective might not be as critical for those libraries' owners.

Long Term: We will share this portion with broader audience after internal discussions. Any suggestion is welcome.

As XRay currently doesn't support koa, I try to create an instance of the client manually and without a framework-assignment:

import * as XRay from 'aws-xray-sdk';
XRay.setLogger(console);
XRay.config([ XRay.plugins.EC2Plugin, XRay.plugins.ECSPlugin ]);
XRay.enableAutomaticMode();
const segment = new XRay.Segment("root");
XRay.setSegment(segment);

However this snippet results in an error in this line: XRay.setSegment(segment);

Error is:

Error: No context available. ns.run() or ns.bind() must be called first.

The current MySQL instrumentation only captures the url and query type of each query made.

It would be nice to update the instrumentation to also capture the actual query being run.

Obviously there would be privacy concerns attached to this, since you probably don't want to log out every query.

I'm happy to implement this if I can get some direction from the core team as to how you want to handle this.

I was thinking you can provide a predicate as part of the options initialising the instrumentation. This predicate that would be given the query being run, and return whether or not this query should log the query being run.

I came to look for this after seeing this forum thread: https://forums.aws.amazon.com/thread.jspa?messageID=809685

We have a custom logger, run nodejs and use lambda. We cannot suppress require time log messages from the javascript SDK. [email protected]

Every lambda invocation starts with

AWS_XRAY_DAEMON_ADDRESS is set. Configured daemon address to 169.254.79.2:2000.
AWS_XRAY_CONTEXT_MISSING is set. Configured context missing strategy to LOG_ERROR.

These messages are factually correct but are logged at the info level which we wish to suppress in production. Our logger suppresses info messages however these are logged at require time before our code has a chance to set the logging object.

You can find the culprit in context_utils.js:200.

This bug doubles our logging output and causes us extra cloud watch costs as well as making searching our logs for "error" very difficult.

Recently a change was completed to move aws-sdk out of dependencies into dev dependencies in the parent package.json. Is it possible to move it in the aws-xray-sdk-core package.json as well? Or will that cause issues?

Hello,

The ExpressJs middleware does not set the X-Amz-TraceId header on the response. The Java ServletFilter does. Seems like this would be universally useful, please add.

Currently we use CLS to track information, and it does not work well with async/await, hence support of some frameworks like Koa is not good.

If we migrate from CLS to cls-hooked (like this) we can support async/await without too much code changes.

When deploying on AWS Lambda, aws-sdk does not need to be bundled since it is already present in the Node.js runtime. This yields significant bundle savings, but... not if you're using aws-xray-sdk-core.

The reason for this is aws-xray-sdk-core requires aws-sdk as a normal dependency. It should be a peerDependency and possibly a devDependency.

We are trying to use the express wrapper in conjunction with a redis database.

We use node-redis. But it does not seem to be working:

const router = require('express').Router();
const redis = require('redis');
const xray = require('aws-xray-sdk');

const cache = redis.createClient({host: 'localhost'});

router.get('/test', (req, res) => {
    console.log(xray.getSegment()); // this is fine
    cache.get('test', () => {
        console.log(xray.getSegment()); // this throws
        res.json({});
    });
});

/Users/rahmen01/workspace/ibl-episodes-api/node_modules/aws-xray-sdk-core/lib/context_utils.js:21
        throw new Error(message);
        ^

Error: Failed to get the current sub/segment from the context.
    at Object.contextMissingRuntimeError [as contextMissing] (/Users/rahmen01/workspace/ibl-episodes-api/node_modules/aws-xray-sdk-core/lib/context_utils.js:21:15)
    at Object.getSegment (/Users/rahmen01/workspace/ibl-episodes-api/node_modules/aws-xray-sdk-core/lib/context_utils.js:92:45)
    at Command.cache.get [as callback] (/Users/rahmen01/workspace/ibl-episodes-api/routes/nik.js:12:26)
    at normal_reply (/Users/rahmen01/workspace/ibl-episodes-api/node_modules/redis/index.js:732:21)
    at RedisClient.return_reply (/Users/rahmen01/workspace/ibl-episodes-api/node_modules/redis/index.js:835:9)
    at JavascriptRedisParser.returnReply (/Users/rahmen01/workspace/ibl-episodes-api/node_modules/redis/index.js:193:18)
    at JavascriptRedisParser.execute (/Users/rahmen01/workspace/ibl-episodes-api/node_modules/redis-parser/lib/parser.js:574:12)
    at Socket.<anonymous> (/Users/rahmen01/workspace/ibl-episodes-api/node_modules/redis/index.js:275:27)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10)
    at TCP.onread (net.js:607:20)

Same route does work fine using ioredis:

'use strict';

const router = require('express').Router();
const Redis = require('ioredis');
const xray = require('aws-xray-sdk');

const cache = new Redis();

router.get('/test', (req, res) => {
    console.log(xray.getSegment()); // this is fine
    cache.get('test', () => {
        console.log(xray.getSegment()); // this is fine
        res.json({});
    });
});

module.exports = router;

Hi! Im trying to capture mssql driver connections but no result is going out in XRay. I guess this package is not yet supported. Any ideas about this? Thanks!

Hi, we are trying to pass trace-id's between multiple apps that talk to each other via SQS. It looks like this library will automatically generate a subsegment for talking to SQS, but there's nothing on the other end (say in MessageAttributes) to know the traceId and parentId of that subsegment. Is there something I'm missing? For now, we are just adding it to the MessageBody but this means I can't use this library to instrument AWS clients, as I need to know the subsegment id.

I found this link referencing a "convention" to use MessageAttributes: https://forums.aws.amazon.com/thread.jspa?messageID=794108 but it doesn't look like this library does it.

It looks like the xray sdk instruments the handler with the assumption that the handler will follow the callback pattern required in node 4 and 6. With node 8, async functions are available (and encouraged in the node 8 on lambda blog post), but including the sdk without wrapping the async handler fails.

Working:

exports.handler = async() => ({statusCode: 200});

HTTP/1.1 200 OK

Broken:

require('aws-xray-sdk');

exports.handler = async() => ({statusCode: 200});

HTTP/1.1 502 Bad Gateway

{
    "message": "Internal server error"
}

Workaround:

require('aws-xray-sdk');

exports.handler = (event, context, callback) => {
  run()
    .then((res) => callback(null, res))
    .catch((err) => callback(err));
};

async function run() {
  return {statusCode: 200};
}

HTTP/1.1 200 OK

Using promises instead of callbacks fail.

Probably related to #12

Using the SDK with Winston v.3.0.0 produces the following error:

node_modules/aws-xray-sdk-core/lib/logger.js:17
    logger = new (winston.Logger)({});

TypeError: winston.Logger is not a constructor

Winston had a breaking change in 3.0 which changes how the logger should be constructed:

https://github.com/winstonjs/winston#creating-your-own-logger

winstonjs/winston@eff6f00#diff-fc1a687308b45c41cf7efa7aff4253dd

Not sure if updating the Winston version in this project is an option?

According to npm, lodash has around 14.5 million downloads per week, whereas underscore has less than 5 million downloads per week. Both libraries have around the same capabilities, but lodash is massively popular.

The reason for this request is that having both libraries demands more npm download time and generates larger bundles; the sdk core is used by lambda functions, and any extra libraries loaded increase the size of the bundle. Also, according to AWS own tips, developers should be using tools like Browserify to bundle up the functions, so this is even more important (see video). Also, requesting the target functions instead of the whole library would help.

I've been trying to find a way to add in support for the API Gateway Context and event data (URL, Path, Headers, etc).

I have a function which should be working to add in the HTTP data to the currently open segment, but this data does not appear to be passed along to XRay:

export function initTracking(event: any) {
   const segment = XRay.getSegment();
   if(segment){
      const headers: any = {};
      if(event.headers){
         for(let header_name in event.headers){
            headers[header_name.toLowerCase()] = event.headers[header_name];
         }
      }
      const incoming_request_data = new XRay.middleware.IncomingRequestData({
         url: event.path,
         connection: { 
            encrypted: true,
         },
         headers, 
         method: event.httpMethod,
      });
      segment.addIncomingRequestData(incoming_request_data);
   }
   return segment;
}

I also tried creating a new segment, but that also failed (and trying to use XRay.setSegment(segment) caused an error):

export function initTracking(event: any) {
   const parent_segment = XRay.getSegment();
   if(parent_segment){
      const segment = new XRay.Segment(parent_segment.name, parent_segment.rootId, parent_segment.id);
      const headers: any = {};
      if(event.headers){
         for(let header_name in event.headers){
            headers[header_name.toLowerCase()] = event.headers[header_name];
         }
      }
      const incoming_request_data = new XRay.middleware.IncomingRequestData({
         url: event.path,
         connection: { 
            encrypted: true,
         },
         headers, 
         method: event.httpMethod,
      });
      segment.addIncomingRequestData(incoming_request_data);
      return segment;
   }
}

Is there something I'm missing? Is it possible to have this data automatically picked up by XRay, or if not is there a way to add this into my function so I can add the additional HTTP data available from Lambda when executed from API Gateway?

After investing plenty of time on trying to make XRay tracing work for our Postgress client we realised that, it might be a version compatibility issue between aws-sdk and pg library.

We are using

"aws-xray-sdk": "1.2.0",
"pg": "7.4.1",

and the below code snippet to enable tracing for our PG Client

const AWSXRay = require('aws-xray-sdk');
const pg = AWSXRay.capturePostgres(require('pg'));
client = new pg.Client(config);
AWSXRay.capturePromise();

But we get error with our client instance when invoked on AWS saying,

{ TypeError: query.on is not a function
at Client.captureQuery [as query] (/var/task/node_modules/@gdw/postgres-client/node_modules/aws-xray-sdk-postgres/lib/postgres_p.js:67:13)
at PostgresClient.findBy (/var/task/node_modules/@gdw/postgres-client/lib/PostgresClient.js:130:28)
at CustomerRepository.findActiveCustomers (/var/task/lib/CustomerRepository.js:68:35)
at CustomerService.findCustomerAccounts (/var/task/lib/CustomerService.js:109:40)
at /var/task/lib/Customers.js:76:38
at /var/task/node_modules/@gdw/postgres-client/lib/PostgresClient.js:176:20
at /var/task/node_modules/@gdw/postgres-client/node_modules/continuation-local-storage/context.js:84:17
...

Can you please confirm is there is an version compatibility limitation between pg and aws-sdk?

The following code does not work:

const DynamoDB = require('aws-sdk/clients/dynamodb')
const xray = require('aws-xray-sdk-core')
const ddb = xray.captureAWSClient(new DynamoDB())

In theory, it should be just the same as encapsulating new AWS.DynamoDB() coming straight from aws-sdk, but it does not work.

This is specially problematic in AWS Lambda environment, where loading more than you need makes you use more memory and more time.

Does it work with koa?

Details

this.addFaultFlag();

AWS X-ray concepts regarding error categorization

X-Ray tracks errors that occur in your application code, and errors that are returned by downstream services. Errors are categorized as follows.

Error – Client errors (400 series errors)
Fault – Server faults (500 series errors)
Throttle – Throttling errors (429 Too Many Requests)

Issue

if addError refers directly to client errors then it is a bit confusing that it flags it as a "Fault" and not an "Error" flag.

If addError is more of an abstract method for all x-ray error types, then I don't have the option to specifiy which kind of x-ray error it is per the categorizations above. And I'm forced to have it flagged as a "Fault".

I'm trying to capture the request and response for every API request and save it as metadata to a segment.. since attaching metadata to the main segment doesn't work I created a custom subsegment as follows:

import { APIGatewayEvent, Callback, Context, Handler } from 'aws-lambda'
import { getUserLocation } from './api/get-user-location'
import { handleFindNearbyUsers } from './api/handle-find-nearby-users'
import { handleSendSMS } from './api/handle-send-sms'
import { lambdaResponse } from './helpers/lambda-response'

const awsXRay = require('aws-xray-sdk')
awsXRay.captureAWS(require('aws-sdk'))
awsXRay.captureHTTPsGlobal(require('http'))
awsXRay.captureHTTPsGlobal(require('https'))

export const handle: Handler = async (event: APIGatewayEvent, context: Context) => {
  // const key = event.headers['x-api-key']
  const path: string = event.resource
  let response
  if (path === '/user-location') {
    response = await getUserLocation(event)
  } else if (path === '/send-sms') {
    response = await handleSendSMS(event)
  } else if (path === '/nearby-users') {
    response = await handleFindNearbyUsers(event)
  } else {
    response = lambdaResponse(new Error('Path not found'), 404)
  }

  console.log('request', event)
  console.log('response', response)

  capture(event, response)

  return response

}

function capture (event, response) {
  awsXRay.captureFunc('Metadata', (subsegment) => {
    if (subsegment) {
      subsegment.http = new awsXRay.middleware.IncomingRequestData({
        method: event.httpMethod,
        url: event.path,
        connection: {
          secure: event.headers['CloudFront-Forwarded-Proto'] === 'https'
        },
        headers: {
          'x-forwarded-for': event.headers['X-Forwarded-For'],
          'user-agent': event.headers['User-Agent'],
          'host': event.headers.Host
        }
      })
      subsegment.addMetadata('request', event)
      subsegment.addMetadata('response', response)
      // subsegment.close()
    }
  })
}

I tried both captureFunc and captureAsyncFunc, However I don't see the subsegment sent with the trace. What's the correct way to do this?

As far as I can see, if you want to build a segment around function that returns something, you're forced to do this:

let result;
AWSXRay.captureFunc('the thing', function(subsegment) {
  const returned = doSomething();
  result = result;
  subsegment.close();
});

Am I missing something?

Hey,

every single of 400 our Lambda function randomly crashes after adding X-Ray SDK.

⬇️ Stack trace ⬇️

Error: socket hang up
    at TLSSocket.onHangUp (_tls_wrap.js:1124:19)
    at TLSSocket.g (events.js:292:16)
    at emitNone (events.js:91:20)
    at TLSSocket.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at /var/task/node_modules/async-listener/glue.js:188:31
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickDomainCallback [as _tickCallback] (internal/process/next_tick.js:128:9)

I suspect this is because of async-listener package which monkey-patches stuff and causes problems with Node 8.10 in Lambda (see #27)

When I run yarn why async-listener and xray-sdk is the only package depending on it.

$ yarn why async-listener 
yarn why v1.6.0-20180405.1011
[1/4] 🤔  Why do we have the module "async-listener"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "[email protected]"
info Reasons this module exists
   - "aws-xray-sdk-core#continuation-local-storage" depends on it
   - Hoisted from "aws-xray-sdk-core#continuation-local-storage#async-listener"

Upgrading from node 6 to 8 didn't help.

Any clue how to fix it? We want to use AWS X-Ray but we don't want our Lambda to crash randomly and confuse users.

In the above function, a header of the form Root=2134; (notice the extra semicolon at the end) will cause the function to throw a TypeError, as there is no content after the semicolon.

I'm more than happy to send a pull request to add a check for values being present before trimming/splitting them, do let me know what you think.

The SDK is having issues when an X-Ray instrumented express web application is running in AWS Lambda by utilizing https://github.com/awslabs/aws-serverless-express, reported in a comment at #30.

The SDK vended express middleware needs to work better with Lambda generated segments to provide better customer experiences.