Implement TagSpan request. See documentation

During the alpha project a scout-e2e-tests repo was created, it would be good to re-validate that it's still working as expected with the current version of the beta (until we have an application that can always be using it).

The API key will probably have to be provided to the local tests to get them to work.

If the socket drops we're going to want to connect.

Should include a variable for retry attempts that's unbounded to start and can be bounded.

Node is the only language agent with a client-side validation hash (in download-configs.ts) of core agents.

In its current form, it makes it impossible to set arbitrary new versions via config (SCOUT_CORE_AGENT_VERSION), since they won't match the prebuilt listing in the agent. Adhoc versions are used sometimes to help debug customer situations.

Additionally, it's just annoying to update. In the other language agents, it's simply specifying the version.

Should be possible to probe the core-agent for status, and combine it's status with the AgentStatus type (add an optional probe property? keep track of when the probe was taken?) .

Agent should be able to send a shutdown command to core-agent, likely going to be useful in a graceful shutdown especially in the case of #4

The Node agent should be as configurable and functionally equivalent as our Python agent implementation.

E.g. configurable to download different versions, from different URL, disable download, disable launching, etc.

Implement sending StartRequest requests to the agent. See documentation

The requests created by the express integration should automatically have a span of Controller/<METHOD> <PATH> created so they're recorded properly by the backend.

Note that PATHs should not have dynamic segments' values -- users/:userId is preferable to /users/1. It remains to be seen whether it's possible to retrieve/reconstruct the dynamic segments from express when a request has started.

@cschneid Has seen the agent hang on the first request. @cschneid Can you provide any information you have?

Send the app metadata once on boot. Refer to the Python agent for the payload and key names.

Add E2E test using the Controller/ span as per debugging session with Chris

Implement sending FinishRequest requests to the agent. See documentation

The NodeJS agent should support communicating with an existing external agent process, if a process domain socket path is provided.

Tasks:

• Remove ChildProcess type

It would be nice to build an automated test suite that runs X node versions versus Y versions of scout (possibly including #44 ), producing some nice compatibility table (that could be used on the README).

The current version is 1.2.6, See-also #64

Implement StartSpan request. See documentation

Ensure that monitoring=false shuts off all monitoring -- no requests should be traced.

A common need in deployments is to have a generic logger implementation that the agent logs to, which then either handles it directly, or more often, passes onward to the same logging system that the user has setup.

Scout Agent -> Scout Log Handler -> User's Log Handler

This is mostly useful for filtering Scout's log level, even though the application is at DEBUG or TRACE type levels, the user who's running Scout happily does not need a constant stream of detailed debug. But those same detailed debug messages are very handy to emit when there is an issue. So allowing a SCOUT_LOG_LEVEL config to override our local logging level is handy.

It also can act as a Null / STDOUT logger when needed, and can do some utility stuff like prepending [Scout] to each message.

Along with the general (Promise API) & integrated (ex. express), it would be great to offer a solution based on async hooks.

There are a more than a few gotchas, however:

• interplay with other async hooks
• nodejs version support
• monkey patching as an integration approach
• cross thread support (possibly some edge cases with the node worker API)

Even with the risk however, the ability to offer tracing and metrics gathering without code changes for the simplest cases is worth checking into.

Allowing specification of a download cache directory & caching the downloads (let's say in /tmp, or at least a well known directory) would likely greatly speed up the e2e tests and help in real world usecases.

It looks like responses don't include the request_id, just so correlation has to be done at the connection/channel level. It's likely not a good idea to create a connection every time for every request lifetime (w/ associated span) so a pool should likely be used.

A deployable link shortening application would be a good target on which to use the nodeJS agent as a sort of evergreen E2E test. With this application it should be possible to log into the scout dashboard and see that the application is performing as expected and that requests/spans are being collected properly.

The link shortener application should be deployed adjacent to the scout website (if possible), or some other webiste that will use it's links, so that the dashboard can produce realistic metrics.

The NodeJS agent should be able to spawn a core-agent child process to run and be controlled by the current process. Some gotchas:

• Ensuring the child process shuts down (gracefully if possible) when the node process exits (process.on)

core-agent needs to be launchable (w/ the start subcommand), with the following options:

• --log-level
• --log-file
• --socket

See command line options for details

Documentation for the lower level Agent needs to be written. It should include at the minimum:

• Architecture break down
• Explanations of applicable interfaces (Agent, AgentDownloader, etc)
• Usage examples
• Model after python README.md

For the express integration:

• How to instantiate the middleware
• Available options, default values
• Basic Usage example
• Advanced usage example (in which the req.scout object gets pulled out and used)

Support ApplicationEvents used for sampling. See documentation.

Similar to python agent's tracked_request.py, annotation for traceback should be done at the stop/close of a span.

The key needs to be stack, and the value should be the array of filenames and line numbers.

The configuration also needs to be double checked to ensure support for the slowness threshhold

The Client-facing API will likely consist of a ScoutAgent object which has at least the following methods:

• startRequest(opts: RequestTraceOptions): Promise<Request>

• stopRequest(req: Request): Promise<Request>

• startSpan(request: Request, opts: SpanTraceOptions): Promise<Span>

• stopSpan(span: Span): Promise<Span>

With the Request object something like the following:

• tag(tags: string[]): Promise<this> (might fail)
• stop(): Promise<this> (might fail)
• `addSpan(SpanTraceOpt

Implementation notes:

• These properties can probably be extracted out to Taggable and Stoppable interfaces, though not strictly necessary
• Spans should probably have a pointer to the request (or at least it's ID) they were started with
• not 100% sure on whether Request/Span should contain the tagging/stopping functionality (this is what python does, so probably good to maintain consistency) or if it should be on the ScoutAgent (or both)
• ScoutAgent should likely maintain a cache of open requests and allow cloning them from the user-facing agent (i.e. tagRequestWithId(...)) (this is likely not a good idea, as it might impose an unexpected memory burden on client code)

Here's what it looks like in python:

span = myAgent.start_span("Template/render")
span.tag("file", "user.mustache")
span.stop

Additionally:

• Span nesting should be super easy -- probably a good idea to just make Span and Request both SpanInjectable or some other class that makes it easy for people to just chain .addSpans
• It should be impossible for a user to forget to close a span and also similarly requests, a default request timeout should be all that's required along with depth-first closing of spans

Make sure the configuration of the Node agent happens in the same order as our Python agent config, and that the configuration option name/keys are identical.

Order is:

• Environment Variables
• In-code config
• Derived values
• Defaults

The first key/value found should be used.

The option names/keys should be identical.

The environment keys should be detected when prefixed with SCOUT_. E.g. env SCOUT_MONITOR=true.

Ensure the configuration values that need to be converted from string into bool or list values are handled properly. See the Python example here.

Based on the review of the library done, there are a bunch of points of improvements that could be made:

- [ ] Remove requirement of waiting in send (here, we can assume that a version with request_id's is in), might make sense to introduce a sendSync just in case someone wants it
I think this is fine -- sendAsync and send are both available, and the change to delay request sending should be enough.

- [ ] Remove connection pool, or limit it to one (multiple connections to the same local socket does not help) The current code is correct, the connection pool is needed due to request/response correlation issues (this will be fixed by request_id being added).

• Add test to ensure that multiple in-flight requests can indeed coexist (this is a sanity test -- just have two concurrent startRequests and end them and make sure nothing goes wrong).

• Implement and add test to ensure that data reading handles possibly buffered output (might have to reduce the socket size to something small to mimic the effect of a partial message)

• Ensure that Scout requests do not immediately attempt to send all their information (save in memory first (request & span creation should be instant)

• Add function to "report"/"settle" requests & spans (which does the actual sending, batched if possible) -- maybe this should be configurable (eager/lazy)

• Remove areas where errors are thrown -- errors from the monitoring client should not take down the client application (an invalid trace/silent failure is better in this case)

• Ensure promises are returned properly to outer promises, check for promises without catches

• Change naming for scout middleware and instance (both are named scout in the examples)

• Return promises in README example

• After calling startChildSpan() getChildSpans() should reflect the right state (this goes hand-in-hand with the change to lazy settlement)

• Remove V1 prefix from requests/response objects

• Ensure 2 levels of stringification are really necessary (when value members are passed the value is stringified then the whole object is stringified)

• Use v4 UUIDs instead of v1 (system time could cause issues)

• Add appropriate types for this.json

• Add a test to ensure that a downstream error (possibly executed when next() is called in the express middleware) does not cause multiple catch block executions in the express middleware

• Since onFinished has a bug, add one more method of closing out requests

• Ensure that timeout code does not still run if the request finishes early (setTimeout is never cancelled)

• Add exact versions for package.json

Looks like 32 bit build platform was left out of initial code, would be good to quickly go through and ensure all supported versions are at least valid values of the Platform enum.

• i686-unknown-linux-gnu
• x86_64-apple-darwin
• x86_64-unknown-linux-gnu

See builds documentation for (possibly changing) complete list

Similar to scout-test-apps there should be an example folder with simple scripts that show how to get up and running quickly with the available scout integrations

Documentation for Express integration needs to be written. It should contain at least:

• How to instantiate the middleware
• Available options, default values
• Basic Usage example
• Advanced usage example (in which the req.scout object gets pulled out and used)

The configuration code currently features filter-params as the default, but right now the code does not process the URLs that it attempts to store.

In the same vein ignore setting is currently collected but not used for filtering the reported route URLs by prefix.

The core-agent binary should be downloadable in a reasonably secure fashion from the S3 buckets specified in the API documentation.

See python agent for example.

Currently routes are re-calculated once per request, see express.ts:

                        // Find routes that match the current URL
                        const matchedRoutes = req.app._router.stack
                            .filter((middleware: any) => {
                                return middleware.route
                                    && middleware.regexp
                                    && middleware.regexp.test(req.url.toString());
                            });

                        // Create a Controller/ span for the request
                        const path = matchedRoutes.length > 0 ? matchedRoutes[0].route.path : "Unknown";
                        const reqMethod = req.method.toUpperCase();

It would be better to cache this value and maybe invalidate the cache based on whether the middleware stack changes (just in case people do dynamic middleware stacks).

Support sending ApplicationEvents that are app metadata (used at startup). See documentation.

This would be a good time to ensure this is accessible through the nodejs agent init phase -- people would likely want to send this information @ init time.

Documentation mentions a feature for the express integration that is about timing out requests (and marking them timed out) -- this should be implemented

Support core-agent registration via the Register command

There needs to be an E2E tests for testing nested spans, current E2E tests only cover leaves.

Implement TagRequest request. See documentation

Load balancers timestamp incoming requests into one of several headers. It indicates the amount of time spent waiting for an available server, and is good for determining if web servers are over or under provisioned.

We need to capture that, find the diff from "now" and record that time.

Record it as a RequestTag, with name: scout.queue_time_ns, and value of the integer number of nanoseconds. Get whatever accuracy is available, and then convert to nanoseconds.

See also the python code: https://github.com/scoutapp/scout_apm_python/blob/b7a9b9a9d2d028906c0ec6e5058ee9b56cb11533/src/scout_apm/core/queue_time.py#L37

See also the ruby code:
https://github.com/scoutapp/scout_apm_ruby/blob/master/lib/scout_apm/layer_converters/request_queue_time_converter.rb

Rename tag to context where appropriate in the application to match the other agents and scout.

Node package publishing scout-agent (and/or scout-agent-express if the projects are split out) could be set up to automatically publish packages to NPM when repository is tagged with some semantic version x.x.x[-beta|rc|...]

The Ruby/Python based agents also record time spent routing -- it would be good to implement this if express will support it.

It would be nice to know the rating for concurrent # of in-flight requests/span-depth (w/ different thread pool settings), and relevant additional memory usage.

Configuration needs to be derived from the following sources,

1. YAML with well-known path
2. Platform specific cfg file (.env is common in the node world, so maybe read scout.env for env vars?)
3. Environment variables

Implement StopSpan request. See documentation

Now that we've got some code written, it would be good to set up the CI