The OTLP export endpoint and Prometheus metrics endpoint both try to bind to port 9090 by default.

OTel collector switched from default of 9090 to 4317, likely because of this conflict.

Inside the sample rules.toml there are sample values (commented out) for setting these:

DryRun = true
...
DryRunFieldName = refinery_kept

If you uncomment them the second needs quotes:

% docker run --rm -it --volume $(pwd):/etc/refinery --expose 8080 --publish 44444:8080 honeycombio/refinery:latest /usr/bin/refinery
unable to load config: While parsing config: (10, 19): no value can start with r

Started the discussion on the pollinators slack, here.

Im trying to use refinery as one of the tools to achieve "trace routing" or "trace multiplexing" capabilities, it is a unusual use case, but here is the context of the ask.

this might be an unconventional use case, and seems like it runs into some problems here, because it uses only traceID to group spans.

Details 👇
Context:
Using istio and a shared ingress gateway for multiple applications/environments/teams (each mapping to its own dataset)
This means i have to split the destination of the traces based on content they carry (not doable in istio alone, or any app)..

What am i trying to do?
Get those istio traces to the correct dataset, by means of:
Shared istio gateway -> otel-collector (fan out to 2 exporters, one for each dataset) -> refinery (rulesbasedSampler - drop traces of other environments/namespaces) -> HC
This effectively duplicates all traces generated by istio, and adds the meta of different datasets to each,
After that, the thought was to drop based on Refinery rules the ones going to the wrong dataset.

What ends up happening?
For a single exporter, it works great.. when multiple exporters are enabled, traces don’t get evaluated properly, some being kept/dropped, seemly in a random fashion.

What i think the problem is:
(No batching is enabled in otel)
When refinery collect the spans in memory and collate them to form a trace here, only the TraceID is taken into account, so this is combining (sometimes) both exporters data, with different datasets as upstream into one single trace, making the sampling decision look wrong.

To The actual questions
Does this sound right? or did i get it all wrong 😃
Was this a conscious decision or i am trying to use the tool on an unexpected use case?

Possible solutions

• Adding sp.Dataset to the cache object key as a prefix should fix it..
• using multiple refinery deployments for each exporter

Would prefer to do 1, but wanted to hear your thoughts before working on a PR 😃

I was looking at the docker hub available image and I see it is a bit old. Would you recommending using the 0.7.0 image that is on docker hub?

Changes since last release:
ba8a5b7...main

If using version 0.7.0 is not recommended, would it be possible to tag a more recent commit as a release so docker hub is updated?

When using Redis for peer discovery, if Redis has in-transit encryption enabled, Samproxy will fail with an i/o timeout:

time="..." level=error msg="registration failed" err="read tcp 172.16.145.223:44362->172.16.4.75:6379: i/o timeout" name="http://ip-000-00-000-000.eu-west-1.compute.internal:8081" timeoutSec=10
time="..." level=error msg="failed to register self with peer store" error="read tcp 172.16.145.223:44362->172.16.4.75:6379: i/o timeout"

This is due to the Redis client used not having the option enabled:
https://github.com/honeycombio/samproxy/blob/7ad0d4372207be9e09d43678d5a7549a76e0ea8b/internal/peer/redis.go#L60-L65

As per Redislabs' "Redis Enterprise and Go" which also uses github.com/garyburd/redigo/redis, this requires specific parameters to be sent when initializing the client:

conn, err := redis.Dial("tcp","<endpoint>:<port>",
    redis.DialPassword("<password>"), // password support would be nice too!
    redis.DialTLSConfig(&tlsConfig),
    redis.DialTLSSkipVerify(true),
    redis.DialUseTLS(true))

Relevant links:

• AWS Elasticache for Redis In-Transit Encryption (TLS) docs: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html
• Redislabs' "Redis Enterprise and Go": https://redislabs.com/lp/redis-enterprise-go/

Hey,

Seems like this is sort of implemented on the metrics, but the values is hardcoded to "" without an option to change it here

Regards,

When running more than 1 refinery cluster there is no way to differentiate the telemetry from each cluster.

Allow the refinery operator to configure attribute name/values that will be added to all metrics and logs emitted by Refinery.

From a hnycon/o11ycon talk on Refinery:

"There are more sample rule types in Refinery than are documented. Which feels like a bad thing to complain about. Like, why complain about more features? But we did have to read the source code to find these. And I might not have thought to look for them if I hadn’t used Legacy Refinery previously and known that some of these were exposed in the UI."

If the environment variable HTTPS_PROXY is set, samproxy should use that proxy to send traffic. In the current state, traffic is sent directly to Honeycomb regardless of the setting of the environment.

Something that we've struggled with since implementing samproxy/refinery is being able to surface infrequently occurring and non-erroneous traces, such as canary deployments or services that just don't receive much traffic.

We've tried experimenting with using additional fields for the sampling key, such as including the service name or a "boost" attribute that can be controlled by the service, but it's mostly resulted in an unpredictable cardinality and throughout of all spans.

Some colleagues (@emauton and @conormcd) had an idea that was inspired by Fred Hebert's Recon to provide a mechanism of forcing the capture of all interesting spans for a limited time period. For example, with a hypothetical CLI and API, you'd be able to:

$ capture-traces 30 service:circle-www-api-canary1
or
$ capture-traces 15 service:circle-www-api-v1 name:circle.permissions/user-can-view-builds

There's some similarity with the LaunchDarkly proposal that was discussed in Slack.

What do you think about logging stacktraces when they are caught by the middleware?

#61 took a bit more effort to debug because I had to rebuild and ship a copy of Docker image that called debug.PrintStack() to figure out where the panic was coming from. Dumping straight to STDERR amongst structured logging is pretty messy though. Should each of the frames be formatted as a logging field?

Should this behaviour always be enabled, or could it generate too many logs and would stopping-the-world be too detrimental to non-failing handlers?

I'd be happy to raise a PR once we've figured out the implementation details.

config:

[PeerManagement]
Peers = []
Type = "redis"
RedisHost = "example.com"

unable to load config: Key: 'configContents.PeerManagement.Peers' Error:Field validation for 'Peers' failed on the 'required' tag

if set to the default value (["http://127.0.0.1:8081"]) it will be overwritten by the peer list from redis (as expected)

Refinery's Honeycomb Metrics are sending two or three events with the same exact timestamp (to the millisecond) but only one of the events has fields populated (the other one or two events are empty)

Version: Refinery 1.2.1 (AMD64)

Handling of int64 sample rates was added in #209. We should extend the tests in route_test.go to test int64 sample rates are handled as expected and are not accidentally broken in the future.

Add test for the following cases:

• int64 value lower/equal to than int32 max - uses sample rate value
• int64 value greater than int32 max - uses int32 max value

In GKE, pods can be resolved using their FQDN (if a headless service is set up) but unless I'm missing something I think the self-reported hostnames don't resolve?

I've forked the project to use FQDN in my own deployment and everything works perfectly. Is it was worth introducing this as a configurable option?

I would like to suggest adding a counter metric that is incremented when TraceTimeout is hit.

We are trying to tune SendDelay to avoid spans arriving after the trace has already been flushed from the Samproxy instance. We know that we have at least some spans arriving late because we see trace_sent_cache_hit being incremented. What we don't know is how often spans arrive after the trace has been evicted from the sent trace cache.

We know in this case that the span will be treated by Samproxy as a new trace. We also know that the root span for the trace has already come and gone, so the "new" trace will only be flushed when TraceTimeout is hit. If Samproxy provided a trace timeout metric then we'd have a proxy metric for late arriving spans.

Note, this metric would also be valuable for us in setting CacheCapacity as we don't know what portion of the cache is currently going towards caching these late arriving spans.

gRPC has a set of ServerParameters that we should make configurable.

Specifically, the MaxConnectionAge and MaxConnectionAgeGrace are of interest to help keep gRPC connections balanced across a pool of refinery hosts.

An issue can arise when a host dies that all the connections will failover to the other hosts and remain there due to the long-lived nature of gRPC connections. The new host that spins up to replace the dead host will never receive any incoming connections and is then underutilized while the other hosts are overburdened.

Are the images published to Docker Hub "supported" for public consumption?

• https://hub.docker.com/r/honeycombio/samproxy/tags

Are they built and pushed manually at the moment? If so, could they be automated by CI, at the same time as creating a GitHub release, so that they remain consistent?

This is related to #265 .

It would have been a LOT easier to understand why the rules were not being applied/evaluated properly if we had the indication that those are not supported.

This does not need to fail the parse, or could, this depends on the team preference for those failure modes. Let me know what you guys would prefer 👍

Regards,

My docker build fails because it cannot find package "github.com/vmihailenco/msgpack/v4". Have you experienced this issue? Any advice on how to resolve it?

docker build -t samproxy .
Sending build context to Docker daemon  9.204MB
Step 1/6 : FROM golang:alpine
alpine: Pulling from library/golang
df20fa9351a1: Already exists 
ed8968b2872e: Pull complete 
a92cc7c5fd73: Pull complete 
00abe362642d: Pull complete 
e0491c15e88f: Pull complete 
Digest: sha256:d6deb50437547fd7226890aa19b075fbf7f5063e3c0d7a2eaf7fb41d11069013
Status: Downloaded newer image for golang:alpine
 ---> 30df784d6206
Step 2/6 : RUN apk add --update --no-cache git
 ---> Running in 74496b0f0285
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/5) Installing nghttp2-libs (1.41.0-r0)
(2/5) Installing libcurl (7.69.1-r0)
(3/5) Installing expat (2.2.9-r1)
(4/5) Installing pcre2 (10.35-r0)
(5/5) Installing git (2.26.2-r0)
Executing busybox-1.31.1-r16.trigger
OK: 22 MiB in 20 packages
Removing intermediate container 74496b0f0285
 ---> 4e7c00c4eeb8
Step 3/6 : RUN go get github.com/honeycombio/samproxy/cmd/samproxy
 ---> Running in 7dc3b072ccf1
package github.com/vmihailenco/msgpack/v4: cannot find package "github.com/vmihailenco/msgpack/v4" in any of:
        /usr/local/go/src/github.com/vmihailenco/msgpack/v4 (from $GOROOT)
        /go/src/github.com/vmihailenco/msgpack/v4 (from $GOPATH)
The command '/bin/sh -c go get github.com/honeycombio/samproxy/cmd/samproxy' returned a non-zero code: 1

Libhoney-go supports output in msgpack format. We'll want to support receipt of this in samproxy. In general, this should give us higher throughput.

https://circleci.com/blog/building-docker-images-for-multiple-os-architectures/

we build arm64 binary but currently don't build arm64 docker images. circleci builders are mostly amd64 at the moment so...

We have traces that vary in in length substantially, and want to use TotalThroughputSampler to help us stay within rate limits, but having it sample in terms of traces per second results in too much variability in the output events per second.

Calls to the underlying dynsampler are done one per trace, which to my understanding increments the internal counter once per trace:

https://github.com/honeycombio/refinery/blob/v1.4.0/sample/totalthroughput.go#L58

To make this in terms of events per second, I see the only way to do this would be to call that line above in a loop, where the iterations == the number of spans in the trace, but that would be a pretty inefficient way of making this work. What approach would you recommend?

We have a theory that the memory problems that we're seeing in #94 may be a result of larger traces (by quantity and size of spans) being cached for longer during our low traffic periods. It occurred to us that traces (and their spans) aren't currently removed from the cache and eligible for GC until their position in the ring buffer is used again, regardless of whether the trace has already been sent.

Could traces be removed from the cache at the time of sending? Or, if the concurrent access is likely to cause race conditions, maybe zero out the references to spans, as they aren't size bound by the CacheCapacity?

Refinery Metric Events are running in three separate go routines (metrics, upstreamMetrics, and peerMetrics) which can lead to instances where a single millisecond has three different metrics events published (each containing their own data along with the dynamic fields).

Ideally, these three different events would get pre-aggregated into a single coherent event.

Originally opened as #293, which was closed after further investigation.

It'd be great to have some better metrics about how the cache is being used, both to use when sizing the cache and investigating memory usage. We'd particularly like to know how many traces in the cache are sent/free vs unsent/used and how many spans they contain. This might only be possible by periodically iterating through the cache - would you foresee this causing any performance problems?

We've seen some strange increases in memory usage that seem to correspond with … 🤔 … less traffic. We're running 10 nodes with a CacheCapacity of 150k each. They normally consume 6.5-7G of memory each, so we've set a limit of 8G. On two occasions now they've ramped up to the limit and have been OOM killed. This appears to repeat until the traffic levels return to normal.

We could increase the memory limit, but it's not clear how much would be required and we haven't been able to reproduce it within our control. We've thought about capturing pprof data from #82 but we're not sure that it will be possible in the small timeframe before the OOM. Is there anything else we should be looking at?

Metrics from during an incident:

Metrics from over a weekend (#89 was causing some additional restarts):

My org only deploys PaaS-y instances of Redis, specifically Azure Cache for Redis and I'd like to be able to use an instance from that service to support Refinery.

This would require the ability to specify a password and to require TLS/SSL. I'll poke around to see if I can assist but it felt useful to open an issue first.

It seems refinery checks the API host option in outgoing events to see if the message is destined for a peer, or the honeycomb API:

During a conversation in the pollinators slack we noticed that log lines emitted by refinery for some messages showed that the event's api_host option was empty, even though the event was being sent to honeycomb:

Apr 01 12:13:01 production-refinery-i-00c73eea958d0b46c refinery[11758]: time="2021-04-01T12:13:01Z" level=error msg="non-20x response when sending event" api_host= dataset= error="Post \"https://api.honeycomb.io/1/batch/production.traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" event_type= status_code=0 target=

This caused timeout errors in the upstream honeycomb API to be treated as peer errors:

This is a bit misleading as it causes the operator to believe there's a problem with their cluster, when the issue is really coming from the honeycomb API.

Given metrics are already prefixed with upstream_ or peer_, I'm wondering whether we really need the _api and _peer suffixes, as it's a bit confusing. e.g. How should upstream_response_errors_peer differ from peer_response_errors_peer?

Would it be possible to remove this suffix so that operators can determine whether a metric is related to the honeycomb API, or the peer, by its prefix?

Right now there is no option to use InsecureSkipVerify. It would be nice if this could be a config option, like UseTLS.

The addition of SkipVerify in the options here is a noop as InsecureSkipVeridy is only set if tlsConfig is nil here.

I proposed a solution in #249, in this commit e1620f0. @vreynolds requested our use case for InsecureSkipVerify. Heroku Redis uses self signed certs so in order to run refinery in a Heroku private/shield space along with our application we need to set InsecureSkipVerify=true. This will also apply to Heroku's customers.

The default bucket sizes for Prometheus histograms are too small to usefully represent metrics like trace_duration_ms or trace_span_count. I imagine that one size will never suit all and that all of the histograms need to be reviewed independently for appropriate values.

It looks as if span.Events is never looked at here, meaning span events are just dropped on the floor 😢

https://github.com/honeycombio/refinery/blob/main/route/route.go#L446-L478

We've been monitoring our AWS bill as we've adopted refinery and one thing we've been surprised by is the jump in our AWS inter-az data transfer costs, which are more significant than we expected. We're currently running 3 c5.2xlarge instances of refinery, in separate AZs, and each one seems to be sending ~7MB/s of data to its other 2 peers. Some back of the envelope math suggests this could generate a bill of (7MB/s * 3600 * 24 * 30)/1000MB per GB * $0.02/GB = $362.88 for 1 node to talk to one peer. Given each node has 2 peers, and there are three nodes, it seems plausible that the bandwidth alone could cost in the ballpark of $2,172 - about 3 times the cost of running the cluster, without even considering the cost of transmitting the sampled-in data to Honeycomb.

While looking through the source code I noticed that refinery explicitly opts out of using gzip compression when communicating with peers. Do you have any context for why that decision was made (e.g. how much overhead did it create), and whether it would be possible to make this value configurable?

Our cluster is currently running at about 30-40% CPU utilization, so I'd much rather use a few more CPU cycles if it could save us a fair chunk on our bandwidth bill.

To make querying easier, it would be good if we tagged each event that flows through refinery with a specific field.

We have derived columns that can attempt to work it out that utilise the user agent. These would also need to be updated to look for the event field.

The other day in the pollinators slack several folks noticed that their refinery clusters all had buffer overruns at the same time. After a bit of discussion it came to light that the honeycomb API had been having a bit of a slowdown around that time, which caused requests to the honeycomb API to stall and put backpressure on refinery's cache.

It seems like the metrics that honeycomb emits don't include statistics about how long upstream API requests are taking - would it be possible to add support for these metrics so that operators can understand if performance/capacity issues are related to the honeycomb API?

On a related note, it appears that libhoney configures a 60s timeout on all upstream requests. Given the decision to change the timeout from 10 seconds to 60 seconds was made 2 years ago I was wondering if that's still an appropriate timeout, or if a lower value would be more suitable?

For reference, here're some charts of what our refinery cluster looked like during the incident. You can see how our refinery instance started dropping incoming spans almost as soon as the API slowdown occurred (the upstream_response_errors_peer is actually for the honeycomb API, not refinery peers - see #244). The point where memory usage drops is where the instance OOMs and restarts:

As an aside, it's not really clear to me why the collect_cache_entries_max metric just disappears as soon as the system starts experiencing backpressure - is that something you would expect to happen?

As span events in the past #258, span links are not taken into account.

https://github.com/honeycombio/refinery/blob/main/route/route.go#L446-L478

Right now there's no way of verifying that our binaries have not been tampered with after leaving circleci (e.g. by someone malicious who has access to our github account)

In the honeycomb-hosted refinery product there was a meta.dynamic_sampler_key field that was added to the root span of every trace that went through the dynamic sampler, and contained the sampling key calculated from all spans in the trace. This field was really helpful for evaluating different sampling configurations, and without it, it's really hard for us to understand the decisions the sampler is making.

Is it possible to have this field added to the open source version of refinery?

Expected Behavior

A Refinery process is able to receive an OTLP span, encode it into a Span event with a trace ID, and use the sharder to route it to the correct Refinery member.

Current Behavior

The Refinery process segfaults and is killed when, as part of creating the Event, it tries to access an uninitialized Status field of a span:

refinery-7666b8c94d-k2hkp refinery panic: runtime error: invalid memory address or nil pointer dereference
refinery-7666b8c94d-k2hkp refinery [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xbc6844]
refinery-7666b8c94d-k2hkp refinery
refinery-7666b8c94d-k2hkp refinery goroutine 53981 [running]:
refinery-7666b8c94d-k2hkp refinery github.com/honeycombio/refinery/route.(*Router).Export(0xc000332020, 0xed4d78, 0xc00095bfb0, 0xc0003ac9c0, 0xc000332020, 0xc00095bfb0, 0xc00024bba0)
refinery-7666b8c94d-k2hkp refinery 	/app/route/route.go:452 +0x944
refinery-7666b8c94d-k2hkp refinery github.com/honeycombio/refinery/internal/opentelemetry-proto-gen/collector/trace/v1._TraceService_Export_Handler(0xd901a0, 0xc000332020, 0xed4d78, 0xc00095bfb0, 0xc0000b2ae0, 0x0, 0xed4d78, 0xc00095bfb0, 0xc000224000, 0x10db)
refinery-7666b8c94d-k2hkp refinery 	/app/internal/opentelemetry-proto-gen/collector/trace/v1/trace_service.pb.go:190 +0x214
refinery-7666b8c94d-k2hkp refinery google.golang.org/grpc.(*Server).processUnaryRPC(0xc000430700, 0xedbb58, 0xc000b56000, 0xc0009a3000, 0xc0004dea20, 0x1399250, 0x0, 0x0, 0x0)
refinery-7666b8c94d-k2hkp refinery 	/go/pkg/mod/google.golang.org/[email protected]/server.go:1194 +0x52b
refinery-7666b8c94d-k2hkp refinery google.golang.org/grpc.(*Server).handleStream(0xc000430700, 0xedbb58, 0xc000b56000, 0xc0009a3000, 0x0)
refinery-7666b8c94d-k2hkp refinery 	/go/pkg/mod/google.golang.org/[email protected]/server.go:1517 +0xd05
refinery-7666b8c94d-k2hkp refinery google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000b966f0, 0xc000430700, 0xedbb58, 0xc000b56000, 0xc0009a3000)
refinery-7666b8c94d-k2hkp refinery 	/go/pkg/mod/google.golang.org/[email protected]/server.go:859 +0xab
refinery-7666b8c94d-k2hkp refinery created by google.golang.org/grpc.(*Server).serveStreams.func1
refinery-7666b8c94d-k2hkp refinery 	/go/pkg/mod/google.golang.org/[email protected]/server.go:857 +0x1fd

Possible Solution

• PR #223

Steps to Reproduce

1. An otel-collector configured with the OTLP exporter to forward spans to Refinery:

    exporters:
      otlp/refinery:
        endpoint: "refinery.tracing.svc.cluster.local:9090"
        insecure: true
        headers:
          x-honeycomb-team: some-key
          x-honeycomb-dataset: some-dataset

2. A Refinery service running on the endpoint configured above.

Context (Environment)

• The issue is present on the following versions:
  • honeycombio/refinery:1.0.0
  • otel/opentelemetry-collector-contrib:0.11.0

Detailed Description

The Opentelemetry spec defines the Status attribue of a span as only optional.

The PR referenced above guards against a potential undefined memory access by defensively checking for a null pointer dereference.

Hello Refinery team! 👋

When deploying refinery on Kubernetes with multiple instances and file-based peer management, during the initialization phase where refinery tries to lookup it’s own hostname in the peer list, the lookup fails with “no such host” exception (because the first address in the peer list is not this specific instance, and the chosen address is not initialized yet).
When this occurs, the instance is then terminated.

I can see that when the lookup fails for an address, no retry mecanism is implemented here: https://github.com/honeycombio/refinery/blob/main/sharder/deterministic.go#L133

To add info, I used the official helm chart provided https://github.com/honeycombio/helm-charts/tree/main/charts/refinery, but changed the deployment type to a StatefulSet with a headless service (to have stable hostnames).

Please tell me if I am missing something here, maybe it's due to a lack of understanding on my part.

if the issue is real however, it would me my pleasure to contribute!

I want to be able to have rules based sampling then fall back to the dynamic sampler when not matched.

Though we want to encourage the use of /1/batch, we should still cover this gap.

With honeycombio/dynsampler-go#10 we got an exponential decaying average sampler algorithm - a huge improvement over the window-based dynsampler. We should add using it to a config and maybe even switch to it as the default.

Hej,

tldr; make all configuration properties available via environment variables to provide the means to dynamically adjust Refinery

we are introducing Refinery into our setup and stumbled upon the problem that the environment variables are not evaluated within the configuration file.

For example a configuration like following

 18 [PeerManagement]
 19 Type = "redis"
 20 RedisHost =  "refinery-leader-redis-${ENVIRONMENT}.mydomain:6379"

will not work. (I don't know if this is the behaviour of viper).

Looking at the implementation of Refinery we can see that only four environment variables are exposed for usage

Taking the 12factor principle into account, I can't think about a design decision why not all properties are exposed through environment variables. It seems to be possible with viper by defining a SetEnvPrefix and running AutomaticEnv.

A use case in our scenario is following:

• we are running different environments (e.g. dev, stage, prod)
• each of them uses a dedicated Refinery instance (plus redis)
• we want to send the metrics of Refinery to our analyse backend but into different datasets
  --> at this moment it is not possible to do this without baking in the different dataset names

Thanks for any input and maybe I am missing something simple here :)

Hello,

Seems like not all ui/api operators from the platform are supported on the RuleBasedSampler, here.

When using one of the unsupported operators, there is no indication of error, it just doesn't match the desired trace/span.
It is also not documented anywhere (that i could find).

Im looking specifically for that one, but would be nice to have them all =)

Regards,

I was looking to use this sampler in https://github.com/honeycombio/dynsampler-go -- however it doesn't seem like this is currently recognized by samproxy.

When using RulesBasedSampler, some long running traces can be evaluated while they are still incomplete (missing root span), this ends up creating results that could unexpected by the users.

A workaround is to have longer TraceTimeout but when those traces are not the norm, this could be detrimental to the size of cache and the the inner workings of sample/transmit on other traces that might be broken for other reasons.

Since the information that the trace is incomplete already exists on the trace object, one could just add a log indication or a field to the spans that the evaluation was executed on like refinery_incomplete or refinery_evaluated.

Let me know if you have any concerns or preferences on the implementation.

PS. There are metrics for these, but those don't help the users knowing the why/what, only showing that it happens, sometime, somewhere.

A common question from customers is how the sampling goal works with the EMADSaampler. This ticket is to document that relationship, along with metrics to look at to understand it.