When log.file is set to stderr, all log messages are duplicated in pretty and not pretty format.

@DariaKunoichi,
agent_group is kind of redundant here and here in the heartbeat spec since it's already present in the header message

Originally posted by @tanveergill in #92 (comment)

Running make generate-docs (which is run via pre-commit) changes svg files despite no changes in .mmd files.

Some input ports are required, some are optional. We need required annotation for required ports, so validator can check them and we can display in docs

As discussed on slack with @tanveergill

The goal of this ticket is to update the data format which we're sending in logs and traces.

To do that the following changes should be applies to the metricsprocessor:

1. Add workloads and dropping_workloads attributes with format:

   policy_name:<policy_name>,component_index:<component_index>,workload_index:<workload_index>,policy_hash:<policy_hash>
   

2. Change concurrency_limiters and dropping_concurrecy_limiters format to:

   policy_name:<policy_name>,component_index:<component_index>,policy_hash:<policy_hash>
   

3. Change flux_meters format to:

   <flux_meter_name>
   

4. Add classifiers attribute with format (depends on #268):

   policy_name:<policy_name>,classifier_index:<classifier_index>
   

Useful links:

1. Function in metricsprocessor which needs to be changed

We don't want to provide Istio and Envoy Filter as chart dependencies and will be assuming that it would be already present on user's infrastructure or they'll do the installation on their own.

Describe the solution you'd like

• The change applies to following fieds:
  • The operator field in ArithmeticCombinator
  • The operator field in Decider
  • The traffic field in ControlPoint
• Use formal protobuf enums instead of strings. The last time I tried, Buf was imposing unreasonable restrictions e.g. first field needs to be unspecified and all fields need to begin with the same prefix etc. See if we can get around these constraints in Buf.
• Make sure the user facing spec is similar, in the sense that they still specify a human readable string in the YAML as the operator.

Describe alternatives you've considered

• Retain string based fields. If the enum path is un-tractable for whatever reason we can stay with the string based approach.

Additional context

• The YAML spec still should not be too off from what's present currently
• This change simplifies the unmarshal code.
• Easier to document enum fields than string values
• Validation is built-in

Please delete paragraphs that you did not use before submitting.

Describe the solution you'd like

• Put matched FlowLabelKeys in flowcontrolv1.CheckResponse
• Make sure the FlowLableKeys propagate to Envoy logs and SDK traces via AuthAPI dynamic metadata and trace header respectively
• OTEL metrics processing pipeline reads FlowLabelKeys from logs and traces and adds a multi-value field flow_label_keys in Druid events. Refer how other multi-value fields such as flux_meters get populated.

Rationale

• Users need feedback to create Selector. It's hard to write one without knowing labels are available. Exposing flow_label_keys in Druid is a quick and effective way to achieve that.

• Send agent key as a label on sentry events

• Sentry plugin needs to subscribe to agent key from FN Cloud plugin via Fx

• Refactor of Crashwriter included: Keep raw zerolog events in the CrashWriter circular buffer and only do the sentry parsing when panic happens
  (i.e. Convert zerolog to sentry only when it's ready to send those logs to sentry to reduce the unnecessary processing)

Migrate the demoapp and bookinfo inside the aperture repo.
Also, migrate the aperture-operator in aperture repo to remove the symlinks.

Sentry Issue: OSS-APERTURE-9

runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/src/plugins/service/aperture-plugin-sentry/sentry/sentrywriter.go", line 223, in (*SentryWriter).SentryPanicHandler
  File "/src/pkg/panichandler/panichandler.go", line 93, in (*PanicHandlerRegistry).Crash.func1.1

Is your feature request related to a problem? Please describe.

• There is repetition of metric names, labels in the codebase which is hard to maintain.

Describe the solution you'd like

• Strings for metrics, labels key and label values can be defined in a common file shared by both agent and controller under pkg/metrics/schema.go
• The advantages of this approach:
  • Fewer chances of errors via typos
  • Changing metric names and labels is as easy as changing it one place
  • Possible to map dependencies of a metric/label via LSPs and code indexers like ctags

Additional context

• Example references to metric, labels which need to be referred from schema:
  • flux_meter_name
  • Default workload's index

• Optimize status registry for writing instead of reading
  This change will get rid of flat maps that were being maintained on every write.

• Create a new registry updater object when NewRegistry is called, making each object hold its own tree

The goal of this issue is to fill README.md file with all the needed information, i.e.:

• What is aperture?
• Resources - docs, SDKs etc.
• Public roadmap
• How to report issue
• How to contribute
• Slack community link

Some of the example code from our autogenerated markdown files seem broken:

Example:

````yaml
selecto…
* [v1ControlPoint](#v1-control-point) – Identifies control point within a service that the rule or policy should apply t…

there is even some embedded HTLM code later in the same code block.

Or this one:

Example:

```yaml
Example of Declarative JSON extractor:

Add some extra buttons to make tilt interactions more enjoyable:

• Button to stop the traffic (as it is running the scenario continuously by default).
• Button to start the traffic.
• Button to remove the default policy (as it's loaded by default).
• Button to add the default policy.
• Button to open Grafana UI -- also see if we can open it by default when we open Tilt UI.
• Buttons to add/remove a classification rule (though default policy is working on headers).

• Migrate crash writer into Sentry
• Keep the buffer of last 100 Sentry events (not sending them until the Crash happens)
• When the Crash happens pick up and make buffered sentry events, dump status registry, (service version information - what we are currently collecting in the platform onCrash) into Sentry event and send them in the large event of Crash (single Crash event is sent to Sentry)

Migrate back to descriptions in comments – this will cause the title field to appear in swagger some times – check if this isn't causing a regression.

Currently, due to https://github.com/FluxNinja/RM/issues/5402, we cannot install the controller in any other namespace than aperture-system.

Modify the installation steps to mention the above requirement and add requires steps for the same.

• Add test for basic and prefix notifier, ran tests on default trackers
  through channels, created write events with a transform function that
  reverse the content of any given bytes

• Add more notifiers having same id, or generating random id to check
  corner case scenarios

• Add tests for unmarshaller notfier, retrieved keyNotifier trough Get
  function and through constructor to check fuctionality of both
  functions.

• Run tests function creates a default tracker, starts it, add's all
  prefix notfiers, key notifiers and events, sleeps for a tiny amount of
  time so that events can be written and then matches the content of
  bytes with expected results. Lastly, it removes the prefix, key
  notifiers and events.

Split from #143. That task handled adding comments to most of the components, this task is about documenting Ratelimiter and Scheduler (actuators)

See eg. https://aperture-tech-docs.netlify.app/docs/development/reference/configuration/policies#v1-rule. Since rego and extractor are required (mutually exclusive) variants of rule, they should go first and optional properties should go last. Now there's a sandwich.

Or see https://aperture-tech-docs.netlify.app/docs/development/reference/configuration/policies#-v1scheduler – the first fields are auto_tokens and default_workload, which require understanding workloads first, which is the last field.

Order of properties in docs should be the same as in definition. See envoy docs. This allows to control the reading order and put more important stuff on front.

This bug was fixed some time ago (https://github.com/fluxninja/cloud/pull/4349) (before moving to github issues), but now I see it was reintroduced.

Issue copied from https://fluxninja.atlassian.net/browse/FLUX-2487
Created by: krzysztof.klimonda

Describe the solution you'd like

• Eliminate GetAgentGroup function from Limiter and FluxMeter interfaces and implementations
• Wherever required, get the AgentGroup from agentinfo.AgentInfo instead

We are missing TLS client configuration for K8S client that we use for discovery. We need to take this task as a priority.

Make sure this config is exposed in the operator etc. as well.

Describe the bug
Agent should not be aware of deployment mode; Configuration field names should be understandable in the context of the agent itself. Thus, the "sidecar_mode" configuration field should be removed and its functionalities moved to new fields.

Steps to reproduce
Just look at the code

What did you expect to see?
Configuration with fields unrelated to deployment mode

What did you see instead?
Configuration with fields related to deployment mode

What version did you use?
ce556a3

What config did you use?

Environment

Additional context

1. When we run k6 in demo app testing, only one of the pods gets the traffic.

• Hold off on the daemonset for demo app, it might mess up the k6 tuning (i.e. Need different number of VUs to saturate the app if we change replica counts)
• Use affinity rules in k8s to make sure only one pod of deployment runs per worker (podAntiAffinity rule based on labels)

2. When the traffic goes to all the pods, test the rate limiter in the distributed mode.

Right now some information is repeated in docs of a Component and the Policy.

Decide what should go where and deduplicate docs (inserting links where appropriate)

Discovered when working on #143

We need to separate the Agent and Controller from the current Aperture repo as there can be scenarios where user might want to install the agent in one cluster and controller in other.

Describe the solution you'd like

• FluxMeters change from the existing list to a map where the key of the map is the FluxMeter name. Remove the explicit name parameter in FluxMeter spec.
• Update the FluxMeter parsing code on the cloud side in a separate PR.
• Update the Aperture Policy Jsonnet lib and Mixin recipes, cc: @fluxninja/devops

Describe alternatives you've considered

• Retain the existing semantics of defining the FluxMeter name as an explicit field in the FluxMeter message.

Additional context

• This will make it consistent with classification rule set.
• User defined keys are usually defined in a map format in other projects e.g. CircleCI jobs, Kubernetes labels
• Kubernetes has complex requirements that the label key should adhere to. Similarly, FluxMeter name needs to be a valid Prometheus metric name format. Please document this via the Proto spec.

When official docs page is public, it should be linked in the README.md instead of local docs directory.

Right now it display as:

we should:

• mark clearly that it's validation constraint
• link to https://pkg.go.dev/github.com/go-playground/validator/v10 to explain syntax

As reported on slack by @harjotgill

Describe the solution you'd like

• Auto-generate documentation for Prometheus metrics using golang code annotations:
  • May use metrics2docs tool from GrafanaLabs
  • Integrate auto-generated documentation for metrics with Aperture Docs

It should deploy k6-operator with a test traffic pattern that can be used to test aperture in local development environment based on our current tests.

Currently,
$$gradient = \frac{setpoint \cdot tolerance}{signal}$$

Proposal:

$$ gradient = \left(\frac{signal}{setpoint}\right)^{slope} $$

Rationale:

• current tolerance parameter is just pre-multiplying setpoint and could be done outside using math mul component,
• slope parameter can be used to fine-tune the "aggresiveness" / slope of the gradient,
• current design allows only to use the gradient controller for cases where signal is positively correlated with control variable.

Migration:

• add a mul node if $\text{tolerance} \neq 1$
• set the slope to $-1$.

TODO

• code
• proto docs
• example policies using this controller (wherever they are commited)

As discussed with @tanveergill

I tried to run the rate limiter policy with different scenarios as below but the request decisions where not as expected.

1. I used the below rate limiter policy where I changed the limit value to 100 requests/minute per user based on the user header but the drop rate is very high and very few requests got accepted.

kind: ConfigMap
metadata:
  name: flowcontrol
  namespace: aperture-system
  labels:
    aperture.tech/validate: "true"
apiVersion: v1
data:
  rate-limit-policy.yaml: |
    evaluation_interval: "0.5s"
    circuit:
      - constant:
          value: "100.0"
          out_ports:
            output:
              signal_name: "RATE_LIMIT"
      - rate_limiter:
          in_ports:
            limit:
              signal_name: "RATE_LIMIT"
          selector:
            service: "demo1-demo-app.demoapp.svc.cluster.local"
            control_point:
              traffic: "ingress"
          label_key: "request_header_user"
          limit_reset_interval: "60s"

I have used the same load-test.js using K6 available on main and tilt under demoapp1 pod.

Below is the output from Druid:

2. In this scenario, I have configured the classification for demoapp using the example available on main and changed the policy as below:

kind: ConfigMap
metadata:
  name: flowcontrol
  namespace: aperture-system
  labels:
    aperture.tech/validate: "true"
apiVersion: v1
data:
  rate-limit-policy.yaml: |
    evaluation_interval: "0.5s"
    circuit:
      - constant:
          value: "100.0"
          out_ports:
            output:
              signal_name: "RATE_LIMIT"
      - rate_limiter:
          in_ports:
            limit:
              signal_name: "RATE_LIMIT"
          selector:
            service: "demo1-demo-app.demoapp.svc.cluster.local"
            control_point:
              traffic: "ingress"
          label_key: "request_header_flow_user"
          limit_reset_interval: "60s"

In this case, all the requests got accepted and none got dropped, which I feel is also not correct as I tried different values like 10, 50 etc and still it was accepting all. I tried flow_user as value for label_key and that too had same output.

Describe the solution you'd like

• OTEL metrics processing pipeline needs to parse the status from Span and put it into Druid events.
  • Span in metricsprocessor component
  • Method to get span status

Additional context

• This is the Feature control point equivalent of status_code and status_message fields present in HTTP based control points.
• Status got added to Spans in this PR for golang SDK

Currently selector docs are duplicated in the classification and policies pages.

We should have a single, properly cross-linked page, in which we can just describe the selector and label matcher.

It would be nice to be able to deploy/destroy kind cluster (single and multi node) from a script, that then can be used to deploy aperture with tilt for local development.

Perhaps https://goreleaser.com/ can be made to work with our building setup?

Current CrashWriter's manual circular buffer using array is not garbage collector friendly - not optimized.
Take another circular buffer library which is optimized with the go garbage collector and update the crash writer.

To prevent log flood, like one fixed in #260, some log sampling needs to be introduced.

BurstSampler should be introduced to the platform logger, with appropriate config. See https://pkg.go.dev/github.com/rs/zerolog#section-readme

What

• Takes policylangv1.Policy as input
• Invokes controlplane.CompilePolicy to compile Policy
• Returns output to stdout in dot file format
• Refer #179

I was running the latency-gradient policy with load test and observed that the drop rate for requests from different workload having different priorities are almost same. Below are 2 scenarios with I checked:

1. Executed the load test with 2 users having different workload config and different priorities in policy but their drop_rate was same:

apiVersion: v1
kind: ConfigMap
metadata:
  name: classification
  namespace: aperture-system
  labels:
    aperture.tech/validate: "true"
data:
  default.classification.yaml: |
    # Example of Flow Classification rules file, showcasing using "extractors" and raw rego.

    selector:
      service: demo1-demo-app.demoapp.svc.cluster.local
      control_point: { traffic: ingress }

    rules:
      # An example rule using extractor.
      # See following RFC for list of available extractors and their syntax.
      # https://www.notion.so/YAML-classification-policies-extractors-a7f36469fdf14c1bbc79c6ba963680a5#9ae103e0c9034c4aba30228d05d91391
      ua:
        extractor:
          from: request.http.headers.user-agent

      # The same rule using raw rego. Requires specifying rego source code and a query
      also-ua:
        rego:
          source: |
            package my.rego.pkg
            import input.attributes.request.http
            ua = http.headers["user-agent"]
          query: data.my.rego.pkg.ua

      user:
        extractor:
          from: request.http.headers.user

kind: ConfigMap
metadata:
  name: flowcontrol
  namespace: aperture-system
  labels:
    aperture.tech/validate: "true"
apiVersion: v1
data:
  latency-gradient.yaml: |
    evaluation_interval: "0.5s"
    flux_meters:
      - name: "demo1_latency"
        selector:
          service: "demo1-demo-app.demoapp.svc.cluster.local"
          control_point:
            traffic: "ingress"
    circuit:
      - promql:
          query_string: "sum(increase(demo1_latency_sum[5s]))/sum(increase(demo1_latency_count[5s]))"
          evaluation_interval: "1s"
          out_ports:
            output:
              signal_name: "LATENCY"
      - constant:
          value: "2.0"
          out_ports:
            output:
              signal_name: "EMA_LIMIT_MULTIPLIER"
      - arithmetic_combinator:
          operator: "mul"
          in_ports:
            lhs:
              signal_name: "LATENCY"
            rhs:
              signal_name: "EMA_LIMIT_MULTIPLIER"
          out_ports:
            output:
              signal_name: "MAX_EMA"
      - ema:
          ema_window: "300s"
          warm_up_window: "10s"
          correction_factor_on_max_envelope_violation: "0.95"
          in_ports:
            input:
              signal_name: "LATENCY"
            max_envelope:
              signal_name: "MAX_EMA"
          out_ports:
            output:
              signal_name: "LATENCY_EMA"
      - gradient_controller:
          tolerance: "1.1"
          min_gradient: "0.1"
          max_gradient: "1.0"
          in_ports:
            signal:
              signal_name: "LATENCY"
            setpoint:
              signal_name: "LATENCY_EMA"
            max:
              signal_name: "MAX_CONCURRENCY"
            control_variable:
              signal_name: "ACCEPTED_CONCURRENCY"
            optimize:
              signal_name: "CONCURRENCY_INCREMENT"
          out_ports:
            output:
              signal_name: "DESIRED_CONCURRENCY"
      - arithmetic_combinator:
          operator: "sub"
          in_ports:
            lhs:
              signal_name: "INCOMING_CONCURRENCY"
            rhs:
              signal_name: "DESIRED_CONCURRENCY"
          out_ports:
            output:
              signal_name: "DELTA_CONCURRENCY"
      - arithmetic_combinator:
          operator: "div"
          in_ports:
            lhs:
              signal_name: "DELTA_CONCURRENCY"
            rhs:
              signal_name: "INCOMING_CONCURRENCY"
          out_ports:
            output:
              signal_name: "LSF"
      - concurrency_limiter:
          scheduler:
            selector:
              service: "demo1-demo-app.demoapp.svc.cluster.local"
              control_point:
                traffic: "ingress"
            workload_config:
              label_key: "flow_user"
              auto_tokens: true
              default_workload:
                priority: 20
                timeout: "0.005s"
              workloads:
                - label_value: "A"
                  priority: 50
                  timeout: "0.005s"
                - label_value: "B"
                  priority: 200
                  timeout: "0.005s"
            out_ports:
              accepted_concurrency:
                signal_name: "ACCEPTED_CONCURRENCY"
              incoming_concurrency:
                signal_name: "INCOMING_CONCURRENCY"
          load_shed_actuator:
            in_ports:
              load_shed_factor:
                signal_name: "LSF"
      - constant:
          value: "2.0"
          out_ports:
            output:
              signal_name: "CONCURRENCY_LIMIT_MULTIPLIER"
      - constant:
          value: "10.0"
          out_ports:
            output:
              signal_name: "MIN_CONCURRENCY"
      - constant:
          value: "5.0"
          out_ports:
            output:
              signal_name: "LINEAR_CONCURRENCY_INCREMENT"
      - arithmetic_combinator:
          operator: "mul"
          in_ports:
            lhs:
              signal_name: "CONCURRENCY_LIMIT_MULTIPLIER"
            rhs:
              signal_name: "ACCEPTED_CONCURRENCY"
          out_ports:
            output:
              signal_name: "UPPER_CONCURRENCY_LIMIT"
      - max:
          in_ports:
            inputs:
              - signal_name: "UPPER_CONCURRENCY_LIMIT"
              - signal_name: "MIN_CONCURRENCY"
          out_ports:
            output:
              signal_name: "MAX_CONCURRENCY"
      - sqrt:
          scale: "0.5"
          in_ports:
            input:
              signal_name: "ACCEPTED_CONCURRENCY"
          out_ports:
            output:
              signal_name: "SQRT_CONCURRENCY_INCREMENT"
      - arithmetic_combinator:
          operator: "add"
          in_ports:
            lhs:
              signal_name: "LINEAR_CONCURRENCY_INCREMENT"
            rhs:
              signal_name: "SQRT_CONCURRENCY_INCREMENT"
          out_ports:
            output:
              signal_name: "CONCURRENCY_INCREMENT_NORMAL"
      - constant:
          value: "1.2"
          out_ports:
            output:
              signal_name: "OVERLOAD_MULTIPLIER"
      - arithmetic_combinator:
          operator: "mul"
          in_ports:
            lhs:
              signal_name: "LATENCY_EMA"
            rhs:
              signal_name: "OVERLOAD_MULTIPLIER"
          out_ports:
            output:
              signal_name: "LATENCY_OVERLOAD"
      - constant:
          value: "10.0"
          out_ports:
            output:
              signal_name: "CONCURRENCY_INCREMENT_OVERLOAD"
      - decider:
          operator: "gt"
          in_ports:
            lhs:
              signal_name: "LATENCY"
            rhs:
              signal_name: "LATENCY_OVERLOAD"
            on_true:
              signal_name: "CONCURRENCY_INCREMENT_OVERLOAD"
            on_false:
              signal_name: "CONCURRENCY_INCREMENT_NORMAL"
          out_ports:
            output:
              signal_name: "CONCURRENCY_INCREMENT"

import http from "k6/http";
import { check } from "k6";

export let options = {
  discardResponseBodies: true,
  scenarios: {
    userA: {
      executor: "ramping-vus",
      stages: [
        { duration: "30s", target: 5 }, // simulate ramp-up of traffic from 0 to 5 users over 30 seconds
        { duration: "2m", target: 5 }, // stay at 5 users for 2 minutes
        { duration: "30s", target: 60 }, // ramp-up to 60 users over 30 seconds
        { duration: "4m", target: 60 }, // stay at 60 users for 4 minutes (peak hour)
        { duration: "1m", target: 0 }, // ramp-down to 0 users in 1 minute
      ],
      env: { USER: "A" },
    },
    userB: {
      executor: "ramping-vus",
      stages: [
        { duration: "30s", target: 5 }, // simulate ramp-up of traffic from 0 to 5 users over 30 seconds
        { duration: "2m", target: 5 }, // stay at 5 users for 2 minutes
        { duration: "30s", target: 60 }, // ramp-up to 60 users over 30 seconds
        { duration: "4m", target: 60 }, // stay at 60 users for 4 minutes (peak hour)
        { duration: "1m", target: 0 }, // ramp-down to 0 users in 1 minute
      ],
      env: { USER: "B" },
    },
  },
};

export default function () {
  let user = __ENV.USER;
  const url = "http://demo1-demo-app.demoapp.svc.cluster.local/request";
  const headers = {
    "Content-Type": "application/json",
    Cookie:
      "session=eyJ1c2VyIjoia2Vub2JpIn0.YbsY4Q.kTaKRTyOIfVlIbNB48d9YH6Q0wo",
    User: user,
  };
  const body = {};
  let res = http.request("POST", url, JSON.stringify(body), {
    headers: headers,
  });
  check(res, {
    "http status was 200": res.status === 200,
  });
}

Below is the result from druid:

2. In this case, I have added a new user and used the same policy and classification as above. Here also, the drop rates were almost similar for different priority loads.

Below is the load-test.js for this scenario:

import http from "k6/http";
import { check } from "k6";

export let options = {
  discardResponseBodies: true,
  scenarios: {
    userA: {
      executor: "ramping-vus",
      stages: [
        { duration: "30s", target: 5 }, // simulate ramp-up of traffic from 0 to 5 users over 30 seconds
        { duration: "2m", target: 5 }, // stay at 5 users for 2 minutes
        { duration: "30s", target: 60 }, // ramp-up to 60 users over 30 seconds
        { duration: "4m", target: 60 }, // stay at 60 users for 4 minutes (peak hour)
        { duration: "1m", target: 0 }, // ramp-down to 0 users in 1 minute
      ],
      env: { USER: "A" },
    },
    userB: {
      executor: "ramping-vus",
      stages: [
        { duration: "30s", target: 5 }, // simulate ramp-up of traffic from 0 to 5 users over 30 seconds
        { duration: "2m", target: 5 }, // stay at 5 users for 2 minutes
        { duration: "30s", target: 60 }, // ramp-up to 60 users over 30 seconds
        { duration: "4m", target: 60 }, // stay at 60 users for 4 minutes (peak hour)
        { duration: "1m", target: 0 }, // ramp-down to 0 users in 1 minute
      ],
      env: { USER: "B" },
    },
    userC: {
      executor: "ramping-vus",
      stages: [
        { duration: "30s", target: 5 }, // simulate ramp-up of traffic from 0 to 5 users over 30 seconds
        { duration: "2m", target: 5 }, // stay at 5 users for 2 minutes
        { duration: "30s", target: 60 }, // ramp-up to 60 users over 30 seconds
        { duration: "4m", target: 60 }, // stay at 60 users for 4 minutes (peak hour)
        { duration: "1m", target: 0 }, // ramp-down to 0 users in 1 minute
      ],
      env: { USER: "C" },
    },
  },
};

export default function () {
  let user = __ENV.USER;
  const url = "http://demo1-demo-app.demoapp.svc.cluster.local/request";
  const headers = {
    "Content-Type": "application/json",
    Cookie:
      "session=eyJ1c2VyIjoia2Vub2JpIn0.YbsY4Q.kTaKRTyOIfVlIbNB48d9YH6Q0wo",
    User: user,
  };
  const body = {};
  let res = http.request("POST", url, JSON.stringify(body), {
    headers: headers,
  });
  check(res, {
    "http status was 200": res.status === 200,
  });
}

Below is the result from druid:

As we have different priorities for different users, the drop rate shouldn't be same and should be much lower for higher priority loads.

We have examples in attributes in proto definitions, but our swagger template doesn't include them.

Possible fix: just put example as regular doc comment (as before)