nexogen-international / nexogen.libraries.metrics Goto Github PK

I am pushing a set of Gauges that I recorded but when scraping the metrics in my push gateway I get the following error:

expected counter in metric process_cpu_seconds_total label:<name:"instance" value:"" > label:<name:"job" value:"test_job" > label:<name:"test_id" value:"978e7cdb-c017-4e69-a9ac-719f96deee91" > gauge:<value:0.9375 >

I am not capturing nor sending this (actively) so it seems like the set up of the metric in the coreclrexporter class is not done properly.

I would like to have an option to add CollectMetricsMiddleware without being forced to have metrics are being exposed on /metrics path in AspNetCore.

Would prefer to use StandaloneService or my own on another port for security reasons.

Making CollectMetricsMiddleware / HttpMetrics classes public and simple helper for middleware registration would be enough.

Do you accept PRs?

Can we drop dependency on Mvc in Nexogen.Libraries.Metrics.Prometheus.AspCore; we can use Middleware to serve metrics.

Hello.
Great library, by the way, thanks for publishing it.
I'm trying to start using PushGateway functionality for our metrics, and at the moment it doesn't look like it's possible to identify where a given metric is coming from.

Here's how the path is configured:

var path = String.Format("/metrics/job/{0}", WebUtility.UrlEncode(job));

https://github.com/nexogen-international/Nexogen.Libraries.Metrics/blob/master/Nexogen.Libraries.Metrics.Prometheus.PushGateway/PushGateway.cs#L115

My understanding is that if it were in the form

var path = String.Format("/metrics/job/{0}/instance/{1}", WebUtility.UrlEncode(job), WebUtility.UrlEncode(instance));

it would be possible to identify the instance where the metric is coming from.

I see there's a way to set custom metrics, but having an option to set instance would still be handy.

Though recommended by prometheus's documentation, sometimes it's not useful to collect CLR metrics. Add an option to disable it.

Currently only the text format is supported. The code should be refactored to handle both the text and protobuf formats, possibly by introducing an immediate representation and a pluggable backend.

https://prometheus.io/docs/instrumenting/exposition_formats/#protocol-buffer-format-details

The way it works currently is that you define label names statically at metric creation time:

            var planningTime = metrics.Gauge()
                .Name("planning_time").Help("The processing time in seconds").LabelNames("solver").Register();

And then you have to specify label values every time you do anything with this metric:

planningCount.Labels(solver).Increment();

There's no way to set label values once, which is a bit of a pain for things that don't change within the lifetime of a given instance (for example, environment or instance ID).

The readme says to install Nexogen.Libraries.Metrics.Prometheus.Standalone for a standalone server but there is no release of this package for 3.1.0 (the last is 3.1.0-rc.1).

Commit 5c4e3cc seems to have removed the standalone server code. Does this mean that this feature is no longer supported?

Perhaps it could be reimplemented using .NET Core Hosted Services and HttpListener, for example like this:

public class PrometheusServer : IHostedService
{
    private readonly HttpListener _listener;
    private readonly IExposable _metrics;

    public PrometheusServer(IOptions<PrometheusServerOptions> options, IExposable metrics)
    {
        _listener = new HttpListener {Prefixes = {$"http://*:{options.Value.Port}/"}};
        _metrics = metrics;
    }

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _listener.Start();
        BeginContext();
        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _listener.Abort();
        return Task.CompletedTask;
    }

    private void BeginContext()
    {
        _listener.BeginGetContext(ListenerCallback, _listener);
    }

    private async void ListenerCallback(IAsyncResult result)
    {
        var context = _listener.EndGetContext(result);
        context.Response.StatusCode = 200;
        context.Response.Headers.Add("Content-Type", "text/plain");
        await _metrics.Expose(context.Response.OutputStream, ExposeOptions.Default);
        context.Response.Close();
    }
}

I could create a Pull Request if you're interested.

First of all, thank you for your work!

It is a great feature to be able to create new metrics by just using a new label. However that would be great to be able to remove metrics with a specific deprecated label. Please see the following example.

this.Something = metrics.Gauge()
    .Name("something_items_total")
    .Help("Total number of items")
    .LabelNames("someLabel")
    .Register();

# HELP something_items_total Total number of items
# TYPE something_items_total gauge
something_items_total{someLabel="label1"} 5 1513340498878
something_items_total{someLabel="label2"} 2 1513340498878
...

At some point items labeled with "label1" becomes obsolete and therefore its metric.. I can reset it to 0, but cannot remove it :(

This is what I would like to see:

# HELP something_items_total Total number of items
# TYPE something_items_total gauge
something_items_total{someLabel="label2"} 2 1513340498878
...

Correct me if I am wrong and it is a misuse of metrics, however if I do not remove it, it becomes kind of a tiny memory leak :)

Markdown file for contribution guidelines.

Nexogen.Libraries.Metrics.Prometheus.PushGateway project contains inconsinstent namespaces:
e.g. both class PushGateway and PushGatewayException are under the namespace Nexogen.Libraries.Prometheus.PushGateway, which is inconsistent according to NEXOGEN naming conventions.

i'm trying to run this on .net 4.6.1 and getting this error:

System.MissingMethodException occurred
  HResult=0x80131513
  Message=Method not found: 'System.Collections.Generic.IEnumerable`1<!!0> System.Linq.Enumerable.Prepend(System.Collections.Generic.IEnumerable`1<!!0>, !!0)'.
  Source=Nexogen.Libraries.Metrics.Extensions
  StackTrace:
   at Nexogen.Libraries.Metrics.Extensions.Buckets.BucketGenerator.Buckets(Double[] bounds)
   at Nexogen.Libraries.Metrics.BucketExtensions.Buckets(IHistogramBuilder builder, Double[] bounds)
   at Nexogen.Libraries.Metrics.Prometheus.AspCore.HttpMetrics..ctor(IMetrics m)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(SingletonCallSite singletonCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.<>c__DisplayClass16_0.<RealizeService>b__0(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass3_0.<UseMiddleware>b__0(RequestDelegate next)
   at Microsoft.AspNetCore.Builder.Internal.ApplicationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at OrleansDashboard.Dashboard.<Init>d__15.MoveNext() in C:\Users\tal\dev\OrleansDashboard\OrleansDashboard\Dashboard.cs:line 99

i see you have a compat library, but it only adds the required extensions in .net 4.5.2

this needs to be done all the way to .net 4.7.x

Nexogen.Libraries.Metrics.Prometheus.AspCore.HttpMetricsMiddleware measures endpoint response times and collects them labelled by http methods, response codes and endpoint paths.

However these paths are actual paths (eg.: order/321item/123), not logical ones (eg.: order/{orderId}/item/{itemId}) identifiing the controller in MVC, thus if [PathParam] is used on a controller this causes an explosion of labels by each actual call. This has storage and performance costs in Prometheus.

The middleware level approach currently used doesn't have acces to MVC routing, so the logical path is not available, only actual HTTP request path.

This is usually not desired, so this measurement should either be moved to MVC level, or an MVC level alternative needs to be added, so the consumers can weigh the costs and benefits of each approach.

.NET Framework library is missing in NuGet package. Any particular reason for that?
I mean the .NET Standard is cool and all that, but it would drag a ton of references to a .NET Framework project.

AddPrometheus should return IServiceCollection like most of the ASP.NET Core extension methods of the same kind.

NuGet packages should be built from repository and they should be uploaded to https://www.nuget.org/ for better user experience.

I'm building a background service that I wanted to collect metrics from the background service & expose them using http endpoint. Unfortunately there is no easy way to share the metrics unless you resolve metrics from the service provider, can we provide a method that has an optional metrics parameter?

 AddPrometheus(this IServiceCollection services, IMetrics metrics = null)
{
    metrics = metrics ?? new PrometheusMetrics();
    //rest of code
}

Would be useful in some cases when need to dinamically create multiple labels.

Currently the synchronization between the 2 repositories are done manually. We may consider automating this process using a tool like Google Copybara (https://github.com/google/copybara) or similar.

Based on the Prometheus documentation: https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-details

metric_name and label_name have the usual Prometheus expression language restrictions. label_value can be any sequence of UTF-8 characters, but the backslash, the double-quote, and the line-feed characters have to be escaped as \, ", and \n, respectively

Actual Output:

tokens_login{username="domain\username"} 1 1510911080610

Expected Output:

tokens_login{username="domain\\username"} 1 1510911080610