Support built in System.Diagnostics.Meters for instrumenting metrics

[kcrandall]

Problem Statement

I think it would make more sense in the dotnet world to suggest to users to add meters to their app for metric use cases instead of using sentry sdk for collection that would allow for instrumentation via one path instead of having to duplicate SentrySDK capture for meters. This is more of the dotnet way to do things regarding metrics. Add something like this.AddMeter(nameof(ResponseMetrics)) or .AddMeter("System.Runtime") to the sentry sdk setup so users can opt in to meters that should be tracked by sentry.

Another alternative path would be to collect metrics via OpenTelemetry bridge like traces are done which already has support for this.

Solution Brainstorm

.AddMeter(nameof(ResponseMetrics)) on Sentry.Init() would be the best way to do this

[linear-code]

NET-675

[jamescrosswell]

Thanks @kcrandall - it's a good suggestion. We had something similar in an earlier beta version of Sentry metrics:

• Capture built in metrics from System.Diagnostics.Metrics API #3052
• Added OpenTelemetry.Instrumentation.Runtime metrics to built in metrics #3133

@Flash0ver we could probably reuse some of that old code from:

• src/Sentry/Internal/SystemDiagnosticsMetricsListener.cs
• src/Sentry/BuiltInSystemDiagnosticsMeters.cs
• possibly even src/Sentry/ExperimentalMetricsOptions.cs

[Flash0ver]

Also from my side: thank you, @kcrandall, for your feedback ... this is much appreciated!

Based off of #4798, we are currently internally exploring auto-collection / auto-exporting (and to some degree auto-instrumentation, but that indeed may not be necessary in modern .NET as we got plenty of built-in metrics from the Runtime, ASP.NET Core, Microsoft.Extensions.*, .NET MAUI, Entity Framework Core, and pretty much the rest of the ecosystem) across all SDKs and platforms.

To be fair, there won't be a one-size-fits-all solution.
And the devil is in the details ... in particular when it comes to ObservableInstrument<T> (including ObservableCounter<T>, ObservableGauge<T> and ObservableUpDownCounter<T>): in which frequency shall we emit Metrics for Observable Instruments? Perhaps choosing a sensible default + allow configuration. But at what granularity should we allow configuration? Per Meter and/or per Instrument? Maybe respect OTEL_METRIC_EXPORT_INTERVAL ... but this would indicate that this should be an OpenTelemtry-Exporter, rather than a Sentry-SDK integration ... but we are about to deprecate our Sentry.OpenTelemetry package (where this would fit in) in favor of the new OTLP-based Sentry.OpenTelemetry.Exporter, and Sentry itself currently only supports OTLP endpoints for Logs and Traces. Probably we should have both variants.

Apart from Observable Instruments and the question of SDK-integration and/or OTLP-endpoint, there are a couple of points of potential friction:

Value / Measurement

• .NET supports: byte, short, int, long, double, float, and decimal
• Sentry currently supports
  • Integral numeric types up to 64-bit signed
  • Floating-point numeric types up to 64-bit

Instrument<System.Decimal> and ObservableInstrument<T><System.Decimal> are currently unsupported by Sentry.

Units

.NET/Otel uses UCUM units.
Sentry currently only supports these units: https://develop.sentry.dev/sdk/foundations/state-management/scopes/attributes/#units

We could map some UCUM units to Sentry-supported units.
Other units would be dropped (not the entire Metric, just the unit).

Also, .NET's Counter<T>/ObservableUpDownCounter<T> supports units, but Sentry's Counter does currently not support units.

Tags to Attributes

In addition, the conversion from .NET Tags/TagList (which is a KeyValuePair<string, object?> collection) would be lossy, as Sentry's Attributes currently only support

• string
• boolean
• integer (64-bit signed integral)
• double (64-bit floating point)
• homogeneous single-dimensional array of the above

Additionally, tags with a null-object value will be dropped client-side.

Long story short

Sentry's Application Metrics, being trace-connected and query-time aggregated,
are not quite synonymous to .NET / OpenTelemtry Metrics,
so there will be some losses that we need to document accordingly.

---

@jamescrosswell where there similar differences in "fidelity" between the .NET/OTel and Sentry Metrics from the "old beta"?

[Flash0ver]

Considering all of the above, we could really use and would very much appreciate additional feedback to understand common scenarios and use cases:

• Do you create our own Meters and Instruments ... or entirely resort to Built-in metrics in .NET?
• Which Metrics in particular are the most valuable to you, or would you expect per default?
• At what frequency do you collect / report (observable) instruments?
• Something else that we haven't covered yet?

We will gather all feedback and use it as driver to our auto-collection integration(s).

[kcrandall]

So I think most metrics i would capture with this are counters/histograms and infact most of them are trace connected. I would call something in the flow of code like myCounterMeter.Add(1, tags); Only guages would be something that would need sampling outside of traces for the most part. Id prefer it be built into the meters and not be connected to Open telemetry, but obviously a OTeL endpoint would be needed for collecting metrics like CPU/Memory for things that arent trace connected (think sending metrics from AWS cloudwatch where you dont write the code), but that is out of scope of what this issue is asking for. Id prefer OTeL instead of the legacy way of sending data through Open telemetry, i think depreciating that is a good idea. But most meters are already instrumented in a way that you would be able to connect their data collection to part of a span (Counters/Histograms for sure, guagues sometimes i would sample them outside of a trace but technically they are still needed to be done as part of the flow of code. You could create a trace around them if one doesnt already exist).

I use both built in metrics instrumented in other packages (System, etc) and I use a lot of my own instrumented metrics. In dotnet your are supposed to be able to listen to meters you care about from packages you import.

If they get rounded because the datatype isnt supported I would have no issue with that. For some if they are a not roundable datatype and wouldnt map to sentry dropping that meter with a warning log upon registration is fine with me (until sentry can support them). For me this is more of a nice to have as additional span debugging data instead of the primary metrics store.

I also dont really use units right now as much. For people that do maybe you could put the unit as a tag on that sample?

this is the only example of a meter i have in my code that is not captured within the flow of a normal span/activity. Its a guauge that is sampled every so often instead of when something happens in code. If this was not able to be captured in V1 of this feature i wouldnt be upset. Its impossible to send these metrics to Sentry right now anyways, I think it is a flaw in sentry's metrics design to not allow non span connected metrics and until Sentry does support that people would still need something like Grafana/Datadog to collect things like cloud metrics outside of you code so its fine if some meters like this are dropped for now. So Sentry is just a duplicated place to look at metrics for now until this is fixed.

public static void CreateQueueDepthGauge(string handlerName, Func<int> getQueueDepth)
    {
        _meter.CreateObservableGauge(
            "batched_handler.queue.depth",
            () =>
                new Measurement<int>(
                    getQueueDepth(),
                    new KeyValuePair<string, object?>("handler", handlerName)
                ),
            "events",
            "Current number of events waiting in queue"
        );
    }