Add flag evaluation metrics via OTel counter and OpenFeature Hook by typotter · Pull Request #11040 · DataDog/dd-trace-java

typotter · 2026-04-02T15:34:34Z

What Does This Do

Records a feature_flag.evaluations OTel counter metric on every flag evaluation via an OpenFeature finallyAfter hook. The hook captures all evaluation paths including type mismatches that occur above the provider level in the OpenFeature SDK pipeline.

Creates a dedicated SdkMeterProvider with an OtlpHttpMetricExporter that sends metrics directly to the DD Agent's OTLP endpoint (/v1/metrics). This avoids the agent's OTel class shading (io.opentelemetry.api.* → datadog.trace.bootstrap.otel.api.*) which prevents using GlobalOpenTelemetry from the published dd-openfeature jar.

Metric attributes:

Attribute	When present	Value
`feature_flag.key`	Always	Flag key
`feature_flag.result.variant`	Always	Variant key (empty string if null)
`feature_flag.result.reason`	Always	Reason lowercased
`error.type`	On error	ErrorCode lowercased
`feature_flag.result.allocation_key`	When present	Allocation key from flag metadata

New files: FlagEvalMetrics.java, FlagEvalHook.java, FlagEvalMetricsTest.java, FlagEvalHookTest.java
Modified files: Provider.java (adds getProviderHooks()), ProviderTest.java, build.gradle.kts

Motivation

Evaluation metrics allow tracking how many times flags are evaluated, with which results, across sessions. This is the Java implementation of the evaluation logging spec (FFL-1942), matching the existing Python (dd-trace-py#17029) and Go (dd-trace-go#4489) implementations.

System tests: 11/17 pass. The 6 remaining failures are pre-existing DDEvaluator gaps (reason mapping, parse error codes) addressed in separate PRs (#11036, #10971).

References:

Python implementation: feat(openfeature): add flag evaluation metrics dd-trace-py#17029
Go implementation: feat(openfeature): add flag evaluation tracking via OTel Metrics dd-trace-go#4489
Go cross-tracer consistency: fix(openfeature): improve FFE eval metrics cross-tracer consistency dd-trace-go#4590
System tests: DataDog/system-tests (branch sameerank/FFL-1942/add-flag-eval-metrics)

Additional Notes

OTel SDK dependencies (opentelemetry-sdk-metrics, opentelemetry-exporter-otlp) are compileOnly — applications must include them on the classpath for metrics to flow. Falls back to silent no-op when absent.
Export interval: 10s (matching Go SDK and EVALLOG.4 spec)
Endpoint resolution follows OTel spec: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT → OTEL_EXPORTER_OTLP_ENDPOINT + /v1/metrics → http://localhost:4318/v1/metrics

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels
Avoid using close, fix, or any linking keywords when referencing an issue

Jira ticket: FFL-1942

pr-commenter · 2026-04-08T20:15:44Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	typo/evaluations-logging
git_commit_date	1775744045	1775756575
git_commit_sha	`b266e2d`	`da92198`
release_version	1.62.0-SNAPSHOT~b266e2d0c2	1.62.0-SNAPSHOT~da921984b2

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1775758317	1775758317
ci_job_id	1582619156	1582619156
ci_pipeline_id	106928527	106928527
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-dkv4gx2z 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-dkv4gx2z 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 13 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1056454
Total [baseline] (11.049 s) : 0, 11049497
Agent [candidate] (1.062 s) : 0, 1061670
Total [candidate] (11.19 s) : 0, 11190343
section appsec
Agent [baseline] (1.248 s) : 0, 1248213
Total [baseline] (11.194 s) : 0, 11194058
Agent [candidate] (1.249 s) : 0, 1249481
Total [candidate] (11.121 s) : 0, 11120859
section iast
Agent [baseline] (1.243 s) : 0, 1242843
Total [baseline] (11.288 s) : 0, 11287911
Agent [candidate] (1.226 s) : 0, 1226436
Total [candidate] (11.355 s) : 0, 11355108
section profiling
Agent [baseline] (1.185 s) : 0, 1184820
Total [baseline] (11.045 s) : 0, 11044846
Agent [candidate] (1.185 s) : 0, 1184512
Total [candidate] (11.069 s) : 0, 11069416

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	appsec	1.248 s	191.76 ms (18.2%)
Agent	iast	1.243 s	186.389 ms (17.6%)
Agent	profiling	1.185 s	128.366 ms (12.2%)
Total	tracing	11.049 s	-
Total	appsec	11.194 s	144.561 ms (1.3%)
Total	iast	11.288 s	238.414 ms (2.2%)
Total	profiling	11.045 s	-4.651 ms (-0.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.062 s	-
Agent	appsec	1.249 s	187.811 ms (17.7%)
Agent	iast	1.226 s	164.765 ms (15.5%)
Agent	profiling	1.185 s	122.842 ms (11.6%)
Total	tracing	11.19 s	-
Total	appsec	11.121 s	-69.484 ms (-0.6%)
Total	iast	11.355 s	164.765 ms (1.5%)
Total	profiling	11.069 s	-120.927 ms (-1.1%)

gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.227 ms) : 0, 1227
crashtracking [candidate] (1.227 ms) : 0, 1227
BytebuddyAgent [baseline] (631.826 ms) : 0, 631826
BytebuddyAgent [candidate] (636.472 ms) : 0, 636472
AgentMeter [baseline] (29.315 ms) : 0, 29315
AgentMeter [candidate] (29.528 ms) : 0, 29528
GlobalTracer [baseline] (248.424 ms) : 0, 248424
GlobalTracer [candidate] (249.456 ms) : 0, 249456
AppSec [baseline] (32.074 ms) : 0, 32074
AppSec [candidate] (32.117 ms) : 0, 32117
Debugger [baseline] (60.445 ms) : 0, 60445
Debugger [candidate] (60.249 ms) : 0, 60249
Remote Config [baseline] (608.347 µs) : 0, 608
Remote Config [candidate] (617.152 µs) : 0, 617
Telemetry [baseline] (8.211 ms) : 0, 8211
Telemetry [candidate] (8.131 ms) : 0, 8131
Flare Poller [baseline] (8.201 ms) : 0, 8201
Flare Poller [candidate] (7.589 ms) : 0, 7589
section appsec
crashtracking [baseline] (1.228 ms) : 0, 1228
crashtracking [candidate] (1.226 ms) : 0, 1226
BytebuddyAgent [baseline] (661.04 ms) : 0, 661040
BytebuddyAgent [candidate] (662.049 ms) : 0, 662049
AgentMeter [baseline] (11.929 ms) : 0, 11929
AgentMeter [candidate] (12.051 ms) : 0, 12051
GlobalTracer [baseline] (249.028 ms) : 0, 249028
GlobalTracer [candidate] (249.65 ms) : 0, 249650
AppSec [baseline] (185.024 ms) : 0, 185024
AppSec [candidate] (184.75 ms) : 0, 184750
Debugger [baseline] (66.414 ms) : 0, 66414
Debugger [candidate] (66.163 ms) : 0, 66163
Remote Config [baseline] (610.008 µs) : 0, 610
Remote Config [candidate] (605.266 µs) : 0, 605
Telemetry [baseline] (8.465 ms) : 0, 8465
Telemetry [candidate] (8.545 ms) : 0, 8545
Flare Poller [baseline] (3.55 ms) : 0, 3550
Flare Poller [candidate] (3.548 ms) : 0, 3548
IAST [baseline] (24.665 ms) : 0, 24665
IAST [candidate] (24.51 ms) : 0, 24510
section iast
crashtracking [baseline] (1.249 ms) : 0, 1249
crashtracking [candidate] (1.242 ms) : 0, 1242
BytebuddyAgent [baseline] (816.038 ms) : 0, 816038
BytebuddyAgent [candidate] (802.339 ms) : 0, 802339
AgentMeter [baseline] (11.591 ms) : 0, 11591
AgentMeter [candidate] (11.425 ms) : 0, 11425
GlobalTracer [baseline] (241.858 ms) : 0, 241858
GlobalTracer [candidate] (239.838 ms) : 0, 239838
AppSec [baseline] (30.495 ms) : 0, 30495
AppSec [candidate] (31.229 ms) : 0, 31229
Debugger [baseline] (61.893 ms) : 0, 61893
Debugger [candidate] (59.596 ms) : 0, 59596
Remote Config [baseline] (536.15 µs) : 0, 536
Remote Config [candidate] (1.121 ms) : 0, 1121
Telemetry [baseline] (12.245 ms) : 0, 12245
Telemetry [candidate] (13.766 ms) : 0, 13766
Flare Poller [baseline] (3.591 ms) : 0, 3591
Flare Poller [candidate] (3.62 ms) : 0, 3620
IAST [baseline] (26.824 ms) : 0, 26824
IAST [candidate] (25.84 ms) : 0, 25840
section profiling
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (691.212 ms) : 0, 691212
BytebuddyAgent [candidate] (692.067 ms) : 0, 692067
AgentMeter [baseline] (9.232 ms) : 0, 9232
AgentMeter [candidate] (9.183 ms) : 0, 9183
GlobalTracer [baseline] (207.271 ms) : 0, 207271
GlobalTracer [candidate] (207.061 ms) : 0, 207061
AppSec [baseline] (32.657 ms) : 0, 32657
AppSec [candidate] (32.429 ms) : 0, 32429
Debugger [baseline] (65.625 ms) : 0, 65625
Debugger [candidate] (65.485 ms) : 0, 65485
Remote Config [baseline] (581.128 µs) : 0, 581
Remote Config [candidate] (564.851 µs) : 0, 565
Telemetry [baseline] (7.862 ms) : 0, 7862
Telemetry [candidate] (7.796 ms) : 0, 7796
Flare Poller [baseline] (3.576 ms) : 0, 3576
Flare Poller [candidate] (3.6 ms) : 0, 3600
ProfilingAgent [baseline] (94.37 ms) : 0, 94370
ProfilingAgent [candidate] (93.881 ms) : 0, 93881
Profiling [baseline] (94.933 ms) : 0, 94933
Profiling [candidate] (94.452 ms) : 0, 94452

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053323
Total [baseline] (8.824 s) : 0, 8824452
Agent [candidate] (1.057 s) : 0, 1056745
Total [candidate] (8.88 s) : 0, 8880199
section iast
Agent [baseline] (1.226 s) : 0, 1225847
Total [baseline] (9.576 s) : 0, 9576209
Agent [candidate] (1.235 s) : 0, 1235049
Total [candidate] (9.57 s) : 0, 9570425

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	iast	1.226 s	172.524 ms (16.4%)
Total	tracing	8.824 s	-
Total	iast	9.576 s	751.758 ms (8.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.057 s	-
Agent	iast	1.235 s	178.304 ms (16.9%)
Total	tracing	8.88 s	-
Total	iast	9.57 s	690.226 ms (7.8%)

gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.222 ms) : 0, 1222
crashtracking [candidate] (1.222 ms) : 0, 1222
BytebuddyAgent [baseline] (631.156 ms) : 0, 631156
BytebuddyAgent [candidate] (633.214 ms) : 0, 633214
AgentMeter [baseline] (29.285 ms) : 0, 29285
AgentMeter [candidate] (29.328 ms) : 0, 29328
GlobalTracer [baseline] (248.008 ms) : 0, 248008
GlobalTracer [candidate] (249.366 ms) : 0, 249366
AppSec [baseline] (31.936 ms) : 0, 31936
AppSec [candidate] (32.098 ms) : 0, 32098
Debugger [baseline] (58.91 ms) : 0, 58910
Debugger [candidate] (59.276 ms) : 0, 59276
Remote Config [baseline] (591.024 µs) : 0, 591
Remote Config [candidate] (603.228 µs) : 0, 603
Telemetry [baseline] (8.013 ms) : 0, 8013
Telemetry [candidate] (8.094 ms) : 0, 8094
Flare Poller [baseline] (8.126 ms) : 0, 8126
Flare Poller [candidate] (7.351 ms) : 0, 7351
section iast
crashtracking [baseline] (1.237 ms) : 0, 1237
crashtracking [candidate] (1.251 ms) : 0, 1251
BytebuddyAgent [baseline] (799.975 ms) : 0, 799975
BytebuddyAgent [candidate] (810.011 ms) : 0, 810011
AgentMeter [baseline] (11.453 ms) : 0, 11453
AgentMeter [candidate] (11.703 ms) : 0, 11703
GlobalTracer [baseline] (240.985 ms) : 0, 240985
GlobalTracer [candidate] (240.409 ms) : 0, 240409
AppSec [baseline] (29.864 ms) : 0, 29864
AppSec [candidate] (30.624 ms) : 0, 30624
Debugger [baseline] (62.243 ms) : 0, 62243
Debugger [candidate] (62.995 ms) : 0, 62995
Remote Config [baseline] (550.063 µs) : 0, 550
Remote Config [candidate] (1.101 ms) : 0, 1101
Telemetry [baseline] (13.499 ms) : 0, 13499
Telemetry [candidate] (11.272 ms) : 0, 11272
Flare Poller [baseline] (3.507 ms) : 0, 3507
Flare Poller [candidate] (3.503 ms) : 0, 3503
IAST [baseline] (26.161 ms) : 0, 26161
IAST [candidate] (25.754 ms) : 0, 25754

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	typo/evaluations-logging
git_commit_date	1775744045	1775756575
git_commit_sha	`b266e2d`	`da92198`
release_version	1.62.0-SNAPSHOT~b266e2d0c2	1.62.0-SNAPSHOT~da921984b2

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1775758700	1775758700
ci_job_id	1582619157	1582619157
ci_pipeline_id	106928527	106928527
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-r9qokdea 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-r9qokdea 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 5 performance improvements and 2 performance regressions! Performance is the same for 13 metrics, 16 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:petclinic:tracing:high_load	better [-2.390ms; -1.333ms] or [-12.431%; -6.932%]	better [-2.525ms; -0.795ms] or [-8.297%; -2.610%]	unstable [-5.228op/s; +46.728op/s] or [-2.177%; +19.457%]	17.365ms	28.779ms	260.906op/s	19.227ms	30.439ms	240.156op/s
scenario:load:petclinic:profiling:high_load	worse [+1.278ms; +2.356ms] or [+7.130%; +13.145%]	worse [+1.836ms; +3.454ms] or [+6.371%; +11.987%]	unstable [-48.066op/s; +4.316op/s] or [-18.767%; +1.685%]	19.742ms	31.464ms	234.250op/s	17.925ms	28.819ms	256.125op/s
scenario:load:petclinic:iast:high_load	better [-1.939ms; -1.217ms] or [-10.420%; -6.538%]	better [-2.262ms; -0.873ms] or [-7.584%; -2.928%]	unstable [-5.914op/s; +46.539op/s] or [-2.395%; +18.844%]	17.030ms	28.251ms	267.281op/s	18.608ms	29.819ms	246.969op/s
scenario:load:petclinic:no_agent:high_load	better [-2.243ms; -0.666ms] or [-12.576%; -3.732%]	unstable [-3.104ms; +0.170ms] or [-10.525%; +0.576%]	unstable [-9.182op/s; +49.807op/s] or [-3.592%; +19.482%]	16.382ms	28.027ms	275.969op/s	17.836ms	29.494ms	255.656op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.248 ms) : 18060, 18437
.   : milestone, 18248,
appsec (18.733 ms) : 18546, 18920
.   : milestone, 18733,
code_origins (18.134 ms) : 17956, 18311
.   : milestone, 18134,
iast (18.899 ms) : 18710, 19089
.   : milestone, 18899,
profiling (18.215 ms) : 18038, 18393
.   : milestone, 18215,
tracing (19.434 ms) : 19239, 19630
.   : milestone, 19434,
section candidate
no_agent (16.903 ms) : 16736, 17070
.   : milestone, 16903,
appsec (18.472 ms) : 18285, 18659
.   : milestone, 18472,
code_origins (18.019 ms) : 17844, 18194
.   : milestone, 18019,
iast (17.456 ms) : 17284, 17629
.   : milestone, 17456,
profiling (19.925 ms) : 19721, 20129
.   : milestone, 19925,
tracing (17.88 ms) : 17704, 18056
.   : milestone, 17880,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.248 ms [18.06 ms, 18.437 ms]	-
appsec	18.733 ms [18.546 ms, 18.92 ms]	484.191 µs (2.7%)
code_origins	18.134 ms [17.956 ms, 18.311 ms]	-114.743 µs (-0.6%)
iast	18.899 ms [18.71 ms, 19.089 ms]	650.926 µs (3.6%)
profiling	18.215 ms [18.038 ms, 18.393 ms]	-33.174 µs (-0.2%)
tracing	19.434 ms [19.239 ms, 19.63 ms]	1.186 ms (6.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	16.903 ms [16.736 ms, 17.07 ms]	-
appsec	18.472 ms [18.285 ms, 18.659 ms]	1.569 ms (9.3%)
code_origins	18.019 ms [17.844 ms, 18.194 ms]	1.116 ms (6.6%)
iast	17.456 ms [17.284 ms, 17.629 ms]	553.641 µs (3.3%)
profiling	19.925 ms [19.721 ms, 20.129 ms]	3.022 ms (17.9%)
tracing	17.88 ms [17.704 ms, 18.056 ms]	977.604 µs (5.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.237 ms) : 1226, 1248
.   : milestone, 1237,
iast (3.302 ms) : 3258, 3346
.   : milestone, 3302,
iast_FULL (6.014 ms) : 5953, 6075
.   : milestone, 6014,
iast_GLOBAL (3.745 ms) : 3675, 3815
.   : milestone, 3745,
profiling (2.196 ms) : 2176, 2215
.   : milestone, 2196,
tracing (1.833 ms) : 1818, 1848
.   : milestone, 1833,
section candidate
no_agent (1.251 ms) : 1239, 1263
.   : milestone, 1251,
iast (3.232 ms) : 3196, 3269
.   : milestone, 3232,
iast_FULL (6.211 ms) : 6147, 6275
.   : milestone, 6211,
iast_GLOBAL (3.682 ms) : 3620, 3743
.   : milestone, 3682,
profiling (2.171 ms) : 2150, 2192
.   : milestone, 2171,
tracing (1.891 ms) : 1874, 1908
.   : milestone, 1891,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.237 ms [1.226 ms, 1.248 ms]	-
iast	3.302 ms [3.258 ms, 3.346 ms]	2.065 ms (166.9%)
iast_FULL	6.014 ms [5.953 ms, 6.075 ms]	4.777 ms (386.0%)
iast_GLOBAL	3.745 ms [3.675 ms, 3.815 ms]	2.508 ms (202.7%)
profiling	2.196 ms [2.176 ms, 2.215 ms]	958.451 µs (77.5%)
tracing	1.833 ms [1.818 ms, 1.848 ms]	595.427 µs (48.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.251 ms [1.239 ms, 1.263 ms]	-
iast	3.232 ms [3.196 ms, 3.269 ms]	1.981 ms (158.4%)
iast_FULL	6.211 ms [6.147 ms, 6.275 ms]	4.96 ms (396.5%)
iast_GLOBAL	3.682 ms [3.62 ms, 3.743 ms]	2.431 ms (194.3%)
profiling	2.171 ms [2.15 ms, 2.192 ms]	920.145 µs (73.5%)
tracing	1.891 ms [1.874 ms, 1.908 ms]	639.851 µs (51.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	typo/evaluations-logging
git_commit_date	1775744045	1775756575
git_commit_sha	`b266e2d`	`da92198`
release_version	1.62.0-SNAPSHOT~b266e2d0c2	1.62.0-SNAPSHOT~da921984b2

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1775758546	1775758546
ci_job_id	1582619158	1582619158
ci_pipeline_id	106928527	106928527
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-0he8mxyh 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-0he8mxyh 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.612 s) : 15612000, 15612000
.   : milestone, 15612000,
appsec (15.071 s) : 15071000, 15071000
.   : milestone, 15071000,
iast (18.269 s) : 18269000, 18269000
.   : milestone, 18269000,
iast_GLOBAL (18.338 s) : 18338000, 18338000
.   : milestone, 18338000,
profiling (15.133 s) : 15133000, 15133000
.   : milestone, 15133000,
tracing (14.646 s) : 14646000, 14646000
.   : milestone, 14646000,
section candidate
no_agent (14.976 s) : 14976000, 14976000
.   : milestone, 14976000,
appsec (15.15 s) : 15150000, 15150000
.   : milestone, 15150000,
iast (18.303 s) : 18303000, 18303000
.   : milestone, 18303000,
iast_GLOBAL (18.119 s) : 18119000, 18119000
.   : milestone, 18119000,
profiling (14.909 s) : 14909000, 14909000
.   : milestone, 14909000,
tracing (14.73 s) : 14730000, 14730000
.   : milestone, 14730000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.612 s [15.612 s, 15.612 s]	-
appsec	15.071 s [15.071 s, 15.071 s]	-541.0 ms (-3.5%)
iast	18.269 s [18.269 s, 18.269 s]	2.657 s (17.0%)
iast_GLOBAL	18.338 s [18.338 s, 18.338 s]	2.726 s (17.5%)
profiling	15.133 s [15.133 s, 15.133 s]	-479.0 ms (-3.1%)
tracing	14.646 s [14.646 s, 14.646 s]	-966.0 ms (-6.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.976 s [14.976 s, 14.976 s]	-
appsec	15.15 s [15.15 s, 15.15 s]	174.0 ms (1.2%)
iast	18.303 s [18.303 s, 18.303 s]	3.327 s (22.2%)
iast_GLOBAL	18.119 s [18.119 s, 18.119 s]	3.143 s (21.0%)
profiling	14.909 s [14.909 s, 14.909 s]	-67.0 ms (-0.4%)
tracing	14.73 s [14.73 s, 14.73 s]	-246.0 ms (-1.6%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~da921984b2, baseline=1.62.0-SNAPSHOT~b266e2d0c2
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.498 ms) : 1487, 1510
.   : milestone, 1498,
appsec (2.553 ms) : 2498, 2608
.   : milestone, 2553,
iast (2.292 ms) : 2222, 2362
.   : milestone, 2292,
iast_GLOBAL (2.346 ms) : 2275, 2416
.   : milestone, 2346,
profiling (2.536 ms) : 2372, 2700
.   : milestone, 2536,
tracing (2.118 ms) : 2064, 2172
.   : milestone, 2118,
section candidate
no_agent (1.498 ms) : 1486, 1510
.   : milestone, 1498,
appsec (3.862 ms) : 3637, 4088
.   : milestone, 3862,
iast (2.307 ms) : 2237, 2378
.   : milestone, 2307,
iast_GLOBAL (2.332 ms) : 2262, 2403
.   : milestone, 2332,
profiling (2.543 ms) : 2378, 2709
.   : milestone, 2543,
tracing (2.12 ms) : 2066, 2174
.   : milestone, 2120,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.498 ms [1.487 ms, 1.51 ms]	-
appsec	2.553 ms [2.498 ms, 2.608 ms]	1.054 ms (70.4%)
iast	2.292 ms [2.222 ms, 2.362 ms]	793.333 µs (52.9%)
iast_GLOBAL	2.346 ms [2.275 ms, 2.416 ms]	847.149 µs (56.5%)
profiling	2.536 ms [2.372 ms, 2.7 ms]	1.037 ms (69.2%)
tracing	2.118 ms [2.064 ms, 2.172 ms]	619.615 µs (41.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.498 ms [1.486 ms, 1.51 ms]	-
appsec	3.862 ms [3.637 ms, 4.088 ms]	2.364 ms (157.8%)
iast	2.307 ms [2.237 ms, 2.378 ms]	809.224 µs (54.0%)
iast_GLOBAL	2.332 ms [2.262 ms, 2.403 ms]	834.127 µs (55.7%)
profiling	2.543 ms [2.378 ms, 2.709 ms]	1.045 ms (69.8%)
tracing	2.12 ms [2.066 ms, 2.174 ms]	622.005 µs (41.5%)

Record a `feature_flag.evaluations` OTel counter on every flag evaluation using an OpenFeature `finallyAfter` hook. The hook captures all evaluation paths including type mismatches that occur above the provider level. Attributes: feature_flag.key, feature_flag.result.variant, feature_flag.result.reason, error.type (on error), feature_flag.result.allocation_key (when present). Counter is a no-op when DD_METRICS_OTEL_ENABLED is false or opentelemetry-api is absent from the classpath.

Replace GlobalOpenTelemetry.getMeterProvider() with a dedicated SdkMeterProvider + OtlpHttpMetricExporter that sends metrics directly to the DD Agent's OTLP endpoint (default :4318/v1/metrics). This avoids the agent's OTel class shading issue where the agent relocates io.opentelemetry.api.* to datadog.trace.bootstrap.otel.api.*, making GlobalOpenTelemetry calls from the dd-openfeature jar hit the unshaded no-op provider instead of the agent's shim. Requires opentelemetry-sdk-metrics and opentelemetry-exporter-otlp on the application classpath. Falls back to no-op if absent. System tests: 11/17 pass. 6 failures are pre-existing DDEvaluator gaps (reason mapping, parse errors, type mismatch strictness).

- Add explicit null guard for details in FlagEvalHook.finallyAfter() - Add OTEL_EXPORTER_OTLP_ENDPOINT generic env var fallback with /v1/metrics path appended (per OTel spec fallback chain) - Add comments clarifying signal-specific vs generic endpoint behavior

When the OTel SDK jars are not on the application classpath, loading FlagEvalMetrics fails because field types reference OTel SDK classes (SdkMeterProvider). This propagated as an uncaught NoClassDefFoundError from the Provider constructor, crashing provider initialization. Fix: - Change meterProvider field type from SdkMeterProvider to Closeable (always on classpath), use local SdkMeterProvider variable inside try block - Catch NoClassDefFoundError in Provider constructor when creating FlagEvalMetrics - Null-safe getProviderHooks() and shutdown() when metrics is null

FlagEvalHook references FlagEvalMetrics in its field declaration. On JVMs that eagerly verify field types during class loading, constructing FlagEvalHook outside the try/catch could throw NoClassDefFoundError if OTel classes failed to load. Moving it inside the try block ensures both metrics and hook are null-safe when OTel is absent.

Documents the published artifact setup, evaluation metrics dependencies (opentelemetry-sdk-metrics, opentelemetry-exporter-otlp), OTLP endpoint configuration, metric attributes, and requirements.

System.getenv() is forbidden by the project's forbiddenApis rules. Replace with ConfigHelper.env() which is the approved way to read environment variables. Add config-utils as compileOnly dependency.

sameerank

Thanks for helping with this! I agree it was a good idea to break out the system test fixes into separate PRs to keep this one brief and focused

manuel-alvarez-alvarez · 2026-04-10T15:55:19Z

products/feature-flagging/feature-flagging-api/README.md

+    <artifactId>dd-openfeature</artifactId>
+    <version>${dd-openfeature.version}</version>
+</dependency>
+<dependency>


This is a transitive dependency of our provider, I think it might be better to skip it (otherwise customers will need to ensure compatibility).

manuel-alvarez-alvarez · 2026-04-10T15:57:43Z

...-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/FlagEvalHook.java

+      String flagKey = details.getFlagKey();
+      String variant = details.getVariant();
+      String reason = details.getReason();
+      dev.openfeature.sdk.ErrorCode errorCode = details.getErrorCode();


nit: we probably want to import dev.openfeature.sdk.ErrorCode

typotter added type: feature request tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Apr 2, 2026

typotter added 5 commits April 9, 2026 09:29

typotter force-pushed the typo/evaluations-logging branch from 4cb7bab to 69c5529 Compare April 9, 2026 15:30

Add README for dd-openfeature with eval metrics setup

18c0441

Documents the published artifact setup, evaluation metrics dependencies (opentelemetry-sdk-metrics, opentelemetry-exporter-otlp), OTLP endpoint configuration, metric attributes, and requirements.

typotter marked this pull request as ready for review April 9, 2026 17:41

typotter requested a review from a team as a code owner April 9, 2026 17:41

typotter requested review from leoromanovsky and sameerank and removed request for a team April 9, 2026 17:41

Use ConfigHelper.env() instead of System.getenv()

da92198

System.getenv() is forbidden by the project's forbiddenApis rules. Replace with ConfigHelper.env() which is the approved way to read environment variables. Add config-utils as compileOnly dependency.

typotter requested a review from manuel-alvarez-alvarez April 9, 2026 17:52

sameerank approved these changes Apr 10, 2026

View reviewed changes

manuel-alvarez-alvarez reviewed Apr 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flag evaluation metrics via OTel counter and OpenFeature Hook#11040

Add flag evaluation metrics via OTel counter and OpenFeature Hook#11040
typotter wants to merge 7 commits intomasterfrom
typo/evaluations-logging

typotter commented Apr 2, 2026 •

edited by atlassian bot

Loading

Uh oh!

pr-commenter bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

sameerank left a comment

Uh oh!

manuel-alvarez-alvarez Apr 10, 2026

Uh oh!

manuel-alvarez-alvarez Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

typotter commented Apr 2, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

sameerank left a comment

Choose a reason for hiding this comment

Uh oh!

manuel-alvarez-alvarez Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

manuel-alvarez-alvarez Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

typotter commented Apr 2, 2026 •

edited by atlassian bot

Loading

pr-commenter bot commented Apr 8, 2026 •

edited

Loading