Telemetry: stop double-retrying, fix user-agent, tune circuit breaker

[samikshya-db]

Summary

After v1.11.0 enabled telemetry by default via the server feature flag, high-QPS workloads produced excessive 429s on /telemetry-ext. Three issues compounded:

1. Double-retry. The telemetry exporter ran its own retry loop on top of the retryablehttp-wrapped HTTP client (internal/client.RetryableClient), which already retries 429/5xx with Retry-After. Result: up to RetryMax × (MaxRetries+1) HTTP attempts per export, all collapsed into a single circuit-breaker outcome — so the breaker barely opened against persistent throttling.
2. Untraceable in access logs. Telemetry POSTs and feature-flag GETs sent no User-Agent, so 429s landed in access logs tagged as Go-http-client/1.1 and could not be attributed to godatabrickssqlconnector by driver version.
3. High request volume. FlushInterval=5s / BatchSize=100.

Changes

Retry behavior

• telemetry/exporter.go — Removed doExport's retry loop entirely. doExport now makes a single HTTP request; transient retries (429/5xx, Retry-After) are owned by the underlying retryablehttp client. Each export() call now corresponds to exactly one HTTP transaction = one breaker outcome.
• telemetry/config.go, telemetry/driver_integration.go — Removed MaxRetries / RetryDelay from telemetry.Config and TelemetryInitOptions. telemetry_retry_count / telemetry_retry_delay DSN params still parse without error for backwards compatibility but are no-ops.

Identifiability

• connector.go — New buildUserAgent helper mirroring internal/client/client.go:295-302 exactly: DriverName/DriverVersion + optional UserAgentEntry + agent product.
• telemetry/exporter.go, telemetry/featureflag.go — Set User-Agent on telemetry POST and feature-flag GET. Plumbed via TelemetryInitOptions.UserAgent.

Cadence and breaker tuning

• telemetry/config.go — FlushInterval 5s → 30s, BatchSize 100 → 200.
• telemetry/circuitbreaker.go — minimumNumberOfCalls 20 → 10 (so low-traffic clients can trip the breaker now that each export is one signal), waitDurationInOpenState 30s → 60s (respect typical Retry-After).

Tests

• Removed obsolete retry/backoff tests (TestExport_RetryOn5xx, TestExport_ExponentialBackoff, TestIsRetryableStatus, retry-config parsing tests).
• Added TestExport_SingleAttemptPerExport covering 4xx/429/5xx, asserting the exporter never retries.
• Added TestExport_SetsUserAgent and TestFetchFeatureFlag_SetsUserAgent.

Mitigation

While this rolls out, users can opt out via DSN: enableTelemetry=false. Server-side: disable enableTelemetryForGoDriver for affected workspaces.

Test plan

• go test ./... — all green locally.
• Exporter never retries (4xx, 429, 500, 503).
• User-Agent set on telemetry POST and feature-flag GET.
• Verify in Lumberjack post-deploy that /telemetry-ext and /api/2.0/connector-service/feature-flags/GOLANG/... requests carry godatabrickssqlconnector/<version> in http_user_agent.
• Confirm 429 rate against /telemetry-ext drops after rollout.

This pull request and its description were written by Isaac.