Add request trace IDs to Winston logs independent of OTEL

[djwhitt]

Add request trace IDs to Winston logs independent of OTEL

Problem

Currently, request trace IDs only appear in Winston log output when the OpenTelemetry SDK is active (i.e., OTEL_EXPORTER_OTLP_ENDPOINT is set or OTEL_FILE_EXPORT_ENABLED=true). The WinstonInstrumentation in src/tracing.ts injects trace_id and span_id into log entries via OTEL's log correlation, but for operators who don't run an OTEL collector, there's no way to correlate log lines to a specific HTTP request.

This makes debugging production issues significantly harder — when multiple concurrent requests are being served, it's difficult to trace which log lines belong to which request without grepping for data IDs or other indirect identifiers.

Context

Current tracing setup:

• src/tracing.ts configures OTEL with WinstonInstrumentation for log correlation (trace_id/span_id injection)
• Route handlers in src/routes/data/handlers.ts and src/routes/chunk/handlers.ts create OTEL spans per request
• Other route handlers (graphql, arns, root, ar-io, datasets, x402, rate-limit) do not create spans
• When OTEL SDK is not started (no endpoint, no file export), the tracer is effectively no-op and no trace IDs appear in logs
• There is no Express middleware that assigns a request ID independent of OTEL

Requirements

Must Have

• Every HTTP request gets a unique request ID assigned at the Express middleware level, independent of OTEL configuration
• The request ID is included in all Winston log entries emitted during that request's lifecycle
• The request ID is returned in a response header (e.g., X-Request-Id) so clients can reference it in bug reports
• If an incoming request already carries an X-Request-Id header, honor it (useful for tracing across reverse proxies / CDN layers)

Should Have

• The request ID should use a compact, fast-to-generate format (e.g., nanoid or crypto.randomUUID())
• Minimal performance overhead — this runs on every request
• The field name in logs should be requestId (or similar) to distinguish from OTEL's trace_id

Nice to Have

• When OTEL is active, the OTEL trace_id continues to be injected alongside the requestId (both are useful — trace_id for distributed tracing, requestId for local log grep)

Technical Notes

Approach

The typical pattern is an Express middleware early in the chain that:

1. Reads X-Request-Id from the incoming request headers (if present) or generates a new one
2. Stores it on the request object (e.g., req.requestId)
3. Sets the X-Request-Id response header
4. Uses a continuation-local storage mechanism (e.g., AsyncLocalStorage) to make the ID available to Winston without explicit passing

Winston supports custom formats and defaultMeta — a log format that reads from AsyncLocalStorage can inject the requestId automatically.

Files likely to modify

File	Change
New: src/middleware/request-id.ts	Express middleware to assign/propagate request IDs
src/log.ts (or equivalent)	Winston format/transport to inject requestId from AsyncLocalStorage
src/app.ts or route registration	Register the middleware early in the Express chain

Considerations

• AsyncLocalStorage is stable in Node.js 16+ and has negligible overhead for this use case
• This should be orthogonal to OTEL — both can coexist (OTEL injects trace_id/span_id, this injects requestId)
• The middleware should be registered before any route handlers so all downstream logs pick up the ID