feat(server): Track request rejections with a unified server.rejected metric

[jan-auer]

Add a RejectionReason enum in a new rejection module that classifies why a request was rejected and emits a server.rejected counter tagged with a reason value on every rejection.

Previously, killswitch and rate limit rejections were only logged at debug level — no metric was emitted. The web concurrency limit had its own standalone web.concurrency.rejected counter with no equivalent for other rejection types.

Refs FS-275

[linear]

FS-275 Increase visibility into Rate Limits

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[matt-codecov]

i think this is a bit messy. this metric and logic are essentially duplicated from our response code metric and logic, except they:

• omit <400
• conflate 401 and 403
• conflate 400 and 413
• (your goal) distinguish between 429s caused by rate limits, killswitches, task concurrency limits, and web concurrency limits

what i'd prefer (and what i think we need anyway) is:

• a pair of gauges (limit, current value) for each rate limit using tags to distinguish between global, usecase, and scope limits
• a pair of gauges (limit, current value) for task concurrency and web concurrency
• a counter for killswitches using tags to distinguish between killswitch rules

it's more involved to implement (changes to TokenBucket, maybe other things), but we can see which things are trending toward their limits and intervene before the 429s happen, or if it's a surge we can see with more granularity which limits are being tripped. also sentry product teams can track their rate limit usage on their own dashboards

[jan-auer]

Thanks, this is raising good points. Let me reconsider the approach.

conflate 401 and 403
conflate 400 and 413

These were very intentional. Since we have status codes tracked already, I'm now looking to create a high-level classification of silent, "expected" errors. If any of these occur too often, we are facing a systemic issue that needs to be fixed. This can be related to auth, the client violating the spec, or one of our limits.

Some of these may be valuable to report to the usecase, hence the separation of the limits. Use cases will be able to define their own limits soon and need to be able to act on those.

a pair of gauges [...]

Agreed. These exist and are exposed to KEDA currently; some also via statsd. My goal right now, however, is to focus on rejection reasons rather than capacity.

a counter for killswitches

I'm struggling to find a central place where we can count all rejections and classify them. Instrumenting them individually like before means we can easily miss some.