fix unhandler error -172 on entry committable causing the server to crash by SylvainSenechal · Pull Request #2731 · scality/backbeat

SylvainSenechal · 2026-04-08T18:20:02Z

Seen in zenko ci :

…rash BB-758

bert-e · 2026-04-08T18:20:06Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-04-08T18:20:11Z

Incorrect fix version

The Fix Version/s in issue BB-758 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.4.0

Please check the Fix Version/s of BB-758, or the target
branch of this pull request.

codecov · 2026-04-08T18:20:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.53%. Comparing base (9c21188) to head (f308a2a).

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/BackbeatConsumer.js	`93.88% <100.00%> (+0.04%)`	⬆️

... and 3 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	`80.37% <ø> (ø)`
Core Library	`80.72% <100.00%> (-0.42%)`	⬇️
Ingestion	`70.53% <ø> (-0.62%)`	⬇️
Lifecycle	`79.01% <ø> (ø)`
Oplog Populator	`85.83% <ø> (ø)`
Replication	`59.65% <ø> (ø)`
Bucket Scanner	`85.76% <ø> (ø)`

@@                 Coverage Diff                 @@
##           development/9.4    #2731      +/-   ##
===================================================
- Coverage            74.75%   74.53%   -0.23%     
===================================================
  Files                  200      200              
  Lines                13610    13613       +3     
===================================================
- Hits                 10174    10146      -28     
- Misses                3426     3457      +31     
  Partials                10       10

Flag	Coverage Δ
api:retry	`9.14% <0.00%> (-0.01%)`	⬇️
api:routes	`8.95% <0.00%> (-0.01%)`	⬇️
bucket-scanner	`85.76% <ø> (ø)`
ft_test:queuepopulator	`10.01% <0.00%> (-0.57%)`	⬇️
ingestion	`12.49% <0.00%> (-0.07%)`	⬇️
lib	`7.62% <60.00%> (+<0.01%)`	⬆️
lifecycle	`18.84% <60.00%> (-0.01%)`	⬇️
notification	`1.02% <0.00%> (-0.01%)`	⬇️
oplogPopulator	`0.14% <0.00%> (-0.01%)`	⬇️
replication	`18.49% <60.00%> (+<0.01%)`	⬆️
unit	`51.16% <100.00%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

claude · 2026-04-08T18:22:12Z

LGTM

Review by Claude Code

bert-e · 2026-04-08T18:22:23Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

SylvainSenechal · 2026-04-08T19:08:31Z

lib/BackbeatConsumer.js

+            try {
+                this._consumer.offsetsStore([{ topic, partition, offset: committableOffset }]);
+            } catch (e) {
+                // ERR__STATE (-172) means the consumer is not in a valid state


So this stuff seems to happen on rebalance or weird kafka status.

Before, the pod would crash, the offset wouldn't be commited, and another one pod would reconsume and re process it

Now : the pod doesn't crash, but the offset is still not commited. and reconsumed later by another pod, or maybe the same pod since now it doesn't crash anymore

do we know when this happens?

in order to properly handle rebalance, we actually need to commit before the rebalance (hence the logic implemented here). If there are some cases where it does not work, we need to understand why/how, and not only mask the problem....

(there will always be corner cases where we cannot commit and will re-play the message, that is fine... but we need to understand why/when it happens, to make sure this is really the unavoidable corner case and not a deeper issue)

could the root cause be the same as 11fe6ee ?

I don't really know when, and more importantly why this happen :/
Maybe it could be interesting to wait for the pr from Thomas to be merged and run in zenko ci to see if this issue still exists

Okay, I repulled the zenko ci logs from a successful run and asked claude to analyze and pay attention to the timing of the logs.

What was found was that this is happening during a rebalance, we first have one pod handling all the partitions, then the operator deployed another pod, and and when that new pod joined the Kafka consumer group, a rebalance was triggered.
The old pod had 5 partitions revoked and was re-assigned only 2 of them. The problem is: some tasks that were already in-flight finished processing after the revoke happened. When their callback tried to call offsetsStore() to mark the offset as ready to commit, librdkafka threw ERR__STATE because those partitions were no longer assigned to this pod so it crashed

francoisferrand · 2026-04-09T07:22:54Z

lib/BackbeatConsumer.js

+                // state transition) — treat as non-fatal and log at debug level.
+                const logger = this._consumer.isConnected() &&
+                    e.code !== kafka.CODES.ERRORS.ERR__STATE
+                    ? this._log.error : this._log.debug;


this is never a debug log : we don't expect this to happen at all... So always ouptutting an error log may be appropriate

francoisferrand · 2026-04-09T07:35:36Z

lib/BackbeatConsumer.js

-            this._consumer.offsetsStore([{ topic, partition,
-                                           offset: committableOffset }]);
+        if (committableOffset !== null && !this.isPaused() && this._consumer.isConnected()) {
+            try {


since you added the isConnected() guard, do we also need to catch the exception?

I think we could still have cases where is connected returns true, but then inside offsetStore something wrong happens 🤔

fix unhandler error -172 on entry committable causing the server to c…

f308a2a

…rash BB-758

SylvainSenechal requested review from a team, benzekrimaha and maeldonn April 8, 2026 18:51

SylvainSenechal marked this pull request as ready for review April 8, 2026 18:51

SylvainSenechal commented Apr 8, 2026

View reviewed changes

francoisferrand reviewed Apr 9, 2026

View reviewed changes

SylvainSenechal requested a review from francoisferrand April 10, 2026 15:16

Conversation

SylvainSenechal commented Apr 8, 2026

Uh oh!

bert-e commented Apr 8, 2026

Hello sylvainsenechal,

Uh oh!

bert-e commented Apr 8, 2026

Incorrect fix version

Uh oh!

codecov bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

claude bot commented Apr 8, 2026

Uh oh!

bert-e commented Apr 8, 2026

Waiting for approval

Uh oh!

SylvainSenechal Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

francoisferrand Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

francoisferrand Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

francoisferrand Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Apr 8, 2026 •

edited

Loading

francoisferrand Apr 9, 2026 •

edited

Loading