Fix command queue corruption on encoding failures

[yangy0000]

Summary

This PR fixes an issue where encoding failures could corrupt the outstanding command queue, causing subsequent responses to be matched to wrong commands and leading to out-of-sync command/response handling.

Closes #2012

Root Cause: When command encoding fails, the failed command remains in the outstanding command stack without proper cleanup, corrupting the command-response matching mechanism.

Solution:

• Add encoding error tracking to RedisCommand interface with markEncodingError() and hasEncodingError() methods
• Mark commands with encoding errors in CommandEncoder when encode failures occur
• Implement lazy cleanup of encoding-failed commands in CommandHandler response processing
• Add comprehensive unit and integration tests for encoding error scenarios

Changes Made

• RedisCommand Interface: Added encoding error tracking methods
• Command Class: Implemented encoding error flag with volatile boolean for thread safety
• CommandEncoder: Mark commands on encoding failures
• CommandHandler: Enhanced response processing to handle encoding-failed commands
• All RedisCommand Implementations: Updated to support encoding error tracking
• Tests: Added comprehensive unit tests (EncodingErrorHandlingTests) and integration tests (EncodingErrorIntegrationTests, CommandEncodingErrorIntegrationTests)

Test Plan

• Unit tests covering encoding error tracking and cleanup logic
• Integration tests simulating real encoding failure scenarios
• Existing test suite passes
• Manual testing with deliberate encoding failures

Fixes

This resolves command queue corruption issues that could occur when Redis commands fail during encoding, ensuring proper command-response synchronization is maintained.

[tishun]

oopsie

[jit-ci]

❌ The following Jit checks failed to run:

• secret-detection-trufflehog
• static-code-analysis-semgrep-pro

#jit_bypass_commit in this PR to bypass, Jit Admin privileges required.

More info in the Jit platform.

[tishun]

Addressing the most visible places where out-of-order processing might be caused due to invalid state of the driver or the JVM, most notably :

Encoding exceptions in the RedisCodec implementations

Based on the original PR

Observation:

The CommandEncoder is called after the commands are assumed to be written on the wire by the CommandHandler and thus put in the stack of issued commands. However the CommandEncoder uses different encoders, which could be custom user-provided encoders, and they might throw an Exception. As explained by @mp911de in #2012 (comment) the driver enters an invalid state and a disconnect is advised in this case.

Test case:

CodecFailureIntegrationTests

Solution:

Users are expected to close the connection, however nobody reads exception messages in their error handlers.
Furthermore some exceptions, such as - but not limited to - OutOfMemoryErrors, could not be avoided in the code.

As such a more radical approach should be taken by directly closing the connection.

Alternatives:

The original suggestion by @yangy0000 (as well as my first attempt) were aiming to recover from this error state. However the driver is very complex and there are a myriad of invariants involving time-outs and batching that might be hard to handle.

To avoid reaching and inconsistent state we assume that such errors are not recoverable.

Throwable errors in the Reactive chain

Observation:

The Lettuce driver - by default - uses the IO thread pull (unless configured otherwise) to process the reactive streams.
As a result an OutOfMemory error would essentially break the IO thread loop's processing for both the stream and the whole command.

Test case:

ReactiveStreamErrorIntegrationTests

Solution:

Catch Throwable instead of Exception and process the failure.

[tishun]

Suggest crossporting this to:

• 7.2.x
• 7.1.x
• 7.0.x
• 6.8.x