Skip to content

Move key encoding to socket api#15

Merged
bisho merged 1 commit into
mainfrom
03-12-move_key_encoding_to_rust
Mar 25, 2026
Merged

Move key encoding to socket api#15
bisho merged 1 commit into
mainfrom
03-12-move_key_encoding_to_rust

Conversation

@bisho
Copy link
Copy Markdown
Member

@bisho bisho commented Mar 13, 2026

Motivation / Description

This change enhances the memcache client to handle arbitrarily long keys by implementing automatic key hashing with Blake2b. Previously, keys were limited to 250 bytes (or 187 bytes for binary keys due to base64 encoding).

This was previously implemented in meta-memcache library in python, but supporting it at the socket level makes for a more robust api. The implementation is backwards compatible, and usages can implement their own key encoding if they wish to or they need backwards compatibility. The keys are unmodified as long as they are non-binary and < 187 bytes, which key encoders should guarantee.

Moving the default encoding to rust provides a small performance gains as well.

Changes introduced

  • Added Blake2b hashing dependency to automatically hash keys >= 187 bytes into compact 18-byte digests
  • Implemented key encoding logic that handles both string and bytes key types, with automatic base64 encoding for binary keys
  • Updated all key parameters in function signatures and type hints to accept Union[str, bytes] instead of just bytes
  • Modified command building functions to use the new key encoding system that ensures wire keys are always under the 250-byte memcache limit
  • Added comprehensive test coverage for key encoding scenarios including short keys, long keys, binary keys, and edge cases
  • Updated error messages to reflect that empty keys are rejected rather than "too long" keys, since long keys are now automatically handled

@bisho bisho marked this pull request as ready for review March 13, 2026 15:00
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch 2 times, most recently from 669508e to 5f8b812 Compare March 16, 2026 17:59
@bisho bisho force-pushed the 03-11-direct_meta_commands_api_in_socket branch from 05a41dd to 8f8b51e Compare March 16, 2026 17:59
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from 5f8b812 to c1493e0 Compare March 16, 2026 22:06
@bisho bisho force-pushed the 03-11-direct_meta_commands_api_in_socket branch from 8f8b51e to 53428a9 Compare March 16, 2026 22:06
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from c1493e0 to a566b68 Compare March 16, 2026 23:13
@bisho bisho force-pushed the 03-11-direct_meta_commands_api_in_socket branch 2 times, most recently from 978f263 to e10435b Compare March 17, 2026 02:09
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from a566b68 to 3f2d7e2 Compare March 17, 2026 02:09
@bisho bisho changed the title Add blake2 hashing for long keys and accept str/bytes keys Move key encoding to socket api Mar 17, 2026
@bisho bisho requested a review from a team March 17, 2026 08:39
@bisho bisho force-pushed the 03-11-direct_meta_commands_api_in_socket branch from e10435b to 6ebdcec Compare March 17, 2026 08:55
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from 3f2d7e2 to 9f5a863 Compare March 17, 2026 08:55
This was referenced Mar 17, 2026
@ethervoid
Copy link
Copy Markdown

@claude review this PR

@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from 9f5a863 to f9ac819 Compare March 24, 2026 09:16
@bisho bisho force-pushed the 03-11-direct_meta_commands_api_in_socket branch from 6ebdcec to 7815d98 Compare March 24, 2026 09:16
@bisho bisho changed the base branch from 03-11-direct_meta_commands_api_in_socket to graphite-base/15 March 24, 2026 17:08
@bisho bisho force-pushed the graphite-base/15 branch from 7815d98 to 01deb21 Compare March 24, 2026 17:10
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from f9ac819 to f714a41 Compare March 24, 2026 17:10
@graphite-app graphite-app Bot changed the base branch from graphite-base/15 to main March 24, 2026 17:11
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch 3 times, most recently from 09802be to c5237c1 Compare March 25, 2026 09:28
Copy link
Copy Markdown

@xmartinez xmartinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG!

Added a comment re: test coverage (non blocking).

Comment thread tests/test_memcache_socket.py
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from c5237c1 to e46fe95 Compare March 25, 2026 14:18
@bisho bisho force-pushed the 03-12-move_key_encoding_to_rust branch from e46fe95 to 3cc05e3 Compare March 25, 2026 14:21
Copy link
Copy Markdown
Member Author

bisho commented Mar 25, 2026

Merge activity

  • Mar 25, 2:26 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Mar 25, 2:26 PM UTC: @bisho merged this pull request with Graphite.

@bisho bisho merged commit c103ed4 into main Mar 25, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants