Skip to content

Migrate MemcacheSocket implementation to Rust#80

Merged
bisho merged 1 commit into
mainfrom
03-11-adopt_socket_from_meta-memcache-sockey-py
Mar 26, 2026
Merged

Migrate MemcacheSocket implementation to Rust#80
bisho merged 1 commit into
mainfrom
03-11-adopt_socket_from_meta-memcache-sockey-py

Conversation

@bisho
Copy link
Copy Markdown
Member

@bisho bisho commented Mar 11, 2026

Upgrade to meta-memcache-socket 0.2 and migrate MemcacheSocket implementation

This pull request upgrades the meta-memcache-socket dependency from version 0.1.7 to 0.2 and migrates the codebase to use the external MemcacheSocket implementation.

Changes Made

  • Dependency upgrade: Updated meta-memcache-socket from >= 0.1.7, < 0.2 to >= 0.2, < 0.3 in pyproject.toml
  • MemcacheSocket migration: Replaced the Python MemcacheSocket implementation with a simple import from the meta-memcache-socket package, removing ~250 lines of code
  • Protocol imports: Updated protocol.py to import response types (Value, Success, Miss, NotStored, Conflict) and server version constants from the external package
  • Value handling: Modified executor to use result.value directly instead of calling conn.get_value(), as the external implementation populates the value field automatically, reducing the number of calls into rust code.
  • Test refactoring: Updated tests to use real socket pairs and mock fixtures instead of manual socket mocking, providing more realistic test coverage

Performance

Common config: server 127.0.0.1:11211, 10 threads, 1.00M requests (4 runs × 25,000 ops × 10 threads), 200 key space, GC disabled, 160/200 keys populated (95% small 80-250B, 5% large 100KB)                                                                                
                                                                                                                                                                                                                                                                              
  ┌───────────────┬────────┬─────────┬──────────────────────┐                                                                                                                                                                                                                                     
  │               │        │         │    Socket in Rust    │                                                                                  
  │   Operation   │        │ BEFORE  ├──────────┬───────────┤                                                                                                                                                                                                                                     
  │               │        │         │  Result  │  Speedup  │                                                                                                                                                                                                                                     
  ├───────────────┼────────┼─────────┼──────────┼───────────┤                                                                                                                                                                                                                                     
  │               │ RPS    │  46,600 │   76,652 │           │                                                                                                                                                                                                                                     
  │  SET          ├────────┼─────────┼──────────┤   1.65x   │                                                                                                                                                                                                                                     
  │               │ us/req │   21.46 │    13.05 │           │                                                                                                                                                                                                                                     
  ├───────────────┼────────┼─────────┼──────────┼───────────┤                                                                                                                                                                                                                                     
  │               │ RPS    │  46,135 │   81,401 │           │                                                                                                                                                                                                                                     
  │  GET          ├────────┼─────────┼──────────┤   1.76x   │                                                                                
  │               │ us/req │   21.68 │    12.28 │           │                                                                                                                                                                                                                                     
  ├───────────────┼────────┼─────────┼──────────┼───────────┤                                                                                                                                                                                                                                     
  │               │ RPS    │   2,677 │    3,626 │           │                                                                                                                                                                                                                                     
  │  MULTI_GET(5) ├────────┼─────────┼──────────┤   1.35x   │                                                                                                                                                                                                                                     
  │               │ us/req │  373.52 │   275.81 │           │                                                                                
  ├───────────────┼────────┼─────────┼──────────┼───────────┤                                                                                                                                                                                                                                     
  │               │ RPS    │  48,080 │   82,886 │           │                                                                                
  │  DELETE       ├────────┼─────────┼──────────┤   1.72x   │                                                                                                                                                                                                                                     
  │               │ us/req │   20.80 │    12.06 │           │                                                                                
  └───────────────┴────────┴─────────┴──────────┴───────────┘

Technical Details

The external MemcacheSocket implementation now handles value parsing internally, populating the value field of Value responses directly. This eliminates the need for the separate get_value() call and simplifies the response handling logic.

Test fixtures have been consolidated to provide consistent mocking across the test suite, with mock_memcache_socket and mock_raw_socket fixtures replacing ad-hoc socket mocking.

Copy link
Copy Markdown
Member Author

bisho commented Mar 11, 2026

@bisho bisho marked this pull request as ready for review March 11, 2026 14:56
@bisho bisho force-pushed the 03-11-adopt_socket_from_meta-memcache-sockey-py branch from 5d1d82c to 0a3c9d8 Compare March 11, 2026 15:59
@bisho bisho force-pushed the 03-11-improve_benchmark_to_test_set_get_and_delete_and_populate_values branch from 04a0c2a to abc3e7c Compare March 13, 2026 14:43
@bisho bisho force-pushed the 03-11-adopt_socket_from_meta-memcache-sockey-py branch from 0a3c9d8 to 55bdd4c Compare March 13, 2026 14:43
@bisho bisho force-pushed the 03-11-adopt_socket_from_meta-memcache-sockey-py branch from 55bdd4c to 275397f Compare March 16, 2026 16:12
@bisho bisho requested a review from a team March 24, 2026 17:00
@bisho bisho force-pushed the 03-11-adopt_socket_from_meta-memcache-sockey-py branch 2 times, most recently from e9b01f1 to e462f68 Compare March 25, 2026 12:38
@bisho bisho force-pushed the 03-11-improve_benchmark_to_test_set_get_and_delete_and_populate_values branch from abc3e7c to 135ffbb Compare March 25, 2026 12:38
@bisho bisho force-pushed the 03-11-adopt_socket_from_meta-memcache-sockey-py branch 2 times, most recently from df722d9 to 5b095d3 Compare March 26, 2026 14:11
@bisho bisho force-pushed the 03-11-improve_benchmark_to_test_set_get_and_delete_and_populate_values branch from 135ffbb to b420700 Compare March 26, 2026 14:11
Copy link
Copy Markdown

@xmartinez xmartinez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG! Just one nit (optional).

Comment thread src/meta_memcache/connection/memcache_socket.py
Comment thread src/meta_memcache/protocol.py
@bisho bisho force-pushed the 03-11-adopt_socket_from_meta-memcache-sockey-py branch from 5b095d3 to 0b04f55 Compare March 26, 2026 15:13
Copy link
Copy Markdown
Member Author

bisho commented Mar 26, 2026

Merge activity

  • Mar 26, 3:31 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Mar 26, 3:34 PM UTC: Graphite rebased this pull request as part of a merge.
  • Mar 26, 3:35 PM UTC: @bisho merged this pull request with Graphite.

@bisho bisho changed the base branch from 03-11-improve_benchmark_to_test_set_get_and_delete_and_populate_values to main March 26, 2026 15:32
@bisho bisho force-pushed the 03-11-adopt_socket_from_meta-memcache-sockey-py branch from 0b04f55 to 41dc43e Compare March 26, 2026 15:33
@bisho bisho merged commit 8f29e0b into main Mar 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants