Skip to content

fix(e2e): send SSE-C key MD5 in Base64 per S3 spec#4

Open
andybrown668 wants to merge 15 commits intomainfrom
fix/ssec-key-md5-base64-e2e
Open

fix(e2e): send SSE-C key MD5 in Base64 per S3 spec#4
andybrown668 wants to merge 15 commits intomainfrom
fix/ssec-key-md5-base64-e2e

Conversation

@andybrown668
Copy link

@andybrown668 andybrown668 commented Mar 12, 2026

Type of Change

  • Bug Fix
  • New Feature
  • Documentation
  • Performance Improvement
  • Test/CI
  • Refactor
  • Other:

Related Issues

N/A

Summary of Changes

E2E tests were sending the SSE-C key MD5 header in hex format (format!("{:x}", md5::compute(key))), while the server and S3 API expect Base64-encoded 128-bit MD5 digest of the encryption key. This caused test_local_kms_key_isolation and any SSE-C flow to fail with "The calculated MD5 hash of the key did not match the hash that was provided."

Updated all e2e tests that use SSE-C to send the key MD5 in Base64:

  • base64::Engine::encode(..., md5::compute(key).0) (or equivalent with existing compute import).

Files touched: kms_local_test.rs, multipart_encryption_test.rs, kms_vault_test.rs, kms_edge_cases_test.rs, kms_comprehensive_test.rs, common.rs.

Checklist

  • I have read and followed the CONTRIBUTING.md guidelines
  • Passed make pre-commit
  • Added/updated necessary tests (existing tests fixed; no new tests)
  • Documentation updated (if needed)
  • CI/CD passed (if applicable)

Impact

  • Breaking change (compatibility)
  • Requires doc/config/deployment update
  • Other impact:

Additional Notes

Verification: cargo test -p e2e_test kms::kms_local_test::test_local_kms_key_isolation passes. Requesting review from Copilot.

@andybrown668 andybrown668 requested a review from Copilot March 12, 2026 11:09
Copilot AI reviewed Mar 12, 2026
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes SSE-C E2E test failures by generating the x-amz-server-side-encryption-customer-key-MD5 header as Base64 of the raw 16-byte MD5 digest (per S3 spec), instead of hex encoding.

Changes:

  • Replace hex-formatted MD5 (format!("{:x}", ...)) with Base64-encoded digest bytes (...encode(..., md5::compute(key).0) / compute(key).0) across SSE-C test flows.
  • Add/adjust inline comments clarifying the S3 SSE-C MD5 header requirements.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
crates/e2e_test/src/kms/multipart_encryption_test.rs Updates SSE-C multipart helper to send Base64 MD5 digest bytes.
crates/e2e_test/src/kms/kms_vault_test.rs Fixes Vault SSE-C key isolation test to send Base64 MD5 digest bytes.
crates/e2e_test/src/kms/kms_local_test.rs Fixes Local KMS SSE-C MD5 headers and adds clarifying comments.
crates/e2e_test/src/kms/kms_edge_cases_test.rs Fixes multiple SSE-C edge-case scenarios to use Base64 MD5 digest bytes.
crates/e2e_test/src/kms/kms_comprehensive_test.rs Fixes comprehensive SSE-C isolation/wrong-key flows to use Base64 MD5 digest bytes.
crates/e2e_test/src/kms/common.rs Fixes shared SSE-C helpers/config to use Base64 MD5 digest bytes.

You can also share your feedback on Copilot code review. Take the survey.

@andybrown668
fix(e2e): send SSE-C key MD5 in Base64 per S3 spec
3ff3193 
E2E tests sent x-amz-server-side-encryption-customer-key-MD5 in hex
while server expects Base64-encoded 128-bit MD5. Fix all SSE-C usages
across kms_local_test, kms_vault_test, kms_edge_cases_test,
kms_comprehensive_test, multipart_encryption_test, and common.

Made-with: Cursor
@andybrown668
e2e: always build rustfs once and only once
2bd6215 
- In common.rs: when CARGO_BIN_EXE_rustfs is unset, always build rustfs
  once per process via Once (remove 'use existing binary if present' path)
  so the e2e harness uses a single, consistent binary.
- Document in tests.mak that e2e setup builds rustfs once and only once.
- Add e2e-test make target; RUSTFS_BUILD_FEATURES=ftps for protocol tests.

Made-with: Cursor
@andybrown668
refactor(e2e): add run_test_* bodies for unified KMS runner
ba5295c 
Extract async test bodies into run_test_* functions callable from both
#[tokio::test] and the unified test runner. Fixes E0277 'not a future'
and '() is not a future' by having the runner await real async futures
instead of #[tokio::test] sync wrappers.

- kms_local_test: run_test_local_kms_key_isolation, run_test_local_kms_multipart_upload
- multipart_encryption_test: run_test_step1..step5
- kms_edge_cases_test: run_test_kms_* (6 tests)
- kms_fault_recovery_test: run_test_kms_* (4 tests)
- kms_comprehensive_test: run_test_comprehensive_* (5 tests)
- test_runner: dispatch to all run_test_* functions

Made-with: Cursor
@andybrown668
chore(e2e): ensure awscurl for e2e tests and improve missing-tool errors
1dc401f 
- Add check-awscurl, install-awscurl, ensure-awscurl in .config/make/check.mak
- Prefer pipx install to avoid externally-managed-environment; fallback to pip --user
- Use ${AWSCURL_PATH:-} so unset var is safe with set -u
- Make e2e-server and probe-e2e depend on ensure-awscurl in tests.mak
- In common.rs, surface clearer errors when rustfs binary or awscurl is missing

Made-with: Cursor
@andybrown668
feat(rio): add ExactLengthReader for decrypted GET body length valida…
e0178cd 
…tion

Errors if stream ends before expected bytes to avoid IncompleteBody when
decryption produces fewer bytes than Content-Length.

Made-with: Cursor
@andybrown668
feat(kms): use UTC timestamps and encryption context for DEK
8fdcb8b 
- Add zoned_now_utc() and ZonedUtcCompatible for consistent serialization
- Pass encryption_context into decrypt_sse_dek for envelope verification
- Add decrypt_data_key_with_context; trim base64 in headers_to_metadata
- SSE-S3: do not store KMS key ID in metadata; key_id fallback to 'default'
- Skip DEK cache when encryption_context is set (context-bound DEKs)

Made-with: Cursor
@andybrown668
feat(storage): SSE encryption context, ExactLengthReader, metadata fi…
c9ed1ec 
…lter

- Use ExactLengthReader in decryption path to avoid IncompleteBody
- Persist S3 SSE type (AES256/aws:kms) and bucket/object_key in context
- decrypt_sse_dek takes optional encryption_context for envelope verification
- Add filter_object_metadata_with_options to include x-rustfs-encryption-* for Head/Get
- Only store x-amz-server-side-encryption-aws-kms-key-id for aws:kms
- Add in-process SSE-S3 roundtrip test with global KMS

Made-with: Cursor
@andybrown668
feat(app): SSE-S3 key ID semantics, metadata filter, versioning, encr…
477d08b 
…ypted GET

- Do not expose ssekms_key_id for SSE-S3 (AES256) in Put/Get/Multipart responses
- Use filter_object_metadata_with_options(..., true) for Get/Head to include encryption metadata
- PutObject: return VersionId when versioning enabled or suspended; use 'null' for null version
- GetObject: set decryption part_number=1 for single-part multipart encrypted objects
- Preload small encrypted GET bodies (<=8MB) to validate decryption before sending

Made-with: Cursor
@andybrown668
fix(ecstore): invalidate file cache after write_all
b44a4a7 
Made-with: Cursor
@andybrown668
feat(policy): allow Resource "*" as IAM shorthand for all resources
bd2e9d1 
Made-with: Cursor
@andybrown668
test(e2e): server env, process group, KMS key format, vault seed
459cce5 
- Add start_rustfs_server_with_env for extra env (e.g. RUSTFS_SCANNER_SPEED)
- Run server in own process group to avoid signal propagation to test runner
- Local KMS: store encrypted_key_material as base64 in key JSON
- Vault: ensure KV v2 at secret, seed default key in KV for backend

Made-with: Cursor
@andybrown668
test(e2e): use shared env and un-ignore tests
5ec19a5 
- Data usage: poll for scanner, RUSTFS_SCANNER_SPEED=fastest, 200 objects
- Group delete: un-ignore, retry on connection reset for set-policy
- Policy: spawn RustFS via RustFSTestEnvironment, use dynamic address
- Reliant/ftps: use common env and dynamic endpoint, un-ignore

Made-with: Cursor
@andybrown668 andybrown668 force-pushed the fix/ssec-key-md5-base64-e2e branch 2 times, most recently from 409838f to 7470016 Compare March 16, 2026 13:59
@andybrown668
ci(build): restrict fork builds to x86_64-unknown-linux-gnu
277a584 
Compute build matrix in build-check; upstream keeps full matrix,
forks only build x86_64-unknown-linux-gnu. Format matrix JSON
multiline for readability.

Made-with: Cursor
@andybrown668 andybrown668 force-pushed the fix/ssec-key-md5-base64-e2e branch from 7470016 to 277a584 Compare March 16, 2026 14:08
@andybrown668
ci: install protoc via direct download instead of arduino/setup-protoc
a10b5e1 
Avoid GitHub API timeouts and Node.js 20 deprecation by downloading
protoc 33.1 from protocolbuffers/protobuf releases for Linux, macOS,
and Windows.

Made-with: Cursor
@andybrown668 andybrown668 force-pushed the fix/ssec-key-md5-base64-e2e branch 2 times, most recently from 99a837e to 2ca6d5e Compare March 16, 2026 15:55
@andybrown668
ci: skip cargo-nextest install when already at pinned version
d80eeb5 
Check for cargo-nextest 0.2.30 before installing to avoid redundant
reinstall when cached.

Made-with: Cursor
@andybrown668 andybrown668 force-pushed the fix/ssec-key-md5-base64-e2e branch from 2ca6d5e to d80eeb5 Compare March 17, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@andybrown668