Skip to content

feat: add allow_custom_keys support, requestBulkUploadUrls mutation, auto-path derivation#1058

Merged
pyramation merged 2 commits intomainfrom
feat/allow-custom-keys
May 5, 2026
Merged

feat: add allow_custom_keys support, requestBulkUploadUrls mutation, auto-path derivation#1058
pyramation merged 2 commits intomainfrom
feat/allow-custom-keys

Conversation

@pyramation
Copy link
Copy Markdown
Contributor

@pyramation pyramation commented May 5, 2026

Summary

Phase 3a + 3b of the storage roadmap: adds allow_custom_keys support to the presigned URL plugin, the requestBulkUploadUrls mutation, and auto-path derivation from custom keys.

Custom key support (Phase 3a):

  • requestUploadUrl now accepts an optional key input field
  • When allow_custom_keys=true on the bucket, clients can provide a custom S3 key (e.g., reports/2024/Q1.pdf)
  • When key is omitted, falls back to content-hash-based dedup (today's behavior)
  • Custom key validation: no traversal (..), no leading /, max 1024 chars, regex-validated
  • Re-uploading to an existing custom key auto-creates a new version (links via previous_version_id)
  • RequestUploadUrlPayload now includes previousVersionId field

Bulk upload mutation (Phase 3b):

  • New requestBulkUploadUrls mutation — batch presigned URL generation
  • Enforces maxBulkFiles and maxBulkTotalSize from storage module config (nullable DB columns with plugin-level defaults: 100 files / 1GB)
  • Shared processSingleFile() logic between single and bulk mutations

Auto-path derivation:

  • When allow_custom_keys=true AND has_path_shares=true, path ltree column auto-derived from S3 key directory (e.g., reports/2024/Q1/revenue.pdfreports.2024.Q1)

Cache/type updates:

  • Storage module queries now select has_path_shares, max_bulk_files, max_bulk_total_size
  • Bucket config queries now select allow_custom_keys

Review & Testing Checklist for Human

  • Verify custom key validation rejects path traversal (..), leading /, null bytes, and keys > 1024 chars
  • Verify dedup works in both modes: content_hash lookup in hash mode, key+version chain in custom mode
  • Verify bulk upload limits enforcement (maxBulkFiles, maxBulkTotalSize) returns correct errors
  • Verify auto-path derivation produces correct ltree values from S3 key directory portions
  • Run upload integration tests with MinIO: cd graphql/server-test && pnpm test -- upload.integration
  • Test that allow_custom_keys=false (default) rejects custom keys with CUSTOM_KEY_NOT_ALLOWED error

Notes

Link to Devin session: https://app.devin.ai/sessions/ffa3ed8652fc412f976accbdc229c88d
Requested by: @pyramation

…auto-path derivation

- Add optional 'key' input to requestUploadUrl for custom S3 keys
- Custom keys require bucket.allow_custom_keys=true
- Re-uploading to existing custom key auto-versions (previous_version_id)
- Dedup by content_hash in both hash-based and custom-key modes
- Auto-derive ltree path from custom key directory when has_path_shares=true
- Add requestBulkUploadUrls mutation with per-storage-module limits
- Add max_bulk_files and max_bulk_total_size to StorageModuleConfig
- Add has_path_shares to StorageModuleConfig for path column awareness
- Validate custom keys: no traversal, no leading slash, alphanumeric start
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@pyramation pyramation merged commit 062ca76 into main May 5, 2026
53 checks passed
@pyramation pyramation deleted the feat/allow-custom-keys branch May 5, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant