feat: org-scoped batch/file filtering with metadata enrichment by hachall · Pull Request #839 · doublewordai/control-layer

hachall · 2026-03-10T18:56:33Z

Summary

- Security: Sensitive metadata (created_by_email, context_name, context_type) only returned when should_enrich = can_read_all || is_org_context. Non-privileged users get these keys scrubbed from responses. member_id filter returns 400 for unprivileged users.

Backend: Resolves individual creator email via api_key_id → api_keys.created_by, determines context (Personal vs org name) from owner's user_type. New find_hidden_key_id and get_creators_by_key_ids methods on ApiKeys repo. Server-side status, created_after, created_before filters passed through to fusillade.
Frontend: User and Context columns on both Batches and Files tables. Searchable member filter combobox (shows org members in org context, all users for PM personal view). Status filter dropdown, DateTimeRangeSelector for time windows, and active-first switch toggle (default on) for batches. Filters reset when org context changes.

Depends on

fusillade hamishhall/cor-193-api-key-id-attribution (api_key_id column + list_batches filter params)

Test plan

PM sees User + Context columns on both tabs; standard user does not
Org context user sees User column, member filter dropdown with org members
PM in personal view sees member filter with all platform users
Standard user does not see member filter, gets 400 if they try member_id via API
Batch status filter works server-side for each status value
Time range filter works with DateTimeRangeSelector
Active-first toggle sorts active batches to the top (default on)
Switching org context clears member and status filters
Search works for batch IDs (not just filenames/endpoints)
~~- [ ] Non-privileged API responses don't contain created_by_email/context_name/context_type~~

security constraint removed - if you have the priviledge to see the resources, you can see who created them (currently)

…g change

…-filtering

…shboard filters Security: Gate sensitive metadata (created_by_email, context_name, context_type) behind should_enrich = can_read_all || is_org_context. Standard users see no enrichment; non-privileged metadata keys are scrubbed from responses. Backend: - Resolve individual creator via api_key_id -> api_keys.created_by - Resolve context (Personal vs org name) from batch/file owner user_type - member_id param translates to api_key_id via find_hidden_key_id, returns 400 for unprivileged users - Add status, created_after, created_before server-side filters for batches - Add get_creators_by_key_ids and find_hidden_key_id to ApiKeys repo - Add created_by_email, context_name, context_type to FileResponse Frontend: - User and Context columns on both Batches and Files tables - Searchable member filter combobox (Popover+Command) shared across tabs - Server-side status filter dropdown for batches - DateTimeRangeSelector for time window filtering - Active-first switch toggle (default on) for client-side sort - Member filter uses org members in org context, all users for PM personal view - Filters reset on org context change

cloudflare-workers-and-pages · 2026-03-10T18:57:46Z

Deploying control-layer with Cloudflare Pages

Latest commit:	`ecde806`
Status:	🚫 Build failed.

View logs

…-filtering

Copilot

Pull request overview

This PR adds organization-scoped per-member attribution and filtering for batch/files by propagating api_key_id into Fusillade metadata, exposing new query parameters, and updating the dashboard UI to support member/status/date filtering and richer display.

Changes:

Backend: create/lookup hidden per-member batch keys with IDs and pass api_key_id into Fusillade for attribution.
Backend: add member_id (and batch status/created_after/created_before) query parameters; enrich list responses with creator/context metadata in privileged contexts.
Dashboard: add member picker + batch filters (status/date) and optional “User/Context” columns; extend API client/types accordingly.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
dwctl/src/db/handlers/api_keys.rs	Adds hidden-key helpers returning key ID and bulk creator lookup by key IDs
dwctl/src/api/models/files.rs	Adds `member_id` query param and new response fields for creator/context
dwctl/src/api/models/batches.rs	Adds `member_id`, `status`, and created-at range filters to query model
dwctl/src/api/handlers/users.rs	Updates `list_batches` call signature
dwctl/src/api/handlers/files.rs	Passes `api_key_id` into file upload metadata; adds member filter translation + enrichment
dwctl/src/api/handlers/batches.rs	Passes `api_key_id` into batch creation; adds member/status/date filters + enrichment/scrubbing
dwctl/Cargo.toml	Switches `fusillade` dependency to a local path
dashboard/src/components/features/batches/FilesTable/columns.tsx	Adds optional User/Context columns for files
dashboard/src/components/features/batches/BatchesTable/columns.tsx	Adds optional Context column for batches
dashboard/src/components/features/batches/Batches/Batches.tsx	Adds member filter combobox, status/date filters, active-first sorting, and column toggles
dashboard/src/api/control-layer/types.ts	Adds new fields and query params to TS types
dashboard/src/api/control-layer/client.ts	Adds query string serialization for new filters
dashboard/public/mockServiceWorker.js	Updates MSW worker package version constant
Cargo.lock	Updates locked `fusillade` package entry

Copilot

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (1)

dwctl/src/api/handlers/batches.rs:1367

api_key_ids is collected with potential duplicates (many batches can share the same api_key_id). Passing duplicates into WHERE id = ANY($1) causes unnecessary DB work and larger query payloads. Consider deduplicating the IDs (e.g., collect into a HashSet first) before calling get_creators_by_key_ids.

        })?;

    // Check if there are more results
    let has_more = batches.len() > limit as usize;
    let batches: Vec<_> = batches.into_iter().take(limit as usize).collect();

Copilot

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 7 comments.

sejori

Looks great 👍

…-filtering

Copilot

Pull request overview

Copilot reviewed 24 out of 32 changed files in this pull request and generated 6 comments.

Files not reviewed (7)

.sqlx/query-0b14bc0b0c7959f6239978fb0ebb575b900c68cc28523febc06c357cc5cbd5a4.json: Language not supported
.sqlx/query-1ee6056408732a6dfb352d87f0f567fbbee435c56775bb041665e874b0dd0fe0.json: Language not supported
.sqlx/query-22d62e4fd39ac3823c45d40afccbd5f559bbe70cd4bb27437af5bc1e87ac9579.json: Language not supported
.sqlx/query-5f3ef24b56af36c470a89750ae098d36d8a0ba0b1435c7136341391226883e2a.json: Language not supported
.sqlx/query-7c95e8ddbc4a6c06373eaabe607172a810d44690d0cce5185f99b115593e95ef.json: Language not supported
.sqlx/query-7cc98fc214085ea12f6bc5d21fa80e8f06f34f645a723c5f58053ee643ed51ac.json: Language not supported
.sqlx/query-d45e2c89b9002fa9dbb60ea6fd39f9756a586129fe3528bb020dd4c612789baf.json: Language not supported

Copilot

Pull request overview

Copilot reviewed 27 out of 35 changed files in this pull request and generated 2 comments.

Files not reviewed (7)

.sqlx/query-0b14bc0b0c7959f6239978fb0ebb575b900c68cc28523febc06c357cc5cbd5a4.json: Language not supported
.sqlx/query-1ee6056408732a6dfb352d87f0f567fbbee435c56775bb041665e874b0dd0fe0.json: Language not supported
.sqlx/query-22d62e4fd39ac3823c45d40afccbd5f559bbe70cd4bb27437af5bc1e87ac9579.json: Language not supported
.sqlx/query-5f3ef24b56af36c470a89750ae098d36d8a0ba0b1435c7136341391226883e2a.json: Language not supported
.sqlx/query-7c95e8ddbc4a6c06373eaabe607172a810d44690d0cce5185f99b115593e95ef.json: Language not supported
.sqlx/query-7cc98fc214085ea12f6bc5d21fa80e8f06f34f645a723c5f58053ee643ed51ac.json: Language not supported
.sqlx/query-d45e2c89b9002fa9dbb60ea6fd39f9756a586129fe3528bb020dd4c612789baf.json: Language not supported

graphite-app · 2026-03-11T20:52:50Z


    # Timeout Settings
-    timeout_ms: 600000 # Timeout per request attempt (10 minutes)
+    # timeout_ms: 600000 # DEPRECATED: splits into 90% first_chunk_timeout_ms, 10% body_timeout_ms


Documentation incorrectly states timeout_ms "splits into 90% first_chunk_timeout_ms, 10% body_timeout_ms". The actual implementation in dwctl/src/config.rs lines 1180-1187 applies the deprecated timeout_ms value uniformly to all three timeout fields (first_chunk_timeout_ms, chunk_timeout_ms, body_timeout_ms), not as a 90/10 split. This documentation mismatch could confuse operators about the actual timeout behavior.

# Change line 294 from: # timeout_ms: 600000 # DEPRECATED: splits into 90% first_chunk_timeout_ms, 10% body_timeout_ms # To: # timeout_ms: 600000 # DEPRECATED: if set, applies uniformly to all three granular timeout fields

Suggested change

# timeout_ms: 600000 # DEPRECATED: splits into 90% first_chunk_timeout_ms, 10% body_timeout_ms

# timeout_ms: 600000 # DEPRECATED: if set, applies uniformly to all three granular timeout fields

Spotted by Graphite

Is this helpful? React 👍 or 👎 to let us know.

Copilot

Pull request overview

Copilot reviewed 27 out of 35 changed files in this pull request and generated 4 comments.

Files not reviewed (7)

.sqlx/query-0b14bc0b0c7959f6239978fb0ebb575b900c68cc28523febc06c357cc5cbd5a4.json: Language not supported
.sqlx/query-1ee6056408732a6dfb352d87f0f567fbbee435c56775bb041665e874b0dd0fe0.json: Language not supported
.sqlx/query-22d62e4fd39ac3823c45d40afccbd5f559bbe70cd4bb27437af5bc1e87ac9579.json: Language not supported
.sqlx/query-5f3ef24b56af36c470a89750ae098d36d8a0ba0b1435c7136341391226883e2a.json: Language not supported
.sqlx/query-7c95e8ddbc4a6c06373eaabe607172a810d44690d0cce5185f99b115593e95ef.json: Language not supported
.sqlx/query-7cc98fc214085ea12f6bc5d21fa80e8f06f34f645a723c5f58053ee643ed51ac.json: Language not supported
.sqlx/query-d45e2c89b9002fa9dbb60ea6fd39f9756a586129fe3528bb020dd4c612789baf.json: Language not supported

Copilot

Pull request overview

Copilot reviewed 27 out of 35 changed files in this pull request and generated 3 comments.

Files not reviewed (7)

.sqlx/query-0b14bc0b0c7959f6239978fb0ebb575b900c68cc28523febc06c357cc5cbd5a4.json: Language not supported
.sqlx/query-1ee6056408732a6dfb352d87f0f567fbbee435c56775bb041665e874b0dd0fe0.json: Language not supported
.sqlx/query-22d62e4fd39ac3823c45d40afccbd5f559bbe70cd4bb27437af5bc1e87ac9579.json: Language not supported
.sqlx/query-5f3ef24b56af36c470a89750ae098d36d8a0ba0b1435c7136341391226883e2a.json: Language not supported
.sqlx/query-7c95e8ddbc4a6c06373eaabe607172a810d44690d0cce5185f99b115593e95ef.json: Language not supported
.sqlx/query-7cc98fc214085ea12f6bc5d21fa80e8f06f34f645a723c5f58053ee643ed51ac.json: Language not supported
.sqlx/query-d45e2c89b9002fa9dbb60ea6fd39f9756a586129fe3528bb020dd4c612789baf.json: Language not supported

Copilot

Pull request overview

Copilot reviewed 27 out of 35 changed files in this pull request and generated 4 comments.

Files not reviewed (7)

.sqlx/query-0b14bc0b0c7959f6239978fb0ebb575b900c68cc28523febc06c357cc5cbd5a4.json: Language not supported
.sqlx/query-1ee6056408732a6dfb352d87f0f567fbbee435c56775bb041665e874b0dd0fe0.json: Language not supported
.sqlx/query-22d62e4fd39ac3823c45d40afccbd5f559bbe70cd4bb27437af5bc1e87ac9579.json: Language not supported
.sqlx/query-5f3ef24b56af36c470a89750ae098d36d8a0ba0b1435c7136341391226883e2a.json: Language not supported
.sqlx/query-7c95e8ddbc4a6c06373eaabe607172a810d44690d0cce5185f99b115593e95ef.json: Language not supported
.sqlx/query-7cc98fc214085ea12f6bc5d21fa80e8f06f34f645a723c5f58053ee643ed51ac.json: Language not supported
.sqlx/query-d45e2c89b9002fa9dbb60ea6fd39f9756a586129fe3528bb020dd4c612789baf.json: Language not supported

Copilot · 2026-03-11T21:27:15Z

+            first_chunk_timeout_ms: 86_400_000,
+            chunk_timeout_ms: 86_400_000,
+            body_timeout_ms: 86_400_000,


The default request timeouts for the batch daemon are effectively changed from the old timeout_ms: 600000 default to 24h for header/chunk/body. Even with the deprecated-field compatibility logic, deployments relying on defaults (or missing config sections) will now allow very long-lived stuck requests, which can tie up connections/workers. Consider keeping the defaults closer to the previous 10-minute behavior (or explicitly documenting this as an intentional operational change and requiring opt-in via config).

Suggested change

first_chunk_timeout_ms: 86_400_000,

chunk_timeout_ms: 86_400_000,

body_timeout_ms: 86_400_000,

first_chunk_timeout_ms: 600_000,

chunk_timeout_ms: 600_000,

body_timeout_ms: 600_000,

Copilot

Pull request overview

Copilot reviewed 27 out of 35 changed files in this pull request and generated 3 comments.

Files not reviewed (7)

.sqlx/query-0b14bc0b0c7959f6239978fb0ebb575b900c68cc28523febc06c357cc5cbd5a4.json: Language not supported
.sqlx/query-1ee6056408732a6dfb352d87f0f567fbbee435c56775bb041665e874b0dd0fe0.json: Language not supported
.sqlx/query-22d62e4fd39ac3823c45d40afccbd5f559bbe70cd4bb27437af5bc1e87ac9579.json: Language not supported
.sqlx/query-5f3ef24b56af36c470a89750ae098d36d8a0ba0b1435c7136341391226883e2a.json: Language not supported
.sqlx/query-7c95e8ddbc4a6c06373eaabe607172a810d44690d0cce5185f99b115593e95ef.json: Language not supported
.sqlx/query-7cc98fc214085ea12f6bc5d21fa80e8f06f34f645a723c5f58053ee643ed51ac.json: Language not supported
.sqlx/query-d45e2c89b9002fa9dbb60ea6fd39f9756a586129fe3528bb020dd4c612789baf.json: Language not supported

Copilot · 2026-03-11T21:40:33Z

+  // Apply client-side filters and sorting to batches
  const filteredBatches = React.useMemo(() => {
-    if (!batchFileFilter) return batches;
-    return batches.filter((b) => b.input_file_id === batchFileFilter);
-  }, [batches, batchFileFilter]);
+    let result = batches;
+
+    // Filter by input file (client-side, from file detail view)
+    if (batchFileFilter) {
+      result = result.filter((b) => b.input_file_id === batchFileFilter);
+    }
+
+    // Sort active batches first if toggled
+    if (sortActiveFirst) {
+      const activeStatuses = new Set([
+        "validating",
+        "in_progress",
+        "finalizing",
+        "cancelling",
+      ]);
+      result = [...result].sort((a, b) => {
+        const aActive = activeStatuses.has(a.status) ? 0 : 1;
+        const bActive = activeStatuses.has(b.status) ? 0 : 1;
+        return aActive - bActive;
+      });
+    }


The “Active first” toggle sorts batches client-side after server-side cursor pagination. With cursor-based paging this only reorders the current page, so active batches on later pages won’t be surfaced, and the visible order won’t match the cursor order users are paging through (can feel inconsistent when navigating pages). Consider implementing this sort server-side (preferred) or switching to a stable secondary sort key within groups and making it explicit that sorting is per-page.

Copilot · 2026-03-11T21:40:33Z

@@ -1009,14 +1133,33 @@ pub async fn list_files<P: PoolProvider>(
                None => Purpose::Batch, // Default to Batch for backwards compatibility
            };

+            // Resolve individual creator email via api_key_id
+            let individual_creator_id = f.api_key_id.and_then(|key_id| api_key_creator_map.get(&key_id).copied());
+            let created_by_email = individual_creator_id.and_then(|uid| user_map.get(&uid)).map(|u| u.email.clone());
+
+            // Resolve context from file owner (uploaded_by field)
+            let owner_id = f.uploaded_by.as_ref().and_then(|id| uuid::Uuid::parse_str(id).ok());
+            let owner = owner_id.and_then(|id| user_map.get(&id));
+            let (context_name, context_type) = match owner {
+                Some(u) if u.user_type == "organization" => {
+                    let name = u.display_name.clone().unwrap_or_else(|| u.email.clone());
+                    (Some(name), Some("organization".to_string()))
+                }
+                Some(_) => (Some("Personal".to_string()), Some("personal".to_string())),
+                None => (None, None),


Creator/context enrichment logic is duplicated between list_files (bulk path) and get_file (single-item path). This increases the risk of the two endpoints drifting (e.g., different fallbacks, context naming, or future fields). Consider extracting a shared helper that accepts the fusillade file(s) and returns the enriched FileResponse(s), reusing the bulk lookups for the single-item case where possible.

Copilot · 2026-03-11T21:40:33Z

+    // Enrich with creator/context metadata (same as list_batches)
+    let mut read_conn = state.db.read().acquire().await.map_err(|e| Error::Database(e.into()))?;
+
+    // Resolve individual creator email via api_key_id
+    let created_by_email = if let Some(api_key_id) = batch.api_key_id {
+        let creator_map = ApiKeys::new(&mut read_conn)
+            .get_creators_by_key_ids(vec![api_key_id])
+            .await
+            .map_err(Error::Database)?;
+        if let Some(&creator_id) = creator_map.get(&api_key_id) {
+            Users::new(&mut read_conn)
+                .get_bulk(vec![creator_id])
+                .await
+                .map_err(|e| Error::Internal {
+                    operation: format!("fetch creator user: {}", e),
+                })?
+                .get(&creator_id)
+                .map(|u| u.email.clone())
+        } else {
+            None
+        }
+    } else {
+        // Fall back to owner email (legacy batches without api_key_id)
+        if let Some(created_by) = batch.created_by.as_ref() {
+            if let Ok(user_id) = Uuid::parse_str(created_by) {
+                Users::new(&mut read_conn)
+                    .get_bulk(vec![user_id])
+                    .await
+                    .map_err(|e| Error::Internal {
+                        operation: format!("fetch owner user: {}", e),
+                    })?
+                    .get(&user_id)
+                    .map(|u| u.email.clone())
+            } else {
+                None
+            }
+        } else {
+            None
+        }
+    };
+
+    // Resolve context from batch owner (created_by field)
+    let (context_name, context_type) = if let Some(owner_id) = batch.created_by.as_ref().and_then(|id| Uuid::parse_str(id).ok()) {
+        let user_map = Users::new(&mut read_conn)
+            .get_bulk(vec![owner_id])
+            .await
+            .map_err(|e| Error::Internal {
+                operation: format!("fetch owner user: {}", e),
+            })?;
+        match user_map.get(&owner_id) {
+            Some(u) if u.user_type == "organization" => {
+                let name = u.display_name.clone().unwrap_or_else(|| u.email.clone());
+                (Some(name), Some("organization".to_string()))
+            }
+            Some(_) => (Some("Personal".to_string()), Some("personal".to_string())),
+            None => (None, None),
+        }
+    } else {
+        (None, None)
+    };
+
+    let mut response = to_batch_response_with_email(batch, None);


Creator/context enrichment logic is duplicated between get_batch and list_batches (and the fallback behavior differs slightly for legacy batches). This duplication makes it easy for the enriched metadata fields to diverge between endpoints over time. Consider extracting a shared enrichment helper (ideally bulk-capable) that both endpoints call so they stay consistent.

🤖 I have created a release *beep* *boop* --- ## [8.19.0](v8.18.1...v8.19.0) (2026-03-12) ### Features * add trace_id to http_analytics ([#865](#865)) ([fb56e2f](fb56e2f)) * org-scoped batch/file filtering with metadata enrichment ([#839](#839)) ([000fcef](000fcef)) ### Bug Fixes * Split timeouts ([#833](#833)) ([7b5d78f](7b5d78f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

JoshC8C7 and others added 6 commits March 10, 2026 13:52

wire up split timeouts to config w/ fallback to kind of avoid breakin…

b33c77d

…g change

lint

d33ec18

header -> first_chunk

3c9069d

backend first pass

7b27d1a

Merge branch 'main' into hamishhall/cor-193-pr-4-org-scoped-batchfile…

e46d416

…-filtering

Copilot AI review requested due to automatic review settings March 10, 2026 18:56

Copilot started reviewing on behalf of hachall March 10, 2026 18:57 View session

hachall changed the title ~~Hamishhall/cor 193 pr 4 org scoped batchfile filtering~~ feat: org-scoped batch/file filtering with metadata enrichment Mar 10, 2026

Merge branch 'main' into hamishhall/cor-193-pr-4-org-scoped-batchfile…

d8b59fc

…-filtering

Copilot AI reviewed Mar 10, 2026

View reviewed changes

linting

6808e53

hachall requested a review from Copilot March 10, 2026 19:27

Copilot started reviewing on behalf of hachall March 10, 2026 19:28 View session

comments

059e139

Copilot AI reviewed Mar 10, 2026

View reviewed changes

comments

1673965

hachall requested a review from Copilot March 10, 2026 19:55

Copilot started reviewing on behalf of hachall March 10, 2026 19:56 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

sejori reviewed Mar 11, 2026

View reviewed changes

Comment thread dashboard/src/components/features/batches/Batches/Batches.tsx

Comment thread dashboard/src/api/control-layer/types.ts

Comment thread dwctl/src/api/handlers/batches.rs Outdated

Comment thread dwctl/src/api/handlers/batches.rs Outdated

hachall added 3 commits March 11, 2026 12:47

fusillade v0.9.0

d858a89

merge from main

360b935

linting

9ecdef2

graphite-app Bot reviewed Mar 11, 2026

View reviewed changes

Comment thread dashboard/src/components/features/batches/Batches/Batches.tsx

second pass

02bf27d

hachall requested a review from Copilot March 11, 2026 13:44

Copilot started reviewing on behalf of hachall March 11, 2026 13:46 View session

Merge branch 'main' into hamishhall/cor-193-pr-4-org-scoped-batchfile…

04d7be1

…-filtering

hachall requested a review from Copilot March 11, 2026 19:47

Copilot started reviewing on behalf of hachall March 11, 2026 19:47 View session