Speed up /vulnerabilities and filtered /software/versions queries#45564
Speed up /vulnerabilities and filtered /software/versions queries#45564dantecatalfamo wants to merge 12 commits into
Conversation
…5415) Address 100% DB CPU spikes observed when paginating these endpoints with filters. Two root causes: cve_meta and vulnerability_host_counts lack indexes for the actual filter shapes, and ListVulnerabilities ran correlated scalar subqueries on every row of vulnerability_host_counts that matched the scope (1.1M rows examined per call). Add three online indexes: - cve_meta(cisa_known_exploit, cve) for the exploit filter - cve_meta(cvss_score, cve) for the CVSS range filter - vulnerability_host_counts(global_stats, team_id, host_count, cve) for the scope filter on the vulnerabilities endpoint Refactor ListVulnerabilities into a two-stage query: an inner query filters, sorts, and paginates vulnerability_host_counts (with an optional LEFT JOIN to cve_meta for sort/filter columns), and an outer query enriches only the paginated page with the created_at and source scalar subqueries. Net effect: the heavy subqueries run perPage times (~20-100) instead of millions. Sort-by-created_at falls back to the single-statement form since the sort key is itself the subquery result. CountVulnerabilities drops the redundant DISTINCT (vhc.cve is already unique within a (global_stats, team_id) scope via the existing UNIQUE KEY) and only joins cve_meta when KnownExploit filtering is requested.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #45564 +/- ##
==========================================
- Coverage 66.74% 66.73% -0.02%
==========================================
Files 2744 2745 +1
Lines 219316 219425 +109
Branches 10794 10794
==========================================
+ Hits 146393 146427 +34
- Misses 59698 59769 +71
- Partials 13225 13229 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
There was a problem hiding this comment.
Pull request overview
This PR improves performance for vulnerability and filtered software version queries by adding supporting MySQL indexes and reducing expensive per-row vulnerability metadata subqueries during pagination.
Changes:
- Refactors
ListVulnerabilitiesinto a paginated inner query plus outer metadata enrichment, with a legacy path forcreated_atsorting. - Optimizes
CountVulnerabilitiesand adds performance indexes for CVE metadata and vulnerability host counts. - Adds a migration, schema updates, migration test, and changelog entry.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
server/datastore/mysql/vulnerabilities.go |
Refactors vulnerability listing SQL and count query behavior. |
server/datastore/mysql/schema.sql |
Adds new index definitions and records the new migration. |
server/datastore/mysql/migrations/tables/20260515000600_AddVulnPerfIndexes.go |
Adds the online index migration. |
server/datastore/mysql/migrations/tables/20260515000600_AddVulnPerfIndexes_test.go |
Verifies the migration creates the expected indexes. |
changes/45415-vulnerability-and-versions-speed-imporvement |
Adds the user-visible changelog entry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR updates the changelog, adds composite indexes to cve_meta and vulnerability_host_counts with a non-blocking migration and test, and refactors ListVulnerabilities into a two-stage SQL plan (inner paginated select from vulnerability_host_counts with optional cve_meta join; outer enrichment adds created_at/source and EE cve_meta with deterministic tie-break). CountVulnerabilities is changed to COUNT(*) over vulnerability_host_counts with conditional cve_meta joins when KnownExploit is requested. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@server/datastore/mysql/vulnerabilities.go`:
- Around line 381-387: The outer ORDER BY only reapplies the primary sort key
and can reshuffle ties; modify the outer ordering code (the block that checks
vulnerabilitiesAllowedOrderKeys and calls outer.WriteString) to append a
deterministic tie-breaker (the cve column) to the ORDER BY clause — e.g. when
orderCol != "cve" add ", cve ASC" (preserving the chosen direction for the
primary key using opt.ListOptions.OrderDirection) so the outer ordering matches
the inner cursor ordering and prevents duplicate/missing rows across pages.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 44813082-5121-412c-82fb-c0e6c9c495c0
📒 Files selected for processing (5)
changes/45415-vulnerability-and-versions-speed-imporvementserver/datastore/mysql/migrations/tables/20260515000600_AddVulnPerfIndexes.goserver/datastore/mysql/migrations/tables/20260515000600_AddVulnPerfIndexes_test.goserver/datastore/mysql/schema.sqlserver/datastore/mysql/vulnerabilities.go
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Converting to draft until we have time to load test it next week |
Address 100% DB CPU spikes observed when paginating these endpoints with filters. Two root causes: cve_meta and vulnerability_host_counts lack indexes for the actual filter shapes, and ListVulnerabilities ran correlated scalar subqueries on every row of vulnerability_host_counts that matched the scope (1.1M rows examined per call).
Add three online indexes:
Refactor ListVulnerabilities into a two-stage query: an inner query filters, sorts, and paginates vulnerability_host_counts (with an optional LEFT JOIN to cve_meta for sort/filter columns), and an outer query enriches only the paginated page with the created_at and source scalar subqueries. Net effect: the heavy subqueries run perPage times (~20-100) instead of millions. Sort-by-created_at falls back to the single-statement form since the sort key is itself the subquery result.
CountVulnerabilities drops the redundant DISTINCT (vhc.cve is already unique within a (global_stats, team_id) scope via the existing UNIQUE KEY) and only joins cve_meta when KnownExploit filtering is requested.
Related issue: Resolves #45415
Checklist for submitter
If some of the following don't apply, delete the relevant line.
changes/,orbit/changes/oree/fleetd-chrome/changes.See Changes files for more information.
SELECT *is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.Testing
Database migrations
COLLATE utf8mb4_unicode_ci).Summary by CodeRabbit