feat: introduce a smarter database for osv.dev zips #323
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
G-Rath
force-pushed
the
scalibr/use-extractors
branch
5 times, most recently
from
05588d9 to
8d4bea2
Compare
G-Rath
force-pushed
the
new-smart-db
branch
2 times, most recently
from
ef16782 to
a66d469
Compare
There was a problem hiding this comment.
Pull Request Overview
Introduces a new "smart" database for osv.dev zips that uses the modified_id.csv file for selective updates, combining the benefits of zip and directory-based databases for improved performance.
- Implements
SmartDBtype that uses CSV file to track modified advisories for incremental updates - Adds cache directory management with fallback from user cache directory to temp directory
- Integrates smart database option into CLI with
--be-smartflag
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/database/smart.go | New SmartDB implementation with CSV-based incremental updates |
| pkg/database/smart_test.go | Comprehensive test suite for SmartDB functionality |
| pkg/database/zip.go | Refactored cache directory handling and extracted setupCacheDirectory function |
| pkg/database/zip_test.go | Updated tests to use CacheDirectory config and added withSummary helper |
| pkg/database/config.go | Added CacheDirectory field and smart database type support |
| main.go | Added --be-smart CLI flag and SmartDB description logic |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment on lines
+233
to
+241
| body, err = io.ReadAll(resp.Body) | ||
|
|
||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| err = db.cacheFile(id+".json", body) | ||
|
|
||
| return err |
There was a problem hiding this comment.
The variable declaration and assignment can be simplified by combining lines 232 and 234-236 into a single statement: body, err := io.ReadAll(resp.Body) and then if err != nil { return err }.
Comment on lines
+231
to
+234
| var body []byte | ||
|
|
||
| body, err = io.ReadAll(resp.Body) | ||
|
|
There was a problem hiding this comment.
Unnecessary variable pre-declaration. Use short variable declaration body, err := io.ReadAll(resp.Body) instead of declaring body as var body []byte and then assigning to it.
Suggested change
| var body []byte | |
| body, err = io.ReadAll(resp.Body) | |
| body, err := io.ReadAll(resp.Body) |
G-Rath
force-pushed
the
new-smart-db
branch
from
16d19bf to
440001b
Compare
G-Rath
force-pushed
the
scalibr/use-extractors
branch
2 times, most recently
from
7ac62bd to
2858bd8
Compare
G-Rath
force-pushed
the
scalibr/use-extractors
branch
from
2858bd8 to
0b63266
Compare
G-Rath
force-pushed
the
new-smart-db
branch
2 times, most recently
from
82d7ffc to
54da9dd
Compare
G-Rath
force-pushed
the
scalibr/use-extractors
branch
6 times, most recently
from
f5babd9 to
d5a7cc0
Compare
G-Rath
force-pushed
the
scalibr/use-extractors
branch
from
d5a7cc0 to
1862eee
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a new "smart" database for zips provided by osv.dev that uses the
modified_id.csvfile stored in the ecosystems bucket to selectively update advisories, effectively combining the zip and directory based databases.If the database isn't available, then it will be initialized using the
all.zipfor the particular ecosystem, otherwise we will just download the advisories that have changed since the database was last modified.This should be at most as slow as the zip database, but hopefully most of the time be faster