Fix getDiskUsage and add getCollectionStats by delthas · Pull Request #2605 · scality/Arsenal

delthas · 2026-04-01T22:35:04Z

Summary

Fix getDiskUsage: Was reading the nonexistent fsFreeSize field from MongoDB's dbStats command, always returning 0 for available/free space. Now correctly computes free space as fsTotalSize - fsUsedSize.
Add getCollectionStats: New method on MongoClientInterface that exposes per-collection stats via the collStats command. Returns the raw stats object including indexSizes (per-index size breakdown) and totalIndexSize.

Context

The lifecycle conductor (BB-753) needs to check available disk space before creating v2 indexes on MongoDB. When the MongoDB PV is nearly full, attempting index creation can fail (ENOSPC) or further pressure the disk. The conductor uses getDiskUsage for filesystem free space and getCollectionStats to get the _id_ index size as an estimate for the cost of creating lifecycle indexes.

Design decisions

getDiskUsage keeps its existing return shape ({ available, free, total }) to avoid breaking the interface contract, just fixes the computation.
getCollectionStats passes stats through directly from MongoDB rather than reshaping — the caller extracts what it needs.
Uses db.command({ collStats }) instead of the deprecated collection.stats() which was removed from the MongoDB Node.js driver type definitions.
Both calls are metadata-only (~3ms and ~8ms respectively on a 1M-document collection), no collection scans or locks.

bert-e · 2026-04-01T22:35:08Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

claude · 2026-04-01T22:36:05Z

LGTM

Review by Claude Code

codecov · 2026-04-01T22:42:57Z

Codecov Report

❌ Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.36%. Comparing base (eefeb8b) to head (e7d02f5).
⚠️ Report is 1 commits behind head on development/8.3.

Files with missing lines	Patch %	Lines
...orage/metadata/mongoclient/MongoClientInterface.ts	73.33%	4 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##           development/8.3    #2605      +/-   ##
===================================================
- Coverage            73.37%   73.36%   -0.01%     
===================================================
  Files                  222      222              
  Lines                18160    18169       +9     
  Branches              3761     3784      +23     
===================================================
+ Hits                 13324    13329       +5     
- Misses                4831     4835       +4     
  Partials                 5        5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bert-e · 2026-04-02T07:58:05Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

bert-e · 2026-04-07T09:47:00Z

Jira issue not found

The Jira issue ARSN-566 was not found.

lib/storage/metadata/mongoclient/MongoClientInterface.ts

tests/unit/storage/metadata/mongoclient/MongoClientInterface.spec.js

francoisferrand · 2026-04-07T16:21:14Z

note: this is an "independent" fix, could be fixed to Arsenal 8.3 if it makes integration easier/simpler...

bert-e · 2026-04-08T10:25:44Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

claude · 2026-04-08T12:25:52Z

LGTM

Review by Claude Code

lib/storage/metadata/mongoclient/MongoClientInterface.ts

tests/unit/storage/metadata/mongoclient/MongoClientInterface.spec.js

SylvainSenechal

Next time try to re add in the PR summary the Issue: ARSN-566, so i can quickly click on it to read the jira ticket, it's also missing in your backbeat pr

delthas · 2026-04-08T15:52:18Z

Next time try to re add in the PR summary the Issue: ARSN-566, so i can quickly click on it to read the jira ticket, it's also missing in your backbeat pr

Indeed, thanks.

lib/storage/metadata/mongoclient/MongoClientInterface.ts

tests/unit/storage/metadata/mongoclient/MongoClientInterface.spec.js

claude · 2026-04-08T16:05:32Z

getCollectionStats uses the collStats command, which was deprecated in MongoDB 6.2 in favor of the $collStats aggregation stage
- Consider using aggregate([{ $collStats: { storageStats: {} } }]) for forward compatibility
getCollectionStats guard checks only !this.db while getDiskUsage checks !this.db || !this.client
- Add !this.client to the guard for consistency
Removed unit tests covered error paths (db not connected, command failure, missing stats fields) that the new functional tests do not cover
- Add unit tests for the new InternalError path when fsTotalSize/fsUsedSize are missing, and for getCollectionStats error cases

Review by Claude Code

lib/storage/metadata/mongoclient/MongoClientInterface.ts

tests/unit/storage/metadata/mongoclient/MongoClientInterface.spec.js

claude · 2026-04-08T16:05:55Z

getCollectionStats guard check only tests !this.db, while getDiskUsage tests !this.db || !this.client — should be consistent.
- Add || !this.client to the guard condition.
Removed unit tests for getDiskUsage error paths without replacement in the new functional test file.
- Add unit tests for the new error branches (InternalError on missing stats fields, db not connected) and for getCollectionStats error paths.

Review by Claude Code

bert-e · 2026-04-08T16:07:05Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

tests/unit/storage/metadata/mongoclient/MongoClientInterface.spec.js

lib/storage/metadata/mongoclient/MongoClientInterface.ts

claude · 2026-04-08T16:37:26Z

Error-path test coverage gap: the removed unit tests for getDiskUsage included "database not connected" and "command error" tests that are still valid for the updated logic and not covered by the new functional tests. The new getCollectionStats method also has no error-path tests.
- Retain/update the two error-path unit tests for getDiskUsage (not-connected + command failure)
- Add unit tests for getCollectionStats error paths (not-connected, command failure, non-existent collection)

Review by Claude Code

lib/storage/metadata/mongoclient/MongoClientInterface.ts

claude · 2026-04-08T16:46:05Z

LGTM — clean fix and solid tests.

- getDiskUsage error log could include the actual stats object for easier debugging (see inline comment)

Review by Claude Code

getDiskUsage was reading the nonexistent fsFreeSize field from MongoDB's dbStats, always returning 0. Fix it to compute free space as fsTotalSize - fsUsedSize, with validation that both fields are present. Add getCollectionStats method that exposes per-collection stats via the collStats command. This is needed by the lifecycle conductor to check disk space before creating indexes (BB-753). Drop mock-based unit tests in favor of functional tests against a real MongoMemoryReplSet, which would have caught the original fsFreeSize bug. Issue: ARSN-566

claude · 2026-04-08T17:29:46Z

LGTM

Review by Claude Code

delthas · 2026-04-08T17:36:37Z

/approve

bert-e · 2026-04-08T17:36:48Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/ARSN-566/disk-space-check-lifecycle into
development/8.3
w/8.4/improvement/ARSN-566/disk-space-check-lifecycle will be merged into development/8.4

The following branches will NOT be impacted:

development/6.4
development/7.10
development/7.4
development/7.70
development/8.1
development/8.2

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e · 2026-04-08T17:46:17Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/8.3
✔️ development/8.4

The following branches have NOT changed:

development/6.4
development/7.10
development/7.4
development/7.70
development/8.1
development/8.2

Please check the status of the associated issue ARSN-566.

Goodbye delthas.

The following options are set: approve

delthas mentioned this pull request Apr 1, 2026

Skip lifecycle index creation when disk space is insufficient scality/backbeat#2729

Merged

scality deleted a comment from bert-e Apr 2, 2026

delthas changed the title ~~ARSN-566: Fix getDiskUsage and add getCollectionStats~~ Fix getDiskUsage and add getCollectionStats Apr 2, 2026

delthas changed the base branch from development/8.3 to development/8.4 April 7, 2026 12:33

delthas requested review from a team, SylvainSenechal and benzekrimaha April 7, 2026 13:13

francoisferrand reviewed Apr 7, 2026

View reviewed changes

lib/storage/metadata/mongoclient/MongoClientInterface.ts Show resolved Hide resolved

francoisferrand reviewed Apr 7, 2026

View reviewed changes

tests/unit/storage/metadata/mongoclient/MongoClientInterface.spec.js Show resolved Hide resolved

francoisferrand approved these changes Apr 7, 2026

View reviewed changes

scality deleted a comment from bert-e Apr 8, 2026

SylvainSenechal reviewed Apr 8, 2026

View reviewed changes

lib/storage/metadata/mongoclient/MongoClientInterface.ts Outdated Show resolved Hide resolved

SylvainSenechal reviewed Apr 8, 2026

View reviewed changes

tests/unit/storage/metadata/mongoclient/MongoClientInterface.spec.js Show resolved Hide resolved

SylvainSenechal approved these changes Apr 8, 2026

View reviewed changes

delthas force-pushed the improvement/ARSN-566/disk-space-check-lifecycle branch 2 times, most recently from dbfb436 to 89e678b Compare April 8, 2026 16:03