Skip to content

Add retrieval_bench: pipeline evaluation for ViDoRe V3 and BRIGHT leaderboards#1498

Open
oliverholworthy wants to merge 24 commits intoNVIDIA:mainfrom
oliverholworthy:oholworthy/agentic-pipeline-evaluation
Open

Add retrieval_bench: pipeline evaluation for ViDoRe V3 and BRIGHT leaderboards#1498
oliverholworthy wants to merge 24 commits intoNVIDIA:mainfrom
oliverholworthy:oholworthy/agentic-pipeline-evaluation

Conversation

@oliverholworthy
Copy link

@oliverholworthy oliverholworthy commented Mar 6, 2026

Add retrieval_bench, a self-contained benchmarking package for evaluating
dense and agentic retrieval pipelines against the ViDoRe V3 and BRIGHT
leaderboards.

What's included:

  • Pluggable dense retrieval pipeline supporting four backends: llama-nv-embed-reasoning-3b, llama-nemoretriever-colembed-3b-v1, llama-nemotron-embed-vl-1b-v2, nemotron-colembed-vl-8b-v2
  • Agentic retrieval pipeline that augments dense retrieval with an LLM agent for iterative query refinement
  • Per-query tracing and result caching
  • CLI via retrieval-bench evaluate dense-retrieval and retrieval-bench evaluate agentic-retrieval
  • Utility commands for listing datasets/backends and comparing results

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.
  • If adjusting docker-compose.yaml environment variables have you ensured those are mimicked in the Helm values.yaml file. N/A

@copy-pr-bot
Copy link

copy-pr-bot bot commented Mar 6, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

oliverholworthy and others added 23 commits March 6, 2026 13:38
@oliverholworthy @radekosmulski
Add retrieval_bench package for ViDoRe V3 and BRIGHT leaderboard eval…
bdfa32f 
…uation

Incorporates agentic and dense retrieval pipeline benchmarking toolkit
as a standalone package. Includes LLM agent loop with tool-calling,
pluggable retriever backends, and evaluation framework with pytrec_eval
metrics.

Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
Co-authored-by: Radek Osmulski <2444926+radekosmulski@users.noreply.github.com>
@oliverholworthy
Add SPDX headers to python files
59bdba5 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Update version of retrieval-bench
d463301 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Reformat and remove unused imports
97cf8ee 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Split long lines
b0bcf84 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Remove trailing whitespace
5d5db61 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Ignore long lines in retrieval_bench
4c3353d 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Remove unused torch functional imports
1dcbb67 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Reformat line in tool_helpers.py
7220f81 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Extrat common singleton methods to shared module
38dfe54 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Replace assertions with runtime errors in singletons
fde1612 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Move infer_bright_task_key to shared module
be32ed5 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Raise exceptions in agentic code instead of assertions
1448625 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Rename logging class names keeping attribution
fd2a636 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Add logger in case cache fails
c521ac1 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Use shared _hash_corpus_ids10 in bm25s_retriever
bbc9d5a 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Update call to _try_preload_corpus_to_gpu in colembed vl singleton
d8a0d7e 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
log debug message if trace fails to load
c4730eb 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Add note on trust_remote_code
36728d3 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Move load_dotenv into pipeline class
037bde6 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Move bench specific .gitignore entries to nested .gitignore
fddaebd 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Add SPDX header to README.md
f59a4ce 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy
Rename top-level retrieval-bench directory
c0efb90 
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
@oliverholworthy oliverholworthy force-pushed the oholworthy/agentic-pipeline-evaluation branch from 5562153 to c0efb90 Compare March 6, 2026 13:39
@oliverholworthy oliverholworthy self-assigned this Mar 9, 2026
@oliverholworthy oliverholworthy marked this pull request as ready for review March 9, 2026 16:30
@oliverholworthy oliverholworthy requested a review from a team as a code owner March 9, 2026 16:30
@oliverholworthy oliverholworthy requested a review from nkmcalli March 9, 2026 16:30
@oliverholworthy oliverholworthy requested a review from a team March 9, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jperez999 jperez999 jperez999 approved these changes

@nkmcalli nkmcalli Awaiting requested review from nkmcalli nkmcalli is a code owner automatically assigned from NVIDIA/nv-ingest-maintainers

Assignees

@oliverholworthy oliverholworthy

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@oliverholworthy @jperez999 @jdye64