Skip to content

Add CustomEvaluator implementation#20

Draft
rlvandenbroek wants to merge 3 commits intomainfrom
feature/custom-evaluator
Draft

Add CustomEvaluator implementation#20
rlvandenbroek wants to merge 3 commits intomainfrom
feature/custom-evaluator

Conversation

@rlvandenbroek
Copy link
Collaborator

Add CustomEvaluator and Basic Custom Evaluator Tutorial

Summary

This PR introduces a flexible CustomEvaluator interface and corresponding CustomEvaluatorConfig, enabling users to supply arbitrary Python callables as scoring functions within the Thompson Sampling (TS) framework.

In addition, this PR adds a minimal custom_evaluator_tutorial Merimo notebook demonstrating:

  1. Standalone usage of CustomEvaluator
  2. Usage of CustomEvaluatorConfig
  3. Integration into a full Thrombin Thompson Sampling workflow

The tutorial uses a simple example scoring function (heavy_atom_count) to provide a transparent, easily verifiable demonstration.


TODO before merging

  • Integrate CustomEvaluator into test suite
  • Integrate into production Thompson Sampling Merimo notebook (thompson_sampling_tutorial.py)

Motivation

Previously, evaluators were primarily lookup-based or hard-coded evaluators. This limited flexibility for:

  • On-the-fly property evaluation
  • Rapid prototyping of scoring functions
  • Algorithmic or model-based scoring

CustomEvaluator enables direct injection of callable scoring logic while preserving compatibility with the existing ThompsonSamplingConfig pipeline.


Implementation Details

1. CustomEvaluator

  • Accepts a scoring_function: Callable
  • Evaluates RDKit mol objects directly

2. CustomEvaluatorConfig

  • Mirrors existing config patterns (e.g., LookupEvaluatorConfig)
  • Stores the scoring function
  • Cleanly integrates with ThompsonSamplingConfig

3. Tutorial Notebook

notebooks/custom_evaluator_tutorial.py

Demonstrates:

  • Small 5-SMILES sanity check
  • Config-based evaluator instantiation
  • Minimal TS run using full reagent dataset

Backward Compatibility

  • No breaking changes.
  • Existing lookup-based evaluators remain unaffected.
  • TS API remains unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant