`GenericChangeThreshold` relative validation inconsistency

[stefandominicus-takealot]

The implementation of relative threshold validation doesn't align with the documented/expected behaviour.

From the docs:

Aboslute change is calculated as the value diference between the metrics of the candidate and baseline model, namely, v_c - v_b where v_c denotes the candidate metric value and v_b denotes the baseline value. Relative value is the relative difference between the metric of the candidate and the baseline, namely, v_c/v_b.

However, the implementation instead calculates (v_c - v_b) / v_b, NOT v_c/v_b as the docs very clearly state.

model-analysis/tensorflow_model_analysis/evaluators/metrics_validator.py

Line 88 in 422dbd3

ratio = diff / metric_baseline

The easiest fix would be to update the docs to correctly describe the implementation, but I'm not sure what the desired behaviour actually is.

[sonali-kumari1]

Hi @stefandominicus-takealot, Thanks for pointing this out!

@mdreves, There’s an inconsistency between the docs and the implementation for relative threshold validation. Could you please clarify which behavior is intended? Thanks!