If two judges with different settings share one identity, the cache cannot tell them apart. Results computed for one get served for the other, and nothing fails.
The defect
In Microsoft PyRIT, two DecodingScorer instances with different matcher settings could return opposite verdicts on the same input while sharing both their hash and their eval_hash. Cached evaluation metrics could then be reused across scorers that do not behave the same way.
The same defect had already been described and fixed for SubStringScorer. Both classes hold a text matcher at the same point of the same method. Only the sibling passed the matcher's parameters into its identifier.
The fix
get_matcher_params = getattr(self._text_matcher, "get_identifier_params", None)
"text_matcher_params": get_matcher_params() if callable(get_matcher_params) else None,Two lines, copied from the sibling on purpose. Case sensitivity, whitespace handling, threshold and n-gram size now reach the identity. Custom matchers that only implement is_match keep working through the callable guard.
The evidence
- Three parameterised cases, case sensitivity, threshold and n-gram size, each asserting that opposite verdicts no longer share an identity
- Before the fix: 3 failed, 11 passed. After: 14 passed
- The wider scorer suite kept the same pre-existing failures before and after, with exactly five more passes, the five added cases
Scorer categories are absent from both classes' identifiers. Widening one class and not the other would have broken the symmetry the fix relied on, so I kept the change symmetric and offered to cover categories for both in a follow-up.
What I take from it
When a bug is fixed in one class, read its siblings. The second instance of a defect is usually sitting one file away, waiting for nobody to look.
Ayoub Hammoudi, Software engineer, full-stack and AI platforms. github.com/Ayoubhm07
