Fifteen pull requests fixed one bug. Two were mine. So I wrote the test.

PyRIT gives every converter and scorer an identifier, a hash of the settings that change what the component does. Registry names come from it and cached results lean on it. When two components that behave differently share a hash, nothing crashes. Their results simply get attributed to the wrong configuration.

The same bug, fifteen times

When I opened the issue, I counted: 15 merged pull requests from 8 authors, since 6 October, had each fixed this same defect for one component. Two of them were mine. The base class already stated the rule in its docstring: "Only include params that affect the component's behavior/output." Nothing checked it, so each gap waited for someone to notice.

One test, the whole registry

The test walks ConverterRegistry, changes one constructor setting at a time, and requires the hash to change. A new converter or parameter is covered the day it lands. A setting must have a sample value or a listed reason to stay out of the identity, so nothing is skipped silently.

assert len(set(hashes)) == len(hashes), (
    f"{converter_name}.{parameter_name} changes the converter but not its identifier{picked}"
)

It found three gaps: PolicyPuppetryConverter identifying a custom template by its name only, TextJailbreakConverter giving every inline template the same identifier, and CodeChameleonConverter whose custom functions never reached the identifier. Each one is a strict xfail linked to its own issue, so fixing it turns the test red until the mark is removed.

Then review broke my test, twice

A test against false identities had false passes of its own. A maintainer changed one line of production code so that offsets 1 and 2 of CaesarConverter recorded the same value. The test stayed green, because it discarded the baseline hash after checking repeatability.

Then the same maintainer added a counter that only ran under a non-default setting. Each changed setting was built once, so the runtime noise looked like a correctly identified setting. Now every configuration is built twice, and the baseline joins the collision check whenever its value is known. Both mutations fail.

Testing the test

The final check did the same thing at scale: 83 identifier-only mutations. 70 fail and name the setting. 3 content fixes fail as XPASS. 6 stay green as they should, 2 change nothing observable, and 2 hit limits the pull request names out loud, along with what it does not catch.

What I take from it

Fifteen fixes for one rule means the rule has no test. I had fixed this bug twice before I counted. The merged change is 1,000 lines of tests and zero lines of production code. In the maintainer's words: "This should help us stop discovering the same identifier problem one component at a time."

Ayoub Hammoudi, Software engineer, full-stack and AI platforms. github.com/Ayoubhm07