The distinction this paper is about
A benchmark measures whether a stage is right on the data it was tested on. It does not measure whether, when that stage is wrong on new data, anything in its output reveals it. Those are different properties, and the second decides whether a pipeline can be trusted without ground truth, which is the situation genome mining is always in.
A method that returns "unknown" for a protein it cannot handle is behaving safely: the failure is legible and a downstream step can route around it. A method that returns a specific, confidently scored, wrong function is behaving unsafely, because its output is formally identical to a correct one.
What we did
We audited a working natural-product genome-mining system of our own. We did not design experiments to elicit these failures; all seven surfaced from a single discipline of re-deriving every number in a manuscript from the files that produced it, rather than from the summary that reported it.
What we found
Seven stages where wrong output was indistinguishable from correct. Five were detectable only by recomputation. Three patterns recur:
- Metrics measure something adjacent to what they claim. An annotation-quality rule counting "hypothetical protein" scored one genome 0.0% where 54.1% of its products were empty: the genome is annotated in a different dialect, and the keyword rule measured which words a submitter chose for their unknowns.
- Evaluations cannot save their own confound. A classifier scoring one protein 0.988 then 0.000 at unchanged ROC-AUC 0.995 reproduced neither label value once rebuilt with documented classes and seeds. The same confound cut a published classification accuracy from 70.4% to 49.5% on unseen genera.
- Provenance records describe the invocation, not the experiment. A seven-method benchmark spanning BLAST/Pfam, Foldseek, CLEAN, ESM-2, ProtRek and DeepFRI graded all seven wrong against a ground truth taken from an unverified code comment. Four were in fact correct. The protein had been misidentified, and correcting it inverts the reading.
A combined recovery figure reported as 15-20% was 14.5% over the 2,455 proteins both tools actually scored, and the intersection the figure implied had never been measured: 11 proteins, a Jaccard index of 0.03.
What this does and does not show
Three claims from our own prior work are withdrawn in full here, including a reported instance of fold-convergent annotation-transfer failure that we no longer have. The paper reports failures in a snapshot of actively maintained tools, so version numbers are not the subject and no claim is made that archaeal biology causes any of them. We release the checks so the audit can be repeated elsewhere.