PHANTOM: Benchmark Validity and Causal Limits in Coordinated Listening

Abstract

High scores in coordination detection can reward benchmark artifacts while concealing unreliable operating thresholds. PHANTOM rejected attractive listening benchmarks when origin, catalog and duplicate shortcuts could explain performance; even perfect raw ranking failed threshold transfer. We develop structure-preserving controlled interventions and a relational detector normalized against a local conditional permutation reference. The population detector targets a declared recurring coordination pattern that the intervention deliberately instantiates. Controlled transfer across separate ListenBrainz populations culminates in a prospectively committed, pre-access holdout evaluation: 378 of 378 treatments and none of 378 matched controls alert across 42 source groups, with AUC 1.000. These labels denote controlled interventions on observed histories, and repeated worlds share background. The holdout tests a new population under the same intervention family, rather than universal coordination detection. Historical reconstruction fully recovers member and target sets for 154 of 168 parents, but common-stimulus and shifted-overlap tests defeat broader operating claims. Applying observational-equivalence reasoning shows why event-only evidence cannot settle intent. PHANTOM therefore supports investigation by presenting structural evidence, alternative explanations and gaps in provenance. Its contribution is the empirical separation of benchmark validity, conditional transfer and causal interpretation.