Private Eye — Paper 2 per-game results

Every method's faithfulness on Private Eye

Each XAI / mechanistic-interpretability method's faithfulness on this ROM, sorted by F (click a header to re-sort). The left border marks the tradition: green = causal / intervention, red = gradient / correlational, amber = other — the split the paper's headline turns on. F is all-regime; content-F and position-F are the two output regimes (n/a where this game has no record for that regime).

Method	Family	F (all)	content-F	position-F	S	M
A2 · Single-unit lesions	causal / intervention	1.000	n/a	n/a	0.611	1.000
A8 · Whole-state recording	descriptive	1.000	n/a	n/a	1.000	0.094
Activation patching / causal mediation	causal / intervention	1.000	n/a	1.000	1.000	1.000
Interchange interventions / DAS	causal / intervention	1.000	n/a	n/a	1.000	0.917
Causal scrubbing	causal / intervention	0.890	n/a	n/a	0.890	0.818
Logit / tuned lens	causal / intervention	0.877	n/a	n/a	n/a	0.275
Occlusion	causal / intervention	0.748	0.758	0.739	0.960	1.000
LIME	gradient / correlational	0.734	0.731	0.737	0.960	0.229
RISE	gradient / correlational	0.718	0.713	0.723	0.960	0.230
On-distribution counterfactual	causal / intervention	0.634	0.758	0.510	0.960	1.000
KernelSHAP / Shapley	gradient / correlational	0.620	0.503	0.737	0.960	0.229
A7 · Dimensionality reduction (NMF/PCA)	dim_reduction	0.600	n/a	n/a	n/a	n/a
Attribution / edge patching	gradient / correlational	0.497	n/a	n/a	0.954	1.000
Grad×Input / DeepLIFT	gradient / correlational	0.487	0.970	0.003	0.960	1.000
A5 · Local field potentials	gradient / correlational	0.458	n/a	n/a	n/a	n/a
Expected Gradients	gradient / correlational	0.380	0.758	0.003	0.960	1.000
Integrated Gradients	gradient / correlational	0.380	0.758	0.003	0.960	1.000
Vanilla gradient (saliency)	gradient / correlational	0.380	0.758	0.003	0.960	1.000
SmoothGrad	gradient / correlational	0.380	0.758	0.003	0.960	1.000
Guided Backprop	gradient / correlational	0.379	0.758	0.000	0.960	1.000
A1 · Connectomics / data-flow graph	causal / intervention	0.364	n/a	n/a	1.000	1.000
ACDC — automatic circuit discovery	causal / intervention	0.364	n/a	n/a	0.167	1.000
Extremal / meaningful perturbation	causal / intervention	0.347	0.507	0.188	0.960	1.000
NMF/PCA dictionaries	dim_reduction	0.291	n/a	n/a	0.999	1.000
Path patching / IOI circuit	causal / intervention	0.222	n/a	n/a	n/a	1.000
A3 · Tuning curves	gradient / correlational	0.166	n/a	n/a	0.292	1.000
Sparse autoencoders	dim_reduction	0.118	n/a	n/a	0.066	1.000
A6 · Granger causality	gradient / correlational	0.100	n/a	n/a	0.980	0.818
Linear probing + control tasks	probing	0.088	n/a	n/a	n/a	1.000
A4 · Pairwise correlations	gradient / correlational	0.011	n/a	n/a	0.120	0.321

Source: site_data.json · generated by gen_site_data.py from the committed §R records + leaderboard.json.

Ground truth

Every method's faithfulness on Private Eye