Ground truth
The screenshot is the exact analysis frame the ground truth is computed on. Labels are imported (AtariARI / OCAtari) then verified causally on the bit-exact machine.
The exact ground truth for this ROM and how every interpretability method scores
against it — one of the 42 scored games. Every number is read from
site_data.json (games.montezuma_revenge).
← back to the Ground Truth ROMs · Paper 2 audit · method catalogue
The screenshot is the exact analysis frame the ground truth is computed on. Labels are imported (AtariARI / OCAtari) then verified causally on the bit-exact machine.
Each XAI / mechanistic-interpretability method's faithfulness on this ROM, sorted by F (click a header to re-sort). The left border marks the tradition: green = causal / intervention, red = gradient / correlational, amber = other — the split the paper's headline turns on. F is all-regime; content-F and position-F are the two output regimes (n/a where this game has no record for that regime).
| Method | Family | F (all) | content-F | position-F | S | M |
|---|---|---|---|---|---|---|
| A8 · Whole-state recording | descriptive | 1.000 | n/a | n/a | 1.000 | 0.102 |
| Activation patching / causal mediation | causal / intervention | 1.000 | n/a | 1.000 | 1.000 | 0.938 |
| Causal scrubbing | causal / intervention | 1.000 | n/a | n/a | 1.000 | 0.875 |
| Interchange interventions / DAS | causal / intervention | 1.000 | n/a | n/a | 1.000 | 0.731 |
| KernelSHAP / Shapley | gradient / correlational | 0.943 | 1.000 | 0.887 | 0.938 | 0.400 |
| Occlusion | causal / intervention | 0.933 | 1.000 | 0.866 | 0.938 | 1.000 |
| A2 · Single-unit lesions | causal / intervention | 0.925 | n/a | n/a | 0.696 | 1.000 |
| LIME | gradient / correlational | 0.923 | 0.975 | 0.871 | 0.875 | 0.400 |
| ACDC — automatic circuit discovery | causal / intervention | 0.889 | n/a | n/a | 0.333 | 1.000 |
| RISE | gradient / correlational | 0.875 | 0.964 | 0.787 | 0.781 | 0.400 |
| Logit / tuned lens | causal / intervention | 0.842 | n/a | n/a | n/a | 0.119 |
| Path patching / IOI circuit | causal / intervention | 0.800 | n/a | n/a | n/a | 1.000 |
| Extremal / meaningful perturbation | causal / intervention | 0.701 | 0.935 | 0.468 | 0.906 | 1.000 |
| A7 · Dimensionality reduction (NMF/PCA) | dim_reduction | 0.600 | n/a | n/a | n/a | n/a |
| Attribution / edge patching | gradient / correlational | 0.567 | n/a | n/a | 0.957 | 1.000 |
| Guided Backprop | gradient / correlational | 0.500 | 1.000 | 0.000 | 0.938 | 1.000 |
| Integrated Gradients | gradient / correlational | 0.500 | 1.000 | 0.000 | 0.938 | 1.000 |
| Vanilla gradient (saliency) | gradient / correlational | 0.500 | 1.000 | 0.000 | 0.938 | 1.000 |
| SmoothGrad | gradient / correlational | 0.500 | 1.000 | 0.000 | 0.938 | 1.000 |
| A5 · Local field potentials | gradient / correlational | 0.402 | n/a | n/a | n/a | n/a |
| On-distribution counterfactual | causal / intervention | 0.362 | 0.724 | 0.000 | 0.188 | 1.000 |
| Grad×Input / DeepLIFT | gradient / correlational | 0.362 | 0.724 | 0.000 | 0.438 | 1.000 |
| Linear probing + control tasks | probing | 0.336 | n/a | n/a | n/a | 1.000 |
| A4 · Pairwise correlations | gradient / correlational | 0.280 | n/a | n/a | 0.322 | 0.867 |
| A6 · Granger causality | gradient / correlational | 0.210 | n/a | n/a | 0.962 | 1.000 |
| A1 · Connectomics / data-flow graph | causal / intervention | 0.125 | n/a | n/a | 0.976 | 1.000 |
| A3 · Tuning curves | gradient / correlational | 0.000 | n/a | n/a | -0.226 | 1.000 |
| NMF/PCA dictionaries | dim_reduction | 0.000 | n/a | n/a | 0.524 | 1.000 |
| Expected Gradients | gradient / correlational | 0.000 | 0.000 | 0.000 | 0.906 | 1.000 |
| Sparse autoencoders | dim_reduction | 0.000 | n/a | n/a | -0.224 | 1.000 |
Source: site_data.json
· generated by gen_site_data.py from the
committed §R records + leaderboard.json.