Nature Machine Intelligence (in preparation)

Paper 2 — An Interpretability Ground-Truth Benchmark

Because the VCS is fully known and exactly intervenable, every explanation can be scored against ground truth. An oracle measures the true causal effect of each state variable; XAI and mechanistic-interpretability methods are then graded by how well they recover it. Headline finding: across all regimes, causal/intervention methods stay well above gradient and correlational methods — a robust faithfulness gap of 0.321 (causal/intervention 0.705 vs gradient/correlational 0.384). On the discrete sprite-position outputs the naive gradient is exactly zero; the emulator's bilinear sampler restores a non-zero position gradient, but its faithfulness stays low, so gradient/correlational methods still collapse there while causal/intervention methods do not. Bootstrapped over the 42 scored games, the position gap is now significant (mean 0.326, 95% CI [0.277, 0.375], excludes zero).

The testbed was redesigned to score every method on the same shared random-action gameplay states — see the experiment redesign note.

paper PDF · Method catalogue & execution stack → · Scoring XAI / mechanistic-interpretability methods against the exact causal truth of a known machine

Headline finding

Aggregated over the 42 scored games, causal/intervention methods stay well above gradient and correlational methods — a robust faithfulness gap across all regimes. On the discrete sprite-position outputs the naive gradient is exactly zero; the emulator's bilinear sampler restores a non-zero position gradient, but its faithfulness stays low, so gradient and correlational methods still collapse there. Bootstrapped over the 42 games, the position-regime gap is now significant — its 95% confidence interval excludes zero (it crossed zero at the original six games).

0.321all-regime faithfulness gap (42 scored games)
0.705causal / intervention mean (all regimes, n=11)
0.384gradient / correlational mean (all regimes, n=14)
31methods on the leaderboard
1773per-game records aggregated

Primary contrast from leaderboard.json, surfaced through site_data.json · headline_contrast.all_regimes (42 scored games).

Position-regime gap — now significant at n≈42 (bootstrap over games, 95% CI excludes zero):

0.326position-regime gap · bootstrap CI [0.277, 0.375] (excludes 0)
0.561causal / intervention mean (position, n=4)
0.235gradient / correlational mean (position, n=9)

headline_contrast.position_regime family means; significance from the game-level bootstrap in position_bootstrap.json (n=42 games, 20 000 resamples). The testbed was redesigned to score every method on shared random-action gameplay states — see the experiment redesign note.

Evidence ledger

Every claim → script → command → artifact → runtime → hardware → verifying gate.

E1 · Ground-truth oracle (intervention + gradient + cross-check) 3 records measured

Exact causal map |Δy(u)| via clamp/resample-and-rerun; content-path gradient ∂y/∂u via the SOFT-STE substrate; a cross-check confirming the two agree. bit_exact_rerun flag set on the records.

Script
tools/xai_study/ground_truth/oracle_intervene.jl
Command
julia --project=jutari tools/xai_study/ground_truth/oracle_intervene.jl --game pong
Artifact
tools/xai_study/ground_truth/out/
Runtime
minutes / game (cluster)
Hardware
LME cluster / local · seed 0
Verified by
oracle_xcheck.jl (intervention↔gradient correlation)
E2 · T3 game-concept labels (import / verify / discover) 17 records measured

Candidate RAM→concept maps from OCAtari/AtariARI, upgraded to causal by verify-by-intervention (does the byte actually move the framebuffer?).

Script
tools/xai_study/t3/verify_labels.jl
Command
julia --project=jutari tools/xai_study/t3/verify_labels.jl
Artifact
tools/xai_study/t3/out/
Runtime
minutes (cluster)
Hardware
LME cluster / local · seed 0
Verified by
intervention check per label
E3 · Phase A — neuroscience battery A1–A8 (Kording, quantified) 54 records measured

Connectomics, single-unit lesions, tuning curves, pairwise correlations, LFP spectra, Granger causality, NMF/PCA, whole-state — each scored against the true read/write graph over the 42 scored games.

Script
tools/xai_study/phaseA_kording/A1_connectomics.jl
Command
julia --project=jutari tools/xai_study/phaseA_kording/A2_lesions.jl --games core
Artifact
tools/xai_study/phaseA_kording/out/
Runtime
4 h walltime / array task
Hardware
LME cluster (CPU array) · seed 0
Verified by
scored vs oracle data-flow graph; A6+A7 spot-re-run bit-exact
E3 · A9 — Visual6502 transistor-level track deferred deferred

Optional circuit-level (transistor) battery; product-owner-gated after the pilot. Not run for this submission — listed for honesty.

Script
xai_paper/xai_2_interpretability/experiment_design.md
Command
—
Artifact
—
Runtime
—
Hardware
—
Verified by
experiment_design.md (PO-gated)
E4 · Phase B — attribution / XAI methods (14) 166 records measured

Vanilla gradient, Grad×Input, Guided Backprop, SmoothGrad, Integrated Gradients, Expected Gradients, Occlusion, extremal perturbation, RISE, LIME, KernelSHAP, counterfactual + the N/A audit — each scored by correlation and deletion/insertion AUC vs the oracle.

Script
tools/xai_study/phaseB_attribution/ig_baseline_sweep.jl
Command
julia --project=jutari tools/xai_study/phaseB_attribution/saliency.jl --games core
Artifact
tools/xai_study/phaseB_attribution/out/
Runtime
up to ~15 min / method / game
Hardware
LME cluster (CPU array) · seed 0
Verified by
scored vs E1 oracle

Every method, its reference and its implementation script are catalogued in the method & execution tour.

E5 · Phase C — mechanistic interpretability (10) 72 records measured

Activation patching / causal tracing, interchange/DAS, attribution patching, path patching, ACDC, sparse autoencoders, NMF/PCA dictionaries, causal scrubbing, linear probing + control tasks, logit/tuned lens.

Script
tools/xai_study/phaseC_mechanistic/activation_patching.jl
Command
julia --project=jutari tools/xai_study/phaseC_mechanistic/sae.jl --games core
Artifact
tools/xai_study/phaseC_mechanistic/out/
Runtime
4 h walltime / array task
Hardware
LME cluster (CPU array) · seed 0
Verified by
scored vs exact patch / true circuit

Every method, its reference and its implementation script are catalogued in the method & execution tour.

E6 · Cross-tradition leaderboard 31 methods, 1773 records measured

Faithfulness-vs-plausibility leaderboard aggregated from every committed per-game record over the 42 scored games. Headline contrast (all regimes): causal/intervention faithfulness 0.705 vs gradient/correlational 0.384 — a robust gap of 0.321. The position-regime gap is now significant: bootstrapped over the 42 games it is 0.326 (95% CI [0.277, 0.375], excludes zero; it crossed zero at six games).

Script
tools/xai_study/compare/leaderboard.py
Command
python3 tools/xai_study/compare/leaderboard.py
Artifact
tools/xai_study/compare/out/leaderboard.json
Runtime
seconds (pure read)
Hardware
any · embedded self-check
Verified by
self-check asserts oracle_copy F=1, uniform at floor
E6 · ROM-free benchmark package 14 records measured

A self-contained scoring harness (magnitude_proxy demonstration) that an auditor can run with no ROM, against the committed oracle records.

Script
tools/xai_study/compare/benchmark/run.py
Command
python3 tools/xai_study/compare/benchmark/run.py --method magnitude_proxy
Artifact
tools/xai_study/compare/benchmark/out/
Runtime
seconds
Hardware
any
Verified by
self-test (oracle_copy F=1; magnitude_proxy reproduces its committed pong/content faithfulness within tolerance)

Figures

Rasterised from the paper’s committed PDFs; click to enlarge.

Platform & oracle

Platform & oracle

The known machine and how the causal oracle is measured.

Faithfulness vs plausibility

Faithfulness vs plausibility

The leaderboard axes. Built from leaderboard.json + faithful_demo.json.

Phase-A battery

Phase-A battery

Kording-style neuroscience methods scored against ground truth.

Attribution vs mechanistic

Attribution vs mechanistic

Phase B vs Phase C method scores.

Representativeness map

Representativeness map

Where each method lands across regimes.

Failure taxonomy

Failure taxonomy

How and where methods fail.

Sampler: faithful, no semantics

Sampler: faithful, no semantics

The differentiable sampler is faithful yet semantically empty.