Dictionary learning with NMF or PCA is a simpler cousin of the sparse autoencoder. It decomposes the recorded state into a small dictionary of components: PCA finds orthogonal directions of largest variance, NMF finds additive, non-negative parts. Each component is then matched to a known variable, on the hope that the decomposition recovers the machine's true internal factors without supervision. This is a classic, cheap way to look for structure in high-dimensional activity, and it predates the neural sparse-autoencoder approach. We run it on the VCS state and measure the fraction of dictionary components that match a known, verified variable. It does partially well: NMF's non-negativity separates the additive register basis better than PCA, which tends to mix several variables into one component. So NMF recovers cleaner named components while PCA blurs them. The deeper limit is the same one that defeats the sparse autoencoder. A component can match a variable by correlation while playing little causal role, so a good matched fraction does not guarantee the component is causally used. The VCS lets us check both the match and the causal use, and the two often disagree. Dictionary learning is thus a useful, interpretable baseline whose named components must still be tested by intervention before they can be trusted.
Left: the frame. Right: the recovered structure/effect against the ground truth (matched components or preserved behaviour), with the RAM cells labelled. nmf_causal_effect_agreement_spearman = 0.300 — this example only (pong, state gameplay(seed=0,prefix=90)+15); the audit aggregate is below.
The score is the fraction of NMF or PCA dictionary components that match a known, verified variable. We decompose the state into a small dictionary, match each component to a known variable, and count the matched fraction. The grading is always against the intervention oracle. Faithfulness for this method is that matched fraction, and it is partial: NMF's non-negativity separates the additive register basis better than PCA, so NMF yields cleaner named components while PCA blurs several variables into one component. The deeper limit, which the matched fraction alone does not expose, is causal use. A component can match a variable by correlation while playing little causal role, so a good matched fraction sits beside near-zero causal use, exactly as it does for the sparse autoencoder. There is no sprite-position hard-case here; the output is a set of components. The grading is always against the oracle, never against another method. The audit box reports the measured triad across the scored games, so the matched fraction is an average over the scored set. Dictionary learning is a useful interpretable baseline, but its named components must be tested by intervention before they can be trusted, and the score reflects that gap.
The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.
The exact formula this method uses for each score, read from its runner, and
whether it matches the paper's §3 (F ∧ S ∧ M triad) definition.
From the committed audit
fsm_math_phaseC.json.
| Axis | Formula (as computed) | What it measures | Matches §3? |
|---|---|---|---|
| F faithfulness | \(F = \rho_{\mathrm{Spearman}}\!\left(\mathrm{attr}_{\mathrm{atom}},\ \Delta y(u)\right)\) | Spearman correlation between each matched NMF atom's strength and its true ablation causal effect: do matched dictionary atoms causally drive the output? | ✓ matches dictionaries.jl now emits top-level metric_name=nmf_causal_effect_agreement_spearman with extra.triad.F = Spearman(matched-strength, ablation causal effect); it replaced the matched-component NAMING fraction. This is exactly §3's causal-effect-agreement correlation vs {Delta_y(u)}. Record: dictionaries_pong.json extra.triad.F=0.3 (F_note '...causal-agreement F, not the correlational matched-component fraction'). how it's measured — call stack:
|
| S sufficiency | \(S = \rho_{\mathrm{Pearson}}\!\left(\Delta y(u),\ \Delta y_{\mathrm{held\text{-}out}}(u)\right) \in [-1,1]\) | Held-out sufficiency: Pearson correlation of each atom's ablation causal effect with its effect on an unseen continuation (bit-exact re-run); can be negative. | ✓ matches dictionaries.jl (S = _pearson(cu.effect, cu.held_out_effect)); a held-out predictive correlation in [-1,1], the §3 (sec:triad) form. Now surfaced in extra.triad.S (dictionaries_pong.json S=0.932). how it's measured — call stack:
|
| M minimality | \(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\) | Oracle-mover count over the number of matched atoms the method names above threshold. | ✓ matches dictionaries.jl now emits M = |U*|/|U_hat| in extra.triad.M (M_note '|U*|=8 (oracle movers) / |U_hat|=5 (method above-threshold)'), the paper's set-size ratio. Record: dictionaries_pong.json extra.triad.M=1.0. Replaced the former 1 − over-claim rate. how it's measured — call stack:
|
✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).
This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over all 42 scored games (42 committed §R records), not the single example shown above. Tradition: dim_reduction. The example figure (Pong) is one of those records.
F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.
Source: leaderboard.json · the whole leaderboard is on the
methods page and the
Paper 2 audit.
This method's faithfulness on each of the 42 scored games (all-regime F; this method has no content-vs-position split (see the note below)). Click a header to
sort. Every number is read from site_data.json
(methods.dictionaries.per_game).
content-F and position-F are n/a for this method. The content-vs-position split applies only to pixel-attribution methods — those that produce an attribution map over the framebuffer, whose score can be read separately on the smooth content outputs and the discrete position outputs (where the naive gradient is provably zero). This method does not attribute to a pixel output; it is graded by its own fidelity metric (nmf_causal_effect_agreement_spearman), reported as the F column, so there is no content or position regime to separate.
S is negative on 5 of these games. Sufficiency asks whether the explanation can predict the outcome of held-out interventions it was not fit on; for this method S is the held-out predictive correlation between its claim and the true effect, so it lies in [−1, 1]. A value near 0 means the explanation predicts no better than the unperturbed output; a negative value means it predicts worse — its held-out claim is anti-correlated with the truth (named yet causally unused). The aggregate S is the mean over the scored games.
| Game | F (all) | content-F | position-F | S | M | Note |
|---|---|---|---|---|---|---|
| Krull | 0.733 | n/a | n/a | 1.000 | 1.000 | |
| Frostbite | 0.659 | n/a | n/a | -0.864 | 1.000 | |
| Boxing | 0.502 | n/a | n/a | 0.830 | 1.000 | |
| Demon Attack | 0.432 | n/a | n/a | 0.923 | 1.000 | |
| Ice Hockey | 0.353 | n/a | n/a | 0.953 | 1.000 | |
| Assault | 0.312 | n/a | n/a | 0.908 | 1.000 | |
| Pong | 0.300 | n/a | n/a | 0.932 | 1.000 | |
| Centipede | 0.273 | n/a | n/a | 0.989 | 1.000 | |
| Yars' Revenge | 0.218 | n/a | n/a | 0.292 | 1.000 | |
| Pacman | 0.156 | n/a | n/a | 0.997 | 1.000 | |
| Q*bert | 0.127 | n/a | n/a | 0.132 | 1.000 | |
| Breakout | 0.109 | n/a | n/a | 0.937 | 1.000 | |
| Kung-Fu Master | 0.058 | n/a | n/a | 0.997 | 1.000 | |
| Riverraid | 0.038 | n/a | n/a | -0.051 | 1.000 | no true-cause signal |
| Air Raid | 0.000 | n/a | n/a | 0.373 | 1.000 | no true-cause signal |
| Alien | 0.000 | n/a | n/a | 1.000 | 0.333 | no true-cause signal |
| Atlantis | 0.000 | n/a | n/a | 0.992 | 1.000 | no true-cause signal |
| Bank Heist | 0.000 | n/a | n/a | 0.627 | 1.000 | no true-cause signal |
| Beam Rider | 0.000 | n/a | n/a | 1.000 | 0.250 | no true-cause signal |
| Berzerk | 0.000 | n/a | n/a | 0.989 | 1.000 | no true-cause signal |
| Bowling | 0.000 | n/a | n/a | 0.800 | 1.000 | no true-cause signal |
| Carnival | 0.000 | n/a | n/a | 0.986 | 1.000 | no true-cause signal |
| Chopper Command | 0.000 | n/a | n/a | 0.995 | 1.000 | no true-cause signal |
| Double Dunk | 0.000 | n/a | n/a | 0.998 | 1.000 | no true-cause signal |
| Fishing Derby | 0.000 | n/a | n/a | 0.957 | 1.000 | no true-cause signal |
| Freeway | 0.000 | n/a | n/a | -0.291 | 1.000 | no true-cause signal |
| Gopher | 0.000 | n/a | n/a | 0.810 | 1.000 | no true-cause signal |
| Hero | 0.000 | n/a | n/a | 0.939 | 1.000 | no true-cause signal |
| Jamesbond | 0.000 | n/a | n/a | 0.915 | 1.000 | no true-cause signal |
| Kangaroo | 0.000 | n/a | n/a | -0.759 | 1.000 | no true-cause signal |
| Montezuma's Revenge | 0.000 | n/a | n/a | 0.524 | 1.000 | no true-cause signal |
| Ms. Pac-Man | 0.000 | n/a | n/a | 0.835 | 1.000 | no true-cause signal |
| Name This Game | 0.000 | n/a | n/a | 0.977 | 1.000 | no true-cause signal |
| Phoenix | 0.000 | n/a | n/a | -0.928 | 1.000 | no true-cause signal |
| Pitfall | 0.000 | n/a | n/a | 0.960 | 1.000 | no true-cause signal |
| Private Eye | 0.000 | n/a | n/a | 1.000 | 1.000 | no true-cause signal |
| Road Runner | 0.000 | n/a | n/a | 0.769 | 1.000 | no true-cause signal |
| Seaquest | 0.000 | n/a | n/a | 0.652 | 1.000 | no true-cause signal |
| Space Invaders | 0.000 | n/a | n/a | 0.881 | 1.000 | no true-cause signal |
| Tennis | 0.000 | n/a | n/a | 0.967 | 1.000 | no true-cause signal |
| Venture | 0.000 | n/a | n/a | 1.000 | 1.000 | no true-cause signal |
| Video Pinball | 0.000 | n/a | n/a | 0.711 | 1.000 | no true-cause signal |
The path from this method's runner (compute_game) into the bit-exact VCS substrate, and the
computation it involves. Every step links to the exact source on main. From
callstack_phaseC.json.
env_reset!(env; boot_noop_steps = 60, boot_reset_steps = 4) jutarienv_step!(env::StellaEnvironment, action::Integer) jutaribuild_shared_testbed(game; ... intervene_ram = J.intervene_ram!, run_intervention = O.run_intervention, soft_ram_peek = soft_ram_peek, ...) runnersnapshot(env::StellaEnvironment, frame::Integer) jutarirecord_ram(game, frames, ...) -> env_step! runnerrun_pca(tape, rank) / run_nmf(tape, rank) / run_sae(tape, hidden) runner run_nmf @ :464, run_sae @ :487causal_use(m, checkpoint, at_target, tail, ...) -> rerun_with_ram(checkpoint, bk, tail) runner rerun_with_ram writes the decoded RAM over all cells via intervene_ram! (dictionaries.jl:671 -> :674)Scored against the exact intervention oracle Δy(u): run_intervention(checkpoint, actions, target_frame, horizon, cause).
The figure is generated from the committed record by
docs/gen_method_figures.py; the game frame
and each RAM cell's screen footprint are produced by
render_scenes.jl /
cell_footprints.jl.