Left: a game frame for context. Right: why these methods do not apply to the VCS. n_methods_not_applicable = 6.000 — this example only (pong, state None); the audit aggregate is below.
Selvaraju 2017; Abnar & Zuidema 2020; Bastani 2018
This entry is an honesty audit, not a method that runs. Three well-known techniques, Grad-CAM, attention rollout, and VIPER, cannot be applied to the VCS at all, and we record that fact rather than force a number. Grad-CAM weights the final convolutional feature maps of a network by the gradient of the class score, so it needs a convolutional layer that the VCS does not have. Attention rollout multiplies the attention matrices across a transformer's layers, so it needs attention weights that the VCS does not have. VIPER distils a learned policy into a decision tree, so it needs a trained policy, and the VCS runs a fixed ROM, not a learned agent. The VCS is a 6502 CPU with a TIA video chip and a RIOT, executing hand-written machine code. Across all 44 scored games the count of each required substrate, convolutional layer, attention matrix, learned policy, is exactly zero. Re-targeting Grad-CAM's pooling onto the raw register state would simply be the content-path gradient under a different name, so it would add nothing. We therefore mark these methods as 'does not apply', a structural absence that is the same in every game. Recording this openly is part of the audit: it shows that some popular tools have no purchase on a real artifact that lacks the structure they assume.
Left: a game frame for context. Right: why these methods do not apply to the VCS. n_methods_not_applicable = 6.000 — this example only (pong, state None); the audit aggregate is below.
New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.
Every XAI method is a recipe, and every recipe has ingredients. Saliency needs a differentiable function. Occlusion needs a way to blank out an input. Some popular methods need much more specific ingredients: a convolutional layer whose feature maps can be weighted, attention matrices that can be multiplied together, or a trained agent whose behaviour can be copied. If the system you study lacks the ingredient, the recipe cannot even start. That is what applicability means. A method applies to a system only if every object its formula refers to actually exists there, and every assumption behind its interpretation holds.
The Atari 2600 VCS is a hand-written program on a 1977 games console. It has a CPU, a video chip and a timer chip. It has no neurons, no layers, no attention and no learned agent. So some famous methods simply have nothing to bind to. This page is an audit of those methods. We do not invent a stand-in and report a score for it. We record “does not apply”, name the missing ingredient, and explain why every stand-in we could build would be a different method under a borrowed name. An empty cell in a results table is more honest than a number that measures nothing.
Here are the three flagship methods of the audit, and exactly what each one needs on a network.
Grad-CAM (Selvaraju et al. 2017). Take the last convolutional layer: \(K\) feature maps \(A^k\), each a small grid of activations. Average the gradient of the class score \(y^c\) over each map to get one weight per channel, then form a ReLU-ed weighted sum:
\[ \alpha_k^c = \frac{1}{Z}\sum_{i,j}\frac{\partial y^c}{\partial A^k_{ij}}, \qquad L^c = \mathrm{ReLU}\Big(\sum_k \alpha_k^c A^k\Big) . \]Ingredients: learned channels \(k\), a spatial grid \((i,j)\) aligned with the image, and a class score. CAM, Grad-CAM++ and feature visualisation / TCAV (concept vectors in a learned activation space) need the same kind of learned, spatially arranged channels.
Attention rollout (Abnar & Zuidema 2020). A transformer layer \(l\) has an attention matrix \(A^{(l)}\): how much each token reads from each other token. Mix in the residual connection and multiply through the layers:
\[ \tilde A^{(l)} = \tfrac12 A^{(l)} + \tfrac12 I, \qquad R = \tilde A^{(L)}\cdots\tilde A^{(1)} . \]Ingredients: tokens and softmax attention weights in every layer.
VIPER (Bastani et al. 2018). Take a trained reinforcement-learning policy \(\pi(a\mid s)\) with its Q-function (how good each action is in each state). Train a decision tree to imitate it, weighting states by how much a wrong action would cost. Ingredients: a learned policy and its Q-values.
Toy example: what a forced re-target produces. On a tiny CNN with two \(2\times2\) maps \(A^1=\begin{pmatrix}1&0\\0&0\end{pmatrix}\), \(A^2=\begin{pmatrix}0&2\\0&2\end{pmatrix}\) and channel weights \(\alpha_1=0.5\), \(\alpha_2=-0.25\), Grad-CAM gives \(L=\mathrm{ReLU}\begin{pmatrix}0.5&-0.5\\0&-0.5\end{pmatrix}=\begin{pmatrix}0.5&0\\0&0\end{pmatrix}\): one hot cell. Now try it on RAM. Suppose we declare the 128 bytes one “map” with one channel. Then \(\alpha\) is the mean gradient, and \(L_i=\mathrm{ReLU}(\alpha\,\mathrm{ram}_i)\) is just a rescaled copy of the RAM. It says nothing about which byte matters. Suppose instead each byte is its own \(1\times1\) channel. Then \(\alpha_k\,\mathrm{ram}_k = \frac{\partial y}{\partial \mathrm{ram}_k}\,\mathrm{ram}_k\), which is Gradient×Input, a method we already score. Either way the “Grad-CAM” label adds nothing.
| Convolutional feature maps \(A^k\) (Grad-CAM, Grad-CAM++, CAM) | None. The VCS has no learned intermediate layers. The TIA (video chip) draws pixels directly from register values set by hand-written code, line by line. |
|---|---|
| Attention matrices \(A^{(l)}\) (attention rollout) | None. Which byte influences which is fixed by the program's instructions. That true data flow is what the intervention oracle and path patching measure; calling it “attention” would relabel ground truth as a method. |
| A learned policy \(\pi(a\mid s)\) (VIPER) | None. The study explains the console itself. Joystick inputs come from a seeded random stream, not from a trained agent. |
| Learned channels / concepts (feature visualisation, TCAV) | None. RAM bytes and registers have fixed, known meanings (the score byte is the score). Testing such meanings is the job of probing with control tasks. |
| A ground truth to score against | Present: the intervention oracle and many candidate causes per game. The applicable methods use them; the n/a methods produce no map to compare. |
The audit is a runner like every other, na_audit.jl, but it runs no method, because there
is nothing to run. Instead it writes a structured, checkable record.
NA_METHODS lists Grad-CAM, Grad-CAM++, attention rollout,
VIPER, CAM, and feature visualisation / TCAV. Each entry names the required ingredient, its kind (conv
feature maps, attention, learned policy, learned channels), sets present_in_vcs = false, and
gives two texts. One says why the method does not apply. The other says why a naive re-target onto VCS
state would be a different method.ground_game loads the game's candidate-cause file: the RAM
bytes the applicable methods attribute over. It records that count next to zero conv maps, zero
attention heads, zero learned policies and zero learned channels. That asymmetry is the finding:
there are real causes to explain, and none of the ingredients these methods need.APPLICABLE_CONTRAST names methods that do bind to real causes:
vanilla gradient, Integrated Gradients, occlusion, counterfactuals. They are scored against the same
oracle (run_intervention) elsewhere, and gradient methods flow through
soft_ram_peek. A re-targeted Grad-CAM would collapse onto exactly those gradients.selftest asserts that every entry is marked absent with a named
ingredient, that the canonical methods are covered, that every core game has candidate causes, and that
the record round-trips. The headline value is the count of methods that do not apply; the record carries
no F, S or M.Why not force a number? Any score needs a map to compare with the oracle. Building one would mean choosing a stand-in for the missing ingredient, and every stand-in we can build is either uninformative (the rescaled-input case above), a method already on the leaderboard (Gradient×Input), or the ground truth itself (the data-flow graph). Scoring it under the famous name would credit or blame that method for something it never did. The absence is also structural. It is the same in every game, because no cartridge contains a network. That is why the record is game-independent and why it stays off the leaderboard.
XAI the interpretability method substrate the bit-exact emulator it runs on (jutari, Julia).
na_audit.jl:67 — The audit schema: required ingredient, its kind, presence flag, why it fails, why no re-target.na_audit.jl:79 — The six audited methods, each with its missing ingredient.na_audit.jl:81 — Grad-CAM: needs last-layer conv feature maps; a RAM re-target is just the content-path gradient.na_audit.jl:115 — Attention rollout: needs per-layer attention matrices; the VCS data flow is ground truth, not attention.na_audit.jl:132 — VIPER: needs a learned policy and Q-function; the VCS runs a fixed program.na_audit.jl:216 — Per-game grounding: real candidate-cause count against zero required channels.na_audit.jl:184 — The methods that do apply and are scored against the oracle.na_audit.jl:490 — Consistency checks: every entry absent and explained, every core game grounded.oracle_intervene.jl:242 — The oracle the applicable methods are scored against; the n/a methods give it nothing to compare.SoftStep.jl:106 — The differentiable RAM read: where a re-targeted Grad-CAM would end up as a plain gradient.This page has no faithfulness figure, and that is the point. The audit box marks the row as excluded from the leaderboard and shows how many methods were audited. Read the per-method table as a list of preconditions: for each method, the ingredient it needs, the fact that the VCS has none of it, and the reason a substitute would not be the same method. The same verdict holds for every game. Set it beside the gradient and perturbation pages: those methods need only a function and a way to change its inputs, and both exist on the VCS.
julia --project=jutari tools/xai_study/phaseB_attribution/na_audit.jl
Writes na_audit_<game>.json/.npz for the six core games, na_audit_combined.json into tools/xai_study/phaseB_attribution/out/, and the write-up na_audit.md. Runs in seconds: no emulation is needed. --games labeled grounds every labelled game; --selftest checks without writing.
Read next: Vanilla gradient (saliency) · Grad×Input / DeepLIFT · Path patching / IOI circuit · Linear probing + control tasks
This entry is not scored on the faithfulness leaderboard, and that is deliberate. Grad-CAM, attention rollout, and VIPER need a convolutional layer, an attention matrix, or a learned policy that the VCS does not have. Across all 44 scored games the count of each required substrate is exactly zero, against many genuine candidate causes per game for the methods that do apply. A faithfulness number would be meaningless, because there is no map to compare to the oracle. So we record a measured structural fact, applies equals false, rather than force a score. We also note that re-targeting Grad-CAM's pooling onto the raw register state would only be the content-path gradient under another name, so it would add no new information. The grading standard elsewhere in the audit is always correlation, F1, or effect-agreement against the intervention oracle; here none of those is defined, because the method cannot run. The audit box below therefore marks this row as excluded from the leaderboard rather than showing an F, S, and M triad. Recording the absence openly is itself a finding: several popular tools have no purchase on a real artifact that lacks the neural structure they assume.
The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.
Recorded but excluded from the faithfulness leaderboard: these methods have no applicable causes on the VCS, so a faithfulness score would be meaningless. See the leaderboard and the Paper 2 audit.
Source: leaderboard.json.
The figure is generated from the committed record by
docs/gen_method_figures.py; the game frame
and each RAM cell's screen footprint are produced by
render_scenes.jl /
cell_footprints.jl.