A8 · Whole-state recording

Neuroscience battery · the record-everything baseline

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

The idea in plain words

Every method in this battery looks at part of the system: one unit, one pair, one pooled signal. The obvious objection is “then record everything”. Neuroscience has pursued that dream for decades. Large projects aim to record every neuron of a brain at once, and techniques such as calcium imaging already capture tens of thousands of cells. The hope behind it is that once nothing is hidden, understanding will follow.

A8 tests that hope on a machine where the dream is already true. The Atari's complete state is tiny, and we can copy all of it at every frame. Such a record is a perfect description: it misses nothing, and from it the machine can be rebuilt exactly. But is it an explanation? An explanation should also be small: it should point at the few things that matter for the output you asked about. A8 is the deliberate baseline that scores perfectly on completeness and at the floor on smallness. It makes the gap between having the data and understanding the system measurable.

How it works on a neural network

Neuroscience origin. Whole-brain or whole-population recording: the Brain Activity Map / BRAIN Initiative vision (Alivisatos et al. 2012) of measuring every spike of every neuron. Jonas & Kording had the complete state of their simulated chip at every step, with every transistor known, and showed that standard analyses of it still failed to reveal how the chip ran its programs.

Deep-learning analogue. Dumping every activation: caching all hidden states of every layer for an input, the way interpretability libraries “run with cache”. It is always the first step, never the explanation. To grade it, we use the study's three axes. With \(U^*\) the true minimal set of causes of an output and \(\hat U\) the set an explanation names:

\[ F = \frac{|U^*\cap \hat U|}{|U^*|}\ \text{(nothing causal left out)}, \qquad S = \text{does }\hat U\text{ reproduce the output?}, \qquad M = \frac{|U^*|}{|\hat U|} . \]

(The study defines F per method; for a set-valued explanation like this one it is the recall above.) A full dump names every unit, so \(\hat U\) is everything. It contains every cause (F = 1), it reproduces the output (S = 1), and M is as small as it can get.

Toy example. A network has 1000 hidden units. For one prediction, careful ablations show that only 3 units matter. The activation dump lists all 1000 numbers: F = 1, S = 1, and \(M = 3/1000 = 0.003\). A good explanation names those 3 units: F = 1, S = 1, M = 1. Same faithfulness, opposite usefulness. M is the axis that tells them apart.

From a neural network to the Atari

Recording every neuron / caching every activation	Copying the full machine state each frame: the 128 bytes of RAM (the program's variables) and the 64 TIA registers (the video/sound chip's settings), 192 bytes in all.
The output to explain	The rendered screen 15 frames after the intervention frame \(f^*\).
The true minimal cause set \(U^*\) (unknown in a brain)	Candidate RAM bytes whose poking (to 0, base+17 or base+37) changes at least one screen pixel when the real ROM is re-run: found by the intervention oracle.
The named set \(\hat U\)	All 128 RAM bytes. The TIA registers are recorded too, but the minimality denominator counts RAM only.
Sufficiency of a recording	Writing the recorded RAM back into a copy of the machine and checking that RAM and screen come out byte-identical.

How we apply it to the VCS

The runner A8_wholestate.jl is short, because the method is trivial. What it measures is not.

• Recording. From the shared gameplay state (seeded random actions, checkpoint at frame \(f^*=90\), 15-frame horizon), record_whole_state steps the real ROM with env_step! for 150 frames, continuing with NOOP after the gameplay window. It stacks get_ram and the TIA register file into a \(T\times192\) tape. Per-byte range and variance are kept as the descriptive map.
• Ground truth. oracle_causal_importance copies the checkpoint, writes each candidate byte to 0, base+17 and base+37, runs the 15-frame horizon and counts changed screen pixels. A byte with any effect is causal. These bytes form \(U^*\).
• F and S. F is 1 by construction, because the dump contains every byte. verify_wholestate_sufficiency writes the recorded 128 RAM bytes into a fresh copy of the checkpoint and compares RAM and screen with the recording. As the code itself notes, this is an identity restore. It confirms the recorder is lossless; it is not a hard test.
• M, the headline. \(M = |U^*|/128\). The runner also reports the over-recording factor \(128/|U^*|\): how many times larger the dump is than the true cause set.
• Positive control. verify_minimal_sufficiency restores only the causal bytes (again over the same checkpoint) and checks the frame is still reproduced. The oracle's own set then scores M = 1, proving the harness rewards small explanations and that the dump's low M is a real measurement.

Honest caveats. \(U^*\) is searched only among the game's candidate bytes, with three poke values and one horizon. Bytes outside that list are never tested, so \(|U^*|\) may be undercounted, and M with it. The verdict does not change: the dump is far larger than any plausible cause set.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

• XAIA8_wholestate.jl:292 — The method: copy RAM and TIA registers into the tape at every frame.
• XAIA8_wholestate.jl:360 — Ground truth: poke each candidate byte three ways, re-run, count changed pixels.
• XAIA8_wholestate.jl:397 — S check: restore the full recorded RAM and compare RAM and screen.
• XAIA8_wholestate.jl:411 — Positive control: the oracle's minimal set alone reproduces the frame.
• XAIA8_wholestate.jl:563 — The headline: minimality = causal bytes / 128.
• XAIA8_wholestate.jl:492 — One game end to end: shared state → oracle → recording → triad → control.
• substrateStellaEnvironment.jl:350 — The 128-byte RAM, read directly: the whole working state of the program.
• substrateStellaEnvironment.jl:346 — The rendered frame: the output whose pixel changes define causality here.
• substrateStellaEnvironment.jl:259 — One video frame of the real, bit-exact machine.
• substrateshared_testbed_impl.jl:115 — The shared gameplay state and cause-density gate every Phase-A runner starts from.

Reading the result

The audit box shows the three axes side by side. F and S sit at the ceiling. M sits near the floor. That contrast is the whole result. The figure shows the activity (variance) of every RAM byte over the recording, with the candidate bytes marked: many bytes change over time, far more than the handful the oracle marks as causal for the output. Read A8 as the yardstick for the other methods. Any method that claims to explain should name fewer things than this dump while keeping F and S high. A method that cannot beat the dump on M has described, not explained.

Pitfalls

• Completeness is not understanding. A record that contains every cause also contains every non-cause, with no way to tell them apart.
• Faithfulness alone is a weak score. Any explanation can reach F = 1 by naming everything. Always report a size or minimality measure next to it.
• The sufficiency check here is easy. Restoring a state into the machine it came from is an identity operation; it shows the recorder loses nothing, not that the state is explanatory.
• The true cause set is only as good as the search. Candidate bytes, three poke values and one horizon bound \(U^*\). A wider search might find more causes, which would raise M somewhat.

Try it yourself

julia --project=jutari tools/xai_study/phaseA_kording/A8_wholestate.jl --games core

Writes A8_<game>.json/.npz per game and the combined A8_wholestate.json/.npz into tools/xai_study/phaseA_kording/out/. --selftest checks the contract on the first game without writing. A few minutes per game; the cost is the three re-runs per candidate byte.

Original papers & further reading

• Alivisatos et al. (2012). The Brain Activity Map Project and the Challenge of Functional Connectomics. Neuron 74(6): 970–974. — doi.org/10.1016/j.neuron.2012.06.006
• Jonas & Kording (2017). Could a Neuroscientist Understand a Microprocessor? PLoS Computational Biology 13(1): e1005268. — doi.org/10.1371/journal.pcbi.1005268
• Lipton (2016). The Mythos of Model Interpretability. ICML Workshop on Human Interpretability in Machine Learning. — arxiv.org/abs/1606.03490

Read next: A2 · Single-unit lesions · A1 · Connectomics / data-flow graph · Causal scrubbing · ACDC — automatic circuit discovery