A5 · Local field potentials

Neuroscience battery · pooled signals and their rhythms

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

The idea in plain words

Put a microphone in a busy restaurant. You cannot follow any single conversation, but you still hear something: a loud hum that swells and fades. Neuroscientists do the same with the brain. An electrode in brain tissue records the local field potential (LFP): the summed electrical activity of thousands of nearby neurons. Nobody sees a single cell. What people study instead is the rhythm of this hum: how strong the slow waves are, how strong the fast ones. That breakdown of a signal into rhythms is called its power spectrum, and LFP bands such as “theta” or “gamma” are read as windows onto what the region computes.

The worry is simple. A strong rhythm might be the computation. Or it might be a pacemaker: a global clock that ticks everywhere and makes every part of the system hum at the same beat, whether or not that part does useful work. Activity that is caused by the clock but carries no computation is called epiphenomenal. In a brain you cannot easily tell the two apart. On the Atari you can, because we know every clock of the machine exactly. A5 measures how much of the “LFP” spectrum is just those clocks.

How it works on a neural network

Neuroscience origin. The LFP is a mesoscale signal: coarser than a single neuron, finer than an EEG over the whole skull. It is the analysis you can run when single-cell access is impossible. Jonas & Kording (2017) recorded the analogous “LFP” of a simulated 6502 chip and found rich rhythms that said nothing about the program it ran.

Deep-learning analogue. In a network, the LFP corresponds to pooled or aggregate activation statistics: the mean activation of a layer, the norm of a block's output, or the average over a group of units, tracked over time steps (tokens, video frames, RL steps). Write the activation of unit \(i\) at time \(t\) as \(a_i(t)\). Standardise each unit (subtract its mean, divide by its standard deviation: a z-score \(z_i\)) and average over a region \(R\):

\[ L(t) \;=\; \frac{1}{|R|}\sum_{i\in R} z_i(t), \qquad P(f_k) \;=\; \Big|\sum_{t=0}^{T-1} L(t)\,w(t)\,e^{-2\pi i k t/T}\Big|^2 . \]

\(L(t)\) is the pooled signal, \(w(t)\) a smooth window that tapers the ends, and \(P(f_k)\) the power at frequency \(f_k = k/T\) cycles per step (the discrete Fourier transform). A peak in \(P\) at period \(p = 1/f\) means the pooled activity repeats every \(p\) steps.

Toy example. Take two units over \(T=8\) steps. Unit 1 is a counter that flips every step: \(z_1 = (+1,-1,+1,-1,+1,-1,+1,-1)\). Unit 2 holds a real variable that jumps once: \(z_2 = (-1,-1,-1,-1,+1,+1,+1,+1)\). Their pool is \(L = (0,-1,0,-1,1,0,1,0)\). Its spectrum has a peak at period 2 (from unit 1) and a slow component at period 8 (from unit 2). If we know unit 1 is a clock, we can fit sin/cos waves of period 2 to \(L\) and ask what fraction of \(L\)'s energy they explain. Here it is exactly half (energy 2 of 4): half of this “LFP” is pure clock, however striking the period-2 peak looks.

From a neural network to the Atari

Neurons in a brain region / units in a layer	The 128 bytes of the VCS's RAM (random-access memory: the program's working variables, e.g. ball position, score, timers), read once per frame (one full TV picture, 1/60 s).
The LFP: pooled, z-scored activity	The mean z-scored value of a set of RAM bytes per frame: the global pool (every byte that changes) and one regional pool per game-variable family (all score bytes, all enemy bytes, …).
Theta / gamma bands	Peaks in the per-frame power spectrum, reported as periods in frames (a period-2 blink, a period-8 animation cycle, …).
A global pacemaker	The machine's known clocks: the frame cadence itself, plus every RAM byte that is an exact free-running counter (it ticks by a fixed step, or repeats with a fixed short period). Games use these as animation and blink timers.
“Is this rhythm computation?”	Answered exactly: the fraction of the pooled signal's energy that lies in the span of the known-clock waves. That part is causally inert timing, not game logic.

How we apply it to the VCS

The runner A5_lfp.jl follows the recipe on the real, bit-exact emulator. Concretely:

1. The trajectory. It builds the study's shared gameplay state: a seeded stream of random joystick actions (90 frames of play, then a 15-frame horizon). It drives the game with that stream and keeps going with NOOP (no button pressed) until the trajectory is long enough for a spectrum (256 frames for most games). Two fresh runs from power-on must give byte-identical RAM and screen, or the runner refuses to analyse. Then a5_record_ram calls env_step! and get_ram once per frame to fill a \(T\times128\) tape.
2. Finding the clocks without labels. detect_clock_cells scans each RAM byte. A byte is a ramp if its frame-to-frame difference (mod 256) is one constant non-zero step: a frame counter. It is periodic if it repeats exactly with some period between 2 and 64 frames: a blink or animation timer. No game knowledge is used.
3. The clock basis. clock_basis stacks sin/cos waves at two slow frame-cadence periods (\(T\) and \(T/2\), always present) plus the first three harmonics of every detected clock period. Each wave goes through the same detrend and window as the data. A QR factorisation makes the basis orthonormal.
4. Pooling and spectra. pool_region builds the global pool and one pool per variable family from the candidate-cause list. dft_power computes a plain \(O(T^2)\) DFT (no FFT package is added to the emulator environment). The headline number is the global pool's clock-explained fraction, from variance_explained_by.
5. Controls. The self-test projects a pure clock wave (must score above 0.9) and a mid-band sinusoid placed between frequency bins (must score below 0.5). So a high clock fraction is a measurement, not a basis that explains everything.

Where we differ from the neuroscience original. We sample once per frame. The VCS also has a scanline clock (262 TV lines per frame) and a CPU clock, but those tick far faster than our sampling rate and cannot appear in a per-frame spectrum. The clocks A5 actually removes are the frame cadence and the frame-driven counter bytes. A5 is also descriptive. It never intervenes; the oracle is used only to build and gate the shared gameplay state. Its score asks “how much of this signal is known-inert timing?”, not “does it find the causes of an output?”.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

• XAIA5_lfp.jl:629 — One game end to end: shared gameplay stream → bit-exact check → RAM tape → clocks → regional spectra.
• XAIA5_lfp.jl:197 — The recording: step the ROM one frame at a time and copy the 128 RAM bytes into the tape.
• XAIA5_lfp.jl:538 — The LFP itself: z-score every varying byte of a region and average them per frame.
• XAIA5_lfp.jl:342 — A dependency-free one-sided DFT power spectrum (cycles per frame).
• XAIA5_lfp.jl:417 — Label-free detection of the known clocks: constant-stride ramps and fixed-period timers.
• XAIA5_lfp.jl:460 — Frame-cadence waves plus harmonics of every clock period, detrended, windowed and orthonormalised.
• XAIA5_lfp.jl:511 — The score: the share of a pooled signal's energy that lies in the clock subspace.
• substrateshared_testbed_impl.jl:115 — The shared gameplay state: seeded random actions, checkpoint at frame 90, oracle cause-density gate.
• substrateStellaEnvironment.jl:259 — One video frame of the real, bit-exact machine.
• substrateStellaEnvironment.jl:350 — Direct read of the 128-byte RAM: the “electrode” that records every unit.

Reading the result

The figure shows the global pool's power spectrum in cycles per frame. Expect most of the power at the lowest frequencies, slow sweeps over the whole recording (periods of \(T\) or \(T/2\) frames), and in some games sharp peaks at short integer periods: these look like the “brain rhythms” of the game. Then compare them with the clock basis. The slow sweeps are the frame cadence, the short peaks sit on a detected timer period or its harmonics, and in most games the clock-explained fraction is large. The rich-looking spectrum is mostly the machine's timing. Regional pools for real game variables (score, enemies) are often just as clock-dominated, or flat when their bytes never change; either way the spectrum cannot tell you what they compute or what causes them. The live faithfulness number is in the audit box below.

Pitfalls

• Rhythm is not computation. A strong, clean peak is often the easiest thing to see and the least informative: a pacemaker shows up everywhere it is wired.
• Sampling hides fast clocks. A per-frame recording cannot see the scanline or CPU clocks. They do not vanish; anything faster than half the sampling rate folds into lower frequencies or is averaged away. The same holds for an LFP filtered at a few hundred hertz.
• Pooling destroys identity. Averaging many units cancels opposite signals and lets one loud unit dominate. A flat pooled signal does not mean the units are idle.
• The clock basis is a choice. A larger basis explains more of any signal. That is why the runner checks that an off-grid sinusoid stays below 0.5; any LFP analysis should run a similar control.
• Descriptive only. Nothing here is an intervention. The spectrum says how activity varies, never which byte drives which output.

Try it yourself

julia --project=jutari tools/xai_study/phaseA_kording/A5_lfp.jl --games core

Writes A5_<game>.json/.npz (pooled signal, spectrum, clock cells, per-region clock fractions) into tools/xai_study/phaseA_kording/out/. Add --selftest to run the controls on the first game without writing. A few minutes per game on one CPU core, most of it building the shared gameplay state.

Original papers & further reading

• Jonas & Kording (2017). Could a Neuroscientist Understand a Microprocessor? PLoS Computational Biology 13(1): e1005268. — doi.org/10.1371/journal.pcbi.1005268
• Buzsáki, Anastassiou & Koch (2012). The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes. Nature Reviews Neuroscience 13: 407–420. — doi.org/10.1038/nrn3241

Read next: A4 · Pairwise correlations · A6 · Granger causality · A7 · Dimensionality reduction (NMF/PCA) · A2 · Single-unit lesions