A3 · Tuning curves

Phase A · neuroscience battery · what does this unit respond to?

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

The idea in plain words

Put an electrode next to one neuron in a cat's visual cortex. Show the cat a bar of light at different angles. At one angle the neuron fires a lot; at others it is quiet. Plot firing rate against angle and you get a bump: the neuron's tuning curve. A sharp bump makes the neuron selective, or “tuned”, to that angle. Hubel and Wiesel won a Nobel prize for such curves. The tempting conclusion is that the neuron encodes the angle and that the brain uses it to see.

Deep learning does the same thing under other names: finding the “cat neuron”, the “edge detector” in layer 1, or a unit that lights up for one class. The method is purely observational. It watches a unit and a variable change together; it never asks whether the unit is needed. On the VCS we can check that second question with the oracle. We find that many strongly tuned bytes play no causal role at all. Being correlated with a variable and being used to produce behaviour are different things.

How it works on a neural network

Neuroscience origin. A tuning curve is the mean response \(r_i(s)\) of unit \(i\) as a function of a stimulus variable \(s\) (bar angle, sound pitch, head direction). Its sharpness is often summarised by a selectivity index. The extreme form is the “grandmother cell” that fires only for one person.

Neural-network analogue. Network dissection (Bau et al.) measures how well the activation map of a unit overlaps a labelled concept (“dog”, “wood”). Class-selectivity indices compare a unit's mean activation on its favourite class with the others. A simple, general version is the absolute Pearson correlation between a unit's activation over many inputs and a variable \(g\):

\[ s_i \;=\; \max_{g}\; \big|\rho\big(x_i(t),\, g(t)\big)\big|,\qquad \rho(x,g) = \frac{\operatorname{cov}(x,g)}{\sigma_x\,\sigma_g}. \]

A unit is strongly tuned if \(s_i \ge \tau\). Leavitt & Morcos then ablated highly selective units and found that removing them often did not hurt, and sometimes helped: selectivity is a poor guide to importance.

Toy example. A game runs for 10 frames, \(t = 1,\dots,10\). The ball moves right at a steady speed: \(\text{ball\_x}(t) = 20 + 2t\). A separate byte is a frame counter: \(\text{timer}(t) = t\). The timer is never read when the ball is drawn. Still, \(\rho(\text{timer}, \text{ball\_x}) = 1\) exactly, because both are straight lines in \(t\). The tuning method calls the timer perfectly tuned to ball position. If you overwrite the timer, the ball does not move differently. In the language of statistics, the shared driver \(t\) is a confounder: it makes two unrelated things move together.

From a neural network to the Atari

The neuron	One candidate RAM byte of the VCS's 128-byte working memory, taken from published RAM annotations (OCAtari / AtariARI) for that game.
The recording	A 150-frame trajectory: RAM and screen logged after every frame while the bit-exact emulator plays the shared seeded random input (then no-ops).
The stimulus variable	Two families: game variables (the other labelled bytes' own time series, e.g. ball x, plus a raw score byte) and luminance (mean palette value of the screen) as a low-level “hardware” feature.
The tuning curve	Absolute Pearson correlation of the byte's value with each variable over the 150 frames; the frame clock is kept as a confound control.
“Is it used?”	The oracle: overwrite the byte at frame \(f^*\) with \(0\), base\(+17\) or base\(+37\), re-run the real ROM 15 frames, count changed screen pixels. Any change means causal.

How we apply it to the VCS

The runner A3_tuning.jl combines one observational part (the method) and one interventional part (the truth) for each game.

1. Shared state. The cartridge (the ROM) is booted with the game's own settings, then played for 90 frames of seeded random joystick input; that frame is \(f^*\). Two fresh replays must match byte for byte. An oracle gate checks that enough causes move the screen.
2. Recording. record_parity_trajectory steps the emulator frame by frame for 150 frames along the same input stream (no-op after the 105 gameplay frames) and stores the 128 RAM bytes and the full screen after every frame. This is the “electrode recording”.
3. Tuning. For each candidate byte the runner correlates its time series with (a) every other candidate byte's time series and a raw score byte, (b) mean screen luminance, and (c) the frame index. The byte's own column is excluded, since a byte trivially correlates with itself. Selectivity is the maximum over (a).
4. Oracle. For each candidate byte, three interventions at \(f^*\), each followed by a real 15-frame re-run. The causal importance is the largest number of screen pixels that differ from the unperturbed run. A byte is causal if that number is above zero.
5. Scores. The spurious-tuning rate (strongly tuned but not causal), for game variables and for luminance. The triad: F is the Spearman rank correlation between selectivity and oracle importance; S correlates selectivity with the effect of the held-out base\(+37\) intervention alone; M compares how many bytes the method names with how many truly matter. A positive control scores the oracle against itself.

Why the trap is so easy to fall into on the VCS. Everything in a game is locked to the 60 Hz frame clock. Timers, animation counters and positions of moving objects all change in step with it, so over a short window many bytes look like straight lines in time and correlate strongly with each other. Some of those bytes are also never used for what reaches the screen within 15 frames. The game variables themselves are other RAM bytes, not hand-measured quantities, so a “tuned” pair may simply be two outputs of the same routine.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

• XAIA3_tuning.jl:234 — The recording: step the ROM frame by frame and stack RAM plus screen into a (T × n) tape.
• XAIA3_tuning.jl:390 — The stimulus variables: each labelled byte's time series plus a raw score byte.
• XAIA3_tuning.jl:370 — The hardware variable (mean screen luminance) and the frame-clock confound.
• XAIA3_tuning.jl:413 — The method: |Pearson| tuning, self-column excluded, threshold τ = 0.7, spurious-tuning rates.
• XAIA3_tuning.jl:325 — The ground truth: do(0 / +17 / +37) per byte, 15-frame re-run, max changed pixels.
• XAIA3_tuning.jl:477 — F (Spearman vs oracle), S (held-out +37 effect), M (named vs true movers).
• XAIA3_tuning.jl:582 — One game end to end: shared state, oracle, recording, tuning, triad.
• substrateshared_testbed_impl.jl:115 — The shared seeded-gameplay state and its oracle cause-density gate.
• substrateStellaEnvironment.jl:259 — One video frame of the bit-exact emulated VCS (called once per recorded frame).
• substrateStellaEnvironment.jl:350 — Read the 128 RAM bytes out of the console: the “electrode”.

Reading the result

The figure is a scatter plot with one dot per candidate byte. The x-axis is the oracle's true causal importance, the y-axis the byte's tuning strength to a game variable. Red dots are spurious: strongly tuned (high on the y-axis) yet at zero on the x-axis. Green dots are genuine. If tuning were a good guide, the dots would rise from bottom-left to top-right. In Pong the top-left corner is occupied: bytes that track a game variable almost perfectly but do not change the screen when overwritten. Also look at the bottom-right: causal bytes, such as a score byte that does not change during the window, show no tuning at all. Tuning misses them as well.

Pitfalls

• Present is not used. A unit can carry information about a variable without the system ever reading it. This is the same trap that defeats linear probing.
• Shared drivers. Anything that changes steadily with time (a clock, a scrolling position) correlates with everything else that does. Always include a clock or time control.
• Constant causes are invisible. A byte that is crucial but does not change during the recording has zero variance, so zero correlation, and is never called tuned.
• Threshold and window. The rate depends on \(\tau\) and on which 150 frames are recorded. A different phase of play can tune different bytes.
• Linear only. Pearson's \(\rho\) sees straight-line relations. A byte that encodes a variable through a table or a bit mask can look untuned.

Try it yourself

julia --project=jutari tools/xai_study/phaseA_kording/A3_tuning.jl --games core

Writes A3_tuning_<game>.{json,npz} per game and the index A3_tuning_all.json into tools/xai_study/phaseA_kording/out/. --tau 0.7 sets the tuning threshold, --traj-frames 150 the recording length, --selftest runs the checks without writing.

Original papers & further reading

• Hubel & Wiesel (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physiology 160: 106–154. — pmc.ncbi.nlm.nih.gov/articles/PMC1359523/
• Bau, Zhou, Khosla, Oliva & Torralba (2017). Network Dissection: Quantifying Interpretability of Deep Visual Representations. CVPR. — arxiv.org/abs/1704.05796
• Leavitt & Morcos (2020). Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs. ICLR 2021. — arxiv.org/abs/2003.01262
• Jonas & Kording (2017). Could a Neuroscientist Understand a Microprocessor? PLOS Computational Biology 13(1): e1005268. — doi.org/10.1371/journal.pcbi.1005268

Read next: Linear probing + control tasks · A4 · Pairwise correlations · A2 · Single-unit lesions · A7 · Dimensionality reduction (NMF/PCA)