Phase B · attribution / XAI

Vanilla gradient (saliency)

Simonyan et al. 2014

← back to the method catalogue

What it does?

Vanilla gradient saliency is the oldest attribution method for neural networks. It computes the gradient of the output with respect to each input, and reads a large gradient as 'this input mattered'. The intuition is a first-order sensitivity: if nudging an input would move the output a lot, that input is important. The result is a heat-map over the inputs, which for images highlights the pixels the network is most sensitive to. We apply the same idea to the VCS. The 'inputs' are the candidate causes, RAM cells and registers, and we take the gradient of a chosen output, a pixel, a score, or a game event, with respect to each one. On outputs that flow smoothly into the picture, a colour or a register value, the gradient is meaningful and points at the true cause. The hard case is a sprite's position. The VCS sets a sprite's screen position by the timing of a strobe write, which is a discrete step, so the naive gradient there is exactly zero: a tiny change in the position byte moves the sprite by no fraction of a pixel. Saliency is fast, model-agnostic in appearance, and everywhere in the literature, which makes it the natural baseline to test against a known answer.

Vanilla gradient (saliency) result

Top row (image domain, as in Paper 1): the game frame, then the oracle's true causal region and this method's attributed region — each painted onto the frame through the screen footprint of the RAM cells it implicates (brighter = more important). A faithful method's heat matches the oracle's. Bottom: per-cell importance — oracle (green) vs method (blue) — and the deletion/insertion faithfulness curves (perturb the ranked causes and watch the output move). Note: the image-domain overlay footprints are illustrative, computed on the pre-redesign boot frame; the bars, curves and all reported numbers come from the re-run records on the shared gameplay states. pearson_corr_with_oracle = 0.464 — this example only (pong, state f90+15); the audit aggregate is below.

Reading this example's causal region

This example explains the content of RAM $36 (byte 54) — the most causally-active concept byte at this state. Its strongest true-causes are: RAM $36 (the play area — ball / paddles); RAM $31 (17% of its footprint sits in the score band, the rest in the play area). The score digits appear in the region because RAM $31 reaches them: perturbing it over the 30-frame NOOP window changes the game outcome, and hence the score — a downstream effect, not direct rendering.

Gradient family · the starting point

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

Saliency mapGradientAutomatic differentiationBackpropagation

The idea in plain words

Imagine a network that looks at a photo and says “dog: 92 %”. You want to know which pixels it looked at. The simplest question you can ask is: if I change this one pixel a tiny bit, how much does the dog score move? A pixel whose small change moves the score a lot is a pixel the network is sensitive to. Ask that question for every pixel at once and paint the answers as a heat-map: that picture is the saliency map.

The “how much does the output move per tiny change of the input” number has a name from calculus: the partial derivative. Collected over all inputs it is the gradient. Deep-learning frameworks already compute gradients for training (that is what backpropagation does), so a saliency map costs just one backward pass. That is why it is the first attribution method everyone learns, and the baseline every other gradient method (Grad×Input, SmoothGrad, Integrated Gradients, …) tries to improve.

How it works on a neural network

Write the network as a function \(y = f(x)\): the input \(x = (x_1,\dots,x_n)\) is e.g. the pixel vector, the output \(y\) is one scalar (the logit of the class you care about). Saliency assigns input \(i\) the score

\[ S_i(x) \;=\; \left|\,\frac{\partial f}{\partial x_i}(x)\,\right| . \]

The derivative is evaluated at the actual input \(x\). It is a local, first-order statement: it says how \(f\) behaves for infinitesimally small changes around \(x\), via the Taylor approximation \(f(x+\delta) \approx f(x) + \nabla f(x)\cdot\delta\). Nothing more.

Toy example. Let \(f(x_1,x_2) = 3x_1 + x_2^2\) at \(x = (1, 0)\). Then \(\partial f/\partial x_1 = 3\) and \(\partial f/\partial x_2 = 2x_2 = 0\). Saliency says “only \(x_1\) matters”. Yet setting \(x_2 = 5\) changes the output by 25, far more than any reasonable change of \(x_1\). The gradient is blind to \(x_2\) because, at this point, the curve is flat in that direction. Keep this example in mind: it is exactly what happens on the Atari.

Recipe

  1. Pick the input \(x\) you want explained and the scalar output \(y\) (one logit, one pixel, one RAM byte).
  2. Run the forward pass and record it on an autodiff tape (PyTorch requires_grad, JAX grad, Julia Zygote.gradient).
  3. Run one backward pass from \(y\): this yields \(\partial y/\partial x_i\) for every input \(i\) at once.
  4. Take the absolute value (sign is usually dropped) and display it as a heat-map over the inputs.

From a neural network to the Atari

The trained network \(f\)The game cartridge's program (the ROM), run by the 6507 CPU, together with the TIA video chip and RIOT timer: the whole Atari 2600 VCS, emulated bit-exactly by jutari.
The input \(x\)The machine's internal state at the intervention frame \(f^*\): the 128 bytes of RAM (plus registers). This is the “activation vector” we explain.
The output \(y\)Two outputs per game: a content output (the value of one candidate RAM byte, the one the oracle finds most causally active) and a position output (how many screen pixels differ from the un-intervened frame 15 frames later).
BackpropagationThe differentiable SOFT path of jutari (JuTari.Diff): the same computation, written so Julia's Zygote can differentiate it. Its forward pass is bit-identical to the real machine (Theorem 1 of Paper 1).
“Ground-truth importance”Something you never have for a real network. Here we do: the intervention oracle sets each candidate cause to a new value, re-runs the real ROM, and measures how much \(y\) actually changed: \(|\Delta y(u)|\).

How we apply it to the VCS

On a neural net we never know whether a saliency map is right. On the VCS we do, because we can simply re-run the machine. The experiment therefore has two halves:

  1. The method. We stop the game at frame \(f^*\) (after 90 frames of seeded random play), take the RAM vector there and compute \(|\partial y/\partial \mathrm{ram}_u|\) with Zygote through the differentiable read of the output. Each candidate cause \(u\) (a RAM byte the game is known to use for a score, a position, …) gets that number as its attribution; the oracle's one TIA colour register and joystick cause, which the RAM gradient cannot see, get 0.
  2. The truth. For every cause \(u\) we poke a new value into the real emulator, play the same 15 frames, and measure the true effect \(|\Delta y(u)|\). That is 1 re-run per cause, no approximations.

The score is how well the two agree: Pearson / Spearman correlation, precision@3 (are the top-3 causes the same?) and deletion/insertion curves run on the real machine.

Content output. \(y\) is a RAM byte read through soft_ram_peek, a one-hot dot product \(y = e_{\text{idx}}\cdot \mathrm{ram}\). Its gradient is exactly the one-hot vector \(e_{\text{idx}}\): saliency points at the byte itself and nothing else. That is correct for “which byte holds the score”, but it says nothing about which other bytes will change the score over the next 15 frames.

Position output. This is the toy example's \(x_2\) for real. The VCS places a sprite by when the program writes a strobe register during the scanline: a discrete event. Moving the position byte by 0.001 moves the sprite by zero pixels, so the honest derivative is exactly zero everywhere: the vanishing gradient. We report that naive zero and, side by side, a gradient through a bilinear sampler that draws the sprite at a continuous column. This is the same trick spatial-transformer networks use, and it restores a non-zero gradient.

The end-to-end arm. The two reads above differentiate one line of code at \(f^*\). The e2e_gradient.jl runner instead differentiates the content byte at \(f^*{+}15\) back through every CPU instruction of the 15 frames (about 105 instructions, checkpointed every 200). That is the analogue of backpropagating through a very deep, very discrete network. The renderer (TIA) is still not differentiated.

What it found (all 44 scored games, forward pass bit-exact in every one, 8–23 hours of backward pass per game on one core). On most games the answer is the same map as the one-line read. On the content byte the per-cause map is unchanged in 35 of the 44 games, and in the position output in 35 as well. The six core games show the patterns one by one:

The lesson for neural networks: a deeper, more “honest” gradient is not automatically a more faithful explanation. Through a computation made of decisions rather than smooth arithmetic, backpropagation sees only the smooth sliver.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

Reading the result

In the figure, the bars are saliency per cause and the dots/bars beside them are the oracle's true effect. A faithful method puts its tall bars where the oracle's tall bars are. For the content output you will see one spike, at the byte that is the output. For the naive position output you will see nothing at all, because the gradient is zero. With the sampler you see a single spike at the position byte. The oracle, meanwhile, usually lights up several causes: the gradient is local, the oracle is global.

Pitfalls

Try it yourself

julia --project=jutari tools/xai_study/phaseB_attribution/saliency.jl --games core

Writes saliency_<game>_{content,position}.json/.npz into tools/xai_study/phaseB_attribution/out/. For the end-to-end variant run e2e_gradient.jl --game pong (8–23 hours per game on one CPU core; records e2e_{saliency,gradxinput}_<game>_* and e2e_core_summary.json).

Original papers & further reading

Read next: Grad×Input / DeepLIFT · SmoothGrad · Integrated Gradients · Occlusion

How it's scored

The score is the Pearson correlation of the saliency map with the oracle's exact causal map, the set of true effect sizes over the candidate causes, together with deletion and insertion area-under-curve as quality measures. Faithfulness for this method is that raw correlation, reported without any rescaling, so a map that carries no causal signal lands at zero rather than at a spurious floor. The grading is always against the intervention oracle, never against another method. The decisive hard case is a sprite's position. The VCS sets a sprite's screen position by the timing of a strobe write, a discrete step, so the naive gradient there is provably exactly zero and its correlation with the oracle is zero. To make the audit as strict as possible we also turn on the differentiable bilinear sampler, which interpolates that discrete boundary and restores a non-zero position gradient. Even then the restored gradient is faithful on only a minority of games, so the position score stays low. On smooth content outputs, by contrast, the gradient points at the true causal byte and scores high. The audit box below reports the all-regime faithfulness, which averages the strong content result and the weak position result across the scored games.

The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.

How F, S, M are computed here

The exact formula this method uses for each score, read from its runner, and whether it matches the paper's §3 (F ∧ S ∧ M triad) definition. From the committed audit fsm_math_phaseB.json.

AxisFormula (as computed) What it measuresMatches §3?
F
faithfulness		\(F = \rho_{\mathrm{Pearson}}\!\left(\mathrm{attr},\ |\Delta y(u)|\right)\)Raw Pearson correlation between the method's per-cause attribution magnitudes and the oracle's true absolute causal effects |Delta_y(u)|, scored separately for the content and position output regimes.✓ matches saliency.jl: sal_attr = abs.(gradient) per cause, pr = pearson(sal_attr, odelta); triad_extra_dict(f.pearson, f.sal_attr, f.oracle_abs_delta). Record: out/saliency_air_raid_content.json (value=0.79); position record out/saliency_air_raid_position.json has F=-0.144 (negative preserved). Scorer pilot_ig_vs_oracle.jl pearson() (raw Statistics.cor, zero-variance->0); F = pearson(attr, oracle_abs_delta) where oracle_abs_delta = abs.(cmap.delta) (true |Delta_y| per cause). Also reports spearman, precision@k, deletion/insertion AUC (paper's auxiliary F metrics). Per-regime split: separate *_content.json and *_position.json records, each carrying its own extra.triad.{F,S,M}.
how it's measured — call stack:
  1. per-cause attribution |∂y/∂u| over the RAM tape — saliency.jl:429
  2. raw Pearson of attribution vs oracle |Δy(u)| — pilot_ig_vs_oracle.jl:173
  3. assemble the F∧S∧M triad record — saliency.jl:697
  4. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
S
sufficiency		\(S = \dfrac{\#\{\,u_{\mathrm{held}}:\ |\hat{y}-y|\le\varepsilon\,\}}{|\mathrm{held\text{-}out}|}\in[0,1]\)Fit Delta_y = a*attr + b on a calibration half of the do(u) causes, predict the disjoint held-out half, and report the fraction of held-out causes whose predicted output lands within an epsilon band of the oracle's bit-exact re-run.✓ matches common/triad_sm.jl sufficiency_score() (interleaved calib/held split by seed, least-squares 1-D fit Delta_y~a*attr+b on calib, epsilon = max(0.5, 0.10*heldout_spread)); assembled in triad_extra_dict(). The new §3 (sec:triad) admits the fraction-within-tolerance [0,1] special case of the held-out predictive test, which this held-out do(u) fit-then-predict estimator satisfies, so it matches. (Detail: epsilon is a self-scaled band 0.10*spread; the [-1,1] correlation form is available but this method reports the [0,1] hit fraction.)
how it's measured — call stack:
  1. sufficiency_score: fit Δy=a·attr+b on a calib half, predict held-out within ε — triad_sm.jl:111
  2. assemble the F∧S∧M triad record (calls sufficiency_score) — triad_sm.jl:173
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
M
minimality		\(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\)Ratio of the number of oracle causal movers (causes with nonzero true Delta_y) to the number of cells the method names above 1e-6 of its own max attribution; null when the oracle finds no mover or the method names nothing.✓ matches common/triad_sm.jl minimality_score() with name_frac=1e-6, mover_floor=0.0; the above-threshold named set is used (topk kept but not passed), see triad_extra_dict(). This is the paper's M = |U*|/|U_hat| in (0,1] (standardized everywhere), so it matches the new §3. (Detail: |U*| is taken as all oracle movers with Delta_y>0, an upper bound on the strictly-smallest reproducing subset, so M can be optimistic.)
how it's measured — call stack:
  1. minimality_score: |U*| oracle movers / |U_hat| named cells — triad_sm.jl:57
  2. assemble the triad record (calls minimality_score) — triad_sm.jl:178
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242

✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).

In the audit

This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 44 of the 44 scored games (88 committed §R records), not the single example shown above. Tradition: gradient. The example figure (Pong) is one of those records.

0.361F — faithfulness vs oracle (mean over 44 games, ±0.083 CI95)
0.554S — sufficiency: held-out predictive (n/a where the paper does not define this axis)
0.980M — minimality: true-minimal-set / named-set (n/a otherwise)
88committed records aggregated
0.00human-plausibility proxy

F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.

Source: leaderboard.json · the whole leaderboard is on the methods page and the Paper 2 audit.

Results per game

This method's faithfulness on each of the 44 scored games (all-regime F, and the content vs position split). Click a header to sort. Every number is read from site_data.json (methods.saliency.per_game).

GameF (all)content-Fposition-FSMNote
Krull0.8041.0000.6080.5381.000
Gopher0.7610.7760.7460.9291.000
Pacman0.6290.7760.4810.6111.000
Bowling0.5180.6880.3480.9051.000
Enduro0.5001.0000.0000.5001.000position gradient vanishes
Space Invaders0.5001.0000.0000.5281.000position gradient vanishes
Tennis0.5001.0000.0000.5311.000position gradient vanishes
Montezuma's Revenge0.5001.0000.0000.9381.000position gradient vanishes
Riverraid0.5001.0000.0000.5001.000position gradient vanishes
Kangaroo0.5000.9990.0000.5001.000position gradient vanishes
Q*bert0.5000.9990.0000.5001.000position gradient vanishes
Boxing0.4870.8170.1560.5421.000
Ice Hockey0.4770.7670.1880.5451.000
Demon Attack0.4770.9550.0000.8751.000position gradient vanishes
Fishing Derby0.4740.9480.0000.4501.000position gradient vanishes
Video Pinball0.4650.7700.1600.5001.000
Phoenix0.4550.9090.0000.5771.000position gradient vanishes
Bank Heist0.4500.9000.0000.8331.000position gradient vanishes
Ms. Pac-Man0.4050.8100.0000.5001.000position gradient vanishes
Skiing0.4050.8100.0000.4501.000position gradient vanishes
Double Dunk0.3990.7970.0000.5381.000position gradient vanishes
Freeway0.3980.7470.0490.5001.000
Name This Game0.3940.7880.0000.4171.000position gradient vanishes
Alien0.3900.7790.0000.5711.000position gradient vanishes
Jamesbond0.3880.7470.0280.8931.000
Atlantis0.3860.7730.0000.4001.000position gradient vanishes
Yars' Revenge0.3850.7710.0000.4641.000position gradient vanishes
Private Eye0.3800.7580.0030.9601.000position gradient vanishes
Breakout0.3750.7500.0000.6001.000position gradient vanishes
Kung-Fu Master0.3720.7450.0000.8001.000position gradient vanishes
Venture0.3720.7430.0000.4741.000position gradient vanishes
Pong0.3710.4640.2780.4441.000
Seaquest0.3640.0170.7100.4791.000holds up on position
Air Raid0.2940.3600.2280.0361.000
Frostbite0.2720.4840.0590.1501.000
Berzerk0.0580.0960.0200.4811.000position gradient vanishes
Assault0.0000.0000.0000.9051.000position gradient vanishes
Asterix0.0000.0000.0000.1001.000position gradient vanishes
Carnival0.0000.0000.0000.5261.000position gradient vanishes
Centipede0.0000.0000.0000.5001.000position gradient vanishes
Chopper Command0.0000.0000.0000.4760.750position gradient vanishes
Hero0.0000.0000.0000.4440.750position gradient vanishes
Pitfall0.0000.0000.0000.8821.000position gradient vanishes
Road Runner0.0000.0000.0000.1050.500position gradient vanishes

Call stack — how it runs on jutari / jaxtari

The path from this method's runner (main) into the bit-exact VCS substrate, and the computation it involves. Every step links to the exact source on main. From callstack_phaseB.json.

Applied on the substrate

  1. env reset + per-frame step of the bit-exact VCS — env_reset! / env_step! jutari
  2. shared gameplay-state testbed (seeded random-action prefix, oracle cause-density gate, position cell + sampler geom) — build_shared_testbed jutari
  3. drive one game: build causes, pick content byte, score content headline + position contrast — compute_game runner

The computation (differentiable / gradient path)

  1. CONTENT output = forward-exact one-hot RAM read; its Zygote gradient is the content-path dy/du (Theorem 1) — content_read -> soft_ram_peek jutari-diff call site tools/xai_study/phaseB_attribution/saliency.jl:316
  2. vanilla saliency |dy/du| over the 128-byte RAM tape — saliency_over_ram runner
  3. autodiff yielding the attribution (the surrogate backward of Theorem 1) — Zygote.gradient(readf, x) Zygote
  4. POSITION path: bilinear (triangular-kernel) sampler read whose column is an affine fn of RAM byte pidx via soft_ram_peek — restores the non-vanishing position gradient — _st_sampler_position_read (wired as st.sampler_read, used at compute_game:512) runner-diff bound to sampler_read at shared_testbed_impl.jl:174; naive position_read is identically zero (vanishing baseline)

Scored against the exact intervention oracle Δy(u): run_intervention.

On jaxtari (JAX / GPU): the same differentiable path has a bit-exact JAX sibling (Theorem 1, forward-exact) for batched GPU runs — JAX-equivalent differentiable path (bit-exact per Theorem 1, for GPU batching): jaxtari/jaxtari/diff/soft_step.py, soft_mem.py, straight_through.py

Implementation
tools/xai_study/phaseB_attribution/saliency.jl
Reference
Simonyan et al. 2014
Record
saliency_pong_content.json
All records
phaseB_attribution/out

The figure is generated from the committed record by docs/gen_method_figures.py; the game frame and each RAM cell's screen footprint are produced by render_scenes.jl / cell_footprints.jl.