Phase B · attribution / XAI

SmoothGrad

Smilkov et al. 2017

← back to the method catalogue

What it does?

SmoothGrad is a denoising wrapper around gradient saliency. Raw gradients are visually noisy, so SmoothGrad adds a small amount of random noise to the input many times, computes the gradient for each noisy copy, and averages the results. The averaging cancels the high-frequency noise and leaves a cleaner map, on the intuition that the true signal is stable under small perturbations while the noise is not. It is a simple, widely used way to make any gradient method look better. We apply it to the VCS by perturbing each cause, averaging the gradients, and correlating with the true causal map. On smooth content outputs the averaging gives a modest improvement over plain saliency. The important negative result is on a sprite's position. There the naive gradient is exactly zero because the position is a discrete step set by strobe timing, and averaging many zeros is still zero: adding input noise cannot manufacture a gradient that the hard machine does not provide. Only a differentiable surrogate can restore one, and even then its faithfulness stays low. SmoothGrad is a good test of whether the popular 'just denoise the gradient' fix rescues attribution on the discrete game logic. It does not.

SmoothGrad result

Top row (image domain, as in Paper 1): the game frame, then the oracle's true causal region and this method's attributed region — each painted onto the frame through the screen footprint of the RAM cells it implicates (brighter = more important). A faithful method's heat matches the oracle's. Bottom: per-cell importance — oracle (green) vs method (blue) — and the deletion/insertion faithfulness curves (perturb the ranked causes and watch the output move). Note: the image-domain overlay footprints are illustrative, computed on the pre-redesign boot frame; the bars, curves and all reported numbers come from the re-run records on the shared gameplay states. pearson_corr_with_oracle = 0.464 — this example only (pong, state f90+15); the audit aggregate is below.

Reading this example's causal region

This example explains the content of RAM $36 (byte 54) — the most causally-active concept byte at this state. Its strongest true-causes are: RAM $36 (the play area — ball / paddles); RAM $31 (17% of its footprint sits in the score band, the rest in the play area). The score digits appear in the region because RAM $31 reaches them: perturbing it over the 30-frame NOOP window changes the game outcome, and hence the score — a downstream effect, not direct rendering.

Gradient family · saliency averaged over noise

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

Normal distributionGaussian noiseMonte Carlo methodSmoothing

The idea in plain words

Vanilla saliency maps of image networks look speckled. Neighbouring pixels get wildly different values. Smilkov et al. argued that this is not what the network “thinks”. It is because the gradient of a deep ReLU network jumps around from point to point: move the input a hair and the gradient changes a lot. The map is accurate for that exact point, but noisy as a summary.

SmoothGrad fixes the look with a simple trick: make many copies of the input, add a little random noise to each, compute the saliency map of every copy, and average them. The random jumps cancel out; what is consistently there survives. In the authors' words, it removes noise by adding noise. It works as a wrapper around any gradient method. The catch is in the word look. Smoothing makes maps cleaner to the eye, and whether it makes them more faithful is a separate question. On the VCS we can answer that question exactly.

How it works on a neural network

With \(y = f(x)\) as in vanilla saliency, SmoothGrad averages the gradient over Gaussian (normal-distribution) noise of standard deviation \(\sigma\):

\[ \mathrm{SG}_i(x) \;=\; \frac{1}{N}\sum_{k=1}^{N} \left|\frac{\partial f}{\partial x_i}\big(x + \varepsilon_k\big)\right|, \qquad \varepsilon_k \sim \mathcal{N}(0, \sigma^2 I). \]

This is a Monte Carlo estimate (an average over random samples) of the expected gradient in a fuzzy neighbourhood of \(x\). Smilkov et al. recommend \(\sigma\) at 10–20 % of the input range and \(N\) around 50. Some variants average the signed gradient; ours averages absolute values.

Toy example: smoothing can wake up a dead gradient… Take a single ReLU, \(f(x) = \max(0, x)\), at \(x = -0.1\). The vanilla gradient is 0: saliency says the input does not matter. Now draw \(N = 4\) noise values \(\varepsilon = (-0.8, 0.3, 1.2, -0.2)\). The noisy inputs are \((-0.9, 0.2, 1.1, -0.3)\), with gradients \((0, 1, 1, 0)\). SmoothGrad gives \(0.5\). With many samples and \(\sigma = 1\) it converges to \(P(x + \varepsilon > 0) \approx 0.46\). Because the kink is within reach of the noise, some samples land where the slope is non-zero.

…but not a flat one. Now take a step, \(f(x) = \mathbb{1}[x > 0]\). Its derivative is 0 on both sides of the jump. Every noisy sample also has gradient 0, and the average of zeros is zero. And for a straight line, \(f(x) = 3x\), every sample has gradient 3: smoothing changes nothing. Remember both cases: the VCS gives us exactly these two.

Recipe

  1. Choose the noise level \(\sigma\) (relative to the input range) and the sample count \(N\).
  2. Draw \(N\) noise vectors \(\varepsilon_k \sim \mathcal{N}(0, \sigma^2 I)\), one value per input.
  3. For each noisy copy \(x + \varepsilon_k\), run one forward and one backward pass and take \(|\partial y/\partial x_i|\).
  4. Average the \(N\) maps. Optionally also keep the noise-free map for comparison.
  5. Display the averaged map. Cost: \(N\) backward passes instead of one.

From a neural network to the Atari

The network \(f\)The Atari 2600 VCS running the ROM, emulated bit-exactly by jutari; its differentiable twin is JuTari.Diff.
The input \(x\)The 128 RAM bytes at the intervention frame \(f^*\) (after 90 frames of seeded random play).
Input noiseGaussian noise with \(\sigma = 8\) raw byte units (about 3 % of the 0–255 range) added to every RAM byte, \(N = 50\) copies, fixed seed.
The output \(y\)A content output (one RAM byte read one-hot) and a position output (screen pixels changed versus the un-intervened frame, made differentiable by the bilinear sampler).
“Is the smoothed map right?”The intervention oracle: poke each cause, re-run the real ROM, measure \(|\Delta y|\).

How we apply it to the VCS

The runner computes \(\mathrm{SG}(u) = \frac1N\sum_k |\partial y/\partial u|_{u+\varepsilon_k}\) over the RAM vector \(u\) with Zygote, and returns the noise-free vanilla \(|\partial y/\partial u|\) next to it. Both are scored against the oracle, so the effect of smoothing on faithfulness is measured directly, not just its effect on looks.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

Reading the result

Put this page next to vanilla saliency. On the content output the two maps are the same, and so is their agreement with the oracle: a single spike at the read byte, against an oracle that usually lights up several causes. On the position output, the smoothed sampler map is often fainter than the raw sampler map, and its agreement with the oracle stays weak. The lesson is general. SmoothGrad changes how a map looks, but here it does not change how faithful the map is. It cannot add causal information the gradient never carried.

Pitfalls

Try it yourself

julia --project=jutari tools/xai_study/phaseB_attribution/smoothgrad.jl --games core

Writes smoothgrad_<game>_{content,ball_pixel}.json/.npz and smoothgrad_core_summary.json into tools/xai_study/phaseB_attribution/out/. Defaults: --sigma 8 --n-samples 50. Add --sigma-sweep to reproduce the \(\sigma/N\) robustness grid (it re-runs every game per grid point).

Original papers & further reading

Read next: Vanilla gradient (saliency) · Integrated Gradients · Guided Backprop · RISE

How it's scored

The score is the Pearson correlation of the SmoothGrad map with the oracle's exact causal map, reported raw, and the grading is always against the intervention oracle. Averaging the gradient over input noise denoises the map, so on smooth content outputs it gives a small improvement over plain saliency. The hard case is a sprite's position. The position is a discrete step set by strobe timing, so the naive gradient there is provably exactly zero, and averaging many zeros is still zero: adding input noise cannot create a gradient the hard machine does not provide. To be strict, the audit turns on the differentiable sampler, which restores a non-zero position gradient, but that restored gradient is faithful on only a minority of games, so the position score stays low. The content score stays high. So the all-regime faithfulness sits in the middle, as an average of the strong content result and the weak position result across the 44 scored games. The audit box reports that measured number. SmoothGrad is a clean test of whether the popular denoise-the-gradient fix rescues attribution on discrete game logic, and the score shows that it does not.

The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.

How F, S, M are computed here

The exact formula this method uses for each score, read from its runner, and whether it matches the paper's §3 (F ∧ S ∧ M triad) definition. From the committed audit fsm_math_phaseB.json.

AxisFormula (as computed) What it measuresMatches §3?
F
faithfulness		\(F = \rho_{\mathrm{Pearson}}\!\left(\mathrm{attr},\ |\Delta y(u)|\right)\)Raw Pearson correlation between the method's per-cause attribution magnitudes and the oracle's true absolute causal effects |Delta_y(u)|, scored separately for the content and position output regimes.✓ matches smoothgrad.jl: noise-averaged saliency per cause, F = pearson(attr, oracle_abs_delta); triad_extra_dict(f.pearson, attr, f.oracle_abs_delta). Record: out/smoothgrad_air_raid_content.json (metric_name=pearson_corr_with_oracle, value=0.79). Scorer pilot_ig_vs_oracle.jl pearson() (raw Statistics.cor, zero-variance->0); F = pearson(attr, oracle_abs_delta) where oracle_abs_delta = abs.(cmap.delta) (true |Delta_y| per cause). Also reports spearman, precision@k, deletion/insertion AUC (paper's auxiliary F metrics). Per-regime split: separate *_content.json and *_position.json records, each carrying its own extra.triad.{F,S,M}.
how it's measured — call stack:
  1. per-cause noise-averaged saliency attribution — smoothgrad.jl:439
  2. raw Pearson of attribution vs oracle |Δy(u)| — pilot_ig_vs_oracle.jl:173
  3. assemble the F∧S∧M triad record — smoothgrad.jl:732
  4. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
S
sufficiency		\(S = \dfrac{\#\{\,u_{\mathrm{held}}:\ |\hat{y}-y|\le\varepsilon\,\}}{|\mathrm{held\text{-}out}|}\in[0,1]\)Fit Delta_y = a*attr + b on a calibration half of the do(u) causes, predict the disjoint held-out half, and report the fraction of held-out causes whose predicted output lands within an epsilon band of the oracle's bit-exact re-run.✓ matches common/triad_sm.jl sufficiency_score() (interleaved calib/held split by seed, least-squares 1-D fit Delta_y~a*attr+b on calib, epsilon = max(0.5, 0.10*heldout_spread)); assembled in triad_extra_dict(). The new §3 (sec:triad) admits the fraction-within-tolerance [0,1] special case of the held-out predictive test, which this held-out do(u) fit-then-predict estimator satisfies, so it matches. (Detail: epsilon is a self-scaled band 0.10*spread; the [-1,1] correlation form is available but this method reports the [0,1] hit fraction.)
how it's measured — call stack:
  1. sufficiency_score: fit Δy=a·attr+b on a calib half, predict held-out within ε — triad_sm.jl:111
  2. assemble the F∧S∧M triad record (calls sufficiency_score) — triad_sm.jl:173
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
M
minimality		\(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\)Ratio of the number of oracle causal movers (causes with nonzero true Delta_y) to the number of cells the method names above 1e-6 of its own max attribution; null when the oracle finds no mover or the method names nothing.✓ matches common/triad_sm.jl minimality_score() with name_frac=1e-6, mover_floor=0.0; the above-threshold named set is used (topk kept but not passed), see triad_extra_dict(). This is the paper's M = |U*|/|U_hat| in (0,1] (standardized everywhere), so it matches the new §3. (Detail: |U*| is taken as all oracle movers with Delta_y>0, an upper bound on the strictly-smallest reproducing subset, so M can be optimistic.)
how it's measured — call stack:
  1. minimality_score: |U*| oracle movers / |U_hat| named cells — triad_sm.jl:57
  2. assemble the triad record (calls minimality_score) — triad_sm.jl:178
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242

✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).

In the audit

This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 44 of the 44 scored games (88 committed §R records), not the single example shown above. Tradition: gradient. The example figure (Pong) is one of those records.

0.361F — faithfulness vs oracle (mean over 44 games, ±0.083 CI95)
0.558S — sufficiency: held-out predictive (n/a where the paper does not define this axis)
0.982M — minimality: true-minimal-set / named-set (n/a otherwise)
88committed records aggregated
0.00human-plausibility proxy

F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.

Source: leaderboard.json · the whole leaderboard is on the methods page and the Paper 2 audit.

Results per game

This method's faithfulness on each of the 44 scored games (all-regime F, and the content vs position split). Click a header to sort. Every number is read from site_data.json (methods.smoothgrad.per_game).

GameF (all)content-Fposition-FSMNote
Krull0.8041.0000.6080.5381.000
Gopher0.7610.7760.7460.9291.000
Pacman0.6290.7760.4810.6111.000
Bowling0.5180.6880.3480.9051.000
Enduro0.5001.0000.0000.5001.000position gradient vanishes
Space Invaders0.5001.0000.0000.5281.000position gradient vanishes
Tennis0.5001.0000.0000.5311.000position gradient vanishes
Montezuma's Revenge0.5001.0000.0000.9381.000position gradient vanishes
Riverraid0.5001.0000.0000.5221.000position gradient vanishes
Kangaroo0.5000.9990.0000.5001.000position gradient vanishes
Q*bert0.5000.9990.0000.5001.000position gradient vanishes
Boxing0.4870.8170.1560.5421.000
Ice Hockey0.4770.7670.1880.5451.000
Demon Attack0.4770.9550.0000.8751.000position gradient vanishes
Fishing Derby0.4740.9480.0000.5001.000position gradient vanishes
Video Pinball0.4650.7700.1600.5001.000
Phoenix0.4550.9090.0000.5771.000position gradient vanishes
Bank Heist0.4500.9000.0000.8331.000position gradient vanishes
Ms. Pac-Man0.4050.8100.0000.5001.000position gradient vanishes
Skiing0.4050.8100.0000.4501.000position gradient vanishes
Double Dunk0.3990.7970.0000.5381.000position gradient vanishes
Freeway0.3980.7470.0490.5001.000
Name This Game0.3940.7880.0000.4171.000position gradient vanishes
Alien0.3900.7790.0000.5711.000position gradient vanishes
Jamesbond0.3880.7470.0280.8931.000
Atlantis0.3860.7730.0000.4501.000position gradient vanishes
Yars' Revenge0.3850.7710.0000.4641.000position gradient vanishes
Private Eye0.3800.7580.0030.9601.000position gradient vanishes
Breakout0.3750.7500.0000.6001.000position gradient vanishes
Kung-Fu Master0.3720.7450.0000.8001.000position gradient vanishes
Venture0.3720.7430.0000.4741.000position gradient vanishes
Pong0.3710.4640.2780.4441.000
Seaquest0.3640.0170.7100.4791.000holds up on position
Air Raid0.2940.3600.2280.0361.000
Frostbite0.2720.4840.0590.1501.000
Berzerk0.0580.0960.0200.4811.000position gradient vanishes
Assault0.0000.0000.0000.9051.000position gradient vanishes
Asterix0.0000.0000.0000.1001.000position gradient vanishes
Carnival0.0000.0000.0000.5261.000position gradient vanishes
Centipede0.0000.0000.0000.5001.000position gradient vanishes
Chopper Command0.0000.0000.0000.4760.750position gradient vanishes
Hero0.0000.0000.0000.4440.750position gradient vanishes
Pitfall0.0000.0000.0000.8821.000position gradient vanishes
Road Runner0.0000.0000.0000.1320.750position gradient vanishes

Call stack — how it runs on jutari / jaxtari

The path from this method's runner (main) into the bit-exact VCS substrate, and the computation it involves. Every step links to the exact source on main. From callstack_phaseB.json.

Applied on the substrate

  1. env reset + per-frame step of the bit-exact VCS — env_reset! / env_step! jutari
  2. shared gameplay-state testbed — build_shared_testbed jutari
  3. drive one game (content headline + position contrast) — compute_game runner

The computation (differentiable / gradient path)

  1. CONTENT output = forward-exact one-hot RAM read — content_read -> soft_ram_peek jutari-diff call site tools/xai_study/phaseB_attribution/smoothgrad.jl:307
  2. SmoothGrad: average |dy/du| over n_samples Gaussian-perturbed RAM copies (sigma noise) — smoothgrad_over_ram runner
  3. autodiff on each noisy sample x + sigma*randn — Zygote.gradient(readf, xn) Zygote
  4. clean (noise-free) reference gradient for the vanilla-vs-smooth contrast — Zygote.gradient(readf, x) Zygote
  5. POSITION path: bilinear sampler read restoring the non-vanishing position gradient — _st_sampler_position_read runner-diff

Scored against the exact intervention oracle Δy(u): run_intervention.

On jaxtari (JAX / GPU): the same differentiable path has a bit-exact JAX sibling (Theorem 1, forward-exact) for batched GPU runs — JAX-equivalent differentiable path (bit-exact per Theorem 1, for GPU batching): jaxtari/jaxtari/diff/soft_step.py, soft_mem.py, straight_through.py

Implementation
tools/xai_study/phaseB_attribution/smoothgrad.jl
Reference
Smilkov et al. 2017
Record
smoothgrad_pong_content.json
All records
phaseB_attribution/out

The figure is generated from the committed record by docs/gen_method_figures.py; the game frame and each RAM cell's screen footprint are produced by render_scenes.jl / cell_footprints.jl.