Phase B · attribution / XAI

Guided Backprop

Springenberg et al. 2015

← back to the method catalogue

What it does?

Guided backpropagation is a variant of saliency that changes how the signal flows backward through the network. At each rectifier it keeps only the positive contributions and zeroes the negative ones, on both the forward activation and the backward gradient. The effect is a much sharper, cleaner-looking saliency map, which is why the method became popular for visualising what a network 'sees'. The intuition is that suppressing the negative evidence isolates the features that positively drive the output. We apply the same rectified-backward rule to the VCS and correlate the resulting map with the true causal map. A well-known concern, raised by Adebayo and colleagues, is that guided backprop can act more like an edge detector than a true explanation, producing similar maps even when the model is randomised. We run exactly that sanity check on the machine itself. On smooth content outputs the method partly tracks the true cause. On a sprite's position it hits the gradient wall: the position is a discrete strobe-timed step, so the underlying gradient is zero, and the rectified backward path leaves it at zero here rather than restoring any signal. Combined with its failure of the randomisation check, this makes guided backprop a revealing case: it looks sharp and convincing yet does not depend on what the program actually computes.

Guided Backprop result

Top row (image domain, as in Paper 1): the game frame, then the oracle's true causal region and this method's attributed region — each painted onto the frame through the screen footprint of the RAM cells it implicates (brighter = more important). A faithful method's heat matches the oracle's. Bottom: per-cell importance — oracle (green) vs method (blue) — and the deletion/insertion faithfulness curves (perturb the ranked causes and watch the output move). Note: the image-domain overlay footprints are illustrative, computed on the pre-redesign boot frame; the bars, curves and all reported numbers come from the re-run records on the shared gameplay states. pearson_corr_with_oracle = 0.464 — this example only (pong, state f90+15); the audit aggregate is below.

Reading this example's causal region

This example explains the content of RAM $36 (byte 54) — the most causally-active concept byte at this state. Its strongest true-causes are: RAM $36 (the play area — ball / paddles); RAM $31 (17% of its footprint sits in the score band, the rest in the play area). The score digits appear in the region because RAM $31 reaches them: perturbing it over the 30-frame NOOP window changes the game outcome, and hence the score — a downstream effect, not direct rendering.

Gradient family · a modified backward pass

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

Rectified linear unit (ReLU)BackpropagationEdge detectionSaliency map

The idea in plain words

Vanilla saliency maps of image classifiers look grainy. Positive and negative gradient signals mix as they flow back through the network. Guided backpropagation changes one rule in the backward pass to clean this up. At every ReLU (rectified linear unit, \(\max(0, z)\), the standard on/off nonlinearity of deep nets) it lets a gradient through only if the unit was active in the forward pass and the gradient arriving from above is positive. Negative signals (“this feature argues against the class”) are thrown away.

The result looks spectacular: crisp, edge-like pictures of the object. That is why the method became popular. But it is no longer the gradient of anything. It is a hand-modified signal. Adebayo et al. (2018) asked a simple question: does the map change if we destroy the model? For guided backprop it barely does. The map behaves more like an edge detector applied to the input than an explanation of the network. This test, the model-randomisation sanity check, is part of our experiment, run on the machine itself.

How it works on a neural network

Take one ReLU unit \(h = \max(0, z)\) and let \(g = \partial y/\partial h\) be the gradient arriving from above. The three backward rules differ only in what they send down to \(z\):

\[ \text{backprop: } \mathbb{1}[z>0]\,g, \qquad \text{deconvnet: } \mathbb{1}[g>0]\,g, \qquad \text{guided: } \mathbb{1}[z>0]\,\mathbb{1}[g>0]\,g . \]

Guided backprop combines the two masks. Applied at every ReLU of the network it yields a map \(G(x)\) that, unlike saliency, is not the derivative \(\nabla f(x)\).

Toy example. Two inputs, two hidden ReLUs, one output: \(h_1 = \max(0, 2x_1 + x_2)\), \(h_2 = \max(0, x_1 - x_2 + 1)\), \(y = h_1 - h_2\). At \(x = (1, 1)\): \(h_1 = 3\), \(h_2 = 1\), \(y = 2\). Both units are active. From above, \(\partial y/\partial h = (1, -1)\).

The ranking flipped. \(h_2\) inhibits the output and \(x_2\) acts partly by reducing that inhibition. Guided backprop discards inhibition, so it cannot see this. Check the truth directly: raising \(x_2\) by 0.1 raises \(y\) by 0.2, raising \(x_1\) by 0.1 raises it by only 0.1. Here the vanilla gradient was right and the “cleaner” map was wrong.

Recipe

  1. Run the forward pass and record which ReLUs are active (\(z > 0\)).
  2. Start the backward pass from the scalar output \(y\), as in vanilla saliency.
  3. At every ReLU, pass the gradient down only where \(z > 0\) and the incoming gradient is \(> 0\); set everything else to 0.
  4. Continue to the input. Display the (usually absolute) result as a heat-map.
  5. Sanity check (Adebayo): randomise the model's weights, recompute the map, and verify it changes. A map that stays the same cannot be explaining the model.

From a neural network to the Atari

The network \(f\)The Atari 2600 VCS running the ROM, emulated bit-exactly by jutari; its differentiable twin is JuTari.Diff.
The ReLUsThere are none. The soft VCS's nonlinearities are straight-through round/clamp (identity backward), the stop-gradient branch estimator, and the distance-softmax read used only in the relaxation study. So the guided rule is applied once, to the assembled gradient over RAM.
The input \(x\)The 128 RAM bytes at the intervention frame \(f^*\).
The output \(y\)A content output (one RAM byte read one-hot) and a position output (screen pixels changed versus the un-intervened frame, made differentiable by the bilinear sampler).
Randomising the weightsRandomising the program: every ROM byte is replaced by a random byte, except the CPU's reset/interrupt vectors so the console still boots.
“Is the map right?”The intervention oracle: poke each cause, re-run the real ROM, measure \(|\Delta y|\).

How we apply it to the VCS

Our substrate has no deep stack of ReLUs. The runner says so plainly, and implements guided backprop as the rectification rule \(g \mapsto g\odot\mathbb{1}[g>0]\) applied to the gradient \(\partial y/\partial u\) over the RAM vector \(u\), followed by an absolute value. It returns the guided map and the vanilla \(|\partial y/\partial u|\), so the effect of the rule can be measured directly.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

Reading the result

On the content output the guided bars are the vanilla saliency bars, exactly. Any agreement with the oracle is saliency's agreement. On the position output the map is often empty, either because the naive gradient is zero or because the rule removed a negative sampler slope. Then look at the sanity-check box. A cosine similarity near 1 between the real and the scrambled-program maps means the map does not depend on the program. A faithful map must change when the model is destroyed; this one stays the same. The oracle, by contrast, re-runs the program and would see the difference at once.

Pitfalls

Try it yourself

julia --project=jutari tools/xai_study/phaseB_attribution/guided_backprop.jl --games core --sanity

Writes guided_backprop_<game>_{content,ball_pixel}.json/.npz and guided_backprop_core_summary.json into tools/xai_study/phaseB_attribution/out/. The sanity check boots three randomised ROMs per output; pass --no-sanity for a faster smoke run.

Original papers & further reading

Read next: Vanilla gradient (saliency) · SmoothGrad · Grad×Input / DeepLIFT · Occlusion

How it's scored

The score is the Pearson correlation of the guided-backprop map with the oracle's exact causal map, reported raw, and the grading is always against the intervention oracle. On smooth content outputs the rectified backward pass partly tracks the true cause. The hard case is a sprite's position: it is a discrete strobe-timed step, so the naive gradient is provably zero, and here the method's rectified path leaves it at zero rather than restoring any signal. On top of that, guided backprop fails the Adebayo program-randomisation sanity check on the machine itself. We scramble the ROM so the program boots to a completely different state, and the content-path map stays essentially unchanged, because its Jacobian is a constant one-hot pointing at the read index, independent of what the program computes. A faithful explanation must change when the model changes; this one does not. So even where it looks sharp, it is model-invariant, which the intervention oracle exposes at once by distinguishing the two programs. Between the zero position gradient and the failed sanity check, its cross-game faithfulness stays low. The audit box reports the measured all-regime faithfulness across the scored games.

The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.

How F, S, M are computed here

The exact formula this method uses for each score, read from its runner, and whether it matches the paper's §3 (F ∧ S ∧ M triad) definition. From the committed audit fsm_math_phaseB.json.

AxisFormula (as computed) What it measuresMatches §3?
F
faithfulness		\(F = \rho_{\mathrm{Pearson}}\!\left(\mathrm{attr},\ |\Delta y(u)|\right)\)Raw Pearson correlation between the method's per-cause attribution magnitudes and the oracle's true absolute causal effects |Delta_y(u)|, scored separately for the content and position output regimes.✓ matches guided_backprop.jl: attr = guided-backprop per cause, F = pearson(attr, oracle_abs_delta); triad_extra_dict(f.pearson, attr, f.oracle_abs_delta). Record: out/guided_backprop_air_raid_content.json (metric_name=pearson_corr_with_oracle, value=0.79). Scorer pilot_ig_vs_oracle.jl pearson() (raw Statistics.cor, zero-variance->0); F = pearson(attr, oracle_abs_delta) where oracle_abs_delta = abs.(cmap.delta) (true |Delta_y| per cause). Also reports spearman, precision@k, deletion/insertion AUC (paper's auxiliary F metrics). Per-regime split: separate *_content.json and *_position.json records, each carrying its own extra.triad.{F,S,M}.
how it's measured — call stack:
  1. per-cause guided-backprop attribution — guided_backprop.jl:601
  2. raw Pearson of attribution vs oracle |Δy(u)| — pilot_ig_vs_oracle.jl:173
  3. assemble the F∧S∧M triad record — guided_backprop.jl:909
  4. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
S
sufficiency		\(S = \dfrac{\#\{\,u_{\mathrm{held}}:\ |\hat{y}-y|\le\varepsilon\,\}}{|\mathrm{held\text{-}out}|}\in[0,1]\)Fit Delta_y = a*attr + b on a calibration half of the do(u) causes, predict the disjoint held-out half, and report the fraction of held-out causes whose predicted output lands within an epsilon band of the oracle's bit-exact re-run.✓ matches common/triad_sm.jl sufficiency_score() (interleaved calib/held split by seed, least-squares 1-D fit Delta_y~a*attr+b on calib, epsilon = max(0.5, 0.10*heldout_spread)); assembled in triad_extra_dict(). The new §3 (sec:triad) admits the fraction-within-tolerance [0,1] special case of the held-out predictive test, which this held-out do(u) fit-then-predict estimator satisfies, so it matches. (Detail: epsilon is a self-scaled band 0.10*spread; the [-1,1] correlation form is available but this method reports the [0,1] hit fraction.)
how it's measured — call stack:
  1. sufficiency_score: fit Δy=a·attr+b on a calib half, predict held-out within ε — triad_sm.jl:111
  2. assemble the F∧S∧M triad record (calls sufficiency_score) — triad_sm.jl:173
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
M
minimality		\(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\)Ratio of the number of oracle causal movers (causes with nonzero true Delta_y) to the number of cells the method names above 1e-6 of its own max attribution; null when the oracle finds no mover or the method names nothing.✓ matches common/triad_sm.jl minimality_score() with name_frac=1e-6, mover_floor=0.0; the above-threshold named set is used (topk kept but not passed), see triad_extra_dict(). This is the paper's M = |U*|/|U_hat| in (0,1] (standardized everywhere), so it matches the new §3. (Detail: |U*| is taken as all oracle movers with Delta_y>0, an upper bound on the strictly-smallest reproducing subset, so M can be optimistic.)
how it's measured — call stack:
  1. minimality_score: |U*| oracle movers / |U_hat| named cells — triad_sm.jl:57
  2. assemble the triad record (calls minimality_score) — triad_sm.jl:178
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242

✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).

In the audit

This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 44 of the 44 scored games (88 committed §R records), not the single example shown above. Tradition: gradient. The example figure (Pong) is one of those records.

0.315F — faithfulness vs oracle (mean over 44 games, ±0.085 CI95)
0.546S — sufficiency: held-out predictive (n/a where the paper does not define this axis)
0.966M — minimality: true-minimal-set / named-set (n/a otherwise)
88committed records aggregated
0.00human-plausibility proxy

F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.

Source: leaderboard.json · the whole leaderboard is on the methods page and the Paper 2 audit.

Results per game

This method's faithfulness on each of the 44 scored games (all-regime F, and the content vs position split). Click a header to sort. Every number is read from site_data.json (methods.guided_backprop.per_game).

GameF (all)content-Fposition-FSMNote
Enduro0.5001.0000.0000.5001.000position gradient vanishes
Space Invaders0.5001.0000.0000.5561.000position gradient vanishes
Tennis0.5001.0000.0000.5001.000position gradient vanishes
Montezuma's Revenge0.5001.0000.0000.9381.000position gradient vanishes
Riverraid0.5001.0000.0000.5001.000position gradient vanishes
Kangaroo0.5000.9990.0000.5001.000position gradient vanishes
Krull0.5001.0000.0000.5001.000position gradient vanishes
Q*bert0.5000.9990.0000.5001.000position gradient vanishes
Demon Attack0.4770.9550.0000.8751.000position gradient vanishes
Fishing Derby0.4740.9480.0000.4501.000position gradient vanishes
Phoenix0.4550.9090.0000.5771.000position gradient vanishes
Bank Heist0.4500.9000.0000.8331.000position gradient vanishes
Boxing0.4080.8170.0000.5001.000position gradient vanishes
Ms. Pac-Man0.4050.8100.0000.4751.000position gradient vanishes
Skiing0.4050.8100.0000.4501.000position gradient vanishes
Double Dunk0.3990.7970.0000.5381.000position gradient vanishes
Name This Game0.3940.7880.0000.4171.000position gradient vanishes
Alien0.3900.7790.0000.5001.000position gradient vanishes
Pacman0.3880.7760.0000.5561.000position gradient vanishes
Gopher0.3880.7760.0000.9291.000position gradient vanishes
Atlantis0.3860.7730.0000.4001.000position gradient vanishes
Yars' Revenge0.3850.7710.0000.4641.000position gradient vanishes
Video Pinball0.3850.7700.0000.5001.000position gradient vanishes
Ice Hockey0.3840.7670.0000.5001.000position gradient vanishes
Private Eye0.3790.7580.0000.9601.000position gradient vanishes
Breakout0.3750.7500.0000.5501.000position gradient vanishes
Jamesbond0.3740.7470.0000.8931.000position gradient vanishes
Freeway0.3730.7470.0000.5671.000position gradient vanishes
Kung-Fu Master0.3720.7450.0000.8001.000position gradient vanishes
Venture0.3720.7430.0000.4741.000position gradient vanishes
Bowling0.3440.6880.0000.8811.000position gradient vanishes
Frostbite0.2420.4840.0000.1501.000position gradient vanishes
Pong0.2320.4640.0000.3891.000position gradient vanishes
Air Raid0.1800.3600.0000.0361.000position gradient vanishes
Berzerk0.0480.0960.0000.4811.000position gradient vanishes
Seaquest0.0090.0170.0000.4791.000position gradient vanishes
Assault0.0000.0000.0000.9051.000position gradient vanishes
Asterix0.0000.0000.0000.1501.000position gradient vanishes
Carnival0.0000.0000.0000.5001.000position gradient vanishes
Centipede0.0000.0000.0000.4501.000position gradient vanishes
Chopper Command0.0000.0000.0000.4760.500position gradient vanishes
Hero0.0000.0000.0000.4440.500position gradient vanishes
Pitfall0.0000.0000.0000.8821.000position gradient vanishes
Road Runner0.0000.0000.0000.1050.500position gradient vanishes

Call stack — how it runs on jutari / jaxtari

The path from this method's runner (main) into the bit-exact VCS substrate, and the computation it involves. Every step links to the exact source on main. From callstack_phaseB.json.

Applied on the substrate

  1. env reset + per-frame step of the bit-exact VCS — env_reset! / env_step! jutari
  2. shared gameplay-state testbed — build_shared_testbed jutari

The computation (differentiable / gradient path)

  1. CONTENT output = forward-exact one-hot RAM read; a LINEAR unit (dy/du = e_idx >= 0) — content_read -> soft_ram_peek jutari-diff call site tools/xai_study/phaseB_attribution/guided_backprop.jl:320
  2. Guided Backprop: gradient then keep positive part (guided_rule) => |positive part|; returns (guided, vanilla) — guided_backprop_over_ram runner
  3. autodiff yielding the raw gradient the guided rule rectifies — Zygote.gradient(readf, x) Zygote
  4. POSITION path restored via bilinear sampler (shared testbed) — _st_sampler_position_read runner-diff

Scored against the exact intervention oracle Δy(u): run_intervention.

On jaxtari (JAX / GPU): the same differentiable path has a bit-exact JAX sibling (Theorem 1, forward-exact) for batched GPU runs — JAX-equivalent differentiable path (bit-exact per Theorem 1, for GPU batching): jaxtari/jaxtari/diff/soft_step.py, soft_mem.py, straight_through.py

Implementation
tools/xai_study/phaseB_attribution/guided_backprop.jl
Reference
Springenberg et al. 2015
Record
guided_backprop_pong_content.json
All records
phaseB_attribution/out

The figure is generated from the committed record by docs/gen_method_figures.py; the game frame and each RAM cell's screen footprint are produced by render_scenes.jl / cell_footprints.jl.