Phase B · attribution / XAI

Grad×Input / DeepLIFT

Shrikumar et al. 2017

← back to the method catalogue

What it does?

Grad times Input, in the DeepLIFT family, sharpens plain saliency by multiplying the gradient at each input by the input's own value. The product answers a slightly different question than the bare gradient: not just how sensitive the output is to an input, but how much that input, at its actual value, contributes to the output. DeepLIFT frames this as a difference from a reference, and the multiplication gives the method a completeness-like property, so the attributions sum to the change in the output. For images this tends to produce cleaner, less diffuse maps than raw saliency. We apply it to the VCS by multiplying each cause's gradient by its recorded value and correlating the result with the true causal map. On outputs that flow smoothly into the picture, the input factor concentrates the attribution onto the genuine causal byte and improves on plain saliency. On a sprite's position it inherits the same wall as every gradient method: the position is a discrete step set by strobe timing, so the underlying gradient is zero and multiplying by the input cannot create signal where there is none. The method is popular because it keeps the speed of a gradient while adding a contribution interpretation, so it is a fair, stronger member of the gradient family to test.

Grad×Input / DeepLIFT result

Top row (image domain, as in Paper 1): the game frame, then the oracle's true causal region and this method's attributed region — each painted onto the frame through the screen footprint of the RAM cells it implicates (brighter = more important). A faithful method's heat matches the oracle's. Bottom: per-cell importance — oracle (green) vs method (blue) — and the deletion/insertion faithfulness curves (perturb the ranked causes and watch the output move). Note: the image-domain overlay footprints are illustrative, computed on the pre-redesign boot frame; the bars, curves and all reported numbers come from the re-run records on the shared gameplay states. deeplift_pearson_corr_with_oracle = 0.822 — this example only (pong, state f90+15); the audit aggregate is below.

Reading this example's causal region

This example explains the score (score@RAM $31). Its strongest true-cause is: RAM $31 (17% of its footprint sits in the score band, the rest in the play area). Because it explains the score, its only true-cause is RAM $31, so the region covers the score and the ball but not the paddle: the paddle cell (RAM $36) is not causal for the score, so it is legitimately absent.

Gradient family · saliency weighted by the input

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

GradientHadamard product (element-wise product)Linear approximationTaylor series

The idea in plain words

Vanilla saliency answers one question: how sensitive is the output to each input? That is useful, but it ignores how big the input actually is. A pixel can be very sensitive and still be black (value 0). Then it contributes nothing to the score right now. Gradient×Input fixes this by multiplying the two: the attribution of input \(i\) is its gradient times its own value. The result reads as “how much of the output this input is responsible for, at the value it has”.

DeepLIFT (Deep Learning Important FeaTures) generalises the idea. It picks a reference (or baseline) input that means “nothing there”, for example an all-black image. It then splits the difference between the output at the real input and the output at the reference among the inputs. The pieces add up exactly to that difference. This property is called completeness (DeepLIFT calls it “summation-to-delta”). With an all-zero reference and a linear model, DeepLIFT and Grad×Input give the same numbers. That is the setting we use on the Atari.

How it works on a neural network

As in vanilla saliency, write the network as \(y = f(x)\). Grad×Input and DeepLIFT (rescale rule) assign input \(i\)

\[ A^{\text{G}\times\text{I}}_i = x_i\,\frac{\partial f}{\partial x_i}(x), \qquad A^{\text{DL}}_i = (x_i - x^0_i)\; m_i ,\]

where \(x^0\) is the reference and \(m_i\) is a multiplier. For a linear unit \(m_i = \partial f/\partial x_i\). Through a nonlinearity such as a ReLU, the rescale rule uses the finite slope \(m = \Delta \text{out}/\Delta \text{in}\) between reference and input instead of the infinitesimal one. The product of input and gradient is the first-order Taylor term \(\nabla f(x)\cdot x\): for a linear function with zero bias it is exactly \(f(x) - f(0)\).

Toy example 1 (linear). Let \(f(x) = 2x_1 - x_2 + 0.5x_3\) at \(x = (4, 1, 0)\). The gradient is \((2, -1, 0.5)\). Saliency says \(x_1\) matters most, then \(x_2\), then \(x_3\). Grad×Input gives \((8, -1, 0)\): \(x_3\) gets zero, because it is switched off. The sum is \(8 - 1 + 0 = 7 = f(x) - f(0)\). Completeness holds.

Toy example 2 (nonlinear). Let \(f(x) = x^2\) at \(x = 3\), reference \(x^0 = 0\). The gradient is 6, so Grad×Input says 18. But the output only changed by \(f(3) - f(0) = 9\). DeepLIFT's rescale multiplier is \((9 - 0)/(3 - 0) = 3\), giving \(3 \cdot 3 = 9\): complete. So Grad×Input is complete only for linear pieces; DeepLIFT repairs this by using finite differences.

Recipe

  1. Pick the input \(x\), the scalar output \(y\), and (for DeepLIFT) a reference \(x^0\). The usual default is all zeros.
  2. Compute the gradient \(\partial y/\partial x_i\) with one backward pass, exactly as in vanilla saliency.
  3. Multiply element-wise: \(x_i \cdot \partial y/\partial x_i\) (Grad×Input) or \((x_i - x^0_i)\cdot m_i\) (DeepLIFT).
  4. Optionally check completeness: \(\sum_i A_i\) should equal \(y(x) - y(x^0)\).
  5. Keep the sign (positive = pushes the output up) or take \(|A_i|\) for a heat-map.

From a neural network to the Atari

The network \(f\)The Atari 2600 VCS running the game ROM, emulated bit-exactly by jutari; its differentiable twin is the SOFT path JuTari.Diff.
The input \(x\)The 128 RAM bytes at the intervention frame \(f^*\) (after 90 frames of seeded random play), as floats 0–255.
The reference \(x^0\)The all-zeros RAM vector, the “absent” state. With this reference DeepLIFT and Grad×Input coincide byte for byte.
The output \(y\)A content output (one RAM byte read one-hot) and a position output (screen pixels changed versus the un-intervened frame 15 frames later, made differentiable by the bilinear sampler).
“Is the attribution right?”The intervention oracle: poke each cause, re-run the real ROM, measure \(|\Delta y|\).

How we apply it to the VCS

The runner computes three maps from one gradient: Grad×Input \(u \odot \partial y/\partial u\), DeepLIFT \((u - 0)\odot \partial y/\partial u\), and vanilla saliency \(|\partial y/\partial u|\) as the contrast. Here \(u\) is the RAM vector. Because the reference is all zeros, the first two are identical. The record's headline number is the DeepLIFT correlation with the oracle.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

Reading the result

Compare the Grad×Input bars with the vanilla saliency page. On the content output both show one spike at the same byte; Grad×Input's spike is just taller or shorter, scaled by the byte's value. So its agreement with the oracle tracks saliency closely. A one-byte map can only partly match an oracle that lights up several causes over 15 frames. On the position output, expect much weaker agreement: the sampler restores a gradient, but it points at one byte, and that byte is not always the one the oracle ranks first. Completeness holds exactly on content, yet it does not make the map more faithful. Summing to the right total says nothing about whether the total is split among the right causes.

Pitfalls

Try it yourself

julia --project=jutari tools/xai_study/phaseB_attribution/gradxinput.jl --games core

Writes gradxinput_deeplift_<game>_{content,position}.json/.npz and a per-output summary into tools/xai_study/phaseB_attribution/out/. Each game re-runs the real ROM once per cause for the oracle and twice per cause for the deletion/insertion curves, so expect minutes per game on one CPU core.

Original papers & further reading

Read next: Vanilla gradient (saliency) · Integrated Gradients · Expected Gradients · SmoothGrad

How it's scored

The score is the Pearson correlation of the Grad-times-Input, DeepLIFT-style attribution with the oracle's exact causal map, reported raw. Faithfulness for this method is that correlation, and the grading is always against the intervention oracle rather than another method. Multiplying the gradient by the input concentrates the attribution onto the genuine causal byte, so on smooth content outputs it improves on plain saliency and scores well. The hard case is the same as for the whole gradient family. A sprite's position is a discrete step set by strobe timing, so the naive gradient there is provably zero, and multiplying a zero gradient by the input is still zero. The audit turns on the differentiable sampler to give the method a fair chance on position, but the restored gradient is faithful on only a minority of games, so the position score stays low while the content score stays high. The completeness property, that attributions sum to the output change, does not rescue the position regime. The audit box reports the all-regime faithfulness, which averages the strong content result and the weak position result across the 44 scored games, so the single number sits between the two.

The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.

How F, S, M are computed here

The exact formula this method uses for each score, read from its runner, and whether it matches the paper's §3 (F ∧ S ∧ M triad) definition. From the committed audit fsm_math_phaseB.json.

AxisFormula (as computed) What it measuresMatches §3?
F
faithfulness		\(F = \rho_{\mathrm{Pearson}}\!\left(\mathrm{attr},\ |\Delta y(u)|\right)\)Raw Pearson correlation between the method's per-cause attribution magnitudes and the oracle's true absolute causal effects |Delta_y(u)|, scored separately for the content and position output regimes.✓ matches gradxinput.jl: pr = pearson(attr, cmap.abs_delta); metric_name=deeplift_pearson_corr_with_oracle. Record: out/gradxinput_deeplift_air_raid_content.json (value=0.90). Scorer pilot_ig_vs_oracle.jl pearson() (raw Statistics.cor, zero-variance->0); F = pearson(attr, oracle_abs_delta) where oracle_abs_delta = abs.(cmap.delta) (true |Delta_y| per cause). Also reports spearman, precision@k, deletion/insertion AUC (paper's auxiliary F metrics). Per-regime split: separate *_content.json and *_position.json records, each carrying its own extra.triad.{F,S,M}.
how it's measured — call stack:
  1. per-cause DeepLIFT grad×input attribution — gradxinput.jl:478
  2. raw Pearson of attribution vs oracle |Δy(u)| — pilot_ig_vs_oracle.jl:173
  3. assemble the F∧S∧M triad record — gradxinput.jl:895
  4. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
S
sufficiency		\(S = \dfrac{\#\{\,u_{\mathrm{held}}:\ |\hat{y}-y|\le\varepsilon\,\}}{|\mathrm{held\text{-}out}|}\in[0,1]\)Fit Delta_y = a*attr + b on a calibration half of the do(u) causes, predict the disjoint held-out half, and report the fraction of held-out causes whose predicted output lands within an epsilon band of the oracle's bit-exact re-run.✓ matches common/triad_sm.jl sufficiency_score() (interleaved calib/held split by seed, least-squares 1-D fit Delta_y~a*attr+b on calib, epsilon = max(0.5, 0.10*heldout_spread)); assembled in triad_extra_dict(). The new §3 (sec:triad) admits the fraction-within-tolerance [0,1] special case of the held-out predictive test, which this held-out do(u) fit-then-predict estimator satisfies, so it matches. (Detail: epsilon is a self-scaled band 0.10*spread; the [-1,1] correlation form is available but this method reports the [0,1] hit fraction.)
how it's measured — call stack:
  1. sufficiency_score: fit Δy=a·attr+b on a calib half, predict held-out within ε — triad_sm.jl:111
  2. assemble the F∧S∧M triad record (calls sufficiency_score) — triad_sm.jl:173
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
M
minimality		\(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\)Ratio of the number of oracle causal movers (causes with nonzero true Delta_y) to the number of cells the method names above 1e-6 of its own max attribution; null when the oracle finds no mover or the method names nothing.✓ matches common/triad_sm.jl minimality_score() with name_frac=1e-6, mover_floor=0.0; the above-threshold named set is used (topk kept but not passed), see triad_extra_dict(). This is the paper's M = |U*|/|U_hat| in (0,1] (standardized everywhere), so it matches the new §3. (Detail: |U*| is taken as all oracle movers with Delta_y>0, an upper bound on the strictly-smallest reproducing subset, so M can be optimistic.)
how it's measured — call stack:
  1. minimality_score: |U*| oracle movers / |U_hat| named cells — triad_sm.jl:57
  2. assemble the triad record (calls minimality_score) — triad_sm.jl:178
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242

✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).

In the audit

This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 44 of the 44 scored games (88 committed §R records), not the single example shown above. Tradition: gradient. The example figure (Pong) is one of those records.

0.471F — faithfulness vs oracle (mean over 44 games, ±0.088 CI95)
0.604S — sufficiency: held-out predictive (n/a where the paper does not define this axis)
0.993M — minimality: true-minimal-set / named-set (n/a otherwise)
88committed records aggregated
0.00human-plausibility proxy

F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.

Source: leaderboard.json · the whole leaderboard is on the methods page and the Paper 2 audit.

Results per game

This method's faithfulness on each of the 44 scored games (all-regime F, and the content vs position split). Click a header to sort. Every number is read from site_data.json (methods.gradxinput.per_game).

GameF (all)content-Fposition-FSMNote
Krull0.8041.0000.6080.5381.000
Seaquest0.7620.8130.7100.9581.000
Gopher0.7610.7760.7460.9291.000
Pacman0.6420.8030.4810.6111.000
Bowling0.6280.9080.3480.9051.000
Ice Hockey0.5760.9640.1880.5451.000
Pong0.5500.8220.2780.5001.000
Video Pinball0.5420.9230.1600.5001.000
Centipede0.5001.0000.0000.5501.000position gradient vanishes
Enduro0.5001.0000.0000.5001.000position gradient vanishes
Space Invaders0.5001.0000.0000.5281.000position gradient vanishes
Tennis0.5001.0000.0000.5311.000position gradient vanishes
Kangaroo0.5000.9990.0000.5001.000position gradient vanishes
Q*bert0.5000.9990.0000.5001.000position gradient vanishes
Chopper Command0.4990.9980.0000.9521.000position gradient vanishes
Pitfall0.4980.9970.0000.9121.000position gradient vanishes
Road Runner0.4910.9820.0000.5261.000position gradient vanishes
Boxing0.4870.8170.1560.5421.000
Private Eye0.4870.9700.0030.9601.000position gradient vanishes
Demon Attack0.4770.9550.0000.8751.000position gradient vanishes
Skiing0.4760.9520.0000.4501.000position gradient vanishes
Assault0.4750.9500.0000.9521.000position gradient vanishes
Hero0.4740.9470.0000.8891.000position gradient vanishes
Carnival0.4690.9380.0000.5261.000position gradient vanishes
Bank Heist0.4550.9110.0000.8331.000position gradient vanishes
Phoenix0.4550.9090.0000.5771.000position gradient vanishes
Jamesbond0.4470.8650.0280.8931.000
Alien0.4300.8600.0000.5001.000position gradient vanishes
Yars' Revenge0.4220.8450.0000.4641.000position gradient vanishes
Riverraid0.4110.8230.0000.4781.000position gradient vanishes
Ms. Pac-Man0.4110.8220.0000.5001.000position gradient vanishes
Double Dunk0.4080.8170.0000.5381.000position gradient vanishes
Name This Game0.4070.8150.0000.4171.000position gradient vanishes
Kung-Fu Master0.4020.8050.0000.8001.000position gradient vanishes
Freeway0.3980.7470.0490.5001.000
Fishing Derby0.3840.7690.0000.4501.000position gradient vanishes
Asterix0.3830.7650.0000.5001.000position gradient vanishes
Breakout0.3750.7500.0000.6001.000position gradient vanishes
Atlantis0.3650.7300.0000.4501.000position gradient vanishes
Montezuma's Revenge0.3620.7240.0000.4381.000position gradient vanishes
Venture0.3490.6980.0000.4740.750position gradient vanishes
Berzerk0.2870.5550.0200.4441.000position gradient vanishes
Frostbite0.2720.4840.0590.1501.000
Air Raid0.2200.2110.2280.3931.000

Call stack — how it runs on jutari / jaxtari

The path from this method's runner (main) into the bit-exact VCS substrate, and the computation it involves. Every step links to the exact source on main. From callstack_phaseB.json.

Applied on the substrate

  1. env reset + per-frame step of the bit-exact VCS — env_reset! / env_step! jutari
  2. shared gameplay-state testbed — build_shared_testbed jutari

The computation (differentiable / gradient path)

  1. CONTENT output y = one-hot RAM read at score cell (forward-exact, Theorem 1) — content_output -> soft_ram_peek jutari-diff call site tools/xai_study/phaseB_attribution/gradxinput.jl:268
  2. dy/du over the 128-byte RAM tape (one-hot content gradient e_s) — content_grad_over_ram runner
  3. autodiff of the content read — Zygote.gradient(r -> content_output(r, score_idx), ...) Zygote
  4. attribution maps: GradxInput u*dy/du, DeepLIFT (u-baseline)*dy/du (completeness-exact on the linear content unit), saliency |dy/du| — attributions_over_ram runner
  5. POSITION/pixel output is a discrete argmax of sprite column => differentiable RAM handle identically zero (vanishing gradient, modeled honestly) — position_grad_over_ram runner
  6. POSITION path restored via bilinear sampler (shared testbed) — _st_sampler_position_read runner-diff

Scored against the exact intervention oracle Δy(u): run_intervention.

On jaxtari (JAX / GPU): the same differentiable path has a bit-exact JAX sibling (Theorem 1, forward-exact) for batched GPU runs — JAX-equivalent differentiable path (bit-exact per Theorem 1, for GPU batching): jaxtari/jaxtari/diff/soft_step.py, soft_mem.py, straight_through.py

Implementation
tools/xai_study/phaseB_attribution/gradxinput.jl
Reference
Shrikumar et al. 2017
Record
gradxinput_deeplift_pong_content.json
All records
phaseB_attribution/out

The figure is generated from the committed record by docs/gen_method_figures.py; the game frame and each RAM cell's screen footprint are produced by render_scenes.jl / cell_footprints.jl.