Gradient family · path integral from a baseline
Learn the method — a beginner's guide
New to explainable AI? This block explains the method as it is used on neural
networks, then how we run the very same method on an Atari 2600, where — unlike for a
network — we know the right answer. Every implementation step links to the code. See also the
XAI primer for the shared vocabulary.
The idea in plain words
Vanilla saliency has a blind spot called saturation. Picture a network that is already
99.9 % sure the photo shows a dog. Making the dog's ear a bit brighter cannot raise the score
much more, so the gradient there is almost zero. Yet the ear is a big reason the score is high.
The gradient only sees the flat top of the curve, not the climb that got there.
Integrated Gradients (IG) looks at the whole climb. It chooses a baseline: an input
that means “nothing”, such as a black image. It walks in a straight line from the baseline
to the real input, adds up the gradients along the way, and multiplies by how far each input moved.
The payoff is completeness: the attributions add up exactly to the output at the input minus
the output at the baseline. The price is that the answer depends on which baseline you pick. So our
runner tries four baselines and reports how much the result moves.
How it works on a neural network
With \(y = f(x)\) and a baseline \(x^0\), IG assigns input \(i\)
\[ \mathrm{IG}_i(x; x^0) \;=\; (x_i - x^0_i)\int_0^1
\frac{\partial f}{\partial x_i}\big(x^0 + \alpha\,(x - x^0)\big)\,d\alpha . \]
The integral is a line integral of the gradient along the straight path. By the
gradient theorem (the fundamental theorem of calculus for paths) the pieces sum to the
total change: \(\sum_i \mathrm{IG}_i = f(x) - f(x^0)\). In practice the integral is a
Riemann sum over \(K\) points, e.g. midpoints \(\alpha_k = (k - \tfrac12)/K\).
Toy example (2 inputs). \(f(x_1, x_2) = x_1 x_2\) at \(x = (2, 3)\), baseline \(x^0 = (0, 0)\).
On the path \(x(\alpha) = (2\alpha, 3\alpha)\) the gradient is \((3\alpha, 2\alpha)\). Then
\[ \mathrm{IG}_1 = 2\int_0^1 3\alpha\,d\alpha = 3, \qquad
\mathrm{IG}_2 = 3\int_0^1 2\alpha\,d\alpha = 3, \qquad
3 + 3 = 6 = f(x) - f(x^0). \]
Completeness holds. Compare: saliency gives \((3, 2)\), and Grad×Input gives \((6, 6)\), which sums to
12, twice the real change.
Saturation. Let \(f(x) = \min(1, x)\) at \(x = 2\). The gradient is 0, so saliency says
“irrelevant”. IG from \(x^0 = 0\) integrates slope 1 over the first half of the path:
\(\mathrm{IG} = 2 \cdot \tfrac12 = 1 = f(2) - f(0)\). IG sees the climb.
Baseline dependence. Back to \(x_1 x_2\), now with baseline \(x^0 = (2, 0)\). \(x_1\) does
not move, so \(\mathrm{IG}_1 = 0\), and \(\mathrm{IG}_2 = 3 \cdot 2 = 6\). Same model, same input:
all the credit moved to \(x_2\). The baseline defines the question being answered.
Recipe
- Choose a baseline \(x^0\) (zeros, a mean input, noise, or a real reference input).
- Pick \(K\) points on the straight path, \(x^0 + \alpha_k (x - x^0)\) with \(\alpha_k = (k - \tfrac12)/K\).
- At each point run one backward pass to get \(\partial y/\partial x_i\).
- Average the \(K\) gradients and multiply element-wise by \((x - x^0)\).
- Check completeness: \(|\sum_i \mathrm{IG}_i - (y(x) - y(x^0))|\) should be small; if not, increase \(K\).
- Repeat for several baselines and compare: the spread is part of the answer.
From a neural network to the Atari
| The network \(f\) | The Atari 2600 VCS running the ROM, emulated bit-exactly by jutari; its differentiable twin is JuTari.Diff. |
|---|
| The input \(x\) | The 128 RAM bytes at the intervention frame \(f^*\) (after 90 frames of seeded random play). |
|---|
| The baseline \(x^0\) | Four choices: all-zeros RAM (headline), every byte set to the mean of \(x\), one uniform-random RAM vector, and the real RAM at an earlier frame (\(f^*/2\)) of the same game. |
|---|
| Points on the path | \(K = 64\) midpoint RAM vectors. They hold fractional byte values the real machine cannot execute; only the differentiable read is evaluated there. |
|---|
| The output \(y\) | A content output (one RAM byte read one-hot) and a position output (screen pixels changed versus the un-intervened frame, made differentiable by the bilinear sampler). |
|---|
| “Is the attribution right?” | The intervention oracle: poke each cause, re-run the real ROM, measure \(|\Delta y|\). |
|---|
How we apply it to the VCS
The runner integrates the gradient over the RAM vector with \(K = 64\) Riemann midpoints, for each
of four baselines, and scores each IG map against the oracle. It also records each baseline's
completeness error. The headline number uses the all-zeros baseline; the sweep sits beside it.
- What about non-integer RAM? A point halfway along the path has bytes like 63.5. A real
6507 cannot run on that. The code never asks it to. At each path point it evaluates only the
differentiable read of the output (
soft_ram_peek or the bilinear sampler), which is
defined for any real number. The machine itself only runs during the oracle and the
deletion/insertion curves, always on integer pokes. As in vanilla saliency, the 15 frames between
\(f^*\) and the readout are not differentiated.
- Content output. The read is linear, so the gradient is the same one-hot vector \(e_s\) at
every path point. IG reduces to \(\mathrm{IG}_s = x_s - x^0_s\) and 0 elsewhere. Completeness is exact
for every baseline. Changing the baseline only changes the height of one spike, so the ranking of
causes, and hence the correlation with the oracle, stays put. One exception: if the baseline happens
to equal the byte's value, the spike vanishes. The sweep makes this baseline degeneracy visible.
- Position output. The naive gradient is zero all along the path, and the integral of zero is
zero, for every baseline. With the sampler on, completeness gives
\(\mathrm{IG} = \text{sampler}(x) - \text{sampler}(x^0)\): the difference in the cell's occupancy
between the sprite drawn at the baseline's column and at the real one. If the cell is empty at
both ends, IG is zero even when the sprite passed through the cell on the way.
- The four baselines.
zeros is the classic “absent” input.
mean_ram fills all 128 bytes with one number, the mean of the current RAM.
random is one seeded uniform draw in 0–255. real_alt is the genuine RAM at
frame \(f^*/2\) of the same seeded run. It is the only baseline that is itself a reachable game state.
- Scoring is the shared contract: correlation and precision@3 against the oracle,
deletion/insertion curves on the real ROM, and an oracle-vs-itself control.
Read the code
XAI the interpretability method
substrate the bit-exact emulator it runs on (jutari, Julia).
- XAI
ig_baseline_sweep.jl:365 — The method: \(K\) midpoint gradients along the path, times \((x - x^0)\), plus the completeness error. - XAI
ig_baseline_sweep.jl:372 — The interpolated RAM vector: fractional bytes, fed only to the differentiable read. - XAI
ig_baseline_sweep.jl:337 — The four baselines of the sweep: zeros, mean_ram, random, real_alt. - XAI
ig_baseline_sweep.jl:560 — The real-state baseline: the genuine RAM at half the intervention frame. - XAI
ig_baseline_sweep.jl:403 — Scores one baseline's IG against the oracle (correlation, precision@k, deletion/insertion, completeness). - XAI
ig_baseline_sweep.jl:250 — Deletion/insertion curves: occlude or restore the top-ranked causes and re-run the real ROM. - substrate
SoftStep.jl:106 — The linear one-hot read: the same gradient at every point of the path. - substrate
shared_testbed_impl.jl:357 — The bilinear sampler: defined for any real RAM value, so IG can integrate along the path. - substrate
oracle_intervene.jl:242 — The ground truth: poke a cause (integer value), re-run the real ROM, read the output. - substrate
StellaEnvironment.jl:259 — One frame of the bit-exact machine.
Reading the result
The headline bars use the zeros baseline. On the content output you see one spike at the
read byte, as in vanilla saliency. Its height is the byte's distance from the baseline. Across the
baseline sweep the spike grows or shrinks, but the correlation with the oracle barely moves. It drops to
nothing only when a baseline equals the byte. The completeness error is essentially zero
throughout. On the position output, agreement with the oracle stays low for every baseline.
Completeness is a bookkeeping guarantee about the total; it does not make IG notice the
several causes the oracle finds over 15 frames.
Pitfalls
- The baseline is a hidden choice. Different baselines answer different questions (“compared with what?”). A black baseline gives zero credit to black pixels; Sturmfels et al. show how much maps change with it.
- Off-distribution path. Points along the straight line are blends that no real input looks like. On the VCS they are fractional RAM states the machine cannot run, so IG explains the differentiable read, not the machine's behaviour on them.
- Completeness is not correctness. Attributions that sum to the right total can still credit the wrong inputs. On discrete outputs the total itself can be zero.
- Riemann error. Too few steps break completeness on curved functions. Check the completeness error, as the runner does.
- Integral of zero is zero. Where the gradient vanishes along the whole path (sprite timing, table look-ups), no baseline helps.
Try it yourself
julia --project=jutari tools/xai_study/phaseB_attribution/ig_baseline_sweep.jl --games core --baseline-sweep
Writes ig_baseline_sweep_<game>_{content,ball_pixel}.json/.npz (the sweep is under extra.baseline_sweep) and ig_baseline_sweep_core_summary.json into tools/xai_study/phaseB_attribution/out/. Use --ig-steps N to change \(K\) and --baseline zeros|mean_ram|random|real_alt to run a single baseline.
Original papers & further reading
Read next: Expected Gradients · Grad×Input / DeepLIFT · Vanilla gradient (saliency) · KernelSHAP / Shapley
