Phase B · attribution / XAI

On-distribution counterfactual

cf. Olson 2021; Atrey 2020

← back to the method catalogue

What it does?

An on-distribution counterfactual asks the smallest realistic change that flips the output. Instead of occluding with an artificial value, it edits the input toward a genuine alternative the system could actually produce, and finds the smallest such edit that changes the output. The explanation is the edit itself: 'change these cells to these plausible values and the outcome changes'. The point of the on-distribution constraint is to answer the objection that occlusion and clamping can set an input to a value the running system would never create, making the explanation off-manifold. We apply it to the VCS by substituting candidate cells toward a real alternative state, the RAM of another frame of the same ROM, and re-running the bit-exact program. Because the substituted values are ones the machine genuinely produced, the edit stays on the data manifold by construction, and because we re-render the whole machine it is a valid intervention. We compare the minimal edit to the oracle's minimal cause set. The method works on a sprite's position where gradients fail. Its faithfulness here is limited: on some frames no on-distribution content byte varies between frames, so the search finds nothing and the map is flat, and even when it does edit, the single-edit search only partly recovers the true minimal set.

On-distribution counterfactual result

Top row (image domain, as in Paper 1): the game frame, then the oracle's true causal region and this method's attributed region — each painted onto the frame through the screen footprint of the RAM cells it implicates (brighter = more important). A faithful method's heat matches the oracle's. Bottom: per-cell importance — oracle (green) vs method (blue) — and the deletion/insertion faithfulness curves (perturb the ranked causes and watch the output move). Note: the image-domain overlay footprints are illustrative, computed on the pre-redesign boot frame; the bars, curves and all reported numbers come from the re-run records on the shared gameplay states. pearson_corr_with_oracle = 0.696 — this example only (pong, state f90+15); the audit aggregate is below.

Reading this example's causal region

This example explains the content of RAM $36 (byte 54) — the most causally-active concept byte at this state. Its strongest true-causes are: RAM $36 (the play area — ball / paddles); RAM $31 (17% of its footprint sits in the score band, the rest in the play area). The score digits appear in the region because RAM $31 reaches them: perturbing it over the 30-frame NOOP window changes the game outcome, and hence the score — a downstream effect, not direct rendering.

Perturbation family · the smallest change that flips the outcome

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

Counterfactual conditionalCausal modelExplainable artificial intelligenceManifold hypothesisGreedy algorithm

The idea in plain words

A bank’s model rejects your loan. A heat-map of “important features” does not help you much. What helps is a sentence like: “If your income had been 4 000 higher, the loan would have been approved.” That sentence is a counterfactual explanation. It names the smallest change to the input that flips the output. The word comes from logic: a counterfactual is a “what if things had been different” statement. Wachter et al. made it popular in machine learning because it needs no access to the model’s insides and is easy for a person to act on.

There is a trap. The smallest change in raw numbers can produce an input that could never occur: an image with a few strange pixels, a person aged 12 with a mortgage. Such an input is off-distribution (or off-manifold): it lies outside the region where real data lives. An explanation built on it describes the model’s behaviour on nonsense. A good counterfactual is therefore also plausible: the changed input should look like something the world really produces. On the VCS we get plausibility for free, by copying values from other frames the game really visited, and validity for free, by re-running the real program.

How it works on a neural network

For a model \(f\), an input \(x\) and a desired different output, Wachter et al. solve

\[ x^{\ast} \;=\; \arg\min_{x'} \; \lambda\,\big(f(x') - y_{\text{target}}\big)^2 \;+\; d(x, x'), \]

where \(d\) is a distance (often the \(L_1\) norm, which favours changing few features). The explanation is the difference \(x^\ast - x\). Later work adds a plausibility term that keeps \(x'\) near real data, for example a generative model (Olson et al. use one to create counterfactual Atari frames for RL agents). Two properties are checked: validity (does \(f(x^\ast)\) really reach the target?) and sparsity / minimality (how few features were changed?).

Toy example. A loan model approves when \(f(x) = 2\,\text{income} + \text{savings} - 3\,\text{debt} \ge 10\). Applicant: income 3, savings 2, debt 1, so \(f = 6 + 2 - 3 = 5\): rejected. Real applicants in the data have one of these alternative profiles: (income 5, savings 2, debt 1) or (income 3, savings 4, debt 0). Try copying one feature at a time from the first real profile: income 3→5 gives \(f = 9\), still rejected. Copying two features from the second profile, savings 2→4 and debt 1→0, gives \(f = 6 + 4 - 0 = 10\): approved. So the plausible counterfactual is “two more savings and no debt”. A purely numerical optimiser might instead suggest income 3→5.5, a value no applicant has. Both are valid; only one is grounded in real data.

Recipe

  1. Fix the input \(x\), the output \(y = f(x)\), and what counts as a flip (a class change, or a move larger than a threshold).
  2. Collect real alternative inputs \(x'\) from the data, so every substituted value is plausible.
  3. Greedily build a feature set \(S\): at each step copy the feature from \(x'\) that moves \(y\) furthest, until the move is large enough.
  4. Prune: drop any feature from \(S\) whose removal keeps the counterfactual valid.
  5. Report \(S\) and the new values as the explanation, and check validity and minimality.

From a neural network to the Atari

The model \(f\)The ROM program on the bit-exact emulated VCS (jutari), run for 15 frames from the checkpoint.
The input \(x\) and its featuresThe machine state at frame \(f^*{=}90\); the features are the candidate RAM bytes (the game’s known variables).
A real alternative \(x'\)The genuine RAM at another frame of the same seeded gameplay stream (frames such as 15, 30, 45, … before \(f^*\)).
The output \(y\)A content output (one candidate RAM byte that is causally active and varies across the alternatives) and a position output (how many pixels change in the shared screen region).
“Flip”A move of at least \(\tau{=}0.5\) of the move you get by copying all candidate bytes from \(x'\).
Evaluating \(f(x')\)Write the bytes into a copy of the checkpoint and re-run the real ROM. No surrogate, no generative model.

How we apply it to the VCS

The runner uses the shared gameplay testbed: seeded random actions, a checkpoint at \(f^*\), a 15-frame horizon. Every number below comes from a real re-run of the emulator.

  1. Alternative pool. The runner replays the same action stream to several earlier frames and keeps their full RAM. These are states the game truly reached, so any byte copied from them is an on-distribution value. This answers the off-manifold objection Atrey et al. raised against saliency explanations for Atari agents.
  2. Content byte. The content output is a candidate byte that both reacts to the oracle’s interventions and actually takes different values across the pool. If no such byte exists, the run falls back to the most causally active byte and records that it does not vary.
  3. Choosing \(x'\). For each output, the runner copies all candidate bytes from each pool frame, re-runs, and picks the frame that moves \(y\) the most: the most different real world.
  4. Per-cell map. Copy one byte at a time from \(x'\) and re-run. The size of the change in \(y\) is that byte’s attribution, which is scored against the oracle like every other attribution method (correlation, precision@3, deletion/insertion curves).
  5. Minimal valid edit. Greedy forward selection adds the byte that moves \(y\) furthest until the move reaches \(\tau\) of the full move; a backward pass removes redundant bytes. The runner reports validity and minimality: \(|S|\) divided by the number of bytes that move \(y\) on their own.

Because the method intervenes rather than differentiates, it works on the position output, where the gradient methods see an exact zero. The difference from Wachter et al. is that we never optimise continuous values. We only substitute whole bytes from real frames. That keeps every edit plausible per byte, but a combination of bytes from another frame inside the current state is not guaranteed to be a reachable state.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

Reading the result

The bars show the per-byte counterfactual effect next to the oracle’s true effect. Expect moderate agreement, stronger on content than on position. The two do not test the same edit: the oracle uses fixed pokes (base+17, zero), while the counterfactual copies a value from a real frame. When a byte has the same value in \(x'\) as in \(x\), the counterfactual cannot see it, however important it is. On some states no candidate byte varies, the map is flat, and the run marks the counterfactual invalid rather than inventing one. When an edit is found, it is usually small: one or two bytes suffice.

Pitfalls

Try it yourself

julia --project=jutari tools/xai_study/phaseB_attribution/counterfactual.jl --games core

Writes counterfactual_<game>_{content,ball_pixel}.json/.npz and counterfactual_core_summary.json into tools/xai_study/phaseB_attribution/out/. The greedy search costs a re-run per candidate byte per step; expect minutes per game on one CPU core. --tau sets the validity threshold.

Original papers & further reading

Read next: Occlusion · Extremal / meaningful perturbation · Activation patching / causal mediation · KernelSHAP / Shapley

How it's scored

The score is the Pearson correlation of the minimal counterfactual edit with the oracle's exact causal map, together with the edit's minimality against the oracle's minimal cause set, and the grading is always against the intervention oracle. The method substitutes candidate cells toward a real alternative state, another frame of the same ROM, and re-runs the bit-exact program, so the edit is on-distribution and a valid intervention, and it works on a sprite's position where the gradient family scores zero. That is why there is no position collapse here in principle. Its faithfulness is nonetheless only moderate. On some frames no on-distribution content byte varies between frames, so the search finds no valid edit and the map is flat; we mark those cells invalid rather than score them as important. Even when a valid edit exists, the single-edit search only partly recovers the true minimal set, so the correlation stays modest. Minimality is central here, because the method explicitly seeks the smallest flipping edit. The grading is always against the oracle. The audit box reports the measured all-regime faithfulness across the 44 scored games, so the moderate number reflects both the no-op frames and the partial recovery on the frames where the search does run.

The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.

How F, S, M are computed here

The exact formula this method uses for each score, read from its runner, and whether it matches the paper's §3 (F ∧ S ∧ M triad) definition. From the committed audit fsm_math_phaseB.json.

AxisFormula (as computed) What it measuresMatches §3?
F
faithfulness		\(F = \rho_{\mathrm{Pearson}}\!\left(\mathrm{attr},\ |\Delta y(u)|\right)\)Raw Pearson correlation between the method's per-cause attribution magnitudes and the oracle's true absolute causal effects |Delta_y(u)|, scored separately for the content and position output regimes.✓ matches counterfactual.jl: cf_attr = single-cell on-distribution counterfactual per cause, pr = pearson(cf_attr, odelta); triad_extra_dict(f.pearson, f.cf_attr_per_cause, f.oracle_abs_delta). Record: out/counterfactual_air_raid_content.json (metric_name=pearson_corr_with_oracle, value=0.79). Scorer pilot_ig_vs_oracle.jl pearson() (raw Statistics.cor, zero-variance->0); F = pearson(attr, oracle_abs_delta) where oracle_abs_delta = abs.(cmap.delta) (true |Delta_y| per cause). Also reports spearman, precision@k, deletion/insertion AUC (paper's auxiliary F metrics). Per-regime split: separate *_content.json and *_position.json records, each carrying its own extra.triad.{F,S,M}.
how it's measured — call stack:
  1. per-cause on-distribution counterfactual attribution — counterfactual.jl:305
  2. raw Pearson of attribution vs oracle |Δy(u)| — pilot_ig_vs_oracle.jl:173
  3. assemble the F∧S∧M triad record — counterfactual.jl:713
  4. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
S
sufficiency		\(S = \dfrac{\#\{\,u_{\mathrm{held}}:\ |\hat{y}-y|\le\varepsilon\,\}}{|\mathrm{held\text{-}out}|}\in[0,1]\)Fit Delta_y = a*attr + b on a calibration half of the do(u) causes, predict the disjoint held-out half, and report the fraction of held-out causes whose predicted output lands within an epsilon band of the oracle's bit-exact re-run.✓ matches common/triad_sm.jl sufficiency_score() (interleaved calib/held split by seed, least-squares 1-D fit Delta_y~a*attr+b on calib, epsilon = max(0.5, 0.10*heldout_spread)); assembled in triad_extra_dict(). The new §3 (sec:triad) admits the fraction-within-tolerance [0,1] special case of the held-out predictive test, which this held-out do(u) fit-then-predict estimator satisfies, so it matches. (Detail: epsilon is a self-scaled band 0.10*spread; the [-1,1] correlation form is available but this method reports the [0,1] hit fraction.)
how it's measured — call stack:
  1. sufficiency_score: fit Δy=a·attr+b on a calib half, predict held-out within ε — triad_sm.jl:111
  2. assemble the F∧S∧M triad record (calls sufficiency_score) — triad_sm.jl:173
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
M
minimality		\(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\)Ratio of the number of oracle causal movers (causes with nonzero true Delta_y) to the number of cells the method names above 1e-6 of its own max attribution; null when the oracle finds no mover or the method names nothing.✓ matches common/triad_sm.jl minimality_score() with name_frac=1e-6, mover_floor=0.0; the above-threshold named set is used (topk kept but not passed), see triad_extra_dict(). This is the paper's M = |U*|/|U_hat| in (0,1] (standardized everywhere), so it matches the new §3. (Detail: |U*| is taken as all oracle movers with Delta_y>0, an upper bound on the strictly-smallest reproducing subset, so M can be optimistic.)
how it's measured — call stack:
  1. minimality_score: |U*| oracle movers / |U_hat| named cells — triad_sm.jl:57
  2. assemble the triad record (calls minimality_score) — triad_sm.jl:178
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242

✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).

In the audit

This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 44 of the 44 scored games (88 committed §R records), not the single example shown above. Tradition: intervention. The example figure (Pong) is one of those records.

0.433F — faithfulness vs oracle (mean over 44 games, ±0.077 CI95)
0.610S — sufficiency: held-out predictive (n/a where the paper does not define this axis)
0.986M — minimality: true-minimal-set / named-set (n/a otherwise)
88committed records aggregated
0.00human-plausibility proxy

F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.

Source: leaderboard.json · the whole leaderboard is on the methods page and the Paper 2 audit.

Results per game

This method's faithfulness on each of the 44 scored games (all-regime F, and the content vs position split). Click a header to sort. Every number is read from site_data.json (methods.counterfactual.per_game).

M is n/a on 7 of these games. Minimality is M = |U*|/|Û|, the size of the true minimal cause set over the size of the set the method actually names. On a game where the method names nothing (|Û| = 0 — it discovers no circuit, its attribution map is all-zero, or it decodes no cell), that ratio is undefined, so M is left blank rather than scored. The aggregate M is the mean over the 37 games where the method did name a cause set.

GameF (all)content-Fposition-FSMNote
Fishing Derby0.9570.9480.9670.8501.000recovers the true causes
Road Runner0.8260.7600.8920.8951.000holds up on position
Bank Heist0.8250.9740.6760.8331.000
Boxing0.8090.8170.8010.7501.000
Freeway0.8070.7470.8670.6001.000holds up on position
Name This Game0.7900.7880.7920.5831.000holds up on position
Alien0.7810.8600.7020.4291.000
Atlantis0.7120.7730.6510.4001.000
Skiing0.7000.8090.5910.4500.875
Krull0.6800.7530.6080.5191.000
Bowling0.6610.8960.4250.8811.000
Riverraid0.6520.6900.6130.5651.000
Ice Hockey0.6440.7670.5210.5911.000
Pong0.6410.6960.5860.3890.875
Private Eye0.6340.7580.5100.9601.000
Demon Attack0.6080.9550.2610.8751.000
Jamesbond0.5780.7470.4080.8931.000
Frostbite0.5110.8260.1960.5501.000
Chopper Command0.4990.9980.0000.9521.000position gradient vanishes
Phoenix0.4550.9090.0000.5381.000position gradient vanishes
Pitfall0.4340.5450.3230.9411.000
Centipede0.4250.8510.0000.4501.000position gradient vanishes
Yars' Revenge0.4070.7760.0390.4641.000
Double Dunk0.3990.7970.0000.5381.000position gradient vanishes
Tennis0.3970.7950.0000.5001.000position gradient vanishes
Seaquest0.3870.7750.0000.9581.000position gradient vanishes
Video Pinball0.3850.7700.0000.5001.000position gradient vanishes
Carnival0.3830.7660.0000.4741.000position gradient vanishes
Montezuma's Revenge0.3620.7240.0000.1881.000position gradient vanishes
Hero0.3530.7050.0000.8891.000position gradient vanishes
Venture0.3490.6980.0000.4740.750position gradient vanishes
Air Raid0.2890.1950.3840.7141.000
Breakout0.2720.5440.0000.4501.000position gradient vanishes
Enduro0.2650.0000.5300.0001.000holds up on position
Assault0.1080.2160.0000.9051.000position gradient vanishes
Berzerk0.0490.0980.0000.5741.000position gradient vanishes
Asterix0.0000.0000.0000.1001.000position gradient vanishes
Gopher0.0000.0000.0000.929n/aposition gradient vanishes
Kangaroo0.0000.0000.0000.469n/aposition gradient vanishes
Kung-Fu Master0.0000.0000.0000.800n/aposition gradient vanishes
Ms. Pac-Man0.0000.0000.0000.475n/aposition gradient vanishes
Pacman0.0000.0000.0000.556n/aposition gradient vanishes
Q*bert0.0000.0000.0000.476n/aposition gradient vanishes
Space Invaders0.0000.0000.0000.528n/aposition gradient vanishes

Call stack — how it runs on jutari / jaxtari

The path from this method's runner (main) into the bit-exact VCS substrate, and the computation it involves. Every step links to the exact source on main. From callstack_phaseB.json.

Applied on the substrate

  1. env reset + per-frame step of the bit-exact VCS (counterfactual re-runs step the horizon) — env_reset! / env_step! jutari
  2. shared gameplay-state testbed — build_shared_testbed jutari

The computation (intervention path)

  1. counterfactual re-run: substitute the chosen RAM cells to the alternative RAM values, step the horizon, read y — cf_read_y -> intervene_ram! jutari intervene_ram! call at counterfactual.jl:118 -> jutari_oracle.jl:174
  2. single-cell counterfactual effect per candidate cell (|y_cf - y_x|) — single_cell_cf runner
  3. greedy search for the minimal counterfactual cell set that flips the output past tau — greedy_minimal_cf runner
  4. snapshot RAM+screen after each counterfactual re-run — snapshot jutari

Scored against the exact intervention oracle Δy(u): run_intervention.

Implementation
tools/xai_study/phaseB_attribution/counterfactual.jl
Reference
cf. Olson 2021; Atrey 2020
Record
counterfactual_pong_content.json
All records
phaseB_attribution/out

The figure is generated from the committed record by docs/gen_method_figures.py; the game frame and each RAM cell's screen footprint are produced by render_scenes.jl / cell_footprints.jl.