Phase B · attribution / XAI

RISE

Petsiuk et al. 2018

← back to the method catalogue

What it does?

RISE explains a black box by random masking. It generates many random masks, applies each to the input, records the output for each masked version, and then forms an importance map as the output-weighted average of the masks. Inputs that tend to be present when the output is high receive high importance. The method needs no gradients and no access to the model's internals; it only needs to query the output, which is why it is popular for truly opaque systems. The intuition is a Monte-Carlo estimate of each input's marginal effect. We apply it to the VCS by drawing hundreds of random masks over the candidate causes, re-running the program for each, and averaging. Because it perturbs and re-runs the real machine, it works on a sprite's position where gradients are dead, unlike the gradient family. Its accuracy depends on the number of masks and on how informative random masking is for a given output. On smooth content outputs it does reasonably well. On a discrete position output, masking a single discrete cause is less informative, so the estimate is noisier and its faithfulness drops. RISE is a clean example of a perturbation method that is genuinely causal in its mechanism but pays for using random rather than targeted interventions.

RISE result

Top row (image domain, as in Paper 1): the game frame, then the oracle's true causal region and this method's attributed region — each painted onto the frame through the screen footprint of the RAM cells it implicates (brighter = more important). A faithful method's heat matches the oracle's. Bottom: per-cell importance — oracle (green) vs method (blue) — and the deletion/insertion faithfulness curves (perturb the ranked causes and watch the output move). Note: the image-domain overlay footprints are illustrative, computed on the pre-redesign boot frame; the bars, curves and all reported numbers come from the re-run records on the shared gameplay states. pearson_corr_with_oracle = 0.610 — this example only (pong, state f90+15); the audit aggregate is below.

Reading this example's causal region

This example explains the content of RAM $36 (byte 54) — the most causally-active concept byte at this state. Its strongest true-causes are: RAM $36 (the play area — ball / paddles); RAM $31 (17% of its footprint sits in the score band, the rest in the play area). The score digits appear in the region because RAM $31 reaches them: perturbing it over the 30-frame NOOP window changes the game outcome, and hence the score — a downstream effect, not direct rendering.

Perturbation family · random joint masks

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

Monte Carlo methodBernoulli distributionConditional expectationImportance sampling

The idea in plain words

Occlusion hides one part at a time. RISE hides many parts at once, at random. Throw a random pattern of holes over the photo (about half the pixels hidden), run the network, write down the dog score. Repeat a few hundred times with fresh random patterns. Now ask, for each pixel: when this pixel happened to be visible, how high was the score on average? Pixels the network needs will be visible mostly in the high-scoring runs. Pixels it ignores will be visible in high and low runs alike.

This is a Monte-Carlo estimate: instead of trying every possible pattern (there are \(2^n\)), you average over random samples, and the estimate gets better as you draw more. RISE is a black-box method like occlusion. Because many parts are hidden together, it can notice parts that only matter in combination, which occlusion cannot. The price is noise: with few masks, the map is grainy.

How it works on a neural network

Draw \(N\) random binary masks \(m_1,\dots,m_N \in \{0,1\}^n\). Each entry is an independent coin flip: kept (\(1\)) with probability \(p\), hidden (\(0\)) otherwise (a Bernoulli variable). Run the model on each masked input, \(y_k = f(x \odot m_k)\). The importance of input \(i\) is

\[ S_i \;=\; \frac{1}{N\,p}\sum_{k=1}^{N} y_k \, m_{k,i} \;\;\approx\;\; \mathbb{E}\big[\,y \mid m_i = 1\,\big] . \]

That is the average output over the masks that kept input \(i\): a conditional expectation. The \(1/(Np)\) factor turns the sum into that average, because about \(Np\) masks keep \(i\). On images, RISE draws small grids, upsamples them smoothly and shifts them randomly, so the holes are blobs rather than single pixels.

Toy example. \(f(x) = 4x_1x_2 + x_3\) at \(x=(1,1,1,1)\) (\(x_4\) unused), \(p = 0.5\). Enumerate all 16 masks instead of sampling. When \(x_1\) is kept, \(x_2\) is kept half the time and \(x_3\) half the time, so \(\mathbb{E}[y\mid m_1{=}1] = 4\cdot0.5 + 0.5 = 2.5\). Likewise \(S_2 = 2.5\), \(S_3 = 4\cdot 0.25 + 1 = 2\), \(S_4 = 1.5\). The unused \(x_4\) gets exactly the overall mean \(\mathbb{E}[y] = 1.5\). Subtract it and you get \((1, 1, 0.5, 0)\): the interaction pair ranks first, \(x_3\) second, \(x_4\) is zero. With \(p = 0.5\) this centred score is half the average change caused by switching \(x_i\) on (the Banzhaf value, a close relative of the Shapley value). Note: occlusion on the same toy gives \((4,4,1,0)\), and both agree on the ranking here.

Recipe

  1. Choose the output \(y\), the number of masks \(N\), the keep-probability \(p\) and what “hidden” means (here: set to 0).
  2. Draw a random mask: keep each input with probability \(p\).
  3. Hide the inputs whose mask bit is 0, run the model, record \(y_k\). Repeat \(N\) times.
  4. For each input, average \(y_k\) over the masks that kept it: \(S_i = \frac{1}{Np}\sum_k y_k m_{k,i}\).
  5. Normalise the map (subtract the mean) and display it.

From a neural network to the Atari

The network \(f\)The cartridge program on the bit-exact emulated Atari 2600 VCS (jutari).
The pixels \(x_i\)The candidate RAM cells (the bytes the oracle scores), de-duplicated: 8 to 23 per core game. TIA and joystick causes are not in RISE's universe.
Masked input \(x\odot m\)deepcopy the checkpoint, intervene_ram!(env, i, 0) for every dropped cell, run 15 frames of the real ROM.
\(N\) masks, keep-prob \(p\)\(N = 500\) masks, \(p = 0.5\), from a seeded random generator, so the run is reproducible.
The output \(y\)A content RAM byte at \(f^*{+}15\), or the number of screen pixels that differ from the un-intervened frame.
“Ground-truth importance”The oracle's per-cause |Δy|, from single-cause re-runs (+17 and →0 per byte, plus a TIA and a joystick do).

How we apply it to the VCS

The VCS version follows the paper's formula one-to-one; only the “image” changes. The universe is the list of candidate RAM cells. Each of the \(N = 500\) masks keeps every cell with probability \(p = 0.5\). For each mask, masked_read_y copies the checkpoint, zeroes the dropped cells with intervene_ram!, plays the recorded 15 actions with env_step!, and reads \(y\). Those 500 real-emulator runs are the whole cost of the method. rise_importance then computes \(\frac{1}{Np}\sum_k y_k m_{k,i}\).

From cells to causes. The oracle scores causes, not cells. rise_attr_per_cause centres the cell scores by subtracting their mean across cells, takes the absolute value, and gives each RAM cause the score of its cell (so a byte's :set and :occlude causes get the same value). TIA and joystick causes get 0, because RISE never touches them.

Convergence. RISE is a Monte-Carlo estimate, so the runner also rescores the map using only the first \(N/8\), \(N/4\), \(N/2\) masks of the same pool. That shows how the agreement with the oracle grows with compute, without drawing new masks.

How is this different from the oracle? Three ways. (1) Joint masks: in every run about half the cells are zeroed at once, so each \(y_k\) mixes many effects; the oracle changes one cause at a time. (2) Averaging: RISE reports an average over random contexts, the oracle a single exact difference at the real state. (3) One direction: RISE only ever sets cells to 0. On a content byte that responds smoothly to many cells, the average is informative. On the position output, the effect of one cell is often an all-or-nothing jump that depends on which other cells are zeroed with it, so the average is noisier.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

Reading the result

Bars are RISE scores per cause, next to the oracle's true effect. Two features are built in: TIA and joystick causes are always zero for RISE, and the two causes of one byte always share a height. Beyond that, compare the tallest bars. On content outputs expect reasonable agreement; on position outputs expect a noisier map with a weaker match. The convergence numbers in the record show whether more masks would still help: if agreement is still rising from \(N/2\) to \(N\), the estimate has not settled.

Pitfalls

Try it yourself

julia --project=jutari tools/xai_study/phaseB_attribution/rise.jl --games core

Writes rise_<game>_{content,ball_pixel}.json/.npz into tools/xai_study/phaseB_attribution/out/, including the convergence sweep. Change the budget with --n-masks and --keep-prob.

Original papers & further reading

Read next: KernelSHAP / Shapley · LIME · Occlusion · Extremal / meaningful perturbation

How it's scored

The score is the Pearson correlation of the RISE importance map with the oracle's exact causal map, reported raw, and the grading is always against the intervention oracle. RISE forms its map by averaging the output over hundreds of random masks, each a real re-run of the machine, so it is a genuine perturbation method and works on a sprite's position where the gradient family scores zero. That is why it does not collapse on the position regime. Its faithfulness is moderate: on smooth content outputs the random-mask average recovers the true causes reasonably well, but on a discrete position output masking a single discrete cause is less informative, so the estimate is noisier and the position score drops below the content score. The accuracy also depends on how many masks are drawn, since RISE is a Monte-Carlo estimate. The grading is always against the oracle, never against another method. The audit box reports the measured all-regime faithfulness across the 44 scored games, which averages the stronger content result and the weaker position result. RISE shows that a truly perturbation-based method can beat the gradient family on discrete outputs while still paying for using random rather than targeted interventions.

The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.

How F, S, M are computed here

The exact formula this method uses for each score, read from its runner, and whether it matches the paper's §3 (F ∧ S ∧ M triad) definition. From the committed audit fsm_math_phaseB.json.

AxisFormula (as computed) What it measuresMatches §3?
F
faithfulness		\(F = \rho_{\mathrm{Pearson}}\!\left(\mathrm{attr},\ |\Delta y(u)|\right)\)Raw Pearson correlation between the method's per-cause attribution magnitudes and the oracle's true absolute causal effects |Delta_y(u)|, scored separately for the content and position output regimes.✓ matches rise.jl: pr = pearson(attr, odelta); triad_extra_dict(f.pearson, f.rise_attr_per_cause, f.oracle_abs_delta). Record: out/rise_air_raid_content.json (value=0.76); note M=0.077 there (|U_hat|=26 named vs |U*|=2 movers) illustrating the dense-mask minimality penalty. Scorer pilot_ig_vs_oracle.jl pearson() (raw Statistics.cor, zero-variance->0); F = pearson(attr, oracle_abs_delta) where oracle_abs_delta = abs.(cmap.delta) (true |Delta_y| per cause). Also reports spearman, precision@k, deletion/insertion AUC (paper's auxiliary F metrics). Per-regime split: separate *_content.json and *_position.json records, each carrying its own extra.triad.{F,S,M}.
how it's measured — call stack:
  1. per-cause RISE mask attribution — rise.jl:249
  2. raw Pearson of attribution vs oracle |Δy(u)| — pilot_ig_vs_oracle.jl:173
  3. assemble the F∧S∧M triad record — rise.jl:562
  4. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
S
sufficiency		\(S = \dfrac{\#\{\,u_{\mathrm{held}}:\ |\hat{y}-y|\le\varepsilon\,\}}{|\mathrm{held\text{-}out}|}\in[0,1]\)Fit Delta_y = a*attr + b on a calibration half of the do(u) causes, predict the disjoint held-out half, and report the fraction of held-out causes whose predicted output lands within an epsilon band of the oracle's bit-exact re-run.✓ matches common/triad_sm.jl sufficiency_score() (interleaved calib/held split by seed, least-squares 1-D fit Delta_y~a*attr+b on calib, epsilon = max(0.5, 0.10*heldout_spread)); assembled in triad_extra_dict(). The new §3 (sec:triad) admits the fraction-within-tolerance [0,1] special case of the held-out predictive test, which this held-out do(u) fit-then-predict estimator satisfies, so it matches. (Detail: epsilon is a self-scaled band 0.10*spread; the [-1,1] correlation form is available but this method reports the [0,1] hit fraction.)
how it's measured — call stack:
  1. sufficiency_score: fit Δy=a·attr+b on a calib half, predict held-out within ε — triad_sm.jl:111
  2. assemble the F∧S∧M triad record (calls sufficiency_score) — triad_sm.jl:173
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242
M
minimality		\(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\)Ratio of the number of oracle causal movers (causes with nonzero true Delta_y) to the number of cells the method names above 1e-6 of its own max attribution; null when the oracle finds no mover or the method names nothing.✓ matches common/triad_sm.jl minimality_score() with name_frac=1e-6, mover_floor=0.0; the above-threshold named set is used (topk kept but not passed), see triad_extra_dict(). RISE names a dense mask, so |U_hat| >> |U*| and M is small (e.g. 2/26=0.077 in the air_raid content record) -- the intended discrimination of the above-threshold reading. This is the paper's M = |U*|/|U_hat| in (0,1] (standardized everywhere), so it matches the new §3. (Detail: |U*| is taken as all oracle movers with Delta_y>0, an upper bound on the strictly-smallest reproducing subset, so M can be optimistic.)
how it's measured — call stack:
  1. minimality_score: |U*| oracle movers / |U_hat| named cells — triad_sm.jl:57
  2. assemble the triad record (calls minimality_score) — triad_sm.jl:178
  3. true causal effect Δy(u) by bit-exact re-run — oracle_intervene.jl:242

✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).

In the audit

This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 44 of the 44 scored games (88 committed §R records), not the single example shown above. Tradition: gradient. The example figure (Pong) is one of those records.

0.543F — faithfulness vs oracle (mean over 44 games, ±0.061 CI95)
0.563S — sufficiency: held-out predictive (n/a where the paper does not define this axis)
0.356M — minimality: true-minimal-set / named-set (n/a otherwise)
88committed records aggregated
0.00human-plausibility proxy

F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.

Source: leaderboard.json · the whole leaderboard is on the methods page and the Paper 2 audit.

Results per game

This method's faithfulness on each of the 44 scored games (all-regime F, and the content vs position split). Click a header to sort. Every number is read from site_data.json (methods.rise.per_game).

GameF (all)content-Fposition-FSMNote
Montezuma's Revenge0.8750.9640.7870.7810.400recovers the true causes
Fishing Derby0.8670.9200.8150.6000.278recovers the true causes
Space Invaders0.8310.9660.6950.7780.294
Bowling0.7900.7740.8060.8570.225holds up on position
Gopher0.7650.7460.7850.9290.287holds up on position
Private Eye0.7180.7130.7230.9600.230holds up on position
Skiing0.7000.7810.6200.6000.417
Phoenix0.6840.8470.5200.1920.292
Krull0.6610.9580.3640.5380.190
Bank Heist0.6480.8520.4430.8330.500
Pacman0.6450.7580.5320.4440.344
Jamesbond0.6440.6850.6030.8930.288
Riverraid0.6290.9730.2860.5650.432
Ice Hockey0.6280.7200.5360.5910.500
Q*bert0.6200.9760.2640.4520.188
Frostbite0.6190.5340.7040.2500.395holds up on position
Boxing0.6110.7980.4240.5830.341
Alien0.6110.7430.4790.5710.333
Demon Attack0.6080.5750.6410.8120.350holds up on position
Pitfall0.6010.5520.6490.8820.344holds up on position
Atlantis0.5940.5280.6600.4500.417holds up on position
Video Pinball0.5690.7310.4060.5000.500
Breakout0.5650.7270.4030.5500.361
Kung-Fu Master0.5630.7210.4050.8000.361
Pong0.5340.6100.4580.3890.281
Road Runner0.5100.1710.8490.3680.167holds up on position
Asterix0.5050.5070.5030.5000.417
Kangaroo0.4900.9790.0000.5000.300position gradient vanishes
Tennis0.4770.9540.0000.4060.350position gradient vanishes
Double Dunk0.4720.6980.2460.5380.500
Air Raid0.4410.3810.5010.6070.423holds up on position
Name This Game0.4080.7490.0670.4170.450
Enduro0.4050.7950.0150.0000.500position gradient vanishes
Ms. Pac-Man0.3950.7900.0000.4750.342position gradient vanishes
Assault0.3930.1000.6860.9050.263holds up on position
Yars' Revenge0.3780.7560.0000.4640.211position gradient vanishes
Seaquest0.3460.0000.6930.4790.413holds up on position
Centipede0.3390.6780.0000.4500.472position gradient vanishes
Berzerk0.3350.6700.0000.4810.260position gradient vanishes
Carnival0.3300.6610.0000.4740.417position gradient vanishes
Chopper Command0.3180.0000.6360.4520.400holds up on position
Hero0.2910.0910.4910.4170.265
Freeway0.2590.4710.0470.5670.500
Venture0.2430.4870.0000.4740.500position gradient vanishes

Call stack — how it runs on jutari / jaxtari

The path from this method's runner (main) into the bit-exact VCS substrate, and the computation it involves. Every step links to the exact source on main. From callstack_phaseB.json.

Applied on the substrate

  1. env reset + per-frame step of the bit-exact VCS (masked re-runs step the horizon) — env_reset! / env_step! jutari
  2. shared gameplay-state testbed — build_shared_testbed jutari

The computation (intervention path)

  1. masked forward: drop (mask to 0) the not-kept cells via the oracle's occlude!, step the horizon, read y — masked_read_y -> intervene_ram! jutari intervene_ram! call at rise.jl:131 -> jutari_oracle.jl:174
  2. draw N random Bernoulli(keep_prob) masks and collect the masked outputs — rise_sample runner
  3. RISE importance: output-weighted mean mask (1/(N*p)) * sum_n y_n * m_n[i] — rise_importance runner
  4. snapshot RAM+screen after each masked re-run — snapshot jutari

Scored against the exact intervention oracle Δy(u): run_intervention.

Implementation
tools/xai_study/phaseB_attribution/rise.jl
Reference
Petsiuk et al. 2018
Record
rise_pong_content.json
All records
phaseB_attribution/out

The figure is generated from the committed record by docs/gen_method_figures.py; the game frame and each RAM cell's screen footprint are produced by render_scenes.jl / cell_footprints.jl.