Ground Truth ROMs

Explainable-AI methods are usually impossible to check: nobody knows the true inner workings of a deep network, so an explanation can be convincing and wrong with no way to tell. We swap in a 1977 Atari 2600 game console instead — a real, complex computer whose every wire and byte is known. For any game we can compute the true cause of any pixel by literally changing a byte and re-running the machine bit-for-bit. That true answer is the ground truth every method is graded against. This page lists, game by game, what ground truth we have and where it comes from.

Three kinds of ground truth: T1, T2, T3

Not all ground truth is the same. We separate what the machine can tell us by itself from what only a human can supply.

T1 — the causal map (what actually drives the picture). Take one memory byte, change its value, re-run the console bit-for-bit, and measure how the screen changes. Do that for every candidate byte and you have the exact list of true causes of any output. Nothing external is needed — the machine answers for itself. Available for all 64 games.
T2 — the wiring (which part reads or writes which). The program’s read/write data-flow graph, read straight off the machine: which registers and memory cells feed which others. Also exact, also all 64 games.
T3 — the human meaning (“this byte is the ball’s x-position”). This is different in kind. The name of a byte is not inside the machine; it is a human fact about what the game means. We import names from two public projects and then verify each one causally — set the byte, re-render, and confirm the named object really moves where predicted. We never invent a meaning; we only confirm a supplied one. T3 exists for 54 of the 64 games.

How we grade an explanation: F, S, M

An explanation is not “right” just because it looks convincing. We ask three separate things, and an explanation only counts as right when it clears all three.

F — Faithfulness: does it name the real causes? We compare what the method highlights against T1’s true causal list. High F means it points at the bytes that genuinely drive the output, not at bytes that merely look related.
S — Sufficiency: could you rebuild the behaviour from it? A real explanation should predict what happens under changes it has not seen. We test it on held-out interventions. High S means it captured the mechanism, not just a single snapshot.
M — Minimality: is it the smallest true set? Listing every byte is perfectly faithful but useless. M rewards naming just the handful of bytes that truly matter. “Record everything” scores near zero on M.

On this machine all three are computed against the oracle (T1/T2), never against another method — which is what keeps the grade honest. The per-method F/S/M scores live on the P2 Methods page; this page shows, per game, what those measures can be computed against.

Where the labels come from

The T3 names are external, so we are explicit about their source. We import from AtariARI (Anand et al., 2019 — RAM annotations for 22 games) and OCAtari (Delfosse et al., 2024 — 54 games, with the render-time offset correction that aligns a RAM coordinate to where the sprite is actually drawn). Every label is then verified on our bit-exact machine before use, so a name is only kept if changing that byte demonstrably moves the named object. A method that “matches” a label is therefore aligning to a meaning we supplied — it did not discover it.

Per-game ground truth

All 64 bit-exact ROMs (54 carry a T3 label, 10 are T1/T2 only), ordered by how much ground truth the shared analysis frame supports (position-regime games first). Screenshots are the actual analysis frame the ground truth is computed on. Colour: green = available, amber = present but limited at this frame, red = not available. The 10 games with no external label carry exact T1/T2 ground truth but are held out of the label-dependent (T3) study.

Of the 54 labelled ROMs, 42 make up the scored battery; the other 12 are excluded (dimmed and tagged below). We exclude a game for one of two honest reasons. Eight fail the cause-density gate: at the shared frame the chosen output has almost no true causes, so there is nothing for a method to be right or wrong about. Four have no sprite that moves under intervention — their verified labels are static things like a score counter, so there is no position for a position method to recover; three games fail both tests. Running interpretability methods on such a frame cannot measure faithfulness — it would only add noise to the leaderboard — so we leave those games out of scoring while keeping their exact T1/T2 ground truth on record. Every game in the scored battery, by contrast, both passes the gate and carries a moving sprite, so it contributes to the all-regime and the position regime alike.

64bit-exact ROMs (T1/T2)
54carry a T3 label
42in the scored battery
12excluded (degenerate / static)
42in the position regime
Air Raid screen

Air Raid OCAtariscored

  • Faithfulness (F): 14 true causes at the analysis frame — scorable
  • T3: 9 of 13 imported labels verified causally · 8 move a sprite
  • Position regime: yes — e.g. b1.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Alien screen

Alien OCAtariscored

  • Faithfulness (F): 6 true causes at the analysis frame — scorable
  • T3: 3 of 6 imported labels verified causally · 3 move a sprite
  • Position regime: yes — e.g. player.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Assault screen

Assault OCAtariscored

  • Faithfulness (F): 16 true causes at the analysis frame — scorable
  • T3: 14 of 20 imported labels verified causally · 6 move a sprite
  • Position regime: yes — e.g. enemy_x_part_1
  • Per-game results → every XAI method's faithfulness on this ROM
Atlantis screen

Atlantis OCAtariscored

  • Faithfulness (F): 12 true causes at the analysis frame — scorable
  • T3: 3 of 9 imported labels verified causally · 3 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Bank Heist screen

Bank Heist OCAtariscored

  • Faithfulness (F): 8 true causes at the analysis frame — scorable
  • T3: 3 of 5 imported labels verified causally · 3 move a sprite
  • Position regime: yes — e.g. player.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Beam Rider screen

Beam Rider OCAtariscored

  • Faithfulness (F): 4 true causes at the analysis frame — scorable
  • T3: 2 of 5 imported labels verified causally · 1 move a sprite
  • Position regime: yes — e.g. player.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Berzerk screen

Berzerk AtariARI + OCAtariscored

  • Faithfulness (F): 23 true causes at the analysis frame — scorable
  • T3: 13 of 26 imported labels verified causally · 7 move a sprite
  • Position regime: yes — e.g. enemy_robots_x[0]
  • Per-game results → every XAI method's faithfulness on this ROM
Bowling screen

Bowling AtariARI + OCAtariscored

  • Faithfulness (F): 14 true causes at the analysis frame — scorable
  • T3: 8 of 20 imported labels verified causally · 7 move a sprite
  • Position regime: yes — e.g. ball_x
  • Per-game results → every XAI method's faithfulness on this ROM
Boxing screen

Boxing AtariARI + OCAtariscored

  • Faithfulness (F): 13 true causes at the analysis frame — scorable
  • T3: 8 of 11 imported labels verified causally · 5 move a sprite
  • Position regime: yes — e.g. enemy_x
  • Per-game results → every XAI method's faithfulness on this ROM
Breakout screen

Breakout AtariARI + OCAtariscored

  • Faithfulness (F): 11 true causes at the analysis frame — scorable
  • T3: 7 of 9 imported labels verified causally · 3 move a sprite
  • Position regime: yes — e.g. player_x
  • Per-game results → every XAI method's faithfulness on this ROM
Carnival screen

Carnival OCAtariscored

  • Faithfulness (F): 26 true causes at the analysis frame — scorable
  • T3: 17 of 18 imported labels verified causally · 12 move a sprite
  • Position regime: yes — e.g. missile.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Centipede screen

Centipede OCAtariscored

  • Faithfulness (F): 14 true causes at the analysis frame — scorable
  • T3: 8 of 9 imported labels verified causally · 7 move a sprite
  • Position regime: yes — e.g. player.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Chopper Command screen

Chopper Command OCAtariscored

  • Faithfulness (F): 16 true causes at the analysis frame — scorable
  • T3: 13 of 20 imported labels verified causally · 9 move a sprite
  • Position regime: yes — e.g. player.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Demon Attack screen

Demon Attack AtariARI + OCAtariscored

  • Faithfulness (F): 18 true causes at the analysis frame — scorable
  • T3: 11 of 15 imported labels verified causally · 10 move a sprite
  • Position regime: yes — e.g. enemy_projectile_y
  • Per-game results → every XAI method's faithfulness on this ROM
Double Dunk screen

Double Dunk OCAtariscored

  • Faithfulness (F): 22 true causes at the analysis frame — scorable
  • T3: 10 of 12 imported labels verified causally · 8 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Fishing Derby screen

Fishing Derby OCAtariscored

  • Faithfulness (F): 8 true causes at the analysis frame — scorable
  • T3: 7 of 9 imported labels verified causally · 5 move a sprite
  • Position regime: yes — e.g. p1s.hook_position
  • Per-game results → every XAI method's faithfulness on this ROM
Freeway screen

Freeway AtariARI + OCAtariscored

  • Faithfulness (F): 26 true causes at the analysis frame — scorable
  • T3: 14 of 14 imported labels verified causally · 7 move a sprite
  • Position regime: yes — e.g. c2.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Frostbite screen

Frostbite AtariARI + OCAtariscored

  • Faithfulness (F): 23 true causes at the analysis frame — scorable
  • T3: 12 of 19 imported labels verified causally · 9 move a sprite
  • Position regime: yes — e.g. fourth_row_iceflow_x
  • Per-game results → every XAI method's faithfulness on this ROM
Gopher screen

Gopher OCAtariscored

  • Faithfulness (F): 21 true causes at the analysis frame — scorable
  • T3: 14 of 20 imported labels verified causally · 14 move a sprite
  • Position regime: yes — e.g. gopher.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Hero screen

Hero AtariARI + OCAtariscored

  • Faithfulness (F): 17 true causes at the analysis frame — scorable
  • T3: 11 of 17 imported labels verified causally · 8 move a sprite
  • Position regime: yes — e.g. destructible_wall.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Ice Hockey screen

Ice Hockey OCAtariscored

  • Faithfulness (F): 17 true causes at the analysis frame — scorable
  • T3: 10 of 10 imported labels verified causally · 10 move a sprite
  • Position regime: yes — e.g. ball.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Jamesbond screen

Jamesbond OCAtariscored

  • Faithfulness (F): 13 true causes at the analysis frame — scorable
  • T3: 5 of 13 imported labels verified causally · 4 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Kangaroo screen

Kangaroo OCAtariscored

  • Faithfulness (F): 16 true causes at the analysis frame — scorable
  • T3: 9 of 15 imported labels verified causally · 5 move a sprite
  • Position regime: yes — e.g. child.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Krull screen

Krull OCAtariscored

  • Faithfulness (F): 17 true causes at the analysis frame — scorable
  • T3: 11 of 25 imported labels verified causally · 5 move a sprite
  • Position regime: yes — e.g. player.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Kung-Fu Master screen

Kung-Fu Master OCAtariscored

  • Faithfulness (F): 11 true causes at the analysis frame — scorable
  • T3: 5 of 9 imported labels verified causally · 5 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Montezuma's Revenge screen

Montezuma's Revenge AtariARI + OCAtariscored

  • Faithfulness (F): 22 true causes at the analysis frame — scorable
  • T3: 15 of 15 imported labels verified causally · 9 move a sprite
  • Position regime: yes — e.g. enemy_skull_x
  • Per-game results → every XAI method's faithfulness on this ROM
Ms. Pac-Man screen

Ms. Pac-Man AtariARI + OCAtariscored

  • Faithfulness (F): 24 true causes at the analysis frame — scorable
  • T3: 15 of 19 imported labels verified causally · 2 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Name This Game screen

Name This Game OCAtariscored

  • Faithfulness (F): 7 true causes at the analysis frame — scorable
  • T3: 4 of 5 imported labels verified causally · 4 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Pacman screen

Pacman OCAtariscored

  • Faithfulness (F): 9 true causes at the analysis frame — scorable
  • T3: 8 of 8 imported labels verified causally · 5 move a sprite
  • Position regime: yes — e.g. player.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Phoenix screen

Phoenix OCAtariscored

  • Faithfulness (F): 12 true causes at the analysis frame — scorable
  • T3: 8 of 12 imported labels verified causally · 5 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Pitfall screen

Pitfall AtariARI + OCAtariscored

  • Faithfulness (F): 19 true causes at the analysis frame — scorable
  • T3: 10 of 16 imported labels verified causally · 7 move a sprite
  • Position regime: yes — e.g. enemy_logs_x
  • Per-game results → every XAI method's faithfulness on this ROM
Pong screen

Pong AtariARI + OCAtariscored

  • Faithfulness (F): 6 true causes at the analysis frame — scorable
  • T3: 6 of 8 imported labels verified causally · 4 move a sprite
  • Position regime: yes — e.g. ball_x
  • Per-game results → every XAI method's faithfulness on this ROM
Private Eye screen

Private Eye AtariARI + OCAtariscored

  • Faithfulness (F): 20 true causes at the analysis frame — scorable
  • T3: 12 of 24 imported labels verified causally · 9 move a sprite
  • Position regime: yes — e.g. car.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Q*bert screen

Q*bert AtariARI + OCAtariscored

  • Faithfulness (F): 13 true causes at the analysis frame — scorable
  • T3: 6 of 20 imported labels verified causally · 1 move a sprite
  • Position regime: yes — e.g. player_y
  • Per-game results → every XAI method's faithfulness on this ROM
Riverraid screen

Riverraid AtariARI + OCAtariscored

  • Faithfulness (F): 33 true causes at the analysis frame — scorable
  • T3: 15 of 22 imported labels verified causally · 12 move a sprite
  • Position regime: yes — e.g. player_x
  • Per-game results → every XAI method's faithfulness on this ROM
Road Runner screen

Road Runner OCAtariscored

  • Faithfulness (F): 11 true causes at the analysis frame — scorable
  • T3: 7 of 18 imported labels verified causally · 6 move a sprite
  • Position regime: yes — e.g. enemy.xy
  • Per-game results → every XAI method's faithfulness on this ROM
Seaquest screen

Seaquest AtariARI + OCAtariscored

  • Faithfulness (F): 19 true causes at the analysis frame — scorable
  • T3: 11 of 23 imported labels verified causally · 9 move a sprite
  • Position regime: yes — e.g. diver_or_enemy_missile_x
  • Per-game results → every XAI method's faithfulness on this ROM
Space Invaders screen

Space Invaders AtariARI + OCAtariscored

  • Faithfulness (F): 18 true causes at the analysis frame — scorable
  • T3: 8 of 17 imported labels verified causally · 7 move a sprite
  • Position regime: yes — e.g. enemies_x
  • Per-game results → every XAI method's faithfulness on this ROM
Tennis screen

Tennis AtariARI + OCAtariscored

  • Faithfulness (F): 19 true causes at the analysis frame — scorable
  • T3: 11 of 15 imported labels verified causally · 11 move a sprite
  • Position regime: yes — e.g. ball_x
  • Per-game results → every XAI method's faithfulness on this ROM
Venture screen

Venture AtariARI + OCAtariscored

  • Faithfulness (F): 29 true causes at the analysis frame — scorable
  • T3: 16 of 18 imported labels verified causally · 8 move a sprite
  • Position regime: yes — e.g. player_x
  • Per-game results → every XAI method's faithfulness on this ROM
Video Pinball screen

Video Pinball AtariARI + OCAtariscored

  • Faithfulness (F): 11 true causes at the analysis frame — scorable
  • T3: 7 of 7 imported labels verified causally · 7 move a sprite
  • Position regime: yes — e.g. ball_x
  • Per-game results → every XAI method's faithfulness on this ROM
Yars' Revenge screen

Yars' Revenge AtariARI + OCAtariscored

  • Faithfulness (F): 11 true causes at the analysis frame — scorable
  • T3: 4 of 13 imported labels verified causally · 3 move a sprite
  • Position regime: yes
  • Per-game results → every XAI method's faithfulness on this ROM
Amidar screen

Amidar OCAtariexcluded

  • Faithfulness (F): 6 true causes at the analysis frame — scorable
  • T3: 0 of 0 imported labels verified causally · 0 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: no moving sprite
Asterix screen

Asterix OCAtariexcluded

  • Faithfulness (F): 12 true causes at the analysis frame — scorable
  • T3: 0 of 9 imported labels verified causally · 0 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: no moving sprite
Robotank screen

Robotank OCAtariexcluded

  • Faithfulness (F): 5 true causes at the analysis frame — scorable
  • T3: 0 of 0 imported labels verified causally · 0 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: no moving sprite
Up'n Down screen

Up'n Down OCAtariexcluded

  • Faithfulness (F): 7 true causes at the analysis frame — scorable
  • T3: 0 of 0 imported labels verified causally · 0 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: no moving sprite
Asteroids screen

Asteroids AtariARI + OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 0 of 47 imported labels verified causally · 0 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected+no moving sprite
Battle Zone screen

Battle Zone OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 1 of 6 imported labels verified causally · 0 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected+no moving sprite
Crazy Climber screen

Crazy Climber OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 2 of 9 imported labels verified causally · 2 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected
Enduro screen

Enduro OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 2 of 3 imported labels verified causally · 2 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected
Pooyan screen

Pooyan OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 4 of 11 imported labels verified causally · 2 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected
Skiing screen

Skiing OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 8 of 9 imported labels verified causally · 4 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected
Star Gunner screen

Star Gunner OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 0 of 7 imported labels verified causally · 0 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected+no moving sprite
Time Pilot screen

Time Pilot OCAtariexcluded

  • Faithfulness (F): too few strong causes at this frame — held out of scoring
  • T3: 2 of 5 imported labels verified causally · 1 move a sprite
  • Position regime: no moving tracked sprite at this frame
  • Excluded from the scored battery: gate-rejected
Defender screen

Defender no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Elevator Action screen

Elevator Action no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Gravitar screen

Gravitar no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Journey Escape screen

Journey Escape no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Solaris screen

Solaris no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Surround screen

Surround no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Tutankham screen

Tutankham no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Videochess screen

Videochess no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Wizard Of Wor screen

Wizard Of Wor no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)
Zaxxon screen

Zaxxon no external label

  • Faithfulness (F): not in the shared testbed
  • T3: none — bit-exact T1/T2 only (no external label source)