A reviewer should be able to confirm what every conformance test computes without reading the whole codebase. This page walks the function stack of each harness — from the test assertion, through the replay driver, down into the emulator core where the compared bytes are produced — with a line-anchored link at every step. The documentation in the code explains how each function works; this page explains what is being checked and why it is evidence.
The oracle pipeline · PXC1 (RAM/CPU) · PXC2 (dual-port) · PXC-S (screen) · PXC4 (6502) · the emulator core
Every test compares one of our ports against xitari, the external C++ reference emulator (google-deepmind/xitari). The comparison runs in two stages. First, xitari is driven by a fixed action stream and its per-frame state is dumped to a JSONL trace. Then a port is driven by the same action stream and its state is diffed against that trace, frame by frame.
fixed action stream
│
▼
trace_dump.cpp · main ──▶ JSONL trace (per frame: ram, optional cpu / screen)
(drives the real xitari C++ emulator: ale.act, ale.getRAM, ale.getScreen)
│ committed under tools/fixtures/
▼
check_trace.py · check_trace ──▶ steps a port with the same actions, diffs each frame
│
▼
PXC1 (RAM+CPU) PXC2 (port≡port) PXC-S (screen) PXC4 (6502 ISA)The JSONL record for one frame holds the absolute frame index, the action applied, the
128 B of RIOT RAM as 256 hex chars, and — when the relevant flag is set — the six CPU
registers and the 210×160 framebuffer. The RAM bytes are read out of xitari at
trace_dump.cpp:256 · ale.getRAM() and hex-encoded at trace_dump.cpp:67 · hex_encode; the CPU registers are tapped through a
friend class CpuDebug at trace_dump.cpp:54 · CpuDebug so xitari's headers are never modified.
Does a port reproduce xitari's RAM (and CPU registers) frame-for-frame?
The test loads an xitari trace, replays the same actions through the port, and asserts the port's 128 B RIOT RAM hex-matches the trace at every frame. A mismatch raises at the first diverging frame with a byte-level diff.
| Step in the call stack | What is actually computed |
|---|---|
test_pxc1_conformance.py:61 · test_jaxtari_matches_xitari… | Entry point. Builds the port from the ROM, replays the trace, asserts all 10 frames matched (L80 · assert matched == 10). |
check_trace.py:123 · check_trace(rom, trace) | Constructs the environment, boots it the same way xitari boots (60 NOOP + 4 RESET), then loops the trace's actions. |
check_trace.py:175 · ram_hex != ref['ram'] | The core check. Per frame: encode the port RAM, compare to the reference hex; on mismatch emit a byte diff and raise ConformanceError. |
stella_environment.py:375 · step(action) | Applies the action and runs one frame to the next VSYNC — the port advance. |
stella_environment.py:492 · get_ram() | Returns the 128 B RIOT RAM (console.bus.ram) — the exact bytes compared. |
How to read it. The assertion that matters is the hex-string compare at
check_trace.py:175: the port's RAM is read at get_ram(), hex-encoded, and compared to the trace's
ram field. If they differ the harness reports the frame and the differing byte
addresses, then raises. The Julia port is checked by the mirror harness check_trace.jl:61 · check_trace.
Do the JAX port and the Julia port diverge from xitari identically?
This is the claim featured on the front page. Two independently written ports — one in JAX, one in Julia — are required to produce byte-identical RIOT RAM at every frame. Where they still differ from xitari, they must differ in the same bytes by the same amount. Two separate implementations agreeing to the bit is strong evidence of a shared, correct mechanism rather than two coincidental bugs.
| Step in the call stack | What is actually computed |
|---|---|
test_pxc2…py:281 · test_jaxtari_matches_jutari_per_frame_ram | The headline cross-check. Steps jaxtari live; loads the jutari fixture; compares RAM per frame. |
test_pxc2…py:240 · _jaxtari_ram_per_frame | Boots jaxtari (60 NOOP + 4 RESET) and records the 128 B RAM after every action — the live side of the comparison. |
test_pxc2…py:299 · if jax_ram != jul_ram | The core check. Byte-equality of the two ports' RAM at frame i; on mismatch builds a per-address diff and fails (L306). |
test_pxc2…py:310 · …divergence_pattern_unchanged | Regression guard: the count of bytes still differing from xitari must equal the value pinned per ROM (all 0 today). |
jutari_trace_dump.jl:132 · main | Produces the jutari fixture: same boot, same actions, records RAM per frame — the Julia side, generated once and committed. |
How to read it. The headline test runs jaxtari live and loads the jutari
result from a committed fixture trace (produced by jutari_trace_dump.jl). The compare at
L299 is a plain byte-equality of the two ports' RAM per frame; on any mismatch it
fails at L306 with the frame index and differing addresses. A third test
(the third test) locks in the residual divergence-from-xitari count per ROM, so a real
emulation fix must move both ports in the same commit — they can never drift apart
silently.
Does the rendered 210×160 framebuffer match, pixel for pixel?
RAM equality does not imply screen equality — the TIA renders pixels from register writes whose timing must be exact. PXC-S diffs the framebuffer per frame, both xitari↔jutari (from fixtures) and xitari↔jaxtari (rendered live), counting differing pixels.
| Step in the call stack | What is actually computed |
|---|---|
test_screen_conformance.py:219 · test_jutari_screen_matches_xitari | xitari fixture vs jutari fixture, per frame. |
test_screen_conformance.py:235 · test_jaxtari_screen_matches_xitari | xitari fixture vs jaxtari rendered live. |
test_screen_conformance.py:213 · _per_frame_diffs | The core check. Per frame: (a[i] != b[i]).sum() — the count of differing palette-index pixels; the test asserts the worst is ≤ the pin (L227). |
stella_environment.py:473 · get_screen() | Crops console.bus.tia.framebuffer to the visible rows — the rendered output being diffed. |
How to read it. _per_frame_diffs counts differing pixels per frame; each test asserts
the worst frame is within a pinned threshold (max_screen_diff — set to 0 once a game is
pixel-exact). The live framebuffer comes from get_screen(), which crops the TIA's internal
buffer to the visible window.
Is the CPU core a correct 6502, independent of any game?
This runs Klaus Dormann's widely used 6502 functional test — an external ROM that exercises every opcode, addressing mode and flag — on a flat 64 KB memory, bypassing the TIA, RIOT and cartridge entirely. Success is a hard, unambiguous signal: the program only reaches its success address if every sub-test passed.
| Step in the call stack | What is actually computed |
|---|---|
test_pxc4_klaus_dormann.py · test_klaus_dormann…passes | Loads the ROM into flat 64 KB memory, sets PC to 0x0400, steps until success or stuck. |
test_pxc4_klaus_dormann.py:99 · _step_inner(state, memory) | Executes one 6502 instruction with no TIA/RIOT side effects — pure ISA. |
test_pxc4_klaus_dormann.py:102 · cur_pc == KLAUS_SUCCESS_PC | The pass condition. PC reaches 0x3469 (L49) ⇒ all opcodes/flags correct. |
m6502.py:410 · _step_inner | The shared instruction decoder/executor used by both the game path and this test. |
How to read it. The loop steps the CPU at L99 and watches the program
counter: reaching KLAUS_SUCCESS_PC = 0x3469 (test_pxc4_klaus_dormann.py:49) means every sub-test
passed; a PC stuck on a trap loop means a specific opcode is wrong, and the failure names the
address to disassemble.
Each harness above bottoms out in the same port internals. This is the stack a single
step() runs through, so a reviewer can follow a compared byte back to the CPU
instruction and TIA tick that produced it.
| Step in the call stack | What is actually computed |
|---|---|
stella_environment.py:87 · reset(...) → _boot_burn | Resets the console and burns the boot frames (60 NOOP + 4 RESET + any per-game starting actions) so the port starts where xitari starts. |
stella_environment.py:375 · step(action) | Applies the action to the bus, then runs one video frame. |
console.py:97 · run_until_frame | Steps the CPU (up to a 25,000-instruction budget) until the TIA frame counter advances — one frame, including the partial/grey-frame case. |
m6502.py:365 · step → _step_inner + _tia_post_step | Fetch/decode/execute one 6502 instruction, then advance the TIA and RIOT by the cycles it consumed and resolve any WSYNC stall. |
console.py:49 · console_reset | Reads the reset vector at $FFFC/$FFFD into PC — the boot entry the whole run hangs off. |
The Julia port mirrors this structure (env_step! / get_ram /
run_until_frame); its fixtures are produced by jutari_trace_dump.jl. Because the two ports
share neither code nor language, PXC2 agreement is independent corroboration of this stack.
# regenerate an xitari reference trace
./tools/trace_dump --rom xitari/roms/pong.bin \
--actions tools/fixtures/actions/pong_noop_10.txt > /tmp/pong.jsonl
# replay it against each port
python tools/check_trace.py --rom xitari/roms/pong.bin --trace /tmp/pong.jsonl
julia --project=jutari tools/check_trace.jl --rom xitari/roms/pong.bin --trace /tmp/pong.jsonl
# run the gates (PXC1/PXC2/PXC-S/PXC4 live)
cd jaxtari && .venv/bin/python -m pytest -q tests/test_pxc1_conformance.py \
tests/test_pxc2_jaxtari_vs_jutari.py tests/test_pxc4_klaus_dormann.py
jaxtari/.venv/bin/pytest jaxtari/tests/test_screen_conformance.py # ~23 min