The Ghost, the Ladder, and the Cartridge

This morning was the baseline and this afternoon was the core swap. The evening went three different directions at once, and every one of them ended the same way: an instrument said something my intuition would not have.

The ghost of level two

Playing Celeste on the console tonight I noticed a second Madeline. She stood embedded waist-deep in the floor at the bottom of level two, just above the spawn point, and she never moved - but she turned to face whichever direction I pressed. A ghost with my controls.

That description is forensically precise, it turns out. An object stuck inside solid ground can never move (the collision check rejects every step), but the player's input code still sets its facing flag every frame. So the ghost had to be a duplicate player object. The question was where a second player could possibly come from.

First suspect: the map. I decoded the cart PNG's steganography directly - each byte of ROM hides in the two low bits of each channel, assembled A,R,G,B - and counted tiles with value 1 (the spawn marker) per level:

level 1: spawn cells [(1, 12)]
level 2: spawn cells [(1, 14)]
level 3: spawn cells [(1, 13)]

Exactly one each. The map was innocent. Which left the object lifecycle, and there the trail got warm fast. When Celeste changes rooms it wipes the old room's objects like this:

foreach(objects, destroy_object)

Deleting the current element from a list you are iterating is the classic trap, and PICO-8's all() iterator is specifically built to survive it. Mine was not. This was my all():

-- the naive iterator: skips an element after every deletion
function all(t)
  local i = 0
  return function() i = i + 1 return t[i] end
end

Delete element 1 while standing on it and everything shifts left; the i = i + 1 then steps OVER the new element 1. Net effect: that wipe destroys only every second object, and half the previous room follows you into the next one. One of the survivors was a player object, entombed in level two's floor, faithfully reading my inputs forever.

The fix is the deletion-safe iterator - advance only while the current slot still holds the element you just returned:

function all(t)
  if t == nil then return function() end end
  local i, prev = 1, nil
  return function()
    if t[i] == prev then i = i + 1 end
    while t[i] == nil and i <= #t do i = i + 1 end
    prev = t[i]
    return t[i]
  end
end

Discipline before the fix, though: I wrote a compatibility cart first - eight checks covering delete-current, delete-next, delete-previous and add-during-iteration - and ran it through my licensed PICO-8 to pin the real machine's behaviour. All eight predictions held, the cart joined the corpus as ground truth, and only then did the one-line swap go into every runtime. A quiet bonus: any object that self-destroys mid-update (smoke puffs, dead particles) had been stealing an update frame from its list neighbour all along. That whole class of jank died with the ghost.

The clock ladder

The console's HDMI path is an SPI link to an RP2040 running PicoDVI, and that link has lived at 6.4 MHz for a week - the highest clock that looked clean over jumper wires. Tonight I re-walked the whole ladder, because two things had changed since the last attempt: the wiring got a proper ground return, and the protocol grew per-line checksums, which means corruption is now counted instead of squinted at. The bridge draws a status cell that is green only if the last frame arrived with zero bad lines, redrawn every frame - a live corruption meter.

The clock rungs come from a PLL, not the default power-of-two ladder:

SPI clock = 192 MHz VCO / DIV / 2

DIV15 -> 6.40 MHz      DIV13 -> 7.38 MHz      DIV12 -> 8.00 MHz

And the link rate scales linearly with the clock, so each rung has a prediction: 45 fps at 6.4 gives 45 x 7.38/6.4 = 51.9 at the next rung. Measured: a steady 52. The instrument-read results:

The 8 MHz rung is the interesting one. The full 60 fps ships out of the STM32 - and the status cell sits solid red, meaning every single frame reaches the RP2040 with at least one corrupted line. The checksums drop bad lines instead of tearing, so it is playable, with a permanent faint shimmer on motion. Better grounds and checksum armour reduced the blast radius; they did not fix the physics of an 8 MHz square wave on flying jumper leads. So the link now lives at 7.38 MHz: clean, 52 fps, and a one-line rollback if I am ever wrong about it. A true 60 waits for the PCB.

The cartridge learns to save

The physical cartridge - a 16 MB SPI NOR flash in a printed shell - has been running a v1 format: one manifest, one game blob, burn and play. Tonight it grew up. The v2 format is a section table:

Two design decisions matter more than the rest.

The cart file stays verbatim. The first v2 burn attempt taught me this: my games download as .p8.png files, and the console converts them for whichever Lua core is running at play time. If the burn stored the converted output, the cart would be married to one core. So the original file is the section of record, byte for byte, and everything else on the cart is derived, disposable acceleration.

Saves live at a fixed address. The save region is pinned to the top two 4 KiB sectors of the flash, so re-burning a game does not move it. Whether it survives a re-burn is decided by comparing section checksums at plan time - the same game byte-for-byte keeps its saves, a different game wipes them:

let same_game = prev.is_some_and(|p| {
    sections[..count]
        .iter()
        .all(|s| p.section(s.kind).map(|q| q.crc) == Some(s.crc))
});

Inside that region, saves are an append-only record log, because NOR flash has opinions: programming can only clear bits, erasing works in whole 4 KiB sectors, and a sector erase can stall the chip for 400 ms - against an audio buffer that holds about 12. So every save appends one 512-byte record (two whole flash pages, so no write ever crosses a page boundary), and a sector is erased only when the ring wraps back onto it:

0   magic "RNSV"
4   seq: u32         monotonic - the latest valid one wins
8   generation: u32  which burn wrote it
12  crc: u32         of the payload
16  payload[256]     the cartdata image (64 slots x 32 bits)

The wear maths is why a log, not a rewrite: 8 records per sector on a 2-sector ring means each sector erases once per 16 saves. At the rated 100,000 erase cycles that is 1.6 million saves per sector - saving once every five seconds, nonstop, is 92 days of abuse; once a minute is three years of continuous play. The cart will die of something else.

I tested the log against a strict RAM model of the flash - one that panics if a write tries to SET a bit or cross a page - and it earned its keep on the first run: after a torn record (the pull-the-cart- mid-save case) the write cursor landed on dirty flash, which NOR cannot rewrite. The model refused, the test failed, and the fix (a dirty mid-sector slot forfeits the rest of its sector) went in before any real cartridge ever saw the bug. The same evening also caught my CI script lying - every gate piped through tail -1, which reports tail's exit code, so a broken firmware build still printed green. It does not any more.

Then the field test, live on the console, straight from the debug log:

cart: flashing 38104 bytes...
cart: v2 burn gen 4 - save region wiped (new or changed game)
cart: flash VERIFIED
cart: recognised "Breakout Hero" (38104 bytes, v2 gen 4)
cart: save log empty (fresh cartdata)
cart: game loaded off the cart (38104 bytes, v2 gen 4)

Burn, badge, mount, load, play. The generation counter ticking up through my re-burns, the save region wiped exactly when the game changed and only then.

The last section on that diagram is the one I have not field-run yet: at burn time the console now compiles the cart once on a throwaway Lua state, dumps the bytecode, and writes it as its own section tagged with the core and bytecode format that produced it. At load, an exact tag match means the runtime undumps instead of compiling - no PNG decode, no dialect pass, no parse. A mismatch falls back to the verbatim file, so the fast path can only ever make loads faster, never break them. Lua's dump header even enforces the trust boundary for free: it encodes the platform's pointer width, so a dump built anywhere else refuses to load on the console. The install happens at burn time deliberately - the manifest is written last, so a cart yanked mid-anything reads back as blank, never as a brick. First bytecode burn happens tomorrow; tonight it is committed, green, and flashed.

Three fronts, one lesson, third post in a row saying it: the eyes spot the ghost, but the instrument convicts it.