The Console That Files Its Own Bug Reports

Rusty Nail runs sideloaded PICO-8 carts on a bare-metal STM32H753. Bare metal means when something faults, there is no operating system to write a core dump, no journald, no dmesg to read in the morning. The screen goes dark or the cart spins at full speed drawing nothing, and whatever went wrong is gone the moment the board resets. For weeks my entire crash workflow was "watch the defmt log over the ST-LINK and hope it happened while I was looking."

So I built the console a crash reporter. Not a toy: capture the fault with the smallest possible amount of work inside the handler, park a record somewhere that survives the reset, turn that into a durable append-only log in flash on the next boot, show the player a recovery screen instead of a brick, and - when there is a network - upload the report and email me. The whole thing landed and got verified on hardware in a day, and along the way the H7 taught me three separate ways it will silently lose your data across a warm reset.

Here is a real one, start to finish, from the bench log:

The shape of it

The rule that made this tractable: the fault handler does almost nothing. It cannot allocate, cannot take a lock, cannot format a string, cannot touch flash or SPI or the Lua VM. Anything interesting - parsing, storage, networking - happens later, from healthy code, at a safe time. So the system is layered by when each part runs:

fault / panic          next boot                menu idle
-------------          ---------                ---------
capture registers  ->  validate the record  ->  upload over Ethernet
into a RAM slot        append it to flash        or the ESP32 relay
reset the chip         show a recovery screen    mark it sent

The capture path is the strict part. On a Cortex-M7 the hardware stacks eight registers (r0-r3, r12, LR, PC, xPSR) when it takes a fault; the handler reads those plus the fault-status registers (CFSR, HFSR, BFAR, MMFAR and friends), a bounded copy of the panic message, the newest breadcrumbs, and a small snapshot of the faulting stack, packs them into a fixed 1920-byte record, writes it to a reserved slot in RAM, and resets. The four fault handlers are five instructions each - grab the exception return value, the stack pointers, a kind tag, and branch to the shared capture:

#[unsafe(naked)]
#[no_mangle]
unsafe extern "C" fn HardFault() -> ! {
    core::arch::naked_asm!(
        "mov r0, lr",       // EXC_RETURN: says which stack faulted, FPU frame or not
        "movs r1, #1",      // kind = HardFault
        "mrs r2, msp",
        "mrs r3, psp",
        "b {entry}",
        entry = sym crash_capture,
    )
}

All the byte-level logic - the record codec, the breadcrumb ring, the slot protocol, the flash journal, the upload state machine - lives in a no_std library with no hardware in it at all, so cargo test on my laptop exercises every torn-write interleaving without a board attached. The firmware is a thin layer of registers and linker symbols on top. That split is the only reason the storage code is trustworthy: the test that matters kills a flash write after every single one of its 62 machine words and proves the previous record always survives.

Where do you put a record that has to survive a reset?

This is the whole problem, and it is where the H7 got interesting.

A software reset (SCB::sys_reset()) does not clear RAM - the bytes are still sitting there when the next boot starts. So in principle you write your record to some RAM address, reset, and read it back. The console already does exactly this with a one-word "boot guard" that breaks crash loops, so I knew the technique worked. I just needed a bigger reserved region.

My first instinct was DTCM - the tightly-coupled RAM. It is never cached, so the fault handler needs no cache maintenance, and it is always clocked. Perfect, except for one detail I found by writing a test pattern over the debug probe, resetting, and reading it back:

Thirty of the thirty-two bytes came back fine. Two did not. And it was not random: every corrupted byte was lane 3 of its 32-bit word - the top byte, addr & 3 == 3. I reproduced it with different patterns and different addresses; the top byte lane of scattered DTCM words just reverts across a warm reset. My best guess is the TCM's ECC re-evaluating at reset and losing to a partially-written word, but I did not need the mechanism - I needed a commit marker that could not evaporate one byte at a time, and DTCM could not give me one. Lesson one: do not trust DTCM to survive a reset.

So I moved the region to AXI SRAM, where the boot guard already lived happily, and carved it out as a non-cacheable MPU region so the fault path still needs no cache flush. Flashed it, crashed it, and... the record came back all zeroes. Every time.

That one took longer, because the bytes were clearly written - I could see them over the probe milliseconds before the reset - and clearly gone by the next boot. The answer was in cortex-m-rt's linker script. I had injected my section with INSERT AFTER .bss, which felt right. But cortex-m-rt deliberately extends the "zero this at startup" boundary (__ebss) past any section you insert there, so the C-runtime zeroing loop helpfully wiped my crash record on every single boot before main ever ran. Moving the injection to INSERT AFTER .uninit - the same protected zone the boot guard uses - fixed it. Lesson two: sections injected after .bss get zeroed at startup; that is a feature, and it will eat your data.

And then, with the region finally surviving, one last gremlin: the commit marker occasionally came back wrong even with a barrier and a read-back verify in between. It turned out the very last store before SCB::sys_reset() gets dropped - the write is in flight when SYSRESETREQ lands and it just does not complete, DSB and ISB notwithstanding. The fix is almost funny: after everything that matters is written, do one more throwaway store to a pad word nobody reads. Let the reset eat that one. Lesson three: the last store before a reset is not guaranteed to land, so make the last store a sacrifice.

Three separate data-loss mechanisms, all on the same chip, all found by the same brute-force loop of "write a pattern, reset, read it back, stare at the diff." None of them are in the reference manual in any form I could have searched for beforehand. This is the part of bare-metal work that no datasheet prepares you for and no abstraction saves you from.

From a RAM slot to a durable log

The RAM slot survives one reset. It does not survive a power cycle, and it is only big enough for two records. So the next healthy boot promotes it: validate the record (magic, version, length, CRC - never trust the magic alone), append it to an append-only journal in a spare 128 KB sector of internal flash, and only then mark the RAM slot consumed. If the flash write fails, the record stays in RAM and gets reported again next time. Nothing is lost by a badly-timed reboot in the middle of promotion.

The journal itself has to respect a rule the H7 does not advertise loudly: its flash is programmed in 256-bit words with ECC, and you may not program a word twice between erases without raising ECC errors when you read it back. So every state transition of a journal slot gets its own write-once word: a header word, then the payload, then a commit word written last, then - much later, when the server acknowledges the upload - an "uploaded" word programmed into a slot that was deliberately left blank for it. The slot's state is read purely from which words are still erased. Nothing is ever rewritten. Marking a report uploaded is a single 32-byte program into previously-blank flash, no erase, no rewrite, idempotent.

On the bench this is undramatic, which is the point:

crash: recovered from previous session - slotA=true watchdog=false loop_count=1
crash: journaled emergency slot0 -> journal slot 5 (kind=9 fp=0xeae0dbabb23d79d5)
crash: journal ready - 1 new this boot, 1 awaiting upload, 59 free slots

Watchdog resets never reach a fault handler at all - the chip just resets when the independent watchdog starves. But the reset cause register knows it was the watchdog, and the runtime context that normal code keeps up-to-date at safe points still survives in RAM, so the next boot can synthesize a record for a hang it never got to handle. I verified that one by deliberately spinning forever and watching the 32-second watchdog fire:

reset cause: iwdg=true
crash: recovered from previous session - watchdog=true loop_count=1
crash: synthesized watchdog record -> journal slot 4

Uploading, without ever blocking a frame

A journalled report is safe forever, so uploading is allowed to be lazy and fussy. It happens only when the console is idle - never while a cart is running or a download is in flight - one report at a time, retried with a 30-second, 2-minute, 10-minute backoff until the server answers. The console has two ways onto the network: a direct HTTP POST over the onboard Ethernet PHY when it has a lease, or, failing that, the record is streamed over the UART to the ESP32 co-processor, which reassembles it, checks the CRC, and POSTs it over WiFi. Either way the machine that decides when and whether is a tiny state machine with no I/O in it, tested exhaustively on the laptop, so the frame loop never waits on a socket.

The nicest detail is the duplicate handling. The server deduplicates by a stable crash fingerprint - a hash of the fault kind, the firmware build, the faulting PC and LR, the relevant fault-status bits, and the cart - with nothing time-like in it, so the same bug always produces the same code. The server answers 200 for a new report and 409 for one it has seen before, and the console treats both as delivered. So a crash that happens fifty times uploads fifty times and collapses to one report with a duplicate counter, and the console still gets to tick each one off its journal:

crash: eth upload slot 0 -> http 200 (delivered)
crash: journal slot 0 marked uploaded
crash: eth upload slot 1 -> http 409 (delivered)
crash: journal slot 1 marked uploaded

The server stores the raw bytes and a decoded copy, and on the first sight of a new fingerprint it emails me. The email is assembled server-side - the console never holds a mail credential, never talks to a mail provider, never knows my address:

Subject: Rusty Nail crash: HardFault in PORKLIKE [b007c0de00000004]

Device: bc6f8d66
Firmware build ID: ...
Fault: HardFault
PC: 0x08001234
LR: 0x08005678
CFSR: 0x00000100
Uptime: 3600s
Frame: 216000
Last runtime phase: 4
Message: attempt to index a nil value 'enemy'
Crash fingerprint: b007c0de00000004
Symbolication: run crash-symbolicate <report-id>

Newest breadcrumbs:
  100ms CART_LAUNCH ...
   90ms BOOT_START ...

Getting that first email through was its own tiny mystery: the provider's API kept returning a bare 403 Forbidden that made no sense until I noticed the error code was Cloudflare's, not the provider's - it was blocking the default Python-urllib user-agent before the request ever reached the API. Naming the service in the UA header fixed it, and the notification landed in my inbox for a fault the console had invented ten seconds after boot.

Turning an address back into a line of code

PC: 0x08001234 is not useful on its own. It is only meaningful against the exact binary that crashed, so every firmware image gets a build ID that is a CRC of its own flashed contents, computed on the board at boot and reproduced off the ELF by a host tool that archives the unstripped binary under that ID. Feed a report to the symbolication tool and it resolves the hardware-stacked PC and LR against the matching ELF. Here it is on a real UsageFault I triggered on purpose with a udf instruction:

$ crash-symbolicate 1296637eb1c1d57a-1
kind: UsageFault  fingerprint: 1296637eb1c1d57a
CONFIRMED frames (hardware-stacked):
  pc 0x08024ca2  firmware::crash::fire_dev_trigger
                 firmware/src/crash.rs:1196

Line 1196 is the udf instruction. The tool keeps three tiers strictly apart, because honesty about forensic confidence matters: CONFIRMED is the hardware-stacked PC and LR, which are real; RECORDED is a panic or Lua error's own file and line, which the record carries directly; and HEURISTIC is any flash-range word fished out of the bounded stack snapshot, which might be a return address or might be a coincidence, and is labelled as such. Arbitrary stack words are not a backtrace and the tool never pretends they are.

What the player sees

None of the above is any use to someone holding the console if the screen is just black. So the last layer is a recovery screen, in the same pixel style as the rest of the little OS. After a crash the console comes back to this instead of silently rebooting into the same crash:

And if the same cart crashes three boots running, the console stops auto-launching it and drops into safe mode - the anti-brick. You can still retry, browse the library, send the reports, or clear the loop and carry on. A crashing cart can annoy you; it cannot trap you:

Both of those screenshots are live captures off the panel over the ST-LINK, and both of them are the second draft. The first recovery screen showed two of the four actions as button icons and the other two as typed "B: RETRY" / "Y: SEND" hints, which meant the same letter appeared twice and the typed letters sat next to mismatched keycap glyphs. It read as gibberish - which button does what? The fix was to show every action once, each on its real face button, as a clean four-key legend. And a docked cart used to bury the recovery notice under its "load game" prompt, so the one screen the player most needed to see was hidden behind an offer to play the game that just crashed. Now a crash outranks every other pop-up. Both were found the only way UI bugs ever are: by looking at the actual screen.

The numbers

The crash feature costs about 3.9 KB of flash. It reserves 4 KB of reset-surviving RAM and about 12 KB of working RAM, all funded by shrinking the Lua heap 24 KB (from 624 KB to 600 KB - the heaviest cart I have measured peaks at 512 KB, so there is still headroom). The durable journal is a 128 KB flash sector holding 64 reports. A capture completes in about a millisecond. And the steady-state cost during actual gameplay is zero: there is no per-frame work beyond dropping the occasional 16-byte breadcrumb at coarse milestones, and the upload only wakes up when the console is sitting idle in a menu.

What software cannot do

One honest limitation, stated plainly because the alternative is a lie: none of this survives instantaneous power loss. If the 3.3 V rail drops mid-write, that record is gone. The whole design guarantees the previous committed record survives - the one being written is the accepted casualty

• but no amount of clever handler code saves the write that was in flight when the power died. Doing that needs hardware the console does not have yet: a supply supervisor that fires while the rail is still good, and enough hold-up capacitance to finish one bounded write. A fault handler is not a substitute for a supercapacitor, and pretending otherwise would just move the disappointment to a worse moment.

But for the failure mode I actually have - carts that fault, panics I write, watchdog hangs, my own undefined instructions - the console now notices, survives, remembers, shows me, and mails me. It files its own bug reports. That is a strange sentence to write about a 320-gram board on my desk, and I am delighted by it.