From emulator to real console

Over five months this year, I took Rock Band 3 from a debuggable boot inside a modified emulator, through a from-scratch native container writer, to a shipped mod running on a physical Xbox 360. Three repositories, one connected arc — and nearly every hard bug along the way lived in the same place: the gap between what an emulator does and what the real console actually does.

This is the story of that arc, told through its three hardest problems.

An emulator that can’t play games, on purpose

The work started with a fork of Xenia, the open-source Xbox 360 emulator. My reason for forking it was unusual: I’ve been working on decompilations of the Milo engine — the shared engine behind Rock Band 3 and Dance Central 3 — and a decompilation is only provably correct if you can run a rebuilt binary and show it behaves identically to the original. That demands an emulator that is headless, scriptable, and deeply observable. Xenia ships as a windowed, gamepad-driven application, which is the opposite of a test harness.

So the first commit was a big one: a headless emulator mode for Linux — 3,328 lines across 61 files, adding a windowless Vulkan command processor, a no-op audio backend, scripted controller input, and frame-dump output. Landing a feature that size in one shot into an unfamiliar codebase with seven thousand commits of history, and then keeping it working for five months of continuous change, turned out to be the foundation everything else sat on.

The months after that were a slow detective story. Getting Dance Central 3 to boot at all meant root-causing heap exhaustion in the guest, then discovering that config parsing was pathologically slow under emulation — a bump allocator cut one parsing pass from 15,273 ms to 236 ms — and ended with the main loop stable at 175 fps. Along the way the fork grew the instruments the later work would need: an interactive guest-PowerPC debugger and a function entry/exit tracer woven directly into the x64 JIT.

The segfault at exactly four gigabytes

In July the target shifted to Rock Band 3, and the work got genuinely strange. A host-side segfault kept landing at exactly the guest memory base plus 4 GiB — the telltale signature of a guest NULL-pointer dereference mapped into the emulator’s own address space. It hit both games. The obvious conclusion was a game bug, or a bug in my decompilation-derived understanding of the engine.

It was neither. Tracing the fault against guest disassembly and the retail kernel’s behavior revealed a hardware-fidelity gap in the emulator itself: real Xbox 360 hardware backs guest page zero with actual memory, and certain architecturally special guest code legitimately touches it. Xenia’s policy treated any access there as fatal. The fix was a hardware-faithful zero-page backing, gated behind a default-off flag so the working configuration was never at risk — and the investigation itself is documented across ten cross-linked wiki pages that read like a lab notebook, including the hypotheses that turned out to be wrong.

That investigation set the pattern for everything after it: root-causing faults that span four layers at once — host JIT, PowerPC guest, kernel emulation, and the game engine — from nothing but fault addresses and logs. It also surfaced a bug I have to own: a later crash in the community-modified Rock Band 3 Deluxe traced back to a regression in my own earlier kernel-emulation change, which had been silently wedging its splash screen. Fixing your own regressions honestly is part of the discipline; so is running before/after comparisons of draw counts, frame rates, and crash counts on nearly every change, so a fix for one game never quietly breaks the other.

Writing containers a real console will accept

Emulation proves correctness. It does not get code onto a real console. The mod at the end of this chain ships as a DLL that a development Xbox 360 must accept through Microsoft’s proprietary XEX2 container format — undocumented, and enforced by a loader whose only diagnostic is binary: the module loads, or it doesn’t.

For months, the working hypothesis was that the container format itself was the problem — the compression, the integrity hashing, the signature envelope. One control experiment collapsed all of that: recompressing and re-signing a known-good stock DLL with no content change still loaded fine. The container pipeline was never broken. The real defect was in the packer-synthesized import table — the exact byte encoding one kind of import thunk requires, which the packer was emitting in one of two wrong forms. That narrowing, verified as a five-module load matrix on live hardware, became both the proof and the regression suite.

From there, the project reproduced the console’s signing and integrity pipeline from scratch as a native Linux tool: a reverse-chained per-page SHA-1 hash chain and a 2048-bit development-console RSA signature, matching the reference output bit for bit. The hardest piece was the compressor. The container format uses LZX compression, and the open encoder library I vendored had a single-buffer limit of roughly 2 MB — the mod’s full image is 8.38 MB. Generalizing the encoder’s sliding window to stream past that limit produced output whose decompressed image and all 134 per-page digests are byte-identical to the closed-source reference tool. The compressed bytes differ, but every value the loader actually verifies matches — which is the only claim that matters, and the honest way to state it.

Shipping a feature the game never had

The end of the chain is RB3 Overdrive, a mod players download and run on their own console. Its headline feature — multiple local players on the same instrument type, which retail Rock Band 3 never supported — required coordinated hooks across part selection, track binding, and player-slot assignment, all inert behind a default-off flag so stock behavior is untouched unless you opt in.

Getting it working meant debugging closed-source retail game code live on real hardware over two days of numbered diagnosis cycles: converting each crash into a named root cause — a slot index of exactly negative one, a null dispatch, a missing vtable pointer — rather than a surface patch. The subtlest bug wasn’t input handling at all but a shared display asset, which led to the part of the project I’m proudest of as engineering ethics: rather than ship a modified copy of a copyrighted game asset, the mod reconstructs the needed asset from the player’s own game files at first boot, from a tiny embedded splice plan, verified with SHA-256 gates and automated tripwires that make it impossible to ship a stock byte. It went out as version 1.0.0 in a twelve-day sprint, verified on the same console the container work targets.

What’s done and what isn’t

Honesty about status matters more in a writeup than in a changelog, so: Dance Central 3’s boot-to-gameplay path is stable in the emulator fork, but Rock Band 3’s emulated bring-up is still the live frontier. The container writer’s offline validation is complete and hardware-confirmed, but a live load test of its natively compressed output is still pending. The mod’s 1.0.0 release is shipped and hardware-verified. And none of the three forks has an upstream pull request open yet — several of the emulator fixes are written to upstream quality and gated behind default-off flags precisely so they could go back, but that work remains ahead of me.

The through-line, if there is one, is that emulators and hardware disagree in small, unforgiving ways — a page of memory one backs and the other doesn’t, a byte encoding one loader checks and the other ignores. Finding those gaps from both sides, and closing them with control experiments instead of guesses, turned out to be the whole job.