Forking the decompilation toolchain

There is a moment, early in this kind of project, when you feed your binary to the community’s standard tools and watch each one fail in its own way. Ghidra hits the Xbox 360’s vector instructions and decodes noise. The binary splitter doesn’t recognize the executable container. objdiff renders a beautiful per-instruction diff — for a human who, in my pipeline, isn’t there.

Over about six months I forked nine open-source reverse-engineering tools — Ghidra, objdiff, decomp-toolkit, m2c, wibo, and several others — and extended each one until a fleet of AI coding agents could use them to reconstruct the source code of two Xbox 360-era games, byte for byte. None of these tools was built for that job. Most had never seen the Xbox 360’s CPU at all. This is a story about what happens when your entire toolchain is someone else’s project, and the target you care about sits just outside what any of it supports.

The problem: a console nobody’s tools can read

Matching decompilation means writing new source code that compiles to the exact bytes of an original binary. It is how communities preserve games whose source is long gone: new code, written from the compiled output, verified instruction by instruction against the retail build. The tooling for this is mature — but it grew up around the GameCube and Wii, targeting GCC-family compilers, and it assumes a human at the controls.

My targets were Harmonix’s Dance Central 3 and Rock Band 3, built with Microsoft’s MSVC compiler for the Xbox 360, alongside their Wii counterparts. That combination breaks the existing tools three ways at once. The Xbox 360’s CPU carries a proprietary 128-register SIMD extension called VMX128 that no mainstream disassembler decodes — not stock Ghidra, not even Capstone. MSVC’s PowerPC output looks nothing like the GCC idioms the community’s analyzers were tuned against. And the executables ship in Microsoft’s XEX container format, which most tools cannot open.

Then I added a constraint of my own: the consumers of these tools would be swarms of Claude Code agents running around the clock, not humans at GUIs. A tool that renders a pretty diff for a person is useless to an agent that needs a structured verdict it can act on.

The whole project lives inside one loop:

retail XEX ---[splitter]----> original .obj ---+
                                               |
new C++ ---[MSVC via wibo]--> rebuilt .obj ----+
                                               |
                                               v
                        [objdiff] --> structured verdict
                                               |
                                               v
                         agent edits the C++; repeat

Ghidra sits underneath the whole thing, telling the agents what the original code even does. Every stage of that loop runs through a fork, and every fork had to answer the same two questions. Can you teach this tool the target? And can you teach it to talk to machines?

Teaching Ghidra a new dialect

Ghidra — the NSA’s open-source decompiler — became the backbone. A community contributor had already produced disassembly-only VMX128 support: enough to name the instructions, not enough to reason about them. My first substantial commit added full decompiler semantics for all 77 VMX128 opcodes — 86 p-code operation definitions, plus a correctness fix declaring the vector registers at their true 128-bit width — validated against 13,836 VMX128 instructions in a real binary. The payoff is a new processor language for the Xbox 360’s Xenon CPU: vector-heavy code now decompiles instead of vanishing into undefined-instruction noise. A second language definition covered the GameCube and Wii CPUs.

The most satisfying fix was smaller, and went back upstream. Ghidra’s PowerPC jump-table analyzer silently failed on MSVC-generated switch statements, leaving whole regions of control flow unrecovered. Digging in revealed three independent defects: an access check that unconditionally rejected table reads beyond 4096 bytes, a target list that leaked entries from one switch into the next, and a predecessor walk too shallow to span MSVC’s multi-block pattern. I filed the diagnosis as an issue and the fix as a pull request to the NSA’s repository, with a synthetic-pattern regression test. A maintainer triaged and assigned it; as of this writing it remains open and unreviewed. That is simply how upstreaming to a large project goes, and I would rather say so than pretend otherwise.

Scale was the third front. Running Ghidra’s function-matching and similarity-search engines on a game with roughly 65,000 functions surfaced performance cliffs everywhere — one degenerate case stalled the CPU for over 70 minutes. Fixing them meant parallelizing carefully inside a codebase I did not own, so every parallelization commit carries a written argument for why the output stays deterministic. The honest caveat: I quantified the before state in each commit but did not record measured after numbers, which I regret.

Rebuilding tools for machine consumers

The more interesting transformations were about who the tool serves.

objdiff, written by encounter — whose tools appear so often in this story that they are effectively a co-architect of the whole scene — is the community’s standard object-file diffing tool, built to show a human where their reattempted code diverges from the original. I rebuilt it to diagnose instead of display: 21 pattern detectors that recognize common mismatch causes, six fixability verdict classes, and structured JSON an agent can consume directly. The fork grew by about 19,000 lines across 76 files. Its nastiest bug was uniquely an AI-pipeline bug: a nondeterminism in how MSVC exception-handling funclets were paired meant two byte-identical builds could report different match percentages — quietly corrupting the exact metric the agents optimize against. Deterministic pairing over a masked byte signature fixed it, across three days and four patch releases.

pyghidra-mcp, which exposes headless Ghidra to agents over the Model Context Protocol, needed a different kind of hardening. Its worst failure only exists under swarm load: 125 half-closed connections deadlocking a single-threaded JVM as sync and async handlers raced it. The fix was one process-wide reentrant lock serializing every public method — unglamorous, and exactly right.

decomp-toolkit, encounter’s Rust CLI that splits console binaries into linkable objects, forked into two deliberate lineages. One stays within eight lines of upstream, PR-ready. The other builds on rjkiv’s Xbox 360 port — the work that first taught the toolkit to open XEX containers at all — and carries a full control-flow-analysis rewrite for retail executables — cut over like a zero-downtime database migration, with staged phases, shadow execution, parity digests, and a rollback flag, promoted to default only after it processed 69,336 functions with byte-identical output and zero crashes. Keeping a clean fork and a heavy fork of the same tool sounds like overhead; in practice it is what let me hack aggressively without burning the bridge back to upstream.

Throwing away the clever version

The fork that taught me the most was wibo, a minimal Win32 compatibility layer. To rebuild these games verifiably, I needed the 2010 Xbox 360 linker running on Linux — which meant faking Microsoft’s private, undocumented PDB COM interface. My hand-written stand-in for mspdb80.dll runs about 810 lines and implements eight COM vtable object types; the linker’s output is byte-identical to a run under Wine, apart from timestamps and an embedded GUID.

My first working version generated x86 calling-convention thunks at runtime. It worked. When I opened a pull request, the upstream maintainer called it “technically impressive but incredibly scary” — and he was right. I threw it out the same day and rebuilt it as a boring, reviewable cross-compiled DLL. That PR is also still open, but the lesson shipped immediately: in someone else’s codebase, reviewability beats cleverness every time.

What’s finished, and what isn’t

The status board is honestly mixed. The Ghidra fork, objdiff, the heavy decomp-toolkit lineage, and the MCP server are actively maintained; they carry the pipeline daily. The m2c decompiler fork — which grew a from-scratch VMX128 disassembler for an ISA extension Capstone still cannot decode — has been dormant since March. Two upstream pull requests remain open. The heavy toolkit fork is mid-triage on a third round of binary-splitting edge cases.

Much of this work was itself AI-assisted, and the commit history says so: agent co-authorship is disclosed in trailers and in the upstream PRs, with human direction and review throughout. If there is one takeaway, it is that “fork and extend” is not a lesser mode of open-source work. Every stage of that pipeline diagram started as someone else’s project: encounter’s decomp-toolkit and objdiff, rjkiv’s XEX port and the Dance Central 3 and Rock Band 3 Xenon scaffolds, DarkRTA’s Wii decompilation of Rock Band 3, wibo’s maintainers, the NSA’s Ghidra team, and the wider MiloHax preservation community whose years of unpaid groundwork made an AI-driven campaign even conceivable. Reading nine unfamiliar codebases deeply enough to change what they fundamentally do — while keeping each fork honest, minimal where it can be, and pointed back at upstream — turned out to be the hardest and most transferable engineering in the whole project.