Transformer Language Model Runs Locally on Stock Game Boy Color

A developer has gotten a real transformer language model running on a stock Game Boy Color (GBC) — no phone, PC, Wi-Fi, or cloud inference involved. The entire inference pipeline runs locally on the handheld hardware.

Key Details

• Model: Andrej Karpathy's TinyStories-260K, converted to INT8 weights with fixed-point math — no floating point support required.
• Hardware: Stock Game Boy Color + EZ Flash Junior flash cart + microSD card.
• Build toolchain: GBDK-2020, producing an MBC5 Game Boy ROM.
• Memory architecture: Model weights live in bank-switched cartridge ROM. The KV cache is stored in cartridge SRAM because the GBC's work RAM is tiny.
• Prompt entry: On-device using D-pad/buttons and an on-screen keyboard.
• Inference pipeline: Prompt tokenization on the GBC, then transformer prefill + autoregressive generation with KV caching.
• Performance: Extremely slow; output is gibberish due to heavy quantization and mathematical approximations, but the core transformer loop works.
• Source code: Available on GitHub at github.com/maddiedreese/gbc-transformer. A large portion of the code was built using Codex AI.

The project demonstrates that even severely resource-constrained hardware can execute transformer inference with aggressive quantization and memory management tricks. It's a proof-of-concept, not a practical LLM, but it's a technical curiosity worth examining.