GuppyLM: A 9M Parameter LLM Built from Scratch for Educational Purposes

What GuppyLM Is
GuppyLM is a tiny language model (~9M parameters) that pretends to be a fish named Guppy. It's built from scratch to demonstrate how language models work without requiring a PhD or massive GPU cluster. The project includes data generation, tokenizer creation, model architecture, training loop, and inference—all in about 130 lines of PyTorch code.
Architecture Details
- Parameters: 8.7M
- Layers: 6
- Hidden dimension: 384
- Heads: 6
- FFN: 768 (ReLU)
- Vocab: 4,096 (BPE)
- Max sequence: 128 tokens
- Norm: LayerNorm
- Position: Learned embeddings
- LM head: Weight-tied with embeddings
This is a vanilla transformer with no GQA, RoPE, SwiGLU, or early exit—as simple as possible.
Training and Data
The model trains on 60K synthetic conversations across 60 topics in about 5 minutes on a free Colab T4 GPU. The dataset (arman-bd/guppylm-60k-generic on HuggingFace) contains 57K training and 3K test samples in format: {"input": "...", "output": "...", "category": "..."}.
Topics include: greetings, feelings, temperature, food, light, water, tank, noise, night, loneliness, bubbles, glass, reflection, breathing, swimming, colors, taste, plants, filter, algae, snails, scared, excited, bored, curious, happy, tired, outside, cats, rain, seasons, music, visitors, children, meaning of life, time, memory, dreams, size, future, past, name, weather, sleep, friends, jokes, fear, love, age, intelligence, health, singing, TV, and more.
Personality and Output
Guppy speaks in short, lowercase sentences about water, temperature, light, vibrations, and food. It doesn't understand human abstractions like money, phones, or politics. Sample conversation:
You> hi guppy Guppy> hi there. i just found a nice spot near the rock. the temperature feels nice. You> are you hungry Guppy> yes. always yes. i will swim to the top right now. i promise to eat all of it. You> what is the meaning of life Guppy> food. the answer is always food.
Quick Start
Chat with Guppy (no training needed): Downloads the pre-trained model from HuggingFace. Just run all cells in the notebook.
Train your own: Set runtime to T4 GPU, run all cells—downloads dataset, trains tokenizer, trains model, tests it. Upload to HuggingFace or download locally.
Chat locally:
pip install torch tokenizers python -m guppylm chat
Example local interaction:
You> the cat is looking at you Guppy> i hide behind the plant when the furry one comes. You> it is raining outside Guppy> i think rain is the best thing about outside.
Project Structure
guppylm/ ├── config.py # Hyperparameters (model + training) ├── model.py # Vanilla transformer ├── dataset.py # Data loading + batching ├── train.py # Training loop (cosine LR, AMP) ├── generate_data.py # Conversation data generator (60 topics) ├── eval_cases.py # Held-out test cases ├── prepare_data.py # Data prep + tokenizer training └── inference.py # Chat interface
This project is useful for developers who want to understand transformer architecture fundamentals without dealing with billion-parameter models. The complete implementation shows every piece from raw text to trained weights to generated output.
📖 Read the full source: HN LLM Tools
👀 See Also

Claude for Creative Work: MCP Connectors for Blender, Adobe, Ableton, and More
Anthropic released a set of MCP connectors allowing Claude to interface with creative tools including Blender, Autodesk Fusion, Adobe Creative Cloud, Ableton Live, and Splice, enabling natural-language control, scripting, and pipeline automation.

Context-Engineered Study System for Claude Code Acts as Persistent Tutor
A developer built a study system using Claude Code that tracks progress across sessions, probes understanding, works through exercises, and adapts to learning styles. The system uses structured markdown files to shape agent behavior and includes tools for extracting textbook pages from PDFs.

Why a Single run() Tool with Unix Commands Beats Function Calling for AI Agents
A backend lead with two years of agent-building experience argues that a single run(command="...") tool with Unix-style CLI commands outperforms traditional function calling catalogs. The approach leverages LLMs' existing familiarity with shell commands from training data.

XLI: Open-Source Python Library for Claude Code-Style Terminal UIs
Building a coding agent? Terminal UX is half the work. XLI is an open-source Python rendering engine that replicates Claude Code's streaming markdown, tool cards, inline approvals, and slash commands without hijacking your terminal scrollback.