Fine-tuned Qwen3-0.6B model outperforms 120B teacher on structured function calling
What this is
Distil Labs released a complete pipeline that fine-tunes a small 0.6B parameter Qwen3 model to outperform a 120B parameter teacher model on structured function calling tasks. The pipeline extracts production traces, generates synthetic training data, and trains a specialist model that's 200x smaller than the teacher.
Performance results
- Teacher (GPT-OSS-120B): 50.0% tool call equivalence
- Base Qwen3-0.6B (no fine-tuning): 10.3% tool call equivalence
- Fine-tuned Qwen3-0.6B: 79.5% tool call equivalence
The task is IoT smart home function calling: routing natural language commands like "turn on the kitchen lights" or "make me a coffee at 7am" to the correct function with the right parameters. Scoring is based on exact structured match, not fuzzy scoring.
Why the small model wins
The 120B teacher is a general-purpose model that has never seen these specific function schemas or user phrasing patterns. It often produces verbose or slightly off-format responses. The 0.6B student is a specialist trained exclusively on this task, so it nails the exact output format consistently.
Pipeline architecture
The three-stage pipeline:
- Data extraction: dlt extracts production traces from databases, APIs, cloud storage, or log aggregators and writes them to Hugging Face as clean Parquet datasets
- Automatic curation: An LLM judge scores and filters traces to select high-quality seed examples (no manual annotation required)
- Synthetic data generation and training: Distil Labs uses the traces as domain context, generates ~10,000 synthetic training examples with a large teacher, validates and filters them, then fine-tunes the student model
The key insight: instead of training on raw traces directly, they're used as context so the synthetic data generator produces examples matching real vocabulary, function schemas, and phrasing patterns from actual users.
Dataset and practical details
- Used Amazon MASSIVE dataset (16k+ utterances, 60 intents) as stand-in for production traffic
- Filtered to IoT scenario with 9 smart home functions
- ~75 labeled seed examples were enough (automatic curation, zero manual annotation)
- Training completed in under 12 hours
- Model inference: under 50ms locally vs. 400-700ms for cloud API calls
- Model available in safetensors and GGUF formats on Hugging Face
Production considerations
The model scores 79.5% exact match, meaning roughly 1 in 5 queries may need a fallback. For production use, you'd want a confidence threshold routing low-confidence predictions to a larger model.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Developer builds AI framework with 17 biological principles using Claude Code
A developer created an AI framework called Cognitive Sparks by implementing 17 biological principles like threshold firing and Hebbian plasticity, based on the 1999 book 'Sparks of Genius.' The entire project—22 design docs and 3,300 lines of code—was built in one day using Claude Code, with no human-written code.
10.33 t/s on Qwen 3.5 35B with a $300 Laptop: Full Optimization Breakdown
A developer achieves 10.33 t/s on Qwen 3.5 35B Q4_K_S on a Lenovo Ideapad Slim 3i ($300) using ik_llama.cpp, core pinning, MTP speculative decoding, and BIOS performance tuning.
Open-source pipeline turns Claude Code workflow into reusable skills
A developer who used Claude Code daily for 9 months has open-sourced a pipeline that structures feature development with checkpoints like functional documentation, technical documentation, complexity estimation, and security checks. The pipeline includes /new-feature and /bug-fix entry points that guide implementation.
Krasis: Hybrid CPU/GPU Runtime for Large MoE Models Achieves 3,324 tok/s Prefill on RTX 5080
Krasis is a hybrid CPU/GPU runtime that runs large MoE models by handling prefill on GPU and decode on CPU, achieving 3,324 tokens/second prefill on an RTX 5080 with Qwen3-Coder-Next 80B Q4. It requires ~2.5x model size in system RAM but enables running models too large for VRAM.