Codebook Lossless LLM Compression: 10-25% RAM Reduction with Bitwise Packing
A developer has published proof-of-concept code for lossless LLM compression that reduces memory usage by 10-25% through bitwise generic packing of indexed weights. The technique trades some inference speed for smaller model size, making it possible to run larger models on hardware with limited VRAM.
How It Works
The developer started by asking how many unique values actually exist in LLM layers. Analysis revealed that while fp16 uses 16 bits, most models only utilize about 12-13 bits of unique values. By packing these values into blocks, the technique achieves compression without losing precision.
Performance Characteristics
- RAM reduction: 10-25%+ across tested models
- Speed impact: Inference speed approximately halved in example tests
- Test hardware: NVIDIA P2200 (5GB) and CPU, with updates being developed for AMD MI50 (32GB)
Implementation Details
The developer worked on this project for several weeks using AI coding assistants including Claude, Qwen, and Gemini. The repository includes both lossless and lossy/balanced versions, though the lossy version hasn't been extensively tested yet.
The developer suggests this compression approach might serve as a way to measure a model's "compactness" - how efficiently it uses its parameter space.
Code Availability
The proof-of-concept code is available on GitHub: https://github.com/bigattichouse/Codebook-Quantization
📖 Read the full source: r/LocalLLaMA
👀 See Also
re_gent: Git for AI Coding Agents – Version Control for Agent Activity
re_gent is an open-source tool that provides version control for AI agent sessions, tracking every tool call, storing prompts and file diffs, and enabling commands like `rgt log`, `rgt blame`, and `rgt rewind` (coming soon).
MiniMax Music 2.5 AI Music Generator Released with Studio-Grade Audio Control
MiniMax Music 2.5 is an AI music generation model that creates studio-quality songs with 44.1kHz Hi-Fi output, 100+ instruments, and paragraph-level precision control using 14+ structural tags for directing song structure.
ZSE: Open-source LLM inference engine with 3.9-second cold starts
ZSE is an open-source LLM inference engine that reduces 32B model memory requirements from 64GB to 19.3GB VRAM and achieves 3.9-second cold starts for 7B models using a pre-quantized .zse format with memory-mapped weights.
Cognithor: A Local-First Agent OS with PGE Trinity Architecture
Cognithor is a fully local, autonomous Agent OS built over a year with 16 development phases. It features the PGE Trinity architecture (Planner → Gatekeeper → Executor), 11,609+ tests with 89% coverage, and supports 16 LLM providers including Ollama and LM Studio.