Quantization

A technical correction on FP16 quantization quality loss, revealing that overflowed BatchNorm tensors rather than architectural sensitivity were to blame, with a complete guide to verification and fixes.

A deep dive into architectural strategies, speculative decoding, KTransformers offloading, and KV cache quantization required to run 125B-parameter MoE LLMs locally at 100 tokens per second on single consumer GPUs.

Discover how expert usage sorting, dynamic tier-based quantization, and llama.cpp CUDA kernel patching enable running the 125B Qwen3.8-Flash-Next MoE model on three consumer RTX 3090 GPUs at 80 tokens/sec.

A deep dive into how Transformers now supports GGUF/llama.cpp quantization, enabling efficient local inference for high-performance LLMs.

An in-depth technical analysis of Quantization-Aware Healing (QAH), explaining how compressed 4-bit LLMs can outperform their original 16-bit counterparts through regularized fine-tuning and noise calibration.

A deep dive into the technical mechanisms behind OpenAI's Sol and Luna model tiers, covering distillation, quantization, and speculative decoding for developers.

Explore how Nunchaku's W4A8 quantization brings high-speed 4-bit inference to large-scale diffusion models like Flux.1 and SVD, now integrated with the Hugging Face Diffusers library.

Discover the 'Tiered Decode Law' that predicts LLM inference speed across hardware. Learn how depth-aware quantization allows a 110B model to run on a 2016 desktop with 16GB RAM.

A deep dive into model quantization formats like AWQ, GPTQ, and GGUF, and how to implement high-performance serving using vLLM and Dynamic LoRA for enterprise Small Language Models (SLMs).

A detailed technical breakdown of VRAM requirements for all Google Gemma 4 variants, including quantization impacts and hardware recommendations for developers.

A deep dive into LLM memory math, explaining why model weights are only half the story and how to calculate KV cache requirements for Llama 3 and Gemma 2.

An in-depth look at the latest advancements in local AI: Ollama's improved interoperability, the release of Qwen3.5 35B with native MTP, and groundbreaking 1-bit text-to-image models running in the browser via WebGPU.