What does this PR do?

Opt-in inference path for GGUF models that keeps weights at their native quantization (Q4_0 / Q8_0 / Q4_K / Q5_K / Q6_K / IQ4_NL / IQ4_XS) and runs the forward matmul through the same Metal kernels llama.cpp ships (lifted from candle, packaged as ArthurZ/gguf-kernels via kernel-builder).

Two new pieces:

• integrations/gguf_linear.GgufLinear — drop-in nn.Linear replacement. Forward picks mul_mat_vec_<fmt>_f32 for batch-1 (decode) and mul_mat_<fmt>_f32 for batch>1 (prefill). CPU/CUDA fall back to dequant + nn.functional.linear.
• GGUFQuantizer.linear_mode — when on, the post-load hook walks the GGUF tensor map, re-applies the same renames the loader used, and swaps each matching nn.Linear for a GgufLinear with raw block bytes.

Enabled via from_pretrained(..., gguf_file=..., gguf_linear=True) or TRANSFORMERS_GGUF_LINEAR=1. Default off, no change to existing behaviour.

Builds on top of #44794 (this PR targets update-gguf). #44794 makes load fast; this PR makes inference fit in memory — weights stay at 4.5 bpw (Q4_K) instead of 16 bpw bf16 = 3.5× less RAM at inference time, the qualitative win on Apple Silicon.

Performance vs llama.cpp

Measured on M3 Max (96 GB), Qwen2.5-0.5B-Instruct Q4_0:

llama-bench tg64                                  →  261 tok/s
our matvec kernels, one batched MTLCommandBuffer  →  266 tok/s   (1.02× parity)

The kernels are byte-identical to llama.cpp's (via candle). When dispatch is batched into a single command buffer per token (which PyTorch's MPS stream does naturally for ops on the same forward pass), throughput matches.

Per-kernel ceilings vs PyTorch's MPS bf16 path (same M3 Max, 8M-element shapes, input already on GPU):

Compute-bound prefill is a small throughput loss vs bf16 GEMM; bandwidth-bound batch-1 decode is a clear win. Memory savings are 3.5× regardless of workload.

Verification

• Bit-exact: each dequant kernel matches gguf.dequantize to 0 ULP (Q4_0 / Q8_0 / IQ4_NL) or fp32 reduction noise (~1e-4 for K-quants).
• Forward correctness: tests/quantization/ggml/test_gguf_linear.py checks GgufLinear forward against dequant_gguf_tensor + nn.functional.linear for Q4_0 and Q4_K, batch sizes 0/1/8.
• Real model end-to-end: standalone smoke test in ArthurZ/gguf-kernels (tests/test_real_model_e2e.py) loads real Qwen2.5-0.5B Q4_0 with all 169 linears swapped to GgufLinear; top-5 logits match the dequant baseline to fp32 ULP.

Kernels package

ArthurZ/gguf-kernels ships pre-compiled metallibs for torch 2.8 / 2.9 / 2.10 / 2.11 on Apple Silicon. Built via kernel-builder (Nix + Xcode 26 + Metal Toolchain). Source: same MSL kernels as llama.cpp / candle.

Override the repo with TRANSFORMERS_GGUF_METAL_KERNELS_REPO=... (so the personal-namespace location works ahead of any kernels-community transfer).

What's out of scope

• CUDA matmul kernels (candle ships them; same structure as the MPS port — follow-up).
• K-quant Python quantizer (gguf-py only has Q4_0 / Q8_0; K-quant Linears stay as nn.Linear during the swap. Fix is upstream in gguf-py, not transformers.)
• Direct byte routing in the loader (today we round-trip through dequant + re-quant on Linear weights; the load-time memory win is left for a follow-up).

Test plan

• python -m pytest tests/quantization/ggml/test_gguf_linear.py — 4 passed
• from_pretrained(..., gguf_linear=True) end-to-end (blocked on a separate shape bug in update-gguf's dequant for some configs — surfaces as replace_with_gguf_linear warnings and the layer stays as nn.Linear, so behaviour is safe but the GgufLinear path isn't exercised end-to-end on that config yet)