Qwen3.8-27B on a 24 GB 7900 XTX: the 3-bit quant won my long-context jobs

TL;DR

• I ran two quants of Qwen3.8-27B (dense, 64 layers) on a 24 GB RX 7900 XTX under llama.cpp (Vulkan, build b9999) with llama-swap: Q4_K_M (15.64 GiB) and a 3-bit GSQ-RCO IQ3_S with a real MTP head (11.29 GiB).
• Past ~99k prompt tokens the two are speed-identical (±3%). The difference is the context ceiling: the 4-bit quant’s 262k KV pool OOM’d at ~191k in a boundary probe; the 3-bit quant runs the full 262k tier with 6 GiB of headroom left.
• The MTP head in the 3-bit file is real (verified by file diff — and two other “MTP-named” files I tested were not), and --spec-type draft-mtp --spec-draft-n-max 3 roughly doubled decode on it: 47.7 → 96.2 t/s, byte-identical output at temp 0.
• The 4-bit quant stays the gate model (quality pick, pinned to 185k ctx after the OOM probe); the 3-bit quant takes the long jobs — its day-to-day quality verdict is still open.

The setup

Arch box, AMD RX 7900 XTX 24 GB, pinned via GGML_VK_VISIBLE_DEVICES=0 (vulkaninfo --summary: Vulkan0 AMD Radeon RX 7900 XTX (RADV NAVI31) 24560 MiB). llama.cpp b9999 Vulkan build (47c7869), fronted by llama-swap v253. Flags on both entries: 99 GPU layers, flash attention on, q4_0 KV cache, --jinja, 16 threads.

Every number below came from one harness against the front door, streaming, temp 0, with the payload SHA-256 recorded and identical on both sides, and 1 Hz VRAM sampling. The workloads:

• standard: a fixed coding-review prompt, 1,133 prompt tokens, 256 max tokens
• long-ctx: a long-context payload shaped like a large codebase review — a prompt template plus a big source corpus, 88,170 prompt tokens, 512 max tokens
• curve: the same job scaled to 99,475 / 224,109 / 241,776 prompt tokens

Both models are thinking models; the standard-workload numbers were run model-default (thinking on), the curve with thinking off. Noted where it matters.

The two quants

The two files (Hugging Face):

• Q4_K_M — lmstudio-community/Qwen3.8-27B-GGUF
• 3-bit GSQ-RCO IQ3_S + MTP — ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf)

The MTP verification matters more than it should. Two other “MTP-named” 27B files in my test queue contained no MTP tensors — the name was aspirational. For the GSQ-RCO file I checked by diff: 4,034 nextn/mtp string matches, and the file is exactly 348,470,176 bytes larger than the repo’s non-MTP IQ3_S sibling. That’s a head in the file.

(Anomaly worth a line: b9999’s llama-gguf -l core-dumps on this file — the tensor-list tool crashes on GSQ-RCO per-tensor quants, while llama-server loads and serves it fine. I did the tensor survey with strings instead.)

Speed

Standard (1,133 pt, thinking on):

* The spec run’s prefill was contaminated by the cold start (TTFT 6.0 s); the prefill path is unaffected by spec, which the long-ctx numbers confirm.

Long-ctx (88,170 pt, 512 max, thinking on):

Two things stand out. First, the smaller quant is not slower — it’s usually a point or two faster on prefill (less weight to stream), and with the MTP head + spec flags it decodes faster than the 4-bit gate. Second: the no-spec vs with-spec gap is the loudest number in this post — 47.7 vs 96.2 t/s on the same file. If your serving engine runs speculative decoding and your bench or spawn doesn’t mirror those flags, you’re benchmarking a different machine than the one you run.

The real question: where does the context ceiling actually sit

This is where quants stop being a quality choice and become a budget choice. Same box, same q4 KV, 24 GiB:

The speed curve is a flat tie:

Both quants benched to 241,776 tokens on a good-RAM day (09-05). The 4-bit’s ~191k death is the next day’s boundary probe on a bad-RAM day: past that point the KV overflowed into system RAM and the kernel killed the server mid-prefill. Same box, same 262k pool — the boundary moved with RAM state, which is why the margin past the 185k pin is fatal rather than slow. Prefill degrades smoothly with depth on both (attention scaling), and the 4.3 GiB of weights the 3-bit quant doesn’t carry never shows up as a speed difference — at these depths the GPU is too busy with KV to care. What it does buy is the boundary: the 4-bit quant’s 262k pool died mid-prefill around 191k in the probe (I pinned it to 185,000 ctx + -cram 24576 afterward, and verified a fresh 172k prefill at 312 t/s), while the 3-bit quant sat at 241k with 6 GiB to spare.

So on this box the honest read is: if your jobs fit under ~180k, take the 4-bit for quality. If your jobs are 200k-class, the 3-bit is the one that actually fits, and you’re not paying a speed penalty for it.

What I run now

The decision, concretely:

• Day-to-day — my coding agent’s working sessions — runs on the Q4_K_M gate pinned at 185k ctx. The agent compacts its own session context as it goes, so day-to-day usage stays within ~5% of that cap: the cut from 262k costs nothing in practice, and it removes the OOM boundary from under the daily driver.
• Long-context jobs (200k-class reviews) go to the GSQ IQ3_S entry at 262k — the one with the room.

That gate model’s thinking knob — what off / low / medium / xhigh actually do to it, and why “xhigh” is a duplicate of the default — has its own post: the thinking-levels grid.

Honest caveat: the 3-bit’s quality is not day-to-day tested yet. My GSQ runs so far are the long-context jobs, and the outputs on those have looked right — but that’s one job shape and a small sample. I haven’t lived with it as a daily driver; once I have, I’ll have a much better idea of where the 3-bit actually sits, and this section gets a corrections note if I’m wrong.

How the gate model got here (the flip A/B)

The Q4_K_M entry exists because I A/B’d the model through my llama-swap front door two ways — LM Studio’s internal Vulkan engine (q8 KV + --kv-offload + mlock, spec on) versus local llama.cpp b9999 (q4 KV, FA on, spec flags mirrored) — on the identical SHA-verified payloads:

Statistical tie on prefill, faster decode, ~2 GiB less VRAM at steady state (22.0 vs 24.0). Flip kept.

One trap from that day, recorded in full in the CPU-fallback incident post: the first flip landed while LM Studio’s copy was still loaded in-TTL, the new server spawned into ~1.7 GiB of remainder, and “worked” at pp 4.3 t/s until the TTL cycled it. Offload is decided once, at load, and VRAM residue at spawn time is how you find out.

Anomalies worth keeping

• --fit on is not a budget gate on Vulkan — lazy KV allocation means no up-front reserve, so the flag happily starts a 262k server that will OOM at 191k. Empirical probes are the record.
• Hot-reloading the llama-swap config stops ALL running model servers, not just the changed one. A config edit is a fleet-wide eviction event.
• A flag grid (threads 8/16 × FA × KV f16/q8/q4) moved prefill by ≤3% and decode by ±1.5% — decode on this box is memory-bandwidth-bound and mostly flag-insensitive. The flags that matter are the KV quant (the memory lever) and the spec flags (the decode lever).

A grain of salt

One box, one engine build, one model family, two quants — and the 3-bit quality picture is early evidence from one job shape, not a verdict. The transferable parts: verify MTP heads by file diff, not filename; probe your real context boundary instead of trusting --fit; and remember that on a fixed-VRAM box the quant choice is a context-budget decision first and a quality decision second.

If you’re running 27B-class dense models on 24 GB and know where your ceiling sits — with numbers — post it. Piles of other people’s ceilings are the only way to know what yours should be.