gguf-mtp-graft

Graft an MTP (multi-token-prediction / nextn) head from a donor GGUF onto a base GGUF of the same architecture, so the base can drive llama.cpp's self-speculative decoding (--spec-type draft-mtp) without a separate draft model.

Why

Fine-tunes are frequently published without the MTP head their base model ships, because the head isn't part of what was trained. But the head reads hidden states that a fine-tune often barely moves, so the base model's head can transfer essentially intact — you get the decode speedup for free, at unchanged output quality.

The head is one extra transformer block (attention, shared + routed experts) plus a handful of nextn projections and norms. Grafting means appending those tensors as blk.<n_block>.* and patching two metadata keys. Quantised data is copied byte-for-byte; nothing is dequantised or re-quantised.

Safety of the operation

MTP is speculative. A head that transfers badly lowers draft acceptance — so you get less speedup, or none — but it cannot change which tokens the model emits. The failure mode is "no faster", not "subtly wrong".

Verify accordingly:

1. Greedy output must be token-identical with and without --spec-type draft-mtp.
2. Then look at draft acceptance to decide whether the graft earned its keep.

Usage

gguf-py must be importable — either pip install gguf, or point at a llama.cpp checkout:

export PYTHONPATH=/opt/llama.cpp/gguf-py

Look at what a donor actually carries:

./graft_mtp.py inspect --gguf Qwen3.6-35B-A3B-UD-Q4_K_M_MTP.gguf

Validate a pairing without writing 20+ GB:

./graft_mtp.py graft \
  --base  Ornith-1.0-35B-UD-Q4_K_M.gguf \
  --donor Qwen3.6-35B-A3B-UD-Q4_K_M_MTP.gguf \
  --out   Ornith-1.0-35B-UD-Q4_K_M-MTP.gguf \
  --dry-run

Drop --dry-run to write it. Output size ≈ base + head.

What gets checked

The graft refuses to run unless:

• general.architecture matches between base and donor
• embedding_length, expert_count, head counts and FFN length match where both declare them
• the base declares no existing nextn layers
• the donor's head starts exactly at the base's block_count (same non-head depth)
• no head tensor name collides with a base tensor

Choosing a donor

Prefer the model the target was actually fine-tuned from. A sibling fine-tune of the same architecture also works — the hidden states are close enough — but the true parent is the closer match.

Donor quantisation matters more than the file's name suggests. Check with inspect: a good dynamic quant keeps the head's attention and projections at Q8_0, norms at F32, and MoE routers at BF16, quantising only the routed experts. A head where the projections got dropped to 4-bit will draft worse.

Worked example

Ornith-1.0-35B (an agentic-coder fine-tune of Qwen3.6-35B-A3B) ships from unsloth without an MTP head. Its parent does have one. Both are qwen35moe, 40 blocks, embedding_length 2048, expert_count 256.

The donor head is 20 tensors / 504 MiB:

Component	Type
attn q/k/v/output, shared experts, nextn.eh_proj	Q8_0
all norms (enorm, hnorm, shared_head_norm, …)	F32
MoE routers (ffn_gate_inp)	BF16
256 routed experts (gate/up, down)	Q4_K / Q5_K

Grafted, the result is ~504 MiB larger than the base — small enough that a VRAM-tuned llama.cpp config (offload split, context size, KV quant) generally carries over unchanged.

Notes

• Peak RSS is modest: tensor data is memory-mapped and streamed. The metadata replay holds the tokeniser arrays in memory briefly.
• Split/sharded GGUFs are not handled; join them first.
• Only the last nextn_predict_layers blocks are treated as head, matching how llama.cpp lays out MTP GGUFs.