shipfeedAI news, curated daily

11:18:50 CET
23 SEPT11:18:50shipfeed
pull to refreshlast sync
Just in — 30 new
§ local-llm · storyline

Fixes DFlash speculative decoding for HunyuanOCR

HunyuanOCR fixes DFlash speculative decoding by registering layer-input tensors.

yesterday · · primary fetch1 sourceupdated yesterday ·

spec : support DFlash for HunyuanOCR (#28890) model : add DFlash layer-input taps for HunyuanVL DFlash speculative decoding needs the target graph to expose the residual stream entering each layer (res->t_layer_inp[il]) - the draft model reads those tensors to build its cross-context. Qwen3 and the other DFlash-capable targets register them, but the Hunyuan graphs do not, so serving a DFlash draft against a HunyuanOCR target aborts during the first graph build: GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null") Register the tensor at the top of the layer loop, mirroring qwen3.

The layer input is the residual stream entering layer il, i.e. the output of layer il-1, which is what the draft's target_layers metadata refers to (the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this graph, so it is covered as well; hunyuan-moe has a separate graph and is untouched. The vector is only read when a speculative implementation enables those layer ids, so there is no behaviour change without a draft model. Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now run, draft acceptance is ~0.5 and the OCR output is byte-identical to the…

read full article on github.com
§ sources1 publication · timeline below
  1. github.comllama.cpp b11103primary