shipfeedAI news, curated daily

16:06:53 CET
20 SEPT16:06:53shipfeed
pull to refreshlast sync
Just in — 30 new
§ local-llm · storyline

Fixes NaN in Metal mul_mm_id when activations exceed f16

llama.cpp fixes Metal mul_mm_id NaN issue when activations exceed f16 range.

Sep 16 · · primary fetch1 sourceupdated Sep 16 ·

metal: fix NaN in mul_mm_id when activations exceed f16 range (#26223) test-backend-ops: reproduce MUL_MAT_ID NaN for activations beyond f16 The Metal mul_mm_id path narrows src1 to `half` for the simdgroup MMA (`S1 = half` in every instantiation; ggml-metal.metal:10582 and :10595, mirrored at :10643/:10654 in the tensor-ops path). f16 saturates at 65504, so a model whose activations exceed that produces inf, and `simdgroup_multiply_accumulate` then turns the whole 8x8 accumulator tile into NaN. The mul_mv_id path used below `ne21_mm_id_min` (32) carries the same values in f32 and is correct, as is every CPU path.

This was untestable before: `init_mul_mat_id_tensors` initializes uniform [-1, 1], so no existing case can drive an operand out of f16 range. `test_mul_mat_id` gains an `amax` parameter (default 1.0f, preserving the historical init exactly) that scales only the f32 activations, leaving the quantized weights in their normal range. Six cases: n=16 sits below the mul_mv_id -> mul_mm_id switch and is the control that must stay green; n=32 and n=64 are above it and fail on Metal today. Two shapes, because this is not model- or size-specific — q4_K at 128 experts / 4 active /…

read full article on github.com
§ sources1 publication · timeline below
  1. github.comllama.cpp b10994primary