shipfeedAI news, curated daily

01:17:03 CET
26 JUL01:17:03shipfeed
pull to refreshlast sync
Just in — 30 new
§ local-llm · storyline

Adds CUDA support for quantized embedding lookups

Llama.cpp adds CUDA support for quantized embedding lookups including k-quants, i-quants, and mxfp4.

Jul 22 · · primary fetch1 sourceupdated Jul 22 ·

cuda: GET_ROWS quants (#25962) cuda: add k-quant support to GET_ROWS Device-side embedding lookups require GET_ROWS to handle the k-quants used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without it the backend rejects the op and the scheduler falls back to the host, copying the full embedding matrix back on every token in single-device graphs. Factor the super-block dequantizers out of the dequantize_block kernels in convert.cu into shared device functions in dequantize.cuh and reuse them from a new k_get_rows_kq kernel : one thread block dequantizes one (dst row, super-block) pair with the existing thread layouts, 32 threads for q4_K and 64 for the other k-quants.

Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left as a TODO. cuda: add i-quant support to GET_ROWS Extends the shared super-block dequantizers to the nine i-quants and reuses them from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernels. supports_op gates the k-quant and i-quant path on ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its own (QK4_NL sub-blocks). mxfp4 is left as a TODO. cuda: add mxfp4 support to GET_ROWS Moves the mxfp4…

read full article on github.com
§ sources1 publication · timeline below
  1. github.comllama.cpp b10089primary