shipfeedAI news, curated daily

00:06:16 CET
9 OCT00:06:16shipfeed⋯
pull to refreshlast sync
Just in — 30 new
§ local-llm · storyline

Adds probabilistic sampling to speculative decoding

ONNX Runtime adds probabilistic sampling for speculative decoding with rejection sampling verification.

Oct 3 · · primary fetch1 sourceupdated Oct 3 ·

spec : add probabilistic sampling for simple draft and MTP (#27694) Make the drafter probabilistic and the target verify by rejection sampling Drop stale spec_draft_q before drafting Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy. Support grammar-constrained requests in rejection sampling Fix - renormalize distribution after masking copy rng on sampler copy and re-accept drafted tokens on replay Fix draft sampler sharing the target's rng stream Simplify the rejection sampler's inputs and move replay to the server Truncate the draft candidates along with the draft --------- Co-authored-by: praneshgo Co-authored-by: Pranesh Gonegandla Website: Attestations: macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32)…

read full article on github.com ↗
§ sources1 publication · timeline below
  1. github.comllama.cpp b11368primary