Adds probabilistic sampling to speculative decoding
ONNX Runtime adds probabilistic sampling for speculative decoding with rejection sampling verification.
spec : add probabilistic sampling for simple draft and MTP (#27694) Make the drafter probabilistic and the target verify by rejection sampling Drop stale spec_draft_q before drafting Fallback to argmax sampling for grammar-constrained requests and adding flag for enabling probabilistic draft sampling. Default flag value is greedy. Support grammar-constrained requests in rejection sampling Fix - renormalize distribution after masking copy rng on sampler copy and re-accept drafted tokens on replay Fix draft sampler sharing the target's rng stream Simplify the rejection sampler's inputs and move replay to the server Truncate the draft candidates along with the draft --------- Co-authored-by: praneshgo Co-authored-by: Pranesh Gonegandla Website: Attestations: macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32)…
- github.comllama.cpp b11368primary