TokenSpeed is a brand new inference engine purpose built for speed-of-light agentic workloads
NVIDIA AI(@NVIDIAAI)157 字 (约 1 分钟)
72
TokenSpeed is a new open-source LLM inference engine optimized for agentic workloads, featuring advanced KV caching, an efficient scheduler, and a modular kernel architecture with multi-silicon support.
入选理由：TokenSpeed 实现了媲美 TensorRT-LLM 的性能与接近 vLLM 的易用性。
FeaturedTweet#LLM Inference#NVIDIA#Open Source#KV Cache#Attention Mechanism中英混合