@marcelroed casually just dropped the world’s fastest tokenizer implementation! He's also co-author ...
Stanford AI Lab(@StanfordAILab)132 字 (约 1 分钟)
85
斯坦福AI实验室发布Gigatoken分词器，性能比HuggingFace快500-1000倍，比OpenAI的tiktoken快100倍。
入选理由：Gigatoken性能比HuggingFace快500-1000倍，比tiktoken快100倍
FeaturedTweet#分词器#自然语言处理#性能优化#Rust#斯坦福中英混合