Welcome Le Chonk: Mistral Large 4 on Hugging Face

pip install --upgrade transformers

import os
from transformers import AutoProcessor, AutoModelForMultimodalLM, DistributedConfig

distributed_config = DistributedConfig(
    tp_size=int(os.environ["WORLD_SIZE"]),
    ep_size=int(os.environ["WORLD_SIZE"]),
)

model_id = "mistralai/Mistral-Large-4-1T-A52B"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    distributed_config=distributed_config,
)
# 仅文本
text_messages = [
    {"role": "user", "content": "Who are you?"}
]
# 文本+图像
multimodal_messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg"},
            {"type": "text", "text": "How many cats are in the image?"}
        ],
    },
]
for messages in [text_messages, multimodal_messages]:
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        reasoning_effort="medium",
    ).to(model.device).to(model.dtype)
    input_len = inputs["input_ids"].shape[-1]

    outputs = model.generate(**inputs, max_new_tokens=100)
if int(os.environ["RANK"]) == 0:
    print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))

torchrun --nproc-per-node 8 script.py

vllm serve mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --config-format mistral \
  --load-format mistral \
  --tensor-parallel-size 8 \
  --decode-context-parallel-size 8 \
  --enable-expert-parallel \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --enforce-eager \
  --mm_processor_cache_gb 0.0

curl -s http://127.0.0.1:8000/v1/chat/completions -H
'Content-Type: application/json'
\
  -d
'{"messages":[{"role":"user","content":"What is 17*23?"}],"max_tokens":512}'

curl -LsSf https://llama.app/install.sh | sh

llama serve -hf ggml-org/Mistral-Large-4-1T-A52B-GGUF:Q2_K

curl http://localhost:8080/v1/chat/completions \
    -H
"Content-Type: application/json"
\
    -d
'{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a limerick about python exceptions"}
]
}'

curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H
"Content-Type: application/json"
\
    -H
"Authorization: Bearer <your-key>"
\
    -d
'{
"model": "muse-spark-open",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,..."
}
}
]
}
],
"temperature": 0
}'

pip install --upgrade transformers trl peft

# torchrun --nproc-per-node 8 sft.py
import
os
from
datasets
import
load_dataset
from
peft
import
LoraConfig
from
transformers
import
AutoModelForImageTextToText, DistributedConfig
from
trl
import
SFTConfig, SFTTrainer

world_size =
int
(os.environ[
"WORLD_SIZE"
])
model_id =
"mistralai/Mistral-Large-4-1T-A52B"
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    distributed_config=DistributedConfig(fsdp_size=world_size, ep_size=world_size),
    dtype=
"auto"
,
)

</think>
<|endoftext|>Human: 请将以下 Markdown 文章翻译为中文。直接返回翻译后的 Markdown，不要添加任何额外说明。

### Llama.cpp

Llama.cpp and Llama App already support Mistral Large 4.

You can install Llama App with the following command:

Then you can start the server as follows, indicating the quant suffix after the model ID:

Here’s the repository with the quants derived from the FP8 checkpoint, and this is the NVFP4 one , converted from the official NVFP4 repo.

Once the server has started, you can head to localhost:9931 to chat using the built-in WebUI. From the web UI you can add images for inference and adjust the reasoning lever. You can also query the server as follows. Feel free to check out the API reference for multimodality, tool calls and more.

You can run image-text inference as follows.

The llama.cpp team ships the following quants: IQ2_XXS , Q2_K , Q4_K_M , NVFP4 , Q8_0 , and BF16 for further conversions. The 2-bit quants run on a single Apple Silicon Ultra computer with 512 GB of unified RAM. The others require distributed inference. You can also run on multiple NVIDIA or AMD cards.

Running Q2_K on a single M3 Ultra, we got a prefill throughput of about 90 tokens per second, and a generation speed of 15 tok/s. On 4 x RTX6000 Pro, we observed the following results:

- NVFP4: 32K context size, prefill 130-590 t/s, gen speed ~10 t/s

- IQ2_XXS: 800K context size. At short context sizes, prefill is 500 tok/s and generation runs at 30 tok/s. However, using long contexts performance drops to 129 tok/s (prefill) and 3 tok/s (generation).

## Fine-tuning with TRL

We also ship day 0 support of fine tuning Mistral Large 4 using TRL. All you have to do is upgrade to the latest version of transformers , trl and peft .

In the following example we train the model to perform multilingual thinking. The fine tuned model would be able to reason in the language the system prompt asks for!

The script runs on the FP8 variant on a node of 8xB300s.

curl -s http://127.0.0.1:8000/v1/chat/completions -H
'Content-Type: application/json'
\
  -d
'{"messages":[{"role":"user","content":"What is 17*23?"}],"max_tokens":512}'

curl -LsSf https://llama.app/install.sh | sh

llama serve -hf ggml-org/Mistral-Large-4-1T-A52B-GGUF:Q2_K

curl http://localhost:8080/v1/chat/completions \
    -H
"Content-Type: application/json"
\
    -d
'{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a limerick about python exceptions"}
]
}'

curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H
"Content-Type: application/json"
\
    -H
"Authorization: Bearer <your-key>"
\
    -d
'{
"model": "muse-spark-open",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,..."
}
}
]
}
],
"temperature": 0
}'

pip install --upgrade transformers trl peft

# torchrun --nproc-per-node 8 sft.py
import
os
from
datasets
import
load_dataset
from
peft
import
LoraConfig
from
transformers
import
AutoModelForImageTextToText, DistributedConfig
from
trl
import
SFTConfig, SFTTrainer

world_size =
int
(os.environ[
"WORLD_SIZE"
])
model_id =
"mistralai/Mistral-Large-4-1T-A52B"
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    distributed_config=DistributedConfig(fsdp_size=world_size, ep_size=world_size),
    dtype=
"auto"
,
)

## 推理端点

Mistral Large 4 也可通过推理端点使用，附带可在 8 x NVIDIA H200 GPU 上部署的现成配方。该配方使用 vLLM 并通过张量和专家并行将 FP8 权重分布在所有八块 GPU 上，利用模型内置的 DSpark 草稿头通过推测解码实现更快的生成。它支持最多 131K 个上下文令牌，并同时处理最多 16 个请求，我们观察到以下结果：

- 单个请求：对于 1K 令牌提示，第一个令牌耗时约 0.6 秒，生成速度为 ~65 tok/s，在 DSpark 草稿被接受的聊天提示中可达 120–138 tok/s。
- 长提示：16K 令牌提示，每次四个，总预填充速度约为 4,300 tok/s，第一个令牌耗时 8–9 秒。
- 高负载：在 1,000 个 ShareGPT 对话中，总生成速度约为 480 tok/s。

您可以使用 hf CLI 部署它如下：

一旦端点运行，您可以使用端点 URL 和 Hugging Face 令牌查询其 OpenAI 兼容 API。`reasoning_effort` 字段控制模型在回答前进行推理的程度，可设置为 low、medium 或 high（默认）。

答案在 `choices[0].message.content` 中，推理内容在 `choices[0].message.reasoning` 中。该端点还支持工具调用、结构化输出、图像输入和通过标准 Chat Completions 字段进行流式传输。

## 戴尔企业中心

使用戴尔 AI 服务器（NVIDIA H200、B200、GB200 或 GB300）最简单的方法是安装 dell-ai CLI：

然后使用它在例如 NVIDIA B300 节点上运行 Mistral Large 4 的优化配方。

有关此发布的更多信息，请参阅 Dell 企业中心（DEH）。

## 试用一下

从 Mistral Large 4 集合中获取权重，使用 GGUF 本地运行它们，或通过推理端点部署它们。我们迫不及待想看到您使用 Le Chonk 构建的内容。在 Hub 上分享您的微调、量化和其他作品！

## 本文提到的模型 3

## 本文中提到的集合 1

更多文章来自我们的博客

公告

多模态

transformers

## 欢迎使用 Thinking Machines 的 Inkling

- +1

168

2026年7月15日

vllm

推理

## 原生速度 vLLM transformers 模型后端

77

2026年7月8日

### 社区

编辑

预览

通过在文本输入框中拖放、粘贴或点击此处上传图片、音频和视频

轻点或粘贴此处以上传图片

评论

· 注册或登录以发表评论