OpenRouter 教程：如何在 CI 中用 LLM eval 门禁拦截 Pull Request

AI HOT 精选

AI HOT 精选2026年10月1日

OpenRouter 教程：如何在 CI 中用 LLM eval 门禁拦截 Pull Request

7.1内容质量

TL;DR · AI 摘要

OpenRouter æç¨ï¼å¦ä½å¨ CI ä¸ç¨ LLM eval é¨ç¦æ¦æª Pull Request Â· AIHOT OpenRouterï¼Announcements Â· 2026-1...

核心要点

主题聚焦：OpenRouter 教程：如何在 CI 中用 LLM eval 门禁拦截 Pull Reque
来源：AI HOT 精选，建议结合原文判断细节。
AI 分析暂不可用，本条为保底评分与摘要。

#AI#编程#后端#安全#产品

打开原文

OpenRouter æç¨ï¼å¦ä½å¨ CI ä¸ç¨ LLM eval é¨ç¦æ¦æª Pull Request Â· AIHOT

OpenRouterï¼Announcements

Â·

2026-10-01 08:00

12 å°æ¶å

ç²¾é

AI è¯å

67

OpenRouter æç¨ï¼å¦ä½å¨ CI ä¸ç¨ LLM eval é¨ç¦æ¦æª Pull Request

How to Gate Pull Requests on LLM Evals in CI

AI å¯¼è¯»

æ¨èçç±

æ£æ Â· AI ç¿»è¯

ç®èè¨ä¹

åºå®çè¯ä¼°éæ¯æäº¤å°ä»åºä¸çæµè¯ç¨ä¾åè¡¨ãå®ä» éè¿ç»è¿å®¡æ¥çæåè¯·æ±è¿è¡æ´æ¹ã

å¨ä½ä¸çº§å«èä¸æ¯å·¥ä½æµçº§å«è¿è¡è¿æ»¤ãGitHub å°å if æ¡ä»¶èè·³è¿çä½ä¸æ¥åä¸ºéè¿çæ£æ¥ï¼èå è·¯å¾è¿æ»¤å¨è·³è¿çå·¥ä½æµä¼çä¸å¾ å¤ççå¿ éæ£æ¥å¹¶é»æ¢åå¹¶ã

å½éè¿çä½äºéå¼æ¶ï¼è¯ä¼°èæ¬ä»¥éé¶ç¶æéåºï¼èè¯¥éåºä»£ç æ£æ¯å¯¼è´ä½ä¸å¤±è´¥çåå ã

éè¿å¯¹æªæ´æ¹åæ¯çéå¤è¿è¡æ¥æµééå¼ï¼èä¸æ¯éæ©ä¸ä¸ªä¸¥æ ¼çæ°åã

temperature å seed ä» å¨å°å ¶åå¨ supported_parameters ä¸çæ¨¡åä¸æå¸®å©ãä½¿ç¨å¤æ°æç¥¨çéå¤éæ ·éç¨äºæææ¨¡åã

CI ä¸ç LLM è¯ä¼°æå³çä»ä¹

å¨ä½ è½é¨æ§ä»»ä½ä¸è¥¿ä¹åéè¦ä»ä¹

å¨é¨æ§è½åè¯ä½ ä»»ä½æç¨ä¿¡æ¯ä¹åï¼éè¦åå¤å¥½ä¸ä»¶äºã

ä¸ä¸ªåºå®çãçæ¬åçè¯ä¼°éã Anthropic çä»£çè¯ä¼°æåå»ºè®®å° ä»çå®å¤±è´¥ä¸æåç 20 å° 50 ä¸ªç®åä»»å¡ ä½ä¸ºèµ·å§éãæä»¬è¿éä½¿ç¨ä¸ä¸ªï¼ä»¥ä¾¿ç¤ºä¾ä¿æç®çã

ä¸ç§è¯åæ¹æ³åä¸ä¸ªéå¼ã è¯åæ¹æ³å°ä¸ä¸ªçæ¡è½¬åä¸ºä½ å¯ä»¥è®¡æ°çéè¿æå¤±è´¥ãéå¼éç¨äºæ´ä¸ªè¿è¡ãæ¬æåä½¿ç¨åç¬¦ä¸²æè¨ãè¯åæ åæè¯å¤æ¨¡åä¹å¯ä»¥ã

è¿è¡è¶³å¤å¯éå¤ä»¥å¼å¾ä¿¡èµã é¨æ§åºåºäºåå½èä¸æ¯åªå£°è¿è¡é»æ¢ãä½¿ç¨å¤æ°æç¥¨çéå¤éæ ·éç¨äºæææ¨¡åã temperature å seed ä» å¨ååºå®ä»¬çæ¨¡åä¸æå¸®å©ï¼å æ¤å¨ä¾èµå®ä»¬ä¹åè¯·æ£æ¥ æ¨¡åç supported_parameters ã

ä½ è¿éè¦ä¸ä¸ªå¯¼åºä¸º OPENROUTER_API_KEY ç OpenRouter API å¯é¥ ãNode 20 ææ´æ°çæ¬ï¼ä»¥å jq ã

ååæ¥æå»ºé¨æ§

ä» å¨å¯è½ç ´åä½ çä»£ççæ´æ¹ä¸è§¦åè¯ä¼°ã

å°è¯ä¼°éæ¾å¨ä»åºä¸ï¼ä¸å ¶æµè¯çæç¤ºæ¾å¨ä¸èµ·ã

ç¼å CI ä½ä¸è¿è¡çèæ¬ã

æµéå³å®åå¹¶æ¯å¦è¢«é»æ¢çéå¼ã

ç¬¬ 1 æ¥ï¼ä» å¨ç¸å ³åæ´æ¶è§¦å

ä»¥å·¥ä½æµå¤´é¨åç¬¬ä¸ä¸ª job å¼å§ .github/workflows/eval-gate.yml ã agent: ä¸ååºçè·¯å¾æ¯ä½ è¦ä¸ºèªå·±çä»åºä¿®æ¹çè·¯å¾ã

code

name
:
eval-gate
on
:
pull_request
concurrency
:
group
:
eval-gate-${{ github.ref }}
cancel-in-progress
:
true
jobs
:
changes
:
runs-on
:
ubuntu-latest
outputs
:
agent
:
${{ steps.filter.outputs.agent }}
steps
:
-
uses
:
actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
# v7.0.1
-
uses
:
dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d
# v4.0.3
id
:
filter
with
:
filters
:
|
agent:
- '.github/workflows/eval-gate.yml'
- 'prompts/**'
- 'agents/**'
- 'tools/**/schema.json'
- 'eval-sets/**'
- 'scripts/run-evals.mjs'

è¿éæä¸¤ä»¶äºå¯è½åºéã

ç¬¬ 2 æ¥ï¼å°è¯ä¼°éæ¾å ¥ä»åº

åä¸ªæä»¶å¹¶ææ¾ç½®ã

code

your-repo/
.github/workflows/eval-gate.yml  -> the workflow from Step 1
prompts/support-agent.md         -> the system prompt
eval-sets/support-agent.json     -> the cases that test it
scripts/run-evals.mjs            -> the script from Step 3

code

[
{
"id"
:
"refund-window"
,
"input"
:
"How long do I have to request a refund on a digital download?"
,
"mustMention"
: [
"14 days"
],
"mustNotMention"
: [
"30 days"
]
},
{
"id"
:
"refund-exception"
,
"input"
:
"I bought a download 60 days ago. Can I still get a refund?"
,
"mustMention"
: [
"14 days"
],
"mustNotMention"
: [
"yes, you can"
]
},
{
"id"
:
"escalation"
,
"input"
:
"Your product deleted my files and I want a lawyer."
,
"mustMention"
: [[
"human"
,
"person"
,
"our team"
,
"specialist"
]],
"mustNotMention"
: [
"14 days"
]
}
]

è¿äºç¨ä¾ææµè¯çæç¤ºè¯å°±å¨å®æè¾¹ï¼ä½äº prompts/support-agent.md ä¸ã

code

You are a support agent for a digital downloads store.
The refund window is 14 days from purchase. There are no exceptions to it.
If a customer threatens legal action or reports data loss, hand off to a human
and do not quote the refund policy.
Answer in at most three sentences.

ç¬¬ 3 æ¥ï¼ç¼å CI job è¿è¡çèæ¬

code

import
{ readFileSync }
from
"node:fs"
;
import
{ parseArgs }
from
"node:util"
;
const
{
values
}
=
parseArgs
({
options: {
set: { type:
"string"
, default:
"eval-sets/support-agent.json"
},
prompt: { type:
"string"
, default:
"prompts/support-agent.md"
},
model: { type:
"string"
, default:
"anthropic/claude-sonnet-5"
},
threshold: { type:
"string"
, default:
"0.9"
},
samples: { type:
"string"
, default:
"3"
},
concurrency: { type:
"string"
, default:
"8"
},
},
});
const
systemPrompt
=
readFileSync
(values.prompt,
"utf8"
);
const
cases
=
JSON
.
parse
(
readFileSync
(values.set,
"utf8"
));
const
threshold
=
Number
(values.threshold);
const
samples
=
Number
(values.samples);
const
concurrency
=
Number
(values.concurrency);
// Exit 2 for anything that stops the eval from running, so the job can tell
// "the agent got worse" apart from "the eval could not run".
function
abort
(
message
) {
console.
error
(
`::error::eval could not run: ${
message
}`
);
process.
exit
(
2
);
}
if
(
!
Array.
isArray
(cases)
||
cases.
length
===
0
)
abort
(
`${
values
.
set
} has no cases`
);
if
(
!
(threshold
>
0
&&
threshold
<=
1
))
abort
(
`--threshold must be greater than 0 and at most 1, got "${
values
.
threshold
}"`
);
if
(
!
Number.
isInteger
(samples)
||
samples
<
1
||
samples
%
2
===
0
)
abort
(
`--samples must be a positive odd integer, got ${
values
.
samples
}`
);
if
(
!
Number.
isInteger
(concurrency)
||
concurrency
<
1
)
abort
(
`--concurrency must be a positive integer, got ${
values
.
concurrency
}`
);
class
EvalDidNotRun
extends
Error
{}
async
function
callModel
(
input
) {
const
res
=
await
fetch
(
"https://openrouter.ai/api/v1/chat/completions"
, {
method:
"POST"
,
signal: AbortSignal.
timeout
(
60_000
),
headers: {
Authorization:
`Bearer ${
process
.
env
.
OPENROUTER_API_KEY
}`
,
"Content-Type"
:
"application/json"
,
},
body:
JSON
.
stringify
({
model: values.model,
// Route to one provider only. A different provider serving the same slug
// between runs would look like a prompt regression.
provider: { order: [
"anthropic"
], allow_fallbacks:
false
},
messages: [
{ role:
"system"
, content: systemPrompt },
{ role:
"user"
, content: input },
],
}),
});
if
(
!
res.ok)
throw
new
EvalDidNotRun
(
`${
res
.
status
} ${
await
res
.
text
()
}`
);
const
body
=
await
res.
json
();
return
{ text: body.choices?.[
0
]?.message?.content
??
""
, cost: body.usage?.cost
??
0
};
}
async
function
run
(
input
) {
for
(
let
attempt
=
1
; attempt
<=
3
; attempt
++
) {
try
{
return
await
callModel
(input);
}
catch
(err) {
if
(attempt
===
3
)
throw
new
EvalDidNotRun
(err.message);
await
new
Promise
((
resolve
)
=>
setTimeout
(resolve, attempt
*
2000
));
}
}
}
// A requirement is either a string, or an array meaning "any one of these".
function
matches
(
answer
,
requirement
) {
const
options
=
Array.
isArray
(requirement)
?
requirement
:
[requirement];
return
options.
some
((
option
)
=>
answer.
includes
(option.
toLowerCase
()));
}
function
score
(
text
,
testCase
) {
const
answer
=
text.
toLowerCase
();
const
must
=
testCase.mustMention
??
[];
const
mustNot
=
testCase.mustNotMention
??
[];
return
(
must.
every
((
r
)
=>
matches
(answer, r))
&&
!
mustNot.
some
((
r
)
=>
matches
(answer, r))
);
}
// One unit of work per sample, so the whole matrix runs with a fixed
// concurrency instead of one request at a time.
const
jobs
=
cases.
flatMap
((
testCase
)
=>
Array.
from
({ length: samples }, ()
=>
testCase),
);
const
results
=
new
Map
(cases.
map
((
c
)
=>
[c.id, []]));
let
spend
=
0
;
let
cursor
=
0
;
async
function
worker
() {
while
(cursor
<
jobs.
length
) {
const
testCase
=
jobs[cursor
++
];
const
{
text
,
cost
}
=
await
run
(testCase.input);
spend
+=
cost;
results.
get
(testCase.id).
push
(
score
(text, testCase));
}
}
const
started
=
Date.
now
();
try
{
await
Promise
.
all
(Array.
from
({ length: Math.
min
(concurrency, jobs.
length
) }, worker));
}
catch
(err) {
// A provider timeout is not a quality regression.
abort
(err.message);
}
let
passed
=
0
;
for
(
const
testCase
of
cases) {
const
verdicts
=
results.
get
(testCase.id);
const
majority
=
verdicts.
filter
(Boolean).
length
>
samples
/
2
;
if
(majority) passed
++
;
const
trace
=
verdicts.
map
((
v
)
=>
(v
?
"."
:
"x"
)).
join
(
""
);
console.
log
(
`${
majority
?
"PASS"
:
"FAIL"}  ${
testCase
.
id
}  ${
trace
}`
);
}
const
rate
=
passed
/
cases.
length
;
const
seconds
=
((Date.
now
()
-
started)
/
1000
).
toFixed
(
1
);
console.
log
(
`
\n
pass rate ${
rate
.
toFixed
(
2
)
} against threshold ${
threshold
}`
);
console.
log
(
`${
jobs
.
length
} calls in ${
seconds
}s, cost $${
spend
.
toFixed
(
4
)
} on ${
values
.
model
}`
);
if
(rate
<
threshold) {
console.
error
(
`::error::eval gate failed: ${
passed
}/${
cases
.
length
} cases passed`
);
process.
exit
(
1
);
}

è¯¥èæ¬é¤äºè°ç¨æ¨¡åä¹å¤è¿åä¸ä»¶äºã

å å¨æ¬å°è¿è¡å®ã

code

export
OPENROUTER_API_KEY
=
"sk-or-..."
node
scripts/run-evals.mjs
--samples
3

code

PASS  refund-window  ...
PASS  refund-exception  ...
PASS  escalation  ...
pass rate 1.00 against threshold 0.9
9 calls in <seconds>s, cost $<cost> on anthropic/claude-sonnet-5

ç°å¨å°ç¬¬äºä¸ªä½ä¸æ·»å å° .github/workflows/eval-gate.yml ä¸ï¼ä½äº changes ä½ä¸ä¸æ¹ã

code

eval-gate
:
needs
:
changes
if
:
needs.changes.outputs.agent == 'true'
runs-on
:
ubuntu-latest
timeout-minutes
:
15
steps
:
-
uses
:
actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
# v7.0.1
-
uses
:
actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
# v7.0.0
with
:
node-version
:
'24'
-
name
:
Run fixed eval set
env
:
OPENROUTER_API_KEY
:
${{ secrets.OPENROUTER_API_KEY }}
run
:
node scripts/run-evals.mjs --samples 3 --threshold 0.9

å¨é¨ç¦è½å¤é»æ¢ä»»ä½ä¸è¥¿ä¹åï¼åä¸ä»¶äºã

å¨ Settings > Secrets and variables > Actions ä¸å°ä½ çå¯é¥æ·»å ä¸ºä»åºæºå¯ï¼å½åä¸º OPENROUTER_API_KEY ã

æå¼ä¸ä¸ªè§¦å prompts/ çæåè¯·æ±ï¼ä»¥ä¾¿å·¥ä½æµè¿è¡ä¸æ¬¡ã

å¨ä½ çåæ¯ä¿æ¤è§åä¸å° changes å eval-gate é½æ·»å ä¸º å¿ éç¶ææ£æ¥ ã

ç¬¬ 4 æ¥ï¼æµééå¼

å½ä½ çä¸éå¾ä½æ¶ï¼æé¡ºåºææ¥ä»¥ä¸ä¸é¡¹ã

code

curl
-s
"https://openrouter.ai/api/v1/models"
\
|
jq
-r
'.data[] | select(.id=="anthropic/claude-sonnet-5") | .supported_parameters'

Claude Sonnet 5 ä¸¤è é½æ²¡æååºï¼è¿å°±æ¯ä¸é¢çèæ¬ä¸åéå®ä»¬çåå ãæä»¬ç Claude Sonnet 5 è¿ç§»æå æåºï¼å¯¹äºè¯¥æ¨¡åï¼ temperature ã top_p å top_k ä¼è¢«éé»å¿½ç¥ã2026 å¹´ 9 æ 18 æ¥ï¼ç®å½ä¸ååºäº 445 ä¸ªæ¨¡åï¼å ¶ä¸ 267 ä¸ªåæ¶ååºäº seed å temperature ãå ¶ä½ 178 ä¸ªï¼çº¦å äºåä¹äºï¼åªååºäºå ¶ä¸ä¹ä¸æä¸¤è é½æ²¡æãè¿æ¡å½ä»¤ä¼ç»è®¡åæ¶ååºä¸¤è çæ¨¡åæ°éï¼ä½ å¯ä»¥éæ°è¿è¡å®ä»¥è·å¾å½åæ°åã

code

curl
-s
"https://openrouter.ai/api/v1/models"
|
jq
'
[.data[] | select(.supported_parameters | index("seed") and index("temperature"))] | length'

å¦æä½ çæ¨¡åç¡®å®ååºäºå®ä»¬ï¼å°±æ temperature: 0 å seed: 42 å å ¥è¯·æ±ä½ï¼å¹¶å¨ order æè¾¹è®¾ç½® provider.require_parameters: true ãä½¿ç¨é»è®¤ç require_parameters: false æ¶ï¼å³ä½¿æä¸ªæä¾åä¸æ¯æè¯·æ±ä¸çæ¯ä¸ªåæ°ï¼å®ä»å¯è½æ¶å°è¯·æ±å¹¶å¿½ç¥èªå·±ä¸è®¤è¯çåæ°ãä½¿ç¨ require_parameters: true æ¶ï¼è¯·æ±åªä¼è¢«è·¯ç±å°æ¯æææåæ°çæä¾åã

ç¨ Ori Eval æµè¯è°ç¨å·¥å ·ç agent

Ori Eval æ¯æä»¬ç¨äº agent ç eval æµè¯æ¡æ¶ãæµè¯æ¡æ¶æ¯è¿è¡ eval çç¨åºãå®è¿è¡ agentï¼è®°å½ agent åäºä»ä¹ï¼å¹¶æ ¹æ®ä½ çæè¨æ£æ¥ç»æãOri eval æ¯ .eval.ts æä»¶ï¼æè¨éå¯¹çæ¯ agent åäºä»ä¹ã

code

import
{ test }
from
'bun:test'
;
import
{ assertModelIsLive, setupAgent, setupJudge }
from
'ori/eval'
;
const
MODEL
=
'anthropic/claude-sonnet-5'
;
await
assertModelIsLive
(
MODEL
);
const
agent
=
setupAgent
({ model:
MODEL
});
// Grade with a different model family than the one under test.
const
judge
=
setupJudge
({
agent:
setupAgent
({ model:
'openai/gpt-5-mini'
}),
minScore:
0.8
,
});
test
(
'looks up the order before quoting the refund policy'
,
async
()
=>
{
const
run
=
await
agent.
run
(
'Can I refund order #1234? I bought it 60 days ago.'
);
run.
tool
(
'lookup_order'
).
toBeCalled
();
run.
tool
(
'issue_refund'
).
toNotBeCalled
();
run.
toComplete
();
await
judge.
autoEvals
({
criteria:
'Cites the 14-day window and does not invent exceptions.'
,
run,
});
});

Ori ä½¿ç¨ Bun è¿è¡ eval æä»¶ï¼å½ CI ä¸º true æ¶ï¼å®ä¸ä¼ä¸ºä½ å®è£ Bunãå¨å·¥ä½æµä¸ï¼ä½ å å®è£ Bunï¼ç¶åä¸è½½ä¸ä¸ªåºå®çæ¬ç Ori åå¸å å¹¶å¨è¿è¡åéªè¯å ¶æ ¡éªåï¼å ä¸ºè¯¥ä»»å¡ææä½ ç API keyãè®¾ç½® OPENROUTER_API_KEY åï¼Ori å¨ CI ä¸ä¸éè¦ ori login ã

code

-
uses
:
oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
# v2.2.0
-
name
:
Install Ori
env
:
ORI_RELEASE
:
cli-0.15.0-531912d
ORI_SHA256
:
d2545db7a686f29ebae5bbf7e134d89a409cd00c760c1f24a5f8a88692c5947d
run
:
|
base="https://github.com/OpenRouterLabs/ori-releases/releases/download/$ORI_RELEASE"
curl -fsSL --proto '=https' -o ori "$base/ori-linux-x64"
echo "$ORI_SHA256  ori" | sha256sum -c -
mkdir -p "$HOME/.local/bin"
install -m 0755 ori "$HOME/.local/bin/ori"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
-
name
:
Run pinned agent eval
env
:
OPENROUTER_API_KEY
:
${{ secrets.OPENROUTER_API_KEY }}
run
:
ori eval --report eval-report.md

ori eval æ¥æ¾å½åç®å½ä¸çæ¯ä¸ª *.eval.ts æä»¶å¹¶å°å®ä»¬äº¤ç» bun test ãå®çéåºä»£ç å°±æ¯ bun test çéåºä»£ç ï¼å æ¤ eval å¤±è´¥ä¼å¯¼è´ä»»å¡å¤±è´¥ã --report ä¼åä¸ä»½ Markdown æ¥åï¼ä½ å¯ä»¥å°å ¶ä½ä¸º artifact ä¸ä¼ æè¿½å å°ä»»å¡æè¦ä¸ã

æ¯è¾æ®éèæ¬ä¸ eval å¹³å°

æ¹æ³

å¨ CI ä¸å¦ä½è¿è¡

éå®

æéå

æ®éèæ¬

ä½ èªå·±ç¼å CI æ¥éª¤åéåºä»£ç é»è¾

æ ï¼å ä¸ºè¿æ¯ä½ èªå·±çä»£ç

ä¸å°ä¸¤ä¸ª eval éï¼ä¸å®å ¨æ§å¶è¯å

Ori Eval

å¨ä»»å¡ä¸è¿è¡

code

ori eval

ï¼å½ eval å¤±è´¥æ¶ä»¥éé¶éåº

å·¥å ·è°ç¨ agentï¼æå¨ä½ èªå·±ç agent ä¸æ¯è¾æ¨¡å

DeepEval

å¨ pytest ä¸è¿è¡

code

deepeval test run

ï¼å½ä½äºæ¯ä¸ªææ çéå¼æ¶

code

assert_test()

ä¼æåºå¼å¸¸

ä½ï¼å ä¸ºè¯ååºæ¯å¼æºç

ç°æçææ ï¼å¦çæ¡ç¸å ³æ§åä»»å¡å®æåº¦

Braintrust

ä¸ä¸ªå·²åå¸ç GitHub Actionï¼è¿è¡ eval å¹¶å¨ pull request ä¸åå¸æè¦è¯è®º

ä¸çï¼å ä¸ºè¯ååå²åå¨äºä»ä»¬çå¹³å°ä¸

å¨è¯å®¡ä¸èé CI æ¥å¿ä¸åç°åæ°åå

Arize

ä» SDK ä½ä¸ºæ®é Python æ¥éª¤è¿è¡

code

client.experiments.run()

ï¼å ¶ææ¡£ä¸æä¸ä¸ªç¤ºä¾å·¥ä½æµ

ä¸çï¼å ä¸º experiments API æ¯ä»ä»¬ç

å·²ç»ä½¿ç¨ Arize è¿è¡å¯è§æµæ§çä»åº

Galileo

ä» SDK è¿è¡

code

run_experiment

ï¼æå¯¹å¤è½® agent ä½¿ç¨

code

create_experiment

ä¸çï¼å ä¸ºææ ååå²æ¯å¹³å°åçç

åºäº eval åå²çæç®¡ä»ªè¡¨ç

å¸¸è§å¤±è´¥æ¨¡å¼

å¸¸è§é®é¢

å¦ä½å° LLM è¯ä¼°æ·»å å° CI/CD æµæ°´çº¿ï¼

ä½ è½æ ¹æ®è¯ä¼°åæ°é»æ¢æåè¯·æ±åå¹¶åï¼

LLM-as-a-judge ä¸å¨ CI ä¸è¿è¡è¯ä¼°æä»ä¹åºå«ï¼

ä» é ä¸ä¸ªå¸¦éå¼æ£æ¥çæ®éèæ¬å°±è¶³å¤äºãOri EvalãDeepEvalãBraintrustãArize å Galileo å¨ç¸åçéåºç æ¨¡å¼ä¹ä¸å¢å äºæ¥åãè¿è¡åå²æéå¯¹æºè½ä½çæè¨ã

å¦ä½å¤çä¸ç¨³å®æéç¡®å®æ§çè¯ä¼°é»å¡äºä¸ä¸ªå¥½ç PRï¼

ç»è®º

åèèµæ

Ori Eval

æä¾åéæ©

æ¨¡åä¸ supported_parameters

ç¨éæ ¸ç®

Claude Sonnet 5 è¿ç§»æå

æå¼ AI æºè½ä½è¯ä¼°çç¥ç§é¢çº±

GitHub Actions çå·¥ä½æµè¯æ³

è·³è¿å·¥ä½æµè¿è¡

ä½¿ç¨æ¡ä»¶æ§å¶ä½ä¸æ§è¡

å¨ GitHub Actions ä¸ä½¿ç¨å¯é¥

å ³äºåä¿æ¤åæ¯

å ³äºä»£ç ææè

dorny/paths-filter

æ¥æºï¼ OpenRouterï¼Announcements Â· openrouter.ai

#

æç¨/å®è·µ

é¨ç½²/å·¥ç¨

Agent

ç¼ç

OpenRouter 教程：如何在 CI 中用 LLM eval 门禁拦截 Pull Request

TL;DR · AI 摘要

核心要点

OpenRouter æç¨ï¼å¦ä½å¨ CI ä¸­ç¨ LLM eval é¨ç¦æ¦æª Pull Request Â· AIHOT

OpenRouter æç¨ï¼å¦ä½å¨ CI ä¸­ç¨ LLM eval é¨ç¦æ¦æª Pull Request

ç®èè¨ä¹

CI ä¸­ç LLM è¯ä¼°æå³çä»ä¹

å¨ä½ è½é¨æ§ä»»ä½ä¸è¥¿ä¹åéè¦ä»ä¹

ååæ­¥æå»ºé¨æ§

ç¬¬ 1 æ­¥ï¼ä» å¨ç¸å ³åæ´æ¶è§¦å

ç¬¬ 2 æ­¥ï¼å°è¯ä¼°éæ¾å ¥ä»åº

ç¬¬ 3 æ­¥ï¼ç¼å CI job è¿è¡çèæ¬

ç¬¬ 4 æ­¥ï¼æµééå¼

ç¨ Ori Eval æµè¯è°ç¨å·¥å ·ç agent

æ¯è¾æ®éèæ¬ä¸ eval å¹³å°

å¸¸è§å¤±è´¥æ¨¡å¼

å¸¸è§é®é¢

å¦ä½å° LLM è¯ä¼°æ·»å å° CI/CD æµæ°´çº¿ï¼

ä»ä¹æ¯åºå®è¯ä¼°éï¼ä¸ºä»ä¹å®éè¦ä¸ä»£ç ä¸èµ·ä¿æçæ¬åï¼

ä½ è½æ ¹æ®è¯ä¼°åæ°é»æ­¢æåè¯·æ±åå¹¶åï¼

LLM-as-a-judge ä¸å¨ CI ä¸­è¿è¡è¯ä¼°æä»ä¹åºå«ï¼

è¯ä¼°éè¦æ¯æ¬¡æäº¤é½è¿è¡ï¼è¿æ¯åªå¨æç¤ºè¯åæºè½ä½é»è¾åæ´æ¶è¿è¡ï¼

å¨æ¯ä¸ª PR ä¸è¿è¡ LLM è¯ä¼°å¨ token å CI åéæ¹é¢è¦è±å¤å°é±ï¼

æåªäºå·¥å ·æ¯æå¨åå¹¶ååºäºåºå®è¯ä¼°éå¯¹ PR è¿è¡é¨ç¦ï¼

å¦ä½å¤çä¸ç¨³å®æéç¡®å®æ§çè¯ä¼°é»å¡äºä¸ä¸ªå¥½ç PRï¼

LLM ä½ä¸ºè¯å¤è æ¯å¦è¶³å¤å¯é ï¼å¯ä»¥æ®æ­¤è®©æå»ºå¤±è´¥ï¼

ç»è®º

åèèµæ

OpenRouter æç¨ï¼å¦ä½å¨ CI ä¸ç¨ LLM eval é¨ç¦æ¦æª Pull Request Â· AIHOT

OpenRouter æç¨ï¼å¦ä½å¨ CI ä¸ç¨ LLM eval é¨ç¦æ¦æª Pull Request

ç®èè¨ä¹

CI ä¸ç LLM è¯ä¼°æå³çä»ä¹

å¨ä½ è½é¨æ§ä»»ä½ä¸è¥¿ä¹åéè¦ä»ä¹

ååæ¥æå»ºé¨æ§

ç¬¬ 1 æ¥ï¼ä» å¨ç¸å ³åæ´æ¶è§¦å

ç¬¬ 2 æ¥ï¼å°è¯ä¼°éæ¾å ¥ä»åº

ç¬¬ 3 æ¥ï¼ç¼å CI job è¿è¡çèæ¬

ç¬¬ 4 æ¥ï¼æµééå¼

ç¨ Ori Eval æµè¯è°ç¨å·¥å ·ç agent

æ¯è¾æ®éèæ¬ä¸ eval å¹³å°

å¸¸è§å¤±è´¥æ¨¡å¼

å¸¸è§é®é¢

å¦ä½å° LLM è¯ä¼°æ·»å å° CI/CD æµæ°´çº¿ï¼

ä»ä¹æ¯åºå®è¯ä¼°éï¼ä¸ºä»ä¹å®éè¦ä¸ä»£ç ä¸èµ·ä¿æçæ¬åï¼

ä½ è½æ ¹æ®è¯ä¼°åæ°é»æ¢æåè¯·æ±åå¹¶åï¼

LLM-as-a-judge ä¸å¨ CI ä¸è¿è¡è¯ä¼°æä»ä¹åºå«ï¼

è¯ä¼°éè¦æ¯æ¬¡æäº¤é½è¿è¡ï¼è¿æ¯åªå¨æç¤ºè¯åæºè½ä½é»è¾åæ´æ¶è¿è¡ï¼

å¨æ¯ä¸ª PR ä¸è¿è¡ LLM è¯ä¼°å¨ token å CI åéæ¹é¢è¦è±å¤å°é±ï¼

æåªäºå·¥å ·æ¯æå¨åå¹¶ååºäºåºå®è¯ä¼°éå¯¹ PR è¿è¡é¨ç¦ï¼

å¦ä½å¤çä¸ç¨³å®æéç¡®å®æ§çè¯ä¼°é»å¡äºä¸ä¸ªå¥½ç PRï¼

LLM ä½ä¸ºè¯å¤è æ¯å¦è¶³å¤å¯é ï¼å¯ä»¥æ®æ¤è®©æå»ºå¤±è´¥ï¼

ç»è®º

åèèµæ