John Shahbazian

Testing SubQ Snap on Two Jobs: Picking Tools and Grading Answers

Mon, 12 Oct 2026 00:00:00 -0400

SubQ Snap is a hosted decision model: you send it some context, instructions, and a fixed set of options, and it returns one option plus a probability for each. No prose, nothing to parse. That is the same shape as Jev, the open JevK5 models, and OpenAI’s Decisions API, all of which I have tested here as answer graders. I ran Snap through two jobs: having an agent pick the right tool for a request, and grading a model’s answer against a known fact. It did very well at one and not the other.

Short answer: it was the best tool picker we have measured, and a middling answer grader. Every tool decision in a 250-case test came back right, with a median response time of 155 ms and a total cost under a cent. On the 100 hand-graded answers, it agreed with the human labels 84% of the time without the source document and 86% with it. JevK5-4B, our preferred local grader, scored 90% and 88% on the exact same requests. The rest of this post is about why I think both of those things are true at once, and what I am actually going to do with them.

A note on how the tests were run, since it matters for what follows. Nothing was changed for Snap. We kept the datasets and request formats from earlier runs, added Snap as one more model, and read every competitor’s number from its saved, audited record. No prompt edits after seeing scores, no retries to nudge a result. Both experiments passed independent audits.

The tool-picking test

The dataset is the public BFCL V4 Live routing adaptation. It has 250 requests. In 150 of them the model sees several tools and one is the right one; in fifty it sees exactly one tool, and that tool applies; in the last fifty no offered tool applies at all. For each request the model gets the user’s message and the tool descriptions and picks from four to seven options, three of which are non-tool actions: say no tool is needed, ask a clarifying question, or say it cannot help.

Here are the rows that matter from a fourteen-model comparison:

Model	Agreement	Unnecessary tool calls (of 50)	Median response time
SubQ Snap Preview	100.0%	0	155 ms
JevK5-9B	98.8%	3	239 ms
Tura-S1-4B	98.4%	4	373 ms
JevK5-4B MLX	97.6%	2	168 ms
OpenAI Decisions, gpt-6-luna	94.8%	0	162 ms
Jeff 0.8B	94.8%	1	51 ms

Snap picked the right tool on all 200 cases where a tool applied, and never reached for a tool on the fifty where none did. That is what the 100% means, and it is worth being precise about, because there is a looser and a stricter way to read those fifty negative cases.

The dataset accepts any of no_tool, clarify, or cannot_answer on a negative case. Snap chose no_tool, the canonical label, on 27 of them and one of the other two permitted actions on 23. If you insist on the canonical label, Snap’s agreement drops to 90.8%, and Tura edges it at 92.0%. So “perfect” here means it never called a tool it should not have and never missed one it should have. It does not mean it always chose the best way to say no. If your product cares whether the agent asks a follow-up question versus answering directly versus declining, that is a separate test and this one does not cover it.

The margins are also smaller than the table makes them look. Snap beat JevK5-4B MLX by six judgments, Tura by four, and the 9B by three. Against the 4B that is +2.4 percentage points with a paired 95% interval of +0.8 to +4.4; against the 9B the interval includes zero. These intervals are descriptive and not adjusted for the fact that I am making several comparisons at once. The catalogues are small and public, so I cannot rule out training overlap, and the test says nothing about filling in tool arguments, executing the call, large tool inventories, or finishing an actual multi-step task. Still: every applicable tool, zero wrong calls, 155 ms, fractions of a cent. That earned a second test.

The grading test

Human100 is the set I keep coming back to. It is 100 cases from one document corpus, each with a question, a reference fact, a candidate answer, the entities that have to appear, and a rubric. A person graded each one: 67 accepted, 33 rejected. The judge returns correct, incorrect, or partial; only correct counts as accepting the answer. We run two versions of each case, one with just the fact and rubric and one that also includes the original source text, and we run each version twice to check that the model is stable. For Snap that was 400 judgments. The repeats stay clustered by case in the statistics; they do not turn 100 cases into 400.

Model	Without source	With source	Median response time (without / with)
Jev	90.0%	88.5%	438 / 461 ms
JevK5-4B MLX	90.0%	88.0%	176 / 923 ms
JevK5-4B Q8	89.0%	88.0%	162 / 998 ms
Decider-4B v2.1	87.0%	88.0%	167 / 1,212 ms
SubQ Snap Preview	84.0%	86.0%	154 / 211 ms
OpenAI Decisions, gpt-6-luna	83.0%	82.0%	135 / 136 ms

Snap marked eight of the 33 human-rejected answers as correct in both versions. It marked eight of the 67 human-approved answers wrong without the source text and six with it. Both repeat passes returned identical verdicts on every case.

The error pattern is more interesting than the totals. Without the source text, Snap and JevK5-4B MLX let through the same number of bad answers. The whole six-point gap comes from Snap rejecting good ones: eight distinct cases versus two. With the source text, Snap let through one fewer bad answer than MLX and rejected three more good ones. Those two kinds of mistakes cost different things in a real pipeline, and a single agreement percentage hides which one you are buying.

I would not read the gap as a ranking, either. Snap minus MLX is −6 points without the source, with a 95% interval of −12 to 0, and −2 points with it, with an interval of −9 to +5. On a reused set of 100 cases that is “probably a bit worse, maybe not”. It is also not a reason to switch, which is the question I actually had.

Why the same model would be great at one and ordinary at the other

My working guess is that these two jobs reward different kinds of judgment, and the shared interface disguises that.

Picking a tool from a small catalogue is mostly matching: does this request line up with one of these capabilities, or with none of them? Grading an answer is checking whether specific facts, entities, and conditions survived intact, and deciding whether what got dropped was important enough to fail the answer.

A toy version: deciding that a request should go to the returns-policy lookup tool is easy. Grading the answer that comes back means noticing that “30 days after purchase” is not the same as the reference policy’s “30 days after delivery”. The answer is relevant, fluent, on topic, and wrong. A typed-choice API does not do that noticing for the model; the model still has to.

I want to be clear that this is a hypothesis. We did not isolate anything about Snap’s architecture or training data, the two datasets differ in domain and in how they are scored, and tool selection presumably gets much harder as catalogues grow and tools start to overlap. The honest version of the claim is: on these two tests, the ordering flipped, and I have a plausible story for why.

Speed and money

Even where Snap lost on accuracy, it was quick. With the source text included, its median response time was 211 ms against 923 ms for the JevK5-4B MLX recipe we measured, about 4.37 times faster. Those requests ran roughly 2,300 to 4,300 input tokens, which is real document handling but well short of the long context SubQ advertises. The local timings were on an M5 Max with 128 GB of memory, in different sessions from the hosted ones, so this compares two practical setups rather than isolating anything about the model, and it says nothing about throughput under load.

At SubQ’s published pricing of $0.05 per million input tokens with free output, the entire tool-picking experiment including controls cost an estimated $0.002835. Human100 including controls was $0.032868. Both together: about 3.6 cents. I like that number mostly because it means trying a new model on our full test set is now a rounding error, whether or not the model wins.

One thing that did not pan out was using Snap’s probabilities as a filter. We have a standing rule from earlier posts: if a grader’s top probability clears 0.95, let the verdict stand, otherwise hand the case to a larger model. That screen kept only 21% of the cases without the source text and 12% with it. The 21% still included a wrong answer marked correct. The 12% was error-free but amounted to twelve unique cases, which is nowhere near enough to certify a production cutoff. We did not fit a new threshold and nothing in production changed.

Where this leaves things

For the next round of agent tool-selection work, Snap goes in as the hosted model to beat. For grading answers against documents, I am going with JevK5-4B MLX, same as before, and nothing here moved me off it. What would change that is a test on new cases from more than one corpus, with the cost of each kind of error in our pipeline written down first.

The general point is that a shared interface does not make the tasks behind it interchangeable. Snap, JevK5, and the OpenAI endpoint all take the same kind of request and return the same kind of answer, and the ordering between them still flipped from one job to the other. A result on one of these tasks says little about the other; each needs its own test.

Testing OpenAI's Decisions API Against Jev and JevK5-4B

Tue, 06 Oct 2026 00:00:00 -0400

Setup: Over the past few weeks I have written up TypeSafe’s Jev as that grader, Jev as a first-pass filter in front of Claude, and six open models that copy Jev’s interface. Until this week Jev was the only hosted service of its kind from a vendor we would consider. OpenAI has now released its Decisions API, which does the same job: you send it input and a typed question, and it returns a decision and probabilities instead of prose. We ran it through the same 100 hand-graded cases as everything else.

Short version

OpenAI Decisions with gpt-6-luna agreed with the human accept/reject labels 83% of the time without the source text and 82% with it. Our Jev run scored 90% and 88.5% on the same cases; JevK5-4B, the open model we picked last time, scored 89% and 88%. The OpenAI result is faster than anything we have measured (median 135 ms including the network round trip) and cheap (all 400 judgments cost about $0.063 at list price), but it rejected three to four times as many human-approved answers as Jev or JevK5-4B, and it let through about as many bad ones. It is a useful hosted speed and cost result. It does not replace Jev or JevK5-4B for this grading task.

Background

The Decisions API is a separate endpoint, POST /v1/decisions, with a request made of three parts: a model (currently only gpt-6-luna), an input (text, images, or both), and a list of questions. Each question has a type: predicate (true or false, returns a probability), choice (pick one of your values, returns the choice, a probability per option, and a confidence), or score (rate against ordered levels, returns a weighted average). Nothing is generated, so you pay only for input, at $0.10 per million tokens. There is no explanation in the response. If you have read the earlier Jev post, this is the same shape as Jev’s Noul, Choice, and Score questions.

Our answer judge is one choice question named verdict with the options correct, incorrect, and partial, in that order, with the rubric in the instructions. That is the same question we sent to Jev and to every open model, serialized the same way; the request hashes match the ones from the JevK5-4B run.

The docs describe the service as a public beta. Responses reported gpt-6-luna and no pinned version, so a rerun next month may not return the same answers.

Method

The test set is Human100: 100 document-grounded recall cases, frozen, with 67 answers a person accepted and 33 a person rejected. As before, each case was sent in two setups:

J0: the question, the known fact, the candidate answer, and the named entities, plus the rubric.
E1: the same, plus the original source text.

Each setup was run twice, so each cell below is 200 judgments over the same 100 cases. The second pass checks that the service is stable; it does not add independent cases. The score is accept/reject agreement with the human label: correct counts as accept, and incorrect and partial both count as reject. A wrong answer marked correct is a false accept; a right answer marked wrong is a false reject.

Requests went out serially from one HTTP client with a 60-second timeout and no retries. Two public control cases were run first to confirm the adapter worked, then the 100 private cases. The plan was written down before any private call, and the rubric was not changed after seeing scores. Every competitor number is read from its saved, audited record; nothing else was rerun. An independent audit of the OpenAI run passed 54,672 checks with zero errors.

Results: agreement with human labels

The graphic at the top of the post summarizes this table; there, FA is false accepts (a wrong answer marked correct), FR is false rejects (a right answer marked wrong), and the asterisk on Jev’s timing marks a historical API measurement.

Model / tested recipe	J0 agreement	E1 agreement	J0 false accept / reject	E1 false accept / reject	Median J0 / E1 response time
Jev	90%	88.5%	14 / 6	18 / 5	438 / 461 ms (historical API)
JevK5-4B v0.3, Q8 / Metal	89%	88%	16 / 6	18 / 6	162 / 998 ms
Mapika Decider-4B v2, BF16 / MPS	86%	86%	22 / 6	22 / 6	160 / 1,170 ms
Jeeves 9B, no reasoning / MLX	84%	85%	28 / 4	30 / 0	887 / 3,870 ms
OpenAI Decisions, gpt-6-luna	83%	82%	16 / 18	14 / 22	135 / 136 ms (API)
JevK5-9B v0.3.3, Q8 / Metal	82%	88%	22 / 14	20 / 4	311 / 1,938 ms
Jeff 0.8B, original / MLX	81%	72%	24 / 14	52 / 4	23 / 163 ms
Clef-flash 9B, BF16 / MPS	80%	79%	34 / 6	36 / 6	392 / 2,493 ms
Strands Decider 2B v19, BF16 / MPS	73%	68%	54 / 0	64 / 0	309 / 825 ms

Error counts cover both passes, so the denominators are 66 human-rejected and 134 human-accepted judgments per column. Clef-flash and Strands were tested after the last post and are included for completeness. Accepting everything scores 67% on this set, so 83% is well above the floor but in the middle of the pack, between Jeeves and the 9B JevK5. The local response times are warm measurements on a Mac from separate sessions; the OpenAI figure includes the network. They compare practical setups, not matched hardware.

Results: which way it errs

The error columns are more informative than the totals. Per pass:

Grader	J0 false accepts (of 33)	J0 false rejects (of 67)	E1 false accepts (of 33)	E1 false rejects (of 67)
OpenAI Decisions	8 (24.2%)	9 (13.4%)	7 (21.2%)	11 (16.4%)
JevK5-4B	8	3	9	3

On J0 the two graders let the same number of bad answers through, eight of 33. The six-point gap in agreement is entirely false rejects: OpenAI marked nine good answers wrong where JevK5-4B marked three. Adding the source text moved OpenAI in the usual direction for false accepts (eight down to seven) but made it reject more good answers (nine up to eleven). So it is stricter than Jev or JevK5-4B, not more discerning: a stricter grader that was also catching more bad answers would be a reasonable trade, and this one is not.

Per pass, J0 returned 66 correct, 6 partial, and 28 incorrect; E1 returned 63, 9, and 28. It does use the partial label, which JevK5-4B never did on this set. Both passes returned identical verdicts on every case, with zero refusals and zero API errors across 400 calls.

The responses include a confidence field and per-option probabilities. We stored and checked them but did not fit or test a cutoff for sending uncertain cases to Claude, so nothing here says whether that score is usable the way Jev’s turned out to be.

Is the six-point gap real?

Paired differences, OpenAI minus the comparator, in percentage points, with 95% intervals from resampling the 100 cases (both passes of a case stay together):

Comparator	J0 difference	E1 difference
Jev	−7 [−13, −2]	−6.5 [−13.5, 0.5]
JevK5-4B	−6 [−14, 2]	−6 [−15, 3]
Decider-4B v2	−3 [−11, 5]	−4 [−12, 4]
Jeeves 9B	−1 [−10, 8]	−3 [−12, 6]
Jeff 0.8B	2 [−6, 10]	10 [0, 21]
Strands Decider 2B	10 [−1, 21]	14 [3, 26]

The gap to the Jev run on J0 is the one comparison whose interval excludes zero. Against JevK5-4B, the six-point gaps have intervals that cross zero: on the first pass there were six cases only OpenAI got right and twelve only JevK5-4B got right on J0 (seven versus thirteen on E1), which on 100 cases is suggestive, not conclusive. What can be said with more confidence is the direction of the errors, since the false-reject counts are not close. These are unadjusted comparisons on one reused development set; they do not rank the services in general.

Results: response time and cost

OpenAI HTTP response time	J0	E1
Median	135 ms	136 ms
Mean	153 ms	152 ms
P95	242 ms	232 ms

Timing covers the HTTP round trip and reading the full response, measured serially from one client. The E1 median, with roughly 2,300 to 4,100 input tokens per request, is about one-seventh of JevK5-4B’s saved local E1 median on a Mac, and the response time barely moved between the short and long inputs. That is the strongest result in this test. It is a workload timing, not a throughput benchmark; we did not test concurrency.

Usage and cost at list price	J0 (200 decisions)	E1 (200 decisions)
Input tokens	81,386	549,316
Input tokens per request	338–505	2,257–4,143
Estimated cost	$0.0081	$0.0549
Estimated cost per 1,000 decisions	$0.041	$0.275

The 400 private decisions used 630,702 input tokens, for about $0.063 at the published $0.10 per million input tokens. There are no output, cache, or reasoning token charges; the figure is computed from reported usage, not an invoice, and regional and long-context premiums can apply. For scale, the eval from the cost post graded about 2,350 answers with Opus for about $43.60; at the E1 rate above, 2,350 decisions would be about $0.65. Jev’s list price is $0.042 per million input tokens, so OpenAI’s per-token rate is about 2.4 times higher, though the two services count tokens differently and we did not measure Jev’s usage on this set.

Caveats

Human100 is one reused development set of 100 cases, all straightforward fact-recall questions. The numbers describe our workload.
The repeated passes check stability; they do not turn 100 cases into 200 independent ones.
The service is in public beta with one model alias and no pinned version. The results apply to what gpt-6-luna returned on 2026-10-06.
The six-point gap to JevK5-4B has a 95% interval that crosses zero. The difference in false-reject counts (18 versus 6 on J0) is the more solid finding.
Only the OpenAI run is new. Every other row is a saved result, and the response times were measured on different hardware, in different sessions, with and without a network in the path.
The confidence field was recorded but not tested. We cannot say whether a two-stage setup with Decisions in front of Claude would work.
Cost is estimated from reported token usage at the base rate, not billed.
Nothing in our production grading configuration has changed.

Summary

On Human100, OpenAI Decisions with gpt-6-luna agreed with human accept/reject labels 83% of the time on J0 and 82% on E1, against 90% / 88.5% for the Jev run and 89% / 88% for JevK5-4B.
Per pass it marked 8 of 33 bad answers correct on J0, the same as JevK5-4B, but marked 9 of 67 good answers wrong where JevK5-4B marked 3. With source text that became 7 and 11.
Both passes returned identical verdicts on all 100 cases; 400 calls, zero errors, zero refusals.
Median response time was 135 ms on J0 and 136 ms on E1 including the network round trip, with a P95 under 250 ms. The long-input median was about one-seventh of JevK5-4B’s local figure.
All 400 judgments used 630,702 input tokens, about $0.063 at $0.10 per million, or roughly $0.041 per 1,000 short decisions and $0.275 per 1,000 with source text.
Its confidence score was stored but not tested, so no first-pass-filter result is available yet.

For this grading task we are staying with Jev hosted and JevK5-4B locally. The Decisions API is the fastest and one of the cheapest options we have measured, and if a later version closes the false-reject gap it would be worth a second look, with the confidence score tested at the same time.

Testing JevK5-4B as a Free, Local Alternative to Jev for Answer Grading

Thu, 01 Oct 2026 00:00:00 -0400

Setup: Evaluating a trained model means having a second model grade each of its answers. The grader receives a question, the known fact, the candidate answer, and a rubric, and has to tell a correct answer from one that changes a number, confuses two entities, or leaves out something the question asked for. For few days, we have been using TypeSafe’s Jev for part of that job. Jev is a hosted service, and several open models now copy its interface. We wanted to know whether any of them could do our grading task on a machine we own.

Short version

We tested Jeff, Jeeves, Kev, Decider, and two sizes of JevK5 on the same 100 hand-graded cases we used to evaluate Jev. JevK5-4B agreed with the human accept/reject labels 89% of the time, against 90% for our Jev run, with a median local response time of about 162 ms on a Mac. None of the other open models came as close. Replaying the same cases through a packaged local server reproduced every verdict. We have picked JevK5-4B as our preferred open model for further validation and local experiments. Our production grading configuration has not changed.

Background

Jev does not write prose. You send it the input and a list of questions with fixed answer types, and it returns a decision and a probability for each option. Our answer judge is a single three-way choice: correct, incorrect, or partial. Software can consume that directly, with no verdict to extract from an explanation. I wrote up how Jev compares to Claude as a grader earlier; this post is about the open models that imitate it.

The candidates were:

Jeff, 0.8B parameters.
Jeeves, 9B, tested with and without its reasoning mode.
Kev, 4B, from an earlier study.
Decider, 4B, version 2.
JevK5, 4B (v0.3) and 9B (v0.3.3).

Each was run locally in the serving recipe we would actually use, which means they differ in numeric precision, input formatting, and runtime. Where that matters it is noted below.

Method

The test set is Human100: 100 document-grounded recall cases, frozen and previously inspected, with 67 answers a person accepted and 33 a person rejected. Every model saw the same questions and the same fixed answers in two setups:

J0: the fact, question, answer, named entities, and rubric.
E1: the same, plus the original source text.

The score is accept/reject agreement with the human label. A correct verdict counts as accept; incorrect and partial both count as reject. Each completed local study sent every request twice. The models are deterministic, so the second pass checks stability; it does not turn 100 cases into 200 independent examples.

All response times are warm measurements on an Apple M5 Max with 128 GB of RAM, taken in separate sessions rather than a controlled throughput benchmark. The Kev timings are server-reported model time only; the other local rows include tokenization and reading the result back. The Jev run has no matched local measurement.

Results: agreement with human labels

Model / tested recipe	J0 human agreement	E1 human agreement	Median J0 / E1 response time
Jev	90.0%	88.5%	No matched local measurement
Jeff 0.8B, original release	81.0%	72.0%	23 / 163 ms
Jeeves 9B, without reasoning	84.0%	85.0%	887 / 3,870 ms
Kev-4B, earlier study	82.0%	74.0%	212 / 796 ms*
Decider-4B v2, BF16 / MPS	86.0%	86.0%	160 / 1,170 ms
JevK5-9B v0.3.3, Q8 / Metal	82.0%	88.0%	311 / 1,938 ms
JevK5-4B v0.3, Q8 / Metal	89.0%	88.0%	162 / 998 ms

* Server-reported model time, excluding tokenization and readout.

JevK5-4B is one point behind the Jev run on J0 and half a point behind on E1, and it is the fastest of the models that scored above 85%. “Q8” means the weights were shrunk to 8-bit numbers; “Metal” and “MPS” mean the Mac’s GPU was used. The earlier Kev study used the same Human100 labels but a broader request that also asked secondary score questions, as did the Jev run.

Results: agreement with a saved Opus run

We also ran the models on Recall224, a second set of 224 fixed answers. That set has saved Claude Opus judgments rather than human labels, so these numbers say how often each model agreed with Opus, not how often it was right.

Model	Accept/reject agreement with Opus
JevK5-4B	91.96%
Decider v2	91.07%
JevK5-9B	88.39%
Jeeves	85.71%
Jeff	85.27%

The ordering matches Human100 at the top, with JevK5-4B and Decider v2 close together and the rest behind.

What we found with each model

Jeff

Jeff was the most tempting option on speed. A 0.8B model returning a short decision in roughly 23 ms would be useful if the decisions were good enough. On our task the original release reached 81% J0 agreement, and adding the source text dropped it to 72%.

We tried to close that gap by training it further on public decision data and fictional examples across several domains. Two initial attempts scored 78% on Human100 J0. A broader recipe that retrained the final block reached 80%. An adapter across the whole backbone fell to 47%, despite scoring 100% on the generated development set it was trained against. Along the way we found and corrected a difference in how inputs were formatted in an early synthetic evaluation; the corrected results still showed more wrong answers being marked correct. These experiments rule out the recipes we tried, not what Jeff can learn. They did make us unwilling to pick it on speed alone.

Jeeves

Jeeves without reasoning was our strongest completed local contender before the later comparison. It improved on Jeff and did well on our synthetic tests. On Human100 it reached 84% J0 agreement and accepted 14 of the 33 human-rejected answers per pass. JevK5-4B accepted eight. In the recipes we measured, the 4B was also about 5.5 times faster on J0.

We tried Jeeves with reasoning turned on. We stopped the run after 185 judgments; the median response time was 44.6 seconds on our local BF16/MLX setup. On the cases it completed, reasoning improved J0 by one correct decision and left E1 unchanged. A partial run cannot establish the full model’s accuracy or predict how it would perform on CUDA hardware. It gave us little reason to keep paying that local time cost.

Kev and Decider

Both are credible alternatives with different strengths. Our older Kev-4B run reached 82% J0 and 74% E1. Decider v2 was much closer, at 86% in both setups, with a short-request response time essentially equal to JevK5-4B. Decider also passed a confidence screen on a second set where JevK5-4B failed (see “Confidence” below), so it deserves attention if the goal is deciding which cases a local model can handle alone.

We tested Decider v2 specifically. Its current model card says that v2.1’s confidence scores track its accuracy less well on hard items. Neither v2.1 nor Kev-9B has been tested here; their absence from the table is work not yet done, not a negative result. Older studies also recorded djev at 89% J0 / 85% E1 and Laya at 63% as shipped, rising to 81% after training on Brekko labels. Those runs lack the same complete recipe and timing record, so they are kept separate from the table above.

JevK5-9B

The larger JevK5 was a useful check. The tested 9B v0.3.3 release reached 82% J0 and 88% E1 while taking roughly twice as long as the 4B. More parameters did not improve the aggregate result on this workload.

Replaying through the packaged server

The research harness is one thing; the serving path we would actually use is another. We replayed Human100 through our installed JevK5 MCP server (MCP is a standard protocol for exposing a tool to other software), making all 200 J0 and E1 decisions through that interface. Every verdict matched the previous 4B run. Probability differences were at floating-point rounding scale, with no inference errors and no cache hits. Median response times through the server were 165 ms for J0 and 901 ms for E1. An independent audit of the replay passed 1,237 checks. This shows the packaged service preserves the tested behaviour; it adds no new accuracy evidence.

What “free” means here

JevK5-4B’s code and released weights are Apache-2.0. The Q8 file we tested is about 4.48 GB and runs through llama.cpp on Apple Silicon, other supported GPUs, or a CPU. Our measurements used the Mac’s GPU; we have not benchmarked a CPU deployment.

“Free” means downloadable weights and code with no per-decision fee to a model vendor. Hardware, electricity, operations, and any hosted fallback still cost money, and we have not measured fleet economics. For a model kept resident on hardware you already own, being able to run locally and control the serving environment is worth something before any cost saving is assigned to it.

Confidence

In the earlier Jev post, the useful part turned out to be the confidence score: using it to decide which cases a cheap grader could settle alone, and sending the rest to Opus. We repeated that check for JevK5-4B. In a development replay on Human100, a two-stage setup with JevK5-4B in front of Opus settled 57% of decisions locally and reached 93.5% agreement with the human labels, matching the Jev two-stage setup at similar coverage.

The same cutoff did not hold on Recall224. It failed both our disagreement screen and our screen for wrong answers being marked correct, and a stricter cutoff settled too few cases to be useful. So the bar we wrote down for replacing the production grader is still unmet, and the confidence score needs its own validation before it can be used to route cases.

What still needs work

JevK5-4B still marks eight of Human100’s 33 human-rejected answers as correct on J0.
Its five-point lead over Jeeves has a paired 95% interval of −1 to +11 points. On 100 cases that is not enough to establish broad superiority.
It returned no partial verdicts on Human100. Accept/reject agreement alone cannot certify all three grading labels.
Separate fictional controls exposed a material omission it accepted, and its probabilities changed when equivalent options were reordered.

Caveats

Human100 is one reused development set of 100 cases. The numbers describe our workload, not arbitrary customer domains.
The repeated passes check that the models are deterministic; they do not add independent cases.
Recall224 agreement is against saved Opus judgments, not human labels.
The models were run in different recipes (precision, formatting, runtime) on one Mac in separate sessions. The response times compare practical setups, not a controlled benchmark.
Decider v2.1 and Kev-9B were not tested. The Jeeves reasoning run was partial.
The results apply to the specific released versions named in the table.
The MCP replay confirms the serving path reproduces the harness; it is not additional accuracy evidence.

Summary

On Human100, JevK5-4B v0.3 Q8 agreed with the human labels 89.0% of the time on J0 and 88.0% on E1, against 90.0% and 88.5% for the Jev run.
Median local response time was 162 ms on J0 and 998 ms on E1; the packaged MCP server measured 165 ms and 901 ms and reproduced all 200 verdicts.
The next-best open models were Decider v2 at 86.0% / 86.0% and Jeeves at 84.0% / 85.0%. Jeeves accepted 14 of 33 rejected answers per pass; JevK5-4B accepted eight.
Jeff was fastest at 23 ms but reached only 81% J0 agreement, and our training attempts peaked at 80%.
The 9B JevK5 scored 82% / 88% and took about twice as long as the 4B.
On Recall224, JevK5-4B agreed with Opus 91.96% of the time, ahead of Decider v2 at 91.07%.
A two-stage setup settled 57% of Human100 locally at 93.5% agreement, but the same cutoff failed our screens on Recall224, so the production replacement bar is unmet.

For anyone looking for a free, local Jev-style answer judge, JevK5-4B v0.3 Q8 is the model we would try first. Our next step is fresh human grading focused on omissions, factual near misses, partial answers, and the confidence cutoff.

Cutting LLM Answer-Grading Cost by 75% With a Cheap First-Pass Grader

Wed, 23 Sep 2026 00:00:00 -0400

Setup: we train a small open model on a customer’s own documents. Every training run is evaluated three times: after stage 1, after stage 2, and once more at the end. Each eval asks the trained model 2,350 test questions and has a large commercial model, Claude Opus 5 on Vertex AI, grade each answer against the known fact. For questions answerable from the documents, Opus also sees a window of the source text, and that window was worth about four points of accuracy against human labels in an earlier test. Three evals per run, times a few thousand graded answers each, adds up.

This post is about how we cut the grading cost by about 75%, from roughly $43.60 to about $11 per eval, while the verdicts we keep still agree with Opus about 99 times in 100. Two things up front. This is the grading cost only; GPU training and the other model calls in the pipeline are unchanged. And the dollar figures are estimates from token prices, not billed amounts, because we do not yet record billed grader usage.

Short version

We put TypeSafe’s Jev in front of Opus as a first pass. Jev grades every answer and returns a probability for its choice. If that probability clears a cutoff, its verdict stands; otherwise the answer goes to Opus exactly as before. With a cutoff of 0.95, Jev handles 60–62% of source-document questions and 91–92% of the rest. On the rows it handles, it agrees with Opus 99.1–99.8% of the time, which is about as often as Opus agrees with itself. The first version of this saved 35%; the saving came from coverage, not from Jev’s unit price.

Where the money went

The eval we measured had 2,352 questions in six categories:

Category	Count	What it tests
Recall	854	Facts answerable from the documents
Trick variants, category A	928	Questions designed to catch the model out
Trick variants, category B	344	Same, a second category
Kept-out	148	Answers deliberately left out of training; the right response is to decline
Nearby out-of-scope	50	Questions close to, but outside, what the documents cover
Regression	28	General behaviour has not broken

I will call the last five categories “non-recall” from here on.

An Opus recall verdict costs about $0.0265, because it also reads a source window of roughly 3,300 tokens. A non-recall verdict costs about $0.014, and about 72% of that is output: Opus writes back a structured verdict with scores and a short explanation, and output tokens are the expensive kind.

What Jev is, and why it is not a replacement

Jev is TypeSafe’s decision model. It writes no prose. You give it the allowed verdicts and it returns a choice plus a probability for each. Input costs $0.042 per million tokens and output is free, which works out to about $0.000033 per verdict, several hundred times cheaper than Opus.

I tested Jev against Opus earlier on a 100-case set graded by hand. Without the source document, the two tied at 89.0% agreement with the human labels. But Jev cannot use the source document; give it the window and it gets worse. Swapping graders outright would have handed back the four points the source window bought us. So Jev is a first pass, not a replacement.

What we built

The two-stage setup is a per-run option, off by default. Jev grades first. If the probability of its top choice clears the cutoff, its verdict stands. Otherwise the answer goes to Opus with the same request it would have received anyway.

Failure handling is strict. Jev timeouts, rate limits and server errors send the answer to Opus. Configuration errors, such as a bad key or an unknown model name, fail the eval before any Opus money is spent. A failed Jev call is never turned into a verdict.

We treat a different grader as a different measuring instrument, because it is one. Each grading mode has its own identity string naming both models and the cutoff. The eval job checks at startup that it built the same identity the orchestrator wrote, and refuses to start, before any GPU or grading spend, if they differ. Stability comparisons between runs never mix runs scored by different graders.

Finally, we copied the grading instructions byte for byte from the research harness that measured them rather than retyping them. All 2,352 request bodies are byte-identical to what was tested.

Version 1 covered recall questions only, with the cutoff at 0.90.

Method

Every comparison is a paired eval: the same trained model and the same questions, with only the grader changing, joined question by question.

Two things about the measurement had to be settled first.

Generation at temperature 0 was not deterministic. Only about 80% of answers were byte-identical across two evals of the same model. If you compare verdicts on answers that differ, generation noise looks like grader disagreement. So all agreement figures below are on questions where both evals produced the identical answer.

Opus disagrees with itself. On the same inputs, Opus versus Opus agreed 98.7% of the time on hard recall rows and 99.6% on non-recall. Any gap between Jev and Opus has to be read against that floor.

On the disagreements we ran a standard paired significance test (exact McNemar) and split them two ways. “Lenient” means Jev said the answer was right and Opus said it was wrong. “Strict” is the reverse. Lenient errors are worse, because they flatter the model being evaluated.

For the check on a second trained model, we wrote down the pass bar before looking at the results:

Non-recall agreement with Opus of at least 99.0%.
Recall agreement of at least 98.7%, matching the Opus self-agreement floor.
No category significantly lenient.
At most one lenient error on kept-out questions.

Results

Comparison	Cutoff	Handled by Jev	Agreement with Opus	Lenient errors
Recall, first model	0.90	67.7%	99.2%	3
Non-recall, first model, re-graded offline	0.90	94%	99.6%	2
Non-recall, first model, re-graded offline	0.95	92%	99.7%	0
Non-recall, second model	0.95	91%	99.7%	0
Recall, second model	0.90	67.6%	98.4%	6
Recall, both models, re-cut	0.95	60–62%	99.1% and 99.8%	4 in total

Agreement is measured on Jev-handled rows with identical answers.

Recall only, first model

Jev handled 67.7% of recall verdicts. On those it agreed with Opus 99.2% of the time, with 3 lenient and 1 strict disagreement. The headline recall score moved −0.2 points (p=1.0, too close to tell apart). The saving was only about 35%. The limit was coverage, not price: 74% of the remaining cost was non-recall questions still sent to Opus every time.

Non-recall, offline

Before touching the non-recall categories in a live eval, we had Jev re-grade the exact answers Opus had already graded. Without a cutoff, it was not safe. On kept-out questions Jev accepted 7 of 148 answers as proper declines that Opus said were not (p=0.016). That is the worst kind of error on the product’s weakest skill, knowing when not to answer. It was also stricter than Opus on one of the trick-question categories (p=0.011).

With the cutoff at 0.90, Jev handled 94% with 99.6% agreement and 2 lenient errors. At 0.95 it handled 92% with 99.7% agreement and no lenient errors. We chose 0.95 for non-recall, knowing it was read off the same data it was tested on.

Second model

Non-recall passed. Jev handled 91%, with 99.7% agreement on 1,071 comparable rows, 0 lenient and 3 strict. On kept-out questions, 0 lenient of 95.

Recall failed. At the 0.90 cutoff, agreement was 98.4% on 494 rows, with 6 lenient and 2 strict, under the 98.7% bar. The result was not significantly different from the first model, and the headline recall score moved only +0.1 points. But across both models Jev leaned lenient on recall, 9 lenient against 3 strict (p=0.15). That is the direction we care about.

We tightened the system, not the bar. We raised the recall cutoff to 0.95. Re-applying that cutoff to the recorded probabilities gives 99.1% and 99.8% agreement on the two models, with 4 lenient errors against 1 across both. A stricter cutoff can only send more answers to Opus, so it cannot do worse than the looser one. The price is a saving of 75% instead of 79%.

What it costs now

Setup	Grading cost per 2,352-question eval
All Opus	about $43.60
Two-stage, recall cutoff 0.90	about $9.34 (79% saving)
Two-stage, recall cutoff 0.95 (what we ship)	about $11 (75% saving)

Both two-stage rows use a 0.95 cutoff for non-recall. Per training run, with three evals, that is roughly $130 down to $33.

What the results mean

Coverage sets the saving, not unit price. Jev is several hundred times cheaper per verdict, and version 1 saved 35%. Every question that still goes to Opus costs what it always did. The gain came from safely extending the cheap path to more categories.

The confidence cutoff is what made non-recall safe. Without it, Jev accepted 7 kept-out answers it should not have. At 0.95 it accepted none, on both models. The probability Jev returns finds the rows where it is unsure, without being told what those rows look like.

Lenient and strict disagreements are not interchangeable. Two graders can disagree on 1% of rows in a harmless way or a harmful one. Only the split told us Jev leaned lenient on recall; the headline recall scores moved by −0.2 and +0.1 points and would not have.

The remaining points are method notes, stated as rules we would reuse:

Compare identical outputs only. Temperature 0 gave us byte-identical answers about 80% of the time; the other 20% is generation noise that would have been counted as grader disagreement.
Read disagreement against the grader’s own noise. Opus agrees with itself 98.7% to 99.6% of the time; a 99.2% Jev-to-Opus figure sits inside that.
Fix the pass bar before you look. When you miss it, tighten the system, not the bar. We gave up four points of saving and kept the meaning of the metric.
Label a new grader as a new instrument and never compare across the line. A cheaper grader that quietly inherited the old grader’s baselines would pass checks against numbers it never produced.
Ship the exact bytes you measured. Retyped rubrics are a new experiment nobody ran. Generating the requests from the harness and checking byte equality cost little and removed a class of doubt.
Fail at startup, before spending money. An identity mismatch stops the job before any GPU or grading spend; a bad key stops it before the first Opus call.

Caveats

The 0.95 cutoffs were chosen from the same data we evaluated them on. A third trained model would show whether they hold.
The out-of-scope (50) and regression (28) categories are too small to read on their own.
Agreement is measured against Opus, not against humans. A sealed, hand-graded set covering every category is still to do.
Billed grader cost is not recorded yet; the dollar figures are token-price estimates.

Summary

Grading cost per eval went from about $43.60 to about $11, a 75% saving, on grading only.
Jev handles 60–62% of recall verdicts and 91–92% of non-recall verdicts at a 0.95 cutoff.
On the rows it handles, Jev agrees with Opus 99.1–99.8%, against an Opus self-agreement floor of 98.7–99.6%.
Without a cutoff, Jev accepted 7 of 148 kept-out answers Opus rejected; at 0.95 it accepted 0 on both models.
Recall at the 0.90 cutoff missed the 98.7% bar on the second model (98.4%, 6 lenient vs 2 strict), so we raised the cutoff rather than the bar and gave up four points of saving.
Version 1 saved 35% with the same per-verdict price; the rest came from extending coverage to non-recall categories.

Evaluating Jev as a Replacement for an LLM Answer Grader

Sun, 20 Sep 2026 00:00:00 -0400

Setup: we train small language models on collections of documents, then check how well each one learned the material by asking it a few thousand questions and having a large commercial model (Claude) grade the answers against a rubric. The grading step costs about ~$100 per run in API spend. For scale, an hour on an 8-GPU machine costs $29.40, so grading is roughly 3.5 hours of GPU cost per run, spent deciding whether the GPU time was worth it.

TypeSafe recently released Jev, and I spent a few days testing it as a replacement for that grader. This post is the writeup. The short version: it matches Claude on grading accuracy when neither is shown the source document, falls behind when both are, and returns a confidence score that is reliable enough to make decisions on. Used as a first-pass filter in front of Claude rather than a replacement, it cuts the grading bill by about 78% at the cost of half a percentage point of accuracy.

What Jev is

Jev does not generate text. It is a model that answers structured questions about an input. You send it a chunk of data and a list of questions with fixed answer types, and it returns answers in those types. The question types are Choice (pick one of N options; returns the option, a probability for each option, and an overall confidence score), Score (a number against a rubric), and Noul (yes/no). Because nothing is generated, you only pay for input: $0.042 per million input tokens. Median response time on my calls was about 400ms.

This matters because our grader is already, functionally, a Choice question. We prompt Claude for {"verdict": "correct" | "incorrect" | "partial", "confidence": 0.0-1.0, "reasoning": "..."} and parse the result. Around that we have retry logic and a function that recovers a verdict from cut-off JSON. That function exists because one run had 472 of 6,657 verdicts get truncated at the output length limit and default to incorrect, which silently invalidated the run. None of that code is about grading. It is the cost of asking a text generator for a structured decision.

Method

I ported our four grading rubrics to Jev. Each rubric’s CORRECT/INCORRECT/PARTIAL criteria became the three options of a single Choice question. I then ran both graders against 100 cases that a person had already graded by hand, two passes each, for 200 judgments per grader. The metric throughout is how often the grader agrees with the human.

Two conditions: with and without the source document. Our production grader gets a ~10,600-character window of the original document around the fact being tested. We added that the week before this test; it moved Claude from 89.0% to 93.0%, mostly by catching wrong answers it had previously marked correct.

Results: grading accuracy

Without the source document:

Grader	Source evidence	Human-label agreement
Claude Opus 5	none	89.0%
Jev	none	89.0%

Identical, and not just in total: the same 178 of 200 judgments are correct. The two graders agree with each other on 92 of 100 cases, are both wrong on 7, and each is uniquely right on 4. Across roughly 4,800 Jev calls total there were zero parse failures and zero truncated outputs.

With the source document:

Grader	Source evidence	Human-label agreement
Claude Opus 5	10,600-char window	93.0%
Jev	10,600-char window	88.5%

Jev gets slightly worse when shown the source. TypeSafe’s docs describe this failure mode directly (“large state full of irrelevant detail — filter first; send only what the question needs”). The windows are three neighboring document chunks picked by position, so most of the text has nothing to do with the question.

Trimming the source text did not fix it

If the problem is too much irrelevant text, the obvious fix is to shrink the window. I wrote four filters and cut the windows to 7.5–10.9% of their original length:

Keep the sentences that share the most words with the question. No model involved.
Ask Gemini Flash to extract the relevant sentences, two prompt variants.
Ask Jev a yes/no Noul question (“is this sentence relevant?”) about every sentence at once, in a single call.

Before running anything I wrote down 91.0% as the bar for “partially fixed.” The best filter came in at 90.5%, and all four were too close to each other to tell apart. So: no.

Looking at why was more useful than the result. One case in the set had been flagged by our human reviewer as the key example: the model’s answer applied a rule from paragraph b(6) when the question asked about b(5). All four filters kept both paragraphs. The most aggressive one reduced 74 sentences to four (981 characters), and those four were, consecutively and word for word:

(5) The appropriate qualification standard and any selective certification or
quality ranking factors which may be appropriate;
(6) The appropriate recruitment sources and techniques to use and whether to
use them singularly or in combination.

Jev marked the answer correct with all four filters, both passes. Across the failed cases where the reviewer left a note, the filtered windows kept 87.5% of the specific terms the reviewer pointed to in the source, which is higher than the filters’ average. The relevant text was there.

So this is not a problem of too much irrelevant text. Rejecting that answer requires noticing that the rule came from the paragraph next door, not the one the question asked about. That is a reasoning step Jev does not appear to take. It handles “does the source support this claim” reasonably well; it does not handle “this claim is supported by the wrong part of the source.”

A mistake I almost made

The two graders disagreed on 290 judgments. To break the ties I sent those rows to Claude Opus 5 as a tiebreaker, without telling it which grader said what, using our own grading prompts. Then I repeated it with a model from a different vendor.

Tiebreaker	Sides with Jev	Sides with Claude grader
Claude Opus 5	73	155
Gemini 3.8 Flash	137	100

The result flips depending on which tiebreaker you ask. Both pass the same sanity check: on rows where the two graders already agreed, they side with the agreed verdict 95.8% and 91.7% of the time respectively. But our grader is Claude, and using Claude to judge Claude roughly doubled its apparent advantage. Counting only the rows where both tiebreakers agree gives 79 to 58 in Claude’s favor: a real edge, but less than half of what the first number suggested.

If you are using an LLM to settle disagreements between graders, use one from a different vendor than any grader being tested, or use two and only count the rows where they agree.

Is the confidence score any good?

Every Jev response includes a confidence score. It is computed from the probabilities Jev assigns to each option, not written out as text by the model. I had been ignoring it because I was evaluating verdicts.

Grouping judgments by the confidence the grader reported:

Reported confidence	Jev rows	Jev agreement	Claude rows	Claude agreement
0.2–0.3	59	37.3%	—	—
0.5–0.6	91	58.2%	11	54.5%
0.6–0.7	113	64.6%	25	12.0%
0.7–0.8	131	74.8%	149	25.5%
0.9–1.0	1,587	97.9%	1,833	97.1%

Jev’s accuracy rises steadily with its reported confidence across the whole range: when it says it is unsure, it is usually wrong, and when it says it is sure, it is usually right. Claude’s self-reported confidence field does not behave this way. It never reports a value below 0.5, it puts 79% of rows at 0.9 or above, and it is less accurate in the 0.6–0.7 bucket than in the 0.5–0.6 bucket. We use a threshold of min_judge_confidence = 0.70 to decide which judgments a human should look at. On Claude’s self-report that threshold flags 1.6% of rows for review. On Jev’s it flags 18.9%, and those rows really are the disputed ones.

Jev as a first-pass filter

Given a confidence score you can trust, the natural setup is two stages: accept Jev’s verdict where it is confident, and send the rest to Claude.

Configuration	Jev serves	Human-label	Cost/run
Current: Claude + full window	—	93.0%	$109
Claude + question-conditioned window	—	92.5%	$71
Jev → Claude + full window	57.0%	92.5%	$47
Jev + Claude, one shared window	66.5%	92.5%	$24

0.5 points is one judgment in 200. Cost drops by about 78%.

This works because the confidence threshold picks out exactly the rows where Jev is not worse. On the two thirds of rows Jev is confident about, Jev and Claude both score 95.5%, and they mark the same six wrong answers as correct. On the third Jev flags as uncertain, it trails Claude by 7.5 points. Splitting the rows randomly at the same ratio would not work, since Jev on its own is 3–4 points behind overall.

The second row is worth noting on its own because it requires no new vendor. Cutting the source window down to roughly 900 characters costs Claude nothing in accuracy (92.5% either way) and halves the input tokens, as long as the filter is based on the question rather than on the answer being graded. Since both the question and the document are fixed, the filtered window can be computed once per question set and cached.

Caveats

I picked the confidence threshold after the fact, on the same 100 cases I used to measure Jev’s confidence in the first place. That is circular. The overall shape (higher confidence, higher accuracy) is probably real; the specific threshold and the 92.5% figure are not validated.
The test set is straightforward fact-recall questions only. Jev’s grading agreement was weakest on trick questions designed to catch the model out, which is exactly where marking a wrong answer correct does the most damage.
Only 100 cases. The differences between the four filters, and between Jev and Claude when neither sees the source, are too small to be meaningful.
This saves money; it does not improve accuracy. If you could somehow pick the better grader for each case, you would score 94.0–94.5%. The two-stage setup captures none of that.

A proper test would fix the threshold on one set of cases and then score it on a fresh, human-graded set covering every question type.

Summary

Jev matches Claude on grading accuracy without the source document (89.0% both) and falls behind with it (88.5% vs 93.0%).
The gap is not fixable by filtering the source text. It comes from a reasoning step (noticing a claim is backed by the wrong passage) that the model does not perform.
Settling grader disagreements with a model from the same vendor as one of the graders roughly doubles that grader’s apparent advantage.
Jev’s confidence score means what it says; our Claude grader’s self-reported confidence does not, and tells you almost nothing below 0.9.
Using Jev’s confidence to decide which cases go to Claude keeps accuracy within 0.5 points and cuts grading cost by ~78%, subject to the caveats above.

If you are evaluating a model like this, measure the confidence score, not just the verdict. In our case the verdict was a tie and the confidence was the result.

The Before Column

Sat, 19 Sep 2026 00:00:00 -0400

I thought I’d figured out why our second training stage was making a model forget facts. Several cuts of the data agreed, the explanation made sense, and it pointed to a fairly straightforward fix. Then I looked more carefully at the column showing where each group started.

We train small models on document collections. The first stage teaches the facts. The second teaches things like refusing questions the documents can’t answer and pushing back on false premises. In this run, accuracy on fresh questions fell from about 58% to 43% after the second stage.

Fifteen points is a lot to lose. We’d also just tried changing the composition of the training data, with no measurable improvement. I wanted to understand whether we’d picked the wrong change.

So I grouped facts by how many examples in the second stage taught the model to answer them correctly, then compared accuracy before and after.

Supporting examples per fact	Before	After	Change
1–9	66.4%	47.5%	−18.9 points
10–29	59.2%	44.5%	−14.7 points
30 or more	52.3%	39.2%	−13.1 points

That’s a pretty convincing table. More supporting examples, less damage. Every bucket moves in the expected direction.

Other cuts agreed. Facts with more refusal examples lost more accuracy. Facts that had received extra examples during a repair pass lost less. I could already see the next experiment: rebalance the examples and check whether recall held up.

But look at the before column. Every group that lost more also started higher.

A group starting at 66% has more correct answers available to lose than a group starting at 52%. It also has fewer wrong answers available to fix. Both affect the change score.

Here’s a small example. Say a training stage keeps 60% of previously correct answers and fixes 20% of previously wrong ones. Apply those same rates to two groups of 100 questions:

A group starting with 80 correct keeps 48 and gains 4. It finishes at 52, down 28 points.
A group starting with 40 correct keeps 24 and gains 12. It finishes at 36, down 4 points.

Same retention. Same recovery. A 24-point difference in apparent damage.

Whatever property you used to define those groups can now look like an explanation for that difference. You don’t need measurement noise to produce it. Those transitions could happen in a completely deterministic system.

The arithmetic fits on two lines. If starting accuracy is A, retention is r, and recovery is g:

after  = A*r + (1-A)*g
change = g - A*(1+g-r)

With fixed retention and recovery rates, higher starting accuracy produces a larger expected drop. The statistical term is mathematical coupling: the starting value is part of the change you’re trying to explain.

I reran the analysis using the transitions directly. Of the questions the first checkpoint got right, how many did the final model keep right? Of the ones it got wrong, how many did it fix?

The supporting-example groups retained 61.5%, 58.9%, and 57.0%, respectively. The apparent protective effect disappeared. The direction even flipped, though the differences weren’t statistically significant. The repaired and unrepaired facts retained 58.7% and 59.1%. Almost identical.

Using the overall retention and recovery rates, a model with just those two parameters reproduced the original bucket changes to within about a point and a half. The groups’ starting accuracies explained most of what had looked like a training-data effect.

That still leaves room for smaller effects. This analysis couldn’t establish that adding examples never helps. It did take away the evidence I was about to use to justify doing it.

Question form remained worth investigating. Among initially correct answers, the model retained about 67% on who/what/where-style questions and 50% on yes/no questions. That difference survived the same analysis. We’d already suspected it because of an imbalance in the training data.

The annoying part is that we had a warning about this general problem in our research notes. It covered groups selected by their outcomes. I’d grouped by training inputs, so I didn’t think the warning applied. It needed to say what to check: groups with different starting accuracies have different opportunities to gain and lose.

I also gave too much weight to several cuts agreeing. They all used the same change score, and all carried the same baseline problem. Each extra table made me more confident without adding much evidence.

For the next analysis, I want the counts of answers kept, lost, and recovered alongside the overall score. They’re cheap to compute from the same results. In this case, they would have saved me from planning another training run around a pattern I could reproduce with two probabilities and a subtraction.

Thirty Minutes to Four

Thu, 17 Sep 2026 00:00:00 -0400

Every merge to main that touches shared code rebuilds our GPU training image: torch, vLLM, deepspeed, and the CUDA compatibility libraries that let a CUDA 13 build run on the drivers Vertex AI requires. The build runs on Cloud Build because no GitHub-hosted runner has the disk for it, and it took thirty minutes. Every single time. And one morning I even had three of them running at once, and two of the three were only documentation merges (my fault for letting a README.md change trigger the action).

So I asked Claude Code why it took so long.

TL;DR: Always ask your advanced AI models why

Finding out where the time went

It read the workflow first, which turned out to be a thin shim: authenticate, submit a Cloud Build config, wait. So the question moved to Cloud Build. It pulled the phase timings for the most recent build, then pulled the build log out of Cloud Logging and lined up the timestamp on every Dockerfile step. Then it fetched the image manifest from Artifact Registry and mapped each layer’s compressed size back to the instruction that produced it.

Phase	Time
GitHub Actions setup	0.5 min
Cloud Build queue	0.8 min
`uv sync` (245 packages)	2.2 min
vLLM probe, CUDA debs, gcc	1.0 min
`useradd` + `chown -R`	9.2 min
Push to Artifact Registry (12.4 GB)	15.3 min
Resolve digest, propose pin	0.3 min

Two lines account for 24 of the 29 minutes.

A chown that cost eight gigabytes

The Dockerfile ended the way a lot of Dockerfiles end: create a non-root user, chown -R the virtualenv to it, switch users. Reasonable. Except overlay2 implements a chown by copying every touched file up into a new layer, and the venv held four gigabytes of compressed wheels.

It was worse than a copy. uv’s wheel cache lived inside the image (UV_CACHE_DIR=/tmp/uv-cache), and uv hardlinks the venv to that cache. The chown touched both trees, the copy-up broke the hardlinks, and the new layer held two full copies. The uv sync layer was 4.0 GB compressed. The chown layer on top of it was 8.0 GB. The image we pushed was 12.4 GB for about 4.4 GB of payload.

Claude Code inferred this from the sizes. It never listed the layer’s contents, and it said so. 8,026 MB sitting on top of 4,020 MB is hard to explain any other way, but I appreciated the distinction.

The push took fifteen minutes because it was pushing that layer. Docker pushes layers in parallel, and the 8 GB one finished last.

No layer was ever reused

The second problem was quieter. Every source directory was copied before uv sync, so any code change invalidated the dependency install and the 4 GB layer got rebuilt and re-pushed. The Cloud Build config had no --cache-from, so even an unchanged lockfile started from nothing on a fresh worker. In the push log, only the five base-image layers ever said Layer already exists.

And README.md was in the workflow’s paths: filter, because the Dockerfile copied it for setuptools’ package metadata. Every docs PR touches the README. So every docs merge rebuilt and pushed 12 GB.

The fixes

Claude Code ranked three fixes by payoff. I asked it to implement all three and to double-check its own work.

Create the user first and let uv install as that user. Same venv ownership as before, with no rewrite. UV_NO_CACHE=1 keeps the wheel cache out of the image entirely.
Install dependencies before copying source. uv sync --no-install-project from pyproject.toml and uv.lock alone, then copy the source and run a second uv sync for the editable install. Cloud Build pulls the previous :latest and builds with --cache-from against it.
Stop copying the README. setuptools only needs the file to exist. The Dockerfile writes a stub, and README.md came out of the path filter.

The Dockerfile’s shape, before and after:

# Before
COPY pyproject.toml uv.lock README.md ./
COPY apps/training ./apps/training
COPY packages ./packages
RUN uv sync --frozen --no-dev --group training
# ... CUDA compat libs, nvcc, gcc ...
RUN groupadd --system app && useradd --system --gid app app \
 && chown -R app:app /app/.venv /tmp/uv-cache
USER app

# After
ENV UV_NO_CACHE=1
RUN groupadd --system app && useradd --system --gid app app \
 && mkdir -p /app && chown app:app /app
COPY --chown=app:app pyproject.toml uv.lock ./
USER app
RUN printf '# stub\n' > README.md \
 && uv sync --frozen --no-dev --group training --no-install-project
USER root
# ... CUDA compat libs, nvcc, gcc ...
COPY apps/training ./apps/training
COPY packages ./packages
USER app
RUN uv sync --frozen --no-dev --group training

Everything a code change can invalidate now sits below the CUDA layers, so those stay cached too.

Results

It verified with two live builds on the branch. The first was cold: the previous :latest still had the old layout, so nothing could hit the cache. The second was warm.

Build	Cloud Build	Image	Layers uploaded
Before	28 min	12.41 GB	11
Cold (new Dockerfile)	18.6 min	4.37 GB	11
Warm	2 min 20 s	4.37 GB	0

The cold build spent five minutes pulling the old 12 GB image one last time, built in five and a half with the chown gone, and pushed the new 4 GB dependency layer in eight. The warm build hit the cache on all nineteen Dockerfile steps, uploaded nothing, and produced a byte-identical digest. Its entire cost was two minutes pulling the 4.4 GB cache image. End to end, the GitHub Actions run went from thirty minutes to a little over four.

The warm build was also a better test than I had planned. A teammate’s docs pass landed on the branch while the cold build was running, so the commit it built changed the README and some comments and nothing else. That is exactly the case that used to cost thirty minutes.

Double-checking

“Double check your work” turned out to be worth saying. It re-read the final Dockerfile, ran the contract tests it had added (layer order, no recursive chown, no shipped cache, the cache warm-up step), and trial-merged both branches onto main. Then it went back to the registry and streamed the tar headers of the pushed layers to confirm the venv and the editable install were owned by the runtime user and the source trees were still root-owned, which is what the old chown had produced.

It also caught two of its own mistakes on the way. It had read the tar link-count column as the uid and streamed the wrong layer on the first pass. And a comment a teammate had reworded was accurate for one branch but would become false the moment the companion branch merged. It fixed both and said which.

Takeaways

Map layer sizes to instructions before touching anything. The registry manifest plus the image config’s history gives you a bill of materials per Dockerfile line. The build log never showed the 8 GB layer. That table put it on the first row.
chown -R is a copy. On overlay2 it rewrites every file it touches into a new layer. Create the user first and install as that user, or leave a root-owned, world-readable venv alone.
A layer cache is a contract about instruction order. Dependencies keyed on the lockfile, source after. Without --cache-from on a fresh worker, none of it matters.
Path filters are code. README.md sat in paths: because a Dockerfile copied it. Nobody wrote that down as a decision, and it cost thirty minutes per docs merge.
Ask for verification. The fix was ordinary Docker lore. The loop paid for itself: dispatch, watch, read the log, inspect the layers, find the stale comment.

Final thoughts

Every one of these fixes is in some Docker best-practices document, and I have written variants of that chown line myself. The diagnosis is what I want to remember: four systems (GitHub Actions, Cloud Build, Cloud Logging, Artifact Registry), one sitting, and one thread of reasoning that went from “this action is slow” to “this specific line produced this specific eight-gigabyte layer” with the evidence attached. I would not have sat down to do that between other things. The model did it well.

Same Answers, Different Score

Tue, 08 Sep 2026 00:00:00 -0400

We fine-tune small models on customer documents to answer domain-specific questions. To evaluate how well it works, we run a few thousand questions through it and have a frontier model grade the answers. That score determines whether a model checkpoint is ready to ship.

Last week I upgraded our external LLM calls from Sonnet 4.5 and Opus 4.5 to Sonnet 5 and Opus 5. The model we train remained completely unchanged, and I expected the grader upgrade to be neutral for our benchmark scores.

To verify, I re-graded a set of saved responses using the exact same response files. Only the grader model changed.

Probe set	Sonnet 4.5	Sonnet 5	Δ	Flipped
Varied phrasings (880)	0.599	0.662	+0.064	7.7%
Terse user phrasings (880)	0.406	0.469	+0.064	9.3%
Adversarial contamination (100)	0.930	0.920	−0.010	1.0%

Both independent probe sets jumped by the exact same margin, and almost every flip went upward (incorrect to partial, partial to correct). Sonnet 5 simply graded the same text more leniently.

For comparison, grading the same responses twice with the same model usually flips around 1.1% of verdicts - essentially our noise floor. Here, flips were 7–8x higher and entirely directional.

There was also a potential surprise: Sonnet 5 rejects explicit temperature parameters, so the re-grade ran with the provider’s default temperature. Re-running the old model at that same default shifted scores by only +0.002 (six flips, well inside noise). The model upgrade itself accounted for +0.061 of the +0.064 jump.

Whoever writes your eval questions is making a claim about your users

While digging into this, I ran another test on question formulation using the same model and 220 target facts across five different question sources.

Who wrote the question	Recall	Declined	Median words
Verbatim from training	0.966	0.000	19
Our eval generator	0.807	0.048	18
A different generator	0.599	0.173	14
Wh-questions only	0.714	0.107	14
How people actually type	0.406	0.325	7

"How people actually type" = an LLM prompted to write the question a busy person would type into an internal assistant: terse, context omitted, sometimes a request rather than a question.

We had been reporting an eval recall around 0.89. The model definitely knows these facts - it scores 0.966 when prompted using the exact phrasing it saw during training. But when asked those same facts in seven-word prompts resembling real user queries, recall drops to 41%, and the model declines to answer roughly a third of the time.

A metric like “overall recall” doesn’t mean much without specifying how the questions were generated. Testing multiple generation styles is important to surface any limitations.

Treating graders like instruments

The immediate fixes took under an hour, but the more important takeaway was how easily unversioned evaluation tooling can distort results.

Pin the grader version in eval metadata. If you version probe sets and stability baselines, the grader model belongs in that same configuration so changes require an explicit re-baseline.
Measure baseline variance. Grading the same responses twice with the same model establishes an noise floor. Without knowing our baseline ~1% flip rate, that 7.7% difference could easily have been mistaken for genuine model improvement.
Avoid unpinned model defaults in configs. Our default model string had been updated weeks earlier, and we only avoided accidental shifts because deployment manifests happened to pin the old version explicitly.
Smoke-test parameter compatibility. Checking whether an API key works with a new model isn’t enough; client wrappers need end-to-end tests with the exact payload used in production.
Vary question generation styles. If benchmark questions are generated with the same prompts and model families used for training data, the benchmark often ends up evaluating what it has seen during training instead of actual comprehension.

Final thoughts

Language models end up in important areas all over the pipeline: grading responses, writing test questions, extracting facts. None of them face the customer, but they quietly drive every decision about what ships. The littlest thing can ripple through the system and cause unexpected shifts in outcomes, so it’s crucial to maintain rigorous versioning, testing, and monitoring practices throughout the entire workflow.