# Jev's 95% came from weights fitted on 1,000 labelled emails

**Splitting one broad question into five narrow ones really does take TypeSafe's Jev from 62.6% to 95.0%, but the 95.0% is not what you get when you finish splitting.** It comes from a public, reproducible benchmark published on September 17, 2026, which used 1,000 emails to pick signals, set thresholds and fit weights, then reported on a different 1,000. The version without fitted weights scored 89.4%. A two-line regex in the same benchmark scored 91.8%.

That gap is worth writing down, because "split it into five questions" is circulating in a much cheaper form than the evidence supports.

### The public test is phishing detection, not reply triage

The current wave started with a long-form post by RazeDen on September 21, 2026. He put Jev in front of Grok Bot as a decision layer, reported 62.6% on day one, reported 95% after splitting the question five ways, and wrote that "the public test tells you exactly what I saw."

That public test is [jev-phishing-bench](https://github.com/anisselbd/jev-phishing-bench), published on September 17, 2026 under the MIT licence and runnable end to end. It measures phishing detection on PhishNChips v5.2: 2,000 emails, 1,000 phishing and 1,000 legitimate, 333 of them built as cross-domain false-positive traps.

Two details carry everything downstream. Ground truth "comes from URL reputation feeds, not from a human reading each email," and the bodies are generated by the dataset's authors with a language model. The task is also not the one RazeDen runs. You can borrow the numbers. You cannot borrow the setting.

Asked once, the results look like this.

| Metric | Jev 1.13 | Claude Haiku 4.5 |
|---|---|---|
| Accuracy | 62.6% | 81.3% |
| Recall on phishing | 43.2% | 76.4% |
| False positive rate | 18.0% | 13.8% |
| AUROC | 0.689 | 0.837 |
| Calibration error (ECE) | 0.154 | 0.097 |
| Median latency (measured from France) | 239 ms | 687 ms |
| Cost per 1,000 emails | $0.038 | $0.462 |

The gap is real, at McNemar p < 0.0001. A classifier that is right 62.6% of the time and catches 43.2% of the thing it is looking for misses more than half of it in production.

The run configuration matters if you want to trust or repeat that comparison. The benchmark used `jev-1.13.0` against Claude Haiku 4.5 on the native Messages API with thinking off and temperature 0, one call per email, the same seeded order, and no concurrency.

### TypeSafe documented this as an anti-pattern before the run

The part that gets left out is that jev-1.13 ships with a page listing what not to do, and the last entry on it is **hiding several judgments inside one question**.

That [model jaggedness page](https://docs.typesafe.ai/model-jaggedness/jev-1.13) covers the exact version the benchmark ran. The page grants up front that Jev is "fast, calibrated, and good at common-sense judgment but it is not perfect", then lists nine failure modes. They start with literal reading, arithmetic ("Jev is not a calculator"), dates read as text rather than as ordered quantities, tasks it "may struggle with" because they "require additional levels of indirection", and context rot when the state fills with irrelevant detail. The other four are adversarial input, contradictory instructions and criteria, option ordering, and generation.

For literal reading, the prescribed fix is explicit. Where reading between the lines is unavoidable, "split it into two literal questions and combine them in code."

There is a second page for the general case. [Composite scoring](https://docs.typesafe.ai/patterns/composite-scoring) tells you to "Break a complex judgment into atomic scores, combine with weights you control in code," and it puts the weights in your application code rather than on the platform.

So the discovery is the vendor's own published failure mode, and the fix is the vendor's own published mitigation. That does not make the post worthless. It means its real contribution is a measured price for the mistake on public data.

### The split works, and that page never tells you to hold data back

The benchmark reports two sets of numbers, and the difference between them is whether signal selection and scoring happened on the same emails.

The control the author marks as the honest one splits the 2,000 in half. Half A picks the signals, thresholds and weights. Half B is only used to report. The five signals are domain mismatch, free hosting, lure, urgency and generic sender.

| Rule, scored on half B | Accuracy |
|---|---|
| Jev best single signal (free hosting >= 0.70) | 89.4% |
| Two-line regex rule | 91.8% |
| Jev five-signal fitted regression | 95.0% |
| Haiku best single signal (generic sender >= 0.08) | 94.2% |
| Haiku five-signal fitted regression | 93.2% |

Three things fall out of that table.

The unfitted split loses to the regex, at McNemar p = 0.0032. Those two lines are a list of shorteners, free hosts, IPFS gateways and document-sharing hosts, plus an eTLD+1 comparison between sender and link. On all 2,000 emails the regex reaches 91.6% accuracy, 83.5% recall and a 0.2% false positive rate, without calling a model at all.

The 95.0% does not beat the expensive model either. Against Haiku's 93.2% the difference is not significant, at p = 0.063, and AUROC favours Haiku at 0.991 against 0.982. Haiku asked a single best signal scored 94.2%, which lands within 0.8 points of Jev's fitted five-signal pipeline without any weight fitting at all. Splitting the question rescued Jev. It did not rescue the task.

The author also says the signal questions "were written after reading the dataset's URL-evasion taxonomy, so they target how it was built," which is why he calls the regex the honest floor. He adds that "the dataset largely separates by construction."

![Five rules scored on the half of the data reserved for reporting: Jev best single signal 89.4 percent, two-line regex 91.8 percent, Jev five-signal fitted regression 95.0 percent, Haiku best single signal 94.2 percent, Haiku five-signal fitted regression 93.2 percent](https://s4.tenten.co/learning/content/images/2026/10/english-infographic-1-4.png)

What survives is the line he puts at the end: for signals of comparable quality, Jev is "about 27 times cheaper and 5 times faster than Haiku." Price and speed win. Accuracy ties.

That ratio has a basis you can check. On the five-signal run Haiku cost $1.02 per 1,000 emails against $0.04 for Jev, with a Haiku p50 of 1,199 ms, across 2,000 calls that produced one API error and no format errors.

The composite scoring page, meanwhile, never mentions holdout data, overfitting or calibration anywhere. Its only tuning advice is that if results "are not matching your expectations, you can adjust the weights to find the right balance." The step between 89.4% and 95.0% is the one the documentation leaves to you.

### Three things you cannot skip

**Decide which half of your data you are allowed to look at once.** Split your labelled cases in two. Use one half to pick signals, set thresholds and fit weights, and touch the other half only to report, once. The benchmark's own headline figure, 95.1%, selected and evaluated on the same 2,000 emails, which is why the author marks the 95.0% split version as the control. If you have no labelled cases yet, label some before you ship, not after.

Two things about the request shape are documented, and they are the two that matter. A Noul returns the probability that the answer is yes, so the five signals come back as five numbers rather than five labels. And within one request Jev "ingests the state once and evaluates every question against it in parallel," so the signals and the verdict can travel together against the same state.

What the documentation deliberately leaves to you is the combining step, and that is the part worth writing out. The function below is mine, not the benchmark's:

```python
import math

SIGNALS = ["domain_mismatch", "free_hosting", "lure", "urgency", "generic_sender"]

def verdict(answers, weights, intercept, threshold):
    z = intercept + sum(w * answers[s] for w, s in zip(weights, SIGNALS))
    probability = 1 / (1 + math.exp(-z))
    return ("phishing" if probability >= threshold else "legitimate"), probability
```

`answers` maps each signal name to its Noul probability. `weights`, `intercept` and `threshold` all come from the fit you did on the first half, and all three live in your code rather than in the prompt — the threshold especially, because it is the only place a business decision about false positives can be written down and changed.

The request itself has two traps. Pin the model id rather than the `jev-latest` alias: an alias that gets repointed silently invalidates every threshold you calibrated. And if you also send a one-question `Choice` verdict to keep measuring the single-question baseline next to the composite, put the negative option first. Option ordering is one of the nine failure modes listed above, and the model "leans toward the option that comes first."

![Three steps you cannot skip: split the data, include the dumb baseline, run shadow mode, with Jev pricing, context and rate limit constraints](https://s4.tenten.co/learning/content/images/2026/10/english-infographic-2-3.png)

**Put the dumb baseline in the comparison.** On this dataset two lines of regex beat an unfitted five-signal split. Your task will differ, but until you know what the dumb method scores, you cannot see what the model added. Write the rules first and add the model second.

**Run shadow mode.** Step six of RazeDen's seven-step setup has Jev make every decision while nothing acts on it, so you read the logs instead. He says that step is what saved him, because the logs were bad. It is the same move as the benchmark's split control, in operational clothing: measure the thing before it can affect anything.

Three constraints belong in the design from the start. Jev bills $0.042 per million input tokens with output free. Context is 64k per request and 32k per question. Within one request, Jev "ingests the state once and evaluates every question against it in parallel," and there is no separate cap on how many questions a request may carry, provided both context limits are satisfied. Rate limits are set per organization in tokens per second and requests per second, with the live values in the developer console, so there is no fixed public number to design against.

Four documentation pages are worth reading before you design the request. `patterns/fan-out` covers sending speculative questions in one call, `patterns/intent-routing` covers routing to deterministic logic, a specialist model or a human, and the `parallel_questions` cookbook reports batching at 12.2 times cheaper and 10 times faster. The `confidence-routing` page is the one to read if your ECE looks like the 0.154 in the table above, and it draws the distinction this article has been circling: "The answer tells you what; confidence tells you whether to act."

### Three things have already moved

The source post is from September 21, 2026 and the benchmark from September 17, 2026. Both are a month old now, and the month was eventful.

`jev-1.14.0` shipped on October 14, 2026. `jev-preview` points to it and `jev-latest` still points to `jev-1.13.0`. Every accuracy number here is from the 1.13 era, so recalibrate your thresholds before you move versions.

The funding line has moved too. The $40 million seed led by DCVC was announced with the company on September 15, 2026. On October 9, 2026 TypeSafe announced an $870 million Series A at a $7.5 billion valuation, led by a16z with Sequoia and DCVC participating. Founder Diogo Almeida previously worked at OpenAI on RLHF, InstructGPT, ChatGPT and GPT-4.

There are four question primitives, not three. The documentation lists Choice, Noul (a yes/no that returns the probability the answer is yes), Score, and Box, which locates something in an image and returns four edges as fractions of the image's width and height.

One last phrase to read carefully. Jev's "no hallucinations" property means the output always conforms to the schema you supplied. It does not mean the answer is right. A wrong answer arrives in a perfectly valid shape, and the 62.6% column is what that looks like in practice.

### Sources

- [jev-phishing-bench: Jev against Claude Haiku 4.5 on 2,000 phishing emails, September 17, 2026](https://github.com/anisselbd/jev-phishing-bench)
- [TypeSafe docs: known rough edges and mitigations for jev-1.13](https://docs.typesafe.ai/model-jaggedness/jev-1.13)
- [TypeSafe docs: the composite scoring pattern](https://docs.typesafe.ai/patterns/composite-scoring)
- [TypeSafe docs: models, pricing and context limits](https://docs.typesafe.ai/models)
- [RazeDen, "I Wired TypeSafe's Jev Into Grok Bot", September 21, 2026](https://x.com/razeden0/status/2101981166500815075)