MEASUREMENTS
Every public measurement of Jev found so far: TypeSafe's evals dashboard, independent benchmark repositories, and vendor and builder tests on their own data.
Independent runs and vendor tests side by side; each entry quotes its own sample, task and baseline.
The public dashboard behind the launch numbers: four workflows (security incidents, agent-trace observability, invoice processing, customer service) and ten model configurations compared on accuracy, cost and latency.
A framework comparing hosted Jev with a local GLiNER2.5 on zero-shot single-label classification (AG News, Banking77, DAIR Emotion) for accuracy, calibration, selective-automation coverage and latency, with bootstrap intervals.
An independent comparison on Banking77, SST-5 and IMDB covering accuracy, calibration, latency and cost.
Benchmarks Jev as a router across 13 flagship models on LLMRouterBench and a RouterArena pilot.
A 52-page report with 50 reproducible Chinese test cases and raw API responses that re-measures TypeSafe's headline speed and cost multiples.
A UK events platform compared three models on approving or rejecting 50 real listings (venue hire, products and cancelled events filtered out), publishing the cases as a PDF.
An early-access test in which the author had Jev score his own writing corpus; widely cited by other write-ups as the first non-TypeSafe measurement.
Uses Jev as a four-tier difficulty classifier in place of NVIDIA NeMo Switchyard's router classifiers, with code.
Wires Jev into five decision points of a coding-agent harness; the main test is failure triage on 102 real provider-API failures from session logs.
A team pointed Jev at a private benchmark and reports it came within a few recall points of their fine-tuned Qwen at matched precision, while the fine-tune remains faster and has not been replaced.
Vercel benchmarked the auto-mode safety reviewer of its fx coding agent with Jev in place of GPT Luna and says it is likely to become the default.
A visual comparison places Jev next to Luna, Sonnet and Flash on email classification.
A 706K-parameter form-filling specialist is compared with Jev on a narrow evaluation.
Hundreds of pages are screened for internal-link fit through thousands of yes/no decisions.
Keep a bypass, decompose questions, tie confidence to actions and shadow-run before replacement.
Argues the headline multipliers are real but chosen against the slowest and priciest baselines; against TypeSafe's own intelligence peer the gains shrink.
Uses an open diffusion language model for structured JSON judgments without fine-tuning and benchmarks against Jev's public eval cases.
A thread testing whether many narrow Jev questions beat one direct question on a classification task with trap phrasing.
A roundup that notes the 193.6x / 444.6x come from workflow evals whose reference answers are other models' outputs, while the demo latency comparison is a different workload.
A mini-site (What Shipped / The Evals / The Guarantee / Prior Art) concluding that pricing is verified, "can't hallucinate" is schema-only, and the multiples hold only against frontier reasoning models on TypeSafe's workflows.
A thorough Japanese synthesis of the official docs, the evals dashboard, third-party tests (Every, Near Here, Classmethod) and public skepticism.
An official docs page listing failure modes of jev-1.13 with workarounds: literal reading of implied conditions and negations, unreliable arithmetic and counting, dates treated as text, weak multi-hop indirection, degradation with irrelevant state, prompt-injection vulnerability, and inconsistent invariants between related questions.
Adapts the RuneBench RuneScape agent benchmark to Jev with about fifty bounded actions, a tick-mode controller and a live dashboard.
A bilingual (Traditional Chinese / English) evidence map of where Jev holds up and where it breaks, with API-call receipts under suites/. Its thesis: success tracks whether the answer is derivable from the supplied state rather than from outside knowledge.
Evaluates Jev reranking against bge-m3 embedding search over an Agent Skills Hub catalog, with judge-bias and robustness analysis.
Measures on Banking77 when Jev can answer alone versus escalate to DeepSeek V4-Pro by confidence threshold, with risk-coverage calibration analysis.
Compares one direct Jev question per row against 12–14 Jev-scored dimensions with locally fitted weights across three classification tasks.
A thread benchmarking Jev on public prompt-injection data, thresholding calibrated probabilities in code, with a dashboard and calibration analysis in follow-ups.
A side-by-side of Jev and DeepSeek-flash on invoice extraction, email classification and reranking, with numbers on a hosted dashboard.
A web tool that runs the same task on GPT and Jev and shows money spent, time taken and tokens used side by side.
HiringCafe (2.5M monthly active users) scores resume-to-job-description relevance; the founder ran Jev against a human-labelled set and compared Spearman correlation and cost with three other models.
LangChain tested Jev against LLM judges on accuracy, repeatability, latency and cost to see whether System One models offer a new approach to online agent evaluation; evaluator code in the repository.
An open-source CLI that builds calibrated classifiers from human feedback on top of Jev, using GEPA prompt optimisation — because a foundation model still needs aligning to your own criteria.
An LLM gateway that mimics TypeSafe's structured output so local models can be scored against Jev; Needle 3 on an M4 Pro came out worse than Jev, Qwen3.8 27B and OSS 120B.
A Hugging Face dataset with a RESULTS.md comparing a fine-tuned 4B model against prompting the same model on typed decisions, run against Laya.
A Jev-style 2B decider tested on a consumer RTX 5070 Ti under Windows 11.
The ielab document-ranking toolkit gained a jev module, with code that uses TREC human relevance judgments to test whether Jev's probabilities are actually calibrated.