MEASUREMENTS
Every public measurement of Jev found so far: TypeSafe's evals dashboard, independent benchmark repositories, and vendor and builder tests on their own data.
Independent runs and vendor tests side by side; each entry quotes its own sample, task and baseline.
The public dashboard behind the launch numbers: four workflows (security incidents, agent-trace observability, invoice processing, customer service) and ten model configurations compared on accuracy, cost and latency.
A framework comparing hosted Jev with a local GLiNER2.5 on zero-shot single-label classification (AG News, Banking77, DAIR Emotion) for accuracy, calibration, selective-automation coverage and latency, with bootstrap intervals.
An independent comparison on Banking77, SST-5 and IMDB covering accuracy, calibration, latency and cost.
Benchmarks Jev as a router across 13 flagship models on LLMRouterBench and a RouterArena pilot.
A 52-page report with 50 reproducible Chinese test cases and raw API responses that re-measures TypeSafe's headline speed and cost multiples.
A UK events platform compared three models on approving or rejecting 50 real listings (venue hire, products and cancelled events filtered out), publishing the cases as a PDF.
An early-access test in which the author had Jev score his own writing corpus; widely cited by other write-ups as the first non-TypeSafe measurement.
Uses Jev as a four-tier difficulty classifier in place of NVIDIA NeMo Switchyard's router classifiers, with code.
Wires Jev into five decision points of a coding-agent harness; the main test is failure triage on 102 real provider-API failures from session logs.
A team pointed Jev at a private benchmark and reports it came within a few recall points of their fine-tuned Qwen at matched precision, while the fine-tune remains faster and has not been replaced.
Vercel benchmarked the auto-mode safety reviewer of its fx coding agent with Jev in place of GPT Luna and says it is likely to become the default.
A visual comparison places Jev next to Luna, Sonnet and Flash on email classification.
A 706K-parameter form-filling specialist is compared with Jev on a narrow evaluation.
Hundreds of pages are screened for internal-link fit through thousands of yes/no decisions.
Keep a bypass, decompose questions, tie confidence to actions and shadow-run before replacement.
Argues the headline multipliers are real but chosen against the slowest and priciest baselines; against TypeSafe's own intelligence peer the gains shrink.
Uses an open diffusion language model for structured JSON judgments without fine-tuning and benchmarks against Jev's public eval cases.
A thread testing whether many narrow Jev questions beat one direct question on a classification task with trap phrasing.
A roundup that notes the 193.6x / 444.6x come from workflow evals whose reference answers are other models' outputs, while the demo latency comparison is a different workload.
A mini-site (What Shipped / The Evals / The Guarantee / Prior Art) concluding that pricing is verified, "can't hallucinate" is schema-only, and the multiples hold only against frontier reasoning models on TypeSafe's workflows.
A thorough Japanese synthesis of the official docs, the evals dashboard, third-party tests (Every, Near Here, Classmethod) and public skepticism.
An official docs page listing failure modes of jev-1.13 with workarounds: literal reading of implied conditions and negations, unreliable arithmetic and counting, dates treated as text, weak multi-hop indirection, degradation with irrelevant state, prompt-injection vulnerability, and inconsistent invariants between related questions.
Adapts the RuneBench RuneScape agent benchmark to Jev with about fifty bounded actions, a tick-mode controller and a live dashboard.
A bilingual (Traditional Chinese / English) evidence map of where Jev holds up and where it breaks, with API-call receipts under suites/. Its thesis: success tracks whether the answer is derivable from the supplied state rather than from outside knowledge.
Evaluates Jev reranking against bge-m3 embedding search over an Agent Skills Hub catalog, with judge-bias and robustness analysis.
Measures on Banking77 when Jev can answer alone versus escalate to DeepSeek V4-Pro by confidence threshold, with risk-coverage calibration analysis.
Compares one direct Jev question per row against 12–14 Jev-scored dimensions with locally fitted weights across three classification tasks.
A thread benchmarking Jev on public prompt-injection data, thresholding calibrated probabilities in code, with a dashboard and calibration analysis in follow-ups.
A side-by-side of Jev and DeepSeek-flash on invoice extraction, email classification and reranking, with numbers on a hosted dashboard.
A web tool that runs the same task on GPT and Jev and shows money spent, time taken and tokens used side by side.
HiringCafe (2.5M monthly active users) scores resume-to-job-description relevance; the founder ran Jev against a human-labelled set and compared Spearman correlation and cost with three other models.
LangChain tested Jev against LLM judges on accuracy, repeatability, latency and cost to see whether System One models offer a new approach to online agent evaluation; evaluator code in the repository.
An open-source CLI that builds calibrated classifiers from human feedback on top of Jev, using GEPA prompt optimisation — because a foundation model still needs aligning to your own criteria.
An LLM gateway that mimics TypeSafe's structured output so local models can be scored against Jev; Needle 3 on an M4 Pro came out worse than Jev, Qwen3.8 27B and OSS 120B.
A Hugging Face dataset with a RESULTS.md comparing a fine-tuned 4B model against prompting the same model on typed decisions, run against Laya.
A Jev-style 2B decider tested on a consumer RTX 5070 Ti under Windows 11.
The ielab document-ranking toolkit gained a jev module, with code that uses TREC human relevance judgments to test whether Jev's probabilities are actually calibrated.
An inspectable comparison of Jev Noul and Qwen3-Reranker-0.6B on retrieval ranking tasks, with the harness and raw model responses published.
An independent 0.8B Qwen3.5 model for typed decisions, published with weights and a comparison against Jev.
A Chinese-language learning repository with five runnable Python demos and the author's Jev 1.13.0 experiments on question fan-out and decision boundaries.
A CLI that uses Jev to judge relevance across repository folders, files and declarations, returning file leads and verbatim source excerpts to a coding agent.
A Python library that turns open causal models into Choice, Noul and Score decisions with access to logits or hidden states.
Text and multimodal LoRA adapters on Qwen3.5-4B for closed-option GUI element and action decisions, published as a separate Cua-S1 model version.
A Rust engine derives typed System One judgments from model log probabilities and exposes them through CLI, MCP and a Jev-compatible HTTP server.
A testable System One specification and CLI check Jev-compatible servers and provide a normalizing proxy and fault-injectable reference mock.
A configurable LIBERO robot-control experiment that combines Jev decisions with reversible physics previews. An interactive replay follows intent, motion family and control choices alongside saved trajectories.
A local TypeScript library and CLI where Jev chooses exploratory browser actions and assesses observed results. Playwright executes configured actions, exact assertions decide outcomes, and saved traces support replay.
A browser and macOS computer-use harness that checks completion claims against observed state and gates irreversible actions with Jev. Its dual mode also delegates mechanical steps to Jev while a driver handles broader decisions.
An independent Jev-style project that freezes a language-model backbone and trains a small decision head over cached candidate features. It publishes typed probability outputs, datasets and browser presentations of its benchmark and game policies.
An independent reranking benchmark paired with a live natural-language search over Y Combinator companies. The app uses Jev for query routing and reranking after keyword and embedding retrieval.
An independent evaluation site and data repository for Jev and LLMs answering the same typed questions. Public boards link accuracy, calibration, costs and latency to human-labelled tasks and individual decision logs.
A Playwright reporter asks Jev which discovered browser tests are relevant to a code change before browser execution. It combines changed paths and patches with test metadata and optionally test source.
A task-specific pipeline generates or labels data with an LLM teacher, trains a LoRA and decision head, then evaluates and serves Choice, Score and Noul probabilities. A public support-triage replay shows saved model outputs and review routing.
An end-to-end research toolkit builds decision records, trains and calibrates candidate scorers, and serves a Jev-compatible API. Its released Qwen3.5-0.8B model ranks supplied web actions, while the public demo presents recorded Mind2Web decisions.
An independent family of LoRA adapters and scalar decision heads returns Choice, Noul and Score outputs over pinned Qwen backbones. The project publishes training data, evaluation records, a CPU classification workbench and interactive saved-decision replays.
A Qwen3.5-4B LoRA adapter answers multiple typed questions without generating answer tokens. Its runtime combines label-token readout with a learned candidate head, option-order averaging and saved calibration profiles.
A local NLI backend scores caller-defined Choice, Score and Noul questions, with temperature scaling and low-confidence abstention. Python, CLI and MCP interfaces expose the same decision layer.
A benchmark compares Jev and other typed decision systems using frozen tasks, explicit adapters and published result artifacts. Its score views combine capability, calibration, latency and estimated cost, with public and sealed evaluation material.
A CLI evaluates Jev against labelled application data, chooses per-question thresholds on one split and checks them on held-out examples. It writes a threshold lock and report, supports low-confidence fallback, and fails CI when accepted accuracy or coverage drifts.
A preregistered audit compares English and Spanish state and instruction combinations for Jev on paired human-labelled datasets. The repository publishes the runner, frozen prompts, response-derived results and a CLI for repeating the protocol on other labelled data.
A deterministic restaurant simulator that replays the same seeded dinner under rule-based, camera-assisted and Jev-planned service. Servers choose from feasible task tours, and the app logs the observations, candidates, confidence, latency and resulting service metrics.
An independent evaluation of pinned jev-1.13.0 across classification, routing, inference, moderation, legal clauses and rubric scoring. The paper uses frozen task templates, reports calibration and selective prediction, and releases its harness and raw model responses.
A training-free algorithm that recovers numerical outputs by recursively narrowing a range through a multiway tree of Jev-like choices. The paper preserves the original question in context and studies how range selection differs from direct candidate selection.
Distilled Qwen models answer Choice, Noul and Score questions by reading option-token logits. The project releases 4B and 9B weights, fitted temperatures, a TypeSafe-compatible server and runtimes for CUDA or GGUF.
A training and deployment library adapts language-model backbones to typed decisions with learned pointer heads or direct option-token readouts. It publishes LoRA checkpoints, data preparation, calibration and evaluation tools alongside an archived application replay gallery.
A family of open 4B and 27B models applies stated rules and policies through typed option probabilities. The repository includes data generation, distillation, quantization, evaluation and a TypeSafe-compatible server, with GPU and Apple Silicon builds.
An evaluation library expresses agent, quality and security checks as typed questions and packs compatible checks into shared decision requests. The same definitions can measure recorded traces or gate an agent step, using hosted Jev or supported local decision engines.
An offline linter checks Jev request design before inference, including missing escape options, broken state references and malformed criteria. An optional semantic layer asks Jev whether a structurally valid question has missing or ambiguous answer boundaries.
An independent CUDA runtime scores verified single-token option labels with existing language-model weights. It batches independent fields through vLLM, reuses shared prefixes when supported and assembles typed output in code; browser experiments also use local WebGPU models.
A frozen-protocol study checks Jev's option probabilities and output schema on Banking77 and CLINC150. It publishes corpus pins, raw responses, scoring rules, amendments, controls and a separate blind rescore.
A reproducible zero-shot evaluation freezes the LexGLUE test splits and maps legal classification tasks to Choice or per-label Noul questions. Its runner saves completed requests, checks model-version changes and writes micro/macro-F1, calibration, usage and cost reports.
A retrieval study compares typed Jev scoring strategies with dedicated rerankers and open-model controls on the same passage candidates. Saved provider responses, query-level evidence, bootstrap intervals and an interactive viewer support inspecting each result.
A measurement harness compares many typed questions in one Jev request with the same questions asked separately. It varies state size and question count, repeats requests to estimate noise and publishes response-level usage, billing and latency evidence.
A local workbench runs open decision engines side by side and can compare them with hosted Jev. It includes a visual request builder, dataset benchmarks, truncation diagnostics and resource or concurrency measurements with separate engine processes.
An independent experiment compares Jev's reported probabilities with known uncertainty on fair dice, coins and option sets. It also tests whether typed decisions preserve probabilities stated in forecast documents, publishing recorded outputs and offline analysis scripts.
A MuJoCo robot workbench compares bounded skills, incremental movement and hierarchical control on manipulation tasks. It exposes decision traces, physical checks and alternative providers, including Jev, compatible chat models and a local MiniCPM model.
A Franka Panda simulator separates Jev's choice of manipulation intent from its choice of movement and gripper commands. A local executor applies those commands, and independent physical checks determine whether a task succeeded.
DeepEval offers Jev-based evaluation metrics and a TypeSafe model configuration. Custom evaluation criteria are converted into structured judgments for individual responses and conversations.
Opik instruments synchronous and asynchronous TypeSafe SDK clients. Evaluation states, questions, returned answers and usage can be captured in application traces.
A mathematics-to-animation pipeline uses Jev to review checkpoints while a separate authoring model develops explanations and Manim scenes. Typed questions assess evidence, blocking defects and the proposed repair target.