Building a harness with Jev
Architecture notes on putting bounded Jev judgments inside an agent harness.
ROUTING, GATING, VERIFYING
Where a typed decision sits inside a larger loop: model routing, tool-risk gates, context compaction, rollout rules.
Choose a project, read its requirements, then follow its setup reference or a related guide.
An existing TypeScript/Node project, or Python environment
Set up the Python SDK, make a typed routing decision and keep the result under your application's control.
Let Jev choose a proposed next step, then apply an explicit allow, review or fallback policy before any tool runs.
The same pattern across sources: a generative model writes, Jev decides, code executes.
Architecture notes on putting bounded Jev judgments inside an agent harness.
A long-form explanation of integrating typed judgments into agent workflows.
Jev is framed as a constrained structured-output primitive inside a larger agent harness.
A model or agent route is chosen from a fixed catalog based on the current task.
Per-turn routing chooses a model, reasoning depth and speed mode for Codex.
Jev scores which tool calls and results remain relevant before context is compacted.
A conceptual map of where a low-latency decision model can sit in software.
Keep a bypass, decompose questions, tie confidence to actions and shadow-run before replacement.
A community guide covers typed calls, batching, middleware, confidence gates and migration.
A prompt asks coding agents to inventory every bounded semantic decision where Jev might replace a heavier model call.
An agent skill for finding and designing bounded Jev decisions in software.
Vercel's changelog entry: Jev is callable through AI Gateway from AI SDK 7 (7.0.105+) via the experimental `evaluate` API as model `typesafe-ai/jev`, with Choice, Score and Noul questions.
OpenRouter's beta listing for `typesafe/jev-1.13` with a `jev-latest` alias: $0.042/M input, free output, 32K context, plus the platform's own latency and token-volume telemetry.
Workers AI exposes the model as `typesafe/jev` (jev-1.13.0) with `state` plus noul / choice / score questions and a 32,000-token context window.
Netlify's changelog: the gateway injects credentials so `@typesafe-ai/sdk` runs inside Netlify Functions without managing keys; alias `jev-latest`, Node 20+.
Vercel benchmarked the auto-mode safety reviewer of its fx coding agent with Jev in place of GPT Luna and says it is likely to become the default.
Screens agent tool calls for destructive, exfiltration, privilege-escalation and off-task actions using Jev probabilities, ships as an npm package and a Claude Code PreToolUse hook, and logs an audit trail.
A proof-of-concept MCP server exposing three tools: verify a claim against evidence, screen text for injection or relevance, and rank candidates semantically.
A Claude Code plugin that asks Jev which standing rules and codebase maps are relevant to the current prompt or file edit, injects only those, and falls back to all rules if the API is down.
A Claude Code hook where Jev decides whether a task goes to Haiku, an Opus sub-agent, or an external harness such as Codex or OpenCode.
Wires Jev into five decision points of a coding-agent harness; the main test is failure triage on 102 real provider-API failures from session logs.
A tutorial that replaces an LLM-as-judge with Jev to score Langfuse traces, with a full Python example detecting user disagreement per conversation turn.
Benchmarks Jev as a router across 13 flagship models on LLMRouterBench and a RouterArena pilot.
Uses Jev as a four-tier difficulty classifier in place of NVIDIA NeMo Switchyard's router classifiers, with code.
An experimental semantic router for Hono by its creator: routes are described in plain language and Jev matches incoming requests to them from method, URL, headers and body.
An n8n node with Ask Yes/No, Ask Choice, Ask Score, batch Evaluate and model lookup operations, defaulting to `jev-latest`.
Sits between MCP clients and servers with three layers: structural allowlists, semantic hazard detection via Jev, and session taint tracking; targets poisoned tool descriptions, injection, exfiltration and secrets.
Jev scores each request on eleven dimensions (difficulty, stakes, safety and more) and YAML thresholds map the scores to model tiers; TypeScript and Python SDKs plus an HTTP proxy.
An OpenAI-compatible router that uses Jev to pick a model on top of LiteLLM, filtering candidates by capability and falling back to the cheapest model without a TypeSafe key.
Routes Claude Code tasks to a cheaper or more suitable model based on a Jev decision; listed in the Studio Yebisu roundup.
The agentgateway proxy project ran Jev as a prompt guardrail with distributed tracing and cost tracking, with an example linked.
Tested Jev as a pre-action safety check for agents; the follow-up setup filters everything with Jev and escalates only suspicious cases to Gemini.
Foreman watches coding agents and uses Jev to continuously assess progress, completeness, tests, drift and verification, intervening when needed.
skillbox reads the query and returns the relevant agent skills immediately with Jev, instead of agents spending many turns searching for them.
Adds a `:typesafe` provider to RubyLLM 2 with Noul, Choice and Score primitives, schema building and validation, and an offline models catalog.
A Rust crate with enum responses and confidence, async or blocking backends, batched questions and observable retries; examples cover PR questions and semantic search.
A TypeScript client for batching several typed questions about your data into one Jev request; the example triages GitHub issues.
Validates state locally with Zod-style schemas, bundles all questions into one parallel Jev request and projects typed answers; built on the official SDK.
An async LangGraph workflow that classifies mock emails (invoice versus general) with the Choice primitive and routes them to handlers, with unit tests; pins jev-1.12.
Pydantic AI merged a TypeSafe model provider: `TypeSafeModel('jev-latest')` with confidence thresholds, a FileUnderstanding capability and fallback to an LLM for tool calls; docs cover jev-latest, jev-preview and pinned releases.
The LiteLLM proxy scores older tool results with Jev and replaces low-relevance ones with short removal notices before the request reaches the LLM; a TypeSafe pass-through endpoint is documented too.
Braintrust added Jev as a scorer with a confidence-threshold pattern (accept above 0.95, route 0.70–0.95 to an LLM judge, flag below 0.70 for humans) and auto-instrumentation of TypeSafe client calls in its Python SDK.
A Python package that traces TypeSafe System One calls as OpenInference LLM spans (request, response, model, token counts) for Phoenix and compatible backends; a JS package is documented in Phoenix.
A Claude Code and Codex CLI plugin that intercepts Bash output above roughly 10K estimated tokens, chunks it and asks Jev which lines are still needed, dropping progress logs and boilerplate.
A layer that decides when an agent's stored memory should be invalidated, using Jev checks; a playground is hosted.
An experimental protocol where a worker agent sends requirements, code and test results to Jev, which returns an evidence verdict for the team-lead agent before results are accepted.
A local MCP server that runs Jev "packs" (code audit, PR review, GUI automation) for Cursor, Claude Code and Codex.
A Cloudflare Worker demo where a Flue agent uses Jev through Cloudflare AI Gateway for intent and urgency routing.
A zero-dependency Ruby client for decision models supporting TypeSafe's native API and OpenRouter, with retries, timeouts and usage/cost fields.
A seven-step setup that puts Jev between Grok Bot and its actions: a jev-usage-router skill is consulted before browser, research, retry or extra-bot steps; it starts in shadow mode with logs, then goes active, with a kill switch. Repository included.
Notes from integrating Jev into a custom agent harness: beyond cheap classification, the interesting uses are continuous evals and monitoring, custom verifiers for long-horizon agents, and recursive self-improving harness loops; a guide is in progress.
A small plugin exposing Jev to Nous Research's Hermes Agent as tools: jev_check, jev_route, jev_score and jev_evaluate.
An MCP server for Jev built in about four hours and paired with a local GLM 5.3 Flash model — the author's 'absolute deadly combo'.
A multi-robot harness CLI (Microduck, Open Duck Mini, LeRobot, XLeRobot, AlohaMini and more) where each robot has an LLM brain and Jev handles the cheaper decision steps; a LeRobot SO-101 setup takes about five minutes.
Go HTTP middleware that lets Jev judge and route requests, plus a CTF-style example site inviting people to try injection attacks against it.
New Zammad ticket comes in, Jev decides which team owns it with a confidence score attached, and routes it.
Jev routes each ticket to a specialist agent (general / account / billing / technical) and decides whether a human should take it instead — all typed — while Cerebras writes the reply.
Open-source log triage for OpenTelemetry: Jev scores each log line's signal so only what matters reaches the expensive LLM analysis; a write-up covers Jev vs a fine-tuned alternative.
LangChain's webinar on using Jev to build a better agent harness, with Sydney Runkle, Hunt Lovell and Allie.
No demo: notes from three days running Jev inside production systems, misses included, organised as map → shadow → measure → promote.
Jev-powered model routing, memory, context compaction, skill selection and computer/browser use for Hermes agents (also Claude Code and Codex), with a routing dashboard and shadow mode that shows what Jev routing would have saved before you switch it on.
A portable agent plugin and MCP tool for batch text classification: the agent asks Jev whether a file is worth reading, a tool worth calling, a website worth browsing — before spending tokens on it.
An MCP connector that gives any MCP-capable agent direct Choice / Score / Noul calls to Jev; the Chinese roundups call it the first thing to install after getting an API key.
Jev as a command-line filter: classify, score and filter lines in a shell pipeline, for crawlers, CI gates and data pipelines.
Agents can write code but struggle to understand what they built at runtime; Reticle gives web and desktop apps a Jev-style runtime perception layer so agents know which interface they are looking at.
Many RL tasks need a judge to verify pieces of outputs or trajectories, and at scale that is the bottleneck; a fast, cheap, calibrated judge could remove it. Draws on earlier work with Harvey's LAB benchmark, where harness engineering plus open models cut judging costs by orders of magnitude.
Treg ('OpenRouter for agent tools') with Jev on the decision step across fraud detection and signup screening, buying-signal triage and viral-content monitoring.
An OpenAI-compatible local cache proxy (`npx @kushalicious/jevcache`): agents that re-ask the same thing get the cached answer when Jev judges the intent identical.
A modular gateway: define each agent's scope and confidence threshold, plug it in before the workflow — one reusable guard, many scopes.
A Rust CLI that pipes logs, files or any text through Jev: natural-language question in, structured semantic decision out.
A Cloud Run remote function lets BigQuery call Jev to classify support tickets and expose the resulting judgments to SQL queries.
A Telegram moderation bot that uses Jev for optional text, bio, advertising and language judgments. Image moderation runs through a separate local detector.
A small Python and HTTP example of support-request routing with Choice, frustration scoring with Score, and urgency evaluation with Noul.
Classifies Gmail threads into workflow labels with Jev and can optionally remove them from the inbox. The tool does not send mail.
An Android chat assistant that uses Jev or a compatible decision endpoint to judge message intent and risk, then offers drafts from a separate generative model.
A macOS overlay that reads WeChat on screen, runs an optional TypeSafe Jev or local decision model, and prepares reply suggestions for manual sending.
A workflow for uploading, screening, training and evaluating datasets, with Jev API calls used for the screening stage.
A Command Code mod asks Jev whether a gentle nudge would help an agent continue useful work within the user's existing request.
A DeepSeek Harness plugin asks Jev to choose a supported reasoning-effort level for a task and preserves the previous level when the choice fails.
An MCP proxy and client hooks screen agent tool calls and results using Jev among other checks.
Caliber's agent-configuration CLI includes an optional Jev plugin that scores tool-call history for retention during transcript compaction.
A Java System One client plus Spring AI modules for Jev judging, guardrails, retrieval filtering and reranking, and tool selection.
A local browser workbench for defining, previewing and publishing versioned Jev functions that callers can invoke over HTTP or MCP.
An independent agent skill for spotting Jev opportunities, designing typed questions, preserving source evidence and testing a small decision workflow.
A research-stage coding harness where an LLM proposes an action, Jev answers four bounded questions and code records evidence for the host to review.
Beacon captures sessions across coding-agent harnesses and turns reviewed workflows, corrections and debugging patterns into reusable knowledge.
A local Node and Playwright app streams partial speech transcripts, asks Jev for intent and target choices, then applies code-level action gates.
A Pi extension exposes Jev ranking, classification, local snippet retrieval and general typed evaluation while retaining the normal coding model.
A browser agent gives Jev a numbered page table and executes its chosen operation and row, using a text model when text entry is required.
A headless-browser agent uses Jev to select actions from real page elements and exposes the same navigation loop through MCP, CLI and a library.
A DeepSeek Harness plugin uses Jev for tool relevance, pre-execution call assessment, skill hints and model-route selection through a reusable decision core.
A TypeScript browser SDK combines native Playwright actions with Jev decisions over observed elements. Its CLI, MCP server and library share the same execution and verification core.
An MCP server uses Jev to drive Chrome through observed page elements, either in an existing browser through an extension or in a separate profile. Tasks expose progress, questions and extracted results to the host agent.
A self-hosted router asks Jev to assess fresh coding tasks and applies YAML policies to choose an upstream model and reasoning effort. Strict sessions retain their admitted model, provider and limits through later tool calls.
A Claude Code plugin sends instruction checks, context retention and subagent routing judgments to Jev. Context compaction retains selected original blocks instead of writing a new summary, with a Pi compaction adapter also included.
A CLI, MCP server and library combine ripgrep recall with Jev judgments over candidate files and passages. Results contain original source excerpts with paths, line numbers and explicit omitted ranges.
A Python routing engine combines inspectable YAML rules with Jev judgments to assign requests to agents or skills. Its Claude Code hooks can discover the installed roster and turn confidence bands into routing, suggestion or confirmation verdicts.
A local MCP server gives coding agents Jev judgments for loop routing, patch review, evidence checks and candidate ranking. Its prepared-call router selects an exact host-supplied tool call while retaining the original arguments locally.
A CLI evaluates Jev against labelled application data, chooses per-question thresholds on one split and checks them on held-out examples. It writes a threshold lock and report, supports low-confidence fallback, and fails CI when accepted accuracy or coverage drifts.
A local Rust search tool combines keyword discovery with Jev ranking and delivers source excerpts through a CLI or MCP. Its setup command connects Codex, Claude Code and OpenCode and can supply retrieval context when a code question is submitted.
A Rust CLI and TypeScript SDK turn a sentence into an inspectable plan over installed programs called reflexes. Jev selects offered programs and bounded inputs; deterministic code applies execution and confirmation rules.
Middleware for LangChain deepagents selects the skills a user turn needs instead of listing the entire catalog in every prompt. A provider-neutral routing core includes a Jev adapter that ranks descriptions and verifies shortlisted instructions.
An experimental Pi extension maintains quoted rule and task ledgers and checks side-effecting tool calls against them. The main agent records user receipts; Jev handles bounded semantic checks while code enforces resource, scope and precedence rules.
A Python browser agent uses Jev decisions over observed page candidates and returns structured results or cited answers. Browser execution, spending controls and result verification are handled by separate code paths.
A bounded executor lets a parent agent hand browser work to Jev over agent-browser snapshots. The parent receives structured handoffs when the task is blocked, repeats or reaches its step budget.
A Databricks app discovers model endpoints, MCP servers and skills from Unity Catalog metadata. Jev selects an agent specification before the OpenAI Agents SDK constructs and invokes the selected configuration.
A Rust Anthropic-compatible proxy uses Jev judgments to select Claude model and effort settings. Deterministic routing rules also account for continuations, side requests and prompt-cache considerations.
Claude Code function hooks ask Jev for a model tier and effort at turn start and apply the selection as the turn runs. A sticky cache guard and usage footer expose the selected route and actual model.
A Python CLI and library reduce a skill catalog into bounded candidates before asking Jev which skill fits. Choice and fit signals produce a route, no-skill result or review outcome.
A shared TypeScript decision tool integrates natively with Pi and through local MCP with Codex. It collects bounded conversation context from a known session and returns compact structured results through OpenRouter Decisions.
A Linux and WSL2 tmux controller provides a local web interface for Claude and Codex sessions. Jev maps typed or spoken requests to bounded operations such as sending, switching, reading or interrupting.
An evaluation library expresses agent, quality and security checks as typed questions and packs compatible checks into shared decision requests. The same definitions can measure recorded traces or gate an agent step, using hosted Jev or supported local decision engines.
A Qwen3-0.6B backbone with its language-model head removed scores caller-supplied candidates through a permutation-equivariant head. The project releases training, serving and prefix-sharing code for Boolean, Choice and Score decisions, including a coding-completion checkpoint.
A self-hosted API gateway forwards Jev requests through the native System One endpoint and manages TypeSafe API-key accounts. Choice, Noul and Score answers keep their structured response format.
A Rust agent harness uses Jev to rank deferred tools against the current intent. Its TinyHumans-connected evaluator combines a tool Choice with a Noul question about whether a tool is needed.
AgentScope exposes Jev through its classifier interface, translating binary, choice and score questions into TypeSafe primitives. The adapter converts structured answers back into framework classifier results.
Composio includes a TypeSafe provider for selecting tools and binding arguments with explicit choices. Its implementation separates closed argument sets from values that require an open-ended companion model.
TradingAgents optionally screens social posts with Jev before a sentiment analyst reads them. It removes unrelated company mentions and groups remaining posts by their stated stance.
The investment-agent framework includes a native TypeSafe adapter for its structured result contract. Jev answers are converted into the compatibility JSON expected by the surrounding agent pipeline.
QuantDinger can apply Jev checks before live entry orders, combining strategy context with account exposure and risk information. The application records individual choices, confidence and timing in a decision timeline.
AgentConnect provides a TypeSafe decision service inside its local agent daemon. It validates Noul, Choice and Score responses before using them in agent coordination.
The macOS Agent! application can ask Jev for advisory judgments about its planned actions. A Swift middleware layer defines the questions and converts returned verdicts into application decisions.
JevHarness lets a language model author task-specific code that builds Jev questions and combines typed answers into actions. A frozen harness can execute without calling the authoring model for every decision.
OpenMausBot can use Jev to select an agent for a collaborative room. The decider validates offered choices and probability values before forwarding a decision to the surrounding routing system.
Agent Squad includes JevClassifier implementations in TypeScript and Python. A typed Choice question selects among configured agents using the request and conversation context.
PyGPT offers an inline Jev plugin for structured semantic decisions inside its assistant workflows. It supports classification, selection, verification and scoring over supplied state.
Paca adds per-project Jev decisions for agent selection, task assignment and filling supported empty task fields. Automation conditions can branch on typed answers while keeping human-set values intact.
OpenHarness ships a Jev Sheets harness for trying typed questions on table rows. Its Question Lab compares two formulations on a frozen sample and records answers, provider usage and review notes.