OpenJev: typed decisions with calibrated probabilities from a model you train yourself

What I owned

All of it: the problem framing, the architecture, the 279-task training mixture and its audit, the evaluation harness and contamination guard, every experiment and ablation, the comparison against the commercial reference, the released models and datasets, and the technical report.

The problem

Production routing and guardrail systems ask the same kind of question over and over: which of these intents is this, how urgent is it, does a human need to see it. Each answer comes from a known, closed set. The dominant approach is to ask a language model and parse what comes back, which is expensive in three ways. Every question costs its own generation. The output is unconstrained and has to be coerced onto the menu. And the token probabilities describe a distribution over strings, not a calibrated belief over options, which is a poor basis for the thresholds operational systems actually run on.

The system that made this concrete was Jev, a closed typed-decision API from TypeSafe. It asks the right question: make the typed decision the primitive, take a state and a set of typed questions, return probabilities over fixed menus. OpenJev treats it as an existence proof and a measuring stick rather than a leaderboard rival, and asks whether the same abstraction can be built openly, trained on your own data, and run inside your own network.

Options are input, not output

An ordinary classifier ends in a layer with one column per class, so the label set is baked into the weights. OpenJev writes each option’s text into the input, places a marker next to it, and reads one score per marker with a single shared readout. The model learns a matching function, how well does this option fit this state, rather than “class 37.” Changing the menu changes the input, not the parameters, which is what makes runtime-variable menus and zero-shot transfer possible at all. This idea is not new, and the repository credits the prior work it builds on.

The contribution is answering every question about a state in one pass. The state is packed once as a shared prefix, followed by one block per question, with block-diagonal attention so each question sees the state and its own options but never its siblings. Asking twenty questions costs about the same as asking one, and nothing is generated, so there is nothing to parse.

Three primitives cover most of what production needs: Choice (one of N), Score (a level on an ordered rubric), and Noul (a probability for yes or no). Multi-label is a Noul per label rather than a Choice, because a softmax has no way to say “both.”

Two backbones, chosen by measurement

The default is a rank-16 LoRA adapter on Qwen3-1.7B. “Decoder” names the attention pattern, not the use: inference is one forward pass with a parallel read at the marker positions. It won the architecture ablation against a ModernBERT encoder on all seven held-out benchmarks. The encoder stays supported for three specific trade-offs: p95 latency at batch 1 (20 ms against 56 ms), footprint (0.6 GB against 3.4 GB), and throughput (655 states a second against 84 on an idle H100). You pay for all three in accuracy.

What it does with a few hundred labels

This is the load-bearing claim. On a healthcare router built to have headroom, a LoRA adapter trained from the base weights on 395 examples, in 258 seconds on one H100, produced three wins, four ties, and no losses against the commercial API on paired exact McNemar tests: intent 0.979 against 0.941, multi-label exact set 0.909 against 0.822, pharmacy scope gate 0.978 against 0.880.

The comparison is asymmetric and stated as such: OpenJev is fine-tuned on that task and Jev is zero-shot on it. It is not evidence that an open 1.7B model matches a closed API in general. It shows that a few hundred labels outweigh that gap on one task, which is exactly the situation most teams building a router are in.

The results that went against it

These are kept in the main text, because they change what you should do.

• Zero-shot, the commercial API is ahead on every task measured. Across seven held-out schemas it wins five and ties two, 38.3x chance against 29.6x. On Banking77 it scores 0.863 where the decoder reaches 0.605. That gap was not closed.
• Benchmark transfer did not predict deployment readiness. The decoder wins every public held-out task and still loses to the encoder zero-shot on the real router (intent 0.467 against 0.601). The ranking reverses on the one task with an operational shape.
• Starting from the general adapter hurts the decoder. A router adapter initialised from it scores 0.953 against 0.979 from base Qwen3 (p = 0.012), the opposite of the encoder, where the same move was worth 36 points. Base Qwen3 already reads menus, so the mixture adds interference rather than capability.
• Accuracy can hide a dead class. One rare class scored 0.000 recall while the model looked 0.984 accurate, which is why the guidance is to check per-class recall before trusting anything.

The evidence on the zero-shot gap points at data rather than the method. Every time data was added, generalization rose and never plateaued: the first mixture of 61 tasks transferred nothing (0.9x chance), padding training menus with distractor labels took Banking77 from 0.000 to 0.205, and scaling and auditing to 279 tasks and 323,466 rows reached the numbers above. That is one person and one GPU against instruction mixtures one to two orders of magnitude larger. Because the commercial model’s training data is unobservable, the report does not claim the remaining gap is purely data scale; it only shows that nothing measured suggests the method is the limit.

Measured, not assumed

Several design decisions came from measuring what would otherwise be guessed. Isolation between packed questions was tested adversarially, with sibling questions that assert the wrong answer, and held below the replication noise floor. Packing was benchmarked against one model per question, expecting it to trade accuracy for cost; it won both, likely because the packed model sees ten labels per state rather than one. Calibration temperatures are fitted on the dev split and saved with the checkpoint, alongside the backbone and the prompt format, so inference cannot drift from training.

Equally important are the things deliberately not built. A label hierarchy for large menus was unnecessary: chance-corrected accuracy stayed flat from 10 to 255 options. A per-option token budget was unnecessary: growing descriptions from 25 to 4,000 words cost nothing measurable. Noise-robustness augmentation was unnecessary: clean and degraded inputs scored identically (McNemar p = 1.0). Each of those would have been complexity bought with no measured return.

Lessons

The typed decision is the right abstraction for routing and guardrails, and a generative model wrapped in a parser is the expensive way to approximate it. The more transferable lesson is about evaluation. Public benchmarks ranked the two backbones one way and the real task ranked them the other, so a model is ready when it passes on a task shaped like production, not when it tops a leaderboard. And publishing the results that went against the project is what makes the ones in its favour believable: the honest headline is not “open model beats commercial API,” it is “with a few hundred labels, you can.”