amal.
← Engineering

Engineering ·

Exploring Jev AI for observability: better metrics or just another AI gimmick?

A new AI model that refuses to talk, and why a platform engineer might actually care.

Every few weeks, some new AI tool shows up promising to "revolutionise observability". Most of them, I scroll past. Jev made me stop.

Jev, from TypeSafe AI, came out in September 2026. It doesn't chat. It doesn't write code. It doesn't summarise your logs into a friendly paragraph. It makes decisions. You hand it some state, ask it a few typed questions, and it hands back typed answers with probabilities attached. That's it.

Most of my working day goes into keeping production platforms healthy: Pacemaker clusters, Oracle databases, Jenkins pipelines, and the occasional 2 AM incident bridge. So my first reaction was mixed. One part of me thought, finally, something that speaks the language of software instead of prose! The other part, the one that has been burnt by shiny tools before, asked the obvious question: is this a genuinely better way to define metrics, or just one more gimmick with an observability sticker on it?

This write-up is me thinking out loud on that question.

First, what Jev actually is

TypeSafe calls Jev a "System One Model". The name comes from Daniel Kahneman's Thinking, Fast and Slow: System 1 is the fast, intuitive judgement, System 2 is the slow, deliberate reasoning. Chat models are the System 2 side. Jev is TypeSafe's attempt at building the System 1 side directly (InfoQ).

It is trained with something they call Reinforcement Learning for Calibrated Decisions (RLCD), and the company was founded by Diogo Almeida, an ex-OpenAI researcher who co-invented RLHF (The Register). Instead of predicting the next token one at a time, it returns all its answers to a request together. That is where the speed comes from.

You ask Jev three kinds of questions:

Question typeWhat you get backAn observability-flavoured example
NoulProbability that a yes/no statement is trueDoes this failover need a human to look at it?
ChoiceThe picked category, probabilities for every category, and a confidenceWhat is the most likely failure mode in this trace?
ScoreA probability-weighted score on a rubric you define, plus the distribution and a confidenceHow bad is the customer impact, on a 0–3 scale?

The output space is fixed before the call. There is no JSON to repair, no "please respond only in valid JSON" begging in the prompt, no retry loop because the model decided to add a friendly sentence before the curly brace. One article I read called it a "smart if-statement" (Level Up Coding), and honestly, that's the best mental model I've found. The model evaluates the fuzzy condition. Your normal code decides what happens next.

On speed and cost, TypeSafe claims roughly two orders of magnitude over frontier LLMs, on the kinds of questions a program asks many times a day (NYU Shanghai RITS). Keep that claim in mind. I'll come back to it.

The problem: "is it up" is not "is it right"

Anyone who has run production HA knows this feeling. pcs status says every resource is Started. The database is open. The listener is up. CPU is fine, latency is fine, error rate is flat. And yet, the application is quietly doing the wrong thing, and the first alert you get is an angry email from the customer.

Our metrics are very good at measuring things that are easy to count. They are terrible at measuring things that need judgement.

AI agents make this gap wider. A developer building on Jev put it nicely: an agent that edits, tests and reverts the same file six times in a row looks perfectly healthy in the logs (source). Every span succeeded. Every tool call returned. No log line says "I'm stuck".

So how do we turn a question like "is this reply grounded in the policy documents?" or "is this agent going in circles?" into a metric?

Until now, there were two answers:

  • A human looks at it. Accurate, but doesn't scale beyond a sample.
  • LLM-as-a-judge. Ask a text-generating model for a yes/no, wrap it in a JSON schema, parse it, retry when it breaks, and pay generation prices for what is basically a single bit. Datadog's engineers made exactly this point when they wired Jev into their Agent Observability product (Datadog).

Jev's pitch is a third answer: a model that only produces that single bit (or a small, typed set of them), produces it fast, and tells you how sure it is.

Where it earns its place

The more I read, the less the interesting part was "AI" and the more it was metric design. A few ideas stood out.

Store the probability, not the verdict

The Datadog write-up recommends submitting Jev's raw probability as the metric and keeping the pass/fail cut as a separate field. If you binarise at write time, changing the threshold later means re-scoring your whole backlog. If you keep the probability, changing the threshold is just a query change.

This is the same lesson I keep relearning with Prometheus: record the histogram, not a precomputed "slow: yes/no" flag. Raw data first, opinions at query time.

Thresholds are policy, not model judgement

In their example, the final "was this handled correctly?" verdict could have been one more question to Jev. They deliberately kept it in plain code instead, combining a few probabilities against thresholds. Their reasoning was simple: thresholds are application policy, so they should be easy to read and retune without touching the model.

That's the architect in me nodding along. The model tells you what it sees. Your code, your config, your team decides what to do about it.

Confidence is a signal on its own

In one sample response, Jev picked none as the failure mode, but none was at 0.46 and partial_answer at 0.42, with an overall confidence of 0.34. Flatten that to the string none and you lose the most useful part. A near tie is exactly the trace you want a human to look at.

There's a second-order signal too. If the evaluator's confidence keeps dropping over weeks, it may not be the system getting worse. It may be your rubric no longer matching your traffic. That's drift in the measurement, and very few of our existing metrics can tell you that.

One request, many questions

All questions in a request are answered independently against the same state. So five checks on one trace is one call, not five. For something that runs on every span, that matters.

Pin the version, like you would an image tag

They pin a specific model version (jev-1.13.0) instead of jev-latest, because the thresholds were calibrated against that exact version, and tag every metric with the model that answered. I have never been so happy to see a vendor tell people not to use latest!

What a question might look like in my world

Here is roughly how I'd frame a Noul question over a cluster log. This is illustrative, not something I have run against production:

questions = {
    "needs_human": Noul(
        instructions="Does the failover sequence in `cluster_log` need a human to review it?",
        criteria=NoulCriteria(
            true="A resource failed to start after moving, or a node was fenced without an obvious trigger.",
            false="A clean, expected failover with every resource started on the surviving node.",
        ),
    ),
}

Notice what is not in there: counting how many resources moved, or checking timestamps. That stays in code. More on that next.

Where it starts to look like a gimmick

Now the other side. Because there definitely is one.

Typed is not the same as correct. You will see "can't hallucinate" floating around. That's a claim about format, not truth. Jev cannot return an invalid type, but it can absolutely return a wrong decision in a perfectly valid type (NYU Shanghai RITS). A confidently wrong 0.92 on a dashboard is arguably worse than no metric at all, because people trust graphs.

It never explains itself. That's the design, and it's why it's fast. But during an incident, or in a postmortem, "P(grounded) = 0.31" is not an explanation. You have to store the question, the criteria and the evidence next to every score, otherwise nobody can tell why it fired.

It never abstains. A Choice question always returns the highest-probability option. If you want "I can't judge this one", you have to add that option yourself.

It is the wrong tool for deterministic things. Per the Datadog team, Jev reads dates as text and doesn't count reliably, so anything a parser can compute belongs in code. Using Jev to decide whether CPU is above 80% is the purest form of gimmick. If PromQL can answer it, PromQL should answer it.

More context makes it worse, not better. Accuracy drops as the state fills up with material the question doesn't need. So you can't just throw a whole trace or a 10 MB alert log at it and hope. Filtering in code is part of the job.

The headline numbers are the vendor's own. The "100x faster and cheaper" results come from evals TypeSafe designed. That doesn't make them false. It makes them a starting hypothesis for your own traffic, not a fact about it.

It is one more dependency on the monitoring path. My own ops worry, more than anything I read: if the thing that decides "does this need a human?" is an external API, what happens when that API is slow or down during the exact incident you need it for? Who monitors the monitor?

And it is one more metric. Most teams I've seen don't suffer from too few signals. They suffer from too many that nobody acts on. A new probability metric nobody looks at is just alert fatigue with a nicer name.

My rules before it goes anywhere near a pipeline

If I were to bring Jev into a real observability setup, these would go into the design doc before a single line of code:

  1. Never ask it anything a parser, a regex or PromQL can compute. Counts, dates, thresholds, durations: code.
  2. Store probabilities, not verdicts. The pass/fail cut lives in config, next to the rest of the alerting rules.
  3. Keep the decision in code. Jev answers narrow questions. The combination of those answers into "page someone" is our policy, versioned in Git like everything else.
  4. Pin the model version and tag every metric with it. Same reason we don't deploy :latest to production.
  5. Give every Choice question an escape hatch. An explicit unclear option, and route those to a human.
  6. Filter the state before sending. Send only the fields the question reads, not the whole trace.
  7. Score out of band, never in the request path. A separate worker, asynchronous, so a slow or unavailable judge never slows the system it's judging. The Datadog example does exactly this.
  8. Calibrate against humans first. Label a few hundred real cases by hand, measure agreement and repeatability, and re-run that check whenever the rubric or the model version changes.

None of these rules are about AI, really. They are the same rules we'd apply to any new data source we want to trust at 2 AM.

So, optimised metrics or unnecessary gimmick?

Honestly? Both. It depends entirely on what you point it at.

Point it at things we can already measure deterministically, and it's a gimmick. A slower, costlier, less reliable way of doing what a recording rule already does for free.

Point it at the signals we currently don't measure at all, the "is this right?" and "is this stuck?" questions we leave to customers and gut feeling, and it starts to look like a genuinely better way to define a metric. Not because it's smarter than an LLM-as-judge, but because it's narrower. A typed question, explicit criteria, a probability you can graph, and a confidence that tells you when to stop trusting the graph.

The part I like most isn't even the model. It's the discipline it forces on you. You can't throw a vague prompt at Jev and get a vague paragraph back. You have to decide, upfront, exactly what question you're asking and what each answer means. Most of our existing dashboards would be better if someone had been forced to do that.

I'll be trying it on a small, non-critical use case before forming a stronger opinion. If it holds up, that will be its own write-up.

Jev won't replace an engineer's judgement during an incident. At best, it gives the fuzzy parts of that judgement a number I can graph. And that might be enough.

Sources