Jev vs GPT-6 Astra: near-LLM ranking quality at 1/225th of the cost

TypeSafe recently released Jev, a small model that claims LLM-like judgment at a fraction of the price. I wanted to know whether that claim survives real data, so I went looking for a task where judgment matters and topical similarity is not enough.

Job matching fits well. The best way to decide whether a job suits a candidate is an LLM that reads the candidate’s profile and the job description together and scores the fit. It is also the slowest and most expensive option. Run it on every job that comes in and the bill grows with every job. Skip it and good matches get missed. So I benchmarked Jev against GPT-6 Astra on 241 real jobs from OutRung’s search pipeline.

In short, the claim holds. Jev ranks jobs about as well as GPT-5.6 models that cost 24 to 125 times more, and it answers in a third of a second.

The methods

I ranked all 241 jobs against my CV with GPT-6 Astra and used that ranking as the reference. Astra is far too expensive to run on every job, so the question is how closely each cheaper method reproduces its ranking, and what that costs. The 241 jobs are everything the pipeline scored for my profile over twelve days. Each method saw the same two inputs, a structured candidate profile (CV plus stated preferences) and the full job description.

Embeddings. A bi-encoder embeds the candidate and each job separately, then scores by cosine similarity. It is fast and cheap, but embeddings measure how related two texts are, which is topical overlap and not fit. Model: text-embedding-3-large.

Cross-encoder reranker. A model that reads both texts together and outputs a relevance score. It captures cross-document signals that embeddings miss. Model: ZeroEntropy zerank-2.

Jev. TypeSafe’s System One model, meaning it never generates text. It takes some state and a typed question and returns a typed answer in a single pass. I asked one question, “how suitable is this job for this candidate?”, on a five-level scale from not suitable to excellent fit. Jev returns a continuous score across those levels, not just one of five buckets, so ties are rare (167 distinct values across the 241 jobs). Model: jev-1.13.0.

LLMs. A chat model with a short neutral prompt that returns a 0-100 score and a one-sentence reason. I ran gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol (small, mid, and large), plus GPT-6 Astra as the reference. All four got exactly the same prompt.

Every method is a one-shot, pointwise scorer. Each job is scored on its own, with no retrieval, no agent loop, and no multi-step reasoning. I wanted the raw scoring ability of each approach before adding system complexity.

How closely each method agrees with Astra

The metric is Spearman rank correlation, which asks whether two methods put the same jobs in the same order. A value of 1.0 means identical rankings and 0.0 means unrelated. Order matters more than the raw number because people read a sorted list.

Three groups fall out. Embeddings reach 0.48 and the reranker 0.75. Then there is a large gap. Jev reaches 0.93, Terra 0.93, Luna 0.95, and Sol 0.96.

How tight is that top group? I bootstrapped 95% intervals by resampling the 241 jobs. Jev lands at 0.91 to 0.95, Terra at 0.91 to 0.95, Luna at 0.93 to 0.96, and Sol at 0.95 to 0.97. The four intervals overlap, and all of them sit far above the reranker (0.68 to 0.81) and embeddings (0.37 to 0.58). For practical purposes Jev belongs in the same group as the GPT-5.6 models.

That is more notable than it looks. The three GPT-5.6 models share a vendor and a lineage with Astra, so some of their agreement is family resemblance. Jev comes from a different company and is a different kind of model, and it still lands in the same band.

I also ran gpt-5.6-luna and gpt-5.6-terra at medium reasoning effort, and it did not help. gpt-5.6-luna stayed at 0.95 and gpt-5.6-terra dropped slightly to 0.91. For this task, paying for more thinking bought nothing.

A cheaper embedding model does not rescue embeddings either. text-embedding-3-small costs $0.02 per 1,000 jobs against $0.14 for the large model, but its agreement with Astra drops to 0.41.

What the scores look like

The chart below plots every job. Astra’s score runs across and the selected method’s score runs up, both on a 0-10 scale. A method that agrees with Astra forms a tight rising band. It does not need to sit on any particular line, because only the order matters.

EmbeddingRerankerJevGPT-5.6 Luna

Keep the top20%

Jev picks 41 of 49 of Astra's top 20% (84%)

Toggle between embeddings, the reranker, Jev, and GPT-5.6 Luna. Raw embedding similarity only spans a narrow range (roughly 0.39 to 0.66), so I stretched it to fill the axis to give it a fair chance. Even then it is a loose cloud. It trends upward, which is enough to weed out off-topic jobs, but it cannot tell a 9 from a 6. The reranker gets the same stretch and forms a tighter cloud, though still far wider than Jev’s.

Jev forms a tight rising band, but it marks about a point and a half more strictly than Astra. gpt-5.6-luna does the opposite and marks about a point more generously. Because each method uses the scale differently, a fixed threshold such as “keep everything above 7” means something different for each one. The slider compares them fairly by keeping each side’s top X% instead. A vertical line marks Astra’s cut-off and a horizontal line marks the method’s.

Dots in the method’s colour, top right, are jobs both picked. Red dots, bottom right, are jobs Astra picked and the method missed. Those are the mistake that matters, because a job filtered out early never reaches the user. Jobs in the top left, which only the method picked, are left grey. They are cheap mistakes, because each costs one wasted LLM call and the LLM still has the final say.

At the top 20%, Jev’s cut-off sits at 6.5 and Astra’s at 8.2, yet they choose 41 of the same 49 jobs. gpt-5.6-luna manages 39 of 49, the reranker 26, and embeddings 17. Jev scores lower, but it picks the same jobs.

Is Jev safe to use as a filter?

The practical use for a cheap scorer is as a prefilter. Score every job with the cheap model, pass only the top X% to the LLM, and save the rest of the bill. X is the pass rate. The question is whether the filter throws away jobs the LLM would have loved.

I simulated this. Sort all 241 jobs by each method’s score, keep only the top X%, and check how many of Astra’s top 10% (its 25 best jobs) survived.

At a 50% pass rate Jev keeps all 25. At 30% it keeps 24, against 25 for gpt-5.6-luna. At 20% it keeps 20, against 21 for gpt-5.6-luna. With only 25 jobs in the target set, one job is four percentage points, so Jev and gpt-5.6-luna are effectively level here too. Embeddings keep 10 of the 25 at a 30% pass rate and the reranker 16, which is why neither is a safe filter on its own.

What it costs

These are measured costs from the benchmark run, normalised to 1,000 candidate-job pairs.

Model	Cost per 1,000 jobs	Multiple of Jev	Median latency
Jev	$0.22	1x	0.3s
GPT-5.6 Luna	$5.18	24x	2.1s
GPT-5.6 Terra	$12.29	56x	2.0s
GPT-5.6 Sol	$27.52	125x	3.0s
GPT-6 Astra	about $50	225x	3.7s

Astra is priced at list rates. The embedding ($0.14) and reranker ($0.03) are cheaper than Jev, but they lose too many good matches to be useful as filters.

Why not Jev alone?

If Jev ranks as well as gpt-5.6-luna, why keep an LLM at all? One reason is decisive for OutRung.

Explanations. Users see not just a score but a reason, such as “strong fit because of your 5 years of distributed systems experience” or “weak fit because this role requires on-site in Singapore.” Jev returns a typed answer, not text. Something still has to write the explanation, and people only trust a score when they can see the reasoning.

There is also a calibration point. Jev marks about a point and a half more strictly than Astra, so a job Astra calls an 8 is closer to a 6.5 on Jev’s scale. For filtering that does not matter, because only the order counts and the cut-off simply sits lower. It would matter if Jev’s number were ever shown to users directly.

Where this leads

The benchmark suggests a split architecture in which Jev scores and the LLM explains.

Every incoming job gets a Jev score, which is cheap enough to run on every job at any cadence. Jobs that pass the threshold go to the LLM. The table shows the trade between money and missed jobs, using gpt-5.6-luna with the benchmark prompt as the LLM behind the filter. The production prompt is longer, so absolute costs there are higher and the percentage savings are what carry over.

Jobs passed to gpt-5.6-luna	Cost per 1,000 jobs	Saving	Astra’s top 10% kept
All of them, no filter	$5.18	0%	25 of 25 (100%)
Top 50% by Jev	$2.81	46%	25 of 25 (100%)
Top 30% by Jev	$1.77	66%	24 of 25 (96%)
Top 20% by Jev	$1.26	76%	20 of 25 (80%)
Top 10% by Jev	$0.74	86%	15 of 25 (60%)

On this data the 50% row loses nothing. Below that, each step buys a smaller bill at the price of a few of the best jobs, so the right setting depends on how many jobs are coming in and how much a missed job hurts.

What about an even cheaper filter in front of Jev?

The obvious next idea is a three-stage pipeline, with embeddings or the reranker thinning the list before Jev sees it. I simulated it, and it is not worth it. Jev is already so cheap that there is almost nothing left to save. Putting the reranker first trimmed the bill by $0.03 to $0.08 per 1,000 jobs. Putting embeddings first cost more than Jev alone, because the embedding call costs nearly as much as the Jev call it is meant to avoid.

The risk runs the other way. A cheap filter at the front removes jobs before anything capable has looked at them, and those jobs never come back. In this run the reranker-first pipeline happened to lose nothing, but embeddings-first dropped some of Astra’s best jobs in three of the four settings I tried. Saving a few cents is a poor trade for throwing away a good job early.

A second use: judgment calls that string matching gets wrong

Scoring is not the only place a cheap model with real-world knowledge helps. A lot of job filtering comes down to small judgment calls that look simple and are not. Location is the classic one. A candidate types “Cambridge” as their preferred location. Is a job in Royston in Cambridge? What about Ely? Neither town has Cambridge in its name, and both are an easy commute. Meanwhile Cambridge, Massachusetts matches the text perfectly and is on the wrong continent.

I ran a small probe. Jev was told the candidate prefers Cambridge, UK and will commute but not relocate, then asked one yes or no question per job location: is this within a reasonable daily commute? Alongside it I computed the embedding similarity between “Cambridge, UK” and each location string, which is what a semantic location match would use.

Job location	Jev says yes	Embedding similarity	Contains “Cambridge”
Cambridge, UK	94%	1.00	yes
Ely, Cambridgeshire	78%	0.55	yes
Huntingdon	62%	0.40	no
Newmarket, Suffolk	61%	0.42	no
Saffron Walden, Essex	60%	0.42	no
Royston, Hertfordshire	59%	0.36	no
Stevenage	49%	0.30	no
Peterborough	38%	0.46	no
London (King’s Cross)	21%	0.46	no
Milton Keynes	20%	0.41	no
Manchester	2%	0.36	no
Cambridge, Massachusetts	3%	0.75	yes
Cambridge, Ontario	3%	0.74	yes

Jev’s order is right. Commutable towns land around 60% and above, the debatable ones sit near the middle, faraway cities fall away, and both overseas Cambridges score 3%. The embedding column gets it almost backwards. Its second and third closest matches are the two overseas Cambridges, and it gives Royston, a short train ride away, exactly the same score as Manchester.

The probabilities are also usefully unsure. Jev is not claiming Stevenage is a yes. It is calling it a coin flip, which is about what a local would say.

This is 13 of 17 places and one question, so it is a promising sign and not a result. A geocoder with travel times would answer this particular question properly. The point is broader. There are many of these small calls, such as seniority, visa requirements, and whether a role is really remote, and most of them have no geocoder. At this price a handful of typed questions per job is affordable.

Caveats

This benchmark measures agreement with GPT-6 Astra, not whether people went on to apply for the jobs or get interviews. If Astra has a blind spot, every method that agrees with it shares that blind spot, and this benchmark cannot see it. The real-world check is whether users act on the jobs Jev ranks highly.

All 241 pairs come from one candidate profile. That is enough to separate the three groups of methods, but it is one person’s job search, in one field and one region. I plan to rerun on several profiles before changing production.

The LLMs used a short neutral prompt so that every method saw the same two inputs and nothing else. OutRung’s production scorer uses a longer rubric prompt with score bands. I left production scores out of the comparison on purpose, because they measure the prompt as much as the model.

The split architecture is not built yet. The cost table assumes an explain-only LLM call costs the same as a score-and-explain call, which is the conservative assumption. The next step is a prototype of the explain-only prompt, to check that explanation quality holds up when the model is handed the score instead of producing it.