Jev AI costs less than a cent to run my benchmark

Jev is a new AI "decision model" from TypeSafe. It doesn't write text, it just picks an answer. So is it actually any good at Convex? I turned our Convex coding evals into 108 multiple-choice questions and ran Jev against 14 LLMs, from GPT-6 Astra and Claude Opus 5.5 down to Qwen and Mistral. Jev scored 84.6%. That's lower than the frontier models, but it answers in about 200ms, is roughly 10x faster than GPT-6 Astra and about 180x cheaper. The whole benchmark cost it less than a cent. I also built a Magic Jev Ball on Convex using the new AI Gateway, which you can play with below. Chapters 0:00 The Magic Jev Ball 0:32 How do you test a decision model? 1:46 What are decision models? 2:16 Turning coding evals into decisions 3:43 LLMs vs decision models 4:21 The new Convex evals page 5:16 Decision leaderboard results 6:42 Digging into Jev's mistakes 8:55 What would you actually use Jev for? 10:11 Wrapping up Links Decision evals leaderboard: https://www.convex.dev/evals/decision Magic Jev Ball: https://magic-jev.mikecann.app/ Magic Jev Ball source: https://github.com/mikecann/magic-jev-ball Convex evals repo: https://github.com/get-convex/convex-evals Convex AI Gateway video: https://youtu.be/K4rrd90JJG4 Convex AI Gateway docs: https://docs.convex.dev/ai-gateway/overview Open-source decision models mentioned OpenJev: https://huggingface.co/openjev/openjev Laya: https://huggingface.co/convaiinnovations/laya Decider-2B: https://huggingface.co/Mapika/decider-2b