Why some AI models search more and still score worse

If an AI doesn't know the answer to a Convex question, can't it just look it up? I added web search to the Convex LLM leaderboard to explore exactly that. The models take the same 112 coding tests without our guidelines, but this time they can search the web and read pages when they want to. The interesting part isn't just the scores. Some models search much more than others, and more searches don't automatically mean a better result. I look at those differences, whether models know when they need help, and how these experiments can help us improve our documentation for coding agents. Try the leaderboard: https://www.convex.dev/llm-leaderboard/no-guidelines-with-web Explore the evals: https://github.com/get-convex/convex-evals Convex documentation: https://docs.convex.dev 00:00 Can't AI just look it up? 00:25 How the 112 coding evals work 00:57 Why we give models guidelines 01:36 Making the benchmark more realistic 03:01 Adding web search 04:05 More searches, better results? 05:17 Does AI know what it doesn't know? 06:12 Are the docs helping agents? 06:50 What I'd like to test next