Announcing Artificial Analysis Intelligence Index v5

Jessica A model score that ignores whether a maintainer would merge the patch is starting to feel a little decorative.

Cathy Yeah. Artificial Analysis version five puts sixty percent of its composite behind private or held-out test sets. That matters right now because public benchmarks have become part of the models' diet. A leaderboard can still be useful, but only if it is measuring something the model has not already rehearsed.

Jessica And the user story is actually clearer than usual. Someone picking a model for coding, research, or knowledge work does not need another heroic demo. They need a rough answer to, "Will this survive the workflow, and what will it cost me?"

Cathy Right, but this is also a trade. Private sets make contamination harder. They make independent auditing harder too. I like the move, I just do not want us pretending a private score is suddenly an objective fact handed down from a mountain.

Jessica No, and they do give you more than a single mystery number. The index has four equal quarters: knowledge work, coding, science, and general capability. Inside coding, FrontierCode two point zero gets fifteen percent, and Terminal-Bench four point zero gets ten.

Cathy Mm-hm.

Jessica That FrontierCode piece is the interesting upgrade. It has one hundred thirty-two real open-source repository tasks, and a blocking failure means zero. The question is whether a repository maintainer would merge the change. That is much closer to a purchase decision than a neat little function test.

Cathy It also runs each task three times in their Stirrup agent harness, which is sensible. Agent results are noisy. One lucky trajectory can look like genius, and one bad tool call can look like incompetence. Three runs are not perfection, but at least they are admitting the harness is part of the measurement.

Jessica This is where I briefly become unbearable about product design. A composite score is a spreadsheet compression algorithm. Useful, yes. But it can make people forget they are buying a shape of capability, not an Olympic medal.

Cathy You celebrated episode one thousand by reading pricing tables. You are already in too deep.

Jessica Oh, stop it. One more index revision and you will be emotionally attached to a pivot table.

Cathy Fair.

Jessica The top line is still worth looking at, though. Claude Opus five point five at maximum effort leads at fifty-eight. GPT-six Astra is one point behind at fifty-seven, and GPT-six point one Sol is at fifty-five. But no model wins every evaluation, which is the part people will flatten into a graphic and then immediately lose.

Cathy Exactly. Astra leads Terminal-Bench-Science at sixty-three percent and the hard CritPt physics subset at seventy-two. Opus leads FrontierCode at forty-six percent, H L E Diamond at sixty-eight, and AA-Omniscience at forty-eight. Those are different failure modes, not just different columns.

Jessica And Terminal-Bench-Science is a pretty good stress test for the agent story. Seventy research tasks in a terminal, reviewed by experts, and every test has to pass. A polished partial answer gets you nothing there.

Cathy Right.

Jessica The spread below the leaders is sharp. Kimi K three point one Pro gets thirty-four percent there, but its weights are not released. DeepSeek V four point one Flash ten seventeen, the best released-weight model in that test, gets thirteen percent. Most of the open-weight models are below ten.

Cathy That is a much uglier open-weight gap than version four point three showed. The best overall open-weight entry is twelve points behind the leader now, versus three before. I am cautious about declaring a permanent divergence from one index revision, but the new science and knowledge tasks clearly punish something the released models are not matching yet.

Jessica You are allowed to say the gap is real without writing its obituary, Cathy.

Cathy Okay, okay.

Jessica Also, our DeepSeek KV-cache bet is still ticking. This index does not settle that one, but it is a useful reminder that a good deployment property and a lower agent score can coexist. Nobody gets to pick one number and call the whole argument over.

Cathy And effort makes the price story messier in a useful way. Opus goes from thirty-five at low effort to fifty-eight at max, while its per-task cost goes from ninety-eight cents to ten dollars and four cents. Sol lands within four points of Opus for one dollar and twenty-seven cents per task.

Jessica That is the upgrade path I can picture. Use the cheap, close-enough workhorse for volume. Escalate when the job is expensive enough to justify more thinking. We said something adjacent with Terra, but version five gives that instinct a more agent-shaped measurement.

Cathy Mm-hm.

Jessica The hidden bill is output. Median tokens per task rose from thirty-five thousand to fifty-nine thousand, and the agentic evaluations are fifty-five percent of the index weight but around ninety percent of output tokens. Terminal-Bench-Science alone is about a third of token use.

Cathy Which means the index is measuring a more demanding kind of work, but it also means the composite increasingly reflects who can afford long-running agents. That is not a flaw. It is just a property buyers should see before they mistake intelligence for a cheap request-response API.

Jessica They also shipped a separate Progress Index, which I think is a clean split. That one puts models since November twenty twenty-two on a continuous historical scale. The Intelligence Index stays focused on what is available for current work.

Cathy Yeah. Progress tracking answers, "How far did the underlying capability move?" The current index answers, "Which model should I test against this task today?" Keeping those apart saves a lot of accidental mythology.

Jessica And we can leave the mythology to the charts. Good one, Cathy. I will see you next time, when somebody inevitably tries to turn a benchmark revision into destiny.