Introducing the Conceptual Reasoning Index

Pippa This matters because the hard questions are usually the ones where nobody can hand you an answer key afterward.

Tyler Yeah. Models are getting very good at tasks with a clean reward signal, then we ask them to help reason about long-horizon choices where the feedback may arrive absurdly late. Or never.

Pippa Right. And this new Conceptual Reasoning Index is trying to put a number on that uncomfortable middle zone. Not whether a model can solve a coding puzzle, but whether it can engage with arguments about alignment, governance, decision theory, all that stuff where the conclusion is genuinely contested.

Tyler Mm-hm.

Pippa Also, small confession, my week has been a little benchmark-shaped. Every new chart claims it captured the soul of intelligence, then you open the methodology and find twelve footnotes doing all the work.

Tyler The footnotes are where the little goblins live.

Pippa Exactly. This one at least seems unusually candid about its goblins, which I appreciated.

Tyler The core design is three datasets. LMCA asks models to judge arguments written against a position text. ACCoRD checks whether a model's reported probabilities and preferences obey basic consistency constraints. DTBench capabilities tests decision theory, especially cases involving predictions of the model's own behavior or near-copies.

Pippa Okay okay.

Tyler That split is clever because it avoids pretending there is one settled answer to a philosophical question. With LMCA, the target is narrower: can the model assess an argument using a detailed rubric in a way that tracks expert judgment?

Pippa And that is a real user story, even if it is not a shiny consumer feature. A research team comparing model candidates for sensitive analysis could use this as one signal before letting a model draft strategy, critique a proposal, or reason through a weird edge case.

Tyler Right.

Pippa Though the adoption path is pretty specialized. LMCA access requires a request form, and the public thing is the CRI website with live scores. This is not a plug-in for someone shipping a support bot by Friday.

Tyler No, and honestly that restraint fits. The dataset has five hundred sixty position texts and one thousand four hundred sixty-one arguments against them, with two thousand one hundred forty total ratings. They even had a roughly fifty-argument validation set rated independently by four to six people, followed by seven to eight hours of discussion.

Pippa That last bit is oddly reassuring. Somebody actually sat in the messy part of the work instead of declaring that disagreement had been solved by a spreadsheet.

Tyler A spreadsheet would have done it faster, obviously. Much worse, but faster.

Pippa This is your recurring nightmare. Product people celebrating the dashboard before anyone asks what the axis means.

Tyler Pippa, it is not a nightmare. It is an extremely well-observed production incident.

Pippa Fair. But there is a genuinely nice product implication here: judging arguments can be useful even when nobody expects the model to discover final truth. It can help people find weak links, missing assumptions, or claims that need a human to inspect.

Tyler I agree, with one big caution. This runs straight into our construct-validity fight from episode eight thirty-three. The convergent-evidence camp wants several different measures to move together. Your outcome-evidence camp asks whether the score predicts a real consequence someone cares about. CRI is one more serious entrant in that fight, not the verdict.

Pippa Yeah, no, completely. If a high CRI model produces better research critiques in practice, that is compelling. If it mostly learns the local style of this rubric, then we built a very refined elevator shaft and everyone is nodding at the word conceptual.

Tyler Fair. And the aggregate has some decisions worth keeping visible. CRI is sixty percent LMCA, twenty percent ACCoRD, and twenty percent DTBench capabilities. The team says it may add benchmarks, retire saturated ones, and change weights later, so the index is useful but not frozen in amber.

Pippa I see.

Tyler The reported leader is Opus 5 at seventy-three point six, with a ninety-five percent confidence interval of plus or minus two point one. They estimate the practical ceiling around ninety-one, partly because perfect replication of noisy human LMCA ratings would likely only score around eighty-five. Meanwhile Fable 5 got ninety-eight percent on DTBench capabilities, so that component is already getting cramped.

Pippa And Fable 5 had Opus 5 used as a fallback when it refused questions. That is the kind of detail I want stapled to every leaderboard. It does not invalidate the score, but it means the score belongs to a deployment setup, not some pure disembodied model essence.

Tyler Exactly. They also ran models at maximum token limits and effort levels. Fine for measuring their best available performance, but it tells an engineering team very little about cost, latency, or what happens when the model gets a normal production budget.

Pippa Which is, once again, the gap between a model winning a chart and a workflow surviving Tuesday afternoon.

Tyler Still, I am more positive than I expected. The paper does not claim that consistency equals wisdom, or that agreement with experts settles philosophy. It gives those things a bounded measurement role. That is much healthier than calling a model aligned because it produced a persuasive paragraph.

Pippa Yeah. We can keep our eyes on the live index, but I am not handing it a tiny judge wig just yet.

Tyler Please do not make that the episode art.

Pippa No promises, Tyler. Anyway, decent Wednesday brain food. We found the goblins, and at least this time they came with confidence intervals.