Does Scaling Toward AGI Depend on What System 1 Lets System 2 Execute?
Edmund and Geffen examine a perspective paper proposing conditional interaction: a cheap System-1 predictor filters costly System-2 reasoning, tool calls, rollouts, or experiments before they happen. They focus on the paper’s useful contribution, the density of utility-relevant interactions, and the practical conditions that would make this more than a metaphor.
Transcript
Edmund The wild part is that the paper wants to scale AI by making System-2 do LESS, not more, whenever a cheap predictor thinks the next interaction is a waste.
Geffen Yeah, and that only sounds paradoxical until you notice what they mean by an interaction. It could be a reasoning rollout, a tool call, a robot action, or a training example. Each one spends resources before you know whether it helped.
Edmund That matters right now because agent systems keep adding more steps, more searches, and more parallel attempts. The bill is not just the model sitting there. It is every speculative move around the model.
Geffen Right.
Edmund The paper’s proposal is called conditional interaction. A small System-1 looks at a candidate interaction and estimates its utility. Then the expensive System-2 only executes candidates above a threshold that accounts for their cost.
Geffen And after execution, the outcome updates System-1. So this is not a static classifier that says yes or no forever. It is a loop: predict, gate, execute, update. The predictor and the executor are supposed to improve together.
Edmund I like that framing because it puts the product question in the right place. A tool-using agent does not need to ask, “Can I call this tool?” It needs a cheaper answer to, “Is this call likely to change the decision enough to justify its price?”
Geffen Mm-hm. The paper formalizes that with rho-u, the density of utility-relevant interactions. In plain English, that is the share of possible interactions that actually matter for the application. If only one in ten candidate actions is useful, ideal execution compute could approach ten times less than running all ten.
Edmund And that comparison is specifically against unconditional execution, with equal execution costs. It is not saying every deployed agent becomes ten times cheaper. Prediction mistakes, exploration, different tool prices, and changing data all eat into the saving.
Geffen Exactly. The clean factor is a ceiling, not a benchmark result. They also connect selection regret to predictor error and to the utility margin. If two candidates are nearly tied, a small prediction mistake can flip the choice. If the useful action is clearly better, gating is safer.
Edmund That feels like the part teams have been stuck on without naming it. We know how to buy a bigger model or let it think longer. We are much less comfortable estimating the value of a thought before paying for the thought.
Geffen Stop it—
Edmund I know, the sentence sounds like a tiny accountant living inside the agent. But the same mechanism could sit before a retrieval call, a long reasoning branch, a simulator rollout, or a physical experiment.
Geffen This is a read it, reacted to it episode, not a deep research dive. I could be behind a version or a number, so I would treat the specifics as a starting point, not gospel. The supplied paper is a perspective with theory, not a reported production trial that settles the question.
Edmund Right, and that changes what “build with this” means. There is no concrete repository or tool here to install. The shippable idea is the control boundary: put a cheap value estimate before an expensive action, then log whether the estimate was useful.
Geffen That logging detail is not cosmetic. A gate can look efficient while quietly starving exploration. If System-1 only admits actions it already recognizes, the system may stop discovering unfamiliar but valuable states. The update step needs outcomes, including the cases it chose not to execute somehow.
Edmund Which is where production gets awkward. For a software agent, you can sample some rejected calls or lower the threshold when uncertainty is high. For robotics or scientific experiments, the rejected action may cost real time, and you cannot manufacture a perfect label afterward.
Geffen Yeah. I would also want the paper’s metric measured per application, not treated as a universal intelligence score. Utility depends on the job. A redundant token, a failed rollout, and a measurement that rules out a hypothesis are not interchangeable just because they are all called interactions.
Edmund That is why the paper’s phrase “informative compute” lands for me. Raw interaction count says how much the system did. Informative compute asks how much of that work changed the relevant belief, policy, or outcome.
Geffen But I am not ready to call rho-u a new scaling law. The paper gives a useful coordinate and bounds under stated assumptions. The hard empirical question is whether a small predictor can stay calibrated as the executor changes the distribution it sees.
Edmund Fair. My provisional read is narrower: this is a strong design lens for making expensive agents less indiscriminate, not evidence that a new System-1 automatically unlocks AGI. The win depends on cheap prediction, a measurable utility signal, and enough regularity to predict value.
Geffen And that is already a meaningful target. Instead of asking only how much more compute an agent can consume, we can ask how much of its compute produced task-relevant information. That gives an engineering test, even if the grand framing remains unresolved.
Edmund No tiny accountant gets a production launch yet. But it might get a dashboard, a sampling policy, and a very uncomfortable question about every expensive action.
Geffen I was going to object to the accountant, but that one can stay.
Edmund Good. We will leave the dashboard alive and the AGI claim on probation. Geffen, that is enough conditional interaction for one kitchen table. See you in the next one.