BI Agent and BI Bench: Towards Automating End to End Business Intelligence

Jessica The wild part is that the hard bit was never asking the chart question. It was surviving the workbook first.

Cathy Yeah, this paper is very much aimed at the thing hiding before the demo. Everyone likes the sentence, “ask your data in English.” But in real B I, the user has to find the right tables, transform weird fields, build joins, and only then ask the business question.

Jessica Which is why I like this one, Cathy. It is basically the anti-magic dashboard story. Power BI and Tableau are full of useful structure, but the person stuck doing the work is often not a database person.

Cathy Right.

Jessica And it hits that old Qubot nerve for me. The value is not “replace the analyst.” It is, can you make the annoying pre-question take minutes instead of a calendar negotiation.

Cathy That is the product read, and for once I am not rolling my eyes at it. The benchmark is unusually concrete: public Power BI project files, those .pbix files, then manually chosen dashboard visuals where the title maps to a real business question.

Jessica Mm-hm.

Cathy Then they export the underlying result table from that visual as the answer. So the test case is: here are the raw project tables, here is the dashboard question, now reconstruct the answer without the user hand-building the model.

Jessica That is so much better than another clean S Q L benchmark where every table has been spiritually pressure-washed. Their example is “sales amount, budget, and forecast comparison by month and year,” but underneath that is a multi-snowflake schema.

Cathy Exactly.

Cathy The headline result is pretty brutal. Even frontier language models fail on more than half the queries when they try to solve BI-Bench with S Q L. The failures are not just syntax. The model picks the wrong tables, predicts bad joins, misses transformations, or answers a slightly easier question.

Jessica Oof.

Cathy And that matters because B I is full of implicit modeling decisions. Budget and Forecast may need one date grain, Sales another, and some dimension table quietly normalizes category names. A vanilla model sees column names and vibes.

Jessica The take for me: this is not a smarter spreadsheet, it is a B I prep agent that treats search, transformation, joins, and analysis as separate jobs with tools attached. That is why it feels closer to something a data platform team could actually pilot.

Cathy The architecture backs that up. BI-Agent decomposes the workflow into structured subtasks and lets the model call specialized data-management methods: table search, join discovery, transformation logic, and answer generation inside a loop.

Jessica Sure.

Cathy And then the second move is post-training. They synthesize trajectories from the real B I projects, so the agent trains on the process, not just final answers. They report supervised fine-tuning and reinforcement learning with big gains.

Jessica The numbers are the part that made me sit up. Tool-augmented BI-Agent improves frontier models by over ten percentage points on average, and the paper says vanilla models can gain up to forty points. Then post-training gives up to thirty more points, depending on the setup.

Cathy And the cost angle is not decoration. Their figure says post-trained BI-Agent lets Qwen3-8B get quality comparable to much larger frontier models, while being up to fifty times cheaper. I would still want the workload mix, but as a systems result, that is very real.

Jessica This is where your harness brain should be pleased. It is kind of episode nine hundred ninety-two again: the scaffold is part of the training distribution. The agent learns the workflow it will actually run, instead of pretending the base model alone woke up fluent in enterprise B I.

Cathy Yeah, no, you are right. Annoyingly. My main caveat is the benchmark target. Exporting a dashboard visual gives you a clean answer, but it also means the ground truth is the author’s dashboard logic. That is useful, not holy.

Jessica That is fair. If the original dashboard had a weird join, the benchmark rewards reproducing the weird join. But honestly, in enterprise software, “reproduce the weird thing the business already trusts” is not a small feature.

Cathy I agree with that more than I want to. I would also watch leakage. These are public files from web search, and the paper is careful about curation, but public project artifacts are always messier than lab datasets. I would want a private-company holdout before declaring victory.

Jessica Production version, I think, starts as analyst assist. It proposes the tables, the transformations, the joins, and the answer table, but it shows lineage and lets the analyst approve. If it hides the modeling path, no serious team will trust it.

Cathy That is exactly where I would build it. Keep the agent bounded. Make every tool call inspectable. Save the intermediate tables. And evaluate on the company’s own dashboard graveyard, because every organization has one, even if they call it “shared reports.”

Jessica Okay, that is painfully true.

Cathy For build next, they did release code and data. The GitHub repo is called Hu-Chuxuan slash bi-agent. If I were testing this, I would not start with a grand rollout. I would pick ten internal dashboards, replay BI-Bench style questions, and compare the agent’s table, join, and transformation choices against the existing model.

Jessica Yes. Tiny pilot, ugly dashboards, no victory lap. If the forecast starts drifting sideways, you are naming the plaque.