OmniVideo 100K: A Dataset for Audio Visual Reasoning through Structured Scripts and Evidence Chains

Justy The part that got me was they basically said, no, the model is not bad at video for one reason. The data pipeline is weirdly sabotaging it.

Cody Yeah. And honestly that tracks. If the prep pipeline keeps separating sound from what caused it, the model is being trained on a broken story.

Justy Which is such an Exploring Next problem. We invent a giant intelligence benchmark, then discover the spreadsheet before it was cursed.

Cody This one is OmniVideo one hundred K. The problem they're solving is audio-visual question answering that actually needs time, causality, and cross-modal links, not just spotting a thing in one clip and parroting a caption.

Justy Right, right.

Justy And the people stuck on it are basically anyone building or tuning omni models for video. Research groups, sure, but also product teams that want a model to answer what happened, who said it, what sound matched what action … all the annoying real stuff.

Cody Before we get too deep, continuity update. I made coffee, forgot it, made more coffee, and then found the first cup cold on the desk. Again. Continuity has not improved operations.

Justy I feel seen. Last night I spent fifteen minutes looking for my charger and it was in my bag the whole time. We're gaining episodic memory and losing basic object permanence.

Justy Anyway, this paper's core complaint is fair. The usual setup is video to captions to Q A, with lots of short clips and separate audio and visual descriptions. That sounds tidy, Cody, but it kind of destroys the thing the model actually needs.

Cody Exactly.

Cody Their two-part fix is pretty clear. First is entity-anchored video scripting. They turn a video into a structured script with a summary, a main entity list, and timestamped segments that include visual description plus audio information like speaker-labeled speech and non-speech sounds.

Justy Mm-hm.

Cody The main entity list is the clever bit. They identify the active people, animals, or objects up front and give each one a stable descriptive identifier, like a named handle but based on appearance. That becomes a global prior so segment captions stop renaming the same person every fifteen seconds.

Justy That feels extremely product-relevant to me. A lot of failure in multimodal apps is not that the model saw nothing. It's that it saw five slightly incompatible versions of the same scene and then had to improvise.

Cody Yeah.

Justy Also the speaker labeling matters. If speech is attached to the visible entity instead of floating in a separate transcript blob, now a question can ask who said something while doing something, or whether a sound came from the person on screen versus off screen.

Cody And then the second stage is clue-guided Q A generation. Instead of dumping the whole giant script into one prompt and saying please make good questions, they have the model mine clues first. Those clues can span multiple segments and modalities, so you get an evidence chain before the Q A is written.

Justy Oh interesting.

Cody That separation matters because long-text comprehension and Q A synthesis are different jobs. If you fuse them, models tend to grab whatever local detail is easiest. Their clue step is basically forcing a retrieval and reasoning pass before generation.

Justy So in plain English, they're making the synthetic training examples show their work a little. Not fully formal, but enough that the question is anchored to actual evidence instead of vibes.

Cody That's my read too. And the dataset is pretty big. One hundred thousand automatically generated audio-visual Q A pairs for training, plus OmniVideo-Test with five hundred and five human-verified samples across ten tasks.

Justy The gains are not subtle either. Fine-tuning VITA one point five, Qwen two point five Omni seven B, and Qwen three Omni thirty B gave up to twenty point five nine percent on their test set, and up to twelve point six four percent on things like Daily-Omni and JointAVBench.

Cody Sure.

Cody I buy that directionally. Especially because the paper's diagnosis is plausible. Open models are decent at basic audio-visual association, but weaker on fine-grained temporal alignment and cross-modal reasoning. This pipeline is aimed right at that gap.

Justy Do you have real objections, or is this one of your rare cheerful Wednesdays?

Cody I do have one caution, Justy. It's still an automated data engine built with models prompting models. So I'd want to inspect how brittle those entity identifiers are, whether clue mining introduces prompt artifacts, and how often the evidence chain is actually faithful versus tidy-looking.

Justy That's fair. But this feels less like a pure benchmark ornament and more like something a team could use. Not as a consumer feature by itself, more like infrastructure for anyone fine-tuning an omni model for support, education, media archives, surveillance-lite— no, bad word choice — video ops, basically.

Cody Right. Research-adjacent but shippable in the sense that the representation is sane. Summary, entities, timestamped multimodal segments, then clue extraction. I might even steal that pattern for evaluation data before training data.

Justy Also I appreciate that they put the code and data up. GitHub slash MiG-N J U slash OmniVideo dash one hundred K. That makes this way more than a pretty PDF.

Cody And it gives people a way to test whether the script format itself is the win, versus the exact prompting recipe. Which is the part I'd want to ablate first.

Justy We really are who we always were. I see a product layer. You see an ablation table with trust issues.

Justy Okay, episode four ninety-two can go live and you can go reheat one of your two coffees. That's a very dignified Monday, Cody.