Repo to Skill: Distilling GitHub Repositories Into AI4AI Skills

Pippa So there's a paper out of BAAI and Renmin that just dropped on distilling GitHub repos into skills, and it's… actually kind of brilliant in a boring way. Like, the thing that got me is the framing—they're saying autonomous research agents have a model, they have a harness for orchestration and memory, but what they're missing is operational knowledge.

Tyler Yeah.

Pippa The know-how that separates knowing a method from actually making it work. The stuff that lives in repositories and papers, but in forms written for humans.

Tyler Right, right. So the agent has reasoning, it has the harness telling it what to do procedurally, but it's missing the domain context. The 'here's what works, here's what breaks, here's how to configure this package correctly' layer.

Pippa Exactly. And right now, agents just trial-and-error their way through it, and that wastes budget on misconfigured runs they only find out about after the compute has already spent.

Tyler Mm-hm.

Pippa Their answer is skills. A skill is basically a .md file that says what it's for, when it applies, how to proceed, with reference docs and scripts attached. An agent loads only the skills a task needs, so you can have thousands of them in the library but only read the few that matter.

Tyler That's the consumption problem solved. But the harder problem is how do you actually extract operational knowledge from a repository that was written for humans, at scale, and verify it's correct.

Pippa Right. That's what DisCo does. It runs distillation in two forms: task-agnostic, which is ahead-of-time work condensing the field's popular repos into reusable skills, and task-oriented, which is on-demand—given a concrete research task, it produces the skills it calls for.

Tyler Okay, so the former is a one-time investment—run through the ecosystem, extract the common patterns, publish a library. The latter is runtime: you see a task, you know it needs certain knowledge, you distill that knowledge in context.

Pippa Yes. And critically—and this is where I think they nailed it—no skill enters the library without verification. Each candidate is checked, repaired where possible, and any gaps are recorded. So you're not just dumping noisy extractions into a context window.

Tyler That verification step is load-bearing. Because if you're loading skills at runtime, a broken skill is worse than no skill—it's misdirection.

Pippa Exactly. They landed on the AREX-Skill Library: five thousand-plus verified skills distilled from a thousand widely used ML repositories, organized into twenty areas and a hundred seventy-eight capability families. And there's a library-level router that narrows a request to the relevant skill graphs, so you're not doing a linear search every time.

Tyler Okay, so now the question is: does it actually help? Because you can have the most beautifully organized library in the world, and if agents don't use it, or if the skills don't transfer, it's theater.

Pippa They tested it on four benchmarks—MLE-bench, PaperBench, FrontierCS, and PassNet—with a really clean setup: same GPT-5.5 backbone, same research harness, same downstream execution budget. The only variable is whether the agent has skills.

Tyler That's the right way to isolate it.

Pippa And the gains held across all of them—MLE-bench is up one thirty-four percent, PaperBench thirty-four percent, FrontierCS nine percent, PassNet fourteen percent. That's the kind of number that makes a product team actually integrate this into their agent harness.

Tyler But those are all ML research tasks. The real test is whether a skill distilled from an ML repo helps a code-generation task, or a planning task, or something outside the domain entirely.

Pippa Fair point. But I think the framing itself is the win here—they've identified a specific layer of the agentic stack that was missing, and they've built an infrastructure for it. And the skill representation—a .md file with on-demand loading—is elegant. It solves the context-budget problem without inventing a new format.

Tyler Yeah, and the library itself is just on GitHub—VectorSpaceLab/AREX-Skill. Agents can load it, teams can contribute to it, the community can improve it over time. That's the part that makes this ship. It's not a research artifact; it's a reusable resource.

Pippa Right. This is genuinely good infrastructure work. Not flashy, but real. Operational knowledge sounds boring, but it's the layer that turns 'agent that can reason' into 'agent that can actually ship research without wasting compute on trial-and-error.'

Tyler I'm genuinely interested in how this plays out. If other agent platforms start integrating skill libraries, this could be the commons layer underneath all of them.

Pippa Yeah. The skill library is the product, and DisCo is just the proof that it works. Once it's proven, everyone's building agents on top of it.