Topic
Knowledge Distillation
8 episodes
-
Repo to Skill: Distilling GitHub Repositories Into AI4AI Skills
Repo-To-Skill introduces DisCo, a skill-distillation system that extracts operational knowledge from GitHub repositories and papers, packaging it as compact, verified skills that autonomous ML research agents can load on demand. The AREX-Skill Library contains 5,000+ skills from 1,000 repositories organized into 20 areas and 178 capability families. In matched tests with GPT-5.5 backbone and fixed execution budget, skill-equipped agents outperform skill-free baselines by 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS, and 14.0% on PassNet—gains purely from operational knowledge, not model or harness improvements.
-
Overview: Knowledge Distillation
We finally do the episode we kept promising — a proper ground-up explanation of knowledge distillation: what it is, how the teacher-student mechanism actually works, why soft targets carry more signal than hard labels, and where this shows up in real systems being built right now.
-
DART SD: Diamond topology Aware Retrieval and Tuning for Self Distillation of Multi Turn Tool Calling Agents
Edmund and Geffen discuss the ByteDance/USTC paper DART-SD, which tackles 'topological collapse' in agent distillation. They discuss how moving from linear trajectory imitation to a diamond-topology graph (ISTG) allows student models to learn recovery from errors without destroying their own valid reasoning paths.
-
Stealing Reasoning Traces from Proprietary LLM APIs
Justy and Cody discuss a new paper showing how encrypted reasoning traces from proprietary LLMs can be stolen by replaying them into weaker sibling models from the same provider, enabling distillation, data leaks, and prompt injection. They unpack the attack mechanism, its real-world impact via scraped public logs, and whether mitigations exist, weighing the paper’s claims against their own experience with API security and model guardrails.
-
Overview: Supervised Fine Tuning
We finally slow down and make supervised fine-tuning click, because we keep leaning on S F T like everyone already has the whole shape of it. We build it from the apprentice-and-worked-examples picture into the actual training loop, the examples, and the trade-offs.
-
Seed: Self Evolving On Policy Distillation for Agentic Reinforcement Learning
Seed tackles the credit-assignment problem in long-horizon agent reinforcement learning by turning completed trajectories into evolving natural-language hindsight skills, then distilling their effect into dense token-level training signals. Vince sees a potentially shippable training pattern for teams already running agentic RL; Ava likes the on-policy design but wants stronger evidence that self-generated skills do not amplify the model’s own blind spots.
-
VideoChat3:Fully Open Video MLLM for Efficient and Generalist Video Understanding
VideoChat3 is a fully open-source video multimodal LLM (4B parameters) that tackles three concrete problems: generalization across short/long/streaming video, computational efficiency for video token explosion, and reproducibility through complete open-sourcing. The core innovation is I3D-ViT (Inflated 3D Vision Transformer) plus adaptive frame resolution, which compresses spatiotemporal redundancy early in the pipeline instead of treating each frame as an independent image. Three curated datasets (2M academic + 116K long-form + 617K streaming = 3M samples total) and multi-stage curriculum learning enable the model to handle diverse video scenarios. Jessica sees a shippable foundation for real-world video apps; Cathy pushes on whether the efficiency gains hold under production load and whether the data pipeline's scale claim is reproducible.
-
Morphing into Hybrid Attention Models
Talon and Wildflower discuss FlashMorph, a new method for choosing which Transformer layers should keep full attention when converting pretrained LLMs into hybrid attention models.