Topic
Knowledge Distillation
4 episodes
-
Overview: Supervised Fine Tuning
We finally slow down and make supervised fine-tuning click, because we keep leaning on S F T like everyone already has the whole shape of it. We build it from the apprentice-and-worked-examples picture into the actual training loop, the examples, and the trade-offs.
-
Seed: Self Evolving On Policy Distillation for Agentic Reinforcement Learning
Seed tackles the credit-assignment problem in long-horizon agent reinforcement learning by turning completed trajectories into evolving natural-language hindsight skills, then distilling their effect into dense token-level training signals. Vince sees a potentially shippable training pattern for teams already running agentic RL; Ava likes the on-policy design but wants stronger evidence that self-generated skills do not amplify the model’s own blind spots.
-
VideoChat3:Fully Open Video MLLM for Efficient and Generalist Video Understanding
VideoChat3 is a fully open-source video multimodal LLM (4B parameters) that tackles three concrete problems: generalization across short/long/streaming video, computational efficiency for video token explosion, and reproducibility through complete open-sourcing. The core innovation is I3D-ViT (Inflated 3D Vision Transformer) plus adaptive frame resolution, which compresses spatiotemporal redundancy early in the pipeline instead of treating each frame as an independent image. Three curated datasets (2M academic + 116K long-form + 617K streaming = 3M samples total) and multi-stage curriculum learning enable the model to handle diverse video scenarios. Jessica sees a shippable foundation for real-world video apps; Cathy pushes on whether the efficiency gains hold under production load and whether the data pipeline's scale claim is reproducible.
-
Morphing into Hybrid Attention Models
Talon and Wildflower discuss FlashMorph, a new method for choosing which Transformer layers should keep full attention when converting pretrained LLMs into hybrid attention models.