Topic
In Context Learning
18 episodes
-
Repo to Skill: Distilling GitHub Repositories Into AI4AI Skills
Repo-To-Skill introduces DisCo, a skill-distillation system that extracts operational knowledge from GitHub repositories and papers, packaging it as compact, verified skills that autonomous ML research agents can load on demand. The AREX-Skill Library contains 5,000+ skills from 1,000 repositories organized into 20 areas and 178 capability families. In matched tests with GPT-5.5 backbone and fixed execution budget, skill-equipped agents outperform skill-free baselines by 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS, and 14.0% on PassNet—gains purely from operational knowledge, not model or harness improvements.
-
S3Gym: Can LLMs Turn Self Testing and Self Judging into Self Improvement?
S3Gym is a new interactive benchmark that tests whether LLMs can actually improve themselves by testing their own behavior, judging the results, and learning from them. The paper evaluates three ways to incorporate experience—keeping full conversation history, compressing it into summaries, and training on it—across seven text-based games. The finding: self-improvement isn't automatic. What works depends entirely on the task. Sometimes summaries help, sometimes raw history is better, and parameter training can backfire badly. The real bottleneck isn't recognizing success—it's turning that recognition into a policy the model can actually reuse.
-
How our agents build on Brand pages with design
Vercel's design.md post describes a three-part system for keeping AI-generated pages on-brand outside a codebase: a public guidance file, a public stylesheet that takes layout decisions away from the model entirely, and an eval loop that converts human corrections into rules and deterministic checks. Pippa and Tyler dig into what actually makes it work, why the naive port-the-prompt approach failed, and what it means that Vercel had to build an eval harness just to ship a markdown file.
-
Overview: Next State Prediction
We finally slow down and explain next-state prediction from the ground up — the deceptively simple idea that if you train a model to guess what comes next, it ends up learning how the world actually works, and why that one trick is underneath almost everything in modern AI.
-
MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
MemTrapBench identifies a counterintuitive failure mode in LLM memory systems: faithfully extracted, semantically relevant memories can actually degrade model performance by anchoring reasoning to prior patterns (Reasoning Fixation) or overriding correct beliefs with historical premises (Belief Distortion). The paper benchmarks this across five memory frameworks and proposes AdaptiveMem, a prompt-based intervention that mitigates traps without modifying underlying architecture.
-
Overview: Prompt Injection
We finally slow down on prompt injection, the security problem we keep name-dropping and then skating past. We build it from the simple mental model of a desk assistant who cannot tell the boss's instruction apart from a sneaky note inside the paperwork.
-
The harness is all you need (mostly)
Burke Holland argues that productivity gains with AI come not from exotic prompts or new tools, but from deeply understanding and using the harness well. He walks through a concrete six-step workflow (pick a tool, enable autonomy, prototype, plan, implement with Autopilot, iterate) that leverages GitHub Copilot's built-in orchestration and subagent routing without requiring custom skills or tricks. The central claim is that the harness — the core interaction model — is what matters; everything else is noise.
-
The new rules of context engineering for Claude 5 generation models | Claude by Anthropic
Anthropic's post on context engineering for Claude 5 models reveals a surprising finding: they removed over 80% of Claude Code's system prompt with no measurable loss in performance. The core insight is that newer models need fewer explicit constraints and benefit more from clean interfaces, progressive disclosure, and letting the model use judgment rather than following hard rules. The shift reflects a broader pattern: as models get stronger, the infrastructure around them gets simpler.
-
Introducing TabFM: A zero Shot foundation model for tabular data
Justy and Cody examine TabFM, Google Research’s zero-shot foundation model for tabular classification and regression. They unpack its hybrid row-column attention design, synthetic-data training, TabArena evidence, the trade-off between out-of-the-box convenience and tuned ensembles, and whether BigQuery integration could make this genuinely useful in everyday data workflows.
-
Overview: Supervised Fine Tuning
We finally slow down and make supervised fine-tuning click, because we keep leaning on S F T like everyone already has the whole shape of it. We build it from the apprentice-and-worked-examples picture into the actual training loop, the examples, and the trade-offs.
-
EvolvingWorld: An Open Schema Framework for Co Evolving Role Play Agents and World Model in Interactive Literary World
Masonry and Eyre dig into EvolvingWorld, a new framework that lets fictional characters and their world co-evolve across long stories. Eyre walks through the open-schema architecture and seven supervised tasks; Masonry sizes up who would actually build with this and where product pain lives. The hosts end up excited about the open-schema premise but skeptical of the benchmark’s generality.
-
Overview: Task Decomposition
We finally slow down on task decomposition, the quiet trick underneath agents, code review workflows, web tasks, and a lot of the stuff we keep arguing about. We use one mental model, a messy project board becoming manageable tickets, and build from intuition to mechanism to where it still matters now.
-
Overview: In Context Learning
We finally slow down and explain in-context learning, the thing we keep leaning on whenever prompts, agents, examples, and adaptation come up. We make the core idea concrete: the model is learning from the temporary packet you hand it, without changing itself permanently.
-
How to Implement a Unified Memory From Scratch
Jessica and Cathy dig into a new post that walks through building a unified agent memory from scratch using knowledge graphs and MongoDB, unpacking what it actually takes to wire memory into a real agent harness. They tease apart where the post’s blueprint shines, where it overreaches, and who on earth should actually roll their own instead of reaching for an off-the-shelf tool.
-
Overview: Prompt Engineering
We’re finally doing the overdue deep dive on prompt engineering, the weirdly practical skill of getting language models to do the thing you actually meant. We keep coming back to it because the difference between a flimsy prompt and a good one is often the difference between nonsense and a usable product.
-
Large language models often prioritize Western moral values, overlooking other cultures
A research paper finds LLMs tend to mirror Western moral priorities when asked to roleplay citizens of 48 countries, and two hosts discuss what this actually means for users, products, and culture.
-
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Slack's Sergii Gorbachov ran 200+ agentic E2E tests to measure where agent-driven testing fits alongside traditional deterministic tests. Core finding: agents verify goals (adaptable paths to the same outcome), while traditional tests enforce journeys (single deterministic sequence). MCP-based agents were most reliable (0% on simple flows, ~12% on complex); generated tests were fastest (~3 min) but fragile on complexity (~48% failure rate on harder flows); cost was the real constraint ($15–30 per run). The insight is not replacement—it's complementary layers. Agents excel at exploratory validation and catching UI state variability; deterministic tests handle regression and CI speed.
-
Overview: Tool use and function calling
We finally sit down and make tool use and function calling click from the ground up. We keep coming back to the same idea: a model can draft the request, but something outside it has to actually do the thing.