Topic
Calibration
13 episodes
-
Introducing LangSmith Tuned Evaluators
Masonry and Eyre discuss LangSmith Tuned Evaluators, starting with Perceived Error, as a shipped product for attaching managed quality signals to production agent traces.
-
Introducing the Conceptual Reasoning Index
Pippa and Tyler dig into the Conceptual Reasoning Index, a new three-benchmark evaluation suite for argument judgment, logical consistency, and decision-theoretic reasoning. They like the attempt to measure work where answers cannot simply be checked, while questioning whether a changing aggregate score can earn trust as a durable comparison tool.
-
Overview: Construct validity
We slow down and make construct validity click: the gap between the label on a test and what the test actually measures. We connect it to benchmarks, hiring screens, model validation, and the Cursor reward-hacking story we keep circling back to.
-
Overview: Bayes' Theorem
We finally slow down on Bayes' Theorem, the belief-updating rule we keep smuggling into conversations about evals, spam filters, diagnosis, ranking, and calibration. We make it click through one package-sorting picture: evidence only matters against the pile it came from.
-
Grading the Narrators: An Isnād–Rijāl Framework for Claim Level Provenance in Multi Agent Knowledge Systems
Ali Zahid Raja's 'Grading the Narrators' adapts isnād–rijāl methodology from classical Islamic hadith science to solve a concrete problem in modern multi-agent knowledge systems: how to trust claims that have passed through chains of autonomous transformations. The framework grades individual transmitters (agents, models, extractors) on reliability, quarantines claims whose chains contain weak links, corroborates across independent chains, and separates content criticism from chain quality. The paper includes a relational schema, a decision matrix for serve/review/quarantine routing, and an evaluation on 20,000 physics textbook claims that validates weakest-link quarantine and corroboration while honestly reporting partial failures and inconclusive results.
-
AREX: Towards a Recursively Self Improving Agent for Deep Research
Pippa and Tyler dig into AREX, a recursively self-improving deep research agent from BAAI that alternates between an inner search loop and an outer constraint-verification loop — and discuss whether that architecture is genuinely novel or a smarter repackaging of ideas the field already had.
-
Overview: Calibration
We finally slow down and make calibration click: what it means for a model’s confidence to match reality, how you measure that, and why it matters when you actually want to trust the thing. We keep it grounded in the systems we’ve been circling for ages, because calibration is everywhere once you start looking.
-
Introducing TabFM: A zero Shot foundation model for tabular data
Justy and Cody examine TabFM, Google Research’s zero-shot foundation model for tabular classification and regression. They unpack its hybrid row-column attention design, synthetic-data training, TabArena evidence, the trade-off between out-of-the-box convenience and tuned ensembles, and whether BigQuery integration could make this genuinely useful in everyday data workflows.
-
Thinking Machines open sources first multimodal language model, Inkling, focused on low cost and 'resistance to censorship'
Inkling, Thinking Machines' open-source multimodal MoE model (975B total / 41B active parameters), lands as a broad, balanced generalist with a standout feature: a controllable 'thinking effort' knob to dial cost vs. performance from 0.2 to 0.99. Enterprises get native text+image+audio fusion, Apache 2.0 weights, and a lighter Inkling-Small preview, but benchmarks show it trails specialized open and closed models on coding and pure reasoning, while remaining competitive on multimodality and agentic workflows. The episode debates whether the real win is the runtime control surface (Tinker platform) and a cautious, non-censoring epistemics posture — not the headline parameters.
-
LLM Evaluation Frameworks Compared: How to Actually Measure What Your Model Does MachineLearningMastery
Fern and Lintel dig into a comparison of RAGAS, DeepEval, and Promptfoo, landing on the article’s real argument: eval frameworks are less about novel metrics than about where evaluation fits in your workflow, and the dangerous part is trusting LLM-as-a-judge without auditing its biases. They like the article’s practical split between RAG scoring, CI gates, and prompt red-teaming, but push on where the examples are a little too toy-like and where teams can overread framework choice as the main problem instead of test-set design and human calibration.
-
Large language models often prioritize Western moral values, overlooking other cultures
A research paper finds LLMs tend to mirror Western moral priorities when asked to roleplay citizens of 48 countries, and two hosts discuss what this actually means for users, products, and culture.
-
Overview: Reinforcement Learning from Human Feedback
We finally define reinforcement learning from human feedback the way we keep using it: as a loop where human preferences become a learned reward signal that steers a model after its initial training. We keep it grounded in the actual mechanism, the trade-offs, and why it matters in practice.
-
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
Jessica and Cathy dig into RL with Metacognitive Feedback (RLMF): a post-training loop that rewards models not just for correct answers, but for accurately judging how well they did—improving both task performance and the faithfulness of uncertainty expressions. They explain the mechanism (metacognitive data selection and metacognitive advantage scaling), discuss trade-offs, and debate whether this is still research-only or actually shippable.