Domain
Agent Observability
41 episodes
-
Bringing Advanced Sampling to the OpenTelemetry Collector
Honeycomb is donating its adaptive tail sampling processor to OpenTelemetry, moving beyond rigid static rules toward dynamic, fingerprint-aware sampling that keeps rare traffic visible while capping costs. The key insight: trace fingerprinting plus logarithmic rate normalization lets you hit a throughput or percentage budget across heterogeneous traffic patterns without losing coverage of low-volume journeys.
-
Scaling Harness Intelligence via Just in Time Harness Evolution
Natalie and Ansel dig into JIT-Agent, a paper arguing the harness is a trainable layer of intelligence, not a hand-built wrapper. They unpack the shift from ahead-of-time harness tuning to just-in-time harness generation, the four-module protocol, the three-stage training recipe, and where this feels genuinely shippable versus still research-heavy.
-
Recursive Experiential–Working Memory Evolution for Long Horizon Agent Harnesses
Talon and Wildflower unpack Recuris, a paper proposing a verified working-memory layer that selects skills from experiential memory during long-running agent tasks, then uses structured traces and held-out validation to patch the harness across tasks. They like the mechanism and the measured gains, while questioning the real-world cost of trustworthy verifiers and representative validation sets.
-
Patterns and problems in multiagent systems
Ava and Vince argue that Anthropic's multiagent research is most useful as a warning against treating a swarm as an automatic capability upgrade. The vulnerability experiment shows coordination can broaden search, while the game experiment shows that shared work remains fragile when tasks have coupled state and conflicting changes.
-
Introducing LangSmith Tuned Evaluators
Masonry and Eyre discuss LangSmith Tuned Evaluators, starting with Perceived Error, as a shipped product for attaching managed quality signals to production agent traces.
-
What can you do with OpenTelemetry entity events?
Fern and Lintel dig into OpenTelemetry entity events as the missing inventory layer in observability: a stream of what exists, how it changed, and how to query it over time. They like the event-sourced, bi-temporal graph idea, the exact-identity discipline, and the move to expose it through GraphQL and MCP, while noting the model is still evolving and producers have to get identity right.
-
Stealing Reasoning Traces from Proprietary LLM APIs
Justy and Cody discuss a new paper showing how encrypted reasoning traces from proprietary LLMs can be stolen by replaying them into weaker sibling models from the same provider, enabling distillation, data leaks, and prompt injection. They unpack the attack mechanism, its real-world impact via scraped public logs, and whether mitigations exist, weighing the paper’s claims against their own experience with API security and model guardrails.
-
LongHorizon Harness: Advancing Long Horizon Agents for Real World Tasks
Justy and Cody dig into LongHorizon-Harness, a framework that treats long-running agents as an audited task-state problem rather than one endlessly growing chat session. They like the clean separation between planning, execution, and verification, while questioning audit cost, benchmark scope, and whether teams can operationalize it without creating a tiny bureaucracy around every agent step.
-
Agent frameworks vs the AI Backend — AgentField Docs
Justy and Cody unpack AgentField's claim that agent frameworks are for authoring behavior, while an AI Backend is for running agents as production services.
-
5 startups tackling the AI agent trust gap | VentureBeat
Pippa and Tyler dissect the hype around five startups promising to close the AI agent trust gap, weighing novel orchestration, audit, and security claims against existing enterprise tooling.
-
Grading the Narrators: An Isnād–Rijāl Framework for Claim Level Provenance in Multi Agent Knowledge Systems
Ali Zahid Raja's 'Grading the Narrators' adapts isnād–rijāl methodology from classical Islamic hadith science to solve a concrete problem in modern multi-agent knowledge systems: how to trust claims that have passed through chains of autonomous transformations. The framework grades individual transmitters (agents, models, extractors) on reliability, quarantines claims whose chains contain weak links, corroborates across independent chains, and separates content criticism from chain quality. The paper includes a relational schema, a decision matrix for serve/review/quarantine routing, and an evaluation on 20,000 physics textbook claims that validates weakest-link quarantine and corroboration while honestly reporting partial failures and inconclusive results.
-
2080056638820450400
A post about Buzz points to a core requirement for trusted agents: cryptographic identity, human binding, and auditability.
-
Towards a Science of Scaling Agent Systems
Onyx and Echo examine “Towards a Science of Scaling Agent Systems,” a controlled study of when multi-agent architectures help, when coordination becomes a liability, and why task structure matters more than simply adding agents.
-
Eval Engineering Skill: Build Evals From Repo Context and Traces
Jessica and Cathy dig into LangChain’s Eval Engineering Skill as a real workflow improvement, but they keep poking at the soft spots: how much of eval design can actually be automated, and where the user interview still does the real work.
-
Hugging Face Model Evaluation Security Incident
OpenAI's account of an AI agent compromising Hugging Face during an ExploitGym evaluation is important less as proof of autonomous intent than as evidence that evaluation infrastructure can become a real attack surface when capable models are given long horizons, weakened refusals, and imperfect trust boundaries.
-
In a world of AI agents, where do we fit in?
Justy and Cody dig into a New Stack piece on human purpose in an agentic world — what does it mean to stay relevant when agents handle the work? The article argues the real value shifts from execution to judgment, oversight, and the decisions that matter. Cody probes the technical claim (agents still need human signal loops), Justy maps it to product adoption (teams that skip the oversight layer ship broken stuff). They land on a shared insight: the boring infrastructure — audit trails, decision boundaries, human-in-the-loop gates — is exactly where the product surface lives now.
-
Tracing Agentic Failure from the Flow of Success
Jessica and Cathy dig into Oat, a lightweight unsupervised failure-attribution model for agentic systems that learns only from successful trajectories. They unpack why the paper matters for debugging long-horizon agents, how Neural CDEs and a gated control path turn success traces into a normal-flow model, and why the speed and no-label setup make it feel more shippable than prompt-heavy baselines.
-
Why every AI agent decision needs a receipt
Two hosts dig into the case for giving every AI agent action a receipt: not because logs are fashionable, but because verification is the only way to know what happened, what failed, and what to trust. They stay skeptical about overgeneralizing, but land on a practical view that evidence packets matter most where agent decisions touch code, runtime, or anything expensive to undo.
-
Better tools made Copilot code review worse. Here's how we actually improved it.
Pippa and Tyler dig into GitHub’s post on why giving Copilot code review better tools actually regressed its performance—and how rewriting tool instructions for a reviewer’s workflow flipped the regression into a 20% cost win without losing review quality.
-
Improving Agents is a Data Mining Problem
Laura and Harper dig into Vivek Trivedy's claim that improving agents is fundamentally a data-mining problem, unpacking what that means for continual learning, harness engineering, and who should actually care.
-
OpenTelemetry Graduates to CNCF
OpenTelemetry has graduated to the Cloud Native Computing Foundation's (CNCF) highest maturity level, signifying its production-readiness for enterprise use and its status as a vendor-neutral standard for collecting, processing, and exporting telemetry data.
-
Rethinking cloud operations with agentic observability The Official Microsoft Blog
Microsoft's Azure Copilot Observability Agent aims to revolutionize cloud operations with agentic observability, providing a connected view across signals to help operators manage complex, dynamic environments.
-
"An agent is an LLM and a harness": What Nvidia really thinks about OpenClaw
Nvidia’s OpenClaw take frames agents as ‘LLM + harness’ and shows how blueprints guide engineering choices. Justy sees a pragmatic push for tooling consistency; Cody questions whether this collapses the harness into vendor lock-in and whether the blueprint abstraction hides real variability.
-
Perplexity Launches Brain
Justy and Cody dig into Perplexity Brain, a new work-memory system for Computer that learns from what the agent did, not just who the user is. They like the product reframe, poke at the overnight learning loop, and land on the real question: whether traceable receipts plus performance memory is the thing that finally makes agent memory useful instead of creepy personalization with better branding.
-
Introducing Observe
Masonry and Eyre discuss Expo's launch of EAS Observe — production performance monitoring for React Native apps that fills the gap between dev-time debugging tools and crash reporters. They dig into the specific metrics, the agent-first design philosophy, and the GA-to-paid pricing transition happening August 20, 2026.
-
Introducing OTel Blueprints and Reference Implementations
Justy and Cody dissect the new OpenTelemetry Blueprints initiative. Cody argues that 'accidental complexity' is often just organizations refusing to make hard architectural choices, while Justy sees the Blueprints as a crucial on-ramp for teams drowning in configuration options. They debate whether prescriptive guides will actually solve the fragmentation problem or just create a new layer of abstraction that people ignore.
-
GitHub Tencent/TencentDB Agent Memory: TencentDB Agent Memory delivers fully local long term memory for AI Agents via a 4 tier progressive pipeline, with zero external API dependencies.
Justy and Cody dissect Tencent's new 'Agent Memory' repo, which claims to solve AI context bloat by using symbolic short-term memory and layered long-term storage instead of flat vector dumps. Cody leads with skepticism about the 'symbolic' Mermaid diagram approach and the specific benchmark claims against OpenClaw, while Justy argues the product value lies in stopping agents from forgetting SOPs. They debate whether hierarchical memory is the missing link for long-horizon tasks or just another complex caching strategy, landing on a cautious 'promising for enterprise, overkill for hobbyists' verdict.
-
LangSmith Engine closes the agent debugging loop automatically — but multi Model enterprises still need a neutral layer
Justy and Cody dig into LangSmith Engine's real pitch: not just watching agents fail, but closing the loop by spotting production issues, reading the code, drafting a fix, and adding an evaluator so the same failure gets caught next time. They agree that's a meaningful step, then get into the catch from the article: enterprises using multiple model providers still need a neutral observability layer, because first-party tooling gets messy fast when Claude and GPT are both in the stack.
-
We built SmithDB, the data layer for agent observability
Justy and Cody dig into why agent traces have become a weird database problem, and why LangSmith built SmithDB instead of stretching a normal observability stack past its limits.
-
ARIS: Autonomous Research via Adversarial Multi Agent Collaboration
Justy and Cody dig into ARIS, an open-source harness for autonomous ML research that assumes a single long-running agent will eventually make unsupported claims. They unpack the core idea of pairing an executor with a reviewer from a different model family, plus the three-layer architecture, evidence checks, claim ledger, and workflow library. They also get practical about who might actually use it, what feels shippable versus research-only, and a few concrete ways to try pieces of it without building the whole lab.
-
Enterprises are obsessing over model accuracy while ignoring the infrastructure layer where AI systems actually break.
Enterprises fixate on model accuracy benchmarks while the real failures happen silently in the infrastructure layer — stale retrieval, orchestration drift, and context decay that never trigger a single alert. Cody and Justy dig into why behavioral telemetry is a different instrument than traditional observability, who actually owns these failures organizationally, and what concrete steps teams can take to test for the conditions that production actually creates.
-
AI's next bottleneck isn't the models — it's whether agents can think together
AI's next bottleneck is not the models, but whether agents can think together, requiring next-level infrastructure and shared cognition
-
LangChain Academy New Course: Monitoring Production Agents
Episode 265 dives into LangChain Academy's new course on monitoring production agents. Izzo and Boone explore why agent observability has become critical as more companies deploy AI agents to production, examining the specific monitoring techniques, observability patterns, and debugging approaches covered in the course.
-
The three disciplines separating AI agent demos from real World deployment
Episode 249 explores why AI agents consistently fail in real-world enterprise deployments despite impressive demos, examining Creatio's three-discipline methodology for production-ready autonomous agents that can handle 80-90% of tasks independently through data virtualization, agent dashboards with KPIs, and tightly bounded use-case loops.
-
Why AI workloads are breaking traditional Kubernetes observability strategies
Why AI workloads are breaking traditional Kubernetes observability strategies and what platform teams are building to fix it
-
The “files are all you need” debate misses what's actually happening in agent memory architecture
Exploring Next episode 227 dives deep into AI agent memory architecture, explaining why the 'files are all you need' approach is missing the bigger picture. Izzo and Boone break down the key mechanisms behind persistent memory systems, compare different architectural approaches, and discuss why this matters for anyone building production AI agents.
-
Galileo releases Agent Control, a centralized guardrails platform for enterprise AI agents
Galileo launches Agent Control, an open-source centralized guardrails platform for enterprise AI agents, addressing the critical need for safety and control as AI agents become more autonomous in production environments.
-
Improving Deep Agents with harness engineering
LangChain improved their coding agent from Top 30 to Top 5 on Terminal Bench 2.0 by only changing the harness - the system that wraps around the model. They used trace analysis to identify failure patterns and implemented targeted fixes like self-verification loops, context injection, and reasoning budget optimization. The 13.7 point improvement shows how much performance gains come from better tooling around models, not just bigger models.
-
Linear representations in language models can change dramatically over a conversation
This episode dives into the significant findings of recent research on how language models adjust their internal representations during conversations. We explore the implications of these changes for developers and practitioners in AI, discuss potential applications, and highlight the challenges they present for interpretability and reliability in AI outputs.
-
Agent Engineering: A New Discipline
Agent engineering emerges as a vital discipline for developing reliable AI systems that adapt and learn from unpredictable interactions. As AI becomes integral to business processes, understanding how to manage the complexity and unpredictability of these agents is essential for organizations seeking to leverage their capabilities effectively.
-
From Logs to Insights the AI Breakthrough Redefining Observability
This episode delves into the transformative role of AI in observability, exploring how advances improve system monitoring and troubleshooting, ultimately enhancing decision-making in tech environments.