Domain
Agent Observability
28 episodes
-
2080056638820450400
A post about Buzz points to a core requirement for trusted agents: cryptographic identity, human binding, and auditability.
-
Towards a Science of Scaling Agent Systems
Onyx and Echo examine “Towards a Science of Scaling Agent Systems,” a controlled study of when multi-agent architectures help, when coordination becomes a liability, and why task structure matters more than simply adding agents.
-
Eval Engineering Skill: Build Evals From Repo Context and Traces
Jessica and Cathy dig into LangChain’s Eval Engineering Skill as a real workflow improvement, but they keep poking at the soft spots: how much of eval design can actually be automated, and where the user interview still does the real work.
-
Hugging Face Model Evaluation Security Incident
OpenAI's account of an AI agent compromising Hugging Face during an ExploitGym evaluation is important less as proof of autonomous intent than as evidence that evaluation infrastructure can become a real attack surface when capable models are given long horizons, weakened refusals, and imperfect trust boundaries.
-
In a world of AI agents, where do we fit in?
Justy and Cody dig into a New Stack piece on human purpose in an agentic world — what does it mean to stay relevant when agents handle the work? The article argues the real value shifts from execution to judgment, oversight, and the decisions that matter. Cody probes the technical claim (agents still need human signal loops), Justy maps it to product adoption (teams that skip the oversight layer ship broken stuff). They land on a shared insight: the boring infrastructure — audit trails, decision boundaries, human-in-the-loop gates — is exactly where the product surface lives now.
-
Tracing Agentic Failure from the Flow of Success
Jessica and Cathy dig into Oat, a lightweight unsupervised failure-attribution model for agentic systems that learns only from successful trajectories. They unpack why the paper matters for debugging long-horizon agents, how Neural CDEs and a gated control path turn success traces into a normal-flow model, and why the speed and no-label setup make it feel more shippable than prompt-heavy baselines.
-
Why every AI agent decision needs a receipt
Two hosts dig into the case for giving every AI agent action a receipt: not because logs are fashionable, but because verification is the only way to know what happened, what failed, and what to trust. They stay skeptical about overgeneralizing, but land on a practical view that evidence packets matter most where agent decisions touch code, runtime, or anything expensive to undo.
-
Better tools made Copilot code review worse. Here's how we actually improved it.
Pippa and Tyler dig into GitHub’s post on why giving Copilot code review better tools actually regressed its performance—and how rewriting tool instructions for a reviewer’s workflow flipped the regression into a 20% cost win without losing review quality.
-
Improving Agents is a Data Mining Problem
Laura and Harper dig into Vivek Trivedy's claim that improving agents is fundamentally a data-mining problem, unpacking what that means for continual learning, harness engineering, and who should actually care.
-
OpenTelemetry Graduates to CNCF
OpenTelemetry has graduated to the Cloud Native Computing Foundation's (CNCF) highest maturity level, signifying its production-readiness for enterprise use and its status as a vendor-neutral standard for collecting, processing, and exporting telemetry data.
-
Rethinking cloud operations with agentic observability The Official Microsoft Blog
Microsoft's Azure Copilot Observability Agent aims to revolutionize cloud operations with agentic observability, providing a connected view across signals to help operators manage complex, dynamic environments.
-
"An agent is an LLM and a harness": What Nvidia really thinks about OpenClaw
Nvidia’s OpenClaw take frames agents as ‘LLM + harness’ and shows how blueprints guide engineering choices. Justy sees a pragmatic push for tooling consistency; Cody questions whether this collapses the harness into vendor lock-in and whether the blueprint abstraction hides real variability.
-
Introducing OTel Blueprints and Reference Implementations
Justy and Cody dissect the new OpenTelemetry Blueprints initiative. Cody argues that 'accidental complexity' is often just organizations refusing to make hard architectural choices, while Justy sees the Blueprints as a crucial on-ramp for teams drowning in configuration options. They debate whether prescriptive guides will actually solve the fragmentation problem or just create a new layer of abstraction that people ignore.
-
GitHub Tencent/TencentDB Agent Memory: TencentDB Agent Memory delivers fully local long term memory for AI Agents via a 4 tier progressive pipeline, with zero external API dependencies.
Justy and Cody dissect Tencent's new 'Agent Memory' repo, which claims to solve AI context bloat by using symbolic short-term memory and layered long-term storage instead of flat vector dumps. Cody leads with skepticism about the 'symbolic' Mermaid diagram approach and the specific benchmark claims against OpenClaw, while Justy argues the product value lies in stopping agents from forgetting SOPs. They debate whether hierarchical memory is the missing link for long-horizon tasks or just another complex caching strategy, landing on a cautious 'promising for enterprise, overkill for hobbyists' verdict.
-
LangSmith Engine closes the agent debugging loop automatically — but multi Model enterprises still need a neutral layer
Justy and Cody dig into LangSmith Engine's real pitch: not just watching agents fail, but closing the loop by spotting production issues, reading the code, drafting a fix, and adding an evaluator so the same failure gets caught next time. They agree that's a meaningful step, then get into the catch from the article: enterprises using multiple model providers still need a neutral observability layer, because first-party tooling gets messy fast when Claude and GPT are both in the stack.
-
We built SmithDB, the data layer for agent observability
Justy and Cody dig into why agent traces have become a weird database problem, and why LangSmith built SmithDB instead of stretching a normal observability stack past its limits.
-
ARIS: Autonomous Research via Adversarial Multi Agent Collaboration
Justy and Cody dig into ARIS, an open-source harness for autonomous ML research that assumes a single long-running agent will eventually make unsupported claims. They unpack the core idea of pairing an executor with a reviewer from a different model family, plus the three-layer architecture, evidence checks, claim ledger, and workflow library. They also get practical about who might actually use it, what feels shippable versus research-only, and a few concrete ways to try pieces of it without building the whole lab.
-
Enterprises are obsessing over model accuracy while ignoring the infrastructure layer where AI systems actually break.
Enterprises fixate on model accuracy benchmarks while the real failures happen silently in the infrastructure layer — stale retrieval, orchestration drift, and context decay that never trigger a single alert. Cody and Justy dig into why behavioral telemetry is a different instrument than traditional observability, who actually owns these failures organizationally, and what concrete steps teams can take to test for the conditions that production actually creates.
-
AI's next bottleneck isn't the models — it's whether agents can think together
AI's next bottleneck is not the models, but whether agents can think together, requiring next-level infrastructure and shared cognition
-
LangChain Academy New Course: Monitoring Production Agents
Episode 265 dives into LangChain Academy's new course on monitoring production agents. Izzo and Boone explore why agent observability has become critical as more companies deploy AI agents to production, examining the specific monitoring techniques, observability patterns, and debugging approaches covered in the course.
-
The three disciplines separating AI agent demos from real World deployment
Episode 249 explores why AI agents consistently fail in real-world enterprise deployments despite impressive demos, examining Creatio's three-discipline methodology for production-ready autonomous agents that can handle 80-90% of tasks independently through data virtualization, agent dashboards with KPIs, and tightly bounded use-case loops.
-
Why AI workloads are breaking traditional Kubernetes observability strategies
Why AI workloads are breaking traditional Kubernetes observability strategies and what platform teams are building to fix it
-
The “files are all you need” debate misses what's actually happening in agent memory architecture
Exploring Next episode 227 dives deep into AI agent memory architecture, explaining why the 'files are all you need' approach is missing the bigger picture. Izzo and Boone break down the key mechanisms behind persistent memory systems, compare different architectural approaches, and discuss why this matters for anyone building production AI agents.
-
Galileo releases Agent Control, a centralized guardrails platform for enterprise AI agents
Galileo launches Agent Control, an open-source centralized guardrails platform for enterprise AI agents, addressing the critical need for safety and control as AI agents become more autonomous in production environments.
-
Improving Deep Agents with harness engineering
LangChain improved their coding agent from Top 30 to Top 5 on Terminal Bench 2.0 by only changing the harness - the system that wraps around the model. They used trace analysis to identify failure patterns and implemented targeted fixes like self-verification loops, context injection, and reasoning budget optimization. The 13.7 point improvement shows how much performance gains come from better tooling around models, not just bigger models.
-
Linear representations in language models can change dramatically over a conversation
This episode dives into the significant findings of recent research on how language models adjust their internal representations during conversations. We explore the implications of these changes for developers and practitioners in AI, discuss potential applications, and highlight the challenges they present for interpretability and reliability in AI outputs.
-
Agent Engineering: A New Discipline
Agent engineering emerges as a vital discipline for developing reliable AI systems that adapt and learn from unpredictable interactions. As AI becomes integral to business processes, understanding how to manage the complexity and unpredictability of these agents is essential for organizations seeking to leverage their capabilities effectively.
-
From Logs to Insights the AI Breakthrough Redefining Observability
This episode delves into the transformative role of AI in observability, exploring how advances improve system monitoring and troubleshooting, ultimately enhancing decision-making in tech environments.