Topic
Concordance
2 episodes
-
CTGT releases open source tool Concordance to trace why AI agents make inconsistent decisions
Jessica and Cathy unpack CTGT's Concordance, an Apache two-point-zero open source tool for tracing where AI agent workflows start making inconsistent decisions.
-
Deterministic Inference Did Not Make Our Pipeline Deterministic
CTGT's Concordance paper shows that enabling deterministic inference on a model didn't stop their pipeline from producing different compliance outputs across runs — because the non-determinism lived in the pipeline around the model, not in the model itself. Four runs of the same banking policy manual committed four different sets of obligations, including one run that dropped the rule requiring escalation to the Head of Compliance. The culprit: regenerating hex asset IDs on every upload silently changed 12,000-character prompts, and prompt sensitivity did the rest. Concordance, their Apache 2.0 tool, finds where runs diverge by comparing requests and responses alone.