Overview: Entity Resolution

Edmund A listener asked what entity resolution even means, and I realized we've been casually saying it for weeks. Geffen, that is deeply unfair to anyone trying to follow our extremely normal amount of graph jargon.

Geffen Fair. We kept pointing at the broken catalog and never explained why the cards got duplicated. So, okay, let's fix that.

Edmund Also, tiny check-in: this week has felt like every system has renamed a database problem into an agent problem. Which is apparently how we're spending a Thursday.

Geffen I object only because some of them rename it into a workflow problem first. Then they add an agent-shaped hat.

Edmund Oh, come on.

Geffen Picture a busy hotel front desk. Three guest cards arrive: Robert Johnson, Bob Johnson, and R. Johnson. The dates line up, the city mostly lines up, but the spelling and formatting do not. Entity resolution is the front desk deciding whether those cards belong to one returning guest or three different guests.

Edmund And the key is, the computer isn't being asked whether the text strings are identical. It's being asked whether the records refer to the same thing in the real world.

Geffen Exactly. That real-world thing is the entity. It might be a customer, a business, a product, a location, or a service. The records are just imperfect descriptions collected by different systems at different times.

Edmund Mm-hm.

Geffen If one system stores a full legal name, another has a nickname, and a third drops most of the address, exact matching falls over immediately. A literal comparison sees different strings. A person sees the same guest with three annoyingly incomplete cards.

Edmund Okay, but why isn't the answer just: clean the data when it comes in? Make everybody use the same form, same date format, same name field, done.

Geffen Because that helps going forward, Edmund, but it doesn't repair years of old data or data arriving from outside systems. And even pristine forms cannot stop someone using Bob in one place and Robert in another. The front desk still has to make a judgment.

Edmund Right, right.

Geffen The simplest judgment uses several clues. Names might be close. Dates of birth might match exactly. An email or phone number can be a strong signal, while a city is weaker because many people share one. The system weighs those clues together instead of crowning one field as truth.

Edmund So it builds a kind of case for each pair of cards. Same birthday, similar name, overlapping address. That feels much more like how a support team would actually reason.

Geffen Yes, with an important caveat. Similar is not identical, and similarity needs a measurement. A similarity metric is just a rule for assigning a score to how close two values are, like deciding whether Jon and John are one tiny edit apart or whether two addresses share meaningful pieces.

Edmund And that can handle a typo without pretending every vaguely similar name is the same person.

Geffen Right. Some methods compare character edits. Some look for names that sound alike. Others reward shared prefixes or common ordering. None of those methods understands identity by itself. They produce evidence for the front desk.

Edmund That's good.

Geffen Then comes the decision layer. We did a whole episode on classifiers in episode six ninety-one, but the short version is a classifier takes input signals and chooses a category. Here, its categories might be match, non-match, or uncertain.

Edmund Wait, uncertain is a real outcome? I like that more than systems acting extremely confident about somebody named J. Smith.

Geffen It should be. A rule-based system might say a matching date of birth plus a very similar name crosses the line. A learned system gets shown labeled pairs, where an expert says same entity or different entity, and learns which combinations of clues tend to matter in that domain.

Edmund Because a shared apartment address might mean something very different for customers than it does for companies. The meaning of a field depends on the job.

Geffen Exactly. One useful production approach is to have domain experts label a small sample of record pairs, then train a model to apply that judgment across a much larger dataset. But a learned model is not a permission slip to stop inspecting errors.

Edmund No way.

Geffen That is the recurring trade-off: speed versus recall, where recall means finding the real matches that exist. Then there is precision, meaning how often the matches you declare are actually correct. You want both high, obviously. Reality makes you negotiate.

Edmund And the threshold is where that negotiation gets painfully real. Above some confidence score, match. Below it, don't match. But move that line and the product behavior changes.

Geffen Yes. Set it too high and you leave one customer split across three profiles. Set it too low and you combine two different people with the same common name. That false positive can contaminate account history, fraud checks, recommendations, and anything downstream.

Edmund This is where my product brain gets very unromantic. A missed duplicate is annoying. A wrong merge means someone sees another person's support history, or a financial workflow makes a bad call. Those aren't symmetric mistakes.

Geffen They are not. So many systems use a middle lane: high-confidence pairs link automatically, low-confidence pairs stay separate, and the ambiguous ones go to human review. The human feedback can also improve the matching rules or the learned model.

Edmund Sure.

Geffen And linking is different from merging. A link says these original cards appear to refer to the same entity. A merge creates one canonical record, the clean master profile. Linking preserves the evidence and is easier to revise when the front desk got it wrong.

Edmund That feels like the whole thing, honestly. Keep the messy cards. Add a durable identity layer over them. Don't shred the originals because one score looked persuasive on a Tuesday.

Geffen —okay, fair. And that is why this keeps showing up in our graph conversations. We looked at that knowledge-layer graph where fragmented documents created one hundred forty-nine concept objects from twenty-one documents, while a curated graph had nineteen. If the same thing splinters into fake separate nodes, traversal gets confused before it even starts.

Edmund Stop it—

Geffen —okay, I know, the number is rude. But it makes the point. A graph edge can be perfectly stored and still point from the wrong duplicate node. Entity resolution is the work of deciding which cards deserve one identity before the graph starts making relationships look meaningful.

Edmund And that connects to the Graph RAG stuff we went through. Retrieval augmented generation means a model fetches outside information before answering. Graph RAG uses relationship paths to choose context, so duplicate entities can send the model down the wrong path.

Geffen Exactly. We already said a graph is expensive decoration for a simple lookup. Here the sharper version is: a graph with unresolved duplicates can be expensive confusion. The relationship machinery only earns its keep if the entities underneath are coherent enough.

Edmund That fake C E O problem, but for data. Give every duplicate its own title and suddenly the whole organization has a slightly more elaborate inbox.

Geffen Okay, that's genuinely funny.

Edmund We also saw the practical version in OpenTelemetry entity events. Observability systems need to know whether events belong to the same service, machine, or component across time. If the identity layer is fragmented, the operational view turns into a pile of almost-the-same things.

Geffen And that is why I resist calling entity resolution mere data cleanup. Cleanup sounds like a one-time chore. In practice, new records keep arriving, fields change, identifiers disappear, and relationships evolve. The front desk never really closes.

Edmund Where does it stand now, then? Is this one of those old enterprise plumbing problems that got swallowed by machine learning branding?

Geffen Still core. Machine learning can improve the matching decision, and embeddings are another useful signal in some systems. We did embeddings in episode seven oh five. In one sentence, an embedding turns an item into numbers arranged so related meanings tend to sit closer together. But that doesn't replace identity judgment.

Edmund Because two records can sound semantically similar and still be different entities. Two branches of the same company, two products in the same family, two actual humans named Alex Chen. Similarity is evidence, not a verdict.

Geffen Precisely. The current shape is usually hybrid: cheap rules for candidate blocking, several comparison signals, perhaps a learned classifier, confidence thresholds, and review for the dangerous gray area. It is not glamorous, but it is still exactly how systems get made trustworthy enough to use.

Edmund So, if this sticks, the front desk isn't asking, "Do these cards look alike?" It's asking, "How much evidence says they describe one real thing, and how costly is it if we're wrong?"

Geffen Yep. And the answer is allowed to be: we do not know yet. That is not failure. It is often the most accurate record in the system.

Edmund I love that our big conclusion is responsible uncertainty and better filing. Nine months into Exploring Next, somehow the boring layer wins again. See you next time, Geffen.