You didn't understand the model. You completed it.

In a two-hour meeting to model the collections module, the head of accounting and the head of credit said “balance” fourteen times without noticing they were not talking about the same thing. For accounting, a customer’s balance is the algebraic sum of the entries posted to their account up to the cut-off date. For credit, the balance is what the customer still owes, which leaves out payments in transit and adds authorized charges not yet invoiced. The two numbers differ by several thousand. Both are correct. Neither of them misused the word.

Nobody stumbles in the conversation, because the word admits both readings and both people move on convinced they understood each other. The stumble comes later, when someone has to write Balance and decide what it returns. That is where the word runs out. Whichever reading you pick, half the team will read the model backwards for the next three years.

In an earlier essay I argued that complexity is decided before the code, in the model, and that the code merely inherits a cut that already existed. That leaves the next question open. Who draws that cut, and with what.

AI proposes the model too

The immediate answer today would be that nobody draws it anymore. You paste the transcript of that meeting into a tool, ask for a domain model, and twenty seconds later you have aggregates with reasonable names, declared invariants, a clean separation between the accounting side and the credit side, and even a note warning that the term “balance” appears in two incompatible senses. It is a good result. Often better than what a team in a hurry produces.

So look at what produced that result. Murray Shanahan, in “Talking About Large Language Models,” insists on returning the question to its minimal form. What the system answers is how a fragment of text might continue according to the statistics of human language, not what is true about the domain. And he warns against the intentional vocabulary we use to describe it. Of a question-answering system built on a language model he writes that “[i]n no meaningful sense, even under the licence of the intentional stance, does it know that the questions it is asked come from a person, or that a person is on the receiving end of its answers.”

Emily Bender and Alexander Koller turned that into a thesis in “Climbing towards NLU.” They argue that “a system trained only on form has a priori no way to learn meaning,” and they make the case with a fable more convincing than the statement. Two castaways, A and B, are stranded on separate islands and communicate through an undersea cable. A hyper-intelligent octopus taps the cable. It knows no English, but it detects statistical patterns with enormous skill, until one day it answers instead, passing itself off as B. A receives plausible replies and takes them at face value. The sentence the authors close the experiment with is the one that matters for everything that follows. “A does all the work in attributing meaning to O’s response.”

That describes precisely what happens when you read the model the tool handed back and nod. The meaning of Balance did not come in the answer. You put it there as you read it. You completed it.

That is a position in an open debate, not a settled fact. The counterargument has a primary source. Kenneth Li and his coauthors, in “Emergent World Representations,” trained a GPT-style model solely to predict legal Othello moves and then examined what was inside. They report finding “evidence of an emergent nonlinear internal representation of the board state.” A board, out of nothing but next-move prediction. Anyone claiming that a system trained only on form never builds anything resembling a world model has to answer for that result.

I do not intend to settle that debate, and what follows does not depend on settling it. Grant the counterargument everything it asks for. Suppose there is internal representation, and something that deserves the name understanding. The problem I care about shows up anyway, and it shows up earlier, on the human side.

Gavagai

The interesting part is not that AI has no access to meaning. It is that meaning is not fixed for us either.

W. V. O. Quine demonstrated this in Word and Object, in 1960, with a thought experiment philosophy is still arguing about. A field linguist tries to translate a language with no interpreter and no dictionary. A rabbit runs past, the native says “gavagai,” the linguist writes down rabbit. And there Quine takes the note apart. “For, consider ‘gavagai’. Who knows but what the objects to which this term applies are not rabbits after all, but mere stages, or brief temporal segments, of rabbits?” It can refer to the rabbit, to a temporal stage of the rabbit, to the undetached parts of the rabbit. No observation of linguistic behavior distinguishes those options, and no number of passing rabbits will distinguish them. Quine generalizes the result without softening it. Translation manuals, he writes, can be built in divergent ways, “all compatible with the totality of speech dispositions, yet incompatible with one another.”

Swap the rabbit for the collections meeting and you have your job. When the head of credit says “balance” and points at a screen, you are facing a gavagai. The term can refer to the state of the account at an instant, to the debt collectible today, to the total exposure committed to that customer, to any of those with or without transactions in transit. Asking helps, but it does not close the list, because every answer comes back in more words that carry the same problem. When I wrote that ubiquitous language is not a glossary I was saying a mild version of this. This is the hard version. No glossary is possible because there is no meaning waiting to be transcribed. There is a set of mutually compatible uses, and someone has to settle on one.

The head of credit might answer that this is what business rules are for, written down, approved, signed. Wittgenstein devoted paragraph 201 of the Philosophical Investigations to showing why that is not enough. A rule does not by itself determine its application, because any course of action can be made to accord with it under some interpretation. What stabilizes correct application is not the statement but the practice, the public use of a community that corrects itself. The rule “the balance excludes payments in transit” does not tell you what counts as a payment in transit. The team settles that, case by case, over months.

What the modeler cuts with

If the modeler has to choose, the question is what they choose with. George Lakoff and Mark Johnson opened Metaphors We Live By in 1980 with a claim that sounded excessive at the time and now has four decades of empirical work behind it. “Our ordinary conceptual system, in terms of which we both think and act, is fundamentally metaphorical in nature.” Metaphor is not the poet’s ornament. It is how we think the abstract, and they define it in a line. “The essence of metaphor is understanding and experiencing one kind of thing in terms of another.” Arguing is thought of as war, which is why positions get attacked and claims get defended without anyone feeling they are speaking in figures. Time is thought of as money, and so it gets spent.

Now look at the vocabulary of any financial domain. Flow. Portfolio. Account. Exposure. None of them is a neutral label. Each one carries implications about what operations the object admits and what questions make sense to ask of it, and it does so before anyone writes a line. “Account” shares a root with counting, and English kept the second sense in accountability. It brings with it a historical record that accumulates and that someone answers for. “Exposure” borrows its name from photography, and it treats risk the same way, fixed at an instant, with no history. Modeling a customer’s debt with one or the other produces different operations and different bugs. Choosing the metaphor is the first act of modeling, and it is already done before design begins.

The second mechanism runs deeper, because to abstract is to forget. Rodrigo Quian Quiroga and his coauthors published their discovery of the so-called concept cells in Nature in 2005. These are neurons in the medial temporal lobe that respond to the same person across different photographs, drawings, and even their written name. What they describe is “an invariant, sparse and explicit code, which might be important in the transformation of complex visual percepts into long-term and more abstract memories.” An invariant, sparse encoding. The neuron keeps the concept and discards everything else. In an interview, Quian Quiroga himself put it this way: “to think we have to forget, which is counterintuitive, as forgetting involves abstractions.”

He points out where the intuition came from. Borges had dramatized the limit case in 1942. Ireneo Funes, after falling from a horse, remembers absolutely everything, every leaf of every tree and every shape of every cloud, and precisely because of that he cannot think. It bothers him that the dog seen in profile at three fourteen should carry the same name as the dog seen head-on at three fifteen. Without forgetting there is no concept of dog. There is a list. A domain model is that concept cell of the business, an invariant abstraction that erases everything the modeler decided did not matter. And that decision, the one about what gets forgotten, is the only thing the model cannot show, because the model is what remained.

Who understands

Quine demonstrates the indeterminacy of meaning, Lakoff and Johnson describe the metaphorical structure of thought, Quian Quiroga shows forgetting as the condition of abstraction, Bender and Koller point to who does the work of attributing meaning. None of them writes the conclusion that comes now. I am the one putting it there.

The human is the anomaly in the system, the only node that understands, and understanding consists of arbitrarily fixing a meaning that was never fixed, with a mind that thinks in metaphors and remembers by forgetting. All the complexity of the model is the shadow of that fixing. When someone decides that Balance excludes payments in transit, that decision was not sitting in the domain waiting to be discovered. Whoever made it brought it, along with their metaphor and everything they chose not to look at.

In the earlier essay I called AI the anomaly, in Kuhn’s sense, because automating the writing of code should have reduced the difficulty and instead pushed it upstream. The deeper anomaly, the one no tool will absorb, was always on the other side of the keyboard.

Complexity with no author

A language model is trained on human text. It inherits our metaphors and our ambiguity, and it propagates them at a scale no team ever reached. Why would the human be the anomaly, and not both at once.

The short answer is that inheriting is not originating. But the short answer is not enough, because once ambiguity is in circulation it hardly matters who introduced it first.

Generative AI simulates the result of having understood without the act of understanding. It returns the form of a decision without anyone having decided. When it hands over a domain model, the cuts are there, clean, with names that sound considered, and they look chosen. Nobody chose them. They fall where they fall because that is where the cut lands, statistically, in the text the system learned from. That is not less complexity. It is complexity with no author.

And that is the worst version of the problem. Peter Naur, in “Programming as Theory Building,” argued that a program dies when the team possessing its theory is dissolved. The text goes on executing and producing useful results, and yet it is dead, because no one can intelligently answer a demand for modification. A model generated by a tool is born in exactly that state. No team dispersed, because there was never a theory to lose. You ask why the cut falls there and there is no one to ask. It is Naur’s dead program, dead from birth.

AI does propagate ambiguity, and in better handwriting. The point survives anyway. The difference lies in who can answer for it. You can be asked why the balance excludes payments in transit, and you answer with a reason that can be argued with, revised, and traced to someone who sat in that meeting.

Take that node out of the system and complexity does not disappear. The person who can answer for it does.