From dots to a picture: a Yale-led study finds symbolic order hidden in the numbers behind AI chatbots
Nobody wrote the internal rules of AI chatbots; the systems formed them on their own while learning. According to a Yale-led study, those rules are not chaotic after all: behind the strings of numbers there is a structure that can be described with words, roles and a single equation. It can even be edited.
A puzzle inside a machine that works
Ask a chatbot about a fraction, the structure of a sentence or a bug in a piece of code, and you will most likely get a sensible answer. If we could look inside while it works, though, we would not see words, rules or formulas, only long strings of numbers, and not one of them stands for the idea of “dog” or “denominator”.
Humans think differently. For us, words, numbers and variables are symbols: they stand for something, and we know which one plays what role. A sentence has a subject, a verb and an object; a fraction has a numerator and a denominator.
That is why the question has long been awkward: how can something be good at language, logic and maths if it does not work with symbols inside? There is no programmed rulebook behind these systems either; they learned from vast datasets. In engineering terms: an excellent black box whose internal logic we know only in part.
Dots and the picture
R. Thomas McCoy of Yale and his co-authors (Paul Soulos, Tal Linzen and Paul Smolensky) proposed that there is order behind the strings of numbers, just not in the familiar form.
McCoy’s analogy is the pointillist painting; think of Georges Seurat’s A Sunday on La Grande Jatte, the work that illustrates the Yale article. Up close, we see coloured dots that mean nothing on their own. From a few steps back, a whole scene emerges: people, trees, water, shadow. The researchers suspect the same of the vectors inside language models: the individual numbers are the “dots”, and the meaning lives in the pattern.
Fillers and roles
To test the idea, the team used a tool called the tensor product representation (TPR). It splits information into two components:
- filler: the content, for example a word or a number;
- role: the structural position, for example subject, verb, object, numerator or denominator.
The point of TPR is that this filler–role structure is not stored in a separate list of rules but inside the vector space itself.
The test: replace what is complicated
The experiment is simple, and it makes a strong claim. The researchers replaced the complicated process that generates the networks’ vectors with a closed-form equation based on such symbolic structures. If the approximation is poor, the model falls apart. If it is good, behaviour hardly changes.
Behaviour stayed largely the same.
It was not a single experiment. They started with small neural networks trained to manipulate lists, then repeated it on seven large language models in four domains: language, arithmetic, logic and programming code, the areas at the heart of research on symbolic intelligence.
When editing works too
The most vivid result, though, is not the approximation but the intervention. The researchers deliberately rewrote some components of a vector, and the model’s behaviour changed predictably.
In one example they changed how the model encodes the word clever, so that it moved from modifying the subject of the sentence to modifying the object. The model then responded as if the sentence had been rephrased accordingly.
This is the strongest evidence. The best sign that a description is accurate is that we can act on the system through it. Here that worked: the model’s behaviour really depends on the structures the researchers identified.
In McCoy’s words, LLMs “naturally converge to representations with role-filler structure”. This order forms by itself during training; it was not built in.
The lesson: a black box is not necessarily chaotic
Three conclusions can be drawn from the study.
- The question was badly posed. We often argue that chatbots either “really understand” or are “just doing statistics”. The study points to a third possibility: a system may not handle symbols as such, and yet a symbolic structure can form inside it during learning.
- From understanding to intervention. Anyone who knows the internal structure can not only explain the behaviour but also act on it. That is the difference between an observer and an engineer.
- Reliability starts with understanding. If we do not know how a language model organises information, we cannot know when it will behave in an unexpected or unsafe way. According to the researchers, the result points to a way of controlling these models more precisely.
What the study does not claim
The work is a preprint: it appeared on arXiv on August 30, 2026 (The Emergent Symbolic Structure of Artificial Neural Networks, arXiv:2608.29530) and has not yet been peer-reviewed in a journal.
The researchers themselves are cautious: according to McCoy, the work only scratches the surface of what must be studied to understand how neural networks encode information. It does not prove that chatbots “think like humans”, nor that everything inside them can be described symbolically. It shows that, in the cases examined, behind the strings of numbers there is no chaos but a structure that can be grasped, measured and edited.
That is why it deserves attention. Until now we mostly talked about chatbots in terms of what they output. This research suggests that their insides can be examined, measured and edited too.
Sources: Mike Cummings: How do AI chatbots “reason” like humans? A Yale-led study offers clues, Yale News (October 8, 2026); R. Thomas McCoy, Paul Soulos, Tal Linzen, Paul Smolensky: The Emergent Symbolic Structure of Artificial Neural Networks, arXiv:2608.29530 (preprint, August 30, 2026)
Questions, or an idea to talk through?
Whether it is a security system, the IT network behind it or just a thought on this article, I am happy to hear from you.
Get in touch
