A researcher specializing in the interpretability of artificial intelligence models says he has found an explanation for why certain artificial neurons activate for concepts that seem unrelated. According to Nick Cammarata, these neurons may detect more abstract shared features than previously thought. This approach, dubbed “recursive generalization,” offers prospects for better understanding large language models, but it remains debated and still needs to be evaluated by the scientific community.
Generative artificial intelligence models are often described as “black boxes.” They receive a prompt and produce a response, but the precise mechanisms linking these two steps are difficult to trace. For years, researchers have therefore sought to understand what goes on inside these systems and how their various components contribute to their results.
Some of this work involves studying the artificial neurons that make up networks. In a long article of around 26,000 words, Nick Cammarata, a researcher at Goodfire AI and a former OpenAI employee, describes a method for analyzing these mathematical units. His goal: to shed light on the role of neurons whose activation can sometimes be difficult to interpret.
Why are artificial neurons difficult to interpret?
AI systems rely on different architectures, such as generative adversarial networks, recurrent neural networks, transformers, and convolutional networks. Despite their differences, these architectures use mathematical operations organized into networks. Artificial neurons are therefore not physical cells: they are computational elements whose activations contribute to information processing.
However, observing a neuron’s activation is not always enough to understand what it represents. The same neuron can fire in response to several elements that seem very different. This characteristic makes the network harder to interpret: if a unit responds to distinct concepts, it becomes difficult to assign it one simple, unique function.
Until now, one commonly offered explanation has been superposition. According to this hypothesis, a model might use the same neuron to encode several concepts because it has a limited number of units. The representations would be sufficiently separated in mathematical space to coexist without interfering too much with one another. The neuron would then have several meanings, making it “polysemantic.”
An example linking a cat to a car
To illustrate this difficulty, Cammarata describes a neuron that could activate both in response to a cat and in response to a car. At first glance, these two categories have no obvious connection. Within the framework of superposition, this shared activation could be explained by a single unit being shared between two concepts with no direct relationship.
The researcher, however, proposes another interpretation. He suggests that the neuron might not recognize the cat or the car as such, but rather a shape present in some images of both categories: a V-shaped contour. A cat’s face can have this kind of configuration, as can the front end of some cars.
This hypothesis is also supported by examples that would not trigger the same response. Round objects, such as a ball or a snail, would not have the visual feature being sought and therefore would not activate the neuron in the same way. The neuron could thus be sensitive to a shared visual property rather than to two independent categories.
“Recursive generalization,” another possible explanation
Cammarata gives this mechanism the name recursive generalization. In this approach, a neuron can respond to varied examples because they share a common structure, sometimes more abstract than the objects themselves. The unit would not necessarily be storing several separate concepts: it could be detecting a recurring feature found in different contexts.
This distinction is important for interpretability. If a neuron activates in response to several images, the raw observation does not immediately reveal whether it groups distinct concepts or recognizes a shared property. Examining the examples, visual details, and conditions under which the activation occurs could help distinguish between these possibilities.
Recursive generalization does not prove that superposition does not exist. The two phenomena could coexist in the same model or contribute differently to its functioning depending on the neurons and tasks involved. Cammarata’s proposal instead offers an additional explanation to test when a unit seems to respond to disparate content.
Understanding neurons better to study entire models
Analyzing neurons is part of a field called mechanistic interpretability. It seeks to identify a model’s internal mechanisms, rather than simply comparing the prompts it receives with the responses it produces. Studying individual activations could thus reveal certain stages of information processing and provide clues about the reasons for a given behavior.
This approach is also of interest to those developing tools around AI. New applications are already transforming the professional landscape, with the emergence of innovative professions linked to artificial intelligence. Understanding how models work is therefore both a scientific and practical concern, particularly when it comes to better assessing their capabilities and limitations.
Nick Cammarata believes that studying neurons could be useful for very large models with up to several trillion parameters. If this estimate is confirmed, it would suggest that the internal analysis of complex systems is more accessible than previously thought. However, it does not mean that every decision made by a model can already be explained fully and with certainty.
What is the connection to AI safety and alignment?
Understanding why a model responds in a certain way could help identify unexpected behaviors. Researchers could seek to determine which internal representations contribute to a response, then study the circumstances in which they appear. In time, this knowledge could inform work on alignment, that is, how to bring AI behavior closer to the goals and constraints set by its developers.
This question is becoming particularly important as systems become capable of planning and chaining operations together. For autonomous agents, it is not enough to observe an isolated response: it is also necessary to examine the steps taken to complete a task. Studying internal mechanisms could complement monitoring and control methods, without by itself guaranteeing that a system will always act as intended.
System security also depends on protecting the information that flows through them. Technologies such as blockchain are being explored to strengthen data traceability and security, a subject discussed in this article on data traceability through blockchain. Model interpretability and data security address different concerns, but both can contribute to a broader discussion about trust in digital tools.
Enthusiastic reactions, but also reservations
Cammarata’s publication has prompted mixed reactions among specialists. Sam Altman, head of OpenAI, praised the work, while Tarun Gogineni, also an employee of the company, described it as an important step for science. Other researchers welcomed the approach with interest, while suggesting that it did not necessarily change the field’s major conclusions.
Neel Nanda of Google DeepMind, in particular, expressed reservations about the scope of this proposal. Even if every neuron could be understood individually, predicting the behavior of the network as a whole would remain extremely difficult. The analogy offered is that of the weather: knowing the properties of microscopic elements is not enough to simply predict how a complex system will evolve.
Another reservation concerns the possibility that the explanations may coexist. Discovering that a neuron detects a shared feature does not automatically rule out superposition. Some activations may be linked to shared properties, while others may result from reusing a unit for multiple representations. Researchers will therefore need to determine in which cases each mechanism applies.
A hypothesis that still needs to be examined
The results presented have not yet been validated through a full independent evaluation. Specialists will need to examine the methods used, reproduce the observations, and test whether recursive generalization applies to other neurons, other models, and different types of data. In particular, an important point to verify is the distinction between a genuinely shared feature and a coincidence in the activations.
This caution is all the more necessary because interpretability touches on concrete safety issues. AI models require significant computing resources, and their development takes place in an environment subject to energy constraints. Debates about energy availability and security, such as those discussed in connection with the G7’s commitment amid tensions in the energy market, are a reminder that the growth of these technologies also depends on material and geopolitical conditions.
For now, Cammarata’s proposal is therefore a research avenue, rather than a definitive explanation of how artificial neurons work. It highlights the possibility that some seemingly contradictory activations may be based on common, recognizable patterns. Further analyses will need to establish the true scope of this idea and clarify what it can explain about large language models.
