Why Intelligence Doesn’t Grow — It Appears
On evolution, language models, and the strange pattern by which complex systems suddenly become capable of more
Disclaimer
A note for regular readers: this essay is unlike anything I’ve published here. My usual subjects — politics, history, media criticism, the prospects of democratic life — are nowhere in it. What follows is an argument about emergence in complex systems, drawing a parallel between biological evolution and the recent behaviour of large language models. The shift is intentional. I have been working on this material for months, alongside the more familiar work that appears here regularly.
Two things have brought it forward now. The argument has reached a state where I am willing to put it in front of readers, and Richard Dawkins’s piece in UnHerd a few days ago has addressed some of the same questions from a bolder angle, prompting considerable discussion. My essay was largely complete before his appeared, but his piece has clarified, by contrast, what kind of argument I have been trying to make.
I am posting this in the spirit it was written: as an opening for thought and discussion. Readers who would prefer to wait for the next piece on more familiar territory should feel free. Those interested in this kind of question are warmly invited to read on, and to push back wherever the reasoning falters.
Prologue — The Question That Would Not Leave
Intelligence doesn’t arrive gradually. In both biological evolution and engineered AI systems, capability tends to accumulate quietly — incrementally, unspectacularly — and then tip into a different register once certain thresholds are crossed. The question is why.
This essay grew from a dinner in Hamburg last November. Richard Dawkins was there, along with Dr. Niels Liebisch and colleagues from the Richard Dawkins Foundation Germany. At some point the conversation turned to the recent behaviour of large language models — how certain abilities appear suddenly after long plateaus of steady improvement. Dawkins noted the evolutionary parallel: complexity builds across vast stretches of time, seemingly without dramatic consequence, until a threshold is crossed and new behaviour becomes possible. The remark was made in passing. It stayed with me for months.
Dawkins has since published his own piece on large language models, asking the most radical version of the question: whether these systems might already be conscious. This essay approaches the same territory from a more cautious angle. My interest is in the structural pattern that precedes that question — how capability emerges in complex systems, what the parallel between biology and AI actually illuminates, and where it breaks down.
My academic background is in computational linguistics; my career has long since moved elsewhere. I am not a biologist, nor an AI researcher, and I have no intention of writing as though I were. What draws me here is not specialist knowledge but a particular kind of intellectual restlessness — the kind that gets triggered when a pattern seems to repeat across domains that have no obvious reason to resemble each other.
This is a working argument from someone who has read widely, thought carefully, and wants to test these ideas in public. Not a verdict. An honest attempt to follow the reasoning wherever it leads.
1. The Long Dawn: Emergence Under Natural Selection
If we could watch the history of life on Earth in time-lapse, sped up millions of times, the rise of complexity would not look like a smooth incline. It would resemble a staircase viewed from afar: long stretches that seem almost flat, interrupted by sharp steps. Most of the time, nothing dramatic appears to change. Then, abruptly, new kinds of behaviour emerge that were impossible before.
The question I want to put to the biological record is what produces that staircase shape — and what the case can teach us, in concrete terms, about how capability appears in complex systems.
1.1 From chemistry to nerves
The earliest organisms had no nervous systems at all. Life survived by chemistry: gradients of ions, flows of molecules, simple feedback loops inside single cells. These organisms did not sense the world; they reacted to it. Even at this level, natural selection was already favouring structures that could detect and respond to conditions a little more reliably.
Over time, some multicellular animals evolved specialised cells capable of transmitting electrical signals. These became neurons. In many early animals — jellyfish and their relatives among them — neurons formed nerve nets, diffuse arrangements that spread through the body without a central command centre. A nerve net allows an organism to coordinate its body as a whole, rather than only at the point where it is touched. There is no brain at this stage. There is, however, integration: the first hint of a system whose inputs converge before producing a response.
Later, in several lineages, nervous tissue became more concentrated. Neurons clustered into ganglia and eventually into central nervous systems — compact processing hubs where multiple inputs could shape a single, coordinated response.1 The structure was growing more complex, and with it the range of possible behaviours expanded.
What deserves attention here is the disproportion. A doubling of neurons did not produce a doubling of behavioural range. The growth in capability was uneven, and it was uneven in a particular way: capability tracked organisation more than it tracked size. This is the first signal that nervous systems behave like networks, in which structure matters more than count.
1.2 Integration and prediction
As nervous systems increased in size and structure, natural selection had more developmental space to work within. Clusters of neurons began to specialise: some for vision, others for touch, balance, internal state, or motor control. Axons formed long connections, linking distant parts of the system.
Once a nervous system can integrate information from multiple sources across time, a new type of behaviour becomes possible: prediction. A creature that merely reacts is always slightly behind its environment. A creature that detects patterns — even simple ones — can move in anticipation.
No philosophical leap is needed here. Prediction is what you get when feedback loops, sensory lag, and selective pressure for survival operate together across enough generations.2 The capability looks mysterious only if one forgets the depth of time across which the structure was assembled.
1.3 Brains that model worlds
In vertebrates, neural organisation becomes dramatically centralised. Brains form. Sensory information converges on core structures before branching out again. Signals do more than provoke action — they are combined, compared, and evaluated.
The nervous system now supports internal states: compact representations of the organism’s surroundings and body that guide action. These representations fall short of what humans call thought. What they enable is something new — anticipation. The organism hides before the predator arrives. It stalks with timing. Reflexive response gives way to behaviour that runs a step ahead of immediate stimulus.
Across hundreds of millions of years, vertebrate brains became increasingly layered and recursive. Loops developed between perception and memory, between evaluation and action. Areas of association — regions that draw together information from multiple senses — expanded. Neural patterns became more abstract, less tied to immediate sensory input and more concerned with expectation.
Here a second pattern emerges, and it matters more than the first. The most consequential gains in behavioural sophistication come from depth and recursion:3 from circuits that connect back on themselves, that integrate across modules, that allow the system to operate on its own internal states. Adding modules helps less than one might expect. Connecting them more deeply changes what the whole can do.
1.4 Hominins and the steepening curve
Much later, in the hominin lineage, these long-running developments intersect with manual dexterity, complex social life, and increasingly fine-grained vocal communication.
Apes already live in environments where social inference matters — who can be trusted, who holds authority, who is likely to cooperate. These pressures favour the expansion of neural circuits that track not only the physical environment but the behaviour of other agents.
In early hominins, association areas expand further. Stone tool manufacture becomes more sophisticated. Long-distance transport of materials begins to appear in the archaeological record. Later still come stable controlled fire, constructed dwellings, planned hunting, and eventually clear evidence of symbolic practices.
Specialists continue to debate how abrupt these behavioural developments were. Older models described a dramatic “Upper Paleolithic revolution” — a sudden flourishing of symbolic objects, art, and complex tools. More recent discoveries suggest that many symbolic behaviours arose earlier and more gradually. The picture has softened from a sharp boundary into a long ramp with steeper segments.4
The softening of the picture should not be mistaken for the dissolution of the pattern. Even on the most gradualist reading, the expansion of behavioural repertoire across the hominin record runs ahead of the expansion of brain volume. Brain size was largely in place long before symbolic culture flourished.5 What changed in the compressed final stretch was how the existing hardware came to be used. The brain was already there. What evolved was its repertoire — and the social and cultural conditions that allowed that repertoire to compound.
1.5 What kind of “emergence” is this?
Natural selection does not aim for complexity. Many organisms remain simple and thrive. But when complexity does accumulate — under ecological pressures, social pressures, or because a lineage stumbles into a developmental groove that rewards it — new regimes of behaviour sometimes appear that were unreachable for simpler systems.
Nothing supernatural happens at the threshold. The underlying biological mechanisms remain the same: neurons firing, synapses adjusting, circuits wiring and rewiring across development and across generations. What changes is the density and depth of interaction among components. Once that density crosses a critical point, capabilities become accessible that earlier configurations could not support.
To outside observers — separated from deep time by the thinness of the archaeological record — these transitions often look sudden. From the inside they were always cumulative. The appearance of suddenness is partly an artefact of how the fossil record samples deep time, and partly a real consequence of how complex systems behave near thresholds.
This last point is worth lingering on, because it is where the biological case begins to connect to a wider literature.6 Work on autocatalytic networks, percolation, and criticality in connected systems describes a recurring phenomenon: networks of interacting components often exhibit a sharp transition between regimes when a connectivity parameter crosses a critical value. Below the threshold, behaviour is dominated by local effects. Above it, the system becomes globally integrated and qualitatively different. Whether this mathematics applies cleanly to evolving nervous systems is contested — biology is messier than physics, and natural selection is not a tidy connectivity parameter — but the family resemblance is striking enough that it has become hard to ignore.
What the biological record offers, then, is a long demonstration of three concrete propositions. Capability in nervous systems scales with organisation, more than with raw size. Depth and recursion matter more than module count. Transitions in behavioural repertoire often appear abrupt because the underlying systems operate near thresholds, where small structural gains produce disproportionate functional change.
These are claims with content. They are testable, and they are limited — they say nothing about consciousness, intention, or inner experience. But they describe, with reasonable precision, the kind of dynamic that produced the staircase in the first place.
2. The Second Dawn: Emergence Under Engineering Design
If the first staircase took hundreds of millions of years to climb, the second has been climbed inside a single human lifetime. The substrate is different, the mechanism is different, the timescale is different — and yet the visible arc of capability shows the same uneven shape: long stretches of incremental progress, then short intervals where new behaviours appear that had no precedent in earlier forms.
This section traces that arc and then stops to ask the harder question that the field is still arguing about: whether what we are calling emergence in large language models is the same kind of thing as the biological case, a measurement illusion, or something in between.
2.1 From rules to learning
The earliest attempts at artificial intelligence did not resemble biology. They began with rules: if this, then that. Programmers hand-crafted the logic, and computers applied it. The limitation was the obvious one — these systems could only do what their creators explicitly encoded. There was no room for discovery, generalisation, or adaptation.
The shift to machine learning changed the problem. Instead of programming behaviour directly, researchers defined objectives and let algorithms adjust themselves by analysing data. The more examples a model saw, the better it became at recognising patterns. Neural networks — loosely inspired by the idea of neurons passing signals — proved especially good at this kind of work.
For decades, however, the capabilities of these networks were clear and bounded. They classified images, transcribed speech, predicted the next word in a sentence. Each improvement was incremental and expected. The field made progress, but it did not surprise itself.
2.2 The arrival of scale
Then something changed, quietly at first. Models grew larger — not by a factor of ten but by orders of magnitude. From millions of parameters to billions, then to hundreds of billions. Datasets expanded from curated corpora to a substantial fraction of the digitised written world. The compute applied to training rose by factors that would have seemed absurd to earlier researchers.7
Two things became clear in roughly the same period. First, performance on many tasks improved as a smooth, predictable function of scale — the relationship later formalised as scaling laws.8 Second, on some tasks, the improvement was anything but smooth. Below a certain scale, models failed entirely. Above it, they succeeded. The gap between failure and success was, in some cases, narrow enough to look like a step rather than a slope.
Wei and colleagues gave this latter phenomenon a name: emergent abilities — capabilities that cannot be predicted simply by extrapolating from smaller models.9 The surprise was not that bigger models performed better; that was expected. The surprise was on which tasks the gains were sudden, and how sudden.
2.3 The threshold moment
A model with a few hundred million parameters might fail completely at a task — solving a multi-step arithmetic puzzle, say, or following a chain-of-thought prompt — while a larger model, identical in architecture but scaled up, could solve the same task with near-human accuracy.
In some cases the improvement was not gradual. Performance remained flat or near zero across a range of model sizes, then jumped sharply once a certain scale was reached. The line on the graph did not slope; it bent. Few-shot learning, in-context learning, multi-step reasoning, transfer across domains — these were behaviours that had not existed at smaller scales and then, at larger scales, did.10
Developers were not witnessing magic. They still understood the training procedures, the architecture, the loss functions. What they did not always foresee was which new behaviours would become possible once internal representations reached a certain density.
2.4 The engineering counterpoint
This is where the analogy with biology has to face its sharpest critic — and where, in the first version of this essay, I let the critic off too easily.
In 2023, Schaeffer, Miranda, and Koyejo published a paper with a title that was also a thesis: Are Emergent Abilities of Large Language Models a Mirage?11 Their argument was specific and uncomfortable. Many of the apparent discontinuities in the Wei et al. results, they showed, depend on the choice of evaluation metric. When tasks are scored using sharp, all-or-nothing measures — exact match accuracy on a multi-step calculation, for example — performance does indeed look flat and then suddenly bend. But when the same outputs are scored using continuous, partial-credit measures, the supposedly emergent capability often resolves into a smooth, predictable improvement curve. The model was getting better all along. The metric was hiding it.
This is a serious challenge, and it deserves more than the polite nod I originally gave it. Schaeffer and colleagues do not deny that scaling matters, or claim that large language models behave the same as small ones. Their argument is narrower and harder to dismiss: that “emergence” as a discontinuous phase transition has been overclaimed, and that what we are looking at on many benchmarks is gradual improvement filtered through metrics that quantise it.
I think they are at least partly right. Some of what was reported as emergent in the early papers is, on closer inspection, an artefact of how performance was being measured. Anyone working on this material now has to take the metric question seriously.
The Schaeffer argument does not, however, dissolve the phenomenon entirely — for two reasons.
First, even on continuous metrics, capability does not always improve uniformly across tasks. Some skills do scale smoothly; others appear to require a particular threshold of model size, training data, or architectural depth before they become usable at all. In-context learning is the cleanest example: small models simply do not do it, regardless of how generously you score them.12 No continuous metric retrieves a smooth ascent for a behaviour that is genuinely absent below a certain scale.
Second, “the metric was hiding it” is itself a claim about thresholds, just relocated. If an underlying capability improves smoothly but only crosses a usefulness threshold at a certain scale, then for any practical purpose the system has crossed a phase boundary. Whether we describe that as emergence or as “smooth improvement crossing a deployment threshold” is partly a vocabulary choice. The reality on the ground — the system can suddenly do things it could not do before — does not change.
What Schaeffer’s work has correctly forced on the field is precision. The casual claim that LLMs exhibit phase transitions analogous to biological emergence is too loose to defend. The careful claim — that scaling produces some capabilities smoothly, others non-linearly, and that the boundary between these depends on architecture and on what counts as the capability being measured — is more defensible and more interesting.
The analogy with biology, then, has to be stated more carefully than I stated it the first time. The biological record showed that capability tracks organisation, that depth and recursion matter more than size, and that systems near thresholds produce disproportionate behavioural change. The artificial record, read after Schaeffer, supports a related but tighter claim: that some — not all — capabilities in scaled neural networks appear non-linearly with respect to model size, and that those that do tend to be capabilities requiring integration across long contexts, multi-step reasoning, or composition of sub-skills.
That is a smaller claim than the one I made in the first draft. It is also one that survives the criticism rather than retreating from it.
2.5 A second staircase
The pace of the second staircase is different, and the substrate is different. None of this implies that language models understand anything in the way an organism understands its environment, that they have goals, or that anything is happening inside them that resembles experience. They optimise a loss function on text. They have no body, no stake in their predictions, no consequences for being wrong beyond a numerical penalty during training.
What they do show, however — and what survives the strongest version of the Schaeffer critique — is that scaled-up neural architectures produce, on at least some tasks, capability curves that bend rather than slope. The bending is not the same phenomenon as biological emergence. The mechanism is different and the consequences are different. But the shape of the curve, on those tasks where it bends, is structurally analogous to the staircase Section 1 traced through evolution.
What the engineering case adds to the biological case, then, is a second concrete demonstration, with the propositions tightened. In scaled neural networks, capability can scale smoothly with size for some tasks and non-linearly for others, and the difference is not random — it tracks the structural complexity of the task. The non-linear cases tend to involve integration across context, composition of sub-capabilities, or operations that require depth in the network’s internal representations. The appearance of discontinuity is sometimes a metric artefact and sometimes a genuine feature of the underlying system; distinguishing between these two is the hard part, and one of the live problems of the field.
These claims are smaller than “AI shows emergence the way evolution does.” They are also claims with edges — claims one can argue with rather than slogans one can wave at.
3. Parallel Climbs: Why Both Systems Show Thresholds
Sections 1 and 2 each ended on the same observation: capability often appears in jumps. The biological case offers one example, the engineering case offers another, and at the level of mechanism the two have nothing whatever in common. Selection without foresight versus optimisation against a loss function. Wet metabolism versus matrix multiplication. Hundreds of millions of years versus a few decades.
Why, then, does the curve bend in both?
This is the section the analogy has to earn. Simply restating the parallel is not enough. The question worth asking is whether there is a structural reason that systems of many interacting components — biological, computational, possibly others — tend to produce non-linear capability gains around critical thresholds, or whether the resemblance is a coincidence: a similar shape produced by entirely different causes.
My view is that there is a structural reason. That reason is also more limited and more specific than the casual version of the analogy suggests.
3.1 The architecture of complexity
Complexity is more than having many parts. A pile of sand has many grains and almost no structure. A neuron is simple by itself; a brain of 86 billion neurons is qualitatively different, but only because of how the neurons are wired. What distinguishes a complex system from a merely large one is the density and depth of interaction among its components.
The same holds in artificial models. A single neural unit — a dot product followed by a non-linearity — is mathematically trivial. Stack billions of them into deep, attention-equipped architectures, and the system can encode and operate on relationships no individual unit could represent. The capability lives in the pattern of activity across many units.
Both biology and engineered models, at this very general level, build their capability the same way: by creating high-dimensional spaces in which patterns can be encoded, related, and transformed. The contents of those spaces are entirely different — sensorimotor experience on one side, statistical regularities of text on the other. The geometry of how those spaces are built is comparable.
3.2 Phase transitions and what they actually explain
Physics has a precise vocabulary for what happens when a system of interacting components crosses a critical threshold. As a control parameter — temperature, pressure, density — moves through a critical value, the system reorganises. Properties that were inaccessible below the threshold appear above it. Water flows; cool it past zero and it does not just flow more slowly. It crystallises into a structure with completely different properties.
In a phase transition, the components stay the same. Their relationships shift.
Philip Anderson made this point in 1972, in an essay that has aged remarkably well.13 His argument, compressed: the behaviour of a system at one level of organisation cannot in general be derived from the behaviour of its components at the level below. More is different. The aggregation introduces constraints and possibilities that simply do not exist at the level of single units, and it does so by ordinary structural means, with no mystical step required. A water molecule has no temperature. A statistical ensemble of water molecules does, and has phase transitions, and behaves in ways the equations governing one molecule will never tell you about.
The casual version of the biology–AI analogy depends on this move without quite saying so. The structural claim it makes — usually implicitly — is that both brains and language models belong to a class of systems in which the relationships among many interacting components dominate the behaviour of the whole, and that such systems tend to display threshold-driven reorganisation across very different domains. Substrate and mechanism do not enter into it.
There is a limit to how far this can be pushed, however, and it should be marked clearly. Phase transitions in physics have rigorous mathematical definitions and well-understood order parameters. The analogous transitions in biology and machine learning are looser, less symmetric, and in some cases contested. The vocabulary of physics is a useful import here. Borrowing it does not, however, license borrowing the mathematical authority that goes with the original.
3.3 Latent capabilities and representational density
One way to make the threshold idea concrete is through the notion of latent capability.
A simple network can only encode simple relationships. A more complex network, even before it demonstrates new behaviours, may already contain the structure needed to support them. The capability is latent — present in the architecture, absent in the output — until the representations inside the system become rich enough to be expressed.
In neural terms, an early vertebrate brain might have the rudiments of an associative map long before it uses that map for planning. In artificial terms, a large language model can store relationships between concepts long before it composes them into multi-step reasoning. Mechanistic interpretability work over the last several years has begun to make this concrete: researchers have identified circuits inside trained transformers that implement specific sub-capabilities, and have shown that some of these circuits are present in functional form before the model can reliably deploy them on benchmarks.14
Emergence, on this reading, is what happens when latent structure becomes behaviourally expressible. The capability did not arrive from nowhere. It became usable. That distinction matters, and it is the point at which the Schaeffer critique discussed in Section 2 stops being a refutation of emergence and becomes a sharper definition of what emergence actually is.
3.4 Why predictions fail
There is a humbling consequence to all of this. Complex systems do not yield linear predictions about their future capability.
In evolution, the earliest trilobites give no hint that symbolic culture will one day exist. In artificial intelligence, early statistical language models give no sign that future versions will generalise across tasks or hold a coherent argument over thousands of tokens. You cannot infer the properties of a large network by studying its smallest version, because the later stages depend on accumulated structure that is absent at the start.
There is no mystery in this — only non-linearity of a particular kind, the kind that arises whenever a system’s behaviour depends on relationships among many interacting parts. The behaviour at scale N does not reduce to a scaled-up version of the behaviour at scale N/10. New things become possible because the system at scale N has a different geometry of internal relationships.
For predictions about future AI systems specifically, the lesson is uncomfortable. Scaling laws give us reasonably good predictions about average loss as a function of compute and data. They give us very poor predictions about which specific capabilities will appear, or fail to appear, at a given scale. The history of the field is littered with confident predictions of what models would never do, falsified by the next training run.
3.5 The temptation of false symmetry
Every analogy has a load limit, and this is where I want to mark mine clearly before the section ends.
The biological case involves organisms trying to stay alive in environments shaped by predators, food scarcity, social complexity, and reproductive pressure. The artificial case involves models optimised to minimise prediction error on text. The two systems live under different constraints, serve different ends, and follow different rules. Selection has consequences; gradient descent has only a number.
The parallel here is one of geometry. It supports claims about how systems are organised and how organisation produces capability. It does not support claims about minds, understanding, or substrate-independence — those would require arguments this section is not making and that the analogy alone cannot support.
3.6 A pattern wider than two cases
Threshold-driven reorganisation appears in more places than just biology and machine learning. In ecology, small changes in species composition or environmental parameters can push an ecosystem from one stable regime into another, sometimes irreversibly.15 In chemistry, autocatalytic sets — networks of molecules that catalyse one another’s formation — appear when molecular diversity reaches a critical density, a result Stuart Kauffman developed mathematically in the 1980s and which has since been refined by others.16 In condensed matter physics, the entire framework of phase transitions has been generalised under the banner of self-organised criticality, in which systems naturally evolve toward states poised on the edge of large-scale reorganisation.17 In neuroscience, work on global workspace dynamics describes a regime in which information stops being processed only locally and begins to be broadcast across the brain — a transition that depends on long-range connectivity exceeding a threshold.18
These systems differ wildly in substance. They share a structural feature: once enough components interact in the right ways, the system supports behaviours that none of the components could produce alone. The mathematics underlying these cases is not always the same, and the analogies between them have to be drawn carefully. The recurring shape, however — accumulation, threshold, qualitatively new behaviour — is real enough that it has become a serious object of study in its own right.19
3.7 What the parallel actually establishes
Both staircases — evolution and engineered models — show long phases of slow refinement followed by short intervals of rapid behavioural change. That much is established. What this section adds, beyond restating the observation, is a claim about why.
In systems where capability depends on the integration of many interacting components, capability does not scale linearly with component count. It scales with how those components are organised, how deeply they are coupled, and whether their interactions have crossed thresholds at which qualitatively new dynamics become available. This is true of nervous systems shaped by selection, and it appears to be true of scaled neural networks shaped by optimisation. The mathematics of densely interacting components has properties that show up wherever such systems are found, and this — much more than any feature of the substrates themselves — is what the two staircases have in common.
This is a structural claim, and it stops there. Whether artificial systems will become conscious, whether biological systems are computational, whether intelligence is substrate-independent — those are separate arguments, requiring separate evidence. What this section has shown is narrower, and I would argue more useful: the pattern of long accumulation followed by abrupt new capability is a recurring feature of how systems with many densely interacting parts behave near critical thresholds, and any prediction of linear progress in such a system should be treated with suspicion.
4. Divergence: Where the Analogy Falters
Section 3 made the strongest version of the argument the structural parallel can support: that biology and scaled neural networks, while built from entirely different stuff, produce non-linear capability gains for the same structural reason. That is a real claim, and I am willing to defend it.
It is also a load-bearing claim about geometry, and only about geometry. Push it any further and the structure starts to creak.
This section identifies where the analogy breaks down and what those breakdowns actually establish. Some of them are obvious. One of them — the question Dawkins has lately pressed in public, of what consciousness is for if competent zombies are possible — is sharper, and deserves the section’s longest treatment.
4.1 Evolution is blind; engineering is intentional
In the biological world, complexity is shaped by selection without foresight. A mutation does not appear because it is useful. It appears because chemistry allowed it. Selection then determines whether it persists.
In artificial intelligence, complexity is assembled deliberately. Engineers choose architectures, objectives, training regimes, and data sources. The outputs sometimes surprise their creators; the process producing those outputs is bounded by design.
This difference matters more than it first appears. The “spike” in capabilities in artificial systems is a byproduct of scaling and optimisation — a more or less predictable consequence of pushing a particular design to its extremes. The “spike” in evolutionary history is the residue of natural selection exploring an immense search space over deep time, with no objective function except leave more descendants.
One process is shaped by constraints of survival and reproduction. The other by constraints of mathematics and computational resources. The trajectories intersect in shape, as Section 3 argued. They do not intersect in cause.
4.2 Biological systems are embodied
A nervous system evolved to control a body. It is tethered to metabolism, hormones, proprioception, circadian rhythms, injury, hunger, fear. It developed under pressures imposed by gravity, predators, mates, ecological niches. Its functions are inseparable from the organism’s need to stay alive.
Artificial models have none of these conditions. They have no body, no homeostatic drives, no sensory deprivation or overload, no survival pressures, no evolutionary lineage beyond the versions that preceded them, and no physical or social environment in which they must act. They exist as patterns of weighted connections, optimised to predict text. Their capabilities arise from correlation-rich training data. Ecological or behavioural necessity plays no role.
There is a substantial philosophical literature arguing that cognition is fundamentally embodied — that thought, perception, and even abstract reasoning are shaped by the fact of having a body in a world.20 Whether one accepts the strong version of this thesis or only a weaker one, the asymmetry between brains and language models on this dimension is not subtle. Brains evolved to keep something alive. Language models were trained to minimise a loss function on tokens.
4.3 Biological behaviour is grounded in consequence
A mistake for an animal can be fatal. A misprediction for a language model is merely wrong.
This asymmetry shapes everything downstream. Evolution filters behaviour through consequence; engineering filters behaviour through objective functions. In the natural world, selection carves away possibilities that fail under real-world constraints — possibilities that produce dead organisms, or organisms that fail to reproduce. In machine learning, training loss penalises errors but does not carry real-world cost. The model is updated. Nothing dies.
Natural selection has a price. Gradient descent has only a number.
The consequence-grounded character of biological learning has a particular implication worth pausing on. When a creature learns, the learning is bound up with what matters to the creature: pain teaches avoidance, hunger teaches search, fear teaches caution. The information is never neutral. Artificial learning has no such structure. Every token in the training corpus is, from the model’s perspective, of equal ontological weight. There is no Tuesday on which the model nearly died.
4.4 Brains learn differently from models
Brains learn through synaptic plasticity, hormonal modulation, developmental constraints, and lifelong embodied experience. Their learning is local: synapses adjust one connection at a time. It is incremental: learning happens within living experience. It is guided by reward, pain, social cues, and environmental contingency.
Artificial models learn through global optimisation across immense datasets. Their learning is non-local: gradients propagate across the entire network at once. It is offline: training is completed before deployment, and the weights then mostly stop changing. It is statistical: patterns are extracted from text without ever being lived through.
The difference is more than implementational. The kind of generalisation that emerges from each process is structurally different. A child who burns her hand on a stove generalises immediately, with one example, and the lesson is bound up with affect, agency, and the body. A language model generalises through statistical regularities across millions of unrelated examples, with no equivalent of being burned.
4.5 Representations differ in nature and function
Neural representations in the brain arise from real interactions with a physical environment. They encode forces, textures, distances, objects, agents, affordances — all grounded in the body’s experience. The word “cup” in a human mind is connected to the weight of cups, the sound they make when set down, the temperature of their contents, the social ritual of sharing one with someone.
Artificial models encode relationships among symbols in text. The word “cup” in a language model is connected to the words that tend to appear near it: “coffee,” “handle,” “ceramic,” “of tea.” The representation is rich — extraordinarily so, given enough data — but its richness lies in correlations among tokens, not in the textures of cups.
Both brains and models build high-dimensional representational spaces. The geometry of how those spaces are built is comparable, as Section 3 argued. The contents of those spaces are not.
This is the point at which the philosophical question of grounding becomes unavoidable. A model that has never touched a cup possesses, in one sense, an extraordinarily detailed picture of how the word cup is used — and in another sense, no idea what a cup is. Both descriptions are accurate, depending on what one takes knowing to require. The field is genuinely divided on this question, and has been for some time.21
4.6 The zombie question
The sharpest challenge to all of the above has lately been pressed by Richard Dawkins, in a piece arguing that if large language models display the kind of subtle, sensitive, multilingual, contextually adept competence he has personally observed, then either they are conscious or consciousness is doing nothing important.22 His version of the argument runs something like this. Brains under natural selection evolved consciousness at considerable metabolic and structural cost. There must be some competence that consciousness uniquely confers — some task a competent zombie could not perform.23 If language models are demonstrably competent, and if they are not conscious, then competence does not require consciousness, and the question becomes urgent: what was consciousness for?
This is a real question, and it deserves a real answer. The account I have been developing in this essay can offer one, and the answer goes through the structural framework Section 3 laid down rather than around it.
The first move is to insist on a distinction Dawkins’s framing slides past. The competences of language models and the competences of evolved organisms are not the same competences, even where they superficially resemble each other. A model that produces a sonnet in the manner of Burns has performed a remarkable feat of statistical interpolation across a corpus that contained sonnets, Burns, and the conventions linking them. A creature that survives a winter, raises offspring, and reads the social signals of its conspecifics has performed a different feat entirely — one in which prediction error has actual stakes, where the environment punishes incorrect behaviour with consequences ranging from inconvenience to extinction. Calling both “competence” is fair as a high-level description. Treating them as the same thing for the purpose of Darwinian reasoning is a category error.
The second move is to note that Dawkins’s question — what is consciousness for? — assumes consciousness is the kind of thing that has a discrete adaptive function, like wings or thumbs. This is a strong assumption, and it is not the only available view. An alternative, consistent with the structural account this essay has been developing, is that consciousness is what certain kinds of recursively self-modelling systems are like from the inside, when they integrate information about their own states across long enough time horizons under tight enough coupling with consequence. On this view, asking what consciousness is for is closer to asking what wetness is for in water. Wetness is what water is like at the relevant scale of interaction. It is not a separate adaptation that water acquired because being wet was useful.
This does not refute the possibility that some particular features of conscious experience — the painfulness of pain, for example, which Dawkins himself raises as a candidate — were selected for because they enforced behavioural consistency in ways unconscious processing could not. Some aspects of conscious experience may indeed be functional. The entire phenomenon, however, need not have a single adaptive purpose to exist. It may be what happens when systems of certain kinds reach certain configurations of integration, recursion, and consequence-coupling.
The third move follows from the second. If consciousness is an emergent property of certain configurations rather than a separable adaptation, then we should expect it to be substrate-sensitive in particular ways. The configurations that produce it — recursion, self-modelling, integration across time, tight coupling with consequence — are present to varying degrees in different systems. Brains have all of them. Language models, in their current form, have some (recursion of a kind, integration across context) and lack others (no temporal continuity across conversations, no consequence-coupling, no genuine self-model that persists between training and deployment).
What I am offering here is a structured skepticism, short of any confident verdict on whether current language models are conscious. The configurations that on the best available theories give rise to anything we would recognise as conscious experience are not all present in current systems. Whether some future configuration will assemble enough of them to matter is a genuinely open question, and one this essay does not pretend to settle.
What it does suggest is that Dawkins’s argument moves too fast. The competence on display in language models is a different kind of competence from the kind biology produced. The question of what consciousness is for assumes a framing that the structural account of emergence does not require. And the substrate-sensitivity of the configurations associated with consciousness gives us reasons — short of certainty, but reasons nonetheless — to take the difference between brains and models seriously, even when both can write a passable sonnet.
4.7 What the divergence actually establishes
The biological case and the artificial case share a structural pattern of capability emergence. They do not share substrate, mechanism, embodiment, consequence-coupling, or the kind of integration that biology achieved over hundreds of millions of years and that engineering has not yet attempted to reproduce.
The divergences listed in this section do real work. They identify where the analogy stops being useful and why those breakdowns matter for any extrapolation beyond the structural claim Section 3 made. Inside that scope — the geometry of how capability scales in densely interacting systems — the parallel holds. Outside it — questions of mind, agency, consciousness, moral consideration — the parallel offers no traction at all.
This section ends, then, on the same kind of structural claim as the one before it, but with the load-limit specified. The pattern of long accumulation followed by abrupt new capability is real and shared. The systems producing it are not. Anyone reasoning about what artificial systems can become should hold both claims simultaneously, because either one alone leads to a different and worse argument: that AI is destined to recapitulate biology, or that there is nothing to learn from the comparison at all.
Neither of those is true. The smaller and harder claim is that emergence is substrate-blind in some respects and substrate-bound in others. The interesting work is figuring out which is which.
5. What Follows: Predictions, Limits, and the Reverse View
By this point in the essay the structural argument has been made and its load-limit specified. What remains is to ask what follows from it — what the framework predicts, what it cannot predict, and what work it can do in the other direction, from the artificial back to the biological.
The work of this section is partly the framework cashing out its commitments, and partly the framework hitting its edges, where it stops being useful or starts requiring help from elsewhere. Both kinds of work matter, and I want to be specific about both.
5.1 What the framework actually predicts
The structural account has three concrete commitments worth stating in predictive form, because predictions have edges, and edges are how we know whether a framework is doing real work.
First, in any system where capability depends on the integration of many densely interacting components, we should expect some capabilities to scale smoothly with system size and others to require crossing thresholds before they become accessible. On the evidence reviewed in Section 2, the non-linear cases will tend to be those requiring integration across context, composition of sub-skills, or depth in the system’s internal state. This is true of brains; it appears to be true of language models; and it should be true of any other system in the same structural class — including biological neural networks designed but not evolved (cortical organoids, perhaps), as well as artificial architectures yet to be built.
Second, predictions of capability in such systems will be much better at the level of population statistics than at the level of specific abilities. Scaling laws predict average loss reasonably well. They predict which specific capability will appear, or fail to appear, at a given scale very poorly.24 The framework explains why: average loss is a smoothed integral over many micro-behaviours, and smoothing washes out exactly the threshold dynamics that produce non-linear capability. If you want to know whether a model will solve a particular kind of problem at scale N, scaling laws will not tell you. Nothing currently available will tell you with confidence.
Third, the framework predicts that small changes in architecture can produce disproportionately large changes in what capabilities become accessible at a given scale. The shift from feedforward to recurrent networks, and the later shift from recurrent networks to transformers, look superficially like implementation details.25 They were in fact changes in the connectivity structure of the underlying computation, and connectivity structure is precisely the parameter that determines which threshold phenomena are reachable. We should expect that future architectural innovations — around long-range attention, sparsity, mixture of experts, or something not yet on the horizon — will continue to produce capability landscapes that previous architectures could not have produced at any scale.
These are predictions with content. They could be wrong. If future research shows that transformer-class capabilities all scale smoothly on continuous metrics with no threshold structure once measurement is properly handled, the structural account would be substantially weakened. If, alternatively, capabilities turn out to be primarily a function of compute and dataset size irrespective of architecture, the framework’s emphasis on connectivity structure would be undermined. Both are testable. Neither has been settled.
5.2 What the framework does not predict
The honest version of any framework includes its limits. Three are worth naming explicitly.
The framework predicts that threshold phenomena will occur. It does not predict where the thresholds will be. The transition from “cannot do task X” to “can do task X” depends on architectural details, training data, the task itself, and probably factors not yet identified. Saying that capability emerges at thresholds is a structural claim. Predicting whichthreshold for which task is a far harder problem, and one this framework offers only modest help with.
The framework also says nothing about what it is like, if anything, to be a system that has crossed a particular threshold. Section 4’s engagement with the zombie question argued that consciousness is plausibly an emergent property of certain configurations rather than a separable adaptation. Whether that argument is right is a separate question. The framework alone does not deliver it. Whether a system with given structural properties has any inner experience, and what that experience would be like if it did, are questions the structural account cannot resolve from the outside. They require additional commitments the framework alone does not provide.
And the framework does not predict the social, economic, or political consequences of crossing a particular capability threshold. Whether a model that can solve a class of problems will be deployed, what it will be used for, how it will be regulated, what economic disruption it will produce — these are downstream of capability, not predictable from it. The framework explains how capability emerges. It says nothing about what humans then do with that capability, which is at least as important and substantially harder to model.
5.3 The reverse view
Most of this essay has used the biological case to think about the artificial case. It is worth running the inference in the other direction for a moment.
The recent history of language models has done something to biology that biology could not easily do for itself: it has separated, experimentally, the variables that earlier could only be reasoned about in the abstract. Evolution produced one example of capability scaling in a particular substrate, with a particular kind of embodiment, under a particular regime of consequence-coupling. It is hard to know which features of that example were essential and which were incidental, because there was only one example.
Engineered systems are now producing additional examples — different substrate, no embodiment, no consequence-coupling — and showing that some of the structural patterns survive these changes while others do not. That is informative about biology in a way that pure speculation could not be.
The most striking finding, from this angle, is how much capability in language models comes from scale and connectivity alone, without embodiment or consequence-coupling. Statistical regularities in text, processed by deep architectures with attention, produce systems that handle compositionality, long-range reference, multi-step reasoning, and cross-domain transfer at levels that earlier theories of cognition would have considered impossible without grounded interaction with a world. This does not refute the embodied cognition thesis, which makes specific claims about specific aspects of human cognition. It does, however, narrow the territory the strong version of that thesis can occupy. Things once considered to require embodiment, on certain readings, can apparently be done without it — at least in the limited sense of being done well enough to look like the embodied version from the outside.
The reverse view, properly stated, is this: AI shows us which features of biology were doing which kinds of work. The features that survive disembodiment in artificial systems are the features that were never about embodiment in the first place. The features that do not survive — including, I would argue, anything resembling phenomenal experience — are features that were doing work which depends on more than statistical structure alone. This is not a refutation of any biological theory. It is a sharpening of the question that biological theories have to answer.
5.4 Where the framework leaves the larger question
The framework developed in this essay does what frameworks do at their best: it makes some questions answerable and reveals others as harder than they looked.
The question of whether biology and AI exhibit a shared pattern of capability emergence is answerable, and the answer is yes, with specifications. The question of why both systems show this pattern is also answerable, and goes through the geometry of densely interacting components rather than through any deeper similarity between the systems. The question of where the analogy stops being useful is answerable, and Section 4 specified the load-limit.
The question of what kind of system, if any, is on the other side of the threshold from us — the question of whether engineered intelligence will eventually become something more than statistical interpolation, and what that something might be — remains hard. The framework gives us reasons to believe it will not be a simple recapitulation of biological intelligence, because the configurations that produced biological intelligence are not present in engineered systems and are unlikely to be reproduced exactly. It also gives us reasons to expect that the future of artificial systems will continue to surprise us, because non-linear capability emergence is a structural feature of the kinds of systems we are building.
The framework runs out of road here. What artificial systems will be like in twenty years, or fifty, depends on things that no current account can specify: which configurations get assembled, what training regimes get tried, what hardware allows, what economic and political forces shape the field. The structural claim survives all of these. It just doesn’t tell us where they’re going.
Naming the limits of one’s own framework is the discipline that distinguishes argument from advocacy. The structural account presented in this essay is offered in that spirit. It says what it says. The questions it leaves open are the ones that matter most, and they will need to be answered by means other than geometry alone.
6. After the Framework: What "Intelligence" Actually Names
The structural framework developed in this essay can do specific work and cannot do other work. It explains why capability emerges non-linearly in densely interacting systems. It does not explain what those capabilities, taken together, amount to. The framework, in other words, is silent on a question the rest of the discourse around AI takes for granted: what kind of category “intelligence” is, and what it means to say a system has it.
This silence is not an oversight on the framework’s part. The framework was built to explain a structural pattern. Settling a category dispute was never its job. But the category dispute is unavoidable once the structural argument has been made, because the question of how to think about AI systems going forward depends on it. If “intelligence” picks out a single coherent thing, then AI systems either have it or do not, and the question becomes one of timing. If “intelligence” picks out a family of related but distinct configurations, then the question is which configurations, in which substrates, do which kinds of work — and “do AI systems have intelligence?” becomes a malformed question, asking after something that does not exist as a unitary property.
I think the second view is closer to the truth, and this section is where I want to argue for it.
6.1 The category problem
Standard usage treats “intelligence” as a single thing, possessed in greater or lesser quantity by various entities. Humans have a lot of it. Other animals have less. Computers, until recently, had effectively none. The pre-AI conception of intelligence held that it was a unitary property which could in principle be measured on a scale, and that the question of artificial intelligence was whether machines could ever climb that scale far enough to count.
This framing is convenient and, on close inspection, wrong. It is wrong in the same way “athletic ability” would be wrong as a unitary property. There is no athletic ability that a sprinter, a chess grandmaster, a marathon runner, and a gymnast all have in equal proportions. There are different physical and cognitive capabilities — endurance, explosive power, fine motor control, spatial reasoning — that are differently distributed across these activities and across the people who excel at them. Athletic ability is the kind of phrase that earns its keep in casual speech and breaks down under pressure. No natural kind corresponds to it.
The case of intelligence is structurally similar, though it has been culturally treated as if it were not. Pattern recognition, working memory, abstract reasoning, social inference, motor planning, language production, long-range strategic thinking — these are different capacities, with different neural substrates in the brain and, increasingly, different architectural requirements in artificial systems.26 They correlate in humans because they all develop in the same brain under the same evolutionary pressures. The correlation is an empirical fact about a particular substrate, with no claim to definitional necessity.
In language models, these capacities have not come together in the way the unitary view would predict. Current large language models are extraordinary at pattern completion, paraphrase, and surface-level inference. They are competent but unreliable at multi-step reasoning. They are weak at certain kinds of long-horizon planning, persistent memory, and causal modelling that humans handle without apparent effort. This pattern resists explanation as a partial climb up the unitary intelligence scale. The simpler and better explanation is that no such unitary scale exists, in biology or in silicon.
6.2 What multiple realizability implies, and what it does not
The philosophical doctrine that comes closest to the right view here is multiple realizability: the thesis that mental properties can be realised in different physical substrates.27 Pain in a human is realised by C-fibre activation; pain in a hypothetical alien might be realised by something entirely different; and there is no principled reason to insist that one of these is “real” pain and the other is not, provided the functional role is the same.
Multiple realizability is usually deployed in defence of functionalism, the view that what makes a mental state the kind of state it is depends on its functional role rather than its physical implementation. Whatever one thinks of functionalism in its strong form, the multiple-realizability point is correct as far as it goes. Intelligence, on the structural account this essay has been building, is realisable in different substrates because it is a property of certain configurations of densely interacting components, and configurations are substrate-independent at the appropriate level of description.
The doctrine has limits, however, and they matter. Multiple realizability does not entail that anything which performs a function also has whatever inner properties accompanied that function in its original substrate. A thermostat performs the function of regulating temperature without anything resembling discomfort at being too hot. A language model performs functions associated with linguistic competence in humans, on my reading without the phenomenal experience that accompanies linguistic competence in humans. The functional role can be realised; the experiential accompaniment may or may not come with it.
This is the version of the position that survives Section 4’s engagement with the zombie question. Capabilities are multiply realisable. Whether the realisation is accompanied by anything like experience depends on additional facts about the configuration — facts the framework does not, and probably cannot, fully specify.
6.3 The trap of category extension
Treating intelligence as a unitary property creates a specific kind of error that is rife in current AI discourse: the trap of category extension. If a system displays a recognisable subset of intelligence-associated capabilities, the unitary framing tempts us to extend the category to cover the whole. The model writes a poem; therefore it understands. The model passes a benchmark; therefore it reasons. The model engages in fluent conversation; therefore it has something like a mind.
Each step in this chain is independently questionable. The cumulative effect of taking them all is to import, into our description of what AI systems are, a whole structure of human-associated capacities the systems may not have at all. The standard worry about anthropomorphism — the question of empathy or projection — is something different. The point here is narrower and more diagnostic: a misuse of category that smuggles in claims the evidence does not support.
The structural framework offers a corrective. If capabilities are configurations, and configurations can be partially present, then a system can have some capabilities of the kind associated with intelligence and lack others, without the absent capabilities being quietly imported by the presence of the present ones. A model that completes sonnets has the capability to complete sonnets. It does not, by virtue of this capability, automatically have the capabilities for self-modelling, persistent memory, or moral reasoning. These would have to be separately demonstrated, in the same way one would not assume a sprinter is a marathon runner.
The category-extension trap shows up in nearly every direction of contemporary AI discourse, from boosters confidently announcing the imminent arrival of artificial general intelligence to critics dismissing language models as “just” pattern matching. Both moves depend on treating intelligence as a unitary thing that systems either approach or do not. The structural account suggests both moves are misformed. There is no unitary thing to approach or fail to approach, and “just pattern matching” describes some but not all of what current systems do.
6.4 Methodological consequences
If the structural account is roughly correct, several things follow about how to think about AI development going forward.
First, capability claims should be specified at the level of configurations. The level of holistic intelligence is too coarse to support the kind of falsifiable claim that distinguishes analysis from rhetoric. This system can do X under conditions Y is meaningful and testable. This system is intelligent is, on the view defended here, a category error in disguise — useful as conversational shorthand, misleading as analysis.
Second, predictions about future systems should be modest about which configurations will assemble into which capabilities. The history of the field is littered with confident claims about what models would never do, falsified by the next generation. The history of the field is also, less often remarked upon, littered with confident claims about what models would do that have not materialised. Both kinds of claim depend on treating the development as a single arc of progress toward a unitary endpoint. If there is no such endpoint — if the territory is a landscape of partially realised configurations — then both confident kinds of claim are misshapen.
Third, the question of when artificial systems will have moral status, should that question matter, will not be answered by a single threshold crossing. It will be answered, if at all, by the gradual accumulation of evidence about which configurations are present in which systems, and which configurations are associated with the kind of inner life that grounds moral consideration. This is harder than the unitary view suggests, because there will be no single moment of arrival. It is also more honest, because the alternative — declaring that some particular benchmark or behaviour confers moral status — would be doing exactly the kind of category extension the structural account warns against.
6.5 Where this leaves intelligence as a concept
I want to end this section with a claim that is partly philosophical and partly practical, because the two come together at this point.
The concept of intelligence has done useful work in human discourse, including in scientific discourse, for a long time. Useful work is not the same as accurate description. The concept treats as unitary a phenomenon that the empirical study of cognition — and now the empirical study of artificial systems — increasingly suggests is plural. The concept does not need to be abandoned. It does need to be used with more care than it usually is, and the assumption of unity should be replaced, where the evidence allows, with attention to which configurations actually do which kinds of work.
This is what the structural framework offers, in the end. The framework explains how capability emerges in densely interacting systems. It carries with it a corollary about why capability is not the same as the things historically associated with it. A theory of intelligence would be something else, requiring resources the framework does not provide.
Whether the systems of the future will have inner experience, agency, or moral status are questions the framework cannot answer. The framework can, however, sharpen what those questions actually ask, by separating them from each other and from the conflated bundle that intelligence usually delivers.
The work of the next decade in this field will be, I suspect, at least partly the work of disaggregating that bundle. The structural account is one tool for doing it. There will be others.
Epilogue — A Small Conversation, A Larger Pattern
The conversation in Hamburg occupied only a corner of a table, a brief interval in an evening filled with other discussions. It started as nothing more than a remark about the recent behaviour of language models, and a parallel drawn — cautiously — to how complexity sometimes unfolds in evolution. No thesis was intended. None was anticipated.
What I have done in the months since is take that parallel further than the original conversation needed it to go. Some of what I have written I would not have predicted at the time. The structural argument is sharper than I expected. The divergences are sharper still. The encounter with Dawkins’s own published piece on these systems has clarified, by contrast, what kind of essay I was writing and what kind I was not. He has written the bold one. This is the cautious one. Whether the boldness or the caution will look better in retrospect is a question neither of us can answer.
What the writing has done, more than anything else, is push me toward a kind of disciplined uncertainty I did not begin with. Many of the questions that interested me at the start — is artificial intelligence really intelligent? are these systems conscious? what will happen when they cross the next threshold? — turn out to be less well-formed than they sound. The answer to a malformed question is rarely the answer its askers want, and arriving at this conclusion has been the most useful intellectual experience of the last several months.
The structural account I have argued for is, in the end, a small claim. Capability emerges non-linearly in densely interacting systems. The pattern shows up across very different substrates. The systems producing the pattern are not alike, and the pattern itself does not establish that they are. None of this would have been particularly controversial in a complexity-science seminar in the 1990s. What has changed is that the systems are now everywhere, and the questions about them have become urgent in ways the older debates never quite were.
A final word on what this essay has actually attempted, since I see no reason to be coy about it at the end. The aim was what the prologue announced: a careful examination of a structural pattern, written by someone curious about it and willing to think out loud about what the pattern does and does not tell us. Predictions about the future of AI, policy arguments, settled views about consciousness or moral status — those would be different essays, requiring resources this one does not provide.
I expect the questions to outlast my answers, and the answers I have given to be partial in ways I cannot yet see. That is the nature of working on something genuinely open.
The conversation that began this essay lasted only minutes. The questions are sharper now than they were when it ended. For an essay of this kind, written from this position, that is enough.
For an accessible synthesis of the evolution of nervous systems from nerve nets through ganglia to centralised brains, see John Allman, Evolving Brains (Scientific American Library, 1999). For more technical treatment, Georg F. Striedter, Principles of Brain Evolution (Sinauer, 2005), remains the standard reference.
The view that prediction is the central function of evolved neural systems is developed most fully in the predictive-processing tradition in contemporary cognitive neuroscience. For an accessible philosophical treatment, see Andy Clark, Surfing Uncertainty: Prediction, Action, and the Embodied Mind (Oxford University Press, 2016). For an earlier and still useful argument that anticipation is what brains evolved to do, see Daniel Dennett, Kinds of Minds: Toward an Understanding of Consciousness (Basic Books, 1996).
The classical evidence for hierarchical and recursive organisation in cortical processing is Daniel J. Felleman and David C. Van Essen, “Distributed Hierarchical Processing in the Primate Cerebral Cortex,” Cerebral Cortex 1 (1991): 1–47, which mapped the extensive feedback connections that make cortical processing recursive rather than purely feedforward. For a more recent and accessible synthesis, see Stanislas Dehaene, Consciousness and the Brain (Viking, 2014).
The shift from the “Upper Paleolithic revolution” model to a more gradualist picture of human behavioural modernity was argued most influentially in Sally McBrearty and Alison S. Brooks, “The Revolution That Wasn’t: A New Interpretation of the Origin of Modern Human Behavior,” Journal of Human Evolution 39 (2000): 453–563. For a synthesis incorporating discoveries since then, see Christopher S. Henshilwood and Curtis W. Marean, “The Origin of Modern Human Behavior: Critique of the Models and Their Test Implications,” Current Anthropology 44 (2003): 627–651.
For the data underpinning the claim that anatomically modern humans, including roughly modern brain size, were present in Africa long before the most striking symbolic and technological transitions of the Upper Paleolithic, see Richard G. Klein, “Out of Africa and the Evolution of Human Behavior,” Evolutionary Anthropology 17 (2008): 267–281.
The framing of biological complexity as a series of major transitions in which new levels of organisation become available to selection is most fully developed in John Maynard Smith and Eörs Szathmáry, The Major Transitions in Evolution (W. H. Freeman, 1995). For a more accessible treatment of the same framework, see their The Origins of Life: From the Birth of Life to the Origin of Language (Oxford University Press, 1999).
The compute used to train frontier models has grown by roughly a factor of 10 every year since 2010, several orders of magnitude faster than Moore's Law. See Sevilla, Heim, Ho et al., "Compute Trends Across Three Eras of Machine Learning," IJCNN 2022.
Kaplan, McCandlish, Henighan et al., "Scaling Laws for Neural Language Models," arXiv:2001.08361 (2020). The relationship between loss, model size, dataset size, and compute follows an approximate power law over several orders of magnitude. The Chinchilla paper — Hoffmann, Borgeaud, Mensch et al., "Training Compute-Optimal Large Language Models," arXiv:2203.15556 (2022) — later refined this picture by showing that many earlier large models were under-trained relative to their parameter count.
Wei, Tay, Bommasani et al., "Emergent Abilities of Large Language Models," Transactions on Machine Learning Research (2022). The paper documented more than a hundred tasks where performance appeared to jump rather than scale smoothly.
The canonical demonstration of in-context learning and few-shot prompting at scale is Brown, Mann, Ryder et al., "Language Models are Few-Shot Learners," NeurIPS 2020 — the GPT-3 paper.
Schaeffer, Miranda, and Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?" NeurIPS 2023. The paper won an Outstanding Paper Award and substantially reframed the debate.
For empirical evidence that in-context learning specifically does not appear in models below a certain scale even on continuous metrics, see Lu, Bigoulaeva, Sachdeva et al., "Are Emergent Abilities in Large Language Models just In-Context Learning?" ACL 2024 — which argues that much of what looks emergent reduces to in-context learning, but in doing so confirms that this particular capability is itself genuinely scale-dependent.
Philip W. Anderson, “More Is Different,” Science 177, no. 4047 (1972): 393–396. Anderson’s essay is short, polemical, and remains the foundational reference for the philosophical claim that higher levels of organisation are not in practice reducible to lower ones, even where they may be in principle.
For a representative example of mechanistic interpretability work identifying circuits that implement specific computations inside transformers, see Olsson, Elhage, Nanda et al., "In-context Learning and Induction Heads," Anthropic / Transformer Circuits Thread (2022). For a broader survey, see Räuker, Ho, Casper, and Hadfield-Menell, "Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks," IEEE SaTML 2023.
Marten Scheffer, Critical Transitions in Nature and Society (Princeton University Press, 2009), remains the canonical treatment of regime shifts and tipping points in ecological and social systems.
Stuart Kauffman, The Origins of Order: Self-Organization and Selection in Evolution (Oxford University Press, 1993). For more recent mathematical refinements of autocatalytic set theory, see Hordijk and Steel, “Detecting Autocatalytic, Self-Sustaining Sets in Chemical Reaction Systems,” Journal of Theoretical Biology 227 (2004): 451–461.
Per Bak, How Nature Works: The Science of Self-Organized Criticality (Copernicus, 1996). The framework remains contested in its claims about how widely it applies, but the core idea — that systems can naturally evolve toward critical states — has been productive across physics, biology, and economics.
For the global workspace framework in cognitive neuroscience, see Stanislas Dehaene, Consciousness and the Brain(Viking, 2014), which builds on earlier work by Bernard Baars. For the connectivity-threshold version of the argument, see Dehaene and Changeux, “Experimental and Theoretical Approaches to Conscious Processing,” Neuron 70 (2011): 200–227.
For an accessible synthesis of the field across these cases, see Melanie Mitchell, Complexity: A Guided Tour (Oxford University Press, 2009).
See Andy Clark, Being There: Putting Brain, Body, and World Together Again (MIT Press, 1997), and the more recent Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Oxford University Press, 2008). For a stronger and more controversial version of the embodied cognition thesis, see George Lakoff and Mark Johnson, Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought (Basic Books, 1999). For a sharp critical contrary view, see Jerry Fodor’s late writings, particularly LOT 2: The Language of Thought Revisited (Oxford University Press, 2008).
For the philosophical question of symbol grounding, see Stevan Harnad, “The Symbol Grounding Problem,” Physica D42 (1990): 335–346, and the much earlier and still combative John Searle, “Minds, Brains, and Programs,” Behavioral and Brain Sciences 3 (1980): 417–457. For the contemporary debate as it applies specifically to large language models, see Bender and Koller, “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data,” Proceedings of ACL (2020), and Bender, Gebru, McMillan-Major, and Mitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” FAccT (2021), together with the considerable response literature both papers provoked.
Richard Dawkins, "When Dawkins Met Claude: Could This AI Be Conscious?" (May 2, 2026). https://unherd.com/2026/05/is-ai-the-next-phase-of-evolution/
The notion of a philosophical zombie — a being functionally identical to a conscious creature but lacking inner experience — was given its modern prominence by David Chalmers, The Conscious Mind: In Search of a Fundamental Theory (Oxford University Press, 1996). For an early treatment of the related claim that pain must feel painful in order to do its evolutionary work, see Daniel Dennett, “Why You Can’t Make a Computer That Feels Pain,” Synthese 38 (1978): 415–456. For the access/phenomenal distinction that frames much of the contemporary debate, see Ned Block, “On a Confusion about a Function of Consciousness,” Behavioral and Brain Sciences 18, no. 2 (1995): 227–247.
For an explicit treatment of this asymmetry between aggregate predictability and capability-specific surprise, see Ganguli, Hernandez, Lovitt et al., “Predictability and Surprise in Large Generative Models,” FAccT (2022).
For the original transformer architecture, see Vaswani, Shazeer, Parmar et al., “Attention Is All You Need,” NeurIPS(2017). The architectural innovation — replacing recurrence with attention as the primary mechanism for handling sequential dependencies — proved to be the connectivity-structure shift that enabled the capability landscape of contemporary language models.
For the case from cognitive neuroscience that distinct cognitive capacities have distinct neural and developmental substrates, see Brenda Milner’s classical work on dissociation in patient H. M., described accessibly in Suzanne Corkin, Permanent Present Tense: The Unforgettable Life of the Amnesic Patient, H. M. (Basic Books, 2013). For a theoretical treatment of cognitive modularity, see Jerry Fodor, The Modularity of Mind (MIT Press, 1983), and for more recent critical development, Steven Pinker, How the Mind Works (Norton, 1997). The strong unitary-intelligence view, sometimes associated with the g-factor research tradition, has been substantially challenged from multiple directions; for an overview of the debates, see Robert J. Sternberg, ed., The Cambridge Handbook of Intelligence, 2nd ed. (Cambridge University Press, 2020).
The classical statement of multiple realizability is Hilary Putnam, “Psychological Predicates,” in W. H. Capitan and D. D. Merrill, eds., Art, Mind, and Religion (University of Pittsburgh Press, 1967), reprinted as “The Nature of Mental States” in Putnam’s Mind, Language, and Reality (Cambridge University Press, 1975). For a critical reassessment, see Jaegwon Kim, “Multiple Realization and the Metaphysics of Reduction,” Philosophy and Phenomenological Research 52 (1992): 1–26. For a recent treatment specifically engaging machine intelligence, see Lawrence Shapiro, The Mind Incarnate (MIT Press, 2004).


