Guest Posts

Unfortunately, the deceptiveness of the term “AI” is to a large extent deliberate. The fact of the matter is that this subject has escaped the control of academic researchers, having been for all intents and purposes hijacked by the Tech industry, which soon saw business opportunities in chatbot parlor tricks. The industry’s problem was that by 2022, revenues from tech platforms (FaceBook, Twitter, Google ads, etc.) were no longer growing at the spectacular rates that their leaders felt was the special privilege of their futuristic enterprises. They were in danger of becoming normal companies with normal growth expectations ¹. This prospect outraged the peculiar techno-messianic sensibilities that seem endemic to many in the industry’s leadership class. They settled on AI as the basis for the next phase of their industry’s evolution. In so doing, they began to impress a corrupting influence on machine learning research that is analogizable to the corruption of climate science perpetrated by the Oil industry, or to the earlier corruption of cancer research by the Tobacco industry.

The AI future that Tech leaders imagined is a strange admixture of technical naiveté, utopian ambition, and naked greed. On the evidence of current LLM capabilities for natural language processing (NLP), they persuaded themselves that they could bring about a new type of AI called “Artificial General Intelligence” that would constitute a transformative, disruptive technology, analogous to the steam engine or the telegraph. This they could accomplish by means of very large investments (totalling $3-5T by 2030) in data center construction, in acquisition of computing hardware (GPUs), in securing large power contracts required to run that hardware, and in extensive hires of data science and computer engineering personnel.

This AGI technology would become an engine of disruption, changing every industry or business that it touched. In the Utopic version of this story, it becomes a driver of a sort of “post-scarcity” world in which human resource competition and conflict become things of the past. In the more self-interest-grounded C-Suite discussions of the profitability of AI, a darker story is told, in which business leaders of other industries are offered AI tools that enable them to reduce their labor costs (i.e. to replace much of their work forces with AI tools). This process would re-direct large fractions of the resulting labor cost savings back to suppliers of AI services, thus assuring those firms of the abnormal profitability growth which they regard as their birthright.

What Is Wrong With This Picture?

Set aside the pious self-congratulatory naivetè of the Utopian story, and the obvious self-interested amorality of the business plan. From a practical point of view, what we should ask ourselves is this: Is there any reason to believe that this vision of the future is at all possible?

That’s another rhetorical question. The answer is that for good and sufficient reasons, this plan is going to fail, badly, and with negative consequences for Tech firms, their customers, the global economy, and the U.S. and other governments. Let us review reasons, and consequences:

(1) There Is No AGI Down This Technological Path

For the central driver of a story designed to mobilize trillions of dollars in investments, the term “AGI” is defined with maddening vagueness. But it is generally agreed that a fundamental aspect of AGI is computational reason. And in fact, researchers working in the AI industry lard their technical terminology with allusions to reason, including “reasoning models”, “reasoning tokens”, “chain-of-thought systems” and other such constructions. By such means they have persuaded themselves that they are on the path to solve (or perhaps have already solved) one of the thorniest problems in science: the modeling of human reason.

They have, of course, done nothing of the sort. In Parts 3-4 of this series, I discussed the wrong-headedness of conflating learning with reasoning. I gave a fairly detailed discussion of the type of scientific approach that one might try to connect modern machine learning to reasoning. This set of reflections has the benefit of illustrating the fact that reasoning qualitatively transcends learning in essential ways: recall that reasoning features the discontinuity of “Aha!”, which learning, as a continuous process, cannot emulate. The notion that one can simply train a machine learning system until it “learns to reason” (sometimes referred to as “emergence”) is scientific nonsense of the type characterized by Wolfgang Pauli as “not even wrong”.

Since all current AI is comprised of learning systems, none of it can learn to reason, even in principle. While AGI might be possible in principle, and we may see some version someday, it will certainly not be on a foundation of current DL-based technologies. Which means that the target motivating the colossal current investments in AI doesn’t even exist. That’s Bad News Message Number 1 for the AI enterprise.

(2) AI Hallucinations Are Never Going Away

By now most people are familiar with, or at least have heard about “AI Hallucinations”. AI famously gives dangerously bad financial and medical advice, incorrectly instructs firms’ customers about company policy, writes legal briefs that cites non-existent case law, screws up mathematical reasoning with unflappable didactic aplomb, writes wrong and unusable computer code, and otherwise reliably produces nonsense at a sustained rate whenever asked to produce output. There exists a subject now called “prompt engineering” in which domain experts use their expertise to detect such incorrect output, and tune or steer their AI prompts in order to coax chatbots into producing improved output. Only by such means is it possible to get actual useful output from AI systems.

The Tech industry’s “AGI” narrative holds, among other things, that such hallucinations are minor failings, which will in any event be addressed through more computing power, more training data, and larger models. In effect they believe that through hyperscaling, they can train models to stop hallucinating.

There is not a shred of quality evidence to suggest that this is true, and considerable reason to believe that hallucinations are an ineradicable part of the output of modern LLMs. They arise, in my view, from a very brittle and imperfect model of the probabilistic distribution over language sequences (sentences, paragraphs, and so on) that the models learn in training. The imperfection of the models is in fact structural: it is brought about by the token embedding process shared by all NLP methods. This process endows the “space” of language sequences with an improper notion of proximity that assimilates nonsense sentences to sensible ones, to a degree not justified by the actual sentence distribution sampled by the training data.

Note that this is a different problem from the impossibility of “learning to reason”. What one would hope of a DL-based LLM is that even though it doesn’t reason, it can generate sentences that are at least consistent with the sorts of sentences that it encountered in training. But that is not the nature of many AI hallucinations: they actually contravene the training distribution, because their broken internal model of that distribution places nonsense responses “closer” to sensible ones than they actually are, for technical reasons having to do with the geometry of the embedding process.

What this means is that AI hallucinations are in all likelihood structural to LLMs. They cannot be eliminated by training with more data, because more data would require more model capacity, which in turn would create more opportunities for embedding to find ways to situate nonsense geometrically near sense. And therefore, any “AGI” that is marketed by the industry is certain to be mentally ill at birth. This is Bad News Message Number 2.

(3) Hyperscaling, “Emergence”, and Model Inefficiency

One of the funniest intellectual failures driven by the corruption of machine learning by the Tech industry is practitioners’ faith in the phenomenon of “emergence” of general intelligence, which, they believe, is achieved by scaling model capacity (measured in billions of adjustable parameters) to the point that a model “suddenly” starts to produce reasonable responses to prompts. There is even the notion that some kind of fundamental law of computing has been discovered, relating the size of a training corpus of text to the required model capacity required to achieve this “emergence” of intelligence. One scales linearly with the other: double the size of the training data and you must double the model capacity to achieve “emergence”. This is in fact the “scaling law” at the root of the industry’s mad drive to build out datacenters, buy GPU hardware, secure power contracts sufficient to power medium-sized cities, and hire data science talent, which we now call hyperscaling.

The problem is that “emergence” is bullshit. The scaling law between training data size and model capacity is a fact of deep learning that has been true of all DL systems since the earliest examples of their application. It is as true of an image classifier recursive neural net (RNN) as it is of a transformer LLM. There is no qualitative difference between the two cases.

The qualitative difference that does apply pertains to the data being modeled: the complexity of the distribution of human language sentences is vastly greater than that of the distribution of natural images, or of weather states, or of materials properties. That is, the modeling of human natural language is an enormously more ambitious subject to tackle than any other problem that has been addressed using DL methods. For this reason, vastly more data is required to get a grip on the distribution than has been the case for any other DL modeling problem. An image classifier needs about 30,000 labeled images to achieve state-of-the-art performance. An NLP transformer needs tens of billions of words to achieve even moderately acceptable performance. And because the data requirements are higher, so are the model capacity requirements. You have to hyperscale the model size in order to get to the point where an NLP transformer doesn’t suck anymore. Hilariously, this “point of sucking less” is what practitioners of the subject refer to as “emergence” of intelligence (and even, in their more reckless moments, of AGI). It is nothing of the sort, unless you think that scaling an image classifier’s model capacity to the point that you can train it to classify images is also “emergence”.

While this model capacity scaling is a reasonable tradeoff for essentially all other DL applications, it is madly, unaffordably expensive when applied to language modeling. Unfortunately there are not many research efforts underway to attempt to find alternative modeling methods that break the scaling, because the AI train as currently constituted has an irresistible momentum conferred upon it by the mad level of investment that has been summoned up by the industry. The level of groupthink on this, even among academic scientists, is astounding, as well as depressing.

So, this is Bad News Message Number 3: unaffordable hyperscaling is baked into the methodology underlying AI chatbottery, and there is negligible research effort dedicated to circumventing it.

On “AI Winters”

Here’s the summary of the summary: the Tech industry’s target, AGI, is not achievable even in principle using current technology. And a subsidiary goal, the taming of the stubborn problem of AI hallucinations, is also not on the table. This being the case, the insane build-out of AI infrastructure is to no purpose, or at least to no purpose articulated by the AI industry. There is no pot of gold marked “AGI” at the end of the hyperscaling rainbow, or even a small purse marked “no hallucinations”. But it would take an infinite amount of data, compute, electrical power, effort, and capital to get there and find out. This cannot possibly end well.

We should try to ask ourselves the “How Does This End” question now, so as to be prepared to recognize the end when it happens. As it happens the subject of AI has an instructive history antedating the post-2007 developments that gave rise to modern “AI”. That history points to a certain cyclic pattern of research and development, culminating in “cataclysmic” (to its academic practitioners) collapses that are known as “AI Winters”.

As usual with history, it is difficult to divide up events into neat periodizations. Nonetheless, from the widest possible perspective, there appear to have been two prominent cycles of development and disappointment, culminating in AI Winters. The first Winter is generally held to have set in around 1973-1974, when research funding agencies in the U.S. and the U.K. concluded that the promise of research in machine translation, speech understanding, and “perceptrons” (single-layer neural networks) had been overhyped and was unlikely to deliver anything of value. Millions of dollars of funding was cancelled, leading to a massive contraction of the field, and to the end of many young (untenured) scientific careers.

By the 1980s, interest in AI had revived, carried largely by the advent of “Expert Systems”, which embodied hand-encoded knowledge according to some new knowledge representation schemes. New enthusiasm among researchers also led to a fair amount of media hype concerning the prospects for the new AI. Government funding followed suit, with the Japanese government’s 1981 announcement of its “Fifth Generation Computing” project. The U.S. government responded with DARPA’s Strategic Computing Initiative (SCI) in 1983.

But as early as 1984, in an article coining the term “AI Winter”, researchers Roger Schank and Marvin Minsky had started sounding the alarm about the growing AI hype, recognizing an echo of circumstances that had resulted in the disappointment and funding cutbacks of 1973-1974. As summarized in the Wikipedia article, Schank and Minski “… described a chain reaction, similar to a ”nuclear winter”, that would begin with pessimism in the AI community, followed by pessimism in the press, followed by a severe cutback in funding, followed by the end of serious research. Three years later the billion-dollar AI industry began to collapse.” By 1987, DARPA had wound up the SCI, and Japan did the same with the Fifth Generation project in 1991. Both were judged to be wasteful disappointments. By the 1990s, essentially all development of expert systems ceased, and the term “AI” itself began to seem somewhat disreputable among scientists and science funders. AI Winter II had set in.

Winters, and Bubbles

There are some useful lessons for our current moment in these events. In particular, it seems that Schank and Minski were prescient in their warning, and perceived a clear pattern to AI research and funding leading to the AI Winter Cycle. The phases of that cycle are:

1. Technical progress

2. Excitement

3. Media hype

4. Investment based on hype

5. Disappointment of investor hopes

6. Withdrawal of investment

7. Winter

When I first began thinking about writing a series of essays on AI (mid-summer 2025), it appeared to me that the hype around modern AI was vastly over-wrought, but it was successfully bringing in unprecedented levels of investment ($300B-$400B annualized investment in 2025, anticipated to rise going forward). That is, if, as I suspected, this was an AI Winter cycle, we we at about Phase (4). There were some differences in the historical pattern, the principal one being that the funders in this cycle are private investors to a far greater extent than government, but I thought that there were enough correspondences to suggest the same cresting wave pattern, culminating in an AI Winter. I couldn’t be sure of when the climax (Phase 7) might arrive, however. And very few of my colleagues were willing to entertain the idea that another Winter was coming.

Under current circumstances, with private investors taking the role previously occupied by Government funding agencies, the concept of an AI Winter necessarily becomes linked to that of an AI Bubble. This was also not a common position to take in Summer 2025. At that stage, the people suggesting that there might be a financial bubble connected to AI in the offing were definitely a minority², and it was certainly not a respectable position to take.

As of this writing (January 2026), we appear to have fast-forwarded to Phase 5, and are beginning to see signs of Phase 6. The phrase “AI Bubble” is no longer in bad odor, but rather has featured prominently in mainstream press reporting, including in The New York Times, The FT, the WSJ, Bloomberg, Barron’s and many other sources of both general and financial news. Oracle’s stock dove into the toilet on investor fears of the unsustainability of its datacenter buildout plans. NVidia and Meta’s datacenter financing deals are being compared to the practices that led to the 2001 implosion of Enron.

And, suddenly, people are noticing that no AI company is profitable, and that nobody knows a path to AI profitability. Ed Zitron estimates that OpenAI’s real revenue is a fraction of its inference costs, and therefore cannot even begin to amortize the GPT model family’s enormous training costs. A similar story applies to Anthropic. In a thus far fruitless search for profitability, both of those companies have taken to poaching the business of the downstream vendors who repackage and resell their services, a sure sign of an business ecosystem heading for collapse.

Suddenly, it seems as if we are already past the threshold of Phase 6-7 of the AI Winter Cycle, and this time it comes with an extra helping of Financial Shipwreck. Seven AI companies now make up 30% of the value of the S&P 500, and represent essentially all the year’s growth in the index. U.S. GDP and annual GDP growth from AI are both now in the 1%-1.5% range, which is a crazy level of risk exposure to an industry with a weak grasp on its own business.

If, as now seems very likely, investors start demanding that AI start paying its own way now, instead of waiting for their scheduled 2030 arrival at the Sunny AGI Uplands, the AI world is suddenly going to look like a very different place, because AI can’t pay its own way at current revenue levels. Free ChatGPT/Claude/Gemini is going away this year, I feel pretty sure. I don’t think that even $200/seat/month accounts bring in enough money to pay for the inference costs that they inflict on their suppliers, so at a minimum all those subscription rates need to be raised sharply just to cover their costs, and that is the way to shrink the hell out of that market.

The U.S. government has been making large AI infrastructure and model development investments predicated on partnering with the AI industry, which would still be largely in charge of pretraining models to be fine-tuned by government scientific (and other) customers. That model is not going to work at all if those AI companies start selling off their data centers and power contracts to Crypto miners for pennies on the dollar, and start firing employees as if suffering from a case of corporate dysentery. There would be nobody on the partner side answering email.

Also, the U.S. government cannot bail out the AI industry. There will no doubt be calls for a new Federal program to rescue NVidia, Microsoft, Meta, OpenAI, Anthropic etc., adverting to the “strategic” value that those firms have to U.S. national security. But even if this were not the wretched self-serving horseshit that it clearly is, the U.S. can’t afford to make up the losses of trillions of dollars in mis-allocated capital that these companies have already set on fire. The U.S. government’s financial position is much more precarious than it was in 2009, and in any event nobody believes that the Feds would know what to do with NVidia, or Anthropic, or Oracle, if they somehow wound up with majority shares of those firms.

I think that it’s coming apart now. Not in a few months. It is happening under our eyes, now.

Then What?

The last 8 paragraphs are, I believe, the most derivative and least informed of any that I have written in this series. I am not a financial expert in any sense of the phrase. Most of what I have written in these essays I can defend based on my fairly extensive domain knowledge in the subjects of machine learning and statistics and computational science. I do wish to write about what I think is going to happen, however, and in this moment it appears that to do so I need to connect the things that I do understand well to financial matters that I feel much less certain about. Please do not make any investment decisions based on what I write.

Instead of babbling on about business economics, I’d like to return to the subject of machine learning (and be shot of the term “AI”), and try to understand what residual value will remain from this strange, two-decade adventure, once the now-inevitable Winter brings on its now-inevitable retrenchment.

If you’ve been patient enough to read through these posts, then you probably know that I am an admirer of many of the scientific accomplishments that came out of the 2007 deep learning revolution. Those accomplishments are real. We can now distill weather patterns output by computationally-expensive numerical weather prediction (NWP) codes into computationally-cheap DL models that can reliably forecast weather up to 2 weeks in the future as well as those NWP codes can. We can predict protein structure, which, take my word for it, is a huge advance in biology. We can make very good guesses at chemical and material structures that lead to desirable properties, and that can actually be manufactured, without those chemicals or materials ever having existed before. These are all Nobel-caliber advances, some of which have in fact been rewarded with Nobel prizes. We will still have them, and obtain others like them, after the dust settles on the burst AI bubble.

The AI companies themselves will, incredibly, be forced to bequest to the public some things of great value. Most of the models at the cores of their chatbots, including their trained model parameters, are publicly available and open-source licensed at Huggingface, a public code and data repository dedicated to LLMs. The firms that built those models do not release their training data, but they did train those models on enormous datasets, at enormous expense in compute, electrical power, and capital, and the resulting models can actually be useful when run stand-alone (if one has reasonable expectations). I think that Marc has made this sort of point several times in comments to previous posts, and I basically agree with it.

Moreover, as I wrote in Part 6, I also believe that the LLM adventure has taught us something interesting that we did not previously know: it is in fact possible to model human language on a computer. This is one of the hardest modeling problems ever attempted, and the history of NLP is littered with failed attempts to crack it. The 2017 introduction of the transformer architecture succeeded in demonstrating that the crazy idiomatic quirkiness of human expression is not beyond capture by computers that we can build, now. That’s a fantastic discovery, because previous experience suggested that capturing the distribution of human language on a computer might be one of those things that are possible in principle, but that one could bankrupt the world and still fail if one actually attempted to do it in practice. Now we know that it can be done by nearly bankrupting the world. This is progress! What we need now is to figure out alternative NLP strategies that do not suffer from the crazy scalings of DL methods. I think that there are some real possibilities for this.

For me, an important silver lining in the LLM debacle is that academic research on machine learning may finally recover from the learned helplessness with which it has faced its exclusion from the development of state-of-the-art language models. The eye-watering cost of the infrastucture required to train and operate such models has meant that academic scientist have been sidelined from model development and pretraining. Also (I’m a bit ashamed to say) a certain passivity and vulnerability to groupthink has prevented them from looking for good alternatives to simply becoming customers of the AI industry. Now that one can forsee that the colossal investments by the industry will soon cease to exercise their mesmerizing effects on researchers, perhaps we can get serious about the science of machine learning, and about the uses of machine learning in science. And about recovering our agency in our own scientific endeavor.

These numbers are ridiculous. It seems clear that those projections for cost and power generation are totally unrealistic, unlikely to ever come to pass, and damaging to the economy to the limited extent that they do come to pass.

The damage is not limited to economics, however. This level of investment also moves minds. The Tech industry consensus—that hyperscaling of large language models (LLMs) will bring about Artificial General Intelligence (AGI)—has attracted a critical mass of mindshare from government and from academia. This is certainly a bit of an own-goal for academic institutions, because by buying in to this consensus academic scientists have entirely shut themselves out of LLM development: no university or consortium of universities can afford to build out even a small fraction of the computational infrastructure required to train or operate these models at these scales. And even the U.S. government is struggling to stay in the game.

The interesting question here is why has this consensus developed across the field. And the “why” that I’d like to discuss in not a business strategy “why”, but rather a technical “why”. As a matter of science and technology, what is the peculiarity of LLMs that fuels the drive to hyperscale? After all, we have had “AI” deep learning (DL) methods since 2007. Hyperscaling, however, is peculiar to the subset of DL methods that power LLMs, and has only really gotten underway since 2022. What changed?

Overparametrization

In a sense, the seeds of hyperscaling have been an implicit part of DL ever since the DL revolution began in 2007. The origin of the phenomenon is the practice of model overparametrization.

In normal parametrization, one has a model with some parameters, which may be thought of as knobs that one can dial to arbitrary values. The knobs control the predictions that the model makes of the data. One determines the values of those knobs by adjusting them so as to minimize the misfit between the model and the data. Normally, one has many fewer parameters than data samples.

Overparametrization is the practice of endowing a model with many more parameters than one has data samples to model. It has not always been regarded as a useful practice: before the rise of DL, it was in very bad odor, for good and sufficient reasons. The problem with overparametrization is that it endows a model with too much flexibility and too little predictive power. After all, the purpose of training a model on some data is to obtain a means of making predictions and decisions given new data samples (recall the Statistical Learning Catechism, from Part 1 of this series). But if I have 10,000 data samples, and I naively train a 100,000-parameter model on that data, what will inevitably happen is that the model will predict the training data perfectly, while giving wildly wrong predictions about new data not included in the training set. In statistical parlance, the model will overfit the data.

Basically, as a matter of algebra, you only need 10,000 parameters to “solve” for the 10,000 training data samples. This is sometimes called “memorizing the training data”, i.e. learning to predict those 10,000 training samples perfectly. Despite its perfect fit to the training data, that 10,000-parameter model will give terrible predictions of any new data not included in the training set. And a 100,000-parameter model is—or ought to be—much worse.

The point is that the true process underlying the data always has more smoothness than the data itself, because the data has additional random noise. Any computational model of that process should not be allowed to play connect-the-dots with the data, because that would be tantamount to chasing the random noise. But that is exactly what a naive overparametrized model does. And because it learns to chase random noise in the training data, it cannot properly predict the smooth behavior of the underlying process. We say of such models that they have poor generalization properties, which just means that no interpolation based on such a model can be trusted ¹.

The problem is that in the example of the 100,000-parameter model trained on 10,000 data samples (not an atypical case for a convolutional neural net trained on a corpus of images) there is a roughly 90,000-dimensional parameter subspace of solutions that exactly memorize the training data. That subspace has a thickened neighborhood (in the full 100,000-dimensional space) of solutions that don’t quite memorize the data. In that neighborhood exist some values of the parameters that endow the model with reasonable generalization properties. Those values are the target.

Regularization and Stochastic Gradient Descent

The way to improve the generalization properties of an overparametrized model is to regularize it. Regularization is a very general term for a time-honored, broad family of techniques that deprive an overparametrized model of much of its freedom. By regularizing such a model, we in effect impose some kind of smoothness properties on it that interferes with its ability to play connect-the-dots with the data, thus making it a more realistic representation of the smooth underlying data-generating process. The regularization technique adopted in DL methods is peculiar to the discipline, because it was in DL that extreme overparametrization was first seriously considered.

DL methods find the target parameters through the technique of stochastic gradient descent (SGD). The “gradient descent” means that there is some defined cost function of the parameters (such as some average of the prediction errors over the training set, for example), and that cost function is minimized (“descent”) by following it through the parameter space along the steepest descent direction.

Here’s an analogy to help understand the training process: think of the cost function as a continental landscape. You know that the landscape has a long, broad valley somewhere in the middle of the continent, and you are targeting a destination somewhere near the valley floor. Not the exact floor, because that would correspond to the noise-chasing data-memorization solutions, but somewhere in that neighborhood. To find the right neighborhood, you need to locate the valley floor.

A reasonable first approach is to always follow the local descending direction, until you find the lowest point. But that won’t work. The problem is that that while the valley walls slope down on average (by about a foot per mile, say), the landscape is highly textured, with lots of local structure such as boulders, hills, small and twisty valleys, mountain passes, and so on. The local direction of steepest descent might actually lead you away from the continental valley floor, because you are descending some random twisty valley, or because the direction that you need to follow leads up some mountain pass. The landscape texture in the analogy is the structure added to the cost function by those 10,000 data samples, each one of which acts as a sort of structure-adding knob, contributing the boulders, hills, etc.

You need some strategy to ignore the local texture and find the large-scale average descent direction. If only by squinting you could blur the landscape, only resolving structure to within a mile instead of to a few feet, you might be able to see the average descent towards the valley floor, because the boulders etc. would be fuzzed away.

That is where the “stochastic” part of SGD comes in. By randomly swapping subsamples (“minibatches”) of data at every step of gradient descent, the optimization code’s view of the landscape texture is blurred, because the landscape of fine-scale features changes with every minibatch. As a consequence, the small-scale structure ceases to obscure the large-scale path of descent. Even better, the 90,000-dimensional memorization submanifold—the exact bottom of the valley, corresponding to the family of connect-the-dots solutions—also gets fuzzed out somewhat, and harder to find. The optimization algorithm dwells for longer in the liminal region where good generalization may be found.

Such a solution is identified by computing the cost function on held-out “test” data, which is not used in the gradient computation. At first, both the training and test costs decline in tandem. At some point in the training, however, the test cost stops dropping and either flattens out or starts rising again, while the training cost continues to drop. This is the stopping point of the training loop: a parameter solution has been located with reasonable generalization properties, because the test cost is OK, and there is no point in continuing, because the optimization routine is about to find the data-memorization submanifold (the lowest point of the valley floor).

Overparametrization and Data Types

That was rather a long explanation of the practice of overparametrization in DL. I need it here, because I need to discuss the costs and benefits of the practice, and how those vary depending on the type of data and decisions that one’s model is required to traffic in.

The combination of overparametrization and SGD is key to the success of all DL methods. It should be clear, however, that from a strictly statistical point of view, overparametrization is inefficient. Remember that according to the Statistical Learning Catechism, an important part of the job of any such model is to learn the distribution of the data. That distribution may be quite complex, and many parameters may be required to approximate it. But in a principled statistical model, the required number of parameters should not depend on the number of samples used to determine those parameters. Instead, the number of parameters should be fixed, and determined only by the structure of the data-generating process. As the number of data samples increases, the precision with which that fixed parameter set is determined should get better and better ².

DL models do not have this property. Instead, there is a notion of model capacity, which corresponds to the size of the parametrization, and which is required to grow as the training data grows, in order to maintain the necessary overparametrization. The more data one trains on, the greater the required model capacity. This is the sense in which they are inefficient.

This inefficiency is not, in and of itself, a bad thing. As we have seen, DL methods have been incredibly successful at addressing problems such as image classification, protein folding, materials design, etc. that were previously regarded as intractable. When used in such applications, they could be trained at very reasonable cost in computation and energy. It was definitely not necessary to occupy a 10,000-GPU datacenter for months to train a high-quality image classifier, for example. On the other hand, this is exactly the computational scale required to train an LLM on corpuses of text. Why the difference?

The answer, in my opinion, is that the distribution of text tokens from human language is vastly more complex than any other data distribution that has ever been modeled using DL methods. Billions of tokens of text are required in order to begin to capture the regularities and peculiarities and sheer chaos of human expression. This is larger, by several orders of magnitude, than, for example, the number of photographic images required to train an image classifier. And this being the case, the model capacity of LLMs has of necessity grown to unprecedented size (hundreds of billions, or even trillions of parameters), in order to maintain overparametrization.

But training models of this sort of capacity requires vastly more compute and energy. Hence, hyperscaling.

On “Emergence”

The account that LLM practitioners give of their capacity issues is simultaneously humorous and frustrating. They do not appear to view those issues in the light of the story that I have just presented. They see no continuity with the capacity requirements characteristic of all DL. Instead, they seem to have persuaded themselves that the relationship between data corpus size (in billions of tokens) and LLM capacity (in billions of parameters) is sui generis, and amounts to some kind of natural law of computing wherein intelligent behavior “emerges” naturally in consequence of the growth of the complexity of the model. The linear relationship between corpus size and model size has even been dignified with a scientific name: the “Chinchilla Scaling Law”, named after the Chichilla model discussed in this paperfrom Google Brain. The idea that there are such “scaling laws” for LLMs was originally suggested in this paper from OpenAI.

This is funny, in a grotesque sort of way. What is really happening here is that, as with all DL methods, without sufficient model capacity the model performance sucks. Then, as one grows the model capacity by adding parametrized computational elements, at some point the model becomes trainable, and begins to suck less. This “point of sucking less” is what LLM developers, in a brilliant stroke of marketing, have chosen to call “emergence”. And the advent of emergence is purportedly predicted by the scaling laws, which are felt to hold some deep significance about the nature of intelligence, and whose discovery rivals that of Newton’s law of gravitation in scientific importance.

This is, of course, pure horseshit. It is indicative of the corrupted state of the science of machine learning under the influence of the business imperatives of the Tech industry. A certain answer is required to be true by the industry: AGI is here, or, at least, nigh, which we know because of these “scientific discoveries”. The scaling law points the way to AGI: we will get there through larger models, more compute, higher CapEx. “Science says” that Hyperscaling will bring about AGI. And AGI will be so great that markets will emerge to absorb the $5T in investment required to get there.

This is a ridiculous story, since there is no credible scientific support for any of these claims. It’s a wild adventure in Madness of Crowds capitalism that cannot possibly end well.

What Have We Really Gained With LLMs?

The advent of the transformer architecture in 2017 represented a true breakthrough in natural language processing (NLP), one worthy of the highest scientific praise. Not because of any bullshit about AGI or “emergence” or scaling laws, but because transformers have demonstrated that natural language can be modeled at all. It was not clear prior to the transformer that this was true, or at least that it might ever be possible to model the distribution of human language text on a realistic computer.

There are two types of scientific impossibilities. The first type is that of things that are simply straight-up impossible so far as our current scientific understanding is concerned (faster-than-light travel, for example). The second type consists of things that, while not impossible in principle, are impossible in practice, because you if you tried to accomplish them you would bankrupt the World in the attempt and still fail (human spacefaring travel to nearby stars is in this category).

Some “type 2 impossible” problems have to do with whether something is or is not amenable to computational modeling. There are many problems that have known scientific principles, but which we do not expect to ever be able to model on a real computer. For example, first-principles numerical modeling of strongly turbulent fluid flow is just one such “type 2 impossible” problem, because no computer we can imagine building would have the capacity to resolve all the physical length scales required for such a simulation.

As to NLP, it has always been clear that language has patterns, and that text corpuses issue from some complicated distribution. It was not clear until 2017 however that computational modeling of that distribution was not “type 2 impossibile”. Now we know that it is in fact possible. That is a real scientific accomplishment.

Unfortunately, the realization of such models using transformers comes at a very daunting cost in compute, power, and capital. This is especially true now that the subject has become entangled with a totally unrealistic quest for a goal (AGI, or, at least, artificial reasoning) that is almost certainly not available along the current technological path. This cannot go on forever. And as economists are fond of saying, if something cannot go on forever, it will stop.

Are There Alternatives To Transformers?

Transformer-based LLMs have exhausted their usefulness as research tools. It is past time to start looking for alternatives.

A small number of leading researchers are already unhitching their wagons from the LLM caravan. Yann LeCun, one of the most celebrated DL scientists, made a splash earlier this year by parting ways with Meta to start up a research effort on an approach called “World Modeling”. Song-Chun Zhu, a renown expert in computer vision, has returned to China after 20 years of research in the U.S., to pursue a set of approaches that he characterizes as “Small Data, Big Task” (by implication, DL approaches have it the other way around).

I think it is promising that some very bright folks are breaking away from transformers (and possibly from DL altogether). As a matter of personal outlook, I am not sanguine about purely computational approaches such as World Models and SDBT superseding LLMs. These are, as I understand matters, still computational learning approaches. As I’ve written in past posts, I feel that the most serious defect of DL approaches is that they place little value on reasoning about data distributions, while focusing too much attention on models. In effect, in light of the Statistical Learning Catechism that I’ve expounded upon, they are computational models attempting to do a statistical model’s job, and as a consequence they make inefficient use of their data. That inefficiency is tolerable for most data types of interest, but unaffordable for human language learning. And while learning human language distributions is not sufficient to model human reason, I believe that without learning human language distributions there is no chance of any kind of emulation of the human faculty of reason.

So what I’d like to see, and something I am attempting to do in my own research, is to try to introduce principled statistical models to perform language-learning tasks. I can’t tell yet whether the things that I am trying will work as well as a transformer, or even be competitive with one. However, I do think that it ought to be possible in principle to construct a model whose capacity need not scale with data corpus size, and whose parameters are fixed in number by the structure of the human language distribution that it models. Those parameters should be determined more accurately the more training data the model is fed.

To accomplish something like this, not only must one break away from transformers: it is necessary to give up on DL methods altogether, because the overparametrization required by all DL-based methods makes language learning unaffordable.

This type of approach is in a sense less ambitious than what LeCun and Zhu are attempting to bring about, because it is still strictly concerned with language modeling. But this seems to me a good way to leverage the one solid, useful result that has emerged from modern NLP: natural language can be modeled on a real computer. Now we just need to find a way to do it that doesn’t bankrupt the World.

I could go on, but it gets boring. Finding examples in the media of AI going off the rails in embarrassing ways is easier than finding inebriated people on Chicago streets at noon on Saint Patrick’s day. Just try a web search on “AI hallucinations”. The AI hallucination is a daily phenomenon, affecting programmers attempting to speed up their coding, scientists looking for fast ways to generate or clean up papers and proposals, and anyone in need of text that must precisely reflect some legal constraints.

AI models are also notoriously bad at mathematical reasoning, making elementary arithmetic mistakes as well as serious mathematical errors. I have prompted ChatGPT to perform a certain standard physics derivation ¹ twice now, at a distance of several months, and both times I have obtained careless stupidity that no undergraduate would be capable of producing, presented with professorial polish and total didactic aplomb.

It’s fun to point and laugh, but sometimes it is no joking matter. People have received bad, even dangerous medical advice from ChatGPT. There is a high-profile effort underway to use AI to “democratize” financial advice, which is seemingly innocent of the associated risks. There’s a pending patent for “AI Traffic Control” which is exactly as terrifyingly stupid an idea as it sounds. In fact, we are living through a moment in which the Tech industry is desperately attempting to propose AI for any application offering any prospect of profitability–no firm today makes any money on AI services–so it is not surprising to see such risks minimized or hidden altogether.

To observers of this discipline, the hallucination phenomenon is a very serious problem, and is another reason to question whether “Artificial General Intelligence” (AGI) is even a remote possibility on our current technological path. Certainly it would seem that if the hallucination issue is not understood and corrected somehow, any prospective AGI will babble hilariously, and possibly dangerously, some unpredictable fraction of the time.

The Tech industry consensus on hallucinations, however, is some combination of (a) hallucinations are not really a problem, and (b) more pretraining of improved (i.e. “larger”) models with more data at higher cost in compute and power will make them go away, as AGI finally emerges. I have had conversations with people who really believe in (a) or (b), and at least one person I spoke with appeared to somehow hold both views simultaneously.

View (a) is obviously not even worth discussing, given the high stakes involved in many AI applications. What I’d like to discuss today is view (b): can we really expect bigger models trained at higher expense with more data to do away with the stubbornly persistent phenomenon of AI hallucinations?

In order to address this question, we need to understand where these hallucinations come from. For that, we first need to review what it is that LLMs do.

What Does an LLM Do?

It is helpful to recall the basic definition of statistical learning–the subject that encompasses all of AI–at this point. Here is how all this stuff works:

1. Take a set of data, and infer an approximation to the statistical distribution from which the data was sampled;
   • In the case of LLMs, the data consists of hundreds of billions of words of text
2. At the same time, optimize some decision-choosing rule in a space of such decisions, exploiting the learned distribution;
   • With LLMs, a decision could be response to a text prompt, or a judgment about whether the text expresses positive or negative sentiment, or a translation to another language, etc.
3. Now when presented with a new set of data samples, produce the decisions appropriate to each one, using facts about the data distribution and about the decision optimality criteria inferred in parts 1. and 2.

The reason that I keep bringing these up is that I find this model-agnostic view of the machine learning enterprise extremely clarifying, and helpful in directing attention towards what matters and away from irrelevant aspects of model design.

We should apply the above catechism to what LLMs do. Data from natural language text consists of sequences of words, interspersed with punctuation. LLMs learn features of the distributions over such sequences that allow them to probabilistically predict what the next response word should be, given a prompt and any response words previously supplied.

So, for example, suppose your prompt to be completed by the LLM is “Bob was nervous about his presentation to the board, despite his preparation the night before.” and the LLM completes it with “He had practiced by reading his slides and timing what he said while each one was displayed.” The LLM starts with the prompt as its context, and uses the learned distribution to compute the probability distribution of the next word. From that distribution, it samples (i.e. decides on) the word “He”. It then appends “He” to the prompt to form a new context, and calculates a new distribution for the next word. It turns out that “had” is pretty high in the probability list, and gets selected. The context is now “Bob was nervous about his presentation to the board, despite his preparation the night before. He had”. The LLM repeats the process, and probabilistically samples the word “practiced” from the new distribution. And so on.

No kidding. This is all that is going on. Next response token prediction based on the prompt and all previous response tokens. That is the entire trick. Neat, eh?

Back to hallucinations: there are two places to look for their origin: the approximation to the data distribution, and the next-token decisions founded on that distribution. Let’s take them in order.

Approximating the Distribution of Human Language

The reason that text comprising hundreds of billions of words are required to train an LLM is that the statistical regularities of human language are extremely complex, and not easy to capture in a principled statistical model.

Just scroll your eyes up and down this essay briefly, and then imagine figuring out the rules by which the words are juxtaposed, without being detained by trivialities such as meaning. There are rules: you rarely see the same word repeated immediately (e.g. “immediately immediately”) which is clearly a rule. There are grammar rules, and context rules. Certain clusters of words recur together in certain types of text and not in others: you will find pairings of “octopus” and “cephalopod” within a few hundred words of each other in texts from works on marine biology, but pairings such as “octopus” and “mortgage” are probably very rare. In fact, the occurrence of “octopus” in a page probably means that the probability of encountering “mortgage” within the next 1000 words is considerably reduced from the average rate of occurrence of that word, while the occurrence probability of “shark” is likely enhanced. And so on. How would one go about describing these patterns?

The approach used in natural language processing (NLP) since time immemorial is to begin by breaking down the text into tokens, then describe the text as a sequence of such tokens. This tokenizationis a subtle and arcane art. You might think that it would be logical to break things down into words, numerals, punctuation, etc. While not wrong, this approach is very inefficient. The problem is that the English language (say) has about 500,000 words, which is a huge vocabulary for an LLM to manage. Vocabulary size is a critical parameter to be managed in this game, because the larger the vocabulary, the larger and more expensive the model.

On the other hand, breaking things down into individual letters is also a bad idea. While the vocabulary size is now much smaller (less than 50 for English), the token sequences are much longer, and the patterns much harder to find. The patterns are at the word level, not the letter level. It’s just that there are so many damn words!

The secret sauce is to notice that most of that half-million words are extremely rare. Studies of natural language have shown that knowing 10,000 words in any language allows one to understand 99% of texts in that language. In English, that would be 2% of the full vocabulary. Moreover, the rare words can be built up out of smaller word pieces. Identifying an optimal set of word pieces, most of which are full words in their own right, is the name of the game here. Algorithms exist that can represent all English text using 30,000 to 50,000 tokens, which is a considerable savings in vocabulary size. So tokenization is (largely) a solved problem ².

Embedding

The next thing that essentially every NLP method does with its tokenized text is a process called embedding: Each token is mapped into a vector space of dimension about 1000 (basically, each token gets described by a list of 1000 numbers) endowed with a notion of distance between points similar to the notion of distance between points in 3-dimensions. At this point, all operations on a sequence of tokens become operations on such lists of numbers. So when a transformer operates on a sequence of tokens (a sentence or a set of sentences, including previously-generated text) that sequence gets embedded in a very high-dimensional space: for example, the prompt above (“Bob was nervous…”) consists of about 17 tokens, so it is mapped to a point in an approximately 17,000-dimensional space consisting of 17 copies of the original 1000-dimensional embedding space.

I want to draw attention to embedding for several reasons: one is that it is an essentially universal practice in NLP, preceding the invention of transformers by many years. It turns out to be much easier to model probability distributions by operating on lists of numbers than by operating directly on sequences of discrete tokens sampled from a finite-dimensional set (the vocabulary). So researchers have defaulted to the embedding strategy.

Another reason to emphasize embedding is that when transformers train embedding parameters, they appears to do something magical: the resulting embeddings cluster together words and word fragments with similar meanings or functions, in well-separated clusters in the embedding space. You can see examples of this at Kevin Gimpel’s Bert Embedding Visualization Page, where you will see visualized in 2-dimensions clusters of suffixes, of verbs with similar meanings, of types of enclosed spaces, etc. It is one of those weird effect that persuades some people that LLMs are in fact acquiring a sense of the meaning of words.

The final reason to draw attention to embedding is this: embedding almost certainly poison the approximation to the distribution of language tokens. The embedding step destroys information about that distribution. The reason is that the original native space of token sequences is entirely innocent of vector spaces, and contains no geometric notion of spatial proximity such as arises in the embedding space. That spatial proximity structure is entirely imposed by the NLP architecture. And it almost certainly gives rise to improper notions of proximity between sentences that are sensible (i.e. have a high probability of occurrence) and other sentences that are nonsense (i.e. have a low probability of occurrence).

As an example of improper proximity, consider these two brief sentences: “My dog is fast”, and “My sparrow is fast.” Both are well-formed, grammatically correct, and obey applicable syntactic and semantic rules. The difference is that the first sentence ought to be ascribed a much higher probability than the second one, because nobody actually owns a sparrow.

As embedded points, however, the two sentences are quite similar: a dog and a sparrow are both animals, and hence live in some proximity in the embedding space. Furthermore, dogs are pets, and while sparrows are not pets, they are birds, and some birds are pets. There are enough ways to draw proximity connections in the embedding space to make the second sentence seem plausible in the distribution approximation, despite the fact that it is, obviously, a hallucination.

So embedding is, in my opinion, one of the origins of hallucinations. It is the reason that the approximation made by LLMs to the distribution of language is so brittle. There is nonsense lurking “near” sense in the embedded sequences, because in their native space (token sequences) there was no notion of geometric “nearness”: that property of relative proximity is an artifact of the model.

And if true, this is very bad news for AGI, because it means that hallucinations are a structural feature of all LLMs. They all embed sequences. So you cannot just train your way out of insanity and into “General Intelligence”, because all those new tokens will have exactly the same problem of spurious proximity. The distribution will be corrupted from the outset. It may be that the most likely responses could appear sane, but insane responses will always lurk nearby, waiting to be sampled by the LLM.

The Tyranny of Sampling

I’ve been referring to the process of “sampling” tokens above, and I should say a bit more about that, because while we have seen the origin of hallucinations in a broken estimate of the distribution of language sequences (part 1 of statistical learning) we need to see how the problem is aggravated by an LLM’s response decisions (part 2).

LLMs are often referred to as “generative” models (the “G” in “GPT”). What this means is that their output is, in a sense, random rather than deterministic. They compute probability distributions over the next token, and then exploit that distribution to decide what the identity of the next token should be. They generally do this by choosing the token randomly, with a higher probability of selection ascribed to tokens judged more likely by the calculation.

You might well ask: “Why not simply select the next token by choosing the one with the highest probability?”

This is occasionally tried. It is a strategy called “greedy sampling”. It is very efficient. Unfortunately, it is also a recipe for disaster, a ticket to hallucination pandemonium.

The problem is this: what one really wants is the most likely extended response, to the prompt, according to the learned distribution over language. This might consist of hundreds, or thousands of tokens. The distribution, while imperfectly learned, appears to at least get the most likely extended response right, in the sense that it is the one least likely to contain a hallucination.

Unfortunately, sampling the most likely next-token at every stage does not produce the most likely extended response. This can be a surprise at first, but from a mathematical standpoint it is not surprising at all. The probability of the 17th next-token conditional on the prompt and the previous 16 next-tokens can be very different from the probability of the 17th next-token conditional on the prompt and on entire remaining most-likely response (tokens 1-16 and 18-1000, say). Choosing the most likely token at every stage can, and usually does, lead the LLM into crazy rabbit-holes.

So instead, one attempts to let the probability distribution do its thing by allowing it to somehow sample the next-token distribution. This is better, but more expensive. In principle, what one ought to do is sample the 1000-token response many times (10,000 times, say) and choose the most frequently-occurring response. That strategy would probably abate a good deal of the hallucination phenomenon. Unfortunately, it would be totally unaffordable in inference computation cost, as well as quite slow. So intermediate strategies are adopted, restricting the next-token distribution to the top 90% of candidates, and looking along a tree to the next and next-next tokens for each one of these top tokens (the so-called “beam search”). This is better, but still not great for finding the top 1000-token response.

You might call this the Tyranny of Sampling: one must somehow sample from an LLM in order to defend its output from the worst hallucinatory offenses. But if you try to do the right thing, the computational cost will destroy the usefulness of the method. Rock, hard place.

Hallucinations Are Structural

Here’s the bottom line: Hallucinations are a structural feature of LLMs, produced by a corrupted model of the probability distribution over language sequences learned in training. The corruption is due to embedding, which is a ubiquitous feature of LLMs.

The only available hallucination abatement strategy is some form of generative sampling, which means accepting the unsettling fact that LLMs cannot produce the same output twice to the same prompt. And even accepting this non-determinism as a cost of doing business, the sampling strategy that cleans up the problem to a maximal extent is totally unaffordable. Unsatisfactory look-ahead strategies are better than nothing, but they still let a lot of nonsense through.

There is no hallucination abatement strategy that begins with more token data and larger models. That’s just not a thing, despite what the Tech industry would like to believe (and would certainly like investors to believe). More tokens and larger models likely aggravate the embedding problem, because there will be more improper proximities discovered in the embedding space.

And note that “larger” models are not “more clever” models. This discipline has not produced radical innovations to the transformer architecture since its invention, or at least none that have led to any breakthroughs comparable to what was wrought by the transformer’s first introduction in 2017. A “larger” model simply means “more parameters”³, not new mechanisms that make the model more clever. Given the argument that I make here, I very much doubt that any new cleverness could be built into a transformer that could eliminate the hallucinationatory mechanisms baked into its structure at its most fundamental level.

All of which is to say this: LLM-based “AGI” will be mentally ill at birth.

I want to narrow down these considerations, from AGI (a term for which no accepted scientific definition exists) to artificial reason, which is at least amenable to some specific discussion. What I would like is a model of what we mean by the term “reason” that is specific and detailed, to the point of being amenable, at least in theory, to implementation as software. Such a model would at least get us away from the territory of bullshit claims such as “self-organization” and “emergence” of AGI.

Last week, I discussed human reason in the context of what sort of traces it might leave in natural language text, to examine the plausibility of claims that reasoning states can be recovered from large text corpuses. I pointed out that our own reason rests on a foundation of subrational processes which almost certainly leave no such trace in text. Cognitive scientists have only the vaguest notions of how those processes work, and they can certainly not exhibit any models for them that are sufficiently specific to be represented as software. So trying to build a principled “bottom-up” model that mimics how reason emerges in a human mind is probably hopeless, at least for now.

What is left, then, is a “top-down” approach. What I mean by that is that we must work at an abstract level rather than at a mechanistic one. We must state what we mean by “reason” in general terms, in a way that we cannot directly show to be connected to the mechanisms of human reasoning, but which is motivated by the structure of reasoned thought. Also, we would like a model that can be expressed as mathematically as possible, because the point here is to come up with something that we could imagine translating into computer code.

Oddly enough, we already have one aspect of human cognition that can be represented this way: learning. We have seen that there is a subject called statistical learning, wherein by some method one learns an approximation to the statistical distribution from which some dataset was sampled, and one concomitantly learns to structure reasonable decisions based on that distribution. I’ve been a little vague about how this works, but it is a process that can be represented quite generally by the kind of model that I have in mind here.

So one possible approach (certainly not the only one!) is to take that representation of learning and generalize it, to represent reasoning. This approach has two advantages: it allows us to get a free ride on the existing model, which appears to work for learning; and, it allows us to connect and contrast “reasoning” to “learning”, so that we can begin to see what the relationship might be between the two.

A Cast Of Characters

This is all very abstract, and it will be helpful to provide concrete examples of reasoners (or alleged reasoners) to consult as we go along. I have three such examples for you:

• The astrophysicists who were trying to puzzle out the nature of Gamma-Ray Bursts (GRBs) between 1973 and 1998. The GRB phenomenon consists of bursts of gamma rays (duh!) that arrive at the Earth from random directions on the sky, never repeating. When they were discovered, and for the quarter-century that followed, their nature remained mysterious, because they seemed unconnected to any other astronomical phenomenon. The available data consisted of gamma-ray “light curves” (time traces of gamma-ray intensity), spectra (distributions of gamma-ray energies in the burst), event durations (fractions of a second to hours), and locations in the sky. The latter were only known very inaccurately: the so-called “error boxes”, regions of the sky from which the events might have arrived, were very large by astronomical standards, many degrees across, because it is difficult to create direction-resolving instruments for photons at gamma-ray energies. We will use the story of how the mystery of GRBs was solved to illustrate an aspect of our model of reasoning.
• A DIY home electrician (name redacted to protect the guilty) attempting to install a light fixture into an electrical box. He is following very standard procedures, using techniques, tools, and materials that he has trained to use and understand, and is moderately skilled. However, for some unknown reason, the fixture installation is failing, because of a persistent short-circuit that only manifests itself when the fixture is finally secured to the wall, and the circuit breaker is turned back on. When he turns on the circuit breaker with the fixture not secured to the wall, there is no short circuit, and the fixture works correctly. He is trying to figure out why by inspecting wire nut connections and checking for crimped wires. We will use this story to illustrate another aspect of the our model of reasoning.
• An LLM undergoing training, or a trained LLM making new inferences. It doesn’t reason: it’s just along for the ride.

Bayesian Updating As A Model Of Learning

Let’s get started with learning.

We can exhibit an abstract model of learning using Bayesian statistical theory. I’ll describe how this works without writing down any equations (there aren’t that many equations, and you will find them in that draft article if you care about that sort of thing). There are two elements to consider: a parameterized model, and an evidence stream.

The role of the evidence stream is to provide new information to be assimilated. The evidence is presented sequentially, one discrete piece at a time. It comes from a fixed set of possible pieces of evidence. There may be infinitely-many such pieces, but they are related by some structural relationships.

Examples of such evidence streams are GRB light curves, spectra, durations, and arrival directions; or the results of the DIY electrician’s inspections for faulty wire connections or crimped wires; or pages of text presented to the LLM in training.

The role of the parameterized model is to provide a description of the structure of the evidence. “Parameterized” simply means that the provided description is controlled by a set of numbers (the parameters) that act as control knobs on the model. Twist those knobs, and the model’s description of the evidence structure changes. There may be a half-dozen such knobs, or there may be billions, depending on the model and the evidence. The model is fixed, but we may set the knobs any way we choose.

The model might contain statements such as “the source of the GRB is a neutron star in our galaxy” and the corresponding knobs could be the star’s spin rate and magnetic field intensity, and its distance from Earth; or the model could contain the statement “one of the wires is getting crimped against the box’s mounting strap” and the corresponding knob would be the identity of the offending wire; or the model might be the LLM itself, and the corresponding knobs would be the billions of parameters that must be set in training.

We do not initially know which settings of the knobs provide the highest-fidelity description of the evidence structure, i.e. which settings are most predictive of the evidence. However,once we start viewing evidence, we have a procedure for weighting the knob settings. “Weighting” means that we may view some settings to which we have ascribed higher weights as being more likely than other settings with lower weights, because the higher-weight settings provide better descriptions of the evidence.

This weighting procedure is called Bayesian updating. As the model views each new piece of evidence, this (fairly simple) mathematical procedure describes how the weights shift among the knob settings. Generally speaking, a single piece of evidence produces a relatively small adjustment of the weights. Over time, as evidence accumulates, what may happen in the ideal case is that a small set of knob settings will hog most of the weight while remaining settings will have essentially zero weight, and we will conclude that those highly weighted settings are “preferred” by the evidence (in the sense that they give the most satisfactory predictions of the evidence).

That, in a nutshell, is our model of learning.

When Learning Stalls

One problem with statistical learning is that the happy circumstance where the weights contract to a small set of knob settings can be difficult to obtain. There are two possible problems with it:

1. The evidence may not shed enough light on the model. In this case, we would say that the evidence is not informative about the model.
2. The model may not be sufficiently descriptive of the evidence. In this case, we would say that the model is not explanatory of the evidence.

If either of these circumstances holds, the Bayesian updating process will stall, and the weights will not decisively concentrate on a winning set of knob settings.

In the case of GRB astronomy, a consensus developed in the 1980s that there was a Case (1) problem: the evidence was not informative with respect to any proposed model of GRBs. The problem was that the source location error boxes were too large, and too-tardily reported. It was felt likely that the transient GRB phenomenon was in all likelihood associated with equally-transient phenomena at other wavelengths, and that observing such transients might be the key to unlocking the mystery. But a 4-degree error box on the sky is always crowded with astronomical sources, including time-varying ones, and it was simply not possible to identify any one of them as the culprit. GRB research stalled. Bad evidence!

In the case of the DIY electrician, something was clearly not right with his understanding of the situation inside the box, because after multiple inspections it was increasingly clear that all the connections were fine, and none of the wires were getting crimped. Something else, not suggested by the model, had to be at fault. Bad model!

In the case of a trained LLM’s efforts to respond to prompts, we mostly have a bad model problem, in my opinion. Certainly, the hallucination phenomenon suggests a very brittle model that easily goes off the rails. However, depending on the objective of the training, there might also be a bad evidence problem, particularly in the case of training an AGI: as I discussed last week, the text corpus almost certainly contains no information concerning the origins of human reasoning processes.

Where’s The Aha! ?

Note one characteristic feature of the learning process that I described above: it is in essence continuous. Piece of evidence comes in, small adjustment occurs in weights. Lather, rinse, repeat.

If we are going to base an account of reason on straight up learning, as the LLM research community is attempting to do, this is a very serious (although largely unrecognized) problem, because one of the salient features of reason is that it often operates discontinuously. We have all, I am sure, experienced those moments of “Aha!” revelation, in which suddenly some issue that we have struggled with suddenly seems easily solvable. The problem has suddenly flipped and twisted in such a way that clarity replaces darkness. If there is an aspect of reason that distinguishes it from other cognitive activities, I submit that “Aha!” is that aspect.

That’s the problem with the “learning to reason” approach to AGI. Learning is an essentially continuous process. It simply cannot produce the “Aha!” discontinuity. There is no pure learning path to Artificial Aha! (AA). As a type of cognition, learning is severely limited by restrictions on evidence and model choice. Essentially, all it can do is update its weights across the fixed model’s knob settings, based on evidence drawn from a fixed collection of evidence types, in the hope that some settings are explanatory of the evidence and that the evidence is informative about the model.

It should go without saying that this does not begin to capture reasoning. Anyone reflecting on their own occasions of “Aha!” moments of sudden clarity and insight (not necessarily in the pursuit of natural science, home repair, or computer science, but in solving any puzzle in any field of human activity!) should understand that those moments do not come from a process analogizable to gradual constraining of a model through gradual assimilation of accumulating data. “Aha!” moments are essentially cognitive discontinuities, gestalt shifts that suddently alter the process of assimilating evidence into a model, and are incompatible with the continuous learning process described above. So what are we talking about when we talk about “reason”, and in what way is it related to learning? And, how might we produce AA?

Evidentiary Reform

Suppose that we recognize that we are in Case (1): the evidence is not informative of the model. Then the move is obvious: we change our evidence stream. We cast about for a new stream of more powerful evidence that speaks more clearly to our model, using our knowledge of model features that might be sensitive to other types of evidence, as well as of what new types of evidence might be feasibly acquired. We refer to this shift as Evidentiary Reform.

Evidentiary reform is pretty much the approach taken by astrophysicists to decode the nature of GRBs. Realizing that no GRBs could be associated with a transient counterpart in other wavelengths because of the inaccuracy of GRB locations, GRB scientists developed new high-precision X-ray localization instruments, and arranged for GRB locations to be propagated in real time to ground-based optical and radio observatories. The first transient optical counterparts of GRBs (the so-called “afterglows’’) were detected in 1998, revealing their extragalactic nature through their substantial absorption redshifts¹. By 2003 a core-collapse supernova in a relatively nearby galaxy had been caught in flagrante in a GRB error box (whose size was now about 0.05 degrees), associating GRBs with a certain type of supernova. Case closed. The new stream of evidence, brought into being to correct the weakness of the previous evidence, transformed the mystery into a soluble problem.

The ability to propose evidentiary reform to obtain better model constraints is certainly an example of a true reasoning process. It has the required “Aha!” discontinuous character, embedded in the realization that a new type of information is required for further progress. It is also a highly non-trivial thing to model in a computation, since a successful evidentiary reform needs to take into account not only the nature of the weakness of the previous evidence with respect to model constraint potential, but also practical considerations of how such new evidence can be obtained given real-world feasibility constraints.

Model Reform

Suppose that we recognize instead that we are in Case (2): the model is insufficiently explanatory of the evidence. Then, again, the move is obvious: replace the model with a new model capable of improved predictive power, and endowed with a new set of knobs. The new model might be suggested by the specific form of prediction failures common to the old model. It would likely also satisfy certain criteria of ontological parsimony, embodying some notion of Occam’s Razor-type simplicity so as to exclude model families of weak explanatory/predictive power. We will refer to this process as Model Reform.

The DIY electrician took this approach to finally figuring out his short-circuit problem. After several iterations of taking the fixture off the box and inspecting various electrical elements and connections for defects, and making sure the wires were neatly folded in the box so that they could not become crimped, he started to think of what could produce a short-circuit only when the fixture was secured to the wall. At which point, he realized that the screw securing the fixture to its mounting strap in the electrical box was long enough to reach through the box into the hole in the wall from which the electrical cable emerged, and bury itself among the wires in the cable, potentially crimping and shorting them. And an inspection of the end of the screw showed a dark discoloration that was not present originally, presumably due to the short-circuit passing through the end of the screw. A simple solution—replacing the screw with a shorter screw—immediately produced a satisfactory installation. The problem had been that the original model did not feature any role for the mounting screw. The new model now contained a statement “The mounting screw causes a short at the electrical cable when the fixture is fully secured.” It was induced by the inability of the original model to predict the short-circuits, and supported by new evidence (the discoloration of the end of the screw) which was not interpretable within the the original model.

A reasoner can produce an “Aha!’’ discontinuity through model reform, when a judicious replacement of the model results in improved predictions of the evidence, leading to marked improvements in the concentration of the knob settings weights. Again, this type of reasoning is not straightforward to model in a computation, since formulating a new model requires some sense of the data misfit and a formulation of some kind of Occam Razor conceptual parsimony constraint.

Reasoning and AI

In summary, this high-level account of reason ascribes to it the ability to supervise and intervene upon a learning process, discontinuously altering either the model or the evidence stream, which would otherwise be static features of the learning process. In addition, a reasoning process must be capable of recognizing when a learning process under its supervision stalls. When a stall occurs, it must diagnose whether the failure is more likely due to a bad model or to a bad evidence stream, and it must propose an alteration of one or the other, according to criteria suggested by the failure, while respecting important constraints on possible alternatives.

In other words, in this account, reasoning transcends learning in an essential manner.

This is major trouble for current attempts to obtain AGI, because, as we have been discussing, the entire subject is based on machine learning. Transformer-based LLMs are nothing but computational models that learn to represent an approximation to the probability distribution over token sequences encountered in their training data, which they exploit to construct likely sentence completions, sentence translations, sentence classifications, and so on. They do this so well that their output can belie its origin in probabilistic mimicry (in Emily Bender’s memorable phrase, they are Stochastic Parrots). They can produce the appearance of reasoned discourse at most times. But the process by which such models are trained is the gradual, continuous assimilation of millions of text documents into a stupefyingly large model. LLMs never do “Aha!” They simply aren’t wired that way, becuse their evidence streams and models are fixed.

This is the point that current AGI research appears to miss altogether. The view now gaining currency among practitioners is that the “emergence” of intelligence occurs in consequence of training models with billions, or trillions of parameters, as evidenced by the fact that such models can perform certain “reasoning tasks”. But performing reasoning tasks is not at all the same thing as reasoning: that is the circular argument for AGI again. Some modern AI system have been trained to write very creditable computer code. But the ability to write code does not make one a computer scientist—there are no AI computer scientists today, certainly none capable of proposing new conceptions and models. Similarly, some AI systems can prove mathematical theorems. This does not make them mathematicians, since there is much more to the cognitive activities of a mathematician than just proving theorems—it is far more challenging and useful to know which theorems are interesting to search for, and to create interesting new mathematical frameworks within which theorems can be searched for and proven. And, from the sublime to the ridiculous: an LLM-based AI electrician may know chapter and verse of the National Electrical Code, and be as conversant with tools, materials, and techniques as any licensed electrician. But faced with a situation not previously confronted by any training example it would not be able to reform its model or its evidence stream to suit the unexpected circumstance.

Is This Model Right?

I don’t know whether the model of reason that I argue for here is indeed correct, or in any sense valuable. It has obviously not been implemented in software and validated. As I have indicated, it would be highly non-trivial to represent the model in software. But not, I think, impossible. It is at least a specific model, and it is based on a set of mathematical ideas. One could at least begin building small toy systems that would permit some exploration of its features.

I imagine that AI practitioners would find it easy to reject this model and ignore the conclusions that it forces one to draw, because there is no output that one can judge it’s validity by. But please note that at least this is a model of reason. AI researchers have never deigned to supply such a model, instead relying lazily on vague notions of “emergence” and “self-organization” for which they offer no mathematical theory worthy of the name. Which is to say, they embrace the circular argument for AGI, discovering AGI in LLM output after declaring what AGI should appear as in LLM output. That is a worthless, contemptible scientific argument (Diogenes is getting the better of me again). If you want to tell me that your model “reasons”, show me your model of reason, and we can argue about whose model is better. I would love to have that conversation. It would be on a whole different intellectual plane from where AGI research is today.

The Implausibility of “Learning To AGI”

In order to break down the claim into intelligible pieces, it is useful to adopt the “model-agnostic” outlook on machine learning that I discussed last week. Recall that in that outlook, we draw a veil over the details of the machine learning implementation, and focus on learned distributional structure of training data and on optimality of decision choice. In this case, the training data is vast amounts of text distilled and cleaned and curated, from large-scale Internet scrapes, from large libraries of scanned books, from academic journals, and so on. The decisions are responses to prompts. Whatever the thing behind the veil is, what it does is learn an approximation to the distribution of texts, and approximately optimal responses to prompts.

I need to introduce a concept here that is familiar to most scientists: it is the idea of an inverse problem. The problem is this: given that some data resulting from observations of some process, infer certain attributes of that process. A simple example is weather prediction: given a time-series of observations of weather conditions at thousands of weather stations, and radar and other remote observations, recover an approximation of the current full state of the atmosphere, so as to evolve it using a numerical weather model to predict whether it will rain tomorrow. Another famous (and essentially unsolved) example is from epidemiology: given some time-series of data on infections, hospitalizations, and deaths due to COVID-19, say, infer the current state of the epidemic (how many people are susceptible, exposed, infected, recovered, immune, on a county-by-county basis), and use a numerical epidemiological model predict the epidemic’s future course.

Note the essential elements of such problems: we have a principled model of the process (a numerical weather model, or an epidemiological model) whose state we would like to infer (the atmospheric state, or the state of the epidemic) using data (weather observations, clinical data) so as to make predictions (will it rain during my picnic, is there a new epidemic wave in progress). There is always an assumed “forward model” that describes how the observed data arises, given the state of the process. But that state is unknown, and to estimate it from data one must in some sense “invert” the forward model. Hence “Inverse Problem”.

The process model plays a key role. You need to have some idea of how the process works—a set of equations that governs the process, for example, depending on unknown parameters that you need to infer—for there to even be a well-posed inverse problem. That’s not a sufficient condition, but it is certainly a necessary one.

Inverse problems are ubiquitous in science. In fact, one could, after a few beers, make the claim that most of the daily activities of scientists revolve around solving inverse problems. This is not completely true (where did the principled process models come from, in the first place?) but it is not a grotesque caricature either.

We can view the training of an “AGI” in inverse problem terms: the data is the oceans of text that these things ingest. The process model is the transformer-based “reasoning” model. The “state” to be inferred is the parameter configuration of that model that closely corresponds to a representation of the mental state of a reasoning human. The predictions are reasoned responses to prompts.

OK that’s all I need. Here is the problem: in order to believe that LLMs are achieving “reason” (the minimum requirement for any definition of AGI), we need to accept two big claims:

1. Whatever a reasoning process may be, it leaves a sufficiently informative imprint of its internal state in text data, such that the state may be in some sense recovered and exploited, given a sufficiently large corpus of text, by solving the corresponding inverse problem.
2. Transformer-based LLMs, in some sense, play the role of the process model in this inverse problem, and training such an LLM is tantamount to solving the inverse problem. Moreover, the trained LLM embodies the resulting reasoning entity to the point that at inference time it actually reasons.

Let’s take these in order:

In my opinion, claim (1) is barely sane. Perform any sort of introspection, and I think it is likely that you will find that your spoken or written utterances embody only the most superficial layers of your reason and other cognitive processes. That’s why we all struggle to put our thoughts into words when the occasion arises. We often are not even clear about what our thoughts are, and find, after putting them into words, that they have changed, possibly getting clearer, but also often becoming murkier and less certain as we are forced to articulate our meaning¹.

I simply cannot understand how such subrational processes might embed any interpretable information in our utterances. It is analogous to believing that, given a full, principled model of human physiology, and a data corpus of human footprints together with clinical observations of the humans leaving the footprints, one could train a model that could observe a new footprint and predict the health of the corresponding human. That would be mad: there is not enough information embedded in a footprint to back out a person’s gastric health, or vision acuity, or state of infection from a disease, etc. Similarly, I do not believe that there is enough information impressed in text about the subrational processes whose surface manifestations we call “reason”. I could be wrong about this, but I don’t think so, and in any event the burden of proof is on those researchers who make this kind of claim. Where is that information? How is it encoded?

Claim (2) is actually much worse: it is in the category that physicist Wolfgang Pauli called “not even wrong”—a statement so detached from scientific discourse that classifying it as correct or incorrect is simply a waste of time.

Let’s pull back the curtain concealing the LLM model for a moment. If you read any of the many online descriptions of how a transformer works (The Illustrated Transformer is pretty good, and Wikipedia’s is quite detailed, but Google has many hits for “How does a transformer LLM work”), you may find the level of computational detail off-putting at first. But if you zoom out a bit, what you realize is that it is mostly a giant chain of linear-algebraic operations, interspersed with a few nonlinear “activations”, sandwiched between a linear encoding layer and a nonlinear decoding layer. In this sense it is not different from any DL method. There’s more layers and parameter arrays than most, but not much more structure. It’s a system that grew out of a lot of trial and error, with a pile of late, unlamented errors filling a large dumpster in the back of the lab, and only what more-or-less worked left in.

There is nothing special in that model that is analogous, say, to the model of human physiology that one would need to even attempt to back out a human’s health from that human’s footprint. There isn’t a scrap of theory to motivate the claim that transformer-based models could furnish the basis for solving this inverse problem. Which is to say, a key element of the inverse problem—the principled model embodying actual knowledge of the process under study—is simply not there. Instead there are chains of linear algebra mingled with other ad-hockery, not purporting to model anything. Which means that Claim (2), is in effect, not only that this Rube Goldberg device is capable of inverting the forward model to recover the reasoning process state, but it is also somehow capable of reconstructing the principled model of the reasoning process of which that state is an attribute. That chain of linear algebra is, in effect, a Nobel-caliber cognitive scientist, because the first reasoning task that it carries out is to create a working model of reason itself, a task that still eludes the discipline of cognitive science!

That is just magical thinking. It is literally impossible that this bodged-together system should have accidentally succeeded in modeling reason—an unsolved scientific problem—and then solving the related, probably impossible inverse problem of recovering the model’s state from text input, so as to boot up a reasoning entity. It’s a thoroughgoingly stupid claim.

“AGI” Is A Scientific Scandal

I find it disgraceful and shameful that an entire category of scientists has been moved by enthusiasms and Tech industry funding to lower its intellectual standards to the point that this sort of bullshit floods the journal and conference literature. It’s a scientific scandal, unfolding in plain view. Nothing in the Replication Crisis that afflicts the social sciences comes remotely close to this level of corrupted science.

I can’t emphasize strongly enough that this hubristic nonsense is taken very seriously by the “AI” research community. Sublimely unfazed by the absence of any fundamental explicit understanding of what reason is, and positively glorying in the inscrutable inner complexity of LLMs (“Explainability” is itself a topic for funded research, after all, as we saw last week), this community crows about achieving the “emergence” of intelligence from the models at large scales of data and computation, secure in the knowledge that the models are too unanalyzably complex for any model developer to be expected to explain how this miracle comes about. They just claim that it’s “self-organization” at work. The intellectual laziness of this outlook is simply shocking to me.

At this point, the technical jargon of this discipline has escaped all bounds of propriety. “AI” was bad enough, given the limited amount of “I” in ML (basically, only learning). But now we have “chain of thought”, “knowledge representations”, “mixture of experts”, “agents”, “reasoning models” and “General Intelligence” as well as many other similar allusions to human cognition polluting the technical discourse. Shame is dead in this discipline.

In a sense it’s kind of funny: Silicon Valley Masters of the Universe are directing trillions of dollars in investments to build hundreds of data centers, buy stupefying amounts of computing hardware, and add an estimated 60GW of electical power generation to the U.S. grid, all for the purpose of achieving something that literally cannot be achieved. There is no pot of gold marked “AGI” at the end of this rainbow. But it will take an infinite amount of data, compute, power, effort, and money to get there and find out. What could possibly go wrong?