Machines that learn

Foundations

The question this lesson is allowed to ask

Lesson 6.5 made an argument. What makes our species different is not individual intelligence but collective learning: information accumulating outside individual brains, transmitted well enough that each generation starts where the last one stopped rather than at the beginning.

Chapter 8 then traced the amplifiers. Writing gave storage. Printing gave identical copies. Journals gave a public record that strangers could attack.

So the honest question about machine learning is not whether it is impressive. It is: is this the next amplifier in that series, or is it a different kind of thing?

That question is answerable in part and not in whole, and this lesson will be careful about which is which.

What these systems actually do

Mechanically, the change that produced the current generation is representation learning. Earlier systems were given features by programmers: a spam filter was told to look for particular words. Deep learning systems find the useful features themselves, by adjusting many parameters to reduce error on a large collection of examples.

That is genuinely powerful and it is not mysterious. A system trained this way finds statistical structure in its training data. What it can do is bounded by what that data contains — a constraint that will matter repeatedly below.

The strongest evidence, taken seriously

If you want to know whether something is real, look at its best case rather than its loudest one.

The best case is protein structure prediction.

A protein is a chain of amino acids that folds into a specific three-dimensional shape, and the shape determines what it does. Predicting the shape from the sequence was a defining unsolved problem in biology for about fifty years. Determining a structure experimentally, by crystallography or cryo-electron microscopy, could take a doctorate.

In 2020 AlphaFold produced predictions competitive with experimental accuracy for many targets.

And here is why that is a measurement rather than an announcement. It was assessed in CASP, a blind competition running since 1994. Organisers take proteins whose structures have been solved but not published, hand out the sequences, collect predictions, and only then reveal the answers. You cannot fit to the test set because you do not have it.

That is chapter 8’s method, applied to a machine learning claim, and it is why this result is in a different category from most of what gets announced.

The part of that story that matters most for this course

AlphaFold was trained on roughly a hundred and fifty thousand experimentally determined protein structures.

Those structures were produced over five decades, by tens of thousands of scientists, at enormous public expense, and deposited in a shared open database because a community decided that was the norm.

The system did not replace that work. It compounded it.

Which is precisely the structure of collective learning from lesson 6.5, running one level up: an accumulated public store of human-produced knowledge, exploited by a new tool, producing a result no individual could have reached.

That is the strongest available answer to this lesson’s question, and it points toward amplifier rather than new kind.

The case on the other side

But there is a genuine counter-case, and it deserves its full weight.

In 2016 a system exceeded the strongest human players at Go. It learned substantially by self-play — playing against copies of itself, generating its own training data — and it produced strategies that human experts had not used and initially did not understand.

That is not compounding human knowledge. In that domain, the loop closed. The system added to the store on its own.

So what made it possible? Two conditions. A perfect simulator: the rules of Go are complete, so a machine can generate unlimited valid experience at no cost. And an unambiguous measure of success: you won or you did not.

Both conditions hold in games, in some mathematics, and in a narrow set of engineering problems. Neither holds for history, medicine, law, or most of science, where the world is the simulator, experiments are slow and expensive, and success is contested.

The honest summary is: where a perfect simulator exists, these systems can generate new knowledge independently. Where it does not, they compound ours.

What happens when the loop tries to close anyway

There is now direct experimental evidence on this, and it is elegant.

Train a model on human data. Use it to generate output. Train the next model mostly on that output. Repeat.

The result is model collapse. Distributions narrow with each generation. And the failure has a specific shape: the tails go first. Rare events, unusual constructions, minority cases disappear before the common cases degrade. Each generation is a slightly blurrier copy that has forgotten the edges.

This was published in Nature in 2024, and it bears directly on the largest claim in the field — that these systems will bootstrap themselves. On present evidence, without continued input of new material from the world, the loop degrades rather than compounds.

That may be an engineering problem with an engineering solution. Right now, it is a measured result pointing the other way.

The verification problem, measured

Here is where this lesson gets uncomfortable, and where it connects hardest to lesson 8.2.

During the pandemic, an enormous number of papers proposed machine learning systems to diagnose or prognose COVID-19 from chest images. A systematic review screened over two thousand of them and examined dozens in detail.

None of the ones examined were found to be of potential clinical use.

Not because the idea was impossible. Because of methodological failures that recur constantly in this field: training and test data drawn from overlapping sources, control images from a different population than the patient images, no external validation, insufficient documentation to reproduce anything.

A famous class of failure: a model that appeared to detect disease was actually detecting that sick patients were imaged lying down and healthy ones standing up. The system found real statistical structure. It was structure about hospital logistics.

Then, more recently, a large claim of machine-learning-driven discovery of new inorganic materials was followed by an independent analysis finding that few of the reported compounds were genuinely novel and useful.

Both episodes are the scientific method working. A claim was made, others checked, the claim was revised. That is the procedure from lesson 8.2 doing its job. But notice that the checking took years and specialist effort, while the original claims travelled worldwide in days.

The bottleneck in this field is verification, not generation.

Whom it is for

Now the question this course is obliged to ask about any technology: who does it work for?

A survey of the world’s languages by how much digital text and annotated data exists for them found the distribution extraordinarily uneven. A handful of languages have almost everything. The overwhelming majority of the world’s roughly seven thousand languages have almost nothing, and performance follows resources closely.

Most languages of India sit in the lower classes — including several with tens of millions of speakers, and some with long written literary traditions considerably older than English.

This is not a footnote. It determines whether a farmer in Bihar can use a system in Maithili, whether a court in Tamil Nadu can process documents in Tamil as well as in English, and whether the productivity gains being forecast reach eighty per cent of Indians or twenty.

And this is fixable. It requires deliberate corpus building, annotation and evaluation in those languages, which is unglamorous, expensive, and exactly the kind of public infrastructure work that produced the protein database this lesson praised earlier.

There is also the labour. The annotation and evaluation these systems depend on is substantially performed in lower-income countries, including India, and it is rarely visible in the accounts of how these systems were made.

The answer, kept narrow

So: new threshold, or very fast library?

What the evidence supports. These systems are a real and powerful amplifier of collective learning. They compress, retrieve and recombine an accumulated human store at a speed no previous tool approached. In domains with a perfect simulator and a clean success measure, they generate genuinely new knowledge without us. They are a scientific instrument, and AlphaFold is the proof.

What the evidence does not support. That they constitute a new mechanism of accumulation independent of human input. The strongest results rest on decades of human measurement; the closed loop degrades; and verification remains a human bottleneck.

What nobody knows. Whether the conditions that made Go tractable can be extended to domains that matter more. That is a real open question and this course is not going to resolve it in eight minutes.

Lesson 6.5 identified a threshold after it had already happened, which is the only way thresholds get identified reliably. Whether this is one will be obvious in retrospect and is not obvious now.

What can be said is that the discipline you need for judging it is the one this course has been building all along: ask what was measured, ask who checked it, ask what would show it to be wrong. That works on machine learning claims exactly as it works on cosmology.

Deeper

Why CASP design matters more than the score

The specific virtue of CASP is that the ground truth exists but is withheld. Structures are solved experimentally and held unreleased; predictors receive only sequences; assessment is by independent assessors using metrics fixed in advance. This blocks the two failure modes that inflate most reported machine learning performance — training on the test set, and choosing the metric after seeing results. Any claim in this field should be evaluated by asking whether an equivalent barrier existed. Most of the time it did not.

What model collapse does and does not show

The experiments train successive generations on data sampled from the previous generation, with varying proportions of original data retained. Collapse is fastest when no original data is retained, and slows substantially when a fraction of real data persists in each generation. So the finding is not that any synthetic data is harmful; it is that a fully closed loop is degenerative, and that the degradation begins in the tails of the distribution. The practical implication is that access to genuinely new human-produced data is a continuing requirement, not a one-off cost, which has consequences for who controls these systems.

Why the chest-image failures were predictable

Several of the reviewed studies used a public dataset of paediatric pneumonia images as the control group for adult COVID-19 patients. A model separating those two groups can succeed by detecting the age of the patient, or the scanner, or the acquisition protocol. This is a general property: any systematic difference between classes that is easier to detect than the intended signal will be found, because the optimisation has no notion of which correlations are meaningful. It is the same failure as an experiment without a proper control, which is why lesson 8.2’s insistence on controlled comparison is the right frame for reading these papers.

The energy accounting, with its caveats

The 2019 analysis that first drew wide attention to training energy costs estimated consumption for several natural language models including hyperparameter search, and reported figures large enough to be worth stating. The estimates depend heavily on hardware generation, data centre efficiency and the carbon intensity of the local grid, all of which have changed since. Inference at scale, rather than training, now accounts for a larger share of total energy for widely deployed systems. The honest statement is that the amounts are material and that credible current figures require operator disclosure that is often not provided.

Frontier

Whether the simulator condition can be relaxed

The active research direction is building sufficiently good simulators, or sufficiently reliable automatic verifiers, for domains that currently lack them — formal mathematics, code with executable tests, and some areas of chemistry are the most promising because correctness can be checked mechanically. How far that extends is the central open question behind most current claims about capability growth.

Evaluation itself

As systems are trained on ever larger fractions of available text, keeping benchmark material out of training data becomes harder, and reported scores become correspondingly less informative. Designing evaluations that remain valid under these conditions is an unglamorous problem on which a great deal depends.

Transcript and shot list

Target runtime about 11 minutes. Built to take the biggest claim, test it against the strongest evidence, then against the strongest counter-evidence, and answer narrowly. Shot types: ANIM is rendered from code with Manim, AI is a generated shot from Higgsfield with the shared style reference, REAL is agency imagery with credit, CAM is the narrator.

[00:00] S01 · ANIM · Lesson 6.5 recalled: information accumulating outside individual brains. Then the amplifiers in sequence — writing for storage, printing for identical copies, journals for a public record strangers can attack.

Lesson 6.5 made an argument. What makes us different is not individual intelligence, it is collective learning: information accumulating outside individual brains, passed on well enough that each generation starts where the last one stopped instead of at the beginning. Then chapter eight traced the amplifiers. Writing gave storage. Printing gave identical copies. Journals gave a public record that strangers could attack.

[00:29] S02 · CAM · Narrator direct to camera.

So the honest question about machine learning is not whether it is impressive. It is whether this is the next amplifier in that series, or a different kind of thing. That question is answerable in part and not in whole, and I am going to be careful about which is which.

[00:52] S03 · ANIM · An older system being handed features by a programmer; then a deep network adjusting parameters to find its own features from examples. The bound drawn explicitly: what it can do is limited by what the data contains.

Mechanically, the change behind the current generation is representation learning. Older systems were handed features by programmers — a spam filter told to look for particular words. Deep learning systems find the useful features themselves, by adjusting a very large number of parameters to reduce error on a very large collection of examples. That is genuinely powerful and it is not mysterious. What such a system does is find statistical structure in its training data. Which means what it can do is bounded by what that data contains. Hold on to that, because it comes back three times.

[01:37] S04 · ANIM · A protein: an amino acid chain folding into a specific shape; the shape determining function. Fifty years of the prediction problem unsolved. A single experimental structure costing a doctorate.

If you want to know whether something is real, look at its best case rather than its loudest one. The best case here is protein structure prediction. A protein is a chain of amino acids that folds into a specific three-dimensional shape, and the shape is what determines what it does. Predicting that shape from the sequence was a defining unsolved problem in biology for about fifty years. Getting one structure experimentally could take somebody a doctorate.

[02:13] S05 · ANIM · CASP: structures solved but unpublished, sequences handed out, predictions collected, answers revealed afterwards. Then AlphaFold’s accuracy shown as competitive with experiment. Then the training set: 150,000 structures, five decades, tens of thousands of scientists, deposited openly.

In 2020 AlphaFold produced predictions competitive with experimental accuracy for many targets. And here is why that is a measurement rather than an announcement. It was assessed in CASP, a blind competition running since 1994. The organisers take proteins whose structures have been solved but not published, hand out the sequences, collect everyone’s predictions, and only then reveal the answers. You cannot fit to a test set you do not have. That is chapter eight’s method applied to a machine learning claim. But here is the part that matters most for this course. AlphaFold was trained on about a hundred and fifty thousand experimentally determined structures, produced over five decades by tens of thousands of scientists at enormous public expense, and deposited in a shared open database because a community decided that was the norm. The system did not replace that work. It compounded it. Which is exactly the shape of collective learning, running one level up.

[03:25] S06 · ANIM · 2016: self-play, a system playing copies of itself, generating its own data; exceeding the strongest human players; producing strategies experts had not used.

So that points toward amplifier. But there is a real counter-case and it deserves full weight. In 2016 a system beat the strongest human Go players. It learned substantially by self-play — playing copies of itself and generating its own training data — and it produced strategies human experts had not used and at first did not understand. That is not compounding our knowledge. In that domain the loop closed. It added to the store on its own.

[04:01] S07 · ANIM · Two conditions isolated: a perfect simulator, and an unambiguous measure of success. Then a checklist run against history, medicine, law and most of science, failing both.

So what made that possible? Two conditions. A perfect simulator: the rules of Go are complete, so the machine can generate unlimited valid experience for free. And an unambiguous measure of success: you won or you did not. Both hold in games, in parts of mathematics, in a narrow set of engineering problems. Neither holds for history, medicine, law, or most of science, where the world is the simulator, experiments are slow and expensive, and success is contested. So: where a perfect simulator exists, these systems generate new knowledge independently. Where it does not, they compound ours.

[04:46] S08 · ANIM · The recursive loop: model trained on human data, generating output, next model trained on that output, repeating. Distributions narrowing generation by generation, with the tails vanishing first — rare events and minority cases disappearing before the common ones degrade.

And there is now direct experimental evidence about what happens when you try to close the loop anyway, and it is elegant. Train a model on human data. Use it to generate output. Train the next model mostly on that output. Repeat. What you get is model collapse. The distribution narrows every generation, and the failure has a specific shape: the tails go first. Rare events, unusual constructions, minority cases disappear before the common cases degrade. Each generation is a slightly blurrier copy that has forgotten the edges. That was published in Nature in 2024, and it bears directly on the biggest claim in the field, which is that these systems will bootstrap themselves. On present evidence, without new material coming in from the world, the loop degrades rather than compounds. That might be an engineering problem with an engineering solution. Right now it is a measured result pointing the other way.

[05:56] S09 · CAM · Narrator direct to camera.

Now the uncomfortable part, and the place this connects hardest to lesson 8.2.

[06:02] S10 · ANIM · Over two thousand COVID-19 imaging papers screened, dozens examined in detail, none found clinically usable. Then the failure modes: overlapping train and test sources, controls from a different population, no external validation. Then the lying-down versus standing-up artefact drawn explicitly.

During the pandemic an enormous number of papers proposed machine learning systems to diagnose COVID-19 from chest images. A systematic review screened over two thousand of them and examined dozens in detail. None of the ones examined were found to be of potential clinical use. Not because the idea was impossible — because of methodological failures that recur constantly in this field. Training and test data from overlapping sources. Control images from a different population than the patients. No external validation. Not enough documentation to reproduce anything. There is a famous class of failure where a model that appeared to detect disease was actually detecting that sick patients were imaged lying down and healthy ones standing up. The system found real statistical structure. It was structure about hospital logistics.

[07:02] S11 · ANIM · A large claim of AI-driven materials discovery; then an independent analysis finding few of the reported compounds genuinely novel and useful. Two clocks: the claim travelling worldwide in days, the checking taking years.

More recently, a large claim of machine-learning-driven discovery of new inorganic materials was followed by an independent analysis finding that few of the reported compounds were genuinely novel and useful. Both of those episodes are the scientific method working. A claim was made, others checked, the claim was revised. That is chapter eight’s procedure doing its job. But notice the asymmetry. The checking took years and specialist effort. The original claims travelled worldwide in days. The bottleneck in this field is verification, not generation.

[07:41] S12 · ANIM · The world’s roughly seven thousand languages ranked by available digital text and annotation: a handful with almost everything, the overwhelming majority with almost nothing. Indian languages placed in the lower classes, several with tens of millions of speakers and literary traditions older than English.

And the question this course is obliged to ask about any technology: who does it work for? A survey of the world’s languages by how much digital text and annotated data exists for them found the distribution wildly uneven. A handful have almost everything. The great majority of roughly seven thousand languages have almost nothing, and performance tracks resources closely. Most languages of India sit in the lower classes, including several with tens of millions of speakers and written literatures considerably older than English. That is not a footnote. It decides whether a farmer in Bihar can use one of these systems in Maithili, whether a court in Tamil Nadu can work in Tamil as well as in English, and whether the productivity gains being forecast reach eighty per cent of Indians or twenty.

[08:42] S13 · CAM · Narrator direct to camera.

And it is fixable. It needs deliberate corpus building, annotation and evaluation in those languages — unglamorous, expensive, and exactly the kind of public infrastructure work that built the protein database I was praising four minutes ago. There is also the labour question. The annotation and evaluation these systems run on is substantially done in lower-income countries, India included, and it almost never appears in the story of how they were made.

[09:16] S14 · ANIM · Three columns: WHAT THE EVIDENCE SUPPORTS — a real amplifier, an instrument, independent generation where a simulator exists. WHAT IT DOES NOT SUPPORT — a new mechanism independent of human input. WHAT NOBODY KNOWS — whether the simulator condition can be extended. Then the three questions: what was measured, who checked it, what would show it wrong. Calendar returns.

So: new threshold, or very fast library? What the evidence supports is that these are a real and powerful amplifier of collective learning, that they compress and recombine an accumulated human store faster than any previous tool, that in domains with a perfect simulator they generate genuinely new knowledge without us, and that they are a scientific instrument — AlphaFold is the proof. What the evidence does not support is that they are a new mechanism of accumulation independent of human input. The strongest results rest on decades of human measurement, the closed loop degrades, and verification is still a human bottleneck. And what nobody knows is whether the conditions that made Go tractable extend to the domains that matter more. Lesson 6.5 identified a threshold after it had already happened, which is the only reliable way thresholds get identified. Whether this is one will be obvious in retrospect and is not obvious now. What I can give you is the discipline for judging it, and it is the same one this whole course has been building. Ask what was measured. Ask who checked it. Ask what would show it to be wrong. That works on machine learning claims exactly the way it works on cosmology. Next: the technology that edits the thing chapter five spent eight lessons on.

Shot list

Zero AI shots, eleven code-rendered shots, three narrator shots. Using generated imagery in a lesson about the limits of generated output would be a bad joke, and the joke would cost credibility.