The AGI Threshold
What it would mean to cross a threshold whose other side we cannot model from this side.
Some doors, once opened, are not doors anymore.
Artificial general intelligence is not a product category. It is a phrase that names a threshold: the point at which a synthetic system is, across most cognitive domains that matter, at least as capable as the human beings who built it, and shortly thereafter, more so. The phrase is contested, the timeline is contested, the definitions are contested. The seriousness of the threshold is not.
We do not know, from this side of that threshold, what the other side looks like. That is the entire point of a threshold. The honest position is not confident optimism and not confident catastrophism. The honest position is humility, of the kind that previous generations applied to nuclear weapons after the first use, not before.
What is unforgivable is not uncertainty. What is unforgivable is sprinting toward a threshold whose other side we cannot model, while telling the public that the people sprinting have it under control. They do not. They have said so themselves, in their own papers, in their own resignation letters, in their own testimony.
This chapter records what the people closest to the frontier have said, in their own words, about the threshold they are racing toward.