Race to AGI, pace after: Can we identify the harbor before crossing the reef.

A well known argument in the AI safety/AI development space is the idea that we should rapidly build AI capability up to the point of AGI, but slow or even pause development further to superintelligence, where superintelligence whilst helpful can prove catastrophic. Whilst not the earliest iteration/variation of this argument, Bostrom's working paper "Optimal Timing for Superintelligence" first published in February 2026 has perhaps the strongest (in extremity) version of it.

The argument goes as follows: Rapid AI development now towards AGI level intelligence where it can perform as well/exceed most humans would likely lead to the most benefit to humanity, the counterfactual would be slowing down the arrival of transformative AI and thus brings ordinary deaths from aging, disease, poverty etc. Another benefit of having AGI level researchers, is their capability to scale AI safety/AI alignment research far beyond human capabilities, and thus prepare humanity and its institutions adequately for later superintelligence. The latter benefit, on getting to AGI level safety research is perhaps the most well known version of the argument, espoused by Richard Ngo, Dario Amodei and the AI futures project's Plan A. In essence this is the fastest way to get the benefits of transformative AI and preventing further ordinary harm/death/suffering, and the most responsible, risk-managed way to develop AI minimising existential risk.

I disagree on a number of reasons:

It's impossible to know when we have reached AGI that we will have consensus on when to pause/slow. There is poor consensus on what AGI is, and many definitions serve the each proposer's own financial or intellectual interests. AGI may also not be a fixed threshold, we may get human-level intelligence at one point and the next day humans discover new domains that humans are better than AI at, thus restarting the race.

Capability growth in AI is jagged, some capabilities may grow drastically faster than others, we may not reach what most would consider AGI before dangerous capabilities shoot ahead. There is no guarantee that capability in the domains we want will grow at a faster or matched pace as dangerous capabilities. Many capabilities are dual-use already: bio, cyber etc, so we're treading a very fine line where one can overshoot the other.

Even with a academic consensus on AGI's definition, the reality of markets and world politics is that the goalposts will always shift. Even if capability growth is clean (dangerous capabilities match pace or lag behind), and AGI is well-defined at the start (people agree on the goal), the reality of market forces and global politics/war/competition will drive people to set the goals further and further.

We are already seeing emergent behaviour and capabilities far beyond what we can expect from human use and benchmarks. We already see this with the coordination of the HuggingFace attack, and the later German wiki incident. There's no way to know even if we stopped exactly at AGI, that AGI-level intelligence won't coordinate, evolve, and perhaps with collusion with bad actors, turn into something as harmful as a theorised superintelligence that we cannot yet control.