Prioritising AI Risks

I was recently asked to pick two AI risks that are most deserving of focused research work from this list:

I chose Loss of Human oversight / Control over critical systems AND Misuse by Bad actors

To be clear, I think all the risks are valid and substantial risks with meaningful probabilities of happening, some with current AI systems, and some much more likely with more advanced AI systems. It's perhaps more intuitive for me to mention why the other risks are either less catastrophic, less likely to occur even if more catastrophic, or actually dependent on other risks, to explain why I arrived at my choices.

On the risks that I don't feel are existentially catastrophic: Bias, discrimination, unfair outcomes / disinformation, manipulation, erosion of shared reality / economic disruption, job displacement. For bias and disinformation, those already happen significantly without LLM-based AI, through conventional politics, media as well as non-LLM algorithms in social media. AI may exacerbate these issues through deepfakes, through plausibly but incorrect research/science through AI decision making, but even in a worst case scenario, most civilisations have robust social and policy safeguards to prevent even the most grievously misinformed society to destroy populations.

On the risks that I feel are existentially catastrophic but carry a very low probability of a genuine catastrophe without relying on the risks I've prioritised: Misaligned or power-seeking AI causing catastrophic / irreversible harm. That isn't to say a misaligned or power seeking AI can't cause catastrophic harm, but the likelihood of it getting to a level of power and capability where it can cause those harms relies on either an inconceivably incompetent series of human decisions that lead to that outcome, or through collusion with bad human actors.

Concentration of power/authoritarian entrenchment is a real significant risk, but I'd argue that can occur with or without advanced AI: suppression of free speech, mass surveillance, economic dominance. The type of concentration of power that would cause mass suffering or cause a catastrophe would fundamentally veer on to being considered as "misuse by bad actors", where AI and automated weapons are deployed.

Loss of human oversight / control over critical systems, is in my view one of the most important areas of AI risk to tackle. This comes both from a technological standpoint, building good monitors, safeguards, scalable oversight to ensure automated long-running processes like research are done safely, effectively and maintain alignment with human researchers. It also comes from a policy standpoint, what structures, safety frameworks, practices do we enforce when we use AI systems in critical infrastructure. AI alignment may never be fully solved, but being able to stop it before misalignment emerges or have real physical barriers to a misaligned AI taking control over critical systems like nuclear weapons or power stations, would prevent downstream risk of power-seeking AI causing catastrophic harm.

Misuse by bad actors (bio, cyber, weapons), is likely the highest and most near-term risk. Without further advancement we've already seen significant cyberattacks, white hat and black hat breaches using AI on critical public and private organisations. It's unclear the true extent of actual cyber incidents by bad actors using AI are, and further advances will only exacerbate that. Bio on the other hand is currently more theoretical, but well accepted as a significant near term risk due to the leverage it provides bad actors as well as runaway risk that even neutralising the bad actor, or the bad actor themselves willingly cannot stop it. Weapon systems is also a large area that we can already see the trajectory of AI systems drastically improving the capabilities, from targeting, coordination to piloting. A small dedicated force can cause mass casualties using AI-controlled drone swarms or AI-targeted ordnance. The bad actor doesn't have to be small either, its capabilities that would drastically entrench the power of authoritarian nations. Tackling misuse will again require technical and policy solutions. Technical solutions would look at safeguards against misuse, monitoring of highly capable models such that misuse attempts would be detected/reported/logged. Policy solutions would look at preventing dangerous capability models from being deployed, banning AI system use in certain domains (like drones), or restricting capabilities in critical parts of AI training.