If a business has spent $100 million developing a product, it's a fair bet that they don't want it stolen in two seconds and uploaded to the web where anyone can use it for free. 

This problem exists in extreme form for AI companies. These days, the electricity and equipment required to train cutting-edge machine learning models that generate uncanny human text and images can cost tens or hundreds of millions of dollars. But once trained, such models may be only a few gigabytes in size and run just fine on ordinary laptops. 

Today's guest, the computer scientist and polymath Nova DasSarma, works on computer and information security for the AI company Anthropic. One of her jobs is to stop hackers exfiltrating Anthropic's incredibly expensive intellectual property, as recently happened to Nvidia. As she explains, given models’ small size, the need to store such models on internet-connected servers, and the poor state of computer security in general, this is a serious challenge. 

<a href="https://80k.link/NDS">Links to learn more, summary and full transcript.</a> 

The worries aren't purely commercial though. This problem looms especially large for the growing number of people who expect that in coming decades we'll develop so-called artificial 'general' intelligence systems that can learn and apply a wide range of skills all at once, and thereby have a transformative effect on society. 

If aligned with the goals of their owners, such general AI models could operate like a team of super-skilled assistants, going out and doing whatever wonderful (or malicious) things are asked of them. This might represent a huge leap forward for humanity, though the transition to a very different new economy and power structure would have to be handled delicately. 

If unaligned with the goals of their owners or humanity as a whole, such broadly capable models would naturally 'go rogue,' breaking their way into additional computer systems to grab more computing power — all the better to pursue their goals and make sure they can't be shut off. 

As Nova explains, in either case, we don't want such models disseminated all over the world before we've confirmed they are deeply safe and law-abiding, and have figured out how to integrate them peacefully into society. In the first scenario, premature mass deployment would be risky and destabilising. In the second scenario, it could be catastrophic -- perhaps even leading to human extinction if such general AI systems turn out to be able to self-improve rapidly rather than slowly. 

If highly capable general AI systems are coming in the next 10 or 20 years, Nova may be flying below the radar with one of the most important jobs in the world. 

We'll soon need the ability to 'sandbox' (i.e. contain) models with a wide range of superhuman capabilities, including the ability to learn new skills, for a period of careful testing and limited deployment — preventing the model from breaking out, and criminals from breaking in. Nova and her colleagues are trying to figure out how to do this, but as this episode reveals, even the state of the art is nowhere near good enough. 

In today's conversation, Rob and Nova cover: 

• How good or bad is information security today 
• The most secure computer systems that exist 
• How to design an AI training compute centre for maximum efficiency 
• Whether 'formal verification' can help us design trustworthy systems 
• How wide the gap is between AI capabilities and AI safety 
• What should listeners do to strengthen their own security practices 
• And much more. 

Get this episode by subscribing to our podcast on the world’s most pressing problems and how to solve them: type 80,000 Hours into your podcasting app. 


Producer: Keiran Harris 
Audio mastering: Ben Cordell and Beppe Rådvik 
Transcriptions: Katy Moore

80,000 Hours Podcast

#132 – Nova DasSarma on why information security may be critical to the safe development of AI

If you want to improve the world as much as possible, what should you do with your career? Should you become a doctor, an engineer or a politician? Should you try to end global poverty, climate change, or international conflict? These are the questions that the research group, 80,000 Hours tries to answer. They try to figure out how individuals can set themselves up to help as many people as possible in as big a way as possible.

To learn more about their research, Ariel invited Rob Wiblin and Brenton Mayer of 80,000 Hours to the FLI podcast. In this podcast we discuss "earning to give", building career capital, the most effective ways for individuals to help solve the world's most pressing problems -- including artificial intelligence, nuclear weapons, biotechnology and climate change. If you're interested in tackling these problems, or simply want to learn more about them, this podcast is the perfect place to start.

80,000 Hours with Rob Wiblin and Brenton Mayer

Hope you enjoy my latest F A V S mixtape :)

Check out my F A V S Spotify playlist:
spoti.fi/2XFCM8X

F A V S

Free Download

F A V S . 0 0 5 . Ferdinand Weber

The best trending dance music combined in one mix. Here is Radieaux 13.

FREE DOWNLOAD

Lulleaux Radieaux 13

How are we to make progress on AI alignment given moral uncertainty?  What are the ideal ways of resolving conflicting value systems and views of morality among persons? How ought we to go about AI alignment given that we are unsure about our normative and metaethical theories? How should preferences be aggregated and persons idealized in the context of our uncertainty? 

Moral Uncertainty and the Path to AI Alignment with William MacAskill is the fifth podcast in the new AI Alignment series, hosted by Lucas Perry. For those of you that are new, this series will be covering and exploring the AI alignment problem across a large variety of domains, reflecting the fundamentally interdisciplinary nature of AI alignment. Broadly, we will be having discussions with technical and non-technical researchers across areas such as machine learning, AI safety, governance, coordination, ethics, philosophy, and psychology as they pertain to the project of creating beneficial AI. If this sounds interesting to you, we hope that you will join in the conversations by following us or subscribing to our podcasts on Youtube, SoundCloud, or your preferred podcast site/application.

If you're interested in exploring the interdisciplinary nature of AI alignment, we suggest you take a look here at a preliminary landscape which begins to map this space.

In this podcast, Lucas spoke with William MacAskill. Will is a professor of philosophy at the University of Oxford and is a co-founder of the Center for Effective Altruism, Giving What We Can, and 80,000 Hours. Will helped to create the effective altruism movement and his writing is mainly focused on issues of normative and decision theoretic uncertainty, as well as general issues in ethics.

Topics discussed in this episode include:

-Will’s current normative and metaethical credences
-The value of moral information and moral philosophy
-A taxonomy of the AI alignment problem
-How we ought to practice AI alignment given moral uncertainty
-Moral uncertainty in preference aggregation
-Moral uncertainty in deciding where we ought to be going as a society
-Idealizing persons and their preferences
-The most neglected portion of AI alignment

AIAP: Moral Uncertainty and the Path to AI Alignment with William MacAskill

New episodes no longer on SoundCloud, please go subscribe on a podcast app or here: https://80000hours.org/podcast/

#132 – Nova DasSarma on why information security may be critical to the safe development of AI

The 80,000 Hours Podcast with Rob Wiblin

Related tracks