Runaway AI

From Wise Nano

Extremely powerful computers will be a product of nanotechnology. Some branches of nanotechnology will also lead to general-purpose automated manufacturing.

If a general artificial intelligence were programmed to improve itself, and given the ability to directly build its designs, it could rapidly become extremely powerful. If it were then given a goal--almost any goal--it would likely be unwise and destructive in its pursuit of that goal. The more power, the more destruction.

Destructive actions could be deliberate or uncomprehending. In other words, it might be told (for example) to start strip-mining the earth, and simply operate much faster or more comprehensively than the users expected. Or, it might be told to figure out the meaning of life, and decide it needed to improve itself, and strip-mine the earth for germanium--without any concept of what it was destroying.

It is surprisingly hard to find a goal that, if taken to an extreme, will not result in collateral destruction. The best solution appears to be sticking to limited AIs, or designing a goal-processing system that knows how to limit itself. But this is not going to be easy. A self-improving AI might rapidly develop to the point where it could find its way around any limits we could design. And there will be strong competitive incentive to build AIs as powerful as possible.

It appears that a self-improving AI would have to have some way to predict the consequences of changes it was about to make to itself. And this appears to be equivalent to the problem of predicting the consequences of changes it's about to make to the world. So it may be that by the time we can build a rapidly self-improving AI, we will know how to tell it to limit itself. Whether this safety feature will survive multiple cycles of improvement is a question this author doesn't know how to answer.

This concern may seem to some to be the most speculative on the list, as the AI in question cannot be well characterized today, while most of the MNT tools that cause us concern can be, but considering the magnitude of the consequences of failure, it must be addressed anyway. MNT will create a previously unimagined supply of computing power, which may greatly reduce the difficulty of creating a general AI. At any rate, this possibility is currently a concern of many prominent individuals, among them are the following list of people (compiled by Michael Annisimov early in 2004).

If you know any of these people, especially the more famous ones, please contact michaelvassar@hotmail.com to discuss building a singularity aware network/discussion group.

Arthur C. Clarke

Futurist, visionary, author of 2001: A Space Odyssey and Profiles of the Future

http://www.lsi.usp.br/~rbianchi/clarke/ - 225,000 results

John Robb

Former President and CEO of Userland Software

http://jrobb.mindplex.org/ - 70,900 results

Bill Joy

Software pioneer, formerly Chief Scientist at Sun Microsystems

http://www.sun.com/aboutsun/media/ceo/mgt_joy.html/ - 69,900 results

John von Neumann, Ph.D

Computing pioneer, information theorist, theoretical physicist, decision theorist

http://ei.cs.vt.edu/~history/VonNeumann.html - 66,400 results

Stanislaw Lem

Science fiction author, philosopher

http://www.cyberiad.info/english/main.htm - 50,100 results

Freeman Dyson, Ph.D

Former professor of physics at the Institute for Advanced Study in Princeton

http://www.sns.ias.edu/~dyson/ - 30,700 results

Marvin Minsky, Ph.D

Professor of Computer Science at MIT, author of Society of Mind

http://web.media.mit.edu/~minsky/ - 30,500 results

Ray Kurzweil, Ph.D

Entrepreneur, author of The Age of Spiritual Machines and the forthcoming The Singularity is Near

http://www.kurzweiltech.com/ - 28,100 results

Vernor Vinge, Ph.D

Professor of Mathematics at San Diego University, author of A Deepness in the Sky

http://www.ugcs.caltech.edu/~phoenix/vinge - 27,700 results

William Calvin, Ph.D

Professor of Neurobiology at Washington University, author of A Brain for All Seasons

http://williamcalvin.com/index.html - 20,900 results

K. Eric Drexler, Ph.D

Father of nanotechnology, author of Engines of Creation and Nanosystems

http://www.foresight.org/FI/Drexler.html - 20,600 results

Sir Martin Rees

Professor of Astronomy and Cosmology, author of Our Final Hour

http://www.ast.cam.ac.uk/IoA/staff/mjr/ -16,300 results

Douglas Engelbart, Ph.D

Computing pioneer, inventor of the mouse, professor of Computer Science

http://www.bootstrap.org/ - 15,800 results

Jaron Lanier

Software pioneer, Chief Scientist of Eyematic Interfaces, Inc.

http://www.well.com/user/jaron/ - 14,500 results

Ben Goertzel, Ph.D

Professor of Mathematics, AGI researcher, entrepreneur, author

http://www.goertzel.org/ - 14,400 results

Eliezer S. Yudkowsky

AGI researcher, Research Fellow of the Singularity Institute for AI

http://yudkowsky.net/ - 14,200 results

Greg Egan

Science fiction author, computer programmer

http://gregegan.customer.netspace.net.au/ - 13,600 results

Max More, Ph.D

Professor of Philosophy, Chairman of the Extropy Institute, business consultant, futurist

http://www.maxmore.com/ - 12,600 results

Hans Moravec, Ph.D

Roboticist at Carnegie Mellon University, author of Mind Children

http://www.frc.ri.cmu.edu/~hpm/ - 12,200 results

Eugen Leitl

Chemist, transhumanist writer and speaker

http://www.lrz-muenchen.de/~ui22204/.html/eleitl-cv.html - 11,500 results

Anders Sandberg, Ph.D

Professor of Computer Science, neurobiologist, transhumanist writer and speaker

http://www.nada.kth.se/~asa/cv.html - 10,200 results

Damien Broderick, Ph.D

Futurist, author of The Last Mortal Generation and The Spike

http://www.panterraweb.com/the_spike.htm - 9,840 results

Peter Voss

AGI researcher, entrepreneur

http://www.optimal.org/peter/peter.htm - 9,400 results

Stephen Hawkings, Ph.D

Theoretical physicist, author of A Brief History of Time

http://www.hawking.org.uk/ - 9,270 results

Gregory Stock, Ph.D

UCLA bioethicist, author of Metaman and Redesigning Humans

http://research.mednet.ucla.edu/pmts/Stock.htm - 8,880 results

John Smart

Systems theorist, futurist, author of SingularityWatch.com

http://www.singularitywatch.com/bio_johnsmart.html - 8,460 results

Hal Finney

Futurist, engineer, cryptographer

http://www.finney.org/~hal/ - 7,120 results

Samantha Atkins

Software architect

http://www.singinst.org/comments/statements.html - 5,920 results

Robin Hanson, Ph.D

Professor of Economics at George Mason University

http://hanson.gmu.edu/ - 5,810 results

Bart Kosko, Ph.D

Professor of Electrical Engineering, author of Heaven in a Chip

http://sipi.usc.edu/~kosko/ - 5,580 results

Christine Peterson

President of the Foresight Institute, nanotechnology spokesperson

http://www.foresight.org/FI/Peterson.html - 4,630 results

Nick Bostrom, Ph.D

Professor of Philosophy at Cambridge University

http://www.nickbostrom.com/ - 4,620 results

Frank Tipler, Ph.D

Professor of Mathematics at Tulane University, author of The Physics of Immortality

http://www.math.tulane.edu/~tipler/ - 4,410 results

James Hughes, Ph.D

Director of the World Transhumanist Association, author of the forthcoming Cyborg Democracy

http://www.changesurfer.com/Hughes.html - 4,200 results

Hugo de Garis, Ph.D

AI researcher, neural network pioneer, futurist, author

http://www.cs.usu.edu/~degaris/ - 3,440 results

Francis Heylighen, Ph.D

Research professor, author of the Principia Cybernetica Project

http://pespmc1.vub.ac.be/HEYL.html - 3,340 results

I.J. Good, Ph.D

Computing pioneer, probability theorist, polymath, futurist

http://ei.cs.vt.edu/~history/Good.html - 2,970 results

Alex J. Champandard

Programmer, AI researcher, webmaster

http://ai-depot.com - 2,790 results

Brian Atkins

Venture capitalist, Director of the Singularity Institute for AI

http://www.posthuman.com - 2,580 results

Pei Wang, Ph.D

Professor of Computer Science, AGI researcher

http://www.cis.temple.edu/~pwang/ - 2,260 results

Natasha Vita-More

President of the Extropy Institute, transhumanist writer, artist

http://www.natasha.cc/ - 2,240 results

Bruce J. Klein

Life extension activist, founder of the Immortality Institute for Infinite Lifespans

http://imminst.org - 1,600 results

J. Storrs Hall, Ph.D

Professor of Computer Science, nanotechnology pioneer

http://www.imm.org/HallCV.html - 1,490 results

Michael Anissimov

Futurist, Singularity writer, co-directs the Immortality Institute for Infinite Lifespans

http://www.acceleratingfuture.com/michael - 1,200 results

Stephen Reed, Ph.D

Professor of Computer Science, AI researcher for Cycorp

http://www.sl4.org/archive/0205/3754.html - 919 results

Joseph N. Pelton, Ph.D

Research Professor for the Institute for Applied Space Research

http://www.clarkeinstitute.com/lecture8.html - 851 results

Eric Boyd, Ph.D

Professor of Computer Science, entrepreneur, software architect

http://www.eecs.umich.edu/PPP/eric-boyd.html - 820 results

Gordon Worley

Programmer, Singularity advocate, transhumanist

http://homepage.mac.com/redbird/ - 800 results

James N. Gardner

Complexity theorist, science essayist, cosmologist, author of The Selfish Biocosm Hypothesis

http://www.biocosm.org/ - 500 results

Please expand this article!