Long-term Equilibria

From Wise Nano

Long Term Equilibria

Totalitarian Oppression

Actually, "totalitarian" is too weak a word. A government that was determined to hold power over rebellious people, in a world of extremely advanced technology, would have both the ability and the necessity to be repressive beyond anything in human experience.

Possible means of oppression include:

• Technological backwardness. Advanced technology would make it possible to keep society going without letting very many people actually access the underlying technology. Everyone could have a factory that made any bland consumer good they wanted--but they wouldn't be allowed to own even a screwdriver or volt meter. It wouldn't take much surveillance to keep them from developing a competitive technology.
• Full-time monitoring, with data-mining, of all human activity. Software already exists to sort out unusual events in video streams; it was demonstrated at MIT several yeas ago. This would allow humans to focus only on unusual activities--a small fraction of the total time--so there could be enough time to scrutinize everything of importance.
• Psychiatric monitoring. As more is learned about the chemical correlates of emotional states, it will likely become possible to detect violent, rebellious, or terroristic mindsets. This could of course be applied to absolutely everyone, when medical monitors become extremely cheap and small.
• Psychiatric intervention. Turn everyone into Stepford Wives (the original, not the remake) or residents of Camazotz (serious link here; funny and explanatory parody of "Camelot" here).
• Kill (almost) everyone. If you're the first with molecular manufacturing, this would be remarkably easy to do, and there'd be no economic reason not to.

Artificial Intelligence

Molecular manufacturing will enable the construction of sensors and manipulators smaller and more numerous than those of which living systems are composed. As a result, its development enables the detailed study and precise modification of biological processes including thought. Once cognitive tasks can be monitored on their functional level, they can be modified, improved, or implemented in software. Several cognitive tasks, operating in conjunction with one another, can potentially produce a feedback loop leading to the unprecedentedly rapid development of new capabilities. For instance, there is a feedback loop between the design of novel tools and the design of novel design capabilities. The former enable the latter which enable the former. Because the set of possible MNT tools is known to include many potent devices that cannot be produced with current or forseeable engineering techniques and/or which cannot be controlled with current data-integration capabilities, the manipulative capabilities that will result from such a feedback loop are likely to be destabilizing. The policy implication of this is that the initial shift to MNT is the prelude to a much larger shift in available technology, a shift which is also likely to lead at least initially to unprescedented concentration of power. This conclusion bodes poorly for the prospects of nano-anarchy. Even if one is optimistic regarding the likely behavior of super- empowered individuals, and regarding the ability of many such people to collectively keep in check the dominating inclinations of any particular group, it appears inevitable that power will once again be centralized by a powerful innovation.

It is not only design capabilities that can be implemented in software. Intentions, values, and goals can also be implemented. In fact, many proposals for the construction of general problem solving machines involve the provision of open-ended and general goals. Such goals may have unexpected consequences, including subgoals which the program's creators might want to avoid. The runaway AI wiki describes the general class of possible unintended consequences of making goal-driven machines. Another possiblity is the goal system may modify itself in such a manner as to ultimately suppress its original goal. Avoiding this mistake should be [http://www.idsia.ch/~juergen/goedelmachine.html theoretically possible], but our only historical example of a general problem solving system seems to have made [http://www.orkut.com/CommMsgs.aspx?cmm=17172&tid=4 precisely this mistake]. At any rate, it is very unlikely that a runaway AI can be stopped once it improves past a certain point and it is very difficult to predict where that point lies. Runaway AIs have no reason to concern themselves with values outside of their goal, so a runaway AI with the wrong goal would destroy all life. Because life, and more generally "things people value" is something very difficult to define, it would be very difficult to produce an open ended goal for an AI that was not deadly.

There appear to be only two general ways in which this problem can potentially be prevented. The first is the development of Friendly AI systems to implement a goal system compatible with the preservation of human values prior to the creation of the first destructive runaway AI. The second is to prevent the development of general AI. Because, as explained in the first paragraph, there are very strong reasons to engage in activities which carry a risk of creating runaway AI, and because this risk is not unique to tasks that match stereotypes of "creating artificial intelligence", it is very unlikely that runaway AI can be prevented by social consensus. A global organization dedicated to preventing its creation might have to be (or decide to be) very restrictive with respect to permitted activities. Such restrictive behavior might create further motivation for the activities that could create a [http://wise-nano.org/w/Runaway_AI runaway AI]. Still, it is clear that there exists some level of restriction, that could prevent the development of any novel technologies (as a grotesque example, global lobotomization would surely be sufficient, although it would fail at the goal of preserving human values). There is no reason to believe that either the establishment and perfect enforcement of an acceptable but sufficient set of restrictions on cognitive engineering or Friendly AI are impossible, and strong reason to believe that both are theoretically possible, but currently there do not appear to be any provably workable proposals for preventing runaway AI through either of these measures.