© 2004 Robert A. Freitas Jr. and Ralph C. Merkle. All Rights Reserved.

Robert A. Freitas Jr., Ralph C. Merkle, Kinematic Self-Replicating Machines, Landes Bioscience, Georgetown, TX, 2004.

6.3.1 Molecular Assemblers Are Too Dangerous

The first argument against a molecular assembler design effort is that the end results are too dangerous. According to this argument [2905, 2908], any research into molecular assemblers should be blocked because this technology might be used to build systems that could cause extraordinary damage. The kinds of concerns that nanoweapons systems might create have been discussed elsewhere, in both the nonfictional [2909, 3027-3030, 3113] and fictional [2898, 3031, 3032] literature. Perhaps the earliest-recognized and best-known danger of molecular nanotechnology is the risk that self-replicating nanorobots capable of functioning autonomously in the natural environment could quickly convert that natural environment (e.g., “biomass”) into replicas of themselves (e.g., “nanomass”) on a global basis, a scenario usually referred to as the “gray goo problem” but more accurately termed “global ecophagy” [2909]. As Drexler first warned in Engines of Creation in 1986 [199]:

“Plants” with “leaves” no more efficient than today’s solar cells could out-compete real plants, crowding the biosphere with an inedible foliage. Tough omnivorous “bacteria” could out-compete real bacteria: They could spread like blowing pollen, replicate swiftly, and reduce the biosphere to dust in a matter of days. Dangerous replicators could easily be too tough, small, and rapidly spreading to stop – at least if we make no preparation....We cannot afford certain kinds of accidents with replicating assemblers.

Such self-replicating systems, if not countered, could make the earth largely uninhabitable [199, 666, 695, 2898, 2909] – concerns that motivated the drafting of the Foresight Guidelines for the safe development of nanotechnology [271] (Section 5.11). But as the Center for Responsible Nanotechnology explains [3029]:

Gray goo would entail five capabilities integrated into one small package. These capabilities are: Mobility – the ability to travel through the environment; Shell – a thin but effective barrier to keep out diverse chemicals and ultraviolet light; Control – a complete set of blueprints and the computers to interpret them (even working at the nanoscale, this will take significant space); Metabolism – breaking down random chemicals into simple feedstock; and Fabrication – turning feedstock into nanosystems. A nanofactory would use tiny fabricators, but these would be inert if removed or unplugged from the factory. The rest of the listed requirements would require substantial engineering and integration [2909].

Although gray goo has essentially no military and no commercial value, and only limited terrorist value, it could be used as a tool for blackmail. Cleaning up a single gray goo outbreak would be quite expensive and might require severe physical disruption of the area of the outbreak (atmospheric and oceanic goos [2909] deserve special concern for this reason). Another possible source of gray goo release is irresponsible hobbyists. The challenge of creating and releasing a self-replicating entity apparently is irresistible to a certain personality type, as shown by the large number of computer viruses and worms in existence. We probably cannot tolerate a community of “script kiddies”* releasing many modified versions of goo.

Development and use of molecular manufacturing poses absolutely no risk of creating gray goo by accident at any point. However, goo type systems do not appear to be ruled out by the laws of physics, and we cannot ignore the possibility that the five stated requirements could be combined deliberately at some point, in a device small enough that cleanup would be costly and difficult. Drexler’s 1986 statement can therefore be updated: We cannot afford criminally irresponsible misuse of powerful technologies. Having lived with the threat of nuclear weapons for half a century, we already know that.

* According to cyberjournalist Clive Thompson [3034], elite writers of software viruses openly publish their code on Web sites, often with detailed descriptions of how the program works, but don’t actually release them. The people who do release the viruses are often anonymous mischief-makers, or “script kiddies” – a derisive term for aspiring young hackers, “usually teenagers or curious college students, who don’t yet have the skill to program computers but like to pretend they do. They download the viruses, claim to have written them themselves and then set them free in an attempt to assume the role of a fearsome digital menace. Script kiddies often have only a dim idea of how the code works and little concern for how a digital plague can rage out of control. Our modern virus epidemic is thus born of a symbiotic relationship between the people smart enough to write a virus and the people dumb enough – or malicious enough – to spread it.”

Thompson goes on to describe his early 2004 visit to an Austrian programmer named Mario, who cheerfully announced that in 2003 he had created, and placed online at his website, freely available, a program called “Batch Trojan Generator” that autogenerates malicious viruses. Thompson described a demonstration of this program: “A little box appears on his laptop screen, politely asking me to name my Trojan. I call it the ‘Clive’ virus. Then it asks me what I’d like the virus to do. Shall the Trojan Horse format drive C:? Yes, I click. Shall the Trojan Horse overwrite every file? Yes. It asks me if I’d like to have the virus activate the next time the computer is restarted, and I say yes again. Then it’s done. The generator spits out the virus onto Mario’s hard drive, a tiny 3KB file. Mario’s generator also displays a stern notice warning that spreading your creation is illegal. The generator, he says, is just for educational purposes, a way to help curious programmers learn how Trojans work. But of course I could ignore that advice.”

Apparently top “malware” writers do take some responsible precautions, notes Thompson. For example, one hacker’s “main virus-writing computer at home has no Internet connection at all; he has walled it off like an airlocked biological-weapons lab, so that nothing can escape, even by accident.” Some writers, after finishing a new virus, “immediately e-mail a copy of it to antivirus companies so the companies can program their software to recognize and delete the virus should some script kiddie ever release it into the wild.”

Attempts to block or “relinquish” [2908, 3035] molecular nanotechnology research will make the world a more, not less, dangerous place [3036]. This paradoxical conclusion is founded on two premises. First, attempts to block the research will fail. Second, such attempts will preferentially block or slow the development of defensive measures by responsible groups. One of the clear conclusions reached by Freitas [2909] was that effective countermeasures against self-replicating systems should be feasible, but will require significant effort to develop and deploy. (Nanotechnology critic Bill Joy, responding to Freitas, complained in late 2000 that any nanoshield defense to protect against global ecophagy “appears to be so outlandishly dangerous that I can’t imagine we would attempt to deploy it.” [3035]) But blocking the development of defensive systems would simply insure that offensive systems, once deployed, would achieve their intended objective in the absence of effective countermeasures. James Hughes [3036] concurs: “The only safe and feasible approach to the dangers of emerging technology is to build the social and scientific infrastructure to monitor, regulate and respond to their threats.”

We can reasonably conclude that blocking the development of defensive systems would be an extraordinarily bad idea. Actively encouraging rapid development of defensive systems by responsible groups while simultaneously slowing or hindering development and deployment by less responsible groups (“nations of concern”) would seem to be a more attractive strategy, and is supported by the Foresight Guidelines [271]. As even nanotechnology critic Bill Joy [3037] finally admitted in late 2003: “These technologies won’t stop themselves, so we need to do whatever we can to give the good guys a head start.”

While a 100% effective ban against development might theoretically be effective at avoiding the potential adverse consequences, blocking all groups for all time does not appear to be a feasible goal. The attempt would strip us of defenses against attack, increasing rather than decreasing the risks. In addition, blocking development would insure that the substantial economic, environmental, and medical benefits [228] of this new technology would not be available.

Observes Glenn Reynolds [3038]: “To the extent that such efforts [to ban all development] succeed, the cure may be worse than the disease. In 1875, Great Britain, then the world’s sole superpower, was sufficiently concerned about the dangers of the new technology of high explosives that it passed an act barring all private experimentation in explosives and rocketry. The result was that German missiles bombarded London rather than the other way around. Similarly, efforts to control nanotechnology, biotechnology or artificial intelligence are more likely to drive research underground (often under covert government sponsorship, regardless of international agreement) than they are to prevent research entirely. The research would be conducted by unaccountable scientists, often in rogue regimes, and often under inadequate safety precautions. Meanwhile, legitimate research that might cure disease or solve important environmental problems would suffer.”

Finally, and as previously explained (e.g., Sections 3.13.2.2, 4.9.3, 4.14, 4.17, 4.19, 5.7, and 5.9.4), it is well-known [208] that self-replication activities, as distinct from the inherent capacity for self-replication, are not strictly required to achieve the anticipated broad benefits of molecular manufacturing. By restricting the capabilities of nanomanufacturing systems simultaneously along multiple design dimensions (Section 5.1.9) such as control autonomy (A1), nutrition (E4), mobility (E10) and immutability (L3, L4), molecular manufacturing systems – whether microscale or macroscale – can be made inherently safe. As Drexler [3106] noted in a 2004 paper:

In 1959, Richard Feynman pointed out that nanometer-scale machines could be built and operated, and that the precision inherent in molecular construction would make it easy to build multiple identical copies. This raised the possibility of exponential manufacturing, in which production systems could rapidly and cheaply increase their productive capacity, which in turn suggested the possibility of destructive runaway self-replication. Early proposals for artificial nanomachinery focused on small self-replicating machines, discussing their potential productivity and their potential destructiveness if abused.... [But] nanotechnology-based fabrication can be thoroughly non-biological and inherently safe: such systems need have no ability to move about, use natural resources, or undergo incremental mutation. Moreover, self-replication is unnecessary: the development and use of highly productive systems of nanomachinery (nanofactories) need not involve the construction of autonomous self-replicating nanomachines.... Although advanced nanotechnologies could (with great difficulty and little incentive) be used to build such devices, other concerns present greater problems. Since weapon systems will be both easier to build and more likely to draw investment, the potential for dangerous systems is best considered in the context of military competition and arms control.

Of course, it must be conceded that while nanotechnology-based manufacturing systems can be made safe, they could also be made dangerous. Just because free-range self-replicators may be “undesirable, inefficient and unnecessary” [3106] does not imply that they cannot be built, or that nobody will build them. How can we avoid “throwing out the baby with the bathwater”? The correct solution, first explicitly proposed by Freitas [2909] in 2000* and later partially echoed by Phoenix and Drexler [3106] in 2004,** starts with a carefully targeted moratorium or outright legal ban on the most dangerous kinds of nanomanufacturing systems, while still allowing the safe kinds of nanomanufacturing systems to be built – subject to appropriate monitoring and regulation commensurate with the lesser risk that they pose.

* Freitas (2000) [2909]: “Specific public policy recommendations suggested by the results of the present analysis include: (1) an immediate international moratorium on all artificial life experiments implemented as nonbiological hardware. In this context, ‘artificial life’ is defined as autonomous foraging replicators, excluding purely biological implementations (already covered by NIH guidelines tacitly accepted worldwide) and also excluding software simulations which are essential preparatory work and should continue. Alternative ‘inherently safe’ replication strategies such as the broadcast architecture are already well-known....”

** Phoenix and Drexler (2004) [3106]: “The construction of anything resembling a dangerous self-replicating nanomachine can and should be prohibited.”

Virtually every known technology comes in “safe” and “dangerous” flavors which necessarily must receive different legal treatment. For example, over-the-counter drugs are the safest and most difficult to abuse, hence are lightly regulated; prescription drugs, more easy to abuse, are very heavily regulated; and other drugs, typically addictive narcotics and other recreational substances, are legally banned from use by anyone, even for medicinal purposes. Artificial chemicals can range from lightly regulated household substances such as Clorox or ammonia; to more heavily regulated compounds such as pesticides, solvents and acids; to the most dangerous chemicals such as chemical warfare agents which are banned outright by international treaties. Another example is pyrotechnics, which range from highway flares, which are safe enough to be purchased and used by anyone; to “safe and sane” fireworks, which are lightly regulated but still available to all; to moderately-regulated firecrackers and model rocketry; to minor explosives and skyrockets, which in most states can be legally obtained and used only by licensed professionals who are heavily regulated; to high-yield plastic explosives, which are legally accessible only to military specialists; to nuclear explosives, the possession of which is strictly limited to a handful of nations via international treaties, enforced by an international inspection agency. Yet another example is aeronautics technology, which ranges from safe unregulated kites and paper airplanes; to lightly regulated powered model airplanes operated by remote control; to moderately regulated civilian aircraft, both small and large; to heavily regulated military attack aircraft such as jet fighters and bombers, which can only be purchased by approved governments; to intercontinental ballistic missiles, the possession of which is strictly limited to a handful of nations via international treaties.

Note that in all cases, the existence of a “safe” version of a technology does not preclude the existence of a “dangerous” version, and vice versa. The laws of physics permit both versions to exist. The most rational societal response has been to classify the various applications according to the risk of accident or abuse that each one poses, and then to regulate each application accordingly. The societal response to the tools and products of molecular nanotechnology will be no different. Some MNT-based tools and products will be deemed safe, and will be lightly regulated. Other MNT-based tools and products will be deemed dangerous, and will be heavily regulated, or even legally banned in some cases.

Of course, the mere existence of legal restrictions or outright bans does not preclude the acquisition and abuse of a particular technology by a small criminal fraction of the population. For instance, in the high-risk category, drug abusers obtain and inject themselves with banned narcotics; outlaw regimes employ prohibited poison chemicals in warfare; and rogue nations seek to enter the “nuclear club” via clandestine atomic bomb development programs. Bad actors such as terrorists can also abuse less-heavily regulated products such as fully-automatic rifles or civilian airplanes (which are hijacked and flown into buildings). The most constructive response to this class of threat is to increase monitoring efforts to improve early detection and to pre-position defensive instrumentalities capable of responding rapidly to these abuses, as recommended in 2000 by Freitas [2909] in the molecular nanotechnology context.

The least problematic danger of a new technology is the risk of accident or malfunction. Engineers generally try to design products that work reliably and companies generally seek to sell reliable products to maintain customer goodwill and to avoid expensive product liability lawsuits. But accidents do happen. Here again, our social system has established a set of progressive responses to deal efficiently with this problem. A good example is the ancient technology of fire. The uses of fire are widespread in society, ranging from lightly-regulated matchsticks, butane lighters, campfires, and internal combustion engines, to more heavily regulated home HVAC furnaces, municipal incinerators and industrial smelters. A range of methods are available to deal quickly and effectively with a fire that has accidentally escaped the control of its user. Home fires due to a smoldering cigarette or a blazing grease pan in the kitchen are readily doused using a common household fire extinguisher. Fires in commercial buildings (e.g., hotels) or industrial buildings (e.g., factories) are automatically quenched by overhead sprinkler systems. When these methods prove insufficient to snuff out the flames, the local fire department is called in to limit the damage to just a single building, using fire trucks, water hoses and hydrants. If many buildings are involved, more extensive fire suppression equipment and hundreds of firefighters can be brought in from all across town to hold the damage to a single city block. In the case of the largest accidental fires, like forest fires, vast quantities of heavy equipment are deployed including thousands of firefighters wielding specialized tools, bulldozers to dig firebreaks, helicopters with pendulous water buckets, and great fleets of air tankers dropping tons of fire retardants. (These progressive measures also protect the public in cases of deliberate arson.) The future emergency response hierarchy for dealing with MNT-based accidents will be no less extensive and will be equally effective in preserving human life and property, while allowing us to enjoy the benefits of this new technology. Notes Steen Rasmussen of Los Alamos National Laboratory in New Mexico: “The more powerful technology you unleash, the more careful you have to be.” [3107]

Last updated on 1 August 2005