© 2004 Robert A. Freitas Jr. and Ralph C. Merkle. All Rights Reserved.

Robert A. Freitas Jr., Ralph C. Merkle, Kinematic Self-Replicating Machines, Landes Bioscience, Georgetown, TX, 2004.

5.11 Replicators and Public Safety

It has been pointed out that the development and deployment of artificial replicators capable of unconstrained replication using the natural environment as substrate would raise serious public safety and environmental concerns [199, 2908-2911]. Perhaps one of the earliest scientists to voice this concern was Freeman Dyson, who opined, in a 1972 lecture [2934], that:

I assume that in the next century, partly imitating the processes of life and partly improving on them, we shall learn to build self-reproducing machines programmed to multiply, differentiate, and coordinate their activities as skillfully as the cells of a higher organism such as a bird. After we have constructed a single egg machine and supplied it with the appropriate computer program, the egg and its progeny will grow into an industrial complex capable of performing economic tasks of arbitrary magnitude. It can build cities, plant gardens, construct electric power-generating facilities, launch space ships, or raise chickens. The overall programs and their execution will remain always under human control.

The effects of such a powerful and versatile technology on human affairs are not easy to foresee. Used unwisely, it offers a rapid road to ecological disaster. Used wisely, it offers a rapid alleviation of all the purely economic difficulties of mankind. It offers to rich and poor nations alike a rate of growth of economic resources so rapid that economic constraints will no longer be dominant in determining how people are to live. In some sense this technology will constitute a permanent solution of man’s economic problems. Just as in the past, when economic problems cease to be pressing, we shall find no lack of fresh problems to take their place.

Self-replicating systems may be the key to low cost manufacturing but there is no need to allow such systems to function freely in the outside world. Restricted to an artificial and controlled environment, factory replicators can manufacture simpler and more rugged applications products which are then transferred to the end user. For example, medical devices designed to operate in the human body [228-236] or inside any other living thing should not self-replicate (see Nanomedicine [228], Sections 1.3.3 and 2.4.2). Such devices can be manufactured in a controlled environment, then injected or implanted into the patient as required, and later removed when the job is done. The resulting medical devices will be simpler, smaller, more efficient and more precisely designed for the task at hand than devices designed to perform the same function and to self-replicate [2928]. Similar considerations apply to nonmedical applications of molecular nanotechnology.

Given the potential for accident and abuse [2, 199, 2909] (and see Section 6.3.1), artificial replicators will almost certainly be very tightly regulated by governments everywhere. For example, it is unlikely that the FDA (or its future or overseas equivalent) would ever approve for general use a nonbiological medical nanodevice that was capable of in vivo replication, evolution, or mutation. Guidelines to avoid accidents and foreseeable abuses have been promulgated for biotechnology replicators [2929] and have also been proposed for nanotechnology replicators [271]. There are many technical routes to ensuring “safe replication,” most notably the broadcast architecture for control (Section 4.11.3.3) and the vitamin architecture for materials (Section 4.3.7) which eliminate almost any possibility that the system can replicate outside of a very controlled and highly artificial setting. Interestingly, the Heifer Project [2930] is an example of a safe self-replicating “technology” being philanthropically distributed to impoverished Third World countries to help combat world poverty and hunger.*

* The problem of a runaway replicative factory system producing an uncontrolled overabundance, ostensibly for the benefit of human beings, was explored by Philip K. Dick in his 1955 story “Autofac” [668]:

Cut into the base of the mountains lay the vast metallic cube of the Kansas City factory. Its surface was corroded, pitted with radiation pox, cracked and scarred from the five years of war that had swept over it. Most of the factory was buried subsurface, only its entrance stages visible....

“The Institute of Applied Cybernetics had complete control over the network. Blame the war...we can’t transmit our information to the factories – the news that the war is over and we’re ready to resume control of industrial operations.”

“And meanwhile,” Morrison added sourly, “the damn network expands and consumes more of our natural resources all the time.”

“Isn’t there some limiting injunction?” Perine asked nervously. “Were they set up to expand indefinitely?”

“Each factory is limited to its own operational area,” O’Neill said, “but the network itself is unbounded. It can go on scooping up our resources forever....It’s already used up half a dozen basic minerals. Its search teams are out all the time, from every factory, looking everywhere for some last scrap to drag home....Each factory has its own special section of our planet, its own private cut of the pie for its exclusive use...They’re raw material-tropic; as long as there’s anything left, they’ll hunt it down.”

Fears that self-replicating machines or “machine life” could get out of control and even displace all biological life have been voiced by writers in the 1910s [2931], 1920s [657, 658], 1930s [660, 2932], 1940s [2933], 1950s [663-668], 1960s [669-673], 1970s [2934-2938], 1980s [2, 199, 2896], 1990s [2910, 2939], and 2000s [2898-2909], though others are more hopeful [2887, 2940]. Possible procedures and safeguards for the safe development of molecular-scale self-replicating systems were first discussed in Drexler’s Engines of Creation [199], and later by Forrest [2941] who listed a number of important regulatory issues and noted that “safe designs, safe procedures and methods to test for potentially hazardous assemblers can be incorporated into standards by consensus of interested parties.” Regulation continued to be discussed within the membership of the Foresight Institute throughout the 1990s. These concepts were further developed in a workshop sponsored by Foresight Institute and the Institute for Molecular Manufacturing (IMM), and finally formalized in the “Foresight Guidelines on Molecular Nanotechnology” during 1999-2000. Neil Jacobstein of IMM conducted several open workshops on the Guidelines, and incorporated the resulting feedback into several subsequent draft versions. Both authors of this book (Freitas and Merkle) and many others participated [2942] in the drafting of the final Guidelines document [271].

The core of the Guidelines [271] consists of three parts – first, a set of general principles; second, a set of development principles; and third, a set of specific design guidelines for molecular nanotechnology (MNT). These principles are crudely analogous to the original NIH Guidelines on Recombinant DNA technology [2929] in the field of biotechnology, an example of self-regulation first adopted by the biotechnology community almost three decades ago.

The Foresight general principles recommend that:

(1) People who work in the MNT field should develop and utilize professional guidelines that are grounded in reliable technology, and knowledge of the environmental, security, ethical, and economic issues relevant to the development of MNT.

(2) Access to the end products of MNT should be distinguished from access to the various forms of the underlying development technology. Access to MNT products should be unrestricted unless this access poses a risk to global security.

(3) Accidental or willful misuse of MNT must be constrained by legal liability and, where appropriate, subject to criminal prosecution.

(4) Governments, companies, and individuals who refuse or fail to follow responsible principles and guidelines for development and dissemination of MNT should, if possible, be placed at a competitive disadvantage with respect to access to MNT intellectual property, technology, and markets.

(5) The global community of nations and non-governmental organizations need to develop effective means of restricting the misuse of MNT. Such means should not restrict the development of peaceful applications of the technology or defensive measures by responsible members of the international community.

(6) MNT research and development should be conducted with due regard to existing principles of ecological and public health. MNT products should be promoted which incorporate systems for minimizing negative ecological and public health impact.

(7) Any specific regulation adopted by researchers, industry or government should provide specific, clear guidelines. Regulators should have specific and clear mandates, providing efficient and fair methods for identifying different classes of hazards and for carrying out inspection and enforcement. There is great value in seeking the minimum necessary legal environment to ensure the safe and secure development of this technology.

The set of development principles includes:

(1) Artificial replicators must not be capable of replication in a natural, uncontrolled environment.

(2) Evolution within the context of a self-replicating manufacturing system is discouraged.

(3) Any replicated information should be error free.

(4) MNT device designs should specifically limit proliferation and provide traceability of any replicating systems.

(5) Developers should attempt to consider systematically the environmental consequences of the technology, and to limit these consequences to intended effects. This requires significant research on environmental models, risk management, as well as the theory, mechanisms, and experimental designs for built-in safeguard systems.

(6) Industry self-regulation should be designed in whenever possible. Economic incentives could be provided through discounts on insurance policies for MNT development organizations that certify Guidelines compliance. Willingness to provide self-regulation should be one condition for access to advanced forms of the technology.

(7) Distribution of molecular manufacturing development capability should be restricted, whenever possible, to responsible actors that have agreed to use the Guidelines. No such restriction need apply to end products of the development process that satisfy the Guidelines.

The set of specific design guidelines includes:

(1) Any self-replicating device which has sufficient onboard information to describe its own manufacture should encrypt it such that any replication error will randomize its blueprint.

(2) Encrypted MNT device instruction sets should be utilized to discourage irresponsible proliferation and piracy.*

(3) Mutation (autonomous and otherwise) outside of sealed laboratory conditions, should be discouraged.

(4) Replication systems should generate audit trails.

(5) MNT device designs should incorporate provisions for built-in safety mechanisms, such as: (a) absolute dependence on a single artificial fuel source or artificial “vitamins” that don’t exist in any natural environment; (b) making devices that are dependent on broadcast transmissions for replication or in some cases operation; (c) routing control signal paths throughout a device, so that subassemblies do not function independently; (d) programming termination dates into devices; and (e) other innovations in laboratory or device safety technology developed specifically to address the potential dangers of MNT.

(6) MNT developers should adopt systematic security measures to avoid unplanned distribution of their designs and technical capabilities.

* Freeman Dyson [2943] observes: “The idea which I found most interesting was the emphasis on encryption as a safeguard against hijacking. This raises a new question for biologists. All living creatures are fighting a constant battle against hijacking of their genetic and metabolic apparatus by viruses. Is there any evidence that cells have evolved encryption of vital genes and polymerase enzymes to stop viruses from taking over the apparatus? Could this be a reason for the subdivision of eukaryotic genes into exons and introns?”

In a June 2000 Foresight Institute press release [2944], Zyvex Corp. became the first corporate supporter of the Guidelines principles. James Von Ehr, President and CEO of Zyvex, said: “This is an important document that deserves careful reading by all concerned. As one of the first nanotechnology companies, Zyvex fully supports the Guidelines and is pleased that two of our senior scientists [Merkle and Freitas] were able to participate in their preparation. I expect that a sense of professional ethics will compel nanotechnology developers to individually subscribe to these principles. These Guidelines are so important that grant-making agencies, even military ones, should require a pledge of adherence as a precondition of funding advanced nanotechnology development.” The Foresight Guidelines appear to be considerably more restrictive than, for example, the “Bioethics Statement of Principles” [2945] to which companies at the forefront of engineered biological replicator development, such as Egea Biosciences [2946], subscribe. Indeed, Freeman Dyson [2943] believes that the Guidelines may be “a bit too cautious. But they can always be relaxed later as experience accumulates, just as the recombinant DNA guidelines were relaxed.”

During 2000-2003, Neil Jacobstein moderated several seminars at the Aspen Institute [2947] that incorporated the Foresight Guidelines as part of a larger dialog on “Opportunity, Risk and Responsibility” regarding nanotechnology. These seminars, as well as other workshops and presentations, served to expose the Guidelines to an increasingly larger community, and to lay the foundation for potential future revisions.

In April 2003, the Foresight President (Christine Peterson) [2948] and a Zyvex Board Member (Ray Kurzweil) [2949] were invited to testify before the U.S. House Science Committee, and later to advise on the wording of amendments to the “societal and ethical” part of the House Bill. The Foresight Guidelines were subsequently employed in producing the wording of an amendment to the Bill. H.R. 766, the National Nanotechnology Research and Development Act of 2003 [2950], which passed the House by an overwhelming majority on 7 May 2003 and was passed in similar form by the Senate on 22 June 2003 [2951]. H.R. 766 Section 8(c) specifically calls for “A Study On Safe Nanotechnology” by the National Academy of Sciences and required, in the original* House language:

Not later than 6 years after the date of enactment of this Act a review shall be conducted in accordance with subsection (a) that includes a study to assess the need for standards, guidelines, or strategies for ensuring the development of safe nanotechnology, including those applicable to — (1) self-replicating nanoscale machines or devices; (2) the release of such machines or devices in natural environments; (3) distribution of molecular manufacturing development; (4) encryption; (5) the development of defensive technologies; (6) the use of nanotechnology as human brain extenders; and (7) the use of nanotechnology in developing artificial intelligence.

* Unfortunately, the final language of the bill that was signed into law calls only for a study on “molecular self-assembly,” or more specifically: “Section 5(b) Study on Molecular Self-Assembly – The National Research Council shall conduct a one-time study to determine the technical feasibility of molecular self-assembly for the manufacture of materials and devices at the molecular scale.” [2952] Observes Glenn Reynolds [2953]: “Given that self-assembling nanodevices have already been demonstrated, taking a narrow view of this language seems unlikely to accomplish much. It’s like performing a study to determine the feasibility of integrated circuit chips. Been there, done that. Presumably, the broader interpretation of the language will obtain. If it doesn’t, that may be an early sign that federal officials aren’t really serious about developing what most people would consider to be true molecular manufacturing. Let’s hope it doesn’t.”

The issue of public safety in connection with possible future nanotechnology-based replicators, though still downplayed in the European Community [3114], has already begun to attract National Science Foundation (NSF) funding in the United States. The March 2001 NSF report [2962], Societal Implications of Nanoscience and Nanotechnology, “produced a template for discussion but left particular investigations for the future.” [2963] Soon thereafter, the University of South Carolina (USC) began “the first university-based interdisciplinary initiative to bring close scrutiny to this new area of science and technology,” a series of studies which first received NSF funding in 2002 [2963] and is planned to run through 2003-2007. These studies will specifically examine, among other related topics, the “problems of self-replication, risk, and cascading effects in nanotechnology.” In August 2003, the NSF announced [2954] an award of $1.3 million [2955] funding for this effort.

According to the USC proposal [2963], the Task Area 3 group will consider the management of risk in three clusters of investigation:

(1) Models of Self-Replication and Self-Regulation. The researchers will “articulate the range of meanings encompassed by self-replication and self-regulation, from a simpler, bench-oriented model to the vision associated with assemblers, and everything in between. Nanoscientists have already developed a variety of new materials that show promise for nano-engineered products – nanotubes, electrically conducting compounds, quantum dots, etc. Now, they need ways to organize these materials into larger structures that might be useful to society. In order to do this, they have focused upon mechanisms of replication and the regulation of these mechanisms. But, as the complexity of a self-replicating process increases, the possibility of an undesirable medical or environmental outcome seems likely to increase as well, and there are additional concerns about potential mutations of the original process. In order to help anticipate and prepare for such possibilities, Task Area 3 team members will seek to identify the multiple models and meanings of self-replication and self-regulation, ranging from current techniques (e.g., for growing nanotubes) to universal assemblers. In between, we consider possibilities on the near horizon (e.g., the use of viruses to engineer at the nanoscale) and the more distant horizon (e.g., limited assemblers, the stated goal of the company Zyvex [2956]).”

“Task Area 3 members will approach this work by drawing on analogies between these engineered mechanisms and those found in natural biological systems. In order to appreciate the challenges involved in designing and manufacturing nano-structures capable of self-replication and correction without loss of control, they will examine the properties of natural self-replicating systems. What methods does nature use for self-replication? Will nanotechnologies resemble genetic systems in such a way that an understanding of the natural principles governing the latter might guide the development and application of the former in safe and controllable ways? In what ways will they differ? Armed with this understanding we will be able to explore the philosophical and ethical implications of aspects of self-regulation.” The target publication date for this phase of the USC work is Spring 2005, and an edited volume on these issues will follow the Summer 2004 Workshop and the Spring 2005 Conference – both on these topics – in Fall 2006.

(2) Taxonomy of Risk Assessments for Nanotechnology. “Scientists know that complex, non-linear, self-replicating systems can produce unanticipated medical and/or environmental harm. In some cases statisticians can quantify risks associated with such systems, but in many other cases the uncertainty is too great, and the best that can be done is to provide a less precise qualitative analysis [2957]. Along these lines, Task Area 3 team members will develop a taxonomy of the kinds of risk assessment that could be used in ethical debates on nanotechnology. First, they will identify risk paradigms for possible medical and environmental outcomes (e.g., the way a new virus can pose a public health risk). Then they will consider whether the associated risks could have been anticipated and quantified in a risk analysis. They will examine cases where established methods of quantifying risk worked well and cases where the outcomes could not have been anticipated and quantified. Next they will draw on their earlier work, developing the analogy between engineered and natural nanosystems, and they will extend this analysis to consider the possibilities of quantifying risks associated with the types of self-replicating, artificially engineered nanosystems identified earlier. The goal is to identify and structure the variety of cases posed by nanotechnology in terms of the degree to which the risks can or cannot be quantified. Finally, within this structure they will consider how such risks can and should be incorporated into ethical analysis and communicated to the public [2958].” The target publication date for this phase is Spring 2006.

(3) The Literature and Culture Informing Ethical Analysis of Nanotechnology. “There are several new areas of research that involve significant challenges to our understanding of ourselves and our prospects in the world. These include (i) robotics/cybernetics, (ii) genetics, and (iii) nanotechnology. In most of this literature, these three technologies are considered in isolation. However, some of the most troubling ethical issues occur where all three technologies intersect. Task Area 3 members will explore analogies, similarities, and differences between the ethical discussions in each of these areas and then consider how their combination could raise issues that have been insufficiently considered when viewed in isolation. The focus here will not only be on the substance of the issues, but also on the climate and culture that frames the way the issues are addressed and resolved.” The target publication date for this phase of the USC work is Spring 2007.

The study of the ethical [2959], socioeconomic [2, 2960-2963] and legal [2964-2967] impact of machine replicators is still in its earliest stages, and additional discussion of safety issues may be found in Sections 2.1.5, 2.3.6, 5.1.9(L), 6.3.1 and 6.4.4. However, two important general observations about replicators and self-replication should be noted here.

First, replication is nothing new. Humanity has thousands, arguably even millions, of years of experience living with entities that are capable of kinematic self-replication. These replicators range from the macroscale (e.g., insects, birds, horses, other humans) on down to the microscale (e.g. bacteria, protozoa) and even the nanoscale (e.g., prions, viruses). As a species, we have successfully managed the eternal tradeoff between risk and reward, and have successfully negotiated the antipodes of danger and progress. There is every reason to expect this success to continue.

The technology of engineered self-replication, even at the microscale, is already in wide commercial use throughout the world. Indeed, human civilization is utterly dependent on self-replication technologies. Many important foods including beer, wine, cheese, yogurt, and kefir (a fermented milk), along with various flavors, nutrients, vitamins and other food ingredients, are produced by specially cultured microscopic replicators such as algae, fungi (yeasts) and bacteria. Virtually all of the rest of our food is made by macroscale replicators such as agricultural crop plants, trees, and farm animals. Many of our most important drugs are produced using microscopic self-replicators – from penicillin produced by natural replicating molds starting in the 1940s [228] to the first use of artificial (engineered) self-replicating bacteria to manufacture human insulin by Eli Lilly in 1982 [2968]. These uses continue today in the manufacture of many other drug products such as: (a) human growth hormone (HGH) and erythropoietin (EPO), (b) precursors for antibiotics such as erythromycin [2969], and (c) therapeutic proteins such as Factor VIII. A few species of self-replicating bacteria are used directly as therapeutic medicines, such as the widely available swallowable pills containing bacteria (i.e., natural biological nanomachines) for gastrointestinal refloration, as for example Salivarex^TM which “contains a minimum of 2.9 billion beneficial bacteria per capsule” [2970], and Alkadophilus^TM which “contains 1.5 billion organisms per capsule” [2971], both at a 2003 price of ~$(0.1-0.2) x 10^-9 per microscale replicator (i.e., per bacterium). Some replicating viruses, notably bacteriophages, are used as therapeutic agents to combat and destroy unhealthful infectious bacterial replicators [1757], and for decades viruses have served as transfer vectors to attempt gene therapies [2972-2975]. In industry, bacteria are already employed as “self-replicating factories” [1779-1781] for various useful products, and microorganisms are also used as workhorses for environmental bioremediation* [2976-2979], biomining of heavy metals [2980-2983], and other applications. In due course we will learn to safely harness the abilities of nonbiological kinematic machine replicators for human benefit as well.

* According to Press [2979]: “The first patented form of life produced by genetic engineering was a greatly enhanced oil-eating microbe. The patent [2984, 2985] was registered to Dr. Ananda Chakrabarty of the General Electric Company in 1981 and was initially welcomed as an answer to the world’s petroleum pollution problem. But anxieties about releasing ‘mutant bacteria’ soon led the U.S. Congress and the Environmental Protection Agency (EPA) to prohibit the use of genetically engineered microbes outside of sealed laboratories. The prohibition set back bioremediation for a few years, until scientists developed improved forms of oil-eating bacteria without using genetic engineering. After large-scale field tests in 1988, the EPA reported that bioremediation eliminated both soil and water-borne oil contamination at about one-fifth the cost of previous methods. Since then, bioremediation has been increasingly used to clean up oil pollution on government sites across the United States.”

Second, replicators can be made inherently safe. An “inherently safe” kinematic replicator is a replicating system which, by its very design, is inherently incapable of surviving mutation or of undergoing evolution (and thus evolving out of our control or developing an independent agenda), and which, equally importantly, does not compete with biology for resources (or worse, use biology as a raw materials resource [2909]). One primary route for ensuring inherent safety is to employ the broadcast architecture for control (Section 4.11.3.3) and the vitamin architecture for materials (Section 4.3.7), which eliminate the likelihood that the system can replicate outside of a very controlled and highly artificial setting, and there are numerous other routes to this end (Section 5.1.9 and see Guidelines, above). Many dozens of additional safeguards may be incorporated into replicator designs to provide redundant embedded controls and thus an arbitrarily low probability of replicator malfunctions of various kinds, simply by selecting the appropriate design parameters as described in Section 5.1.9.

Artificial kinematic self-replicating systems which are not inherently safe should not be designed or constructed, and indeed should be legally prohibited by appropriate juridical and economic sanctions, with these sanctions to be enforced in both national and international regimes. In the case of individual lawbreakers or rogue states that might build and deploy unsafe artificial mechanical replicators, the defenses we have already developed against harmful biological replicators all have analogs in the mechanical world that should provide equally effective, or even superior, defenses. Molecular nanotechnology will make possible ever more sophisticated methods of environmental monitoring and prophylaxis. However, advance planning and strategic foresight will be essential in maintaining this advantage. (See also Section 6.3.1.)

Last updated on 1 August 2005