23/47 Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns

GE.13-12776

Human Rights Council Twenty-third session

Agenda item 3 Promotion and protection of all human rights, civil,

political, economic, social and cultural rights,

including the right to development

Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns

Summary

Lethal autonomous robotics (LARs) are weapon systems that, once activated, can select and engage targets without further human intervention. They raise far-reaching concerns about the protection of life during war and peace. This includes the question of the extent to which they can be programmed to comply with the requirements of international humanitarian law and the standards protecting life under international human rights law. Beyond this, their deployment may be unacceptable because no adequate system of legal accountability can be devised, and because robots should not have the power of life and death over human beings. The Special Rapporteur recommends that States establish national moratoria on aspects of LARs, and calls for the establishment of a high level panel on LARs to articulate a policy for the international community on the issue.

Contents

Paragraphs Page

I. Introduction ............................................................................................................. 1 3

II. Activities of the Special Rapporteur ....................................................................... 2–25 3

A. Communications ............................................................................................ 2–3 3

B. Visits ............................................................................................................... 4–6 3

C. Press releases .................................................................................................. 7–15 3

D. International and national meetings ................................................................ 16–24 4

E. Intended future areas of research .................................................................... 25 5

III. Lethal autonomous robotics and the protection of life ........................................... 26–108 5

A. The emergence of LARs ................................................................................. 37–56 7

B. LARs and the decision to go to war or otherwise use force ............................ 57–62 11

C. The use of LARs during armed conflict ......................................................... 63–74 12

D. Legal responsibility for LARs ........................................................................ 75–81 14

E. The use of LARs by States outside armed conflict ......................................... 82–85 16

F. Implications for States without LARs ............................................................. 86–88 16

G. Taking human decision-making out of the loop ............................................. 89–97 16

H. Other concerns ................................................................................................ 98–99 18

I. LARs and restrictive regimes on weapons ...................................................... 100–108 19

IV. Conclusions ............................................................................................................. 109–112 20

V. Recommendations ................................................................................................... 113–126 21

A. To the United Nations ..................................................................................... 113–115 21

B. To regional and other inter-governmental organizations ............................... 116–117 22

C. To States ........................................................................................................ 118–121 22

D. To developers of robotic systems ................................................................... 122 22

E. To NGOs, civil society and human rights groups and the ICRC ................... 123–126 22

I. Introduction

1. The annual report of the Special Rapporteur on extrajudicial, summary and arbitrary executions, submitted to the Human Rights Council pursuant to its Resolution 17/5, focuses on lethal autonomous robotics and the protection of life.1

II. Activities of the Special Rapporteur

A. Communications

2. The present report covers communications sent by the Special Rapporteur between 16 March 2012 and 28 February 2013, and replies received between 1 May 2012 and 30 April 2013. The communications and responses from Governments are included in the following communications reports of special procedures: A/HRC/21/49; A/HRC/22/67 and A/HRC/23/51.

3. Observations on the communications sent and received during the reporting period are reflected in an addendum to the present report (A/HRC/23/47/Add.5).

B. Visits

4. The Special Rapporteur visited Turkey from 26 to 30 November 2012 and will visit Mexico from 22 April to 2 May 2013.

5. The Government of Mali has accepted the Special Rapporteur‟s visit requests and the Syrian Arab Republic views his proposal to visit the country positively. The Special Rapporteur thanks these Governments and encourages the Governments of Sri Lanka, the Republic of Madagascar and Pakistan to accept his pending requests for a visit.

6. Follow-up reports on missions undertaken by the previous mandate holder to Ecuador and Albania are contained in documents A/HRC/23/47/Add.3 and A/HRC/23/47/Add.4 respectively.

C. Press releases2

7. On 15 June 2012, the Special Rapporteur issued a joint statement with the Special Rapporteur on torture deploring the escalation of violence in the Syrian Arab Republic and called on all parties to renounce violence and lay down arms.

8. The Special Rapporteur issued several press releases with other mandate holders amongst others concerning aspects related to the right to life of human rights defenders in Honduras on 4 April 2012 and 1 October 2012; the Philippines, on 9 July 2012; and on 21 June 2012, he issued a press release to urge world governments, the international

1 The assistance of Tess Borden, Thompson Chengeta, Jiou Park and Jeff Dahlberg in writing this

report is acknowledged with gratitude. The European University Institute is also thanked for hosting an expert consultation in February 2013, as well as the Global Justice Clinic, the Centre for Human Rights and Global Justice, and Professor Sarah Knuckey of New York University School of Law for preparing background materials and hosting an expert consultation in October 2012.

2 Press releases of the Special Rapporteur are available from www.ohchr.org/en/NewsEvents/Pages/NewsSearch.aspx?MID=SR_Summ_Executions.

community, journalists and media organizations to act decisively on the protection of the right to life of journalists and media freedom;

9. On 12 October 2012, a statement was sent jointly with other special rapporteurs concerning violence in Guatemala. The same day, the Special Rapporteur issued a joint statement regarding violence against a schoolchild in Pakistan.

10. On 22 October 2012, an open letter by special procedures mandate holders of the Human Rights Council was issued expressing concern at the planned adoption by the Congress of Colombia of a project to reform certain articles of the Political Constitution of Colombia, with regard to military criminal law.

11. On 15 November 2012, the Special Rapporteur, jointly with other mandate holders, called for an investigation into a death in custody in the Islamic Republic of Iran.

12. A joint statement was issued by all special procedures mandate holders on 23 November 2012 to express their dismay at the effect that the escalation of violence had on civilians in the Occupied Palestinian Territory and Israel.

13. On 28 February 2013, the Special Rapporteur together with other mandate holders called for an international inquiry into human rights violations in North Korea.

14. A number of press releases were issued specifically on death penalty cases concerning the following States: the United States of America, on 17 July 2012; Iraq, on 27 July 2012 and 30 August 2012; and the Gambia, on 28 August 2012.

15. Additional joint statements with other mandate holders on the death penalty were issued by the Special Rapporteur:

(a) The Islamic Republic of Iran: on 28 June 2012, concerning the execution of four individuals; on 12 October 2012, calling for a halt to executions; on 23 October 2012, regarding the execution of 10 individuals on drug-related crimes; and on 25 January 2013, urging the Iranian authorities to halt the execution of 5 Ahwazi activists;

(b) Saudi Arabia: on 11 January 2013, condemning the beheading of a domestic worker;

(c) Bangladesh: on 7 February 2013, expressing concern at a death sentence passed by the International Crimes Tribunal which failed to observe all the guarantees of a fair trial and due process.

D. International and national meetings

16. From 14 to 15 September 2012, the Special Rapporteur delivered a paper at the Pan-African Conference on the Safety of Journalists and the Issue of Impunity, held in Addis Ababa, Ethiopia.

17. On the occasion of the 52nd Ordinary Session of the African Commission on Human and Peoples‟ Rights on 9 October 2012, the Special Rapporteur delivered a

statement on the cooperation between the United Nations and African Union special procedures mechanisms.

18. During the sixty-seventh session of the General Assembly, the Special Rapporteur was a panellist in the side-event on the theme “The Death Penalty and Human Rights”, organized by the Special Procedures Branch of the Office of the High Commissioner for Human Rights (OHCHR) in cooperation with the World Organisation Against Torture, Penal Reform International, the Center for Constitutional Rights and Human Rights Watch in New York on 24 October 2012.

19. On 25 October 2012, the Special Rapporteur participated in the weekly briefing entitled “Issue of the Moment: The Death Penalty” for the community of non-governmental organizations associated with the Department of Public Information in New York.

20. On 15 November 2012, the Special Rapporteur presented a lecture on “The Right to Life during Demonstrations” at a seminar organized by the South African Institute for Advanced Constitutional, Public, Human Rights and International Law at the Constitutional Court of South Africa in Johannesburg. On 22 and 23 November 2012, the Special Rapporteur was a panellist during the 2nd UN Inter-Agency meeting on the safety of journalists and the issue of impunity in Vienna, Austria.

21. The Special Rapporteur took part in an Expert Meeting in Geneva entitled “How Countries Abolished the Death Penalty”, organized by the International Commission against the Death Penalty on 5 February 2013, and delivered a presentation on the resumption of the death penalty.

22. On 22 February 2013, the Special Rapporteur participated in a High Level Policy Seminar organized by the European University Institute and Global Governance and Global Governance Programme on “Targeted Killing, Unmanned Aerial Vehicles and EU Policy”,

held at the European University Institute in Florence, where he spoke on “Targeting by

Drones: Protecting the Right to Life”.

23. On 19 March 2013, the Special Rapporteur presented a keynote address at a conference on “The Ethical, Strategic and Legal Implications of Drone Warfare”, organized by the Kroc Institute at the University of Notre Dame in Indiana, United States of America.

24. On 21 March 2013, the Special Rapporteur took part in the Pugwash Workshop at the University of Birmingham, United Kingdom, where he spoke on lethal autonomous robotics.

E. Intended future areas of research

25. The Special Rapporteur will present a report on unmanned combat aerial vehicles (UCAVs) to the General Assembly in 2013.

III. Lethal autonomous robotics and the protection of life

26. For societies with access to it, modern technology allows increasing distance to be put between weapons users and the lethal force they project. For example, UCAVs, commonly known as drones, enable those who control lethal force not to be physically present when it is deployed, but rather to activate it while sitting behind computers in faraway places, and stay out of the line of fire.

27. Lethal autonomous robotics (LARs), if added to the arsenals of States, would add a new dimension to this distancing, in that targeting decisions could be taken by the robots themselves. In addition to being physically removed from the kinetic action, humans would also become more detached from decisions to kill – and their execution.

28. The robotics revolution has been described as the next major revolution in military affairs, on par with the introduction of gunpowder and nuclear bombs.3 But in an important respect LARs are different from these earlier revolutions: their deployment would entail not merely an upgrade of the kinds of weapons used, but also a change in the identity of those

3 Peter Singer, Wired for War (Penguin Group (USA) Incorporated, 2009), p. 179 and further,notably

p. 203.

who use them. With the contemplation of LARs, the distinction between weapons and warriors risks becoming blurred, as the former would take autonomous decisions about their own use.

29. Official statements from Governments with the ability to produce LARs indicate that their use during armed conflict or elsewhere is not currently envisioned.4 While this may be so, it should be recalled that aeroplanes and drones were first used in armed conflict for surveillance purposes only, and offensive use was ruled out because of the anticipated adverse consequences.5 Subsequent experience shows that when technology that provides a perceived advantage over an adversary is available, initial intentions are often cast aside. Likewise, military technology is easily transferred into the civilian sphere. If the international legal framework has to be reinforced against the pressures of the future, this must be done while it is still possible.

30. One of the most difficult issues that the legal, moral and religious codes of the world have grappled with is the killing of one human being by another. The prospect of a future in which fully autonomous robots could exercise the power of life and death over human beings raises a host of additional concerns. As will be argued in what follows, the introduction of such powerful yet controversial new weapons systems has the potential to pose new threats to the right to life. It could also create serious international division and weaken the role and rule of international law – and in the process undermine the international security system.6 The advent of LARs requires all involved – States, international organizations, and international and national civil societies – to consider the full implications of embarking on this road.

31. Some argue that robots could never meet the requirements of international humanitarian law (IHL) or international human rights law (IHRL), and that, even if they could, as a matter of principle robots should not be granted the power to decide who should live and die. These critics call for a blanket ban on their development, production and use.7

To others, such technological advances – if kept within proper bounds – represent legitimate military advances, which could in some respects even help to make armed conflict more humane and save lives on all sides.8 According to this argument, to reject this technology altogether could amount to not properly protecting life.

32. However, there is wide acceptance that caution and some form of control of States‟ use of this technology are needed, over and above the general standards already posed by international law. Commentators agree that an international discussion is needed to consider the appropriate approach to LARs.

4 US Department of Defense, Unmanned Systems Integrated Road Map FY2011-2036, p. 50, available

from http://publicintelligence.net/dod-unmanned-systems-integrated-roadmap-fy2011-2036 5 See http://www.usaww1.com/World_War_1_Fighter_Planes.php4 6 Nils Melzer, “Human rights implications of the usage of drones and unmanned robots in warfare”

Study for the European Parliament‟s Subcommittee on Human Rights (2013), available from

http://www.europarl.europa.eu/committees/en/studies/html, p. 5 (forthcoming). 7 Human Rights Watch, Losing Humanity: The Case Against Killer Robots (2012), p. 2, available from

http://www.hrw.org/reports/2012/11/19/losing-humanity-0. See in response Michael Schmitt “Autonomous Weapons Systems and International Humanitarian Law: A Reply to the Critics”

Harvard International Security Journal (forthcoming 2013), available from http://harvardnsj.org/wp- content/uploads/2013/02/Schmitt-Autonomous-Weapon-Systems-and-IHL-Final.pdf). The International Committee on Robot Arms Control (ICRAC) was formed to promote such a ban. See http://icrac.net

8 Ronald Arkin, Governing Lethal Behaviour in Autonomous Robots (CRC Press, 2009); Kenneth Anderson and Matthew Waxman, “Law and ethics for robot soldiers”, Policy Review, No. 176 (2012), available from http://www.hoover.org/publications/policy-review/article/135336.

33. As with any technology that revolutionizes the use of lethal force, little may be known about the potential risks of the technology before it is developed, which makes formulating an appropriate response difficult; but afterwards the availability of its systems and the power of vested interests may preclude efforts at appropriate control.9 This is further complicated by the arms race that could ensue when only certain actors have weapons technology. The current moment may be the best we will have to address these concerns. In contrast to other revolutions in military affairs, where serious reflection mostly began after the emergence of new methods of warfare, there is now an opportunity collectively to pause, and to engage with the risks posed by LARs in a proactive way. This report is a call for pause, to allow serious and meaningful international engagement with this issue.

34. One of the reasons for the urgency of this examination is that current assessments of the future role of LARs will affect the level of investment of financial, human and other resources in the development of this technology over the next several years. Current assessments – or the lack thereof – thus risk to some extent becoming self-fulfilling prophesies.

35. The previous Special Rapporteur examined LARs in a report in 2010,10 calling inter alia for the convening of an expert group to consider robotic technology and compliance with international human rights and humanitarian law.11 The present report repeats and strengthens that proposal and calls on States to impose national moratoria on certain activities related to LARs.

36. As with UCAVs and targeted killing, LARs raise concerns for the protection of life under the framework of IHRL as well as IHL. The Special Rapporteur recalls the supremacy and non-derogability of the right to life under both treaty and customary international law.12 Arbitrary deprivation of life is unlawful in peacetime and in armed conflict.

A. The emergence of LARs

1. Definitions

37. While definitions of the key terms may differ, the following exposition provides a starting point.13

38. According to a widely used definition (endorsed inter alia by both the United States Department of Defense and Human Rights Watch14), the term LARs refers to robotic weapon systems that, once activated, can select and engage targets without further

9 David Collingridge, The Social Control of Technology (Frances Pinter, 1980). 10 A/65/321. 11 A/65/321, pp. 10-22. 12 International Covenant on Civil and Political Rights, art. 6, enshrining the right to life, and art. 4 (2)

on its non-derogability. 13 Arkin (see note 8 above), p. 7; Noel Sharkey AutomatingWarfare: lessons learned from the drones,

p. 2, available from http://www.alfredoroma.it/wp-content/uploads/2012/05/Automated-warfare- Noel-Sharkey.pdf; Patrick Lin et al, Autonomous Military Robotics: Risk, Ethics, and Design (San Luis Obispo, California Polytechnic State University, 2008) p. 4, available from http://ethics.calpoly.edu/ONR_report.pdf

14 US Department of Defense Directive, “Autonomy in Weapons Systems”, Number 3000.09 of 21 November 2012, Glossary Part II. See also United Kingdom Ministry of Defence “The UK Approach to Unmanned Aircraft Systems” paras. 202-203, available from https://www.gov.uk/government/publications/jdn-2-11-the-uk-approach-to-unmanned-aircraft- systems; see also, Human Rights Watch (see note 7 above), p. 2.

intervention by a human operator. The important element is that the robot has an autonomous “choice” regarding selection of a target and the use of lethal force.

39. Robots are often described as machines that are built upon the sense-think-act paradigm: they have sensors that give them a degree of situational awareness; processors or artificial intelligence that “decides” how to respond to a given stimulus; and effectors that carry out those “decisions”.15 The measure of autonomy that processors give to robots should be seen as a continuum with significant human involvement on one side, as with UCAVs where there is “a human in the loop”, and full autonomy on the other, as with

LARs where human beings are “out of the loop”.

40. Under the currently envisaged scenario, humans will at least remain part of what may be called the “wider loop”: they will programme the ultimate goals into the robotic systems and decide to activate and, if necessary, deactivate them, while autonomous weapons will translate those goals into tasks and execute them without requiring further human intervention.

41. Supervised autonomy means that there is a “human on the loop” (as opposed to “in” or “out”), who monitors and can override the robot‟s decisions. However, the power to

override may in reality be limited because the decision-making processes of robots are often measured in nanoseconds and the informational basis of those decisions may not be practically accessible to the supervisor. In such circumstances humans are de facto out of the loop and the machines thus effectively constitute LARs.

42. “Autonomous” needs to be distinguished from “automatic” or “automated.” Automatic systems, such as household appliances, operate within a structured and predictable environment. Autonomous systems can function in an open environment, under unstructured and dynamic circumstances. As such their actions (like those of humans) may ultimately be unpredictable, especially in situations as chaotic as armed conflict, and even more so when they interact with other autonomous systems.

43. The terms “autonomy” or “autonomous”, as used in the context of robots, can be misleading. They do not mean anything akin to “free will” or “moral agency” as used to

describe human decision-making. Moreover, while the relevant technology is developing at an exponential rate, and full autonomy is bound to mean less human involvement in 10 years‟ time compared to today, sentient robots, or strong artificial intelligence are not currently in the picture.16

2. Current technology

44. Technology may in some respects be less advanced than is suggested by popular culture, which often assigns human-like attributes to robots and could lure the international community into misplaced trust in its abilities. However, it should also be recalled that in certain respects technology far exceeds human ability. Technology is developing exponentially, and it is impossible to predict the future confidently. As a result, it is almost impossible to determine how close we are to fully autonomous robots that are ready for use.

45. While much of their development is shrouded in secrecy, robots with full lethal autonomy have not yet been deployed. However, robotic systems with various degrees of autonomy and lethality are currently in use, including the following:

• The US Phalanx system for Aegis-class cruisers automatically detects, tracks and engages anti-air warfare threats such as anti-ship missiles and aircraft.17

15 Singer (see note 3 above), p. 67. 16 The same applies to “the Singularity”, Singer (see note 3 above), p. 101. 17 See http://usmilitary.about.com/library/milinfo/navyfacts/blphalanx.htm

• The US Counter Rocket, Artillery and Mortar (C-RAM) system can automatically destroy incoming artillery, rockets and mortar rounds.18

• Israel‟s Harpy is a “Fire-and-Forget” autonomous weapon system designed to detect, attack and destroy radar emitters.19

• The United Kingdom Taranis jet-propelled combat drone prototype can autonomously search, identify and locate enemies but can only engage with a target when authorized by mission command. It can also defend itself against enemy aircraft.20

• The Northrop Grumman X-47B is a fighter-size drone prototype commissioned by the US Navy to demonstrate autonomous launch and landing capability on aircraft carriers and navigate autonomously.21

• The Samsung Techwin surveillance and security guard robots, deployed in the demilitarized zone between North and South Korea, detect targets through infrared sensors. They are currently operated by humans but have an “automatic mode”.22

46. Military documents of a number of States describe air, ground and marine robotic weapons development programmes at various stages of autonomy. Large amounts of money are allocated for their development.23

47. It seems clear that if introduced, LARs will not, at least initially, entirely replace human soldiers, but that they will have discretely assigned tasks suitable to their specific capabilities. Their most likely use during armed conflict is in some form of collaboration with humans,24 although they would still be autonomous in their own functions. The question should therefore be asked to what extent the existing legal framework is sufficient to regulate this scenario, as well as the scenario whereby LARs are deployed without any human counterpart. Based on current experiences of UCAVs, there is reason to believe that States will inter alia seek to use LARs for targeting killing.

48. The nature of robotic development generally makes it a difficult subject of regulation, especially in the area of weapons control. Bright lines are difficult to find. Robotic development is incremental in nature. Furthermore, there is significant continuity between military and non-military technologies.25 The same robotic platforms can have civilian as well as military applications, and can be deployed for non-lethal purposes (e.g. to defuse improvised explosive devices) or be equipped with lethal capability (i.e. LARs). Moreover, LARs typically have a composite nature and are combinations of underlying technologies with multiple purposes.

49. The importance of the free pursuit of scientific study is a powerful disincentive to regulate research and development in this area. Yet “technology creep” in this area may

18 See http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA557876 19 See http://www.israeli-weapons.com/weapons/aircraft/uav/harpy/harpy.html 20 See http://www.baesystems.com/product/BAES_020273/taranis 21 See http://www.as.northropgrumman.com/products/nucasx47b/assets/X-

47B_Navy_UCAS_FactSheet.pdf 22 See http://singularityhub.com/2010/07/25/armed-robots-deployed-by-south-korea-in-demilitarized-

zone-on-trial-basis 23 United States Air Force, “UAS Flight Plan 2009-2047” (Washington, D.C., 2009) p. 41, available

from http://www.scribd.com/doc/17312080/United-States-Air-Force-Unmanned-Aircraft-Systems- Flight-Plan-20092047-Unclassified

24 Ronald Arkin “Governing Lethal Behaviour: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture” Technical Report GIT-GVU-07-11 p. 5, available from http://www.cc.gatech.edu/ai/robot-lab/online-publications/formalizationv35.pdf

25 Anderson and Waxman (see note 8 above), pp. 2 and 13 and Singer (see note 3 above), p. 379.

over time and almost unnoticeably result in a situation which presents grave dangers to core human values and to the international security system. It is thus essential for the international community to take stock of the current state of affairs, and to establish a responsible process to address the situation and where necessary regulate the technology as it develops.

3. Drivers of and impediments to the development of LARs

50. Some of the reasons to expect continuous pressures to develop LARs, as well as the impediments to this momentum, also apply to the development of other unmanned systems more generally. They offer huge military and other advantages to those using them and are part of the broader automization of warfare and of the world in general.

51. Unmanned systems offer higher force projection (preserving the lives of one‟s own soldiers) and force multiplication (allowing fewer personnel to do more). They are capable of enlarging the battlefield, penetrating more easily behind enemy lines, and saving on human and financial resources. Unmanned systems can stay on station much longer than individuals and withstand other impediments such as G-forces. They can enhance the quality of life of soldiers of the user party: unmanned systems, especially robots, are increasingly developed to do the so-called dirty, dull and dangerous work.26

52. Robots may in some respects serve humanitarian purposes. While the current emergence of unmanned systems may be related to the desire on the part of States not to become entangled in the complexities of capture, future generations of robots may be able to employ less lethal force, and thus cause fewer unnecessary deaths. Technology can offer creative alternatives to lethality, for instance by immobilizing or disarming the target.27

Robots can be programmed to leave a digital trail, which potentially allows better scrutiny of their actions than is often the case with soldiers and could therefore in that sense enhance accountability.

53. The progression from remote controlled systems to LARs, for its part, is driven by a number of other considerations.28 Perhaps foremost is the fact that, given the increased pace of warfare, humans have in some respects become the weakest link in the military arsenal and are thus being taken out of the decision-making loop. The reaction time of autonomous systems far exceeds that of human beings, especially if the speed of remote- controlled systems is further slowed down through the inevitable time-lag of global communications. States also have incentives to develop LARs to enable them to continue with operations even if communication links have been broken off behind enemy lines.

54. LARs will not be susceptible to some of the human shortcomings that may undermine the protection of life. Typically they would not act out of revenge, panic, anger, spite, prejudice or fear. Moreover, unless specifically programmed to do so, robots would not cause intentional suffering on civilian populations, for example through torture. Robots also do not rape.

55. Yet robots have limitations in other respects as compared to humans. Armed conflict and IHL often require human judgement, common sense, appreciation of the larger picture, understanding of the intentions behind people‟s actions, and understanding of

values and anticipation of the direction in which events are unfolding. Decisions over life and death in armed conflict may require compassion and intuition. Humans – while they are fallible – at least might possess these qualities, whereas robots definitely do not. While

26 Gary Marchant et al, “International governance of autonomous military robots”, Columbia Science

and Technology Law Review, Volume XII (2011) p. 275. 27 Singer (see note 3 above), p. 83. 28 Arkin (see note 8 above), xii.

robots are especially effective at dealing with quantitative issues, they have limited abilities to make the qualitative assessments that are often called for when dealing with human life. Machine calculations are rendered difficult by some of the contradictions often underlying battlefield choices. A further concern relates to the ability of robots to distinguish legal from illegal orders.

56. While LARs may thus in some ways be able to make certain assessments more accurately and faster than humans, they are in other ways more limited, often because they have restricted abilities to interpret context and to make value-based calculations.

B. LARs and the decision to go to war or otherwise use force

57. During the larger part of the last two centuries, international law was developed to constrain armed conflict and the use of force during law enforcement operations, to make it an option of last resort. However, there are also built-in constraints that humans have against going to war or otherwise using force which continue to play an important (if often not decisive) role in safeguarding lives and international security. Chief among these are unique human traits such as our aversion to getting killed, losing loved ones, or having to kill other people.29 The physical and psychological distance from the actual use of force potentially introduced by LARs can lessen all three concerns and even render them unnoticeable to those on the side of the State deploying LARs.30 Military commanders for example may therefore more readily deploy LARs than real human soldiers.

58. This ease could potentially affect political decisions. Due to the low or lowered human costs of armed conflict to States with LARs in their arsenals, the national public may over time become increasingly disengaged and leave the decision to use force as a largely financial or diplomatic question for the State, leading to the “normalization” of armed conflict.31 LARs may thus lower the threshold for States for going to war or otherwise using lethal force, resulting in armed conflict no longer being a measure of last resort32 According to the report of the Secretary-General on the role of science and technology in the context of international security and disarmament, “…the increased

capability of autonomous vehicles opens up the potential for acts of warfare to be conducted by nations without the constraint of their people‟s response to loss of human life.”33 Presenting the use of unmanned systems as a less costly alternative to deploying “boots on the ground” may thus in many cases be a false dichotomy. If there is not sufficient support for a ground invasion, the true alternative to using unmanned systems may be not to use force at all.

59. Some have argued that if the above reasoning is taken to its logical conclusion, States should not attempt to develop any military technology that reduces the brutality of armed conflict or lowers overall deaths through greater accuracy.34 Drones and high- altitude airstrikes using smart bombs should then equally be viewed as problematic because

29 A/65/321, para. 44; John Mueller “The Iraq Syndrome”, Foreign Affairs, Vol. 84, No. 6, p. 44

(November/December 2005). 30 According to military experts, it generally becomes easier to take life as the distance between the

actor and the target increases. See David Grossman On Killing: The Psychological Cost of Learning to Kill in War and Society (Back Bay Books, 1996).

31 Armin Krishnan Killer robots: Legality and Ethicality of Autonomous Weapons (Ashgate, 2009) p. 150

32 Singer (see note 3 above), p. 323; Peter Asaro “How Just Could a Robot War Be?” in P. Brey et al (eds.) Current Issues in Computing And Philosophy (2008), p. 7.

33 A/53/202, para. 98. 34 Asaro (see note 32 above), pp. 7-9. Discussed by Patrick Lin et al “Robots in War: Issues of Risk and

Ethics” in R. Capurro & M. Nagenborg (eds.) Ethics and Robotics (2009) p. 57.

they also lower casualty rates for the side that uses them (and in some cases also for the other side), thereby removing political constraints on States to resort to military action.35

60. This argument does not withstand closer scrutiny. While it is desirable for States to reduce casualties in armed conflict, it becomes a question whether one can still talk about “war” – as opposed to one-sided killing – where one party carries no existential risk, and bears no cost beyond the economic. There is a qualitative difference between reducing the risk that armed conflict poses to those who participate in it, and the situation where one side is no longer a “participant” in armed conflict inasmuch as its combatants are not exposed to any danger.36 LARs seem to take problems that are present with drones and high-altitude airstrikes to their factual and legal extreme.

61. Even if it were correct to assume that if LARs were used there would sometimes be fewer casualties per armed conflict, the total number of casualties in aggregate could still be higher.

62. Most pertinently, the increased precision and ability to strike anywhere in the world, even where no communication lines exist, suggests that LARs will be very attractive to those wishing to perform targeted killing. The breaches of State sovereignty – in addition to possible breaches of IHL and IHRL – often associated with targeted killing programmes risk making the world and the protection of life less secure.

C. The use of LARs during armed conflict

63. A further question is whether LARs will be capable of complying with the requirements of IHL. To the extent that the answer is negative, they should be prohibited weapons. However, according to proponents of LARs this does not mean that LARs are required never to make a mistake – the yardstick should be the conduct of human beings who would otherwise be taking the decisions, which is not always a very high standard.37

64. Some experts have argued that robots can in some respects be made to comply even better with IHL requirements than human beings.38 Roboticist Ronald Arkin has for example proposed ways of building an “ethical governor” into military robots to ensure that

they satisfy those requirements.39

65. A consideration of a different kind is that if it is technically possible to programme LARs to comply better with IHL than the human alternatives, there could in fact be an obligation to use them40 – in the same way that some human rights groups have argued that where available, “smart” bombs, rather than less discriminating ones, should be deployed.

66. Of specific importance in this context are the IHL rules of distinction and proportionality. The rule of distinction seeks to minimize the impact of armed conflict on civilians, by prohibiting targeting of civilians and indiscriminate attacks.41 In situations

35 Anderson and Waxman (see note 8 above), p. 12. 36 According to some commentators, war requires some willingness to accept reciprocal or mutual risk,

involving some degree of sacrifice. See Paul Kahn “The Paradox of Riskless Warfare” Philosophy

and Public Policy Vol. 22 (2002) and “War and Sacrifice in Kosovo” (1999), available from http://www-personal.umich.edu/~elias/Courses/War/kosovo.htm

37 Lin (see note 34 above), p. 50. 38 Marchant (see note 26 above), p. 280; Singer, (see note 3 above), p. 398. 39 Arkin (see note 8 above), p. 127. 40 Jonathan Herbach “Into the Caves of Steel: Precaution, Cognition and Robotic Weapons Systems

Under the International Law of Armed Conflict” Amsterdam Law Forum Vol. 4 (2012), p. 14. 41 Protocol I additional to the Geneva Conventions, 1977, arts. 51 and 57.

where LARs cannot reliably distinguish between combatants or other belligerents and civilians, their use will be unlawful.

67. There are several factors that will likely impede the ability of LARs to operate according to these rules in this regard, including the technological inadequacy of existing sensors,42 a robot‟s inability to understand context, and the difficulty of applying of IHL language in defining non-combatant status in practice, which must be translated into a computer programme.43 It would be difficult for robots to establish, for example, whether someone is wounded and hors de combat, and also whether soldiers are in the process of surrendering.

68. The current proliferation of asymmetric warfare and non-international armed conflicts, also in urban environments, presents a significant barrier to the capabilities of LARs to distinguish civilians from otherwise lawful targets. This is especially so where complicated assessments such as “direct participation in hostilities” have to be made.

Experts have noted that for counter-insurgency and unconventional warfare, in which combatants are often only identifiable through the interpretation of conduct, the inability of LARs to interpret intentions and emotions will be a significant obstacle to compliance with the rule of distinction.44

69. Yet humans are not necessarily superior to machines in their ability to distinguish. In some contexts technology can offer increased precision. For example, a soldier who is confronted with a situation where it is not clear whether an unknown person is a combatant or a civilian may out of the instinct of survival shoot immediately, whereas a robot may utilize different tactics to go closer and, only when fired upon, return fire. Robots can thus act “conservatively”45 and “can shoot second.”46 Moreover, in some cases the powerful sensors and processing powers of LARs can potentially lift the “fog of war” for human soldiers and prevent the kinds of mistakes that often lead to atrocities during armed conflict, and thus save lives.47

70. The rule of proportionality requires that the expected harm to civilians be measured, prior to the attack, against the anticipated military advantage to be gained from the operation.48 This rule, described as “one of the most complex rules of international humanitarian law,”49 is largely dependent on subjective estimates of value and context- specificity.

71. Whether an attack complies with the rule of proportionality needs to be assessed on a case-by-case basis, depending on the specific context and considering the totality of the circumstances.50 The value of a target, which determines the level of permissible collateral damage, is constantly changing and depends on the moment in the conflict. Concerns have been expressed that the open-endedness of the rule of proportionality combined with the complexity of circumstances may result in undesired and unexpected behaviour by LARs, with deadly consequences.51 The inability to “frame” and contextualize the environment

42 Noel Sharkey “Grounds for Discrimination: Autonomous Robot Weapons” RUSI Defence Systems

(Oct 2008) pp. 88-89, available from http://rusi.org/downloads/assets/23sharkey.pdf 43 Peter Asaro “On Banning Autonomous Weapon Systems: Human Rights, Automation, and the

Dehumanisation of Lethal Decision-making” p. 94, International Review of the Red Cross (forthcoming 2013) p. 11.

44 Human Rights Watch (see note 7 above), p. 31. 45 Marchant (see note 26 above), p. 280. 46 Singer (see note 3 above), p. 398. 47 Ibid. 48 Protocol I additional to the Geneva Conventions, 1977, art. 51 (5) (b). 49 Human Rights Watch (see note 7 above), p. 32. 50 Lin (see note 34 above), p. 57. 51 Noel Sharkey, “Automated Killers and the Computing Profession” Computer, Vol. 40 (2007), p. 122.

may result in a LAR deciding to launch an attack based not merely on incomplete but also on flawed understandings of the circumstances.52 It should be recognized, however, that this happens to humans as well.

72. Proportionality is widely understood to involve distinctively human judgement. The prevailing legal interpretations of the rule explicitly rely on notions such as “common sense”, “good faith” and the “reasonable military commander standard.”53 It remains to be seen to what extent these concepts can be translated into computer programmes, now or in the future.

73. Additionally, proportionality assessments often involve qualitative rather than quantitative judgements.54

74. In view of the above, the question arises as to whether LARs are in all cases likely (on the one hand) or never (on the other) to meet this set of cumulative standard. The answer is probably less absolute, in that they may in some cases meet them (e.g. in the case of a weapons system that is set to only return fire and that is used on a traditional battlefield) but in other cases not (e.g. where a civilian with a large piece of metal in his hands must be distinguished from a combatant in plain clothes). Would it then be possible to categorize the different situations, to allow some to be prohibited and others to be permitted? Some experts argue that certain analyses such as proportionality would at least initially have to be made by commanders, while other aspects could be left to LARs.55

D. Legal responsibility for LARs

75. Individual and State responsibility is fundamental to ensure accountability for violations of international human rights and international humanitarian law. Without the promise of accountability, deterrence and prevention are reduced, resulting in lower protection of civilians and potential victims of war crimes.56

76. Robots have no moral agency and as a result cannot be held responsible in any recognizable way if they cause deprivation of life that would normally require accountability if humans had made the decisions. Who, then, is to bear the responsibility?

77. The composite nature of LAR technology and the many levels likely to be involved in decisions about deployment result in a potential accountability gap or vacuum. Candidates for legal responsibility include the software programmers, those who build or sell hardware, military commanders, subordinates who deploy these systems and political leaders.

78. Traditionally, criminal responsibility would first be assigned within military ranks. Command responsibility should be considered as a possible solution for accountability for

52 Krishnan, (see note 31 above), pp. 98-99. 53 Tonya Hagmaier et al, “Air force operations and the law: A guide for air, space and cyber forces”

p. 21, available from http://www.afjag.af.mil/shared/media/document/AFD-100510-059.pdf; Andru Wall “Legal and Ethical Lessons of NATO‟s Kosovo Campaign” p. xxiii, available from http://www.au.af.mil/au/awc/awcgate/navy/kosovo_legal.pdf

54 Markus Wagner “The Dehumanization of International Humanitarian Law: Legal, Ethical, and Political Implications of Autonomous Weapon Systems” (2012), available from http://robots.law.miami.edu/wp- content/uploads/2012/01/Wagner_Dehumanization_of_international_humanitarian_law.pdf note 96 and accompanying text.

55 Benjamin Kastan “Autonomous Weapons Systems: A Coming Legal „Singularity‟?” University of Illinois Journal of Law, Technology and Policy (forthcoming 2013), p. 18 and further, available from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2037808

56 Human Rights Watch (see note 7 above), pp. 42-45.

LAR violations.57 Since a commander can be held accountable for an autonomous human subordinate, holding a commander accountable for an autonomous robot subordinate may appear analogous. Yet traditional command responsibility is only implicated when the commander “knew or should have known that the individual planned to commit a crime yet

he or she failed to take action to prevent it or did not punish the perpetrator after the fact.”58 It will be important to establish, inter alia, whether military commanders will be in a position to understand the complex programming of LARs sufficiently well to warrant criminal liability.

79. It has been proposed that responsibility for civil damages at least should be assigned to the programmer and the manufacturers, by utilizing a scheme similar to strict product liability. Yet national product liability laws remain largely untested in regard to robotics.59 The manufacturing of a LAR will invariably involve a vast number of people, and no single person will be likely to understand the complex interactions between the constituent components of LARs.60 It is also questionable whether putting the onus of bringing civil suits on victims is equitable, as they would have to bring suit while based in a foreign country, and would often lack the resources.

80. The question of legal responsibility could be an overriding issue. If each of the possible candidates for responsibility identified above is ultimately inappropriate or impractical, a responsibility vacuum will emerge, granting impunity for all LAR use. If the nature of a weapon renders responsibility for its consequences impossible, its use should be considered unethical and unlawful as an abhorrent weapon.61

81. A number of novel ways to establish legal accountability could be considered. One of the conditions that could be imposed for the use of LARs is that responsibility is assigned in advance.62 Due to the fact that technology potentially enables more precise monitoring and reconstruction of what occurs during lethal operations, a further condition for their use could be the installation of such recording devices, and the mandatory ex post facto review of all footage in cases of lethal use, regardless of the status of the individual killed.63 A system of “splitting” responsibility between the potential candidates could also be considered.64 In addition, amendments to the rules regarding command responsibility may be needed to cover the use of LARs. In general, a stronger emphasis on State as opposed to individual responsibility may be called for, except in respect of its use by non- state actors.

57 Rome Statute of the ICC, art. 28; Heather Roff “Killing in War: Responsibility, Liability and Lethal

Autonomous Robots” p. 14, available from

http://www.academia.edu/2606840/Killing_in_War_Responsibility_Liability_and_Lethal_Autonomo us_Robots

58 Protocol I additional to the Geneva Conventions, 1977, arts. 86 (2) and 87. 59 Patrick Lin “Introduction to Robot Ethics” in Patrick Lin et al (eds.) Robot Ethics: The ethical and

Social Implications of Robotics (MIT Press, 2012), p. 8. 60 Wendell Wallach “From Robots to Techno Sapiens: Ethics, Law and Public Policy in the

Development of Robotics and Neurotechnologies” Law, Innovation and Technology Vol. 3 (2011) p. 194.

61 Gianmarco Verugio and Keith Abney “Roboethics: The Applied Ethics for a New Science” in Lin, (see note 59 above), p. 114; Robert Sparrow “Killer Robots” Journal of Applied Philosophy Vol. 24, No. 1 (2007).

62 See Ronald Arkin “The Robot didn‟t do it” Position Paper for the Workshop on Anticipatory Ethics, Responsibility and Artificial Agents p. 1, available from http://www.cc.gatech.edu/ai/robot- lab/publications.html

63 Marchant (see note 26 above), p. 7. 64 Krishnan (see note 31 above), 105.

E. The use of LARs by States outside armed conflict

82. The experience with UCAVs has shown that this type of military technology finds its way with ease into situations outside recognized battlefields.

83. One manifestation of this, whereby ideas of the battlefield are expanded beyond IHL contexts, is the situation in which perceived terrorists are targeted wherever they happen to be found in the world, including in territories where an armed conflict may not exist and IHRL is the applicable legal framework. The danger here is that the world is seen as a single, large and perpetual battlefield and force is used without meeting the threshold requirements. LARs could aggravate these problems.

84. On the domestic front, LARs could be used by States to suppress domestic enemies and to terrorize the population at large, suppress demonstrations and fight “wars” against drugs. It has been said that robots do not question their commanders or stage coups d‟état.65

85. The possibility of LAR usage in a domestic law enforcement situation creates particular risks of arbitrary deprivation of life, because of the difficulty LARs are bound to have in meeting the stricter requirements posed by IHRL.

F. Implications for States without LARs

86. Phrases such as “riskless war” and “wars without casualties” are often used in the context of LARs. This seems to purport that only the lives of those with the technology count, which suggests an underlying concern with the deployment of this technology, namely a disregard for those without it. LARs present the ultimate asymmetrical situation, where deadly robots may in some cases be pitted against people on foot. LARs are likely – at least initially – to shift the risk of armed conflict to the belligerents and civilians of the opposing side.

87. The use of overwhelming force has proven to have counterproductive results – e.g. in the context of demonstrations, where psychologists warn that it may elicit escalated counter force66 In situations of hostilities, the unavailability of a legitimate human target of the LAR user State on the ground may result in attacks on its civilians as the “best available” targets and the use of LARs could thus possibly encourage retaliation, reprisals and terrorism.67

88. The advantage that States with LARs would have over others is not necessarily permanent. There is likely to be proliferation of such systems, not only to those to which the first user States transfer and sell them. Other States will likely develop their own LAR technology, with inter alia varying degrees of IHL-compliant programming, and potential problems for algorithm compatibility if LARs from opposing forces confront one another. There is also the danger of potential acquisition of LARs by non-State actors, who are less likely to abide by regulatory regimes for control and transparency.

G. Taking human decision-making out of the loop

89. It is an underlying assumption of most legal, moral and other codes that when the decision to take life or to subject people to other grave consequences is at stake, the decision-making power should be exercised by humans. The Hague Convention (IV) requires any combatant “to be commanded by a person”. The Martens Clause, a

65 Ibid , p. 113. 66 A/HR/17/28, p. 17. 67 Asaro (see note 32 above), p. 13.

longstanding and binding rule of IHL, specifically demands the application of “the principle of humanity” in armed conflict.68 Taking humans out of the loop also risks taking humanity out of the loop.

90. According to philosopher Peter Asaro, an implicit requirement can thus be found in IHL for a human decision to use lethal force, which cannot be delegated to an automated process. Non-human decision-making regarding the use of lethal force is, by this argument, inherently arbitrary, and all resulting deaths are arbitrary deprivations of life.69

91. The contemplation of LARs is inextricably linked to the role of technology in the world today. While machines help to make many decisions in modern life, they are mostly so used only where mechanical observation is needed (e.g. as a line umpire in sporting events) and not in situations requiring value judgements with far-reaching consequences (e.g. in the process of adjudication during court cases). As a more general manifestation of the importance of person-to-person contact when important decisions are taken, legal systems around the world shy away from trials in absentia. Of course, robots already affect our lives extensively, including through their impact on life and death issues. Robotic surgery is for example a growing industry and robots are increasingly used in rescue missions after disasters.70 Yet in none of these cases do robots make the decision to kill and in this way LARs represent an entirely new prospect.

92. Even if it is assumed that LARs – especially when they work alongside human beings – could comply with the requirements of IHL, and it can be proven that on average and in the aggregate they will save lives, the question has to be asked whether it is not inherently wrong to let autonomous machines decide who and when to kill. The IHL concerns raised in the above paragraphs relate primarily to the protection of civilians. The question here is whether the deployment of LARs against anyone, including enemy fighters, is in principle acceptable, because it entails non-human entities making the determination to use lethal force.

93. This is an overriding consideration: if the answer is negative, no other consideration can justify the deployment of LARs, no matter the level of technical competence at which they operate. While the argument was made earlier that the deployment of LARs could lead to a vacuum of legal responsibility, the point here is that they could likewise imply a vacuum of moral responsibility.

94. This approach stems from the belief that a human being somewhere has to take the decision to initiate lethal force and as a result internalize (or assume responsibility for) the cost of each life lost in hostilities, as part of a deliberative process of human interaction. This applies even in armed conflict. Delegating this process dehumanizes armed conflict even further and precludes a moment of deliberation in those cases where it may be feasible. Machines lack morality and mortality, and should as a result not have life and death powers over humans. This is among the reasons landmines were banned.71

95. The use of emotive terms such as “killer robots” may well be criticized. However, the strength of the intuitive reactions that the use of LARs is likely to elicit cannot be ignored. Deploying LARs has been depicted as treating people like “vermin”, who are

68 Geneva Convention Protocol I, art. 1(2). See also the preambles to the 1899 and 1907 Hague

Conventions. Hague Convention with Respect to the Laws and Customs of War on Land and its Annex: Regulation Concerning the Laws and Customs of War on Land (Hague Convention II)

69 Asaro (see note 43 above), p. 13. 70 See http://www.springer.com/medicine/surgery/journal/11701 71 Asaro (see note 43 above), p. 14.

“exterminated.”72 These descriptions conjure up the image of LARs as some kind of mechanized pesticide.

96. The experience of the two World Wars of the last century may provide insight into the rationale of requiring humans to internalize the costs of armed conflict, and thereby hold themselves and their societies accountable for these costs. After these wars, during which the devastation that could be caused by modern technology became apparent, those who had personally taken the central military decisions resolved, “in order to save

succeeding generations from the scourge of war”, to establish the United Nations to pursue world peace and to found it on the principles of human rights. While armed conflict is by no means a thing of the past today, nearly 70 years have passed without a global war. The commitment to achieve such an objective can be understood as a consequence of the long- term and indeed inter-generational effects of insisting on human responsibility for killing decisions.

97. This historical recollection highlights the danger of measuring the performance of LARs against minimum standards set for humans during armed conflict. Human soldiers do bring a capacity for depravity to armed conflict, but they also hold the potential to adhere to higher values and in some cases to show some measure of grace and compassion. If humans are replaced on the battlefield by entities calibrated not to go below what is expected of humans, but which lack the capacity to rise above those minimum standards, we may risk giving up on hope for a better world. The ability to eliminate perceived “troublemakers” anywhere in the world at the press of a button could risk focusing attention only on the symptoms of unwanted situations. It would distract from, or even preclude, engagement with the causes instead, through longer term, often non-military efforts which, although more painstaking, might ultimately be more enduring. LARs could thus create a false sense of security for their users.

H. Other concerns

98. The possible deployment of LARs raises additional concerns that include but are not limited to the following:

• LARs are vulnerable to appropriation, as well as hacking and spoofing.73 States no longer hold a monopoly on the use of force. LARs could be intercepted and used by non-State actors, such as criminal cartels or private individuals, against the State or other non-State actors, including civilians.74

• Malfunctions could occur. Autonomous systems can be “brittle”.75 Unlikely errors can still be catastrophic.

• Future developments in the area of technology cannot be foreseen. Allowing LARs could open an even larger Pandora‟s box.

• The regulation of the use of UCAVs is currently in a state of contestation, as is the legal regime pertaining to targeted killing in general, and the emergence of LARs is likely to make this situation even more uncertain.

• The prospect of being killed by robots could lead to high levels of anxiety among at least the civilian population.

72 Robert Sparrow “Robotic Weapons and the Future of War” in Jessica Wolfendale and Paolo Tripodi

(eds.) New Wars and New Soldiers: Military Ethics in the Contemporary World (2011), p. 11. 73 Jutta Weber “Robotic warfare, human rights and the rhetorics of ethical machines”, pp. 8 and 10,

available from http://www.gender.uu.se/digitalAssets/44/44133_Weber_Robotic_Warfare.pdf 74 Singer (see note 3 above), p. 261-263. 75 Kastan (see note 55 above), p. 8.

99. The implications for military culture are unknown, and LARs may thus undermine the systems of State and international security.

I. LARs and restrictive regimes on weapons

100. The treaty restrictions76 placed on certain weapons stem from the IHL norm that the means and methods of warfare are not unlimited, and as such there must be restrictions on the rules that determine what weapons are permissible.77 The Martens Clause prohibits weapons that run counter to the “dictates of public conscience.” The obligation not to use weapons that have indiscriminate effects and thus cause unnecessary harm to civilians underlies the prohibition of certain weapons,78 and some weapons have been banned because they “cause superfluous injury or unnecessary suffering”79 to soldiers as well as civilians.80 The use of still others is restricted for similar reasons.81

101. In considering whether restrictions as opposed to an outright ban on LARs would be more appropriate, it should be kept in mind that it may be more difficult to restrict LARs as opposed to other weapons because they are combinations of multiple and often multipurpose technologies. Experts have made strong arguments that a regulatory approach that focuses on technology – namely, the weapons themselves – may be misplaced in the case of LARs and that the focus should rather be on intent or use.82

102. Disarmament law and its associated treaties, however, provide extensive examples of the types of arms control instruments that establish bans or restrictions on use and other activities. These instruments can be broadly characterized as some combination of type of restriction and type of activity restricted. The types of restrictions include a ban or other limitations short of a ban.

103. The type of activity that is typically restricted includes: (i) acquisition, retention or stockpiling, (ii) research (basic or applied) and development, (iii) testing, (iv) deployment, (v) transfer or proliferation, and (vi) use.83

104. Another positive development in the context of disarmament is the inclusion of victim assistance in weapons treaties.84 This concern for victims coincides with other efforts to address the harm weapons and warfare cause to civilians, including the practice of casualty counting85 and the good faith provision of amends – implemented for example by

76 Through the Hague Convention of 1907 and the 1977 Additional Protocols to the Geneva

Conventions. 77 See http://www.icrc.org/eng/war-and-law/conduct-hostilities/methods-means-warfare/index.jsp 78 Mine Ban Treaty (1997); and Convention on Cluster Munitions (2008). 79 Protocol I additional to the Geneva Conventions, 1977. art. 35 (2); ICRC, Customary Humanitarian

Law, Rule 70. 80 Protocol for the Prohibition of the Use of Asphyxiating, Poisonous or Other Gases, and of

Bacteriological Methods of Warfare. Geneva, 17 June 1925. 81 Convention on Certain Conventional Weapons, Protocol III on incendiary weapons. 82 Marchant (see note 26 above), p. 287, Asaro see note 43 above), p. 10. 83 Marchant (see note 26 above), p. 300. See also Bonnie Docherty “The Time is Now: A Historical

Argument for a Cluster Munitions Convention” 20 Harvard Human Rights Law Journal (2007), p. 53 for an overview.

84 Mine Ban Treaty (1997), art. 6, and Convention on Certain Conventional Weapons, Protocol V on Explosive Remnants of War (2003), art. 8. The Convention on Cluster Munitions (2008), art. 5 was groundbreaking in placing responsibility on the affected State.

85 S/2012/376, para. 28 (commending inter alia the commitment by the African Union Mission in Somalia).

some International Security Assistance Force States – in the case of civilian deaths in the absence of recognized IHL violations.86 These practices serve to reaffirm the value of life.

105. There are also meaningful soft law instruments that may regulate the emergence of LARs. Examples of relevant soft law instruments in the field of disarmament include codes of conduct, trans-governmental dialogue, information sharing and confidence-building measures and framework conventions.87 In addition, non-governmental organization (NGO) activity and public opinion can serve to induce restrictions on weapons.

106. Article 36 of the First Protocol Additional to the Geneva Conventions is especially relevant, providing that, “in the study, development, acquisition or adoption of a new weapon, means or methods of warfare, a High Contracting Party is under an obligation to determine whether its employment would, in some or all circumstances, be prohibited by this Protocol or by any other rule of international law applicable to the High Contracting Party.”

107. This process is one of internal introspection, not external inspection, and is based on the good faith of the parties.88 The United States, although not a State party, established formal weapons mechanisms review as early as 1947. While States cannot be obliged to disclose the outcomes of their reviews, one way of ensuring greater control over the emergence of new weapons such as LARs will be to encourage them to be more open about the procedure that they follow in Article 36 reviews generally.

108. In 2012 in a Department of Defense Directive, the United States embarked on an important process of self-regulation regarding LARs, recognizing the need for domestic control of their production and deployment, and imposing a form of moratorium.89 The Directive provides that autonomous weapons “shall be designed to allow commanders and

operators to exercise appropriate levels of human judgement over the use of force”.90 Specific levels of official approval for the development and fielding of different forms of robots are identified.91 In particular, the Directive bans the development and fielding of LARs unless certain procedures are followed.92 This important initiative by a major potential LARs producer should be commended and may open up opportunities for mobilizing international support for national moratoria.

IV. Conclusions

109. There is clearly a strong case for approaching the possible introduction of LARs with great caution. If used, they could have far-reaching effects on societal

values, including fundamentally on the protection and the value of life and on

international stability and security. While it is not clear at present how LARs could be

capable of satisfying IHL and IHRL requirements in many respects, it is foreseeable

that they could comply under certain circumstances, especially if used alongside

human soldiers. Even so, there is widespread concern that allowing LARs to kill

people may denigrate the value of life itself. Tireless war machines, ready for

deployment at the push of a button, pose the danger of permanent (if low-level) armed

conflict, obviating the opportunity for post-war reconstruction. The onus is on those

who wish to deploy LARs to demonstrate that specific uses should in particular

86 Ibid., para. 29 (the Secretary General “welcomed the practice of making amends”).

87 Marchant, see note 26, pp. 306-314. 88 Discussed in International Review of the Red Cross vol. 88, December 2006. 89 US DoD Directive (see note 14 above). 90 Ibid, para 4.a. 91 Ibid, paras 4.c and d. 92 Ibid, Enclosure 3.

circumstances be permitted. Given the far-reaching implications for protection of life,

considerable proof will be required.

110. If left too long to its own devices, the matter will, quite literally, be taken out of human hands. Moreover, coming on the heels of the problematic use and contested

justifications for drones and targeted killing, LARs may seriously undermine the

ability of the international legal system to preserve a minimum world order.

111. Some actions need to be taken immediately, while others can follow afterwards. If the experience with drones is an indication, it will be important to

ensure that transparency, accountability and the rule of law are placed on the agenda

from the start. Moratoria are needed to prevent steps from being taken that may be

difficult to reverse later, while an inclusive process to decide how to approach this

issue should occur simultaneously at the domestic, intra-State, and international

levels.

112. To initiate this process an international body should be established to monitor the situation and articulate the options for the longer term. The ongoing engagement

of this body, or a successor, with the issues presented by LARs will be essential, in

view of the constant evolution of technology and to ensure protection of the right to

life – to prevent both individual cases of arbitrary deprivation of life as well as the

devaluing of life on a wider scale.

V. Recommendations

A. To the United Nations

113. The Human Rights Council should call on all States to declare and implement national moratoria on at least the testing, production, assembly, transfer, acquisition,

deployment and use of LARs until such time as an internationally agreed upon

framework on the future of LARs has been established;

114. Invite the High Commissioner for Human Rights to convene, as a matter of priority, a High Level Panel on LARs consisting of experts from different fields such

as law, robotics, computer science, military operations, diplomacy, conflict

management, ethics and philosophy. The Panel should publish its report within a

year, and its mandate should include the following:

(a) Take stock of technical advances of relevance to LARs;

(b) Evaluate the legal, ethical and policy issues related to LARs;

(c) Propose a framework to enable the international community to address

effectively the legal and policy issues arising in relation to LARs, and make concrete

substantive and procedural recommendations in that regard; in its work the Panel

should endeavour to facilitate a broad-based international dialogue;

(d) Assessment of the adequacy or shortcomings of existing international

and domestic legal frameworks governing LARs;

(e) Suggestions of appropriate ways to follow up on its work.

115. All relevant United Nations agencies and bodies should, where appropriate in their interaction with parties that are active in the field of robotic weapons:

(a) Emphasize the need for full transparency regarding all aspects of the

development of robotic weapon systems;

(b) Seek more international transparency from States regarding their

internal weapons review processes, including those under article 36 of Additional

Protocol I to the Geneva Conventions.

B. To regional and other inter-governmental organizations

116. Support the proposals outlined in the recommendations to the United Nations and States, in particular the call for moratoria as an immediate step.

117. Where appropriate take similar or parallel initiatives to those of the United Nations.

C. To States

118. Place a national moratorium on LARs as described in paragraph 114.

119. Declare – unilaterally and through multilateral fora – a commitment to abide by IHL and IHRL in all activities surrounding robotic weapons and put in place and

implement rigorous processes to ensure compliance at all stages of development.

120. Commit to being as transparent as possible about internal weapons review processes, including metrics used to test robotic systems. States should at a minimum

provide the international community with transparency regarding the processes they

follow (if not the substantive outcomes) and commit to making the reviews as robust

as possible.

121. Participate in international debate and trans-governmental dialogue on the issue of LARs and be prepared to exchange best practices with other States, and

collaborate with the High Level Panel on LARs.

D. To developers of robotic systems

122. Establish a code or codes of conduct, ethics and/or practice defining responsible behaviour with respect to LARs in accordance with IHL and IHRL, or

strengthen existing ones.

E. To NGOs, civil society and human rights groups and the ICRC

123. Consider the implications of LARs for human rights and for those in situations of armed conflict, and raise awareness about the issue.

124. Assist and engage with States wherever possible in aligning their relevant procedures and activities with IHL and IHRL.

125. Urge States to be as transparent as possible in respect of their weapons review processes.

126. Support the work of the High Level Panel on LARs.