CCW Report, Vol. 3, No. 5
Out from the shadows: law, ethics, and the prohibition of autonomous weapons
Ray Acheson
15 April 2016
“When we act from afar and from the shadows, we do much more harm than good.” A retired Captain from the US Air Force Reserve, writing a letter to The New Yorker this week, critiqued “automated warfare” for its false sense of precision and separation of body from battlefield. These concerns, as others raised at this CCW meeting, have serious ethical, moral, and human rights implications when it comes to increasing autonomy in weapon systems. Thursday’s plenary meetings featured discussions on many of these implications, as well as the risks for global and regional destabilisation. The compelling solution to these challenges remains a prohibition on weapon systems operating without meaningful human control.
Human rights
Christof Heyns, UN special rapporteur for extrajudicial, summary, or arbitrary executions, argued that weapons operating without meaningful human control pose a range of human rights concerns. In particular, he noted, such autonomous weapon systems (AWS) risk undermining the right to life, which is essential for protecting human beings from the use of force.
Several states and civil society groups have expressed concern that AWS risk lowering the threshold for the use of force. Pablo Kalmanovitz of Universidad de los Andes, Colombia, agreed AWS risk lowering the threshold to go to war, because of the perception of minimised risk to the deploying force, but also that AWS risk lowering the use of violence within war, arguing that more attacks might be made during a conflict for the same reason. Heyns also noted that even where AWS might be employed in capture rather than kill operations, this is still a use of force and would be subject to the same concerns if done without meaningful human control over the weapon system.
Chile and Amnesty International also highlighted what Chile’s delegation described as the “terrible impact” AWS would have on human rights. Amnesty International argued that without effective and meaningful human control, AWS threaten the right to life, right to security of person, right to human dignity, and possibly the right to freedom of peaceful assembly and should be banned. This last element was also recently addressed by Heyns, along with the UN special rapporteur on the rights to freedom of peaceful assembly and of association, in a report on the “proper management of assemblies.” The report recommends that, “where advanced technology is employed, law enforcement officials must, at all times, remain personally in control of the actual delivery or release of force.”
Human dignity
The threat to the right to human dignity has been repeatedly highlighted by lawyers, ethicists, and others participating in these discussions over the past three years. Heyns again reiterated that targeting by AWS reduces human beings to zeroes and ones in a computer, with serious consequences for human dignity. Further, Heyns argued, if dignity is understood also to entail assuming responsibility for one’s action, then the use of AWS can challenge this in various ways. Humans deploying AWS are not necessarily the authors of actual actions that take place—if they do not have meaningful control over the machine, this affects their ability to make “responsible decisions”.
This of course also has implications for accountability and liability. Heyns argued that control and accountability are two sides of the same coin. If one does not have control, one cannot be accountable. This lack of accountability, he suggested, in itself constitutes a violation of human rights.
The US delegation on Thursday argued that “adherence to ethical and moral norms will depend less on the inherent nature of a technology and more on its potential use by humans.” But as many have pointed out over the past three years, it is the inherent nature of AWS that has serious implications for human rights and ethics, not merely “misuse” of such weapons. This is the basic idea behind the fundamental principle of IHL that the choice of methods and means of warfare is not unlimited. It is the affront to human dignity posed by AWS that drives many ethical and societal objections to their development and deployment. The idea of machines killing humans on the basis of software and algorithms alone is, as WILPF has noted, cynically abhorrent, or, as Kalmanovitz said, a nightmare for civilians.
Transfer of risk
Indeed, Kalmanovitz noted that while AWS may minimise risks to the deploying force’s soldiers, it can amplify risks to civilians. He warned that programming in AWS may incorporate preferences of militarily advanced countries to shift risks away from their own forces. Greater damage to civilians could come to be treated as proportional because preferences of the deploying force are programmed in at the expense of civilian protection.
This is a risk, Heyns and Amnesty International argue, not only for civilians living under situations of armed conflict, but also in law enforcement contexts. Heyns and
Eliav Lieblich, of the Radzyner Law School in Israel, both expressed concerns about what might happen if an AWS programmed for use in armed conflict is used in a law enforcement situation. Law enforcement officials have a responsibility to protect the public, argued Heyns, but this obligation is not as strong in armed conflict.
These challenges run counter to what Denise Garcia of Northeastern University described as a global norm for preventative regulations to protect civilians. The development and deployment of AWS, she argued, would jeopardise civilian protection, human rights laws, and the architecture and principles for sustainable peace, including disarmament and reduction of military spending. She highlighted that the development of AWS would divert resources away from peace and disarmament in violation of the UN Charter’s article 26.
Deference to machines
Yet it appears that some states at CCW wish to leave the door open to the development of further autonomy in weapons. The US argued that “human-machine teaming in targeting has brought not only enhanced situational awareness to help reduce the immediate risk to soldiers, but also better discrimination and the ability to exercise tactical patience, where additional time can be taken to ensure accurate target identification and avoid civilian casualties.”
However as Amnesty International points out in an article in this edition of the CCW Report, the “Drone Papers” recently published by The Intercept “paint an alarming picture of the lethal US drones programme. According to the documents, during one five-month stretch, 90% of people killed by US drone strikes were unintended targets.” While the US argues that machines’ participation in targeting can enable humans to “make better decisions,” the actual deployment of such machines appears to have serious precision issues and risks undermining rules of proportionality and increasing the transfer of risk to civilians.
In this context, Heyns highlighted that because machines are faster at processing information and suggesting actions on that basis, we are becoming accustomed to deferring to machines and relying on their determinations. Thus even if humans could act fast enough to override an autonomous delivery of force, there might be an inclination to defer to the machine because the stakes are so high, he warned. We use machines as tools, explained Heyns, yet we sometimes think they might know better. This challenge suggests that human beings need to have meaningful human control not just over each individual attack but also over analysing and selecting targets. It also highlights a key problem with suggestions that AWS should have the “possibility of human control”—this seems to lack the effectiveness of the human being fully responsible for the operation of a weapon and the selection of and engagement with targets.
Prohibition
The need for meaningful human control over targeting and individual attacks is the basis for a prohibition on autonomous weapons systems. Calls for this prohibition are growing. The two special rapporteurs writing on the proper management of assemblies recommended that AWS without meaningful human control should be banned. Heyns reiterated this call during his presentation on Thursday, joining fourteen states, thousands of scientists, and the Campaign to Stop Killer Robots in urging the negotiation of a treaty preventing the development, deployment, and use of AWS. Nearly all of the other panelists on Thursday supported a prohibition, reflecting the growing momentum for serious action on this issue. As Chile’s delegation said, the disarmament and human rights community has a responsibility to be ahead of the curve on AWS and act now to prevent their introduction into our shared world.
[PDF] ()