CCW Report, Vol. 5, No. 1
The urgent need to retain human control of weapon systems
Mary Wareham
13 November 2017
The United States along with China, Israel, South Korea, Russia, and the United Kingdom have been investing in developing weapons systems with decreasing levels of human control in the critical functions of selecting and engaging targets. The fear is that as the human role decreases, machines will eventually take over these critical functions.
Armed drones are an example of the trend toward ever-greater autonomy, but they are still operated by a human who takes the decision to select and fire on targets. A central concern with fully autonomous weapons is that they will cross a moral line that should never be crossed by permitting machines to make the determination to take a human life on the battlefield or in policing, border control, and other circumstances.
After 20 long months, this week states are finally reconvening at the Convention on Conventional Weapons (CCW) at the UN in Geneva to discuss serious questions relating to lethal autonomous weapons systems, also known as fully autonomous weapons or “killer robots”. Since the last substantive CCW meeting on the issue in April 2016, concerns have continued to mount over these future weapons, while questions have multiplied about whether the CCW can adequately address this urgent challenge in a timely and conclusive manner.
In recent months, hundreds of artificial intelligence and robotics experts from around the world have demanded swifter and stronger action by states at the CCW to prevent the development of lethal autonomous weapons systems. This includes calls for Australia and Canada to lead efforts to conclude a new international treaty to prohibit these weapons systems.
While the capabilities of future technology are uncertain, there strong reasons to believe that fully autonomous weapons could never replicate the full range of inherently human characteristics necessary to comply with international humanitarian law’s fundamental rules of distinction and proportionality. Moreover, it is obvious that these weapons have the potential to commit unlawful acts for which no one could be held responsible. Existing mechanisms for legal accountability are ill suited and inadequate to address the unlawful harm that fully autonomous weapons would be likely to cause.
Because fully autonomous weapons would have the power to make complex determinations faster in less structured environments, their use could lead armed conflicts to spiral rapidly out of control. Their ability to operate without a line of communication after deployment is problematic because the weapons could make poor choices about the use of force.
While fully autonomous weapons might create an immediate military advantage for some states, they should recognize that such benefits would be short lived once these weapons begin to proliferate. Ultimately, the financial and human costs of developing such weapons systems would leave each state worse off.
For these and other reasons, non-governmental organizations launched the Campaign to Stop Killer Robots in April 2013 to work for a preemptive ban on development, production, and use of weapons systems that, once activated, would select and fire on targets without meaningful human control.
Since that time, 19 countries have endorsed the call to ban fully autonomous weapons. Dozens of states have affirmed the importance of retaining meaningful or appropriate or adequate human control over critical combat functions of weapons systems. The Campaign to Stop Killer Robots encourages all states to endorse the call for a ban or, at minimum, to make clear their position on the ban call.
Our campaign is disappointed by some of the weak and unambitious proposals that certain states are considering working towards at the CCW. These include non-binding political declarations as well as efforts to ensure greater transparency and identify best practices for weapons reviews that are required under Article 36 of Additional Protocol I to the Geneva Conventions.
Yet weapons reviews are insufficient in addressing the many challenges raised by fully autonomous weapons. While international humanitarian law already sets limits on problematic weapons and their use, responsible governments in the past have found it necessary to supplement existing legal frameworks for weapons that by their nature pose significant humanitarian threats.
A binding, absolute ban on fully autonomous weapons would reduce the chance of misuse of the weapons, would be easier to enforce, and would enhance the stigma associated with violations. Moreover, a ban would maximize the stigmatization of fully autonomous weapons, creating a widely recognized norm and influencing even those that do not join the treaty. Precedent shows that a ban would be achievable and effective.
At these CCW meetings, the Campaign to Stop Killer Robots urges states to acknowledge the need for international regulation to address autonomy in the critical functions of weapon systems and elaborate their position on the call to prohibit systems that would lack meaningful human control.
States should agree to continue the Group of Governmental Experts in 2018, by holding at least four weeks of meetings throughout the year. States should retain the long list of topics contained in the GGE mandate agreed by the Fifth Review Conference. They should focus specifically on lethal autonomous weapons systems rather than broader questions relating to artificial intelligence and autonomy.
Rolling over the mandate for another year is the least that CCW states can do on this topic. After three years of informal talks, our campaign strongly believes that it’s time for states to commit to negotiate and adopt an international, legally binding instrument that prohibits the development, production, and use of fully autonomous weapons.
If that is not possible under the auspices of the CCW, then it may be time to leave this inconclusive talk shop. We stand ready to work with states interested in exploring all possible mechanisms to ban fully autonomous weapons without delay. This is not an esoteric exercise. Rather the future of our humanity depends on it.
Mary Wareham is arms division advocacy director at Human Rights Watch and coordinator of the Campaign to Stop Killer Robots.
[PDF] ()