This is another short extract from the Thesis that I thought was particularly relevant given recent news coverage of the dangers of autonomy and AI, particularly in the field of “killer robots”.

Legal Considerations in Design Trust

If there is one key feature of the application of robotics and autonomy to the defence field that separates it from applications in commercial and civil fields, it is the potential direct impact on life and safety.

The process of some entity, be it human or autonomous, to commit to the decision to fire any kind of weapon is a critical one, and the criticality of this decision making process weighs heavily on the development of autonomous weapons systems and platforms ¹²³. Key challenges to autonomy in this space include accurate friend-or-foe identification, rapid assessment of incoming threat, and discerning “appropriate levels of force”, all of which are key parts of international combat doctrine and law ⁴. Beyond these ethical and operational challenges, one key piece of legislation that makes the dissemination of fire control to autonomous weapons systems particularly challenging; under the 1907 Hague Conventions ⁵, lawful combat requires any combatant “to be commanded by a person.”, and in particular, The Martens Clause of that convention (originally introduced in the earlier 1899 convention) specifically demands the application of “the principle of humanity” in combat ⁶. The problem presented by these principles is that modern remote operation systems perform with the operator “on-the-loop” for the majority of time, with on-board autonomy taking over “simple” on-board processes such as local navigation, collision avoidance, positioning etc.

Once this “on-the-loop” platform is in theatre, direct control must be assumed before any fire-orders are given, otherwise commanders would be in direct breach of the above conventions. Whether such breaches have already occurred is outside the scope of this work, and as discussed in (the Thesis ) , this work is primarily concerned with actions that involve no fire control what so ever. However, there is one potentially relevant area of application to which these doctrines may arguably apply; a significant area of application of maritime autonomy is in Explosive Ordinance Disposal as part of Mine Counter Measures operations, i.e. mine-clearance with explosive devices. Current operational doctrine as well as currently in-testing systems utilise autonomous survey and mine localisation, and then this information is used to send in a human diving team to perform clearance, placing them at extreme risk.

However, none of the above applies to this work and is presented for context.