When Systems Still Work: Training Human–AI Teams to Recognize Degradation and Recalibrate Trust
As military operations become increasingly dependent on AI-enabled decision support, networked sensors and autonomous systems, one of the most difficult failure modes may not be complete system loss but partial degradation: systems that remain available while latency, data freshness, confidence, provenance or communications quality deteriorate. Operators trained primarily for normal operation or obvious failure may continue to rely on outputs whose decision value has changed. This paper develops a scenario-based training framework for preparing human–AI teams to recognise that transition, recalibrate trust and adapt before operational performance deteriorates. The framework maps controlled degradation conditions—including stale or delayed data, conflicting sensor inputs, intermittent connectivity, changing confidence levels and missing contextual information—to observable operator responses. It combines concepts from human-machine teaming and trust calibration with structured scenario design and measurable behavioural indicators, including degradation-recognition time, reliance or override behaviour, decision latency, recovery strategy and the operator’s ability to explain why trust should change. The analysis identifies three training requirements: expose operators to ambiguous degradation rather than only clean failure; provide uncertainty and system-state cues without creating new automation dependence; and use after-action review to assess not only whether the final decision was correct, but when and why trust changed. Rather than treating resilience simply as the ability to continue operating after binary failure, the framework treats cognitive resilience as the ability to recognise that the decision environment has changed and adapt the human-machine relationship accordingly. The contribution is therefore a training design framework for the transition between justified reliance and justified distrust, rather than for system failure alone. Applicable to AI-enabled command and control, autonomous systems, degraded communications and multi-domain operations, the approach aims to prepare personnel for the more ambiguous—and potentially more dangerous—condition in which a system is still functioning but no longer deserves the same level of trust.