Development and Comparison of Operator Strategy Models

Human supervisory control (HSC), in which operators indirectly control autonomous systems by sending and receiving commands, is a commonly-used scheme for various human-automation interaction scenarios. While many studies have investigated how factors, such as different levels of autonomy and interface designs, affect operator performance in HSC scenarios, no previous research has quantitatively evaluated the impact of such factors on operator strategies. Thus, this research focuses on developing a quantitative metric to compare strategy models to determine whether changing specific factors in HSC scenarios would affect operator strategies.

Previous studies have shown that operator strategies can be represented by operator behavior patterns in conducting tasks and achieving goals. Given that hidden Markov models (HMMs) can represent operator strategies, researchers can investigate impacts from technology or process changes on operator strategies by comparing HMM strategy models. However, no quantitative and systematic HMM strategy model comparison metric has been proposed. To resolve this problem, this research uses the divergence distance measure to develop a mesh comparison metric to comprehensively compare strategy models and obtain quantitative model difference measures.

As a part of the comparison metric, the data quantity requirement for model development is determined using a large external dataset from a typical HSC video game. Strategy models were trained based on different data quantities and then compared to benchmark models developed from the whole dataset. Comparison results show that a minimum of 30 data sequences can represent the whole population and be effectively used to model operator strategies. Also, as another part of the metric, an observation alignment approach is proposed to compare strategy models developed from different HSC scenarios with non-equivalent training data elements.

Utilizing this comparison metric, researchers can quantitatively measure differences between strategy models. However, it is not clear how the magnitude of such comparison measures maps to meaningful degrees of difference in HSC scenarios. To address this issue, an initial baseline of strategy difference comparisons was established by comparing strategy models developed from human-subject experiment sessions. Then, a continuum of comparisons was generated to provide references for the magnitude of impacts from different factors on operator strategies. Thus, researchers can apply changes in HSC scenarios and evaluate the impacts from such changes on operator strategies by measuring differences between strategy models and referring to comparison baselines.

In summary, the contributions of this dissertation include 1) proposing an operator strategy model comparison metric to quantitatively measure differences between operator strategies modeled from HSC scenarios and 2) establishing strategy model comparison references across multiple HSC scenarios with varying settings.