Automation resilience.

The level of autonomy is rapidly increasing in Unmanned Vehicle Systems (UVS) and the ability to diagnose and understand human­-UVS teamwork is becoming an even more essential part of developing effective warfighting capabilities. System designers must model a wide range of decisions and actions between human operators and autonomous systems to maximize UVS mission performance. TOPViz provides Automated Performance Evaluation functionality to support the design of performance aids, training interventions, procedures, and avionics tools. It also provides comprehensive metrics to quantitatively assess overall system performance using a probabilistic Bayesian Inference Framework to model and predict performance/autonomy outcomes. Based on a Web-based architecture, TOPViz allows Design and Decision-making teams to share and access data, model and analyze results simultaneously, and utilize a backend inference engine without a need for powerful computing devices for each user.

 
 

Probabilistic Reasoning

Model, simulate, analyze and optimize for probabilistic factors in system and team performance

 

Data-Driven
 

Easily upload or enter relevant data and link it to the performance models

 

Collaboration
 

Share real world data, models and analysis or optimization results with everyone in design and decision making teams simultaneously

 

Always
Online

Web-based architecture for easy access to shared data and computationally intensive modeling and analysis

Problem

A decision needs to be made regarding whether to provide an automated target detection aid for pilots of Scout Unmanned Aerial Vehicles (UAVs) based on pilot task difficulty and workload.

Solution

The design team starts by building a performance model based on user experience literature using the TOPViz model editor, which models effects of workload and task difficulty on user error rates and target detection accuracy. The team then can upload experimental data from a target group of pilots carrying out a series of tasks including target detection either in a simulated environment or the field exercises into the TOPViz application. Relevant metrics such as mission length and error rates are then linked to the model to fill in prior distributions and better identify how each factor in the model influences the next. The effects of automated aids can either be collected and linked to the performance model in the same manner, or the designers can choose to work with hypothetical definitions for such effects based on literature in the absence of tangible data.

 

Finally, the performance model can be analyzed, taking into account both data-based and literature-driven considerations, generating two outputs: 1) An overall utility assessment of whether to provide automated aids for target detection with the expected performance improvement; and 2) A set of metrics that allows designers to analyze the sensitivity of their model. In addition to providing a path to improve human-UV team performance, these metrics can also be used to guide the editing of the performance model for better future assessment. Furthermore the metrics can allow designers to scope out better experimental evaluations to collect more useful data from simulators or from the field.