Metrics and Methods for Real-Time Task Performance Assessment
Principal Investigator:
Kevin R. Duda, Ph.D.
Organization:
The Charles Stark Draper Laboratory, Inc.
Human interaction with any modern spacecraft is enabled through shared control between the operator and the underlying control software or automation that drives the effectors, monitors feedback loops, and provides system stability. However, current approaches to performance evaluation focus on metrics related to the final result of the mission. The quantification of human performance in real-time offers the significant advantage of including these metrics as a feedback parameter for both the human and the control-system software.
Our team is developing and validating robust and objective performance metrics to assess task performance in representative spacecraft control tasks and recommend changes in sufficient time to improve mission outcomes. The identification of critical performance decrements, either in measures of task performance, workload, or situational awareness, may be used to alter the human-automation task allocation or suggest changes to crew resource management. In the field of human spaceflight, the real-time quantification of performance during operationally-relevant tasks and scenarios has the potential for making existing operations more safe and efficient, as well as for improving the design of future vehicles.
NASA Taskbook Entry
Manual control of any modern spacecraft is a task of shared control between a human operator or team, and the underlying control software that drives the effectors, monitors feedback loops, and provides control-system stability. Evaluating human performance in real-time offers the significant advantage of including human performance as a feedback parameter for both the human operators and control-system software. Rather than focusing on the final results of a mission, real-time performance assessments can be valuable diagnostic tools for evaluating more detailed performance successes and failures during the mission. Meaningful real-time operator monitoring can produce a comprehensive assessment of an operator's state and mission performance. This provides important context for interpreting the operator's actions and is critical for presenting intelligent feedback to the operator.
In the field of human spaceflight, the development and validation of real-time performance metrics and quantification of human performance during operationally-relevant tasks and scenarios has the potential for making existing operations more safe and efficient, as well as for improving the design of future vehicles. The identification of critical performance decrements, either in measures of task performance, workload, or situational awareness, may be used to alter the human-automation task allocation or suggest changes to crew resource management. As with any measure, they are highly dependent on the conditions under which the data is collected. Therefore, the development and validation of these metrics requires operationally-relevant tasks with highly trained test subjects.
Our research plan aims to include four operationally-relevant tasks:
• MPCV/Orion docking operations with the International Space Station (ISS)
• Piloted atmospheric entry of the MPCV
• Piloted lunar landing using a generic lunar lander design
• Manual control of the EVA SAFER jet pack near the ISS.
We propose three integrated specific aims to produce a configurable and portable simulation research capability for developing and validating real-time metrics for assessing flight performance, workload, and situational awareness as a function of time:
Aim 1: Define the system architecture for integrating the vehicle and environmental models with the simulation environment for configuring the scenarios, data logging, and data analysis.
Aim 2: Perform a critical analysis of four piloted tasks: MPCV/Orion docking, MPCV/Orion entry, Lunar Landing, and EVA SAFER self-rescue. Simulator data will be analyzed to identify the specifics of candidate metrics for performance, workload, and situational awareness. Options for presenting feedback to the operator will be developed and evaluated, and integrated with the flight displays and performance analysis algorithms.
Aim 3: Conduct a series of experiments using the simulated spaceflight tasks and real-time metrics engine to baseline performance, workload, and situational awareness in each of the tasks in order to develop algorithms and methodologies for alerting the operator to deviations from the nominal. Analysis will be conducted within each scenario and a meta-analysis across scenarios will be conducted to determine commonalities and to set absolute performance thresholds.
Human-in-the-loop control of any complex system benefits from the development, validation, and implementation of objective, unobtrusive real-time performance metrics. These metrics and methodologies have the potential to improve mission outcomes in piloted aircraft and rotorcraft, unmanned aerial vehicle operations, undersea operations, and human interaction with land vehicles and robots.
