Astronauts must perform mission-critical tasks at a high level of functional capability. The ability to unobtrusively detect levels of psychological distress will be necessary on long missions. The goal of Dr. David Dinges project is to develop machine vision that can track and discriminate facial expressions induced by low versus high workload demands. As the system is being calibrated, the investigators are taking into account facial differences, such as the facial swelling, or puffy face, that is common in the weightless environment, gender, age, and ethnicity. The system is currently being validated with subjects exposed to varying degrees of workload-based stressors.
Overview
Optical Computer Recognition of Performance Under Stress
Principal Investigator:
David F. Dinges, Ph.D.
Organization:
University of Pennsylvania School of Medicine
Technical Summary
Original Aims
This project focused on the early objective detection of neurobehavioral problems in spaceflight. The overarching goal was to develop an unobtrusive, automated optical technology to detect psychological distress when astronauts are working during spaceflight. The primary aim was to determine whether optical computer recognition (OCR) algorithms based on changes in facial expressions could discriminate behavioral stress induced by low versus high workload performance demands. Astronauts are required to perform mission-critical tasks at a high level of functional capability throughout spaceflight. There are a number of stressors that can compromise their ability to do so. In order to provide timely countermeasures for stressor-induced impairments in astronauts, objective, unobtrusive measures of the presence of stress reactionsespecially during performance demandsare needed.
The study focused on the ability of an unobtrusive, automated optical technology to detect psychological distress (and the need for countermeasures for it) during operational performance. The findings support the hypothesis that machine vision can be used to as an objective, unobtrusive, automated method for the recognition, monitoring and management of the risks of neurobehavioral dysfunction due to work-related stress in spaceflight and in many Earth-based safety-sensitive occupations, such as transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.
This project focused on the early objective detection of neurobehavioral problems in spaceflight. The overarching goal was to develop an unobtrusive, automated optical technology to detect psychological distress when astronauts are working during spaceflight. The primary aim was to determine whether optical computer recognition (OCR) algorithms based on changes in facial expressions could discriminate behavioral stress induced by low versus high workload performance demands. Astronauts are required to perform mission-critical tasks at a high level of functional capability throughout spaceflight. There are a number of stressors that can compromise their ability to do so. In order to provide timely countermeasures for stressor-induced impairments in astronauts, objective, unobtrusive measures of the presence of stress reactionsespecially during performance demandsare needed.
This project was designed to achieve such a measurement methodology, through collaboration between two laboratories: one with expertise in the evaluation of behavioral and physiological responses to stressful performance conditions (D. Dinges, Unit for Experimental Psychiatry, University of Pennsylvania School of Medicine) and the other with expertise in optical computer recognition of human expressions and gestures (D. Metaxas, Computational Biomedicine Imaging and Modeling Center, Rutgers University). The goal of this collaboration was to further develop and test an optically-based computer recognition algorithm of the face to reliably detect the presence of stress. The computer-based recognition system, developed by Prof. Metaxas, utilized automatic optical tracking of human subjects' subtle anatomical and motoric changes in facial expressions. Video input to the system was provided from experiments performed in the laboratory of Prof. Dinges. In this project, we experimentally tested the extent to which optical computer recognition algorithms based on facial expressions could detect behavioral stress during cognitive performance.
Hypothesis
Optical computer recognition algorithms of the face can reliably discriminate when subjects are undergoing behavioral stress associated with low versus high cognitive workload demands. The project was also intended to serve as an opportunity to improve the technology. Our goal was to deliver an optical computer recognition system capable of identifying stress reactions based on the automated analysis of facial expressions in real-time.
Key Findings
In our initial experiments, the validity of the workload paradigm to induce differential levels of stress in facial expressions was established. Basic stress-related facial expressions required to establish a prototypical OCR algorithm to detect such changes were also identified. We established that OCR identification of stress during performance is possible and that the accuracy of the system could be improved. We then further improved the OCR algorithm and the automated application of the deformable masks to video of the moving face to increase practical utility. In more recent experiments, we completed development of an optical algorithm for real-time dynamic tracking of the face using a deformable model-based tracker and Active Shape Modeling (ASM). To overcome the limitations of previous optical tracking techniques, Metaxas and colleagues developed a formal framework for the integration of edge detection and optical flow into a deformable model framework and applied it to facial shape and motion estimation. This method used a single camera to track the shape of the face and its movement in three-dimensional (3D) space, and it created a deformable model, incorporating optical flow (an approximation of the motion of objects within a visual representation) into the model as a constraint. Our earlier version of the OCR system, which utilized 3D deformable models for face tracking and Hidden Markov Models (HMM) for stress detection, was 68 percent accurate in discriminating between high and low stress situations in 60 subjects. After we reduced computational burden of the OCR system by using Active Shape Models for face tracking and the more efficient Conditional Random Fields for stress detection, accuracy increased to 73 percent.
We have made several other new developments to the OCR system during this period of support:
- The technique was validated with the use of only one camera, where the previous method required two;
- We improved tracking by using a manifold of faces that helped automatically track the face as the head moves;
- We added the use of Conditional Random Fields, in addition to Hidden Markov Modeling, to the algorithm which improved its computational efficiency; and
- GABOR filtering (used for edge detection in image analysis) was incorporated into the ASM algorithm to track changes in facial texture, allowing it to identify features (e.g., furrowed brow).
The study focused on the ability of an unobtrusive, automated optical technology to detect psychological distress (and the need for countermeasures for it) during operational performance. The findings support the hypothesis that machine vision can be used to as an objective, unobtrusive, automated method for the recognition, monitoring and management of the risks of neurobehavioral dysfunction due to work-related stress in spaceflight and in many Earth-based safety-sensitive occupations, such as transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.
Earth Applications
The study focuses on the ability of an unobtrusive, automated optical technology to detect psychological distress (and the need for countermeasures for it) during operational performance. The knowledge gained has the potential to identify an objective, unobtrusive, automated method for the recognition, monitoring and management of the risks of neurobehavioral dysfunction due to work-related stress in spaceflight and in many Earth-based safety-sensitive occupations, such as transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.
This project's funding ended in 2008