Research

Brain

  • Current Research
  • Previous Research

Overview

Optical Computer Recognition of Stress, Affect and Fatigue in Space Flight

Principal Investigator:
David F. Dinges, Ph.D.

Organization:
University of Pennsylvania

The overarching objective of the research is to address the NSBRI Neurobehavioral and Psychosocial Factors goal to “Refine entirely non-obtrusive objective means of detecting and mitigating cognitive performance deficits, stress, fatigue, anxiety and depression for the operational setting of spaceflight”, and in doing so, provide an effective method to predict, detect, and assess decrements in behavioral health and fatigue (which may negatively affect performance) during spaceflight missions. Astronauts must maintain high-level performance while experiencing demanding workload/work schedules, extreme environmental risks, and psychosocial stressors in space (e.g., isolation, confinement). Stress, negative emotions and fatigue can jeopardize their cognitive performance, behavioral health and interpersonal functioning.

We seek to mitigate these behavioral health risks by developing an objective, unobtrusive optical computer recognition (OCR) technology that provides early detection of facial expressions of stress, negative moods and fatigue during spaceflight. To accomplish this, we have been developing (Dr. Metaxas Lab) and validating (Dr. Dinges Lab) a computational model-based tracker of the human face that reliably identifies when astronauts are displaying facial signs of stress, various emotional expressions and ocular signs of fatigue from sleep loss during spaceflight. The project deliverable will be a software OCR algorithm integrated with spacecraft video cameras that provides feedback to astronauts and flight physicians for autonomous selection of countermeasures for stress, negative affect and fatigue. Such a deliverable will help close related risk gaps from the NASA Human Research Program (HRP) Behavioral Health and Performance (BHP) areas by (1) finding an effective method to predict, detect, and assess decrements in behavioral health (which may negatively affect performance) during spaceflight missions (BMed2); and (2) finding an effective tool to detect and assess performance decrements due to fatigue (Sleep1).

A preliminary evaluation of the OCR technology will be conducted in NASA???s Human Exploration Research Analog (HERA) at Johnson Space Center. The HERA facility is a space analog environment in which astronauts and astronaut-surrogates can live and work in the presence of potential spaceflight stressors (i.e., confinement, isolation, etc.).

NASA Taskbook Entry

Technical Summary

This project will advance development of OCR technology using video analysis, software-based algorithms to permit online, real-time tracking in space of astronaut facial expressions of stress and negative affect (BMed2 gap); and ocular (PERCLOS) evidence of fatigue from inadequate sleep (Sleep1 gap). Thus far, OCR algorithms for processing facial expressions of affect and an ocular index of fatigue have been enhanced by our investigative team with new developments in image processing (Metaxas Lab). We have continued to validate their accuracy in adult men and women of various ages and ethnicities, through double-blind randomized trials, using induced conditions of stress, negative and positive emotions, and fatigue, and validating the coherence of OCR emotion detection against what trained (on FACES) human observers rate as the expressed emotion, and OCR PERCLOS detection to PVT performance lapses from sleep loss (Dinges Laboratory).

Recent development of the OCR technology focused on the following specific aims: (1) Create an OCR system capable of monitoring facial displays of specific emotions (i.e. angry, happy and sad). (2) Improve our current OCR system's ability to detect facial expressions of high versus low performance-induced stress. (3) Develop OCR algorithms to identify fatigue due to sleep loss based on slow eyelid closures (PERCLOS). (4) Test the technical feasibility of data acquisition and reliability of the advanced OCR system in spaceflight analogs that contain neurobehavioral stressors relevant to spaceflight.

During the upcoming funding period, a preliminary evaluation of the OCR technology will be conducted in NASA's Human Exploration Research Analog (HERA), a BHP space analog environment in which volunteers live and work in the presence of potential spaceflight stressors (i.e., confinement, isolation, etc.). HERA is currently undergoing engineering preparations to make the study feasible. This includes the addition of video cameras for recording the faces of inhabitants of HERA undergoing simulated space missions. Facial video footage will be obtained from 6-8 crewmembers' faces during missions they undergo in HERA. Cameras will record their faces at work stations during multiday missions. The facial video footage will be analyzed for oculomotor indicators of fatigue from sleep loss and for facial expressions of negative and positive emotions. Validation of these analyses will involve comparing independent (i.e., double blind) scoring of fatigue and facial expressions by human scorers and the computerized OCR algorithm.

The following specific aims will be addressed in the HERA facility: (1) test the technical feasibility of acquiring facial video data for OCR analyses in a space analog facility; (2) determine the reliability of OCR analyses of facial videos across time in mission; (3) determine the validity of OCR algorithm accuracy for detecting oculomotor fatigue in the HERA spaceflight analog; (4) determine the validity of OCR algorithm accuracy for detecting positive and negative facial expressions in the HERA spaceflight analog; and (5) identify changes or improvements to the OCR system to optimize functionality.

Earth Applications

The study focuses on the ability of an unobtrusive, automated optical computer recognition (OCR) technology to detect facial expression of stress, negative and positive emotions, and oculomotor evidence of fatigue during operational activities of healthy adults living and working in a confined environment (i.e., HERA facility at Johnson Space Center). The knowledge gained has the potential to identify an objective, unobtrusive, automated method for the recognition, monitoring, and management of the risks to performance of individuals and crews, from negative emotion and fatigue, in many Earth-based safety-sensitive occupations, such as transportation workers (e.g., truck drivers, train conductors, airline pilots); operators in safety-sensitive industries (e.g., power plant control rooms); and military personnel.