This AI Found Signs Comatose Patients Were Waking Up Days Before Doctors Did

Imagine waking up in a hospital room. The last thing you remember is a terrible car crash. A doctor holds your hand and asks you to squeeze it. You try as hard as you can, but nothing happens—not even a twitch.

“I’m afraid he’s in a coma,” you hear the doctor say. But I’m conscious, you want to yell.

People with traumatic head injuries, often resulting from car accidents, can seem completely unresponsive to the outside world. But many experience “covert consciousness”—in that their brains respond to commands—even though they can’t translate it into eye blinks, finger twitches, or other obvious movements for clinicians and nurses to detect.

Although brain imaging techniques can sometimes capture signs a person is internally awake and trying to follow instructions, these methods are expensive and difficult to use for everyday monitoring while doctors and the patients’ families wait for them to wake up.

“Some people with severe brain injuries may appear unconscious, but still have some awareness and the ability to move,” wrote Sima Mofakham and colleagues at Stony Brook University in a new study. It’s just that “these movements are often too small to be seen by doctors during routine exams.”

The study, led by Mofakham, used computer vision to track tiny facial movements in seemingly unconscious patients. The AI tool, dubbed SeeMe, zeroed in on extremely minute movements, down to the level of single pores in the skin.

Compared to physicians, the tool detected early signs of covert consciousness roughly four days earlier in roughly 90 percent of patients. The study also found the number and strength of these tiny twitches corresponded to how well the patient had recovered at the time of discharge.

Early detection of consciousness could make recovery less distressing for a person who’s just waking up. Knowing the person is aware could help doctors decide when to kickstart rehabilitation associated with better health outcomes. The technology may also one day be used to monitor real-time treatments for brain damage due to stroke and other injuries.

Stairway to Consciousness

We often think of consciousness as a light switch. Flip it on, and you’re aware of both the outside world and yourself; flip it off, and awareness goes dark.

But consciousness is more like a light dimmer. After a blow to the brain, people can fall into a minimally conscious state. Here, they experience intermittent awareness and can follow commands, like if a doctor says “look left” or “squeeze my hand.” More severe is the vegetative state. Patients in this state can open or close their eyes in cycles, but they can no longer respond to outside stimulation.

In especially traumatic injuries, the patient goes into a coma, where they’re not aware of themselves and others, can’t move, and can’t be awakened.

Despite the odds, unresponsive people can recover mental awareness—often sooner than their observable behavior would suggest. In one study, a person in a vegetative state showed relevant brain activity when asked to imagine playing tennis or moving around her house, even though she couldn’t physically respond.

More recently, a landmark brain imaging study found at least a quarter of 353 people with severe brain injuries, who had been deemed unconscious, showed signs of awareness based on brain activity when given voice commands. Most did not react to a battery of standard clinical tests for responsiveness.

But brain imaging tests, while powerful, are expensive and impractical for everyday clinical use. Rather than looking into the brain, the team behind the new study took a page out of the clinician’s playbook by linking tiny facial movements to diagnostics and recovery.

Now You See Me

The face is a window on the brain. Its muscles are controlled by large areas across both of the brain’s hemispheres. Any early signs of recovery are likely to show up first in facial movements.

The team recruited 16 healthy volunteers and 37 people with brain injuries who, outwardly, appeared to be in a coma. They then analyzed video recordings of the participants being asked to do three tasks: “Stick out your tongue,” “open your eyes,” and “show me a smile.”

The tasks chosen involved multiple facial regions and muscles to better gauge brain activity, the authors wrote.

The new AI tool, SeeMe, then tracked facial movements—down to the level of individual pores—in response to the commands. A group of trained medical professionals also reviewed the videos and were asked for their expert opinions.

The AI captured eye responses in 30 patients and nearly all of their mouth movements, with a success rate nearly double that of the physicians. SeeMe was especially sensitive to tiny twitches that evaded the human eye.

The tool also flagged earlier signs of consciousness. In one deeply comatose volunteer, an older man who suffered a car crash, the AI detected mouth movements on day 18 after admission; he finally responded to motor commands on day 37. SeeMe also found signs of eye and mouth movements 19 days after admission in another participant in a coma after a traffic accident. He opened his eyes three days later and went on to gradually recover.

Across the board, SeeMe detected eye-opening responses roughly four days before standard tests picked them up and mouth-related reactions about eight days earlier. The AI’s performance also correlated to how well patients recovered on discharge and at six months—that is, they increasingly regained awareness and could do rehab.

SeeMe is intended to complement, not replace, long-term follow-up and care. Comatose patients are “an exceedingly challenging population to study,” wrote the team. Some people may have had fluctuations in awareness that weren’t captured in the study. Others may simply not have wanted to participate.

A lack of early detection of consciousness “should never be interpreted as the absence of potential” that the patient can regain awareness, the authors explained.

To further fine-tune the AI, the team hopes to gather information on people who regained consciousness but were initially missed by SeeMe. They also aim to incorporate other objective measures of movement, such as electrical signals in muscles. SeeMe could even help people presumed unconscious for longer periods of time than that covered in the study.

For patients and families, further work could result in a “yes or no” system based on facial movements that might allow loved ones to “talk” to each other again.