A recurrent approach for predicting Parkinson stage from multimodal videos

International audience; Parkinson's disease is a neurodegenerative disease that affects more than 6.1 million people worldwide.

In the clinical routine, the main tool to diagnose and monitor disease progression is based on motor impairments, such as postural instability, bradykinesia, tremor, among others.

Besides, new biomarkers based on motion patterns have emerged to describe disease findings.

Nonetheless, this motor characterization has low sensitivity, especially at early stages, and is largely expert-dependent, because protocols are mainly based on visual observations.

However, most of these analyses require complex and some invasive systems that additionally only bring global information of complete recordings.

This work introduces a multimodal approach that integrates gait and eye motion videos to quantify and predict patient stage on-the-fly.

This method starts by computing dense apparent velocity maps that represent the local displacement of the person seen from the gait in a sagittal plane and as micro-movements during the fixation experiment.

Then, each frame is described as a covariance descriptor of deep feature activation maps computed over the motion field at each video time.

Then, the covariance video manifold is mapped to a recurrent LSTM network to learn higher non-local dependencies and quantify a motion descriptor.

Also, an end-to-end scheme allows to lately fuse both modalities (gait and fixational eye) to obtain a more sensitive Parkinson disease descriptor.

In a study with 25 subjects, the proposed approach reaches an average F1-score of 0.83 with an average recall of 0.78.

In a temporal prediction analysis, the approach reports major correlations with the disease considering swing phase.