Ensembles of Bidirectional LSTM and GRU Neural Nets for Predicting Mother-Infant Synchrony in Videos

The importance of positive, healthy and reciprocal interactions between mother and infant cannot be understated as it leaves a lasting impact on the rest of the infant’s life. One way to identify a positive interaction between two people is the amount of nonverbal synchrony - or spontaneous coordination of bodily movements, present in the interaction. This work proposes a neural network and ensemble learning based approach to automatically labelling a mother-infant dyad interaction as high versus low by predicting the level of synchrony of the interaction. Bidirectional Long Short-Term Memory (BiLSTM) and Bidirectional Gated Recurrent Unit (BiGRU) models were trained and evaluated on a dataset consisting of 25 key body position coordinates of mother and infant extracted with an AI specialised tool called OpenPose, from 58 different videos. Ensembles of 30 such bidirectional recurrent neural network base models were built and then post-processed via ROC analysis, to improve prediction stability and performance, both of which assessed in a Monte Carlo validation procedure of 30 iterations. The prediction performances on the unseen test samples for the ensembles of BiLSTM and ensembles of BiGRU models include a mean AUC of 0.781 and 0.796, a mean precision of 0.857 and 0.899, and a mean specificity of 0.817 and 0.872, respectively. In particular our models predict higher probability scores for the high synchrony class versus the low synchrony class in 80% of cases. Moreover the achieved high precision level indicates that 90% of mother-infant dyads predicted to be in the high synchrony class are predicted correctly, and the high specificity level indicates a detection rate of the mother-infant dyads with low synchrony in 87% of cases, suggesting these models’ high capability for automatically flagging cases that may be clinically relevant for further investigation and potential intervention.