▬ Research

Medical Cybernetics and Bioinformatics

Affective Computing

For machines, the capability to recognize user state (mental and affective disposition of the user) and adapt their response accordingly has the potential to improve communication accuracy of these technologies, and promote an engaging user experience, which in turn can improve user's adoption of the technologies.

The objective of my PhD thesis was to identify motion and postural features most salient to affective expressions and to exploit the identified features to develop computational models for affective movement recognition and generation that are robust to kinematic, interpersonal, and stochastic variations inherent to bodily expression of affect. I have adapted a stochastic graphical modeling approach based on the hidden Markov model to devise a novel hybrid discriminative-generative representation of body movements, augmented with a quantitative encoding of the Laban¹ components of the movements. I have developed a new quantification for Laban components in a collaboration with a certified Laban analyst². The resulting hybrid movement encoding was then used for the automatic estimation of affective expressions from movements as well as the adaptation of pre-defined motion paths in order to overlay affective content. Furthermore, I have conducted a series of perceptual user studies to explore the impact of kinematic embodiment and the observer's gender on affective movement perception.

Affective Movement Recognition

The below figure shows a schematic of the proposed recognition approach. The proposed approach derives a stochastic model of the affective movement dynamics using hidden Markov models (HMM)s. The resulting HMMs along with the forward algorithm are used to derive a Fisher score representation of the movements, which is subsequently used to optimize affective movement recognition using the linear support vector machine classification. The Fisher score for a movement is the partial derivative of the log-likelihood of the movement with respect to parameters of the class-specific HMMs.

Schematic of the proposed approach. 1) Class-specific raw movement observations are encoded in separate HMM models, 2) Using the forward algorithm, log likelihood of the movements with respect to class-specific HMMs is obtained, 3) the partial derivative of the resulting log likelihoods with respect to the class-specific HMMs is calculated (a Fisher score representation of the movements), 4) Affective movement recognition in the Fisher score space using SVM classification, 5) A discriminative lower-dimensional embedding of the Fisher scores is derived using sPCA, 6) Affective movement recognition in the resulting sPCA space using kNN classification , 7) Analysing salient discriminative movements features spanning the sPCA subspace.

In addition, I presented an approach to obtain a minimal discriminative representation of the movements using supervised principal component analysis (SPCA) that is based on maximizing Hilbert-Schmidt independence criterion between the movements and their labels in the Fisher score space. The dimensions of the resulting SPCA subspace consist of intrinsic movement features salient to affective movement recognition (Shown in the Figure below).

The identified full-body salient postures in the Fisher score space. Joint variances are also shown and color coded using spheres whose sizes are proportional to the variances. The postures are labeled and ordered according to their significance in constructing the three dimensional Fisher score space (rows of the Figure). The automatically identified postures are congruent with postures manually extracted as the most expressive in the same motion dataset in a perceptual user study.

These salient features enable a discriminative low-dimensional encoding of movements. The efficacy of the proposed approach in recognizing affective movements and identifying a minimal discriminative movement representation was demonstrated using two challenging affective movement datasets³.

Affective Movement Generation

In this work, I presented an approach for automatic affective movement generation that makes use of two movement abstractions: 1) Laban movement analysis (LMA), and 2) hidden Markov modeling. The Laban movement analysis provides a systematic tool for an abstract representation of the kinematic and expressive characteristics of movements. Given a desired motion path on which a target emotion is to be overlaid, the proposed approach searches a labeled dataset in the LMA Effort and Shape space for similar movements to the desired motion path that convey the target emotion. An HMM abstraction of the identified movements is obtained and used in the Viterbi algorithm to generate a novel movement that is a modulated version of the desired motion path conveying the target emotion. The extent of modulation can be varied, trading-off between kinematic and affective constraints in the generated movement. The proposed approach was tested using a full-body movement dataset. The efficacy of the proposed approach in generating movements with recognizable target emotions was assessed using a validated automatic recognition model³ and a user study. The target emotions were correctly recognized from the generated movements at a rate of 72% using the recognition model. Furthermore, participants in the user study were able to correctly perceive the target emotions from a sample of generated movements, although some cases of confusion were also observed.

A schematic of the affective movement generation approach.

^{1. The Laban system is a prominent movement notation system that is used
for writing and analyzing both the structure and expressivity of movements in dance choreography.↩}
^{2. A. Samadani, S. Burton, R. Gorbet, and D. Kulić Laban
Effort and Shape Analysis of Affective Hand and Arm Movements, 5th
International Conference of Affective Computing and Intelligent
Interaction, pp. 343 – 348, 2013 (pdf).
↩}
^{3. A. Samadani, R. Gorbet and D. Kulić, Affective
Movement Recognition based on Generative and Discriminative Stochastic
Dynamic Models, IEEE Transactions on Human-Machine Systems,
vol. 44, no. 4, pp. 454 – 467, 2014 (link). ↩}