Multimodal Communication in LFG: Gestures and the Correspondence Architecture

In this paper we investigate the interaction between verbal language and the non-verbal behaviours that commonly accompany it. We focus on spontaneous hand gestures. We discuss the complex network of interactions between the two modalities and how we can model the interpretation of a multi-channel signal. We embed this model of interpretation in the LFG correspondence architecture and we show how the flow of linguistic information that characterizes the architecture can be used to make the interpretation more precise. The result is an enriched architecture in which a complex signal is first broken up into its component parts. The subcomponents are initially interpreted independently but are then fused at the end into a single meaning object. Our model can capture quite precisely the intuitive meaning associated with multimodal utterances.