Multimodal language processing in human communication

J Holler, SC Levinson - Trends in Cognitive Sciences, 2019 - cell.com
The natural ecology of human language is face-to-face interaction comprising the exchange
of a plethora of multimodal signals. Trying to understand the psycholinguistic processing of …

Depicting as a method of communication.

HH Clark - Psychological review, 2016 - psycnet.apa.org
In everyday discourse, people describe and point at things, but they also depict things with
their hands, arms, head, face, eyes, voice, and body, with and without props. Examples are …

Speech gesture generation from the trimodal context of text, audio, and speaker identity

Y Yoon, B Cha, JH Lee, M Jang, J Lee, J Kim… - ACM Transactions on …, 2020 - dl.acm.org
For human-like agents, including virtual avatars and social robots, making proper gestures
while speaking is crucial in human-agent interaction. Co-speech gestures enhance …

Virtual reality: A game-changing method for the language sciences

D Peeters - Psychonomic bulletin & review, 2019 - Springer
This paper introduces virtual reality as an experimental method for the language sciences
and provides a review of recent studies using the method to answer fundamental …

[HTML][HTML] Origins of vocal-entangled gesture

W Pouw, S Fuchs - Neuroscience & Biobehavioral Reviews, 2022 - Elsevier
Gestures during speaking are typically understood in a representational framework: they
represent absent or distal states of affairs by means of pointing, resemblance, or symbolic …

Gesture–speech physics: The biomechanical basis for the emergence of gesture–speech synchrony.

W Pouw, SJ Harrison, JA Dixon - Journal of Experimental …, 2020 - psycnet.apa.org
The phenomenon of gesture–speech synchrony involves tight coupling of prosodic contrasts
in gesture movement (eg, peak velocity) and speech (eg, peaks in fundamental frequency; …

Acoustic information about upper limb movement in voicing

W Pouw, A Paxton, SJ Harrison… - Proceedings of the …, 2020 - National Acad Sciences
We show that the human voice has complex acoustic qualities that are directly coupled to
peripheral musculoskeletal tensioning of the body, such as subtle wrist movements. In this …

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

W Pouw, JP Trujillo, JA Dixon - Behavior research methods, 2020 - Springer
There is increasing evidence that hand gestures and speech synchronize their activity on
multiple dimensions and timescales. For example, gesture's kinematic peaks (eg, maximum …

Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures

W Pouw, SJ Harrison, N Esteve-Gibert… - The Journal of the …, 2020 - pubs.aip.org
Expressive moments in communicative hand gestures often align with emphatic stress in
speech. It has recently been found that acoustic markers of emphatic stress arise naturally …

Co-thought and co-speech gestures are generated by the same action generation process.

M Chu, S Kita - Journal of Experimental Psychology: Learning …, 2016 - psycnet.apa.org
People spontaneously gesture when they speak (co-speech gestures) and when they solve
problems silently (co-thought gestures). In this study, we first explored the relationship …