Multimodality in language and speech systems (Q1408491)

This book is based on contributions to the 7th European Summer School on Language and Speech Communication that was held in Stockholm, Sweden, in July 1999. It contains eight chapters dealing with various topics related to multi-modality in language and speech systems. The book starts from studies of human-to-human communication behavior involving bodily communication, as well as speech and gesture, and also covers issues of audio-visual speech perception. The section on multi-modality in alternative communication mediated by machine discusses different approaches to solving communication problems of people with sensory, physical or cognitive impairments which can be overcome by multi-modal systems. From the perspective of modality theory, the communication between humans and systems is structured in terms of a taxonomy and systematic analysis of input and output modalities, their representations and functions, and a discussion of practical problems of modality selection. This modality overview is followed by contributions in which various aspects of experimental language and speech systems are described and tested. The architecture of the Chameleon system is presented where input includes spoken language and pointing gestures and output comprises speech and a laser pointing system. A computational model of natural turn-taking in goaloriented face-to-face multi-modal dialogues (including speech, gestures and body language) is introduced, and visual and auditory conversational signals for 3D artificial talking heads as interactive interface agents are dealt with so that speech gestures and face and head movements are properly synchronized.