Noninvasive extraction of audiovisual cues for multimodal applications

Harouna Kabre

doi:10.1117/12.316534

8 July 1998 Noninvasive extraction of audiovisual cues for multimodal applications

Harouna Kabre

Proceedings Volume 3389, Hybrid Image and Signal Processing VI; (1998) https://doi.org/10.1117/12.316534
Event: Aerospace/Defense Sensing and Controls, 1998, Orlando, FL, United States

Abstract

We describe HOPS, a system for extracting some audiovisual cues for the modeling of a computer end-user environment. The objective of the study is to provide some reliable audiovisual cues in order to 'augment' the computer input devices set for multimodal applications. The system accepts an audio-visual scene as input and produces different kinds of events which could contribute to increase the awareness and robustness of interactive system. The described framework for the extraction of cues is ecological and homogenous. On the audio path a cross power spectrum method is applied for extracting different kind of acoustic patterns defined as acoustic segments. The acoustic signal from a microphone and the acoustic segments are firstly FFT- transformed, averaged, and secondly correlated in the spectral domain. The maxima of the inverse Fourier transform of this cross-power spectrum is the criteria for the detection of some acoustic events. On the video path, we define some initial color models of some desired cues such as mouth, eyes, etc. and then track them in the audiovisual scene recorded by a camera.

Citation Download Citation

Harouna Kabre "Noninvasive extraction of audiovisual cues for multimodal applications", Proc. SPIE 3389, Hybrid Image and Signal Processing VI, (8 July 1998); https://doi.org/10.1117/12.316534

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available