Audio-Visual Integration Varies with Stimulus and Background Complexity in a Virtual Environment: Towards a Naturalistic Model of Multisensory Integration

Science Center, Bent Corridor

We are continually bombarded by information arriving to each of our senses; however, we perceive the world around us as unified and ordered. We are able to perceive many objects in our environment through more than one sensory modality, and the brain seems to effortlessly integrate the separate information into a unified percept. Although multisensory integration has been researched extensively using simple computer tasks and stimuli, it has not yet been investigated in a real world context. Virtual reality offers the perfect combination of realism and precise control over the environment to investigate this question. We have chosen to begin this area of investigation with a task known as the detection task. In its computer game version, participants are asked to detect a white circle, white noise burst, or a combination of the two as fast as possible. Participants are faster at detecting multisensory stimuli than either of the unisensory stimuli. In the virtual reality version of this task, participants detected stimuli varying in complexity within a virtual world that also varied in complexity. The environmental complexity was modulated using the three following virtual worlds: a gray room that contains no depth cues, texture, color, or environmental sounds; a room with textured walls that offers depth cues, texture, soft background white noise, but no color or environmental context; a room that mimics the experimental testing room that offers depth cues, texture, color, and identifiable background noise. The stimulus complexity was modulated using the following three conditions: white circle with no 3D shading and/or white noise burst; white circle with 3D shading thus appearing as a white ball and/or noise which is modulated in amplitude and frequency; tennis ball with 3D shading and color and/or a tennis ball sound. The visual and auditory stimulus complexity was always modulated together, and the world and stimulus complexity were combined to produce nine conditions. In all cases, participants were asked to detect the stimuli as fast as possible, and response times for the multisensory stimuli were compared to the response times for the unisensory stimuli within each condition. We found that all stimulus and environment complexity combinations resulted in significant multisensory integration; however, varying levels of stimulus complexity resulted in significantly different “amounts” of integration in the high complexity environment only. The results of this experiment offer tentative support for the real-world applicability of previous multisensory findings. Future studies will investigate how unpredictable stimulus features and location influence multisensory integration.