Interactions of Attention, Stimulus Conflict, and Multisensory Processing

Abstract

At every moment in life we are receiving input from multiple sensory modalities. We are limited, however, in the amount of information we can selectively attend to and fully process at any one time. The ability to integrate the relevant corresponding multisensory inputs together and to segregate other sensory information that is conflicting or distracting is therefore fundamental to our ability to successfully navigate through our complex environment. Such multisensory integration and segregation is done on the basis of temporal, spatial, and semantic cues, often aided by selective attention to particular inputs from one or multiple modalities. The precise nature of how attention interacts with multisensory perception, and how this ramifies behaviorally and neurally, has been largely underexplored. Here, in a series of six cognitive experiments in humans using auditory and visual stimuli, along with electroencephalography (EEG) measures of brain activity and behavioral measures of task performance, I examine the interactions between attention, stimulus conflict, and multisensory processing. I demonstrate that attention can spread across modalities in a pattern that closely follows the temporal linking of multisensory stimuli, while also engendering the spatial linking of such multisensory stimuli. When stimulus inputs either within audition or across modalities conflict, I observe an electrophysiological signature of the processing of this conflict that is similar to what had been previously observed within the visual modality. Moreover, using neural measures of attentional distraction, I show that when task-irrelevant stimulus input from one modality conflicts with task-relevant input from another, attention is initially pulled toward the conflicting irrelevant modality, thereby contributing to the observed impairment in task performance. Finally, I demonstrate that there are individual differences in multisensory temporal processing in the population, in particular between those with extensive action-video-game experience versus those with little. However, everyone appears to be susceptible to multisensory distraction, a finding that should be taken into serious consideration in today's complex world of multitasking.