Prof. Shinsuke Shimojo

Prof. Shimojo

ISP2012 Symposium Talk Abstract:

Sensory substitution, crossmodal plasticity, and the third kind of “qualia”

“Qualia” to some philosophers refers to the absolute, unique and subjective  quality of a conscious sensory experience, which may not be “explained away” by any sort of neurophysiological account or model. Whereas we do not endorse to the opinion that the qualia is the "hard" (i.e. impossible in principle) problem for science, we also agree that the current visual perceptual sciences failed to explain the "absolute, unique and subjective quality of conscious sensory experiences."  In this context, we may be able to find significant insights in the latest progresses in sensory substitution since they raise fundamental questions, such as what precisely are the basic characteristics of “vision-like” sensory processing.

The “vOICe” is one of the currently available visual substitution devices based on auditory inputs, primarily for blind people. It translates video inputs (X- and Y-axes) into auditory parameters (time and pitch, respectively). There are several “super users” who claim “visual” experiences. Moreover, at least one of them showed neural activity in higher-level visual cortical areas in fMRI, when engaged in a shape discrimination task relying on this device. 

In principle, we may come up with a brief list of psychophysical &  neuroscientific criteria to acquire new “visual” experiences, and I will describe some data that our laboratory recently obtained along this line.

The psychophysical results suggest that some aspects of perceptual constancy (such as orientation or shape) can be attained by the vOICe training in sighted subjects, but that the post-training performance is mainly based on top-down cognitive strategy, rather than the typical, vision-like, automatic processes. To make it more automatic and effortless, dynamic sampling, i.e. sensory feedback from self action, turned out to be critical. Our fMRI data, albeit preliminary, indicate V3 activation.

All these results, as well as the vast majority of the literature on sensory substitution, have been base on the assumption that before training, the participants can perform the task only at the baseline chance level. It turned out not to be true, due to some intrinsic (synesthetic) mapping across modalities. Texture discrimination is the best example so far in our case with the vOICe, where untrained sighted subjects can grossly outperform the chance level and training does not necessarily add much.

Considering all these findings together, qualia, if still want to use such a word, should be understood with regard to adaptive behavior. Consequently, the subjective sensory experience which is acquired via excessive training/experiences with the sensory substitution device is not entirely “visual,” nor “auditory.” Instead,  it may be characterized best as “the third kind of qualia.”

Prof. Shimojo's ISP2012 Lecture Talk Abstract
Prof. Shimojo's ISP2012 Profile
Prof. Shimojo's Caltech Profile

ISP2012 Abstracts