Synaesthesia: How can it be used to enhance the audio-visual perception of music and multisensory design in digitally enhanced environments?

Svetlana Rudenko, João P. Cabral


Synaesthesia is a type of sensory cross-activation related to a specific wiring of the brain which has additional neuronal pathways between different cortexes, creating various sensory experiences, such as people “hearing colours” or “seeing sounds”. There is a great potential of applying the knowledge about synaesthesia in different fields. For example, it could lead to innovative developments of educational programs for the young age cross-modal perception, digital applications for children with autism, and technological solutions to train plasticity of ageing brain. Synaesthetes very often benefit from their condition, especially in production of creative works, transposing experience from one sensory modality to another. For example, mixing sounds to create colour palette, as Vincent van Gogh, or composing modes of limited transposition with very clear view of colours, as Olivier Messiaen. Both had the chromaesthesia type of synaesthesia. This paper reviews important achievements in modelling synaesthetes perceptual experiences, with a focus on audio-visual perceptions, such as 4D digital visualisation of musical texture/musical-space synaesthesia, Cognitive Musicology visuals for classical music based on cross-modal associations and composing for animated Art. We also discuss the implications of the cross-modal sensory output on visual digital applications. Our assumption is that applications of cross-modal perception models based on synaesthesia can contribute to more engaging experiences to users, including the possibility to stimulate plasticity of the brain through multisensory design and digitally enhanced environments in Augmented Reality (AR) and Virtual Reality (VR).


 DOI: 10.21437/SMM.2019-13

Cite as: Rudenko, S., Cabral, J.P. (2019) Synaesthesia: How can it be used to enhance the audio-visual perception of music and multisensory design in digitally enhanced environments?. Proc. SMM19, Workshop on Speech, Music and Mind 2019, 60-64, DOI: 10.21437/SMM.2019-13.


@inproceedings{Rudenko2019,
  author={Svetlana Rudenko and João P. Cabral},
  title={{Synaesthesia: How can it be used to enhance the audio-visual perception of music and multisensory design in digitally enhanced environments?}},
  year=2019,
  booktitle={Proc. SMM19, Workshop on Speech, Music and Mind 2019},
  pages={60--64},
  doi={10.21437/SMM.2019-13},
  url={http://dx.doi.org/10.21437/SMM.2019-13}
}