Linguistic and cultural exchanges facilitate understanding. Photo: TUCHONG

At present, research on linguistic communication from a multimodal perspective has become an international hot topic and it is in its ascendency in China as well.

Linguistic communication is fundamentally multimodal, yet traditional linguistic research has largely ignored the fact that the meaning of human linguistic communication is constructed by multiple modalities, leading to insufficient understanding of the essence of linguistic communication. The transition from unimodal to multimodal research in linguistic communication research reflects a brand new view of language. 

Multimodality

Within different fields and approaches, scholars have varying definitions of “modality,” such as the sensory and corresponding neural systems, symbolic resources, and modes of information presentation and so on. However modality is defined, it is based on the fact that the construction, transmission and understanding of meaning in human linguistic communication require the invocation of a variety of symbolic resources and multimodal coordination of various sensory systems and corresponding brain areas. For example, processing textual symbols requires invoking the visual modal system, the olfactory modal system for processing olfactory symbols, and the auditory modal system for processing auditory symbols.

From the perspective of brain mechanisms for language comprehension and production, language studies based on sensory multimodalities suggest that individual language development and decline are closely related to multimodal sensory functions. Children’s language abilities are gradually constructed and cultivated with the support of multimodal sensory inputs after birth. Therefore, it is advisable to investigate topics such as child language acquisition, adult linguistic communication, language attrition among the elderly, and various language disorders from a multimodal perspective.

Starting from a multimodal perspective, we examine the development and evolution of individual linguistic communication abilities, as well as study multimodal discourse in linguistic communication, which greatly expands its scope and dimensions.

Research paradigm and approach

Multimodal research represents a new paradigm of linguistic communication research. This paradigm is based on the premise that the occurrence and development of linguistic communication and the communication between human and the outside world require support from the multimodal sensory system. This paradigm focuses on the mechanisms of meaning expression and transmission that have been overlooked in traditional linguistic research. 

Within this paradigm, the overarching goal of relevant research includes exploring brain mechanisms and behavioral patterns of multimodal linguistic communication, as well as facilitating multimodal research and development (R&D) for applications based on modern technology and future advancements. Such research is both problem-oriented and interdisciplinary in nature.

A research paradigm can encompass various research paths and can be applied to multiple specific research fields. At present, multimodal research paths related to linguistic communication generally include: multimodal analysis of semiotic attributes, multimodal corpus methods, research on neurocognitive mechanisms of multimodal phenomena, and R&D of multimodal human-machine interaction technologies.

Multimodal research with semiotic attributes has matured significantly both domestically and internationally, yielding abundant results. Multimodal corpora refer to integrated collections of various data types, such as text, audios and videos, which researchers can process, retrieve, and calculate using multimodal methods for relevant studies. Research on the neurocognitive mechanisms of multimodal phenomena has become a hot topic. The multimodal nature of linguistic communication requires attention from neurosciences to the synergistic effect of multiple brain regions involved in speech.

Core issues and future development

At present, the connotations and extensions of multimodal language research continue to expand, and efforts should be made to deepen research going forward in the following four aspects.

Firstly, it is suggested to continue to strengthen research on the relationship between multi-symbolic modalities, refining both interpretive theories and analytical frameworks. Secondly, it is necessary to expand research of multimodal expression disorders and compensation strategies in special population. Thirdly, neurocognitive research on multimodal phenomena of linguistic communication should be conducted. Fourthly, it is suggested to optimize the development of human-machine interaction technology or multi-modal intervention methods to promote individual linguistic communication.

Zhang Delu (professor) and Huang Lihe (associate professor) are from the School of Foreign Studies at Tongji University.

Edited by ZHAO YUAN