Does your child ever communicate without using words? Multimodal communication is simply communication through “modes”. Multiwhata? What did you say? Some examples of modes may be verbal, pictures, gestures, sign language, etc. Multimodal communicators use more than one “mode” to communicate.
In the high technology world we live in, we use many modes of communicating daily. Next time you have a chance, ask your cell phone provider how many text messages you send per month. Sometimes, verbally communicating is not the easiest way for us to communicate. A quick, “I forgot eggs for my recipe, can you get them on the way home?” text message is much more efficient than a phone call. The same concept holds true for children who are struggling to speak verbally. Sometimes, other forms of communicating are more efficient at meeting their needs.
“Wait a minute”, you say to yourself. “I want my child to communicate verbally, I don’t want my therapist to stop working on that and I’m afraid if they use one of these ways to communicate they’ll stop wanting to talk”. That’s a common fear of parents, but let me tell you it is a myth. Study after study continues to show that utilizing varying ways to communicate does NOT (I repeat does NOT) hinder or interfere with the development of speech. So why is your therapist talking about introducing all of these non-verbal ways of communicating? Your therapist only wants to alleviate your child’s frustration. The goal of your speech therapist is giving your child a way to communicate their needs and wants in a more efficient way. By teaching your child more than one mode of communicating in a systematic way, you are giving them a greater opportunity to express themselves.
Multimodal Communication and How They Can Help:
Signs: The use of specific hand gestures representing true words can help children to communicate what they want across environments. Signs are typically introduced first to children who are struggling to produce spoken language. Spoken words are a symbolic system, and when we speak we are exchanging “symbols” or words. This can help teach children how to exchange “symbols” or words in the absence of verbal language.
Visuals:Visuals are typically used for quick and easy communication for a specific purpose. For example, your therapist might introduce a YES or NO board to your child. This way, your child can easily communicate their response by pointing to their answer. Before your flags go off, this is a great way to teach the difference between yes and no before it can be used verbally. For children with difficulty communicating verbally, they often misuse the words yes or no. Some other quick visuals can be, “bathroom” or “break”. Another easy way to implement visuals at home is having pictures of items placed on or next to their corresponding real life representations. For example, in the kitchen have a picture of “food” posted on the refrigerator. These pictures are typically placed in an easily accessible place within a specific environment. These visuals can immediately alleviate frustration for children.
PECS: The Picture Exchange Communication System is form of communication where your child will exchange pictures for their desired items (hence the name). This is different from visuals. PECS is intended for use when communicating for a wide variety of items or actions and carried with the child across all environments. PECS is taught in many phases, by a trained PECS speech therapist, each one encouraging your child to become more and more independent with communicating. For children, this is often how they learn their communication and language is meaningful. Over time, this mode of communicating can alleviate frustration and teach children how to use communication to express themselves versus a meltdown.
Speech Generating Devices: These devices are typically in tablet form with a variety of “buttons” that have picture representations embedded within them. A child will then press their desired “button” for their desired object, and the device will produce the verbal output. There are many common misconceptions when talking about SGDs or assistive and augmentative communication devices. However, these devices that generate speech are specifically formatted for your child and can improve or increase verbal language output. Not only do these devices model language constantly, they can also continue to teach children how to use language.
I know that all of these alternative ways of communicating can seem overwhelming. Just remember, your child’s therapist is only trying to immediately alleviate the frustration your child feels in not being able to communicate needs and wants. It is okay for your child to use sign language and PECS, or visuals and an SGD, or any other combination of modes of communication. Studies show that when therapists introduce these modes of communicating early, children can increase vocalizations and improve overall speech abilities. Your child’s therapist will continue to model spoken word when using multimodal communication. Remember, spoken word will still be the target and utilized when teaching and using these alternative modes of communication.
1. Speaker characterization in a multimodal context
Speaker recognition consists of determining who spoke when, where the identity can be that of the true speaker or an identity specific to one document or a set of documents. Different sources of information can be used to identify the speaker in multimedia documents (the speaker's voice, what is said, or what is written. The group is leading the QCOMPERE consortium for the REPERE challenge.
2. Affective and social dimensions of spoken interactions
Affective and social dimension detection are being applied to both human-machine interaction with robots and in the analysis of audiovisual and audio documents such as call center data. The main research subjects in this area are emotion and social cues identification in human-robot interaction, emotion detection based on verbal and non verbal cues (acoustic, visual and multimodal), dynamic user profile (emotional and interactional dimensions) in dialog for assistive robotics, and multimodal detection of the anxiety applied to therapeutic serious games.
3. Perception and automatic processing of variation in speech
The very large corpora used for training statistical models are exploited for linguistic studies of spoken language, such as acoustic-phonetics, pronunciation variation and diacronic evolution. Automatic alignment enables studies on hundreds to thousands of hours of data, permitting the validation of hypotheses and models. This topic also studies human and machine transcription errors via perception experiments.
4. Robust analysis of spoken language and dialog systems
Robust analysis methods for the spoken language are developed in the framework of open domain information retrieval with applications to language understanding for dialog systems, to named-entity recognition, and to interactive question answering systems supporting both spoken and written languages.
5. Automatic translation and machine learning
Research activities on statistical machine translation of speech or text focus on the design and development of novel language and translation models as well as novel decoding strategies; this activity is closely related to the development of machine learning methodologies for multilingual Natural Language Processing applications.
6. Speech recognition
Speech recognition is the process of transcribing the speech signal into text. Depending upon the targeted use, the transcription can be completed with punctuation, with paralinguistic information such as hesitations, laughter or breath noises. Research on speech recognition relies on supporting research in acoustic-phonetic modeling, lexical modeling and language modeling (a problem also addressed for machine translation), which are undertaken in a multilingual context (18 languages). This topic also includes research on language recognition, that is determining the language and/or dialect of an audio document for both wideband and telephone band speech.
7. Language resources
In addition to the collection, annotation and sharing of varied corpora, this research topic addresses more general investigations on Language Resources, covering data, tools, evaluation and meta-resources (guidelines, methodologies, metadata, best Practice), for spoken and written language, but also for multilingual, multimodal, and mutimedia data. Those activities are mostly conducted in collaboration with national and international organizations and networks.