Platforms like Zoom, Skype and Microsoft Teams have made video-based communication an everyday part of many people’s work and personal lives. For many, however, the experience can fail to compare to in-person interactions. New research by professor Nikolaus Troje from York’s Centre for Vision Research reveals why.
Everyday in-person social interactions rely on a wealth of nuanced non-verbal communication cues that enrich verbal communication in the real world. The trouble with video platform communication, as Troje reveals in a journal article published in Trends in Cognitive Sciences, is that visual cues people normally pick up when communicating in-person can become misleading, disruptive and false over video platforms like Zoom and Skype, making communication not only more difficult, but also exhausting.
“The problem is that directional information, eye gaze in particularly, gets disrupted in video calls,” says Troje, who is also Canada Research Chair in Reality Research and director of the BioMotion Lab at York’s Centre for Vision Research. “In real life, if I meet someone’s eyes, I can rely on the fact that the other person experiences the same. The other person knows that I look at her and I know that she looks at me.”
Social gaze, as it is called, informs bonding, trust and rapport, moderates how people communicate emotions through facial expressions and negotiates turn taking. All of this affects the efficiency of communication.
“Social gaze during communication is a sophisticated dance during which eye contact is established and broken repeatedly,” says Troje. “Eye contact events trigger physiological responses, such as changes in pupil size, heart rate and skin conductance, in both partners and synchronize their affective states.”
If these visual cues are absent, such as in a phone conversation, people are able to adjust to their absence. The problem with video communication is that they are present, but misleading and deceptive. “The visual system and the rest of our body respond to them even though they are no longer faithful. Even though I may experience eye contact, my partner may not even see me, as she is looking into her camera instead.
“Real face-to-face meetings are way more efficient, productive and less exhausting,” says Troje. “This lack of efficiency using internet-based video communication extends to teaching, learning, employment interviews and therapy.”
Manipulating only the eyes in the video feed, as some new video filters do, does not fix the main problem and cannot re-establish natural, dynamic eye contact behaviour.
Troje’s paper points not just to the phenomenon, but possible solutions. New generations of video conferencing platforms need to be able to track user’s head location in real time, without lags, to drive a virtual camera. AI and machine learning can then be used to generate the changing views of the communication partner.
“Given the advent of fast on-board computer vision chips and new algorithms in computer vision and machine learning, that approach is feasible," says Troje. "The societal implications of fixing the issues are huge. Trust and rapport building through functional eye contact is essential for telehealth and business meetings alike.”