It was a Friday night. Myself and my two best friends were deciding for what felt like an eternity how we wanted to spend our night. Sally, I will call her, wanted to spend the night in, watching Netflix and eating pizza. Ally wanted to go out and grab some drinks at a local bar. I was indifferent and would have been satisfied with either decision. Sally said she had a long day and just wanted to enjoy some quiet time at home. Ally, on the other hand, said she thought she deserved a night out after a long and stressful week. The two went back and forth until we decided that “Netflix and chilling” was how we would be spending our Friday night. Sally was content with our decision, but she looked over to Ally and noticed she did not look too thrilled. She asked Ally if she was alright with the decision. She smiled. Sally interpreted Ally’s smile as confirmation that she was happy with the decision and so she continued on to enjoy her night. Myself, on the other hand, I did not see contentment in Ally’s overall disposition. Something about her body – the slouching and her head hanging into her chest (Wallbott, 1998) – was telling me otherwise.
There is a lot going on in this above scenario. Sally and Ally were obviously in disagreeance about how they wanted to spend their night. Although Ally’s facial expression appeared to denote that she approved the decision, her body posture clearly communicated something different. Both of these aspects of Ally’s behavior make-up her body language. Many experts, including Joe Navarro, a former FBI agent turned body language expert, define body language as “the unspoken element of communication that we use to reveal our true feelings and emotions. Our gesture, facial expressions and posture, for instance” (“Body Language,” n.d.). Whether intentional or not, Ally used two different forms of body language to communicate how she was feeling. She used a smile to communicate that she appeared to be content with the decision and a sluggish body posture to communicate that she may actually not be. Evidently, Sally was attending to Ally’s smile to uncover how she was feeling – she failed to take into consideration her body posture.
The idea of using someone’s facial expressions to recognize what emotion they are feeling is not new. Emotion researchers Dael, Mortillaro, and Scherer (2012) share that “it has long been assumed that whereas a number of facial muscle configurations are reliable indicators of specific emotions, body movements or posture provide information of only gross affect state or emotion intensity” (p.1085). Essentially, due to the fact that certain emotions are associated with specific facial expressions, we as a species tend to use these expressions to distinguish one emotion from the next. For example, a smile is a globally recognized way of expressing happiness, and a frown, sadness (Ekman & Friesen, 1969).
These tendenices can be attributed to the works of Dr. Paul Ekman, who has been deemed the father of emotions and their expressions. In 1969 and later in 1971, alongside Wallace Friesen, the two men discovered that the emotions of happiness, sadness, anger, disgust, surprise, and fear, known as the basic emotions, each had distinct facial expressions that made them differentiable from one another (Ekman & Friesen, 1969). They also found that people across cultures were able to recognize these expressions with considerable accuracy when provided images of actors expressing these emotions on their faces (Ekman & Friesen, 1971). From these studies, we learn that facial expressions are important in detecting what emotion an individual is portraying. Likewise, they are also important in communicating an emotion. As a result, it might have been fair to assume in earlier days that facial expressions were the meat and bones of detecting an emotion. Nonetheless, Dael et al. (2012) share that “this view is no longer supported by recent studies showing that variations in body movement and posture convey specific information about a person’s emotional state” (p.1086). After all, the word emotion is derived from the latin word emovere which literally means to move (- movere) out (e-). Thus, body movement must play some part in the recognition of emotions (“Emotion,” n.d.).
My skepticism in regards to Sally’s interpretation of Ally’s emotions was in part due to Ally’s face “saying” one thing, and her body “saying” another. I was put off by these differences in her facial and body expressions. Consequently, I truthfully did not know what to make of Ally’s feelings – was she upset, was she happy? Naturally, as a Psychology student, I turned to the research to find some answers. “What insightful and revolutionary answers did you find?,” you may be asking yourself. To put it simply, body expressions such as body postures and gestures, have extremely powerful roles in the recognition of emotions. What I found might sound trivial, perhaps even like common sense, but nonetheless it is an important and often overlooked idea in emotion research and literature (e.g. Schindler, Van Gool & de Gelder, 2008; Dael et al., 2012).
The Power of Incongruence
One way in which body expressions influence the recognition of emotions is illuminated through the idea of incongruence – a mismatch between the emotions displayed in one’s face and body (Meeren, van Heijnsbergen, & de Gelder, 2005). As I briefly mentioned, Ally’s facial display of contentment (i.e. her smile), did not match her sluggish body posture. Now, imagine yourself in a similar and very common situation. Say your boss reads your business proposal, smiles and appears to like what they see, but, they do not make eye contact and continuously look out the window (“Body Language,” n.d.). What do you make of this? Is the body trying to say something? Or is it affecting our processing of the facial expression? Meeren et al. (2005), emotions researchers from Tilburg and Harvard University, share that “incongruent bodily expression hampers facial expression recognition” (p. 16521). Considering all the ambiguities that can arise from a mismatch between facial and body expressions in everyday life, these three researchers were the first to study how such a mismatch between body movements and facial expression can influence the detection of an emotion an individual is trying to display from their face.
Twelve participants were brought into Meeren and colleague’s cognitive and neuroscience laboratory at Tilburg University in the Netherlands. The researchers put together different images showing an individual either portraying fear or portraying anger. However, the images were altered such that some pictures had congruent facial expressions and body postures and some had incongruent facial expressions and body postures. Congruent images included an actor whose fearful facial expression matched their fearful body posture, and likewise for anger. Incongruent images included an actor displaying an angry face, but a fearful body posture, or an actor displaying a fearful face, but an angry body posture. All faces were photoshopped onto the body of an actor. The participants were then simply asked to determine if the actor was expressing anger or fear. But there was a catch, participants had to make these judgments based on the facial expressions, even while they were viewing the whole face-body image (Meeren et al., 2005).
It turns out that participants were not too great in accurately detecting whether the actor was expressing fear or anger when the facial expressions and body posture were incongruent. In fact, they were only right sixty-seven percent of the time (Meeren et al., 2005).
“When observers have to make judgments about a facial expression,” Meeren et al. (2005) observed, “their perception is biased toward the emotional expression conveyed by the body” (p.16521). The idea here is that our overall body is like a sealed container, and our body language, especially our postures and gestures can “leak out” our true feelings (e.g. Ekman & Friesen, 1969). Thus, we become more attuned to these body expressions because they perhaps share an individual’s true feelings. Relating this back to my experience with Ally, it is now clear that I was so focused on Ally’s body expression that it inhibited me from interpreting any emotion from her face. Does this mean I was wrong in suspecting that Ally might have been upset?
Definitely not! If we think of our facial expressions as “emotion seeds,” Aviezer et al. (2008) explain, “that lie dormant in isolated faces, they can be activated by appropriate context….These seeds ‘sprout’ and may override the original expression of the target face” (p.731). This metaphor suggests that context, especially body movement, allows the facial expression to “grow” or “morph” into what the context is suggesting.
In their research, Aviezer et al. (2008) conducted three studies to show how our perception of someone’s emotion can change in relation to the context in which the facial expression is elicited. In one of their studies, undergraduate students at the University of Toronto were presented with images of actors who expressed disgust on their faces. However, this facial display of disgust was then paired with different contexts. Interestingly, the differing contexts were actually different body postures. Thus, the researchers were extending work on previous studies that looked at incongruency in facial and body expressions. Disgust faces were either paired with fear, anger, sadness, or disgust body expressions. Again, participants were simply asked to indicate what emotion the facial expressions elicited, choosing between the basic emotions of sadness, anger, fear, disgust, happiness, and surprise.
Consistent with previous work, when a disgust facial expression was matched with a contextual display of disgust (i.e. the body showed disgust), participants accurately identified this emotion ninety-one percent of the time. However, when the contextual display showed anger, fear, or sadness, accuracy of judging the face as disgusted decreased. “This systematic decline in accuracy indicates that participants did not simply ignore the faces” (Aviezer et al., 2008, p. 726), the researchers explain. Moreover, perhaps most interesting of the results was that when the context (i.e. the body) showed anger, eighty-seven percent of participants identified the facial expression as eliciting anger rather than disgust. The authors continue to explain that “facial expressions can be perceived as conveying strikingly different emotions depending on the bodily context in which they appear” (Aviezer et al., 2008, p. 731). Although Ally’s facial and body expressions showed two extreme ends of emotions, this work suggests that her body provided some context for me to make sense of her facial expression. She was smiling, but to me her body suggested she was upset, and I used that information to re-interpret the emotion she was displaying on her face.
Let us take a pause and recall what we have learned thus far regarding how powerful body expressions can be in the recognition of emotion. Essentially, body expressions can “hamper” (Meeren et al., 2005, p. 16521) our perceptions of an emotion being expressed through the face because we typically attend to these contextual cues to make sense of the emotion. This is especially prevalent when body and facial expressions are incongruent. The next natural question becomes, why are we so influenced by incongruency in body and facial expressions? Is there something that is happening in the brain when facial and body expressions do not line up?
Rajhans, Jessen, Missana, and Grossmann (2016) sought out to determine exactly that. They were interested in learning what brain areas become activated when infants partake in emotion perception – that is, when attached to neural imaging machines, what brain areas light up when infants are trying to recognize an emotion from both the body and face? They were specifically interested in learning how incongruent body expressions impact the processing of facial expressions in the brain. Their work has demonstrated that this incongruency effect can be seen in the neural network of infants as young as 8-months-old, suggesting that this bias is part of our development.
Thirty-two eight-month old babies were shown avatars displaying either a fearful or happy emotional body expression. The avatars were followed either by a happy or fearful facial expression. Like previous body expression studies, there was a twist – some avatars were followed by incongruent facial expressions. This resulted in four possible combinations in which the emotional displays could have been presented: a happy body expression followed by a happy face, a fearful body expression followed by a fearful face, a happy body expression followed by a fearful face, and a fearful body expression followed by a happy face. The first two displays show congruence, meaning that the display of body and facial expressions match up in the emotion they are trying to express, and the latter two displays show incongruence. To measure the infant’s brain activity during these presentations, the researchers attached electrodes to an elastic cap placed around the infants’ heads, and used electroencephalography methods to record activation of brain regions (Rajhans et al., 2016).
The procedures Rajhans and colleagues used to get to their results are rather complex, but nonetheless their work illuminates some important information regarding how body expression influences the processing of facial expressions, especially when incongruent. They explain that “viewing conflicting body expressions hampers the neural discrimination of facial expressions in infants…suggest[ing] that early in development body expressions provide an important context in which facial expressions are processed” (Rajhans et al., 2016, p.119).
Thus, these converging lines of research tell us that one’s body movements can influence our perceptions of one’s emotions, especially if the body expression does not match the facial expression. With this knowledge, I now understand why I was heavily influenced by Ally’s slouchy and passive body posture. Nonetheless, I still do not know if my interpretation of her emotions were correct. What could have Ally done to make it easier for myself and Sally to detect her true emotions?
The Power of Congruency
We have looked at the powerful consequences that incongruent facial and body expressions have on emotion recognition. Paradoxically, body expressions also play a powerful role in regards to emotion recognition when they match one’s facial expressions. Wang, Xia & Zhang (2017) say that “congruent emotional body language facilitates the recognition of facial expressions” (p. 10). In other words, when body expressions are congruent to facial expressions – that is, when facial expressions line up with body expressions – it increases the accuracy of what the face is trying to express. Given the research I am about to present, I speculate that if Ally showed similar emotion displays in her face and body, Sally and I might have been able to accurately detect her feelings.
Much like Rajhans et al. (2016), Wang and colleagues became interested in studying the underlying neural processes involved in integrating both facial and bodily expression cues to recognize emotions, specifically in instances of victory or defeat. Their work follows similar procedures of Aviezer, Trope, and Todorov (2012) who found that when participants rated how intense as well as how positive/negative the emotion of a tennis player who had just won or lost a match was, body expression over facial expression was more influential in accurately recognizing the emotion the player was eliciting when the two modalities were incongruent. This finding seems to contradict what other researchers have found regarding mismatches between body and facial expression, and so Wang et al. (2017) decided to build upon this novel finding.
Three sets of participants were brought to the lab, where twenty-one undergraduate students took part in the neural aspect of the study, fifteen took part in rating valence – how positive or negative an emotion is – and intensity – how strongly an emotion is felt or displayed – of tennis players who had either just won or lost a point, and nineteen took part in identifying what emotion was being displayed by facial expressions. Participants in each study were presented with a range of images falling into one of four categories: win face-win body, lose face-lose body, win face-lose body, and lose face-win body. In the neural study, event-related potentials were recorded using electroencephalography technology as participants determined how positive or negative the facial expression was of the person in the image. In the second study, subjects were asked to rate how positive or negative and how intense both the facial and body expressions were. In the third study, participants were asked to determine what emotion was being portrayed by the faces, even while looking at the whole face-body image (Wang et al., 2017).
Event-related potentials showed that accuracy of facial expression emotions were higher when actors’ facial expressions and body expressions were congruent compared to incongruent. Furthermore, the researchers found that when participants were asked to rate how positive or negative the target’s facial expressions were, congruent images were more accurately rated as positive or negative compared to incongruent images. Likewise, when rating how intense the facial expressions were, congruent images more accurately depicted how intense the expression was compared to incongruent images. Similar results were found for ratings of body valence (how positive or negative the emotion elicited by the body was) and intensity. When it came to identifying the emotions from these congruent or incongruent images, win face-win body pictures were frequently identified as happiness, lose face-lose body pictures were frequently identified as sadness, win face-lose body were identified as happiness and anger, and lastly, lose face-win body pictures were identified as happiness and sadness. Evidently, the incongruence images led to some ambiguity and thus inaccurate emotion detections. Wang et al. (2017) state, like previous researchers, that “body context influence[s] the recognition of facial expressions” (p. 10). If this is the case, similarly displaying the emotion in our bodies to that of the emotion in our face might reduce how influenced we are by the body expressions. This may lead us to believe that the best way to have someone recognize our emotions is to display similar expressions in both our face and body.
If this evidence is not convincing enough, a decade prior to Wang et al.’s (2017) work, emotion researchers Van den Stock, Righart, and de Gelder (2007) were interested in studying how body expressions may influence the recognition of emotion that is being elicited from the face. In their work, Van den Stock and colleagues piggy-backed off the work of Meeren et al. (2005) by using fear and happiness target images, instead of fear and anger. “Furthermore,” the researchers said, “morphed faces were used in order to test whether individuals use information from bodies differently when facial expressions are ambiguously positioned between fear and happy” (van den Stock et al., 2007, p. 490).
In a series of three studies, the second study involved using images of five faces that ranged along a continuum from fear to happiness, with faces starting off as fearful and gradually morphing into happy expressions. These faces were then placed on a fearful body expression and later placed on a happy body expression, creating a total of ten images. Fourteen undergraduate psychology participants were simply asked to indicate if the target was expressing happiness or fear.
Not surprisingly, a happy face and body led to more participants indicating that the face was expressing happiness. However, when a happy face was on a fearful body, the face was recognized less as happy. Furthermore, fearful faces on fearful bodies resulted in participants rating the fearful face as more fearful. Similarly, a fearful face on a happy body resulted in the face being recognized as less fearful.
“The results of [this experiment] provide clear evidence,” the researchers observed, “that recognition of facial expressions is influenced by the accompanying body language” (van den Stock et al., 2007, p.491), again suggesting that the best way to accurately detect an emotion is to ensure facial and body expressions match. Perhaps this is what Ally should have done. If she was upset, it might have been useful to display a frown alongside her sluggish body. If she was content, maybe she should have displayed a more open and erect body posture.
Now, you might have noticed that although these studies tend to converge on the idea that body expressions influence the recognition of an emotion from facial expressions (e.g. Mereen et al., 2005), they all use very posed, unnatural and directed images of actors. For example, viewing an image where the target is directed to express fear in both their face and body leads to a very staged and prototypical depiction of what fear looks like, perhaps making it easier for the participants to distinguish the emotion. If images were more realistic and natural, would the findings still hold? This is exactly what Abramson, Marom, Petranker and Aviezer (2017) were determined to find out!
“We were particularly interested in examining whether the face and body show different patterns of recognizability in real-life versus instructed stimuli” (Abramson et al., 2017, p.558), the researchers shared. Twenty-five participants joined the study where they were presented with a target male or female displaying an emotion. The images they viewed ranged from posed (instructed) to natural (real-life) displays of emotion. The instructed-emotion pictures consisted of preexisting images found in databases that were used in other studies. To construct such images, Abramson and his colleagues used Photoshop to assemble congruent facial and body expressions of a particular emotion. The real-life emotion pictures consisted of images found on Google that either showed fear or anger. “We used Google images and Google videos to search for real-life expressions of emotion during situations that likely elicited fear or anger” (Abramson et al., 2017, p.559), the authors explained. Specific phrases that would typically elicit anger, such as sports fan brawls, or fear, such as haunted house reactions, were typed into a Google search. Once these instructed and real-life images were prepared, Photoshop was used one more time to create three cue conditions: face alone, body alone, and face with body (Abramson et al., 2017).
The participants’ task was relatively easy. They simply had to choose what emotion best described the target in the image. Their choices included anger, fear, happiness, sadness, disgust, surprise, and neutral. However, the researchers were mainly interested in determining how recognition of fear and anger differed in instructed images compared to real-life images.
Focusing on the results of the fear expressions, when asked to determine the emotion from instructed-images, participants were best at recognizing the emotion when the face and body were both shown. The results were similar when looking at real-life images. However, what was at utmost interest to Abramson et al. (2017) was, “to what extent the face and body concurrently expressed the same emotion” (p. 560). They examined this by looking at what percent of participants actually gave the correct emotion response when the face and body were presented. Interestingly, when fear was shown in the instructed-images, sixty percent of participants reported that the face and body showed fear, but in the real-life images this number dramatically dropped to thirty-three percent (Abramson et al., 2017). When looking at the results from anger expressions, similar findings were found, suggesting that the face and body do not concurrently express the same emotion in real-life situations.
“Perhaps the most striking…difference between instructed and real-life stimuli emerged with fearful stimuli” (Abramson et al., 2017, p. 561), explained the authors. The researchers found that in real-life images, fear was better recognized from the body than the face. “This novel finding suggests that in real-life, the fearful face conveys…ambiguity and the expressive body may act as a contextualizing agent” (Abramson et al., 2017, p. 561). Thus, in some emotional situations, like fear, our bodies are responsible for giving away our emotions.
What is extremely interesting about this study is that is subtly contradicts all the current research regarding the effects of congruent face and body expressions on emotion recognition. Previous work shows we are very good at detecting someone’s emotion when their face and body expressions match (e.g. Meeren et al., 2005). However, as mentioned, these results were based on participants looking at very staged photos. In real-life however, it turns out that even when face and body expressions are thought to match, they may not appear to, thus complicating the matter even more. With that being said, should Ally have tried to match her face and body expressions so Sally and I could have accurately detected her emotions? In all honesty, I believe it might have been helpful, however, given what Abramson and his colleagues have found, it is possible that it may not help.
What Does this all Mean for us?
Given this plethora of converging and some contradictory evidence regarding the power of body expression in emotion recognition, I believe there are some key pieces of information we as a society can take away from this. Yes, our facial expressions can communicate an emotion, but our facial expressions are never in complete isolation. Our bodies are always present, also providing some sort of information as well. Thus, I want to stress the importance that body expression has in the domain of communication. Knowing that our body expressions can influence our overall perception of emotions, we should keep this information in the back of our minds when we are trying to communicate with others. How should we communicate? What should we watch out for?
In line with what the research suggests about congruent facial and body expressions, it might serve useful to ensure that your facial and body expressions match when expressing an emotion – especially when you want to properly communicate something to others. A mismatch between your body and facial expressions leads to ambiguity. As a result, the receiver will have a difficult time attempting to uncover what emotion you are trying to express, inevitably leading to a misinterpretation of the intended emotion. This, I believe, is one of the reasons many couples, friends, siblings, parents, and any two people in a relationship for that matter have disagreements. The sender ineffectively communicates their emotions because there is a mismatch between facial and body expressions, and the receiver is now left to use these incongruent expressions to formulate some sort of message the sender is trying to communicate. The receiver may be right in their detection, but chances are, according to the research, that the body will influence and take precedence over what the face is trying to say, perhaps leading to a false identification of emotion.
For example, my sister recently asked me to help her with an assignment. I smiled and told her I would, but apparently my body expression suggested otherwise. She was upset with me because she believed, “[I] was showing no interest in helping.” What we have here is a classic case of miscommunication as a direct result of the incongruency between my face (and in this case my words) and body.
Thus, in order to combat this miscommunication, we must be careful in our facial and bodily expressions. Although we may not always be consciously aware that our body and facial expressions are incongruent, we can take the information we learned here to begin to make a conscious effort to try and fix that. And yes, the research has shown that even when our face and body are congruent in expressions, it may not always lead to correct emotion detection, but it nonetheless is a good starting point and place to try.
In sum, body expressions play a powerful role in the recognition of emotions. Specifically, body expressions that do not match facial expressions alter our perceptions of one’s emotions (e.g. Meeren et al., 2005). Furthermore, when they match, we are very good at detecting the emotion another individual is trying to express. As a result, body expressions in conjunction with facial expressions will make it much easier for us to communicate an emotion, especially if “what the body is saying” matches “what the face is saying.” The famous quote, “they say a picture is worth a thousand words – and the same could be said for gestures,” accurately depicts how important body language, in particular body postures and gestures, is in understanding someone else’s emotions.