IntraFace software able to read emotion

When I stepped into the office of Fernando De La Torre, an associate research professor in the Robotics Institute at Carnegie Mellon, the first thing he did was pull me out of the room. He brought me one flight down from his office and into the lobby on the main floor of Smith Hall.

Inside the lobby was a machine with a television screen that was mapping my movements and displaying my state of emotion based on my appearance. When I smiled, it said “Happy, 98 percent;” when I stopped, it said “Neutral, 56 percent;” when I remembered how much homework I had this past weekend, it said “Sad.”

This machine was not just guessing my emotions, but was algorithmically calculating them based on the appearance of my face to the screen. This machine is the counterpart to the one found in Newell-Simon Hall. Installed in both machines is a special app called IntraFace, created by De La Torre alongside his team. This application is the result of 10 years of facial imaging research done by De La Torre, and it is capable of reading facial images in real time and computing a corresponding emotion based off changes in one’s facial structure.

Generally, only humans and some animals are thought to understand emotions, but De La Torre and his team are adding robots to the list.

When first assessing IntraFace, one of the first questions De La Torre posed was “How can you make a computer understand human emotion?”

He answered himself by saying that the answer is twofold. The first part involves finding a way to have the computer track the user’s facial components so that it can determine their position in a vector space and read them. To do so, De La Torre developed software that allows the application to track facial muscles based on their conformation and location on the face. The process involved feeding the software thousands of photos of facial components and thousands of photos of non-facial components.

Based on those initial inputs, the machine could then process what is a facial component and what is not a facial component. To downsize the storage size of the software, De La Torre reduced the processing algorithm to just four matrix multiplications.

This downsize is well within the memory storage capacity of a phone or other computing device, allowing IntraFace to be accessible across a wide array of electronic media.The second part of the answer to De La Torre’s question, and the more important part, involves developing a way to connect facial features to emotions in an algorithmic way that a computer can understand. To accomplish this task, De La Torre and his research team fed the software thousands of photos with different facial structures and labeled them accordingly; smiling faces were labeled “Happy,” frowning faces were labeled “Sad.”

While expressions have far more psychologically and physiologically complex features, there does exist a set of universal expressions, such as happiness, sadness, and surprise, which look the same on everyone. This means that the same facial structures are used, and this universality allows researchers to use these broad categories.

This is an instance of classification, or the method of providing a machine an index (a list of facial expressions) from which it can develop a solution (stating whether the person is happy, sad, etc.). In classification, a machine is given thousands of different observations, each with an appropriate category. Using these observations, a machine can take an unknown observation and file it into a pre-existing category based on the list of categories it already contains. In the case of IntraFace, the observations are facial structures and the categories are emotions.

After elaborating on the technicalities of the software, De La Torre described some of the endless applications. For example, companies could use IntraFace in market advertising to assess the quality of their advertisements; they could see what parts of the advertisement make people laugh, what parts bore them, and what parts completely disinterest them.

Additionally, De La Torre mentioned integrating this software into vehicles to detect driver distraction. If vehicles had this software within them, cars could detect when a driver is distracted or fatigued or texting based on the expression on his face. Another could be for public speaking, which would allow a lecturer to assess when his crowd is listening and engaged versus when it is tuning out or dozing off. De La Torre even mentioned the possible medical applications of IntraFace; the app could allow doctors and other clinicians to determine how patients are feeling medically and if they are experiencing any pain or discomfort.

In order to advance the medicinal applications of InstaFace, De La Torre is currently partnering with Jeffrey Cohn, an Adjunct Professor of Computer Science at the Robotics Institute at Carnegie Mellon and Professor of Psychology and Psychiatry at the University of Pittsburgh. Another feature of IntraFace is that it is indiscriminate: it can read faces of any skin color as well as faces with facial hair of any kind.

While the application has gained much ground over the past 10 years, De La Torre is hoping to develop it to a point where he can release it to researchers across the nation as a practical research tool. IntraFace could provide researchers in the field of psychology, physiology, and computer science with the tools to create breakthrough research.