(Em)bodied emotional Interaction

In face to face communication it is not only what is being verbalized that carries valuable information. By looking at people's body language, listening to their voice or catching up on their vibes, it is possible to learn so much more of what they actually want to communicate. In digital media, people attempt to communicate as much as possible of these physical and personal experiences of conversations; phoning, sending SMS/MMS, taking photographs, using smilies or emoticons, but still there is little personal and physical experience of the dialogue. Keeping in touch with friends and family to maintain strong social and emotional ties is however as important as before, if not even more important in our busy and technologically driven lives.

In this project, we want to create systems that use both physical and cognitive modalities so that users get strong experiences of the underlying and more physical signs that occur in real life communication. However, we do not believe that these experiences need to be and also could not be exactly the same as in real life. Instead we argue that the digital setting instead has other modalities to offer.

We have for a number of years explored the "affective loop" where we involve both body and mind to express emotions in interaction with and through mobile services/artefacts/media, sometimes through tangible interaction devices. A first attempt was done in collaboration with Ana Paiva and her group in Portugal. We created a doll, SenToy, through which players of a computer game could control their avatar. By gesturing with the doll in ways reminding of different emotional states, the avatar changed emotional state and acted accordingly in the game. This design experience taught us that it was possible to get users involved not only intellectually but in a strong, bodily sense. We used that insight in the eMoto project where users' gestures with the stylus pen that comes with certain mobile phones in order to express their emotions in a texting system. Their gestures render different colourful, animated backgrounds to their SMS, aimed to communicate what emotions they want to communicate. Finally, in collaboration with Microsoft, we created the system Affective Diary, where we pick up on bodily states through sensors worn in a bracelet on users' upper arm. These are then reflected back to users together with a range of materials from their mobile phone (SMSs sent and received, photos taken, Bluetooth presence, etc.). Users can scribble on top of them to create a diary of their past and reflect on their bodily experiences.

In all of these systems, we try to address the physical aspects of emotional experiences in interactions with different mobiles services. In SenToy the domain was a game, in eMoto communication between friends, and Affective Diary, self-reflection. We now want to move on to emotional physical interactions between groups of users. Our experience of the world depends on our human bodies, not only in a strict physical, biological sense, through our experiential body, but also through our cultural bodies. We learn how to behave, appropriate emotional expressions in different contexts, and emotions are shaped and defined by the culture we live in. Certain emotions are only talked about and in a sense only exist in certain cultures. Thus, exploring the social contract for how emotions are initiated, expressed, perceived and replied to, can provide us with a key to how to build for embodied emotional experiences of others in social context.

In this project, we want to combine our previous work with tangible, mobile, emotional interaction with our previous work on social interaction. In particular, we are interested in group behaviours, cf the "Ghost in the Cave"-system, where the collective behaviour of the audience controls an avatar in a game. Our physical bodies are potential arenas for emotional interaction. We have not yet decided on a domain of use as this will be negotiated together with the industrial partners in this project.

We believe that this strand of work that we have performed for several years, and will continue to work on, provides input to not only how to design for emotional interaction, but also to questions of how to make use of our bodies in interacting with mobile devices and ubicomp environments. As mobile interaction is done on such small interfaces with small input possibilities, other modalities need to be further explored. This project will therefore feed into the theme project on "interaction models".

Questions: Research questions for the project lies first and foremost in the design area: can we design for embodied emotional interactions touching both our physical and cognitive experiences?

Technically, we need to realise the interaction through sensors on our bodies or inside the devices we carry, which not only entails technical issues, but also design issues to do with privacy, ease of use, and intrusiveness into our everyday behaviours and social spheres.

Method: The research method applied is that of a design- and technical probe that is implemented to such a level that it can be tested with end-users. End-users are involved in all stages of the design process, ranging from a cultural probe used to make end-users document their lives and thereby provide us with design feedback, via participatory design methods, to finally, end-user evaluation of the final system. Ethnographic studies will be used to lead the investigation to the most interesting artefacts

Results: Expected results are first of all a more or less fully functioning mobile service where users interact physically with the service. Second, it will result in a set of insights on how to design for affective interaction in these kinds of applications such as how to make use of sensors in this context, and a furthered understanding of how users react to physical interactions in mobile contexts.