Abstract

The entertainment industry is going through a transformation, and technology development is affecting how we can enjoy and interact with the entertainment media content in new ways. In our work, we explore how to enable interaction with content in the context of 3D cinemas by means of a mobile phone. Hence, viewers can use their personal devices to retrieve, for example, information on the artist of the soundtrack currently playing or a discount coupon on the watch the main actor is wearing. We are particularly interested in the user experience of the interactive 3D cinema concept, and how different interactive elements and interaction techniques are perceived. We report on the development of a prototype application utilizing smart phones and on an evaluation in a cinema context with 20 participants. Results emphasize that designing for interactive cinema experiences should drive for holistic and positive user experiences. Interactive content should be tied together with the actual video content, but integrated into contexts where it does not conflict with the immersive experience with the movie.

J. Häkkilä, M. Post, S. Schneegass, F. Alt, K. Gültekin, and A. Schmidt, “Let me catch this! Experiencing Interactive 3D Cinema through Collecting Content with a Mobile Phone,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2014. [PDF]

Introduction

During the past few years, new technologies and services have emerged around the entertainment industry, including the rise of 3D movies and an ever-increasing number of movie-related extra content available online. In the era of the mobile Internet, mobile apps, and social media services, extra content related to the movie can be searched, accessed, and discussed easily, virtually anywhere. In this paper, we look beyond the conventional experience of watching a movie and explore the concept of interactive 3D cinemas.

Capturing and assessing interaction around TV and cinema content has so far been possible through Twitter, Facebook, or tailored mobile apps [25]. These interactions are particularly valuable for several reasons. On one hand, they could be used to implicitly identify interesting parts of the content, for example, those that trigger most discussion. This may help producers to enhance the story of future episodes or also provide an automatically created summary, for example, of a sports game. On the other hand, this is interesting for merchandising. While the merchandising industry currently mainly targets the after-movie experience through selling products to fan communities, our approach can complement the instant experience and trigger sales already during the movies.

At the same time, with the rise of mobile apps and social media, we see 3D content becoming more popular – both in cinemas and at home. This allows more immersive experiences to be created by having an additional dimension for information placement. Today, this dimension is used to create special effects and, thus, intensify the user’s experience.

The use case for our work is an interactive movie that could be shown in a cinema. In this movie users can collect different items by catching them through mobile phone based interaction techniques. The main focus of our work is on understanding the UX aspects of interactive 3D cinema. Particularly, we chart what kind of user experiences and preferences different interaction and visualization methods provoke. In order to do so, we compare different concepts, namely

 catching interactive elements in a competitive vs. informative condition,

 touch vs. shake gestures on mobile phones, and

 different types of interactivity cues.

The novelty of our work lies in presenting the first study on interactive S3D cinema where users interact with their mobile phones to collect content and in conducting the study in a real cinema. While our study is artificial in that we used customized, short movie clips, we made the experiment as realistic as possible by conducting it in an authentic environment. We assessed our research questions through logging, questionnaires, and interviews.

The contribution of this work is twofold. First, we investigate the requirements for interaction with 3D content, including means to communicate the interactivity of particular screen elements and suitable interaction techniques to retrieve the content. Second, we evaluate the influence on UX. Our work provides information for future researchers and practitioners who want to create interactive cinema experiences. Particularly, we (a) identify preferred input modalities, (b) show design solutions for presenting the interactive content, and (c) report on user perceptions and lessons learnt on the concept of collecting items in an informative and competitive manner.

Interactive Cinema

Our concept for interactive 3D cinema is focused on showing interactive 3D content on the cinema screen. The interactive objects are integrated into the commercials and the movie clips, and are shown for a limited period of time. The users can then catch these interactive objects with their mobile phones by performing a gesture, and the collected items are stored on the mobile phone application – either for the purpose of an interactive game or for later use. We explore two different concepts:

 Competition, where users compete against each other in trying to catch as many interactive objects as possible, integrated with a commercial video clip.

 Collecting informative content items, where each user can collect any interesting interactive items without any competition, integrated in a movie clip.

We explore two different interaction techniques for mobile phone interaction for catching the interactive objects – tapping the touch screen and gesture input (shake). The technical set-up and mobile phone application are explained in more detail in the following sections.

Technical Prototype

We developed a prototypical system using a client-server architecture

Server

The server consists of a game engine (Panda3D4) that is responsible for displaying the movie as well as rendering the 3D objects in real time and a component that handles client interactions over HTTP. The client mobile application (see next section) requests the state of the clip currently being run and communicates back any user interaction (tap or shake). Success of the interaction, i.e., whether a user managed to catch an item, is then calculated on the server side and the result responded back to the client software.

Since client and server are in the same local Wi-Fi network, the use of HTTP does not introduce a significant latency and the interaction appears to be almost instantaneous. To be able to handle numerous Wi-Fi connections within the same room without causing delays, two commodity wireless access points are configured on the same network. Operating two distinct channels allows for distributing clients evenly between the access points and, thus, balance the load.

Mobile Application

A mobile app enables users to interact with 3D content by retrieving content items as they appear on the screen. Retrieval of content can be triggered (a) by tapping the phone screen, or (b) by performing a catching gesture with the phone.

For the client software, we use an Android app that communicates with the communication server to get the state of the clip running and indicating an interaction (tap or shake). For the tap interaction, we used the entire screen of the phone to provide an easy and eyes-free interaction. For the shake interaction we used the accelerometer data and simply detect shake gestures by defining a certain threshold (i.e., more than 25 m=s2 acceleration). Success of the interaction is calculated on the server and responded back to the client. The phone then displays the caught item for 1 second and vibrates for 2 seconds to provide feedback to the users.

The mobile app currently supports two different modes: a competitive mode and an informative mode. In the competitive mode, content items can be collected. This mode is mainly meant for games, for example, in the form of commercials, where the audience can compete for an incentive (e.g., an ice cream) by collecting as many items as possible. For each collected item a symbol is shown on the screen. Depending on the game setting (server), only the first n users to perform the interaction as content items appear will collect the item. In the informative mode, people can retrieve metainformation. This mode is means to be used during motion pictures to enable viewers to retrieve information on actors, the name of the song currently playing, or a screenshot of the current scene as a background image. In this mode, each user performing the interaction will receive the information. The UI when collecting an interactive item is shown in Figure 4.

The Movies

Interactive Items

During the evaluation we were particularly interested in how interactivity of an item within 3D clips could be communicated to the user, and how users would perform using the mobile phone interaction techniques, tapping and gesture. We use three interactivity cues: 1) glow, 2) particles flying from the object, and 3) no additional visual cues but depth. The interactivity cues are presented in Figure 5.

The depth only cue was included as a baseline condition, as we were first interested whether using depth only would be perceived as a sufficient cue to indicate interactivity. Hence, in this condition, we simply let interactive items float towards the user. In prior art, glow has been found to be a visual effect that is perceived particularly well when it comes to indicating interactive items in 3D virtual world user interfaces [19]. Finally, small particles emitted from the object are selected as the third cue, as this visualization has been earlier used in computer games. Note, that glow and particles are used in combination with depth. In the commercial clip, the we also add a movement of the object towards the viewer.

Only one interactive item appears at a time on the cinema screen with one cue applied. The overall number of interactive items was 30 per clip for the competition case (10 for each visualization technique) and 30 per clip for the informative content item collection. The fraction of the screen space taken by an appearing collectable item is approximately 2% in both cases. Each item is shown for approximately 8 seconds – although in practice the time was shorter in the competitive mode, since catching the item causes it to disappear from the movie screen.

We prepared different pieces of interactive 3D content. A professional designer created all content in Blender.

Commercial

For the competition concept, we use a commercial video clip (appr. 1 min), which was edited to be approximately 6 minutes long by repeating it over. During the commercial, we show altogether 30 interactive items (soft drink cans) that users can catch with their mobile phone. The items utilize three different visualization techniques (10 items with each interactivity cue), and their order is randomized. All items are shown as moving towards the audience, and they become interactive approximately two seconds after their appearance on the screen. When the first user catches the item, it disappears from the screen and is shown on the mobile phone of the “winner”. As we have two interaction techniques, tapping and gesture, we created two randomized clips.

Movie

For the informative content collection concept, we chose the “Big Buck Bunny” movie, an open movie created with Blender. We augmented the movie with certain items, which, if caught by the viewer, would transfer meta-content of the movie (e.g., information on the characters, a soundtrack, background images) to the mobile phone of the user. Altogether, 30 interactive content items appear during the 10- minute movie clip. We created two versions of the interactive movie – one with glow and sparkle effects (no movement), and the other one with interactive items that move towards the audience. This decision to include less content items compared to the commercial was made to leave more space for the immersive experience with the movie, as we found approaching items to be quite dominating when testing.

Discussion

Experiencing Interactive 3D Cinema

Based on our study, the interactive cinema concept, where people interact to collect items into their personal repository, does provide an interesting potential for the future development. In general, the competition concept is perceived as rewarding and playful, whereas collecting informative items is perceived as useful and personal. The interest and positive comments left us with an encouraging overall perception about the concept of interactive 3D cinema. However, there are a number of findings that should be carefully considered before designing such an application or service.

As movie experience is very much content-driven, it is crucial to avoid breaking the immersion or disturbing the movie. Hence, participants want interactive items to be integrated contextually with the actual media content shown on the theater screen. Additionally, extra content should appear close to the objects (e.g., actor) they relate to. An ideal solution to combine these two design requirements would be to use the interactive items in places where they do not interfere with the movie experience – commercials, trailers, or end scripts.

In terms of Forlizzi‘s UX framework [7], our concept of interactive cinema provides means to shift the balance from cognitive towards expressive user experience. Whereas the cinema experience is still essentially linked to the action and duration of watching the movie in the theater, i.e., cognitive interactions, there are already elements which extend the experience before and after the theater, such as trailers, advertisements and discussion forums. The approach of collecting and retrieving items and extra content, adds to the existing means and fosters a more personal relationship with the product (i.e., the movie), thus forming an expressive user experience. Roto highlights, how user experience extends also to the time after use [24], and collecting movie related items and take them away, either as information or as tokens, has the potential to further prolong this connection. In addition, the collective information about the audience can offer interesting reflections. For instance, a ranking list of the most commonly collected information items can support the perception of a shared experience. Interestingly, interactions themselves are not regarded as socially unacceptable, which was somewhat surprising given prior work on interaction with public displays [23]. This may be partly due the context, as people are sitting in a dark room, where the visibility towards other viewers is limited. Also, it may have been the case that our participants were immersed with the content of the film and, hence, paid less attention to their surroundings.

Design Considerations

Our study revealed several findings regarding designing for a competition in the interactive cinema. The feedback on collecting the items should be very clear, and it should preferably be designed to create feelings of excitement, delight, and playfulness. To achieve this, one potential option is to use some special effects in the interactive items. In our study, the glow and sparkle effects were well received, but additional effects could be added, e.g., on how the interactive item appear or disappear on the screen. Here, the graphics and animation design could also be matched to reflect the movie genre. Also, in the competition, participants want to know more about their fellow competitors, not just their own performance. The opportunity to know something about the rest of the audience, for example, winning scores shown on the large cinema screen, would create a common experience and the feeling of being together in the common place.

Another important thing to consider relates to the overall user experience with movies. People often go to the cinema to relax and feel good, or to immerse themselves with emotional or exciting scenes, and it is important to design for such an experience. In the context of competition, this means that the interaction should provide positive experiences to avoid situations, as in our study, where some participants felt very disappointed about not catching interactive items even though they thought they were doing well. A possible solution to this is to reward not just one person but for example the first 30% of the users that catch an item, or provide them with other inspiring or positively surprising UI feedback. In the middle of a (supposedly) relaxing and refreshing situation (the cinema), the user should be able to achieve positive experiences, and not feel ignored or disappointed.

From a design perspective, presenting interactive items in a sequence, i.e., not several items simultaneously, was a good solution, as it did not require switching attention from the movie screen to the mobile phone – tapping anywhere on the touch screen would suffice.

Conclusion

In this paper we have investigated the concept of interaction with 3D cinema content. We explored collaborative as well as competitive forms of interaction. Our results show that people engage themselves with both the content and extra material related to the actual movie. We envision the popularity of such applications to further increase as means are provided for the viewer to retrieve information on the main actor or buy the current soundtrack by just one click during watching the movie or while they play a game prior to the movie.

The finding that interaction was sometimes perceived to be distracting suggests that interaction should be realized in a simple and unobtrusive way that does not negatively influence the cinema experience. For example, retrieved content could be stored for later use and then be brought to the attention of the user as the movie is over. Our findings indicate also that the concept of interactive cinema provides means to enhance both individual and communal experiences in the cinema. Whereas collecting items to your personal devices makes the experience more personal and lasting, the design of interactive games can be used to create the perception of a shared experience. Our findings also emphasize that designers of interactive cinema concepts should pay particular attention to create positive experiences and reward the users, as feeling good or exited is an inseparable part of the overall movie experience and motivation to attend the theater.

Related Publications

J. Häkkilä, M. Post, S. Schneegass, F. Alt, K. Gültekin, and A. Schmidt, “Let me catch this! Experiencing Interactive 3D Cinema through Collecting Content with a Mobile Phone,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2014. [PDF]