Multiple Levels of Interaction, Multiple Levels of Performance Measure

The evaluation of such a system was quite complex because of installation problems within the museum's infrastructure, specific features of the system, and its particular context of use. For these reasons and in order to address the multi-faceted components of the HIPS system, we decided to adopt a multidimensional approach based on storytelling and scenarios. Specifically, we assessed user performance on four levels: phenomenological, cognitive, emotive and sociocultural.

At the phenomenological level the performance measure concerns included:

• Users' perception of the adaptation to the visiting style (personalization of the information, pauses, pace of narration) and the physical movement as a primary means for accessing information.
• Auditory comments (deictics, pronouns, etc) effectiveness in supporting the users' orientation and recognition of artworks.
• Tool flexibility (skill of personalizing and contextualizing the information according to the users' changes of path or visiting style).
• At the cognitive level the performance measure concerns focused on:
• 
• The cognitive effort associated with the use of the tool and the comprehension of the contents.
• Users'conceptual model.

At this level, scenarios were used to question the design of the system. We used Norman's cycle of cognition based on: goals, intentions, planning, execution, perception and evaluation to generate questions such as "How does the artefact evoke goals in the user?" or "How does the artefact make it easy or difficult to carry out the activity?" or "How does the artefact support the user when a shift in the goal occurs?".

At the emotive level the performance measure mainly concerned aspects of experiential cognition:

• Observation of frustration.
• Observation of confusion.
• Expression of satisfaction.
• Expression of engagement. At the socio-cultural level the performance measure concerns included:
• The social aspects of group activity mediated by the system (communication, knowledge sharing, collective memories).
• Appraisal /dislike of contents.
• Impact of narrative styles (male/female voices, accents, music, reading styles).

Even if the narrative approach impacted the evaluation at the four above mentioned levels, it is worth clarifying that the stories mostly provided the structuring framework for reasoning on the system and externalizing implicit knowledge. For the evaluation of more detailed features of the system, from interaction design to contents, we complemented storytelling with other methodologies, including ethnographic observations of the activity and laboratory testing (heuristic evaluation, cognitive walkthrough, scenario-based evaluation on intermediate prototypes.