We needed four staff people to perform the test:
- a technician who took care of the correct system functioningand the data log;
- an observer who took notes using an observation grid and recorded user comments on an audio tape;
- an interviewer who recruited the visitors, introduced the system, supported during the test and facilitated during the storytelling;
- a video operator who video recorded the test. Next, we organized the test procedure into these steps:
- recruiting visitors at the entrance of the museum, who volunteered to try the system;
- conducting pre-test interview to collect data about the users' characteristics;
- describing the test procedure;
- carrying out a test session in the form of a free exploration and re-creating, when possible, pre-defined scenarios (to induce the users to try a specific functionality the interface, to try the system under particular circumstances or constraints);
- storytelling.
User Sampling
The subjects who took part in the evaluation were recruited in the museum on the day of the test. They were chosen at random from a group of museum visitors who had already volunteered to try the system and participate in a debriefing after the test. All subjects were English or English-speaking tourists and were picked to span different characteristics:
- Age: 15-18; 19-25,25-30,31-40,41-50,51-60,61 and over.
- Nationality
- Average/very good knowledge of English.
- Gender: finales and females.
- Poor/average/very good knowledge of art.
- Previous knowledge of Museo Civico: yes/no; in case, how many times the user visited the museum, and how long ago.
Test Session
The evaluation at the four levels was carried out on the basis of direct observation of user activity (video recorded) during the free exploration and scenarios execution. We used scenarios to create a context for the activity, and we artificially provoked users to evaluate the system under specific conditions or exceptional constraints. Such conditions included:
- Time slots for the visit (near the closing time).
- Museum crowd.
- u Distribution of physical affordances in the environment (chairs, light conditions, location of signs, presence of maps).
- Single /group visit.
- Use of other artefacts (book guide, audio guide, paper maps, etc.). The test provided us with a variety of observed data including:
- Time spent in front of each exhibit.
- Time spent in the museum.
- Observation of frustration, confusion, satisfaction, engagement.
- Conformity with a pre-defined visiting style (paths defined on the basis of explicit interactions with the PDA).
- Possible changes in the visiting style.
- Influence of physical affordances (e.g. objects, light, people in front of a specific exhibit).
Further data came from the system log of the interaction with the PDA interface: list of descriptions heard; typology of descriptions heard; number and type of commands used.