Monday, February 18, 2008

Multidimensional Evaluation Based on Story Telling and Scenarios

The interest in narratives has a long tradition. Bruner (1990) considers narrative a primitive function of human psychology, lying at the heart of human thought. The representation of experience in narratives provides a frame which enables humans to interpret their experiences. In this way narrative is a fundamental aspect in the construction of meaning.

Although narratives have been studied in many areas of psychology, the idea of exploiting them for system design and evaluation is quite recent and still not consolidated (Erickson, 1995). It is based on the need to understand user requirements through the collection of implicit knowledge that a user may have gained through experience. In particular, the use of narratives for system evaluation does not aim to provide quantitative results but to structure a framework within with users may express knowledge otherwise difficult to verbalize Stories make it possible to study the complexity of the context (media, internal/external environment, actors); details about critical events that can be observed very seldom; personal involvement; emotional details; and the concreteness and veracity of personal experiences.

Internet 2010

This chapter applies the narrative approach to the evaluation of an electronic tourist guide. The evaluation was carried out in an Italian museum, the Museo Civico, in Siena, with real visitors recruited at the museum entrance. The tourist guide is a prototype system developed within a project called Hyper Interaction within Physical Space (HIPS),' funded by the European Commission within the I-Cube (Is) Programme. The system is very advanced since it exploits cutting edge technologies (positioning technology, dynamic language generation) and visionary interaction design concepts (access to the information space through the physical movement in the museum; user modelling for content adaptation).

The system guides visitors by generating audio messages: users can get instructions on how to find items of interest, hear descriptions with references to items seen earlier and to ones that will follow, or ask for additional information. Information is generated dynamically, adaptive, and integrated with maps and spatial directions. When interacting with users, the system integrates their requests with a customized user-model, the user's browsing history, and their physical location at the moment of the query, providing highly contextual and personalized information (Marti et al, in press). The information content varies according to the user's location, preferences, and to the information already given. From a hardware point of view the system is based on a client-server model; the clients that visitors carry around are pen-driven palmtop computers with a screen, headphones and no keyboard. Localization is performed by various means: infrared, radio and GPS (Global Positioning System). Connectivity to the server is wireless.

Multiple Levels of Interaction, Multiple Levels of Performance Measure

The evaluation of such a system was quite complex because of installation problems within the museum's infrastructure, specific features of the system, and its particular context of use. For these reasons and in order to address the multi-faceted components of the HIPS system, we decided to adopt a multidimensional approach based on storytelling and scenarios. Specifically, we assessed user performance on four levels: phenomenological, cognitive, emotive and sociocultural.

At the phenomenological level the performance measure concerns included:

  • Users' perception of the adaptation to the visiting style (personalization of the information, pauses, pace of narration) and the physical movement as a primary means for accessing information.
  • Auditory comments (deictics, pronouns, etc) effectiveness in supporting the users' orientation and recognition of artworks.
  • Tool flexibility (skill of personalizing and contextualizing the information according to the users' changes of path or visiting style).
  • At the cognitive level the performance measure concerns focused on:
  • Internet 2010
  • The cognitive effort associated with the use of the tool and the comprehension of the contents.
  • Users'conceptual model.

At this level, scenarios were used to question the design of the system. We used Norman's cycle of cognition based on: goals, intentions, planning, execution, perception and evaluation to generate questions such as "How does the artefact evoke goals in the user?" or "How does the artefact make it easy or difficult to carry out the activity?" or "How does the artefact support the user when a shift in the goal occurs?".

At the emotive level the performance measure mainly concerned aspects of experiential cognition:

  • Observation of frustration.
  • Observation of confusion.
  • Expression of satisfaction.
  • Expression of engagement. At the socio-cultural level the performance measure concerns included:
  • The social aspects of group activity mediated by the system (communication, knowledge sharing, collective memories).
  • Appraisal /dislike of contents.
  • Impact of narrative styles (male/female voices, accents, music, reading styles).

Even if the narrative approach impacted the evaluation at the four above mentioned levels, it is worth clarifying that the stories mostly provided the structuring framework for reasoning on the system and externalizing implicit knowledge. For the evaluation of more detailed features of the system, from interaction design to contents, we complemented storytelling with other methodologies, including ethnographic observations of the activity and laboratory testing (heuristic evaluation, cognitive walkthrough, scenario-based evaluation on intermediate prototypes.

Preparation of Test Material, Equipment and Test Procedure

We needed four staff people to perform the test:

  • a technician who took care of the correct system functioningand the data log;
  • an observer who took notes using an observation grid and recorded user comments on an audio tape;
  • an interviewer who recruited the visitors, introduced the system, supported during the test and facilitated during the storytelling;
  • a video operator who video recorded the test. Next, we organized the test procedure into these steps:
  • recruiting visitors at the entrance of the museum, who volunteered to try the system;
  • briefing users on the system and its basic functionality;
Internet 2010
  • conducting pre-test interview to collect data about the users' characteristics;
  • describing the test procedure;
  • carrying out a test session in the form of a free exploration and re-creating, when possible, pre-defined scenarios (to induce the users to try a specific functionality the interface, to try the system under particular circumstances or constraints);
  • storytelling.

User Sampling

The subjects who took part in the evaluation were recruited in the museum on the day of the test. They were chosen at random from a group of museum visitors who had already volunteered to try the system and participate in a debriefing after the test. All subjects were English or English-speaking tourists and were picked to span different characteristics:

  • Age: 15-18; 19-25,25-30,31-40,41-50,51-60,61 and over.
  • Nationality
  • Average/very good knowledge of English.
  • Gender: finales and females.
  • Poor/average/very good knowledge of art.
  • Previous knowledge of Museo Civico: yes/no; in case, how many times the user visited the museum, and how long ago.

Test Session

The evaluation at the four levels was carried out on the basis of direct observation of user activity (video recorded) during the free exploration and scenarios execution. We used scenarios to create a context for the activity, and we artificially provoked users to evaluate the system under specific conditions or exceptional constraints. Such conditions included:

  • Time slots for the visit (near the closing time).
  • Museum crowd.
  • u Distribution of physical affordances in the environment (chairs, light conditions, location of signs, presence of maps).
  • Single /group visit.
  • Use of other artefacts (book guide, audio guide, paper maps, etc.). The test provided us with a variety of observed data including:
  • Time spent in front of each exhibit.
  • Time spent in the museum.
  • Observation of frustration, confusion, satisfaction, engagement.
  • Conformity with a pre-defined visiting style (paths defined on the basis of explicit interactions with the PDA).
  • Possible changes in the visiting style.
  • Influence of physical affordances (e.g. objects, light, people in front of a specific exhibit).

Further data came from the system log of the interaction with the PDA interface: list of descriptions heard; typology of descriptions heard; number and type of commands used.

Saturday, February 16, 2008

Being Flexible

When given the choice between creating a perfect product that is never released and an imperfect project that benefits its customers, we must consistently choose the latter. There is a point at which our insistence on a complete usability process becomes counterproductive. The most successful usability engineers can quickly assess a situation and impose the parts of the process that make the most sense, given project constraints.

In Example A, the usability engineers were alarmed and discouraged by the direction the project was heading. The UCD team was inexperienced, and its visual designers and writers had all worked together on a previous product that was poorly designed and not well accepted by users. Access to users was severely limited, and there was no time to conduct formal usability studies. The usability engineers worried that their data was not good enough and that the users were not involved enough. They spent the first two weeks fighting with the project team and the next two weeks fighting the requirements. By the time the usability engineers were through fighting, their schedule had slipped by nearly four weeks. In Example B, the entire project timeline was six weeks. There was no time to fight, only to adapt. The challenge was figuring out how to provide value despite unchanging, imposed guidelines.

Internet 2010

In both examples, usability engineers realized that they could offer significant help in several areas. They did this by:

  • Clearly understanding the project goals, including scheduling constraints, user constraints, and project ownership.
  • Quickly establishing what work they could provide within the project constraints. If the usability engineers could not conduct full-blown designs sessions and usability tests, what elements of those things could they incorporate into their work? They chose to provide a series of quick heuristic evaluations, to research best practices for similar projects, to post questions to a usability listserv, to interview users under the guise of another project, and to interview subject matter experts when no users were available.
  • Being flexible enough to understand and work within the project constraints. Usability engineers researched best practices and provided immediate feedback whenever possible.
  • Resisting the temptation to suffer through meetings silently, consistently voicing concerns, gently but persistently reminding the rest of the team of their roles and skills.

Understanding that neither project would support the rigo of true user- centred design work, the usability engineers continued to guide team meetings in a more user-centred direction, representing users and incorporating usability concepts and techniques whenever possible. The usability engineers became extremely flexible in their schedules and work styles. In both examples, flexibility and customer focus yielded positive results. By working within the confines of their projects, the usability engineers were perceived as valuable team members, rather than as impractical or uncooperative team players.

Caught Between Real and Virtual Worlds

While agreement was breaking out at this workshop, so too were some concerns. As most HCI practitioners would agree the distinction between design and evaluation in HCI is often blurred, indeed design and evaluation have been described as the different sides of the same coin. The design of the DISCOVER system proved to be no different. As we elicited requirements and undertook early design we kept an eye on evaluation, ultimately as a sanity check. The use of the prototype, described above, had given us clear usability requirements. The affinity diagram had given us detailed requirements of the design, content and substance of the DISCOVER collaborative virtual environment but concerns as to the issue of evaluation (and with it validation) began to emerge.

Internet 2010

Design Tensions: No Magic, Thanks

For many collaborative tasks in virtual environments, the goal is to achieve a particular state of affairs within the virtual world, for example agreeing the layout of an office, as in Hindmarsh et al. (1998). For others, the activity within the CVE is part of a larger collaborative process, but the way collaboration works within the CVE need not exactly replicate real world interaction. In contrast, the DISCOVER environment must support the acquisition of real world skills. In short, skills acquired in the DISCOVER CVE must be transferable to the real world of the ship. This presents a significant constraint: ideally, interaction and collaboration must not be artificially harder or take longer than in the real world, but neither must they be artificially easier or executed faster. Thus the design of DISCOVER should treat with caution "magical" devices such as birds' eye views of the state of environment, Star Trek-like transporting, or visible rubber banding between an avatar and its current focus of attention.

Accordingly, much research which has addressed the problems of ensuring that the users of such environments are aware of their surroundings and of other users cannot be employed directly. Consider, for example, the problems with recognizing other avatars. To maintain realism, avatars cannot be simply labelled. There may also be the presence of dense smoke, and perhaps the need for the avatars to wear vision obscuring breathing apparatus. All of this means that recognizing one's colleagues (as avatars), mediated in the real situation by such characteristics as gait, stance and minor variations in standard issue clothing, becomes far more difficult.

Thus there is a fundamental tension between exploiting the technology to the full to produce a state-of-the-art virtual training environment and creating one which is faithful to the behaviour and constraints of a real ship. This, of course, is not merely a design issue but also presents a corresponding evaluation challenge.

Thursday, February 14, 2008

Building the "Right" Project Team

The best usability engineer for a project is one who is intelligent and flexible, and one who is not controlled by ego. The usability engineer should have excellent negotiation skills, and should be able to present a compelling case for the work.

The right project team includes members with a variety of roles and hard and soft skills. Equally important, the right project team includes at least one firm supporter of usability. This person should be in a leadership position, and should be firmly convinced of the benefits of usability engineering.

In Example A, the usability engineers never became fully integrated with the project team. There were several reasons for this. First, the management team did not include a firm supporter of usability. Although they included usability in the project plan, they did not implement or support making changes to the system or the project schedule based on usability issues. In addition, the usability engineers on the project tended to be timid, and often approached their work complaining about what they could not do, instead of working to figure out how they could provide value to the project. The right project team in this case would have been one with stronger management support and more persistent, positive-thinking usability engineers.

Internet 2010

In Example B, the intranet project team was a combination of two independent, already established teams: the consultants and the clients. The majority of the consulting group was made up of new consultants, many of whom did not have a clear understanding of how to integrate the usability engineer role within their own team. The clients had very few people to dedicate to the project, and had no flexibility with resources if the team failed to gel.

Still, what made this the "right" project team was the consistent, open sharing of information. Despite its shortcomings, the project team worked well together by educating, communicating, and negotiating. Team members listened to each other and shared what they knew about the project. Open communication led to trust within the team. As miscommunications, mis-perceptions and other mistakes occurred, the team talked through them, working towards resolutions without egos getting in the way. This assured the highest level of success on the project, and created a basis for ongoing understanding.

In our next projects for the company in Example A, we have had enormous success by allowing project leaders to interview potential usability engineering candidates. This gives the project leaders a strong sense of resource ownership, and eliminates the resentment they can feel when resources are imposed on their projects without their approval. An initial interview ensures immediate buy-in, and in these situations, the usability engineer starts out as a more integral and integrated team member, laying the groundwork for success.

Develop a Highly Usable VR System - and Do It Yesterday!" Continue...

Phase 2: Hello, I'm Your Avatar

A prototype system was created and demonstrated to trainers and prospective trainees at all three maritime training sites. In use each trainee begins by adopting a role - captain, chief engineer, first mate and so forth. They then put on a lightweight set of combined headphones and microphone to afford communication with other avatars and the trainer. Once a minimum of two trainees have entered the system, the trainer, who has been overseeing this process, starts the training scenario (Fig. 6.1). Avatars are able to communicate with one another by way of (virtual) radio and (virtual) telephone and are able to walk and run, open doors and pick up and operate fire extinguishers.

The trainer (or trainers) have all these facilities but have the God-like powers of being able to watch all trainees simultaneously, set fires and so forth as can be seen in Fig. 3.2. The training itself consists of working through a detailed scenario, usually drawn from a real world incident, which requires the trainees to role play and collectively deal with the problems which arise during the course of playing it out. At the end of the training, the trainer will debrief those involved and may use the play-back facilities of the DISCOVER CVE to illustrate particular points.

Internet 2010

Demonstrations of the software and simple hands-on tasks were followed by interviews and questionnaires. Our intention was to give potential end users a clearer impression of what a CVE is, how it could be used, to elicit feedback which could be used to refine requirements. We had also evidently managed to convince our developers of the importance of usability issues, since they were particularly keen to have any problems of this sort identified. However, our intentions were confounded by the very prominence which usability assumed in these initial trials. Eager for early feedback, and understandably reluctant to undertake development which might be misdirected, our technologists delivered a prototype just as soon as the software could be run independently of its development environment. This meant that although a reasonable impression could be gained of the functionality which could be offered, user interaction was in a very immature state. As a consequence, finer grain usability issues, for example the ability to identify the focus of an avatar's gaze, were obscured by larger difficulties such as moving through the environment. Equally users could not be induced to speculate in depth about how the system might be used, or how training delivered through such a medium might relate to existing practice. For example, it is difficult to convince the captain of one of the world's largest passenger vessels of the possibilities of the new medium after difficulty with movement control has "trapped" him for some minutes in a corner of the virtual bridge. However, some indications did emerge of the type of usage envisaged in each training context, and much debate was triggered about the detailed design features which would be necessary to support these.

The developers now urgently required a unified, detailed, concrete design specification which would nonetheless support each intended context of use. This was to be achieved by means of a workshop involving each of the three maritime organizations. (There was only one offshore organization in the project, whose requirements were relatively unified and straightforward.)

Phase 3: Virtual Reality Meets Paper and Pen

At this two-day event, trainers from the three organizations met with the representatives from one of the software developer organizations and two facilitators from the requirements team.

The agenda was very simple: to agree the detailed functionality of the maritime simulator and how it was expected to be used. We adapted elements of Contextual Design (Holtzblatt and Beyer, 1998) to facilitate these processes, principally a variant of the affinity diagram technique, which supports the identification of common themes from a mass of contextual data. We began by asking each training organization to revisit what they wanted of DISCOVER in terms of the "w" words (familiar to user-centred design practitioners), i.e. why, when, who, where and of course, how. As each organization described their needs we recorded each issue or explicit requirement on a Post-IT® note. At the end of the process we had gathered over 400 Post-ITs of which approximately 10 per cent were subsequently discarded as duplicates or irrelevant on closer inspection.

The trainers were then invited to create an affinity model which required them to sort the requirements/issues into logical groups: emerging groupings included the layout and configuration of the virtual ship, the appearance and functionality of the avatars and the context of use of the completed system. Throughout this process the software designer helped ground the requirements in reality. Informal discussion with the three trainers afterwards revealed that they thought that the day had gone well. The use of the Post-Its and their grouping was a familiar technique from other contexts, and had fostered engagement and apparent consensus. At the end of the first day we had succeeded in co-constructing an affinity model and subsequently a communications model and an artefact/physical model (ibid.) of the environment to be created.

Develop a Highly Usable VR System - and Do It Yesterday!"

The project had identified a number of key areas which needed specialist attention. These were to:

  • Produce a definitive statement of requirements on the DISCOVER system from the training organization. This involved understanding the methods used in current training, which ranged from state-of-the-art physical simulators costing tens of millions of pounds, to simple role playing, and their methods of assessment.
  • Adopt a suitable pedagogic model for the training environment.
  • Understand how we were to validate the DISCOVER system with the appropriate validating bodies. The project had ambitions to sell the system across the world to a number of shipping and oil companies and thus required a suitable seal of approval.
  • Produce for the software designers a detailed description of the virtual environments to be modelled and a list of the objects and their behaviours therein.
  • Specify a collaborative virtual environment with a strong emphasis on fidelity and usability.

Match the system to current and proposed training scenarios. And we had three months to complete this in the first instance!

Internet 2010

Phase 1: Just What Should a VirtualTraining System Do?

Some requirements work had been carried out before our entry to the project, but this had simply taken the form of the collection of high level statements of intent from the training companies and lists of competencies (or skills) to be trained. This provided our first political advantage: working closely with managers and training personnel in the user companies in itself convinced these project partners that their needs were being seriously considered. Thus our activities tended to be viewed in a constructive and receptive way. We will not provide a detailed description of the techniques we used, since they are familiar ones, but we employed a combination of interviews with trainees, trainers, their managers, observation and video- recording of training sessions and the collection of relevant documentation. All this took place at user sites. Again, goodwill was generated simply by the amount of effort very clearly being devoted to the user-centred work.

The results of this fieldwork fed into a large group of requirements relating to the way in which CVE based training could best be used in its particular intended context. Fortunately, although CVE design is a young field, there isan emerging body of literature which addresses generic usability issues for single users (e.g. Stanney et al., 1998; Sutcliffe et al., 2000) and in the collaborative context, the guidelines provided by the COVEN project (Normand et al., 1999). Thus we were able to supplement our own primary findings with secondary material from the literature and produce a large body of user-related requirements material in a relatively short time.

The Use of the MoSCoW Method

At this stage we had a large set of requirements in clear need of prioritization. In keeping with the user-centred philosophy which we had started to foster, this would be done in conjunction with the user companies. The MoSCoW technique from the Dynamic Systems Development Method - DSDM, (Stapleton, 1997) afforded a means of doing this which would be readily understandable to users and familiar to the developer partners in the project. DSDM prioritizes requirements using the MoSCoW rules.

The uppercase letters of the acronym (the o's are only there to make for a memorable acronym) stand for: Must have for requirements that are fundamental o the system. Without them the system will be unworkable and useless. The Must have category defines the minimum usable subset. Should have for important requirements for which there is a work-around in the short term but the system will be useful and usable without them. Could have for requirements that can more easily be left out of the current development. Want to have but won't have this time round for those valuable requirements that can wait till later development takes place. A small sample of the prioritised requirements follows below A substantial subset of the higher priority lists related to usability and other non-functional concerns.

  • lMust: The ability for trainers to increase stress in a safe, controlled manner for the trainees.
  • Should: Provide a more satisfying professional role for trainers than classroom teaching.
  • Could:Courses formally validated by the appropriate standards bodies.
  • Want:Measures of trainingtransfer.

By now, therefore, we had an substantial body of organized and prioritized requirements which formed the main body of the requirements and evaluation deliverables to the funding body. These reports passed their review, user-related concerns were accepted as the main force driving the design of the software and the project moved forward.

Wednesday, February 13, 2008

Post-Test Session: Storytelling

At the end of each session, the visitors were involved in a debriefing (in one of the rooms of the museum) where storytelling was encouraged to further comment, analyse, and interpret events which occurred during the test. The subjects were asked to describe their experience looking at the video recording of the test. This created a common base of discussion and knowledge, and provided concrete data to express impressions and points of view. To facilitate the storytelling, we prepared a set of questions to stimulate the discussion at the four levels of evaluation. These questions spanned over the following dimensions:

  • Global experience
  • Satisfaction /engagement.
  • Contents.
  • Internet 2010
  • Design concepts.

As a general outcome of the evaluation, we can say that stories mostly addressed aspects related to the emotional and socio-cultural levels. Indeed the system created a rich sensory environment that the users perceived but were not able to describe in a more structured form than stories. Stories, being concrete and immediate, do not require abstraction or introspection, so the users were free to tell and to compare their previous experiences, re-creating a context to share with the facilitators. That is why we used storytelling as an expressive means to stimulate communication between people who were not familiar with each other and to encourage them to speak about personal feelings, experiences and impressions.

We were aware that this kind of evaluation cannot provide quantitative data about satisfaction and engagement (such as that obtained using the Differential Emotions Scale, the Semantic Differential Scale or the free labelling method (Kim and Moon, 1998)). However, since the system was oriented to entertainment and leisure, we decided to collect data that could give insights into the capability of the system to intrigue and attract the users. During the debriefing we collected a corpus of about 20 stories that were mapped on user requirements and system specifications. Here is one of those stories:

When we went to Avignone, to visit the Palais des Papes, we had a local guide, a teacher of a school party who explained in detail the artistic and historical features of every room. We were interested in her explanation, but the students (children of the primary school) got suddenly tired. During the visit we passed through a room, where there was a different kind of exhibition where strange and funny animals dangled from the ceiling. Pupils were very curious to know about them, but the teacher was prepared only on the Palais des Papes, so she passed through the room without paying any attention to the animals of the exhibition.

The story contains a number of relevant elements for the design and evaluation. It highlights that visitors have heterogeneous needs: most of the time their activity is "non-goal oriented" since they can be pushed just by curiosity or pleasure, and their behaviour is not predictable. They often do not know ahead of time, or with any specificity, what future state they desire to bring about. Therefore, the situations of use can be various and idiosyncratic, leading the visitors to frequently adjust their goals and objectives during the visit.

Taking these elements into account, we can infer that the HIPS tour guide and systems like it must learn to adapt to visitor inclinations as they arise. The solution designed into the system, which proved to be particularly appropriate in this respect, focused on establishing a closer link between environment and user interactivity through the introduction of adaptive mechanisms.

Defining and Designing "User Experience"

"What seems to be missing is a clear idea about what experience is; what its components or elements are; and, perhaps more importantly, whether it even can be designed or scripted," said Jody Forlizzi at a Usability Professionals' Association conference workshop on experience-based design (EBD) a couple of years ago.

XMod's view is that experience is both definable and designable. In the Fall 1998 issue of The Journal of Design Management an article appeared on experience-based design written by another En Vivo co-founder. EBD involves analysing everyday experience, and making the results useful to design stakeholders. It calls for creating an experience "framework," using ethnographic techniques to study what people "think, do, and use," gain insight into consumer experiences and identify opportunities for new and better ones.

Internet 2010

The author is not aware of any ISO standard defining "user experience"; so what is it? XMod's XVP says: "We use 'experience' to refer to the ways in which people develop habits, assumptions, and routines in a particular domain. We do not believe our clients can or should in most cases be focused on the single event - experience is a repeated event and this means that it implies a history that is invoked each time an individual interacts with the particular product, interface or environment in question."

And Now . . . Whither XMod?

Susan Dray and David Siegel observed difficulties between promoting a vision and implementing the changes it requires. They acknowledge that a vision must be behind any company's effort to become more user-centred, but note that "Vision can get in the way of change when corporate activities that focus on vision are disconnected from the current work in progress. ... By itself, a vision does not tell you what to do next." They also write that "despite the growing awareness of such things as the importance of good user interface (UI) design, usability, and UCD practices, it is extremely rare that companies adopt a fully integrated UCD approach in one grand strategic shift."

Written a year before XMod came into being, Dray and Siegel's observations seem relevant to XMod's first year. It hasn't been painless. Much of the emphasis in the discipline in the first months of its existence was on strategy rather than human-centred design. The nascent discipline seems to have gone through a bit of an identity crisis. There has been some confusion, for example, about what XMod is and what it does. "The notion of 'user experience' has become overexposed. It's moving toward becoming a mere buzzy sentiment," said one information architect. "I think there is a need for clarity. We need a nice, hard-edged conceptual model of XMod."

Design Tensions: Virtually Real or Really Real?

In addition to the technical challenges of designing and implementing this collaborative system, there are the more subtle issues of convincing senior professionals and their employers that the system is easy to use, is appropriately realistic and will deliver the training they require. Would you trust the captain of your passenger ferry who has practised his command and control skills on something which looks suspiciously like an arcade game? Similarly, would you, as personnel manager of a large shipping company buy this product and expect to have your ships' captains to use it?

The DISCOVER trainers believed that the collaborative virtual environment should aim to model a ship as fully and as accurately as possible to bridge this credibility gap. However, we recognize that the DISCOVER presentation of reality must necessarily fail, as it is likely that even small limitations will shatter the illusion. So the design challenge is to identify and abstract from reality those elements which will give a sufficiently good impression of a ship. But how real is real? To date, existing research has focused on achieving a sense of presence and evaluation instruments have been developed to measure just that, but what has not been established is whether "presence" is a good measure of being real. During our requirements work at one of the partners' sites in Denmark, a trainer told us that when a mariner was using their physical simulator they spoke of it as being "their ship" within 30 minutes of use. Physical simulators in contrast to a collaborative virtual environment are equipped with real, physical controls, readouts, charts and manuals with a synthetic display: for CVEs, all is synthetic.

Internet 2010

Evaluating Reality?

From a functional perspective, the environment must (for example) be robust, adequately fast, allow the movement of trainees and their interaction with each other and various objects, support trainer-trainee interaction, the modification of the environment by trainees and provide the numerous other functions specified in the requirements. Evaluation of such features is relatively simple, through inspection against the requirements list combined with simple trials covering the actions necessary to support the training scenarios. Narrow usability evaluation is again fairly unproblematic. Initially, we have used expert cognitive walkthroughs based on the structure suggested by the COVEN project, extended to cover aspects of usability for pedagogic interaction. These results are supplemented by the administration to representative users undertaking task-based trials of Kalawsky's VRUSE questionnaire instrument (Kalawsky, 1999), with additional material to elicit data about usability for collaboration. We are left with the questions, "However how does one evaluate reality?" and "Is presence an appropriate measure?". At the time of writing we are left with a round of iterative evaluation and redesign to determine whether the system is sufficiently real.

Tuesday, February 12, 2008

Communication with Clients

One of the first political stumbling blocks we encountered with our wire- frames in project A was their use in client discussions. Our aim in initial client meetings was to arrive at agreement about how the site would work in terms of navigation, and the kinds of content for each page such as photos, text, tables and so on. However, because of the apparently realistic content and dynamic behaviour, the client tended to focus on small details of content rather than broad views of the navigational and content structure - the very aspects on which we wanted feedback. For example, we put fictional prices with their products, and used fictional attributes to illustrate how real attributes would be displayed. While this approach saves time in developing prototypes, as real values do not have to be researched, the client latched on to these specific parts of the prototype asking questions such as "do we really sell them at that price" and "I didn't realize that product came in that colour". Constantly explaining the rough nature of the prototype meant that we concentrated on small details rather than discussing the overall nature of the proposed web-site.

Internet 2010

In terms of our dimensions we would suggest that at this early stage of our ideas the prototype was too high in fidelity for the target audience (the clients). In retrospect, it would have been better to use rougher prototypes at this stage with the client, e.g. paper prototypes, or clickable paper-looking prototypes using a computer to show the navigational flow.

In Project B the wireframe worked relatively well as a communication tool with the client. It made it relatively easy to explain and discuss ideas of information structure and functionality with the client as the wire frame made it very concrete. The problem with the wireframe in Project B was that theclient thought the wire frame was close to the final site but for us it was not. This led to problems when communicated to the external graphic design agency (see Communicating with Graphic Designers). We had different interpretations of the fidelity of the wireframe which led to political tensions.

Internet Blogosphere