Caught Between Real and Virtual Worlds

While agreement was breaking out at this workshop, so too were some concerns. As most HCI practitioners would agree the distinction between design and evaluation in HCI is often blurred, indeed design and evaluation have been described as the different sides of the same coin. The design of the DISCOVER system proved to be no different. As we elicited requirements and undertook early design we kept an eye on evaluation, ultimately as a sanity check. The use of the prototype, described above, had given us clear usability requirements. The affinity diagram had given us detailed requirements of the design, content and substance of the DISCOVER collaborative virtual environment but concerns as to the issue of evaluation (and with it validation) began to emerge.

Design Tensions: No Magic, Thanks

For many collaborative tasks in virtual environments, the goal is to achieve a particular state of affairs within the virtual world, for example agreeing the layout of an office, as in Hindmarsh et al. (1998). For others, the activity within the CVE is part of a larger collaborative process, but the way collaboration works within the CVE need not exactly replicate real world interaction. In contrast, the DISCOVER environment must support the acquisition of real world skills. In short, skills acquired in the DISCOVER CVE must be transferable to the real world of the ship. This presents a significant constraint: ideally, interaction and collaboration must not be artificially harder or take longer than in the real world, but neither must they be artificially easier or executed faster. Thus the design of DISCOVER should treat with caution "magical" devices such as birds' eye views of the state of environment, Star Trek-like transporting, or visible rubber banding between an avatar and its current focus of attention.

Accordingly, much research which has addressed the problems of ensuring that the users of such environments are aware of their surroundings and of other users cannot be employed directly. Consider, for example, the problems with recognizing other avatars. To maintain realism, avatars cannot be simply labelled. There may also be the presence of dense smoke, and perhaps the need for the avatars to wear vision obscuring breathing apparatus. All of this means that recognizing one's colleagues (as avatars), mediated in the real situation by such characteristics as gait, stance and minor variations in standard issue clothing, becomes far more difficult.

Thus there is a fundamental tension between exploiting the technology to the full to produce a state-of-the-art virtual training environment and creating one which is faithful to the behaviour and constraints of a real ship. This, of course, is not merely a design issue but also presents a corresponding evaluation challenge.

Develop a Highly Usable VR System - and Do It Yesterday!" Continue...

Phase 2: Hello, I'm Your Avatar

A prototype system was created and demonstrated to trainers and prospective trainees at all three maritime training sites. In use each trainee begins by adopting a role - captain, chief engineer, first mate and so forth. They then put on a lightweight set of combined headphones and microphone to afford communication with other avatars and the trainer. Once a minimum of two trainees have entered the system, the trainer, who has been overseeing this process, starts the training scenario (Fig. 6.1). Avatars are able to communicate with one another by way of (virtual) radio and (virtual) telephone and are able to walk and run, open doors and pick up and operate fire extinguishers.

The trainer (or trainers) have all these facilities but have the God-like powers of being able to watch all trainees simultaneously, set fires and so forth as can be seen in Fig. 3.2. The training itself consists of working through a detailed scenario, usually drawn from a real world incident, which requires the trainees to role play and collectively deal with the problems which arise during the course of playing it out. At the end of the training, the trainer will debrief those involved and may use the play-back facilities of the DISCOVER CVE to illustrate particular points.

Demonstrations of the software and simple hands-on tasks were followed by interviews and questionnaires. Our intention was to give potential end users a clearer impression of what a CVE is, how it could be used, to elicit feedback which could be used to refine requirements. We had also evidently managed to convince our developers of the importance of usability issues, since they were particularly keen to have any problems of this sort identified. However, our intentions were confounded by the very prominence which usability assumed in these initial trials. Eager for early feedback, and understandably reluctant to undertake development which might be misdirected, our technologists delivered a prototype just as soon as the software could be run independently of its development environment. This meant that although a reasonable impression could be gained of the functionality which could be offered, user interaction was in a very immature state. As a consequence, finer grain usability issues, for example the ability to identify the focus of an avatar's gaze, were obscured by larger difficulties such as moving through the environment. Equally users could not be induced to speculate in depth about how the system might be used, or how training delivered through such a medium might relate to existing practice. For example, it is difficult to convince the captain of one of the world's largest passenger vessels of the possibilities of the new medium after difficulty with movement control has "trapped" him for some minutes in a corner of the virtual bridge. However, some indications did emerge of the type of usage envisaged in each training context, and much debate was triggered about the detailed design features which would be necessary to support these.

The developers now urgently required a unified, detailed, concrete design specification which would nonetheless support each intended context of use. This was to be achieved by means of a workshop involving each of the three maritime organizations. (There was only one offshore organization in the project, whose requirements were relatively unified and straightforward.)

Phase 3: Virtual Reality Meets Paper and Pen

At this two-day event, trainers from the three organizations met with the representatives from one of the software developer organizations and two facilitators from the requirements team.

The agenda was very simple: to agree the detailed functionality of the maritime simulator and how it was expected to be used. We adapted elements of Contextual Design (Holtzblatt and Beyer, 1998) to facilitate these processes, principally a variant of the affinity diagram technique, which supports the identification of common themes from a mass of contextual data. We began by asking each training organization to revisit what they wanted of DISCOVER in terms of the "w" words (familiar to user-centred design practitioners), i.e. why, when, who, where and of course, how. As each organization described their needs we recorded each issue or explicit requirement on a Post-IT® note. At the end of the process we had gathered over 400 Post-ITs of which approximately 10 per cent were subsequently discarded as duplicates or irrelevant on closer inspection.

The trainers were then invited to create an affinity model which required them to sort the requirements/issues into logical groups: emerging groupings included the layout and configuration of the virtual ship, the appearance and functionality of the avatars and the context of use of the completed system. Throughout this process the software designer helped ground the requirements in reality. Informal discussion with the three trainers afterwards revealed that they thought that the day had gone well. The use of the Post-Its and their grouping was a familiar technique from other contexts, and had fostered engagement and apparent consensus. At the end of the first day we had succeeded in co-constructing an affinity model and subsequently a communications model and an artefact/physical model (ibid.) of the environment to be created.

Defining and Designing "User Experience"

"What seems to be missing is a clear idea about what experience is; what its components or elements are; and, perhaps more importantly, whether it even can be designed or scripted," said Jody Forlizzi at a Usability Professionals' Association conference workshop on experience-based design (EBD) a couple of years ago.

XMod's view is that experience is both definable and designable. In the Fall 1998 issue of The Journal of Design Management an article appeared on experience-based design written by another En Vivo co-founder. EBD involves analysing everyday experience, and making the results useful to design stakeholders. It calls for creating an experience "framework," using ethnographic techniques to study what people "think, do, and use," gain insight into consumer experiences and identify opportunities for new and better ones.

The author is not aware of any ISO standard defining "user experience"; so what is it? XMod's XVP says: "We use 'experience' to refer to the ways in which people develop habits, assumptions, and routines in a particular domain. We do not believe our clients can or should in most cases be focused on the single event - experience is a repeated event and this means that it implies a history that is invoked each time an individual interacts with the particular product, interface or environment in question."

And Now . . . Whither XMod?

Susan Dray and David Siegel observed difficulties between promoting a vision and implementing the changes it requires. They acknowledge that a vision must be behind any company's effort to become more user-centred, but note that "Vision can get in the way of change when corporate activities that focus on vision are disconnected from the current work in progress. ... By itself, a vision does not tell you what to do next." They also write that "despite the growing awareness of such things as the importance of good user interface (UI) design, usability, and UCD practices, it is extremely rare that companies adopt a fully integrated UCD approach in one grand strategic shift."

Written a year before XMod came into being, Dray and Siegel's observations seem relevant to XMod's first year. It hasn't been painless. Much of the emphasis in the discipline in the first months of its existence was on strategy rather than human-centred design. The nascent discipline seems to have gone through a bit of an identity crisis. There has been some confusion, for example, about what XMod is and what it does. "The notion of 'user experience' has become overexposed. It's moving toward becoming a mere buzzy sentiment," said one information architect. "I think there is a need for clarity. We need a nice, hard-edged conceptual model of XMod."

Defining Roles within the Project

Intranet project roles should be clearly defined at the outset of the project. This includes an understanding of job titles and of what each person adds to the project mix. It also includes an understanding of overlapping skills, and boundaries around their expectations of each other.

In Example A, two-thirds of the User-Centred Design group consisted of content writers and artists who were not familiar with the term UCD. The usability engineers were challenged to find their place within the group, to help clarify the difference between roles on the team, and to incorporate usability engineering techniques wherever possible.

To successfully integrate themselves with the rest of the team, the usability engineers worked to understand the value that each team member provided. Each person's role was considered equally valuable and respected, and each team member was encouraged to think outside of his or her role, even if his or her opinions were sometimes overruled. Focusing on the common goal helped promote a combined sense of ownership and teamwork that has extended throughout many different projects to this day. Still, the usability engineers should have gone one step further. Clarifying the role of each team member in writing would have formalized the plan, defining it clearly from the outset. For example:

1. Usability Engineer. Conducts field studies and other observations, creates conceptual models, researches best practices, conducts heuristic evaluations of early design models and concepts, content (text, links, etc.), and navigation, provides the team with design feedback throughout the project.

2. Visual Designer. Creates intranet graphics, selects colours, fonts, and other graphical elements (buttons, links, etc.).

3. Content Provider. Writes all of the text that appears on the site, including text blocks, link names, button names, and banners, if included.

It is less important that the information be well written than that it be clear. In Example A, the usability engineers found that true working definitions were more appropriate than those you might find in a textbook. If the members of the project team feel that there is overlap in roles, for example, writing these issues down is a good way to ensure that they are dealt with. Doing so at the start of a project helps establish working relationships, expectations, and limits. A "roles" document can also help the project leader to understand who is responsible for what tasks.

People's familiarity with these roles is another critical factor. In our Example B, none of the consulting company's project team had worked together before. Although the roles of Project Manager, Technical Manager, and Developer were well established within the consulting company, all of the people filling these roles were new employees. To make matters worse, the concepts of usability engineering and information architecture were new to the consulting company as well, so no one quite knew how to integrate them into the process. The lack of clarity regarding roles within the consulting company was amplified when external clients became involved.

After the usability engineer created several page layouts, the visual designer created three concepts for the client to review. One of these concepts deviated drastically from the layouts — the navigation was on the right side of the screen. Although the usability engineer had made it clear to the visual designer that the right side navigation was not usable, she was on vacation when the concepts were presented. The client loved the uniqueness of the right side navigation model and selected it as the framework for the site. If the visual designer trusted the usability engineer and clearly understood the implications of an unusuable design, he might never have recommended the unorthodox design to the customer.

In addition, the usability engineer should have provided the team with extensive data supporting her page layouts. The team could have referred to this when the usability engineer was out of town.