搜档网
当前位置:搜档网 › Extension of heuristic evaluation method a review and reappraisal

Extension of heuristic evaluation method a review and reappraisal

, 179–197

Ergonomia IJE&HF, 2005, Vol. 27, No. 3

Extension of heuristic evaluation method:

a review and reappraisal

Chen Ling1, Gavriel Salvendy1,2

1 School of Industrial Engineering, Purdue University, West Lafayette, Indiana, USA

2 Department of Industrial Engineering, T singhua University, Beijing, PR China

Abstract

As one of the major discount methods, the heuristic evaluation method

(HE) is the most commonly used usability inspection method. We

introduced the history and procedure of this method, as well as its

strengths and weaknesses. We then reviewed the applications of this

method to different Human-Computer Interaction systems with the

adapted heuristic sets. We also reviewed many studies that extended the

traditional HE method in different ways. Finally, the paper ends with a

reappraisal of these extension methods and future research direction to

improve the HE method.

Key words: heuristic evaluation, usability inspection, extension method,

adapted heuristics

Address for correspondence:

Gavriel Salvendy, Grissom Hall, Room 263, 315 N. Grant Street,

School of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA

e-mail: salvendy@https://www.sodocs.net/doc/4211375578.html,

ergo2005_3.indd 1792005-12-20 12:39:14

180

Extension of heuristic evaluation method: a review and reappraisal

There are many methods for the evalu-ation of information technology products

and s ervices (Niels en, Mack 1994; Lewis 2006) for eas e, joyful and productive us e. According to a major s urvey res ult, heuris -tic evaluation (HE) method is currently the mos t us ed us ability method (Ros enbaum et al. 2000). It is a widely accepted and applied in both academia and indus try. T o achieve better evaluation res ult with the HE meth-ods on various information technology sys-tems, many studies have been conducted to extend the traditional HE method in differ-ent ways. A comprehensive review of studies on extension to HE method is provided.

1. History and procedure The concept of “di s

count u s

ability method” (Niels en 1993) has been around for more than a decade. The as s et of dis -count usability methods is to save time and cos t of carrying out a us ability s tudy while achieving satisfactory study results. Heuris-tic evaluation method is one of the main discount methods because it can effectively detect us ability problems with limited time and resources (Nielsen 1994b). Nielsen and Molich invented the HE method in 1990 (Molich, Niel s

en 1990; Niel s

en, Molich 1990). It is a us ability ins pection method whereby a s et of evaluators produces lis ts of us ability problems in a us er interface by going through it and noting deviations from accepted usability principles (Nielsen, Phillips1993). These accepted usability prin-ciples are als o called heuris tics. The evalu-ation of interface us ed to be difficult and time-con s uming due to the intimidating number of guidelines (in thous ands ) to ob-serve. Nielsen and Molich (1990) cut down the complexity of the extens ive collections of guidelines by two orders of magnitudes, and derived ten “golden rules ” or heuris -tic . The e heuri tic were cho en ba ed on Niels en and Molich’s unders tanding of typical problem areas of us ability, as well a an informal con ideration of exi ting guidelines. They des cribed their method as “the most general of the usability inspection methods and is also the easiest to learn and apply” (Nielsen, Molich1990). The original list of usability heuristics are listed here.? Simple and natural dialogue ? Speak the user’s language

? Minimize the user’s memory load ? Cons is tency ? Feedback

? Clearly marked exits ? Shortcuts

? Precise and constructive error messages ? Prevent errors

? Help and documentation

Nielsen (1994a) later performed a more

formal study on heuristics. He chose 101 us-ability principles , which includes the origi-nal set of heuristics listed above, as well as six other collections of published principles or guidelines . He s tudies how well thes e principles can account for the 249 us abil-ity problems found during the evaluation of 11 interactive systems. He attempts to pick the principles that provided the best expla-nation of the usability problems. The result of a principal component analysis indicates that seven factors could account for 30% of the variability of the usability problems. The seven factors formed the basis for a revised set of 10 heuristics with maximum explana-tory power. Another three heuris tics were added by Niels en bas ed on his own expe-rience. The commonly us ed ten heuris tics for interactive s ys tems are lis ted as follows (Nielsen 1994b).

? Visibility of system status ? Match between s y s

tem and the real world

? User control and freedom ? Consistency and standards ? Error prevention

? Recognition rather than recall ? Flexibility and efficiency of use ? Aesthetic and minimalist design

? Help us ers recognize, diagnos e, and re-cover from errors

? Help and documentation

ergo2005_3.indd 1802005-12-20 12:39:15

181

C. Ling, G. Salvendy

During the heuri s

tic evaluation, the evaluators decide on their own how they want to proceed with the evaluation. It is generally recommended that they go through the interface at least twice. The first pass is intended to get a feel for the flow of the interaction and the general scope of the sys-tem. The second pass then allows the evalu-ator to focus on specific interface elements. Heuris tic evaluators us e their judgment to determine whether an interface violates any of the heuristics (Nielsen, 1994b). Different people find different u s -ability problems and give different s ever-ity ratings to us ability problems us ing HE (Niels en 1994b; Hertzum, Jacobs en 2001). Therefore, it is necessary to involve multiple evaluators in HE. It is recommended to use

around five evaluators becaus e it can cov-er around 75% of total us ability problems and have the highes t benefit-to-cos t ratio (Nielsen, Landauer 1993; Nielsen 1994b). HE method can find both major and minor us ability problems , but mos t prob-lem s

found are minor one s

(Niel s en, 1994b). It has higher thoroughnes s than cognitive walkthrough method becaus e it finds more intermediate and minor prob-lems (Sears 1997). But it can also find more fals e-pos itive problems that will not actu-ally occur with real us ers and need to be eliminated (Sears 1997). Studies show that the us e of HE early in the des ign proces s tends to mis s certain clas s es of problems , s uch as thos e aris ing from perceptual-mo-tor slips (Mack, Montaniz 1994), or miss-ing functionality (Nielsen 1992). This might be compensated by directing the inspectors to pay particular attention to thes e areas. When compared to user testing, HE tends to mis s tas k-bas ed problems , whereas the user testing tends to miss interface feature related problems becaus e heuris tic evalua-tors were not absorbed in using the system to perform a tas k as the us ers in the us er testing (Doubleday et al.1997). HE usually identifies the caus e of the problem while end user testing may indicate the symptom of the problem (Doubleday et al. 1997). Another empirical s tudy s hows that HE is more effective in identifying us ability problems as ociated with s kill-bas ed and rule-bas ed levels of performance whereas user testing is more effective in identifying us ability problems as s ociated with knowl-edge-bas ed level of performance (Fu et al.

2002). Since HE and usability testing com-plement each other by identifying different s ets of us ability problems , it is s ugges ted that they s hould be us ed together to find

more comprehensive usability problem set. The common s ugges tion is to apply HE firs t to get the “lower hanging fruit”, and then perform usability testing to clarify the rest of the problem areas (Nielsen 1994b). 2. Previous studies HE method has been widely us ed to evaluate and improve the usability of many

systems. When first developed, HE was ap-plied to the many low fidelity prototypes of telephone system such as the screen dumps, written specification of information system, and voice response system (Nielsen, Molich 1990). Nielsen then applied the method to other telephone system such as telephone op-erated interface in banking system (Nielsen 1992), and the integrating system for inter-nal telephone company use (Nielsen 1994b). After its effectivenes s in detecting us ability problems has been es tablis hed by res earch (Jeffries et al. 1991; Niels en 1992), it was widely applied to many different comput-ing systems, such as the hypermedia browser (Connell, Hammond 1999), digital libraries (Blandford et al. 2004), World Web Proto-type (Levi, Conrad 1996), and museum web s ite (Bendoly, Goldman 2003) etc. It con-tinues to be used for newly emerged infor-mation s ys tems and devices , s uch as Palm Pilot pers onal organizer (Slavkovic, Cros s 1999), laboratory teaching tool (Avouris et al. 2001), and brokerage platform for E-learning (Law, Hvannberg 2002). It was also used to analyze the learning process of the

ergo2005_3.indd 1812005-12-20 12:39:15

182Extension of heuristic evaluation method: a review and reappraisal

object-oriented language (Warren, 2004) and the programming practice in integrated development environment of C++ (Kline 2002).

HE method has been us ed beyond the traditional information technology indu-tries. It was applied to improve the musical related hardware s uch as electrical guitar amplifier (Fernandes, Holmes 2002). It helps to find us ability problem with an acous tic fis hing dis play (Mills 1995). It improved the usability of many industrial hypermedia applications us ed by s hop floor operators (Fakun, Greenough 2002). Health care in-dustry start to benefit from using it by cor-recting us ability problems with the proc-es s of Internet T elemedicine (Lathan et al. 1999), the medical devices such as infusion pumps (Zhang et al. 2003), and healthcare information systems (McGrow et al. 2004).

T o better address usability issues in com-puting systems from different domains, spe-cific heuristic sets can be developed. These adapted heuris tics s ets will be dis cus s ed in detail in section 4.1.

3. Strengths and weaknesses

Heuris tics evaluation method has both trength and weakne e. A a di count method, HE method has the advantage of effectively identify the us ability problems, and identify the most serious problems (Jef-fries et al. 1991). At the same time, it is easy to learn and fast and cheap to apply. Because it can be applied to low fidelity prototypes, it become an important method early in the development (Nielsen 1994b). Its evaluation procedure is relatively informal due to its free-form s tructure. Only a s mall number of evaluators are needed. Heuristics has key benefits of concise, memorable, meaningful and ins ightful (Paddis on, Englefield 2003). The HE method can be cus tomized to dif-ferent domains by developing domain-s pe-cific heuris tics. (Niels en 1994b). All thes e s trength made HE the mos t us ed us ability method (Rosenbaum et al. 2000).

Des pite its popularity, HE method has many drawbacks. Though HE is s aid to be a s ys tematic ins pection of the us er in-terface des ign for us ability (Mack, Niels en 1994; Niels en, Mack 1994), its evaluation procedure is loos ely s tructured (Jeffries et al. 1991). Studies s how “s ubs tantial unex-plained variability in performance from one evaluation to the next” (Nielsen1994b). The past literature listed at least four weaknesses of the HE method:

(1) HE relies heavily on the expertis e of the evaluator both in usability and in do-main knowledge (Doubleday et al. 1997). Heuris tic valuators can be novice, us ability expert and double expert in both us ability and domain knowledge (Niels en 1994b). Double expert can find mo t problem while novice finds the least. But the experts are always harder to find.

(2) HE lack di covery re ource to dis cover us ability problems. Niels en noted that the heuris tic s et is good at explaining exis ting problems, but he does not know how effective they will be at finding usabil-ity problems. Heuristics are not always able to deeply guide evaluators (Doubleday et al.1997). Cockton and Woolrych (2001) de-veloped a Discovery and Analysis Resources (DR-AR) model to differentiate two distinct evaluator’s activities: discovery and problem analysis. Effectiveness of a method depends on the resources that the evaluator have for these two activities. HE method lacks both dis covery and analys is res ources. The HE method does little to prepare evaluators for inspection. The essential discovery resource that HE lacks is a focus on tas k execution and complex domain goals that are the ori-gin of subtle interaction problems (Cockton, Lavery, Woolrych 2003).

(3) HE method produces a large number of fals e pos itive us ability problems which require evaluators to s pend extra time to eliminate (Miller, Jeffries 1992). With only a lis t of heuris tics providing guidance, the HE process is unstructured and does not fo-cus on user tasks (Sears 1997). Because HE

ergo2005_3.indd 1822005-12-20 12:39:15

183 C. Ling, G. Salvendy

does not have much control over its proc-ess, the less experienced evaluators may lose focus on the users. They may focus on issues that do not impact users, and therefore may find unreal usability problems (Sears 1997). The HE method lacks the analysis resource becaus e it lacks a res pect for us er’s intelli-gence and an understanding of display-based interaction that eliminates logically possible problems as empirically improbable (Cock-ton, Lavery, Woolrych 2003).

(4) HE method doesn’t provide a system-atic way to generate fixes to us ability prob-lems, or a way to assess the probable quality of any redesign, thus, it identifies usability prob-lems without providing direct suggestions for how to solve them (Nielsen 1994b)

4. Extension of heuristic evaluation method

T o produce a more complete set of us-ability problems in a s ys tem, practitioners have modified HE method in different ways to s uit their evaluation needs for different computing s ys tems and try to compens ate for the drawbacks of the HE method. The modification us ually occurs in three ways, extending the heuris tics s et, extending the HE method by modifying the evaluation procedure, and extending the HE method with a conformance rating scale.

4.1 Extended heuristic set

HE method was developed and applied mainly for s ingle us er, productivity-orient-ed des ktop programs, which were the ma-jor computer application in the early 90s. However, with computer technologies get-ting more integrated into everyday life and new types of human computer interaction emerging, Niels en’s 10 heuris tics may not be able to cover usability issues in the new computing y tem. For example, mobile s ys tems need to addres s is s ues of changing context of us e (Vetere et al. 2003). Notifi-cation s ys tems and ambient dis play s ys tem need to s tudy the overall performance of the s ys tem including both the traditionally s tudied primary tas ks, and the s econdary tasks as well (Berry 2003). T eamwork issues involving multiple us ers rather than s ingle user task work issues need to be addressed by groupware systems (Gutwin 2000; Drury 2001). The goal of the traditional system is usually fast and easy, but for the game sys-tem, the goal is eas y to learn, but hard to master (Desurvire et al. 2004). Thus, games de ign need to intentionally contravene Niels en’s heuris tics of prevent errors, but provide pos s ibility of errors for the us ers instead (Johnson, Wiles 2003). Therefore, Nielsen’s ten heuristics are not readily appli-cable to many new domains with different goals and us ability is s ues. As domain-s pe-cific heuristics can be developed to supple-ment the existing heuristics (Molich, Nielsen 1990; Nielsen 1993; Nielsen, Mack 1994), researchers have derived many adapted heu-ristic sets to address the typical requirements and problems in different kinds of applica-tion domain. T able 1 lis ts s tudies that de-veloped new s ets of heuris tics, the domain that the heuris tic s et was developed for, a brief des cription of the heuris tics, and the way that they are developed.

These studies cover many types of heu-ristics sets. Some are intended for use across different technologies (Kamper 2002), and mos t are adapted domain-s pecific heuris-tics intended for specific type of computing systems with different goals and interaction nature. Important design issues that are not covered by the Niels on’s traditional heu-ris tics s et are emphas ized in the heuris tics according to the different purposes. For ex-ample, to design better playability of games (Des urvire et al. 2004), four as pects need to be addres ed: game play, game s tory, mechanics and us ability. Des igning a better electronic news paper needs to s tres s three as pects, including graphics, general layout, navigation (Mariage, Vanderdonckt 2000).

From thes e s tudies, we can s ee that it generally takes two s teps to derive a new set of heuristics: heuristic development and

ergo2005_3.indd 1832005-12-20 12:39:15

184Extension of heuristic evaluation method: a review and reappraisal T able 1. Adapted heuristic sets.

Reference Application

domain

Number and content of heuristics

Developed

based on

Kamper 2002Many

different

domains Lead (6)

Follow(6)

Get out of way (6)

Previous

research

Rau, Liang 2003Websites Information Design (10)

Consistency (2)

Navigation (2)

Operation (6)

Errors (4)

Nielsen’s

heuristic set

and Garzotto

et al.1995

Sutcliffe 2001Website

attractiveness Attractiveness (7)

Content (5)

Nielsen’s

heuristic set

and previous

research

Paddison, Englefield 2003Accessibility9: Provide meaningful and relevant alternatives

to non-text elements; Support consistent and

correctly tagged navigation; Allow complete and

efficient keyboard usage; Ensure appropriate use

of standard and proprietary controls; Do not rely

on colour alone to code and distinguish; Allow

users to control of potential distractions; Allow

users to understand and control time restraints;

Make certain the Web site is content compatible

with assistive technologies.

Previous

research

Muller et al. 1998Participatory15: Revised Nielsen’s set plus Respect the user

and his/her skills; Pleasurable experience with

they system; Support quality work.

T raditional

heuristic set

(Nielsen,

Molich 1990)

Evans, Sabry 2003Web-based

learning

system

9: Making navigation easy; Engaging learner

frequently; Allow for reflection; Using a variety

of interactions; Using multimedia; Applying what

has been taught; Being relevant; Being timely;

Grabbing attention.

Previous

research

Reeves 2002E-learning

system 15: Revised Nielsen (1994b)’s set plus interac-

tivity; Message design; Learning design; Media integration; Instructional assessment; Resources; Feedback.

Nielsen’s

heuristic set

Squires, Preece 1999Educational

software

8: Match between designer and learner models;

Navigational fidelity; Appropriate levels of learn-

er control; Prevention of peripheral cognitive

errors; Understandable and meaningful symbolic

representation; Support personally significant

approaches to learning; Cognitive error recogni-

tion, diagnosis and recovery cycle; Match with

the curriculum is evident.

Nielsen’s

heuristic set

and previous

research

Greenberg et al. 1999Shared

workspace

groupware

system

5: Provide centers; Provide awareness;

Allow Individual views; Allow people to manage

and stay aware of their evolving interactions;

Provide a way to organize and relate locales to

one another.

Previous

research

ergo2005_3.indd 1842005-12-20 12:39:15

185 C. Ling, G. Salvendy

Baker et al. 2002Shared

workspace

groupware

system

8: Provide the means for intentional and appro-

priate verbal communication; Gestural commu-

nication; Provide consequential communication

of an individual’s embodiment; Shared artifacts;

Provide protection; Management of tightly and

loosely-coupled collaboration; Allow people to

coordinate their actions; Facilitate finding col-

laborators and establishing contact.

Previous

research

Drury 2001Synchronous

collaborative

systems 8: Show identities of people in the workspace;

Show the activities of the other participants;

Show the locations where the other participants

are working; Show the changes made by other participants; Show the goals of other participants;

Show the interdependencies of participants’

work; Show the extent to which the system

supports social rule; Provide clues for users to

predict the system’s probable future status.

Previous

research

Mankoff et al. 2003Ambient

displays

12: Sufficient information design; Consistent

and intuitive mapping; Match between system

and real world; Visibility of state; Aesthetic and

pleasing design; Useful and relevant informa-

tion; Visibility of system status; User control

and freedom; Easy transition to more in-depth

information; “Peripherality” of display; Error

prevention; Flexibility and efficiency of use.

Nielsen’s

heuristic set

Berry 2003Notification

system 8: Timely; Reliable; Consistent; Information un-derstandable; Shortcut; Indicate status; Provide

context; Allow adjustment.

Evaluation

results

Somervell et al. 2003Large screen

information

exhibits

8: Appropriate color schemes can be used for

supporting information understanding; Layout

should reflect the information according to its

intended use; Judicious use of animation is neces-

sary for effective design; Use text banners only

when necessary; Show the presence of informa-

tion, but not the detail; Using cyclic displays can

be useful, but care must be taken in implementa-

tion; Avoid the use of audio; Eliminate or hide

configurability control.

Evaluation

results

K?ykk? et al. 19993D multi-user

interface

12: modified Nielsen’s set plus Real world meta-

phors have to be clearly understandable; Provide

support for orientation, navigation and move-

ment; Avoidance of delays and waiting periods in

the performance.

Nielsen’s

heuristic set

and evaluation

results

Vetere et al. 2003Mobile use Provide locales(6)

Provide awareness (5)

Provide individual views (3)

Interaction trajectories (4)

Civic structures (5)

Previous

research

T able 1 cont. Adapted heuristic sets.

ergo2005_3.indd 1852005-12-20 12:39:15

186Extension of heuristic evaluation method: a review and reappraisal

Desurvire et al. 2004Playability of

games

Game play (16)

Game story (8)

Mechanics (7)

Usability (12)

Previous

research

Zhang et al. 2003Healthcare

medical

devices

14: Consistency and standard; Visibility of

system state; Match between system and world;

Minimalist; Minimize memory load; Informative

feedback; Flexibility and efficiency; Good error

messages; Prevent errors; Clear closure; Reversi-

ble actions; Use user’s language; Users in control;

Help and documentation.

Nielsen’s

heuristic set

and Shneider-

man (1998)’s

guideline

McGrow et al. 2004Healthcare

information

system

24: For example: Timely feedback from the

computer; User in control; Easy to move and

navigate; Undo unwanted actions at anytime;

Escape or exit from the program at any time, etc.

Nielsen’s

heuristic set

Fakun, Greenough 2002Industrial

hypermedia

applications

7: Easy retrace mechanism; Overview, zoom,

filter, details search strategy; Information chunks

of single concepts; A sense of entire domain and

a preview device; All elements have same conven-

tions; Judicious use of colors; Known metaphor.

Previous

research

Mariage, Vanderdonckt 2000Electronic

newspaper

Graphics (7)

General layout (7)

Navigation (4)

Previous

research

T able 1 cont. Adapted heuristic sets.

heuri tic validation. In the heuri tic de-velopment s tep, res earchers come up with a new s et of heuris tics for us e in the HE method. If the new set is developed on the basis of the Nielsen’s heuristics (Mankoff et al. 2003, Muller et al.1998), then the com-mon s teps include firs t taking off the not-applicable heuris tics from the Niels on’s 10 heuris tics bas ed on whether the heuris tics met the primary goal of the s ys tem, then modifying the applicable heuris tics to s uit the us e within the domain, and las tly add-ing additional heuris tics to form the new s et. Many s tudies come up with new s et of heuris tics without referencing the Niel-son’s original heuristics. These heuristic sets don’t bear much s imilarity to the original s et in s tructure and content. Regardles s of whether the traditional heuristic set was ref-erenced or not, to come up with new heuris-tics, researchers always need to consult the pas t literatures, or pas t evaluation res ults, and the expert opinions. T wo common ap-proaches of deriving new heuristics include the res earch-bas ed method and the evalu-ation-ba ed method (Paddi on, Englefield 2003). In the res earch-bas ed method, key points of the domain are identified based on past literatures. For example, Drury (2001) constructed eight heuristics for the synchro-nous collaborative s ys tem bas ed on the s ix related theories and metaphors for the type of s ys tem. In the evaluation-bas ed method, common usability problems with the system are categorized into heuristics. For example, Berry (2003) categorized the common us-ability problems with the notification system into eight major heuristics to form the heu-ristic set. Sometimes, second-level heuristics with finer granularity were developed to help evaluators better understand the intent and focus of the heuristics. The whole sets are us ually reviewed and modified by do-main experts before they are used in further validation experiments.

In the validation phas e of the heuris-tic adaptation, the authors of the newly developed heuris tics s et will compare its

ergo2005_3.indd 1862005-12-20 12:39:15

187

C. Ling, G. Salvendy e ffectiveness with Nielson’s original sets by conducting empirical s tudies (Baker et al. 2002; Berry 2003; Mankoff et al. 2003), or

benchmark with user testing results (Desur-vire et al. 2004). The adapted set of heuris-tics can usually obtain better or comparable effectivenes s as the Niels en’s original s ets because their heuristics fit the evaluated do-main better (Baker et al. 2002; Berry 2003; Mankoff et al. 2003).

The adapted heuri s

tic s

can undergo everal rounds of modifications before a final optimal s et of heuris tics emerge. For example, three out of nine heuris tics for web-bas ed learning s ys tems were changed after the evaluation s tudy and dis cus s ions (Evans, Sabry 2003). T o develop a heuristic s et for groupware s ys tem, Greenberg et al. (1999) first derived five heuristics based on the Locales framework of social interaction. Later on, the mechanics of collaboration framework was us ed as the theory bas is to address a narrower focus: the shared visual workspace. Then, a new set of eight heuris-tics was developed (Baker et al. 2001) and empirically s tudied (Baker et al. 2002) for the groupware system.

After the new heuris tics s et is devel-oped, the process of applying the HE meth-od us ually remains the s ame to achieve its benefit of “dis count methods ”. However, there can be s ome additional changes. For example, Vetere et al. (2003) used the heu-ristic walkthrough procedure with their mo-bile heuristic set rather than the HE evalu-ation procedure. In participatory heuris tic evaluation (PHE) (Muller et al.1998), three “participatory” heuris tics were added and validated to cons ider the us age context of the s ys tem. Rather than jus t us ing expert ins pectors as in traditional HEs , the PHE s tipulates involving us ers as ins pectors for the participatory nature.

4.2 Extension methods with modified evaluation procedure In an attempt to overcome HE’s draw-backs and produce better results, many studies extended the original HE method in several

different ways. T able 2 lis ts s even extended methods with a brief description of the major characteristics and the system the method has been applied to. As these methods extend the HE method in their unique ways, we will de-scribe them in detail in the following.

T o give more s tructure to the tradi-tional HE methods , Sears (1997) created a technique called heuristic walkthrough that combines benefits from the HE, cognitive

walkthrough and u s ability walkthrough.

While keeping a free-form evaluation run

from HE, it adds the tas k focus of cogni-tive walkthrough to bring in more structure.

The evaluators evaluate the interface in two

pa e , a ta k-oriented evaluation guided by four thought-focus ing ques tions derived from cognitive walkthrough, followed by a free-form evaluation guided by heuris -tics. When compared to HE, the heuris tic walkthrough methods produced fewer false positives usability problems. Thus its results have higher validity than HE.

The HE-plus method added a con-textualized layer called “u ability prob-lem profile” to aid the evaluation proces s (Chattratichart, Brodie 2002). The us abil-ity problem profile contains the common u s ability problem area s a s s ociated with the s ame type of application/product. For example, profile for e-commerce webs ite includes six problem areas: content, graph-ic , navigation, layout, terminology and matches with us er’s tas ks. In the HE-plus method, a lis t of the problem areas that cons titute “us ability problem profile” was given to the evaluators in addition to the list of heuris tics. Evaluators are as ked to look for problems in the areas given in the us -ability problem profile while they examine the web s ite agains t the heuris tic s et. It is proven to give results with higher reliabil-ity, thoroughness, validity and effectiveness

than the HE method when applied to two online s hopping webs ites (Chattratichart,

Brodie 2002, 2004). It is reasoned that this is because that the usability problem profile

ergo2005_3.indd 1872005-12-20 12:39:16

188

Extension of heuristic evaluation method: a review and reappraisal

T a b l e 2. E x t e n s i o n m e t h o d s t o h e u r i s t i c e v a l u a t i o n .

R e f e r e n c e

E x t e n d e d m e t h o d s I n t e n t i o n C h a r a c t e r i s t i c s

S y s t e m a p p l i e d t o

S e a r s 1997

H e u r i s t i c w a l k t h r o u g h

T o b r i n g i n m o r e s t r u c t u r e t o H E T a s k -b a s e d e v a l u a t i o n w i t h t h o u g h t -p r o v o k -i n g q u e s t i o n s f o l l o w e d b y f r e e -f o r m e v a l u a -t i o n w i t h h e u r i s t i c s

E d u c a t i o n a l a p p l i c a t i o n

C h a t t r a t i c h a r t 2002, 2004

H E -p l u s T o i n c r e a s e e v a l u a t i o n r e l i a b i l i t y U s a b i l i t y p r o b l e m p r o f i l e

O n l i n e s h o p p i n g w e b s i t e

G a r z o t t o 1998; M a t e r a e t a l . 2002; A n g e l i e t a l . 2003

S y s t e m a t i c u s a b i l i t y e v a l u a t i o n (S U E )T o g u i d e n o v i c e e v a l u a t o r s ; a n d t r a n s f e r e v a l u a t o r ’s k n o w -h o w

A b s t r a c t t a s k s ; I n s p e c t i o n f o l l o w e d b y u s e r t e s t i n g

H y p e r m e d i a C D -R O M s

K u r o s u 1997, 1998,

1999

S t r u c t u r e d h e u r i s t i c e a l u a t i o n m e t h o d (s H E M )T o a v o i d o v e r w h e l m i n g n u m b e r o f h e u r i s t i c s a n d h e l p e v a l u a -t o r s f o c u s o n a l i m i t e d i s s u e s a t a s a m e t i m e

F i v e s u b -s e s s i o n s : e a s e o f c o g n i t i o n , e a s e o f o p e r a t i o n a n d p l e a s a n t n e s s , n o v i c e /e x p e r t , a n d d i s a b l e d u s e r s . P o r t a b l e m i n i -d i s k

D e s u r v i r e , T h o m a s 1993

P r o g r a m m e d a m p l i f i c a t i o n o f v a l u a b l e e x p e r t s (P A V E )

T o e n c o u r a g e a b r o a d e r s c o p e o f t h i n k i n g ; t o i m p r o v e n o v i c e e v a l u a t o r ’s p e r f o r m a n c e T e n p e r s p e c t i v e s : s e l f , a h u m a n f a c t o r e x p e r t , a c o g n i t i v e p s y c h o l o g i s t , a b e h a v i o r i s t , a s o -c i a l /c o m m u n i t y p s y c h o l o g i s t , a n a n t h r o p o l o -g i s t , a F r e u d i a n , a h e a l t h a d v o c a t e , a w o r r i e d m o t h e r , a n d a s p o i l e d c h i l d .

F l o w c h a r t o f v o i c e i n t e r f a c e Z h a n g e t a l . 1999

P e r s p e c t i v e b a s e d h e u r i s t i c e v a l u a t i o n

T o g i v e m o r e g u i d a n c e t o e v a l u -a t i o n a n d e n c o u r a g e b r o a d e r t h i n k i n g

T h r e e p e r s p e c t i v e s : n o v i c e u s e , e x p e r t u s e , a n d e r r o r h a n d l i n g ; E v a l u a t i o n p r o c e d u r e f o r e a c h p e r s p e c t i v e

W e b -b a s e d i n t e r -f a c e

H o r n b ?k , F r ??k j ?r 2004M e t a p h o r s o f t h i n k i n g (M O T )

T o f o c u s i n s p e c t i o n o n u s e r ’s m e n t a l a c t i v i t y a n d t o m a k e i n s p e c t i o n e a s i l y a p p l i c a b l e t o d i f f e r e n t d e v i c e s a n d u s e c o n t e x t s

F i v e m e t a p h o r o f h u m a n t h i n k i n g : h a b i t f o r -m a t i o n i s l i k e a l a n d s c a p e e r o d e d b y w a t e r ; t h i n k i n g a s a s t r e a m o f t h o u g h t ; a w a r e n e s s a s a j u m p i n g o c t o p u s i n a p i l e o f r a g s ; u t t e r -a n c e s a s s p l a s h e s o v e r w a t e r ; k n o w i n g a s a b u i l d i n g s i t e i n p r o g r e s s .

P o r t a l f o r s t u d e n t s a t u n i v e r s i t y w e b s i t e ergo2005_3.indd 188

2005-12-20 12:39:16

189

C. Ling, G. Salvendy

helps the evaluators focus their evaluations on important problem areas.

The systematic usability evaluation (SUE) method is especially used for evaluating hyper-media usability (Garzotto et al. 1998; Matera

et al.2002; Angeli et al. 2003). It supplies the evaluators with a structured flow of activities by the means of “abs tract tas ks ”. An evalu-ation pattern called abs tract tas ks des cribes in detail the activities the evaluators s hould perform. SUE is believed to facilitate sharing and transferring evaluation know-how among evaluators. Bas ed on the Hypermedia des ign model (HDM), SUE focuses not only on the s urface elements of the s ys tem, but als o on specific hypermedia aspects such as navigation and information s tructures , s ynchronization etc. Another characteristic of the SUE method is conducting us er tes ting after the problems from ins pection are fixed to validate s ubjec-tive aspects such as learning behavior and user ati faction. Empirical compari on between the SUE method and HE shows that, the SUE method is better than the HE in terms of ef-fectiveness, efficiency and satisfaction (Angeli et al. 2003).

In order that the evaluators are not over-whelmed by the large number of heuristics, Kuros u et al. (1997) developed structured heuristic evaluation method (sHEM) by in-troducing structure into the set of usability heuristics. The usability heuristics (Nielsen, Molich 1990) are s plit into three s ub-cat-egories: ease of cognition, ease of operation and plea antne . T wo additional catego-ries concerning the us ers includes novice vs. expert, and users with special care. The evaluation is divided into sub-sessions, with each sub-session devoted to finding usability problems concerning heuristics belonging to one of the s ub-categories. Each s ub-s es s ion takes 30 minutes and there is 15 minutes break between s ub-s es s ions. Becaus e with-in each s

ub-s e ion, s

ubject s concentrate on finding us ability problems with limited amount of heuris tics within the range of human memory chunk s ize (7 plus /minus 2), and several sub-sessions can cover more

us ability is s ues than HE, it is believed that this type of evaluation will be more produc-tive than the HE method. The results show that sHEM can find twice as much usability problems than the traditional HE method (Kurosu et al. 1998, 1999).

Programmed amplification of valuable experts (PAVE) method (Desurvire, Thomas 1993) was developed to encourage a broader scope of thinking by including ten perspec-tives to the evaluation: self, a human factor expert, a cognitive psychologist, a behavior-ist, a social/community psychologist, an an-thropologist, a Freudian, a health advocate, a worried mother, and a s poiled child. It intends to help novice evaluator reveal as many usability problems as experts by aug-menting the evaluator’s existing knowledge and stimulating them to think about usabil-ity more broadly with the help of different perspectives. In this method, evaluators will study the interface for ten times, each time with one of the ten perspectives. When ap-plied to a flow chart of a voice interface (Desurvire, Thomas 1993), it improved the performance of a novice evaluator by giv-ing less false positives, finding real problems and offering more s ugges tion for improve-ments. But this method doesn’t improve the performance of the expert evaluators.

Similar to the PAVE approach, Zhang et al. (1999) proposed the perspective-based method. The method us es pers pectives to focus the evaluator’s attention on a s pecific subset of usability issues during each evalua-tion session. It asks the evaluators to inspect the interface focus ing on one of the three defined perspectives: novice use, expert use, and error handling. Each perspective is asso-ciated with a set of inspection questions based on modified “Seven Stages of Action” Model (Norman 1988). T as k s cenarios are us ed to guide the evaluation. For the “novice us e” pers pective, evaluators think about novice us ers and ans wer ques tions as to whether a novice user can successfully go through each tas k s teps. For the “expert us e”, evaluators think about expert us ers and notice is s ues

ergo2005_3.indd 1892005-12-20 12:39:16

190Extension of heuristic evaluation method: a review and reappraisal

such as short-cuts, appearances, information organizations as they perform tasks. For the “error handling” pers pective, evaluators de-rive possible error types and check the inter-face against these error types to see how well it minimizes errors and assists error recovery . The pers pective-bas ed method gives more s tructure to the HE by as s igning evaluators different res pons ibilities us ing pers pectives , and s tipulate evaluation procedures within each pers pective. The evaluation res ult of two web-based interfaces shows that the per-spective-based method can find more usabil-ity problems than the HE method.The metaphor of thinking (MOT) method (Hornb?k, Fr??kj?r 2004) didn’t explicitly s ay that it is an extens ion of HE method, but its strong similarity with the HE procedure well qualify it as an extension. It was developed bas ed on the clas s ical intro-spective psychology . It aims to focus inspec-tion on users’ mental activity by incorporat-ing five metaphors of human thinking: habit formation, stream of thought; awareness and as s ociation, the relations hip between utter-ances and thought, and knowing. During the evaluation, evaluators firs t get familiar-ized with the application, and then they try to finish three tasks and meanwhile take the perspective of each of the metaphors to find us ability problems in two pas s es. They can think of additional tasks and continue to find problems with the help of the metaphors if time allows. While HE provides simple guide-lines to encourage s traightforward interpre-tation, MOT provide complex guideline and require evaluator’s active interpretation. Experimental results show that MOT discov-ers more s evere us ability problems that are

more complex to fix than HE. It als o takes less time to conduct the MOT evaluation.

4.3 Extension method with a conformance rating scale A method that extended heuri s tic evaluation with a conformance rating s cale has been us ed in s ome s tudies (Mariage, Vanderdonckt 2000; Avouris et al. 2001; Sutcliffe 2001; Agarwal, Venkatech 2002; Berry 2003; McGrow et al. 2004). Evalu-ators are pres ented with a s et of heuris tics and reques ted to rate the interface bas ed on degree of conformance to each heuris -tic with a rating s cale. Sometimes , evalua-tors were also instructed to write down the rationale for their rating (Sutcliffe 2001) or s ugges tions for des ign (Avouris et al 2001) which can be the basis for finding and fixing

us ability problems The rating s cale can be

dichotomous , 5-point or 10-point ranging from lowes t to highes t conformance level. In additional to a list of usability problems found by traditional HE, this extended form

produce quantitative data on the conform-ance rating of the evaluated system to each heuristic. This approach provides an overall assessment of the evaluated system in terms of its weakness and strength, and helps pin-point the area of problems with the system to direct further corrective efforts . It can also give a quality index of the system which helps choosing among competing design op-tions. For example, Agarwal and Venkatech (2002) developed an evaluative ins trument to rate the usability of firm websites of dif-ferent indu trie ba ed on Micro oft u -ability guideline and this type of heuris tic evaluation. Evaluators assume the role of a consumer or an investor when assessing us-ability. They firs t as s ign weight to each of five us ability categories and dis tribute the weight over each s ubcategories. Then they rate webs ite in terms of its quality regard-ing each subcategory. A weighted rating for each s ubcategory can be calculated to get the overall rating of the website. 5. T ool support for heuristic evaluation Software tools and ins truments have

been developed to a i t HE proce to improve the efficiency of evaluators and

achieve more rigorou s re s

ult s

. IBM de-veloped a Heuri tic Evaluation Databa e (HEDB) to s upport the collaborative work of the evaluation manager and the evalua-

ergo2005_3.indd 1902005-12-20 12:39:16

191 C. Ling, G. Salvendy

tors. HEDB implements the key tasks of en-tering findings, as s igning s everity to prob-lems, and editing duplicates in the HE proc-ess (Paddison, Englefield 2003). It has been well received by practitioners.

Since HE can be described as a creative brainstorming process, it might benefit from anonymous, parallel production with the help of collaborative s oftware. Lowry and Roberts (2003) examined ways to increas e the productivity of HE by us ing collabora-tive s ys tem. Experiment res ult s hows that, with the collaborative s oftware, evaluators are aware of us ability problems found by others. So duplicate work is avoided, and a cons ens us of aggregate problem s et can be found much faster.

Another ins trument that can improve the HE res ult is the s tructured report for-mat for us ability problems (Cockton et al. 2003). The us e of the report format can change the evaluator’s behavior by demand-ing more reflection, and in turn res ult in fewer fals e pos itives and more appropriate heuristic usage.

6. Reappraisal

The free-formed s tructure of HE has made it a widely us ed dis count method. But it als o leads to many of its drawbacks (Cockton, Woolrych 2002). HE method has left plenty of room for improvement. Much research has been conducted to improve HE method.

A s et of heuris tics is intended as mne-monic framework that can cue the deeper knowledge body held by an evaluator de-fined by the guidelines and existing expertise (Paddison, Englefield 2003). But for novice evaluators, the us ability related knowledge body is not well developed. Therefore, the ole re ource for their evaluation i the heuris tic s et, which is not enough in many cases. They may pick user tasks and system features randomly to evaluate. Literatures have indicated four ways to give more struc-ture to the HE method and better orient the evaluators in the s earch s pace. When the evaluation is more structured, the evaluators will feel better guided.

(1) Provide structure with modified heuristics set

Some heuristic set was explicitly divid-ed into several parts. For example, heuristic set for websites design was divided into five parts: information design, consistency, navi-gation, operation, and errors (Rau, Liang 2003). The latent s tructure in the heuris tic set might help evaluators to form more struc-tured approach during evaluation. They may focus on detecting us ability problems from each us ability as pect at a time. The s HEM (Kurosu et al. 1997, 1999) method is a good example for providing structured heuristics. By dividing evaluation into sub-session, the method enables the evaluators to focus on is s ues in one us ability s ub-category in each s ub-s es s ion at a time. Therefore, a larger body of usability aspects can be considered in a non-threatening way.

(2) Provide structure with problem areas

This approach tells the evaluator the mos t important problem areas to look at. For example, the HE-plus method provides evaluators with a us ability problem profile containing problem area (Chattratichart, Brodie 2002, 2004). This makes s ure that evaluators will examine the important prob-lem areas. Hence, comprehensive evaluation results can be produced.

(3) Provide structure with evaluation procedure

This approach tells the evaluators s pe-cifically what to do during the evaluation. For example, the perspective-based method has s tipulated exact tas ks to do and ques-tions to answer with each perspective. The abs tract tas ks us ed in SUE (Matera et al. 2002; Angeli et al. 2003) also provide very stringent structure by stipulating the system elements to examine, and the related ques-tions to answer regarding each elements.

(4) Provide structure with tasks

Six out of the seven extended HE meth-ods lis ted in table 2 provide typical tas k

ergo2005_3.indd 1912005-12-20 12:39:16

192Extension of heuristic evaluation method: a review and reappraisal

s cenarios to evaluators. For example, the heuristic walkthrough method introduced a task-based evaluation pass to make sure ele-ments related to important tasks are evalu-ated (Sears 1997). T asks give the evaluators better focus and unders tanding of the s ys-tem, which, in turn, improve the evaluation result.

The heuristic evaluation is a subjective process (Doubleday et al. 1997). In addition to the checklis t of general heuris tics to be considered, the evaluator is also allowed to cons ider any additional us ability principles or results that come to mind which may be relevant for any s pecific interface element (Nielsen 1993). Thus, the HE method some-times serves as a method for inspiring evalua-tors (Cockton, Lavery, Woolrych 2003). But the inspiration that the evaluators get from the HE method is a bit limited. Another way that has proved to be able to improve the evaluation result is to inspire evaluators with different perspectives. This is like ask-ing the evaluators to put themselves into the s hoes of other type of us ers. It is believed that this approach can enlarge the discovery scope by providing incentives for the evalu-ators to think actively and broadly. It tries to fully tap the cognitive capacity of the evalu-ators by keeping their eyes wide open and mind actively running during the evaluation. This approach has been us ed by many HE extens ion methods. In two evaluative s ub-sessions of the sHEM method (Kurosu et al. 1997, 1998), evaluators are as ked to think whether different users (novice, expert and us ers with s pecial care) will have problems us ing the s ys tem. Similarly, two pers pec-tives us ed in the pers pective bas ed heuris-tic evaluation (Zhang et al. 1999) include the novice us e and expert us e. Evaluators think from the pers pective of novice and expert users and follow the evaluative pro-cedure to detect possible usability problems for them. In the PAVE method (Des urvire, Thomas 1993), the evaluators need to think from ten different pers pectives while find-ing us ability problems, from the mos t con-ventional views like human factor expert to most unusual view of a worried mother and a s poiled child. The metaphor of thinking method (Hornb?k, Fr??kj?r 2004) al o try to guide the evaluator’s mental thinking with five metaphors.

In the HE method, the evaluators in-s pect every interface element agains t a lis t of heuristics. The result of applying the HE method is a system conforming to all of the heuristics if all identified problems are fixed. Therefore, the quality of the heuris tics is directly related to the quality of evaluation res ults. If the heuris tics does n’t cover all important as pects of the computing s ys tem us ability, then the res ultant s ys tem may not be good enough even if the evaluation went well. The lis t of heuris tics s hould be devel-oped carefully. Niels en and Molich (1990) recommended that a good s et of heuris tics should be small (e.g., around 10) in number, s o that the ins pectors could have an eas y time remembering, and being reminded of the heuris tics while they are detecting us-ability problem. Nielsen’s heuristic set has a manageable size of 10. It was speculated that Nielsen limit the number of heuristics to fit the limitation of human memory with the chunk size of 7 plus or minus 2 (Kurosu et al. 1997). Though larger number of heuris tics may addres s more us ability problems, they may not be as easily maintained in the evalu-ator’ working memory. Therefore, when developing domain specific heuristic set, the total number should not be too large.

Many heuristic sets listed in table 1 are still under revision. We can notice that dif-ferent heuristic sets were developed for the same domain of e-learning websites (Reeves et al. 2002; Evans, Sabry 2003). The reason is that they use different theoretical basis for heuris tic s et development. With more ap-plication cas e s tudies and experiments, we would expect that the disparate sets can be merged into a comprehensive set. Heuristic set should be refined iteratively until a vali-dated heuristic set emerge for each domain. Practitioners can then choose these domain

ergo2005_3.indd 1922005-12-20 12:39:17

193

C. Ling, G. Salvendy

s pecific heuris tics as tools to evaluate the corresponding systems.

As the usability inspection method such

as the heuristic evaluation develop and ma-ture, we can understand better the facts that

affect the quality of evaluation res ults , and can design better methods to achieve result with high thoroughnes s , validity and reli-ability.

7. Future research

This review s tudy s hows the plethora of research devoted to optimize the usability inspection method: HE method. With better supported evaluation procedure, and better fitted heuristic set, the HE and its extensions s hould be more ready to s erve as effective discount method. The future research direc-tion related to HE method can be s umma-rized in the following aspects:

(1) Extens ion methods to HE need to be applied to various computing systems to test their applicability and generalizability.

(2) More res earch is needed to com-pare the effectivenes s among the different extension method.

(3) Get better under s tanding of the differences between the extended methods and the traditional HE method in terms of evaluator’s cognition process.

(4) Study the difference between the expert and novice evaluators while using the HE method and extended HE methods.

(5) Us ability ins pection methods need to be applied to more domains to improve their usability. More domain-specific heuris-tics need be developed and refined to help give precise and relevant evaluation results.

(6) More extended method can be de-veloped to further improve the HE meth-od. A typical HE procedure includes four phas es : pre-evaluation training, the actual evaluation, debriefing s es s ion, and a s ever-ity rating phase (Nielsen 1994b). Currently, mos t extended methods aimed to improve the actual evaluation phase of the HE meth-od. More improvement may be achieved by improving more or all phas es of the HE method.

(7) More s oftware tools can be devel-oped to assist the heuristic evaluation proc-ess.

(8) With the pre ence of data from many empirical studies, some kind of meta-analys is can be performed to better deter-mine the factors that play important roles in detecting usability problems.

References

Agarwal R, Venkatesh V , 2002, Assessing a firm’s web presence: a heuristic evaluation procedure for

the measurement of usability. Information Systems Research , 13, 168–186.Angeli AD, Matera M, Costabile MF, Garzotto F, Paolini P , 2003, On the advantages of a system-atic ins pection for evaluating hypermedia us ability. International J ournal of Human-Computer

Interaction , 15, 3, 315–335.Avouris NM., T selios N, T atakis EC, 2001, Development and evaluation of a computer-based labora-tory teaching tool. Computer Applications in Engineering Education, 9, 1, 8–19.Baker K, Greenberg S, Gutwin C, 2001, Heuristic evaluation of groupware based on the mechanics

of collaboration. Proceedings of the 8th IFIP Working conference on Engineering for Human-Computer Interaction , May 11–13, T oronto, Canada.Baker K, Greenberg S, Gutwin C, 2002, Empirical development of a heuristic evaluation methodol-ogy for s hared works pace groupware. Proceedings of the 2002 ACM conference on computer supported cooperative work , November 16–20, New Orleans, Louisiana.Bendoly L, Goldman KH, 2003, Investigating heuristic evaluation: a case study. Proceedings of the

Museums and the Web 2003. March 19–22, Charlotte, North Carolina.

ergo2005_3.indd 1932005-12-20 12:39:17

194Extension of heuristic evaluation method: a review and reappraisal

Berry B, 2003, Adapting heuristics for notification systems. Proceedings of 41st Annual ACM South-east Conference, Savannah, GA, 144–149.

Blandford A, Keith S, Connell I, Edwards H, 2004, Analytical usability evaluation of digital librar-ies: a case study. Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, June

7–11, T uscon, Arizona, USA.

Chattratichart J, Brodie J, 2002, Extending the heuristic evaluation method through contextualiza-tion. Proceedings of the 46th Annual Meeting of the Human Factors and Ergonomics Society,

September 30–October 4, Baltimore, MD.

Chattratichart J, Brodie J, 2004, Applying user testing data to UEM performance metrics. Proceed-ings of the CHI 2004 Conference on Human Factors in Computing Systems, April 24-29, Vienna,

Austria, 1119–1122.

Cockton G, Woolrych A, 2001, Unders tanding ins pection methods: les s ons from an as s es s ment of heuristic evaluation. [in:] Blandford AJ, Vanderdonckt J (eds.) People & Computers XV, Springer-

Verlag, 171–192.

Cockton G, Woolrych A, 2002, Sales must end: should discount methods be cleared off HCI’s shelves?

Interactions, September + October, 13–18.

Cockton G, Lavery D, Woolrych A, 2003, Inspection-based evaluation. [in:] Jacko JA and Sears A (eds.) The Human-Computer Interaction Handbook. Lawrence Erlbaum As s ociates, Mahwah,

New Jersey, 1118–1138.

Cockton G, Woolrych A, Hall L, Hindmarch M, 2003, Changing analys t’s tunes: the s urpris ing impact of a new instrument for usability inspection method assessment. [in:] Staphanidis C and

Jacko J (eds.) Proceedings of HCI 03 Conference, June 22–27, Crete, Lawrence Erlbaum Associ-

ates, 145–161.

Connell IW, Hammond NV, 1999, Comparing usability evaluation principles with heuristics: problem instances vs. problem types. [in:] Sasse MA and Johnson C (eds.) Proceedings of the IFIP INTER-

ACT ’99 Conference on Human-Computer Interaction, Amsterdam: IOS, 621–629.

Desurvire H, Caplan M, T oth JA, 2004, Using heuristics to evaluate the playability of games. Pro-ceedings of the CHI 2004 Conference on Human Factors in Computing Systems, April 24–29,

Vienna, Austria.

Desurvire H, Thomas JC, 1993, Enhancing the performance of interface evaluators using non-em-pirical us ability methods. Proceedings of the 37th Annual Meeting of the Human Factors and

Ergonomics Society, October11–15, Seattle, Washington.

Doubleday A, Ryan A, Springett M, Sutcliffe A, 1997, A comparison of usability techniques for evalu-ating design. Proceedings of Designing Interactive Systems (DIS’97): Processes, Practices, Methods,

and Techniques, Amsterdam, the Netherlands, ACM Press, 101–110.

Drury J, 2001, Developing heuristics for synchronous collaborative systems. Proceedings of the CHI 2001 Conference on Human Factors in Computing Systems. March 31–April 5, Seattle, Wash-

ington.

Evans C, Sabry K, 2003, Evaluating of the interactivity of web-based learning systems: principles and process. Innovations in Education and Teaching International, 40, 1, 89–99.

Fakun D, Greenough RM, 2002, Us er-interface des ign heuris tics for developing us able indus trial hypermedia applications. Human Factors and Ergonomics in Manufacturing, 12, 2, 127–149.

Fernandes G, Holmes C, 2002, Applying HCI to mus ic related hardware. Proceedings of the CHI 2002 Conference on Human Factors in Computing Systems, April 20–25, Minneapolis, Min-

nesota.

ergo2005_3.indd 1942005-12-20 12:39:17

195

C. Ling, G. Salvendy

Fu L, Salvendy G, T urley L, 2002, Effectiveness of user testing and heuristic evaluation as a function of performance classification. Behaviour & Information Technology, 21, 2, 137–143.

Garzotto F, Mainetti L, Paolini P, 1995, Hypermedia des ign, analys is, and evaluation is s ues. Com-munication of ACM, 38, 74–86.

Garzotto F, Matera M, Paolini P, 1998, Model-bas ed heuris tic evaluation of hypermedia us ability.

Proceedings of the Working Conference on Advanced Visual Interfaces, May 24–27, L’Aquila, Italy.

Greenberg S, Fitzpatrick G, Gutwin C, Kaplan S, 1999, Adapting the Locales framework for heuristic evaluation of groupware. Proceedings of OZCHI’99, 28–30.

Gutwin CG, 2000, The mechanics of collaboration: developing low cost usability evaluation methods for shared workspaces. Proceedings of the IEEE 9th International Workshops on Enabling Technolo-

gies: Infrastructure for Collaborative Enterprises.March 14–16, 2000. Gaithersburg, Maryland.

Hertzum M, Jacobsen NE, 2001, The evaluator effect: a chilling fact about usability evaluation meth-ods. International Journal of Human-Computer Interaction, 13, 4, 421–443.

Hornb?k K, Fr??kj?r E, 2004, Usability inspection by metaphors of human thinking compared to heuristic evaluation. International Journal of Human-Computer Interaction, 17, 3, 357–374.

Jeffries R, Wharton C, Uyeda KM, 1991, User Interface evaluation in the real world: a comparison of four techniques. Proceedings of CHI’91 Conference on Human Factors in Computing Systems,

April 27–May 2, New Orleans, LA.

Johnson D, Wiles J, 2003, Effective affective user interface design in games. Ergonomics, 46, 13-14, 1332–1345.

Kamper RJ, 2002, Extending the usability of heuristics for design and evaluation: lead, follow, and get out of the way. International Journal of Human Computer Interaction, 14, 3-4, 447–462.

Kline RS, Javahery H, Donayee M, Rilling J, 2002, Quantifying developer experiences via heuristic and psychometric evaluation. Proceedings of the IEEE 2002 Symposia on Human Centric Com-

puting Languages and Environments, September 03–06, Arlington, Virginia

K?ykk? M, Ollikainen R, Ranta-aho M, 1999, Usability heuristic guidelines for 3D multi-user worlds.

Proceedings of the 1999 Conference on Computer Human Interaction Special Interest Group of

the Ergonomics Society of Australia, November 28–30, Wagga Wagga, Australia, 52–57.

Kurosu M, Matsuura S, Sugizaki M, 1997, Categorical inspection method-structured heuristic evalu-ation (sHE). Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics

“Computational Cybernetics and Simulation”, October 12–15, Orlando, Florida.

Kurosu M, Sugizaki M, Matsuura S, 1998, Structured heuristic evaluation (sHEM): A more produc-tive method to find out usability problems. Proceedings of the Usability Professionals’ Association

Conference, June22–26, Washington DC.

Kurosu M, Sugizaki M, Matsuura S, 1999, A comparative study of sHEM (structured heuristic evalu-ation method). [in:] Bullinger HJ, Ziegler J (eds.) Human-computer interaction: Ergonomics and

user interfaces. Proceedings of HCI International 99, Lawrence Erbaum Associates, Mahwah, NJ,

1, 938–942.

Lathan CE, Sebrechts MM, Newman DJ, Doarn CR, 1999, Heuristic evaluation a Web-based inter-face for Internet T elemedicine. Telemedicine Journal, 5, 2, 177–185.

Law LC, Hvannberg ET, 2002, Complementarity and convergence of heuristic evaluation and usabil-ity test: a case study of UNIVERSAL brokerage platform. Proceedings of the 2nd Nordic Confer-

ence on Human-Computer Interaction, October 19–23, ?rhus, Denmark.

ergo2005_3.indd 1952005-12-20 12:39:17

196Extension of heuristic evaluation method: a review and reappraisal

Levi MD, Conrad, FG, 1996, A heuris tic evaluation of a world wide web prototype. Interactions, July+August, 51–60.

Lewis J, 2006, Usability testing. [in:] Salvendy G. (eds.) Handbook of Human Factors and Ergonomics, 3rd Edition, John Wiley & Sons, New Y ork, NY.

Lowry PB, Roberts TL, 2003, Improving the us ability evaluation technique, heuris tic evaluation, through the us e of collaborative s oftware. Proceedings of the 9th Annual Americas Conference

on Information Systems (AMCIS), August 4–5, T ampa, 2203–2211.

Mack RL, Montaniz F, 1994, Observing, predicting, and analyzing usability problems. [in:] Nielsen J and Mack RL (eds.) Usability Inspection Methods. John Wiley & Sons, New Y ork, NY, 295–

339.

Mack RL, Nielsen J, 1994, Executive summary. [in:] Nielsen J and Mack RL (eds.) Usability Inspec-tion Methods. John Wiley & Sons, New Y ork, NY, 1–23.

Mankoff J, Dey AK, Hsieh G, Kientz J, Lederer S, Ames M, 2003, Heuristic evaluation of ambient displays. Proceedings of the CHI 2003 Conference on Human Factors in Computing Systems,

April 5–10, Ft. Lauderdale, Florida.

Mariage C, Vanderdonckt J, 2000, A Comparative usability study of electronic newspapers. P roceed-ings of International Workshop on Tools for Working with Guidelines, October 7-8, Biarritz,

Springer-Verlag, London, 325–337.

Matera M, Costabile MF, Garzotto F, Paolini P, 2002, SUE inspection: An effective method for sys-tematic usability evaluation of hypermedia. IEEE T ransactions on Systems, Man, and Cybernet-

ics-Part A: Systems and Humans, 32, 1, 93–103.

McGrow K, Brennan AH, Preece J, 2004, Development of a tool for heuristic evaluation of health-care information systems. Computers, Informatics, Nursing, Journal of Hospice & Palliative Nurs-

ing, in press.

Miller JR, Jeffries R, 1992, Interface-usability evaluation: science of trade-offs. IEEE Software, 9, 5, 97–98, 102.

Mills S, 1995, Usability problems of acoustical fishing displays. Displays, 16, 3, 115–121.

Molich R, Niels en J, 1990, Improving human-computer dialog. Communications of the ACM, 33, 3, 338–348.

Muller MJ, Mathes on L, Page C, Gallup R, 1998, Participatory heuris tic evaluation. Interactions, September+October, 13–18.

Nielsen J, 1992, Finding usability problems through heuristic evaluation. Proceedings of CHI’92 Con-ference on Human Factors in Computing Systems, May 3–7, Monterey, California, 373–380.

Nielsen J, 1993, Usability Engineering. Boston: Academic Press.

Nielsen J, 1994a, Enhancing the explanatory power of usability heuristics. Proceedings of the CHI’94 Conference on Human Factors in Computing Systems “Celebrating interdependence”, April 24–

28, 1994, Boston, Massachusetts.

Nielsen J, 1994b, Heuristic evaluation. [in:] Nielsen J, Mack RL (eds.) Usability Inspection Methods.

John Wiley & Sons, New Y ork, NY, 24–62.

Nielsen J, Landauer TK, 1993, A mathematical model of the finding of usability problems. Proceed-ings of the InterCHI Conference on Human Factors in Computing Systems, April 24–29, Amster-

dam, The Netherlands.

Nielsen J, Mack RL, 1994, Usability Inspection Methods. John Wiley & Sons, New Y ork, NY.

ergo2005_3.indd 1962005-12-20 12:39:17

197

C. Ling, G. Salvendy

Nielsen J, Molich R, 1990, Heuristic evaluation of user interfaces. Proceedings of CHI’90 Conference on Human Factors in Computing Systems, April 1–5, Seattle, Washington, 249–256.

Niels en J, Phillips VL, 1993, Es timating the relative us ability of two interfaces: heuris tic, formal, and empirical methods compared. Proceedings of the InterCHI Conference on Human Factors in

Computing Systems, April 24–29, Amsterdam, The Netherlands.

Norman DA, 1988, The Design of Everyday Things. Basic Books, New Y ork.

Paddison C, Englefield P, 2003, Applying heuristics to perform a rigorous accessibility inspection in

a commercial context. Proceedings of the CUU’03, November 10–11, Vancouver, Britis h Co-

lumbia, Canada.

Rau P, Liang SF, 2003, Internationalization and localization: evaluating and testing a website for Asian users. Ergonomics, 46, 1-3, 255–270.

Reeves TC, Benson L, Elliott D, Grant M, Holschuh D, Kim B, Kim H, Lauber E, & Loh S, 2002, Usability and instructional design heuristics for E-Learning evaluation, Proceedings of World Con-

ference on Educational Multimedia, Hypermedia & Telecommunications, Denver, Colorado.

Rosenbaum S, Rohn JA, Humburg J, 2000, A toolkit for strategic usability: results from workshops, panels, and surveys. [in:] Little R, Nigay L (eds.) Proceedings of ACM CHI 2000 Conference on

Human Factors in Computing Systems, April 1–6, The Hague, The Netherlands, ACM pres s,

New Y ork, 337–344.

Sears A, 1997, Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9, 3, 213–234.

Shneiderman B, 1998, Designing the user interface. Strategies for effective human-computer interac-tion. (3rd Edition). Addison-Wesley, Reading, MA.

Slavkovic A, Cross K, 1999, Novice heuristic evaluations of a complex interface. Proceedings of the CHI’99 Conference on Human Factors in Computing Systems, May 15–20, Pittsburgh, Pennsylvania.

Somervell J, Wahid S, McCrickard DS, 2003, Usability heuristics for large screen information exhib-its. Proceedings of the Ninth IFIP TC13 International Conference on Human Computer Interac-

tion, Zurich, Switzerland, 904–907.

Squires D, Preece J, 1999, Predicting quality in educational software: Evaluating for learning, usabil-ity and the synergy between them. Interacting with Computers, 11, 467–483.

Sutcliffe A, 2001, Heuristic evaluation of website attractiveness and usability. Proceedings of the 8th International Workshop on Interactive Systems: Design, Specification, and Verification, Springer-

Verlag, London, UK, 183–198

Vetere F, Howard S, Pedell S, Balbo S, 2003, Walking through mobile us e: Novice heuris tic and their application. Proceedings of OzCHI2003: New directions in interaction, information environ-

ments, media and technology, November 26–28, Brisbane, Australia.

Warren P, 2004, Learning to program: spreadsheets, scripting and HCI. Proceedings of the Sixth Aus-tralasian Computing Education Conference, Dunedin, New Zealand.

Zhang J, Johns on TR, Patel VL, Paige DL, Kubos e T, 2003, Us ing us ability heuris tics to evaluate patient safety of medical devices. Journal of Biomedical Informatics, 36, 1/2, 23–30.

Zhang Z, Basili V, Shneiderman B, 1999, Perspective-based usability inspection: An empirical valida-tion of efficacy. Empirical Software Engineering, 4, 1, 43–69.

ergo2005_3.indd 1972005-12-20 12:39:18

相关主题