How to Conduct Heuristic Evaluation?

The usability testing is one of the essential stages of the user experience design process where the prototype is tested and evaluated to ensure that the final product is user-friendly in terms of usability factors such as functionality, user control, consistency, and error prevention. While there are different usability testing methods that could be used in the testing and evaluation. One of the methods that have been part of a large debate is the heuristic evaluation. While many companies favor this tool because of its cheap cost and affordability comparing with other testing methods, heuristic evaluation is suitable as an evaluation tool that need to conducted by a team of experts.

Unlike the other usability testing methods that depend on a user sample to test the product, the heuristic evaluation depends on UX Design experts (evaluators) who review the product based on predefined characteristics. In this method, the evaluator checks the design based on an assessment document that evaluates different elements such as the layout elements, function, flow, and system. The aim of heuristic evaluation is to identify the common usability problems in the design.

Related articles:

What is Heuristic Evaluation in Usability Testing?

The heuristic evaluation was first introduced by Jakob Nielsen and Rolf Molich in 1990. In their method, they suggested a number of guidelines that can be tested by a group of experts to evaluate the usability of the product. These guidelines act like a rule of thumbs which can identify the common usability problems in the design. There are different guidelines that can be implemented in the heuristic evaluation such as the following (Figure 1):

  • Visibility of system status
  • Match between system and the real world
  • User control and freedom
  • Consistency and standards
  • Error prevention
  • Recognition rather than recall
  • Flexibility and efficiency of use
  • Aesthetic and minimalist design
  • Help users recognize, diagnose and recover from errors
  • Help and documentation
Heuristic Evaluation Factors
Figure 1. Jackob’s Heuristic Evaluation Factors (Source: NN Group) 

In the heuristic evaluation, a number of UX evaluators test the product based on the above guidelines to uncover usability problems during the testing process. The aim of the heuristic evaluation is to identify the common usability issues rather than the user-specific problems as it doesn’t include the user sample in the testing process, which is the main reason behind the debate of the viability of the tool to use in the usability testing process.

The Advantages of Using the Heuristic Evaluation

Companies may depend on the heuristic evaluation rather than another usability testing because of the following advantages:

  • It is an inexpensive usability testing methods that can test the product based on number of in-house UX experts
  • It is a quick testing tool as it doesn’t require to prepare a representative user sample to do the testing
  • It can be used to element the common usability problems that don’t need a feedback from the end user
  • It can be used prior to other usability testing methods to focus on the user-specific usability issues

The Disadvantage of Using the Heuristic Evaluation

In contrast to the above advantages, the heuristic evaluation comes with different disadvantages that may affect its viability and a trusted tool to use in the usability testing. These disadvantages may include the following:

  • It doesn’t involve the user opinion in the testing. As the UX experts who do the testing are not the same as the end userIt should depend in more than one UX expert evaluators in order to ensure accurate results for the testing
  • It should depend on more than one UX expert evaluators in order to ensure accurate results for the testingIt may be difficult to find the experts who can do the evaluation in-house. Also, it may be expensive to hire external evaluators
  • It may be difficult to find the experts who can do the evaluation in-house. Also, it may be expensive to hire external evaluatorsIt only works to identify general usability testing issues. Some issues can be identified as a problem when it is not actually a problem for the end users
  • It only works to identify general usability testing issues. Some issues can be identified as a problem when it is not actually a problem for the end users
  • It can’t be used alone to end up with accurate usability testing, other methods with user involvement should be used as well

How to Conduct Heuristic Evaluation in Usability Testing?

Although there is a long debate about using the heuristic evaluation in usability testing, it can still use in an early stage of the testing phase before depending on other testing methods that involve the user sample. Using the heuristic evaluation can reduce the cost of the usability testing as well as the tested factors. As in-house UX evaluators can start by testing the product based on the guidelines highlighted above, then the company may use other external usability testing methods that involve the end user in the testing process.

In order the to implement the heuristic evaluators, the company need to prepare the following:

  • Between 3-5 UX evaluators
  • The heuristic guidelines
  • The user tasks that need to be tested
  • The product prototype or wireframe that needs to be tested
  • A form to collect testing results

And to ensure a successful heuristic evaluation process, in particular, and a usability testing process in general, the company need to consider the following:

  • A good knowledge about the user behavior, needs, and demographics in order to consider while doing the evaluation
  • Clear definition of the user tasks and use cases that will be evaluated by the heuristic evaluation
  • Depend on more than one evaluator to ensure accurate output for the usability testing. If one issue missed by one evaluator, it can be highlighted in the other evaluators’ reports
  • Once the evaluators test the product and collect the results, Each evaluator analyze the results and present it in a slide presentation to discuss with the rest of the evaluators and stakeholders to reach final reports

In this example, we will explore how to apply Heuristic Evaluation on BBC Bitesize homepage and see how the ten criteria applied to the webpage.

The visibility of the system status refers to keep the user informed about the system, the current status or if there are any changes. The alerts at the top of the page provides updated information for the user. The Picks of the week as also informing the user about if there is added resources (Figure 2).

system status visibility
Figure 2 Visibility of system status.

Match between system and the real world criterium is represented in the icons and symbols. The parents section is represented by the backpack and lunch. The teachers section is represented with the illustrations of the notebook and coffee (Figure 3).

Match system to real world
Figure 3. Match between system and the real world

The consistency and standards is represented in the whole design elements and structure. The links related to the whole BBC site is allocated in the top menu, while the links relate to the education website is located under the logo. The layout of each learning section (i.e. the Learn & Revise, Extra resources and KS3 Biology) is having the same design style (Figure 4).

Consistency and standards
Figure 4. Consistency and standards

The user control and freedom factor allows the user to easily identify where they are and be able to return the correct path. This factor is represented in the top menu which allows the user to navigate between different sections in the website (Figure 5).

User control and freedom
Figure 5. User control and freedom

The Error prevention factor may not applicable in this page as error isn’t likely to happen.

Recognition rather than recall is applied in using different colours to differentiate the content such as the primary, secondary and post-16 content. Also, the use of background colours to separate between sections such as KS3 Biology and The Regenerators (Figure 6).

Recognition rather than recall
Figure 6. Recognition rather than recall

Flexibility and efficiency of use is reflected in the All Bitesize link which allows users to directly find links instead of using the links in each page. If the users are aware of the content of the website, they can simply jump to the content needed through this sitemap page (All Bitesize) (Figure 7).

Flexibility and efficiency of use
Figure 7. Flexibility and efficiency of use

Aesthetic and minimalist design appears on the usage of layout, colours, fonts and icons. The design layout is clear and easy to understand and navigate. The colours are used to separate between sections and identify each part of the website (Figure 4).

The next criterium is helping users to recognize, diagnose, and recover from errors. This applies in having the navigation system available for users to easily return to the main page (Figure 5).

The final criterium is Help and documentation. The system provides help for the users to know how they can use it (i.e. hoe to use the Bitesize website to learn at home) (Figure 8).

Help and documentation
Figure 8. Help and documentation

The above quick overview provides a brief about how to identify each criteria of the heuristic evaluation to assess a system. Evaluators can rate each of the above factors based on a scale from one to ten. You can use a heuristic evaluation template, which is available in the article URL in the video description. Finally, you can download our free Heuristic Evaluation Template below.

In addition to the above guidelines that need to be considered before doing the heuristic evaluation, other factors may depend on each project characteristics, For example, if the product requires specific skills or experience to test it. then, the company may consider reviewing the evaluator resume to ensure they have the proper experience required to test the product prototype.

Although there is still a debate on the validity of using the heuristic evaluation method in the usability testing phase, it can be used in an early stage of the testing phase before moving with other user testing methods to reduce the external testing costs and reduce the tested features. The heuristic evaluation tool can be very tricky especially with its end results. Therefore, companies need to consider the above guidelines in order to ensure the accuracy of the results of the tool.

Bibliography 

Nielsen, J. (2020). 10 Usability Heuristics for User Interface Design. Access URL: https://www.nngroup.com/articles/ten-usability-heuristics/#poster. [Accessed on: 12/2021]

Nielsen, J. (1994). How to Conduct a Heuristic Evaluation. Access URL: https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/. [Accessed on: 12/2021]

Nielsen, J., & Molich, R. (1990, March). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 249-256).

Tan, W. S., Liu, D., & Bishu, R. (2009). Web evaluation: Heuristic evaluation vs. user testing. International Journal of Industrial Ergonomics, 39(4), 621-627.

What is Heuristic Evaluation?

Heuristic Evaluation is a tool used for usability testing and evaluation. It allows experts to evaluate websites and mobile applications based on ten generic criteria. These criteria ensure that the website is user friendly for the end-user. Nilson (1994) defined 10 criteria, including:
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose and recover from errors
Help and documentation

How to Conduct Heuristic Evaluation?

  1. Define the aim of the evaluation session

    Identify the website or the mobile application that will be evaluated and the target part to evaluate.

  2. Assign the evaluators

    Create a team of evaluators and introduce the task for them.

  3. 1st evaluation session

    In the first session, the evaluators explore the website without evaluating it to get familiar with it and its structure.

  4. 2nd evaluation session

    In this session, the team evaluate the website based on Nelison’s 10 criteria or they can set a customised criteria if needed. Each member rank the application of each criteria.

  5. Final evaluation

    The team collect the answers and calculate the average which will be used to identify the final recommendations.


Dr Rafiq Elmansy

Academic lecturer, researchers, and author. I have eight years of academic teaching experience for undergraduate and postgraduate modules, including digital design, interactive design, design thinking, and design process. My research interest involves design theory and practice including design process and the role of design in healthcare technology. I'm the inventor of the Adherence Canvas, an evidence-based design tool to improve patient adherence to health tech. Additionally, I am a published author in industry books in the fields of design and technology. I am the founder of Designorate.com. I am a fellow and mentor for the Higher Education Academy (HEA) and the Royal Society of Arts (FRSA), an accredited lecturer from the British Charter Institute of IT (BSC), and an Adobe Education Leader. My industry experience involves 15 years of working with international clients.

Leave a Reply

Your email address will not be published. Required fields are marked *