Paired comparisons, sometimes called pairwise comparisons, is a hierarchical ranking scale assessment method or test instrument which enables design practitioners to gauge the success of their design solutions against other similar products within the target market. The technique presents each participant with every possible pair of similar components taken from a set of design solutions and then requires them to state a preference for one item in each pair. This measurement method or instrument was first proposed by Thurstone. [5]
Böckenholt [6] highlights three benefits of using this test method/instrument:
- It imposes minimal constraints on the judge (participant), especially when differences between items are small, and is less prone to the influence of context;
- Internal consistency checks are available that identify judges (participants) who discriminate (choose) poorly; and,
- It provides rich data about the effects of individual differences and perceived similarity relationships among items.
Böckenholt also highlights that there are drawbacks in using this method to test products. When multiple paired comparisons are undertaken by each judge (participant) the data may contain not only variations between each participant, but also variations at each moment during the test of all the items.
Variation or variability is a common issue within quantitative (non-parametric) testing; unidirectional variability arising from a confounding variable is called bias. Greer and Mulhern [7] define four generic causes of variability that may be found in the application of the paired comparison method/instrument:
- Sample variation
- Individual variation
- Situation variation
- Measurement variation
The structure of a paired comparison should include:
- Participant information sheet, why the survey is being done, what will happen to the information given, contact details if further questions are asked or complaints made.
- Screening protocol (to ensure correct participants are being surveyed)
- Consent form for the participant, or their advocate, to sign;
- Keep samples out of sight of the participant, unless being compared (to avoid bias);
- ‘round robin’ comparison sheet to ensure each two samples are shown in the correct order, until all samples have been compared with each other;
- Rotate the order of the samples presented to the next participant to avoid bias (Torrens and Smith, 2013);
- Ask each participant: ‘taking first impressions only..’ choose one of the two samples.
- Comments made as the participant chooses can be recorded for cross-referencing of results;
Take advice from a statistician about which non-parametric method to use to process your data (which will be transferred from surveys or inputted directly into an electronic spreadsheet;
Get a statistician to review and help you interpret the results.
The outcomes of a paired comparison can provide a priority order of samples. When combined with qualitative comments about the reasons behind their choice, the approach can provide strong evidence for the effectiveness of a particular new AT product with clinicians and healthcare professionals.
Disadvantages are that a larger sample, (a minimum of fifty participants), are needed and to ensure reliable results, a statistician is required.
Useful links
Böckenholt U. 2002. A Thurstonian analysis of preference change. Journal of Mathematical Psychology; 46, pp300–314
Creswell JW., 2009. Research design. Qualitative, quantitative, and mixed methods approaches. 3rd ed. Sage, London.
Greer B, Mulhern G. 2002. Making sense of data and statistics in psychology. Palgrave, Baskingstoke.
Siegel S, Castellan NJ. 1988. Nonparametric statistics. McGraw-Hill, London, pp 174-183.
Sinclair, M. 1999. Subjective assessment. In: J.R. Wilson & E.N. Corlett (eds). Evaluation of human work, a practical ergonomics methodology. 2nd ed. London: Taylor & Francis, pp78-83.
Thurstone, Thurstone LL. 1927. A law of comparative judgment. Psychological Review; 34, pp273–286.
Torrens G, McDonagh-Philp D, Newman A. 2001. Getting a grip, Ergonomics in Design: The quarterly of Human Factors Applications; 9, (2). pp7-13
Torrens, G.E. and Smith, N.C.S., 2013. Evaluation of an assistive technology product design using a paired comparisons method within a mixed methods approach: A case study evaluating preferences for four types of cutlery with 34 upper limb impaired participants. Disability and Rehabilitation: Assistive Technology, 8 (4), pp 340 – 347. Available at: (https://hdl.handle.net/2134/14862), Accessed: [21/01/2021]