How to Ensure Fairness and Equality in Your Pre-Employment Testing

How to Ensure Fairness and Equality in Your Pre-Employment Testing

One of the biggest concerns preventing organisations from exploring pre-employment testing is its potential impact on diversity and inclusion. This is ironic, given the research on résumé sifting and interviews regarding discrimination, which shows them to be significantly more vulnerable to bias than any pre-employment test. In lieu of pre-employment testing, employers often rely exclusively on résumé sifting and interviews to select candidates, introducing tremendous bias into the process. Nevertheless, many employers are simply unaware of the work done in this area and are consequently resistant to changes in their selection processes.

In this article, I will outline how organisations can showcase the fairness of pre-employment tests, making them more palatable to nervous stakeholders.

Review an Adverse Impact Analysis

One of the major advantages of pre-employment testing over interviews and résumé sifting is the ability to thoroughly research these tests. Because psychometric tests are standardised products, psychologists can conduct “adverse impact” analyses across different protected groups, identifying differences in scores between them. For example, if a provider wants to demonstrate fairness based on sex, they could trial the assessment on a group of males and females and then compare the differences. If the differences are small or non-existent, this suggests the assessment does not cause adverse impact on the basis of sex.

Typically, test publishers conduct adverse impact analyses during the initial development of the assessment and before its launch. This reassures early adopters regarding the assessment's level of fairness while also demonstrating a commitment to the principles of fairness and equality. Publishers normally conduct adverse impact analyses on several protected groups, ensuring fairness across different variables of interest. To standardise differences between groups, psychologists typically rely on the “Cohen’s d” statistic to represent standardised effect sizes.

Adverse impact analyses can also be performed for specific organisations using their live data. This can be particularly reassuring to organisations with concerns, providing hard data regarding the assessment’s fairness. This analysis could be conducted using candidate data collected during screening or the employing organisation's existing employees. In either case, local data is always more convincing than general data, and a thorough and well-sampled adverse impact analysis will ease the concerns of even the most sceptical decision-makers.

Explore Differential Item Functioning

A step beyond adverse impact analyses is differential item functioning analysis, which represents a more powerful and scientific approach to assessing fairness. One limitation of adverse impact analysis is its inability to differentiate between genuine differences between groups and unfair bias. For example, comparing candidates with PhDs in economics to those without qualifications on a test measuring economic knowledge would reveal significant differences in an adverse impact analysis, even though these differences are expected and valid. Similarly, adverse impact analysis could theoretically mask bias in situations where groups are expected to score differently but do not.

Differential item functioning, however, examines individual questions within the test and identifies bias at the question level. This analysis involves controlling for candidates' overall ability and investigating whether a question seems disproportionately difficult for high-performing candidates from a particular group. For example, if an assessment is written almost entirely in English but contains one question in French, that question would be flagged for bias via differential item functioning when trialled on monolingual English speakers.

While adverse impact analyses can be conducted directly by a client organisation, differential item functioning can only be conducted by the test publisher. Question-level data is typically not shared with employing organisations, making this analysis impossible to conduct without the provider's involvement. However, these analyses are conducted during test development and should be available for review in the assessment's technical manual. These considerations are particularly relevant in the context of recruitment tests for diversity and inclusion, where fairness and unbiased assessment are critical to achieving equitable outcomes.

Request Insight from Inclusion Experts

Beyond psychometric evaluations, test content can also be reviewed qualitatively by experts in cultural sensitivity. Certain content may cause offence or upset without affecting scores, thus flying under the radar of psychometricians. For example, questions could reference concepts, terminology, or situations that unintentionally cause offence to specific groups. Psychometricians are primarily statisticians and psychologists, not experts in cultural sensitivity.

Hiring a consultant to advise on cultural sensitivity can provide significant reassurance to nervous stakeholders. Many freelance, contract, or agency-based consultants operate in this space and can review questionnaire content quickly. Larger organisations may already employ diversity and inclusion specialists who are experienced in this type of work, potentially saving costs. In any case, after review, the consultant can recommend practical changes to the questionnaire to avoid cultural insensitivity, promoting fairness and inclusion.

Conclusion and Summary

Compared to traditional hiring practices, pre-employment tests represent a fairer and more inclusive approach to hiring. One key reason for this is the ability to review, investigate, refine, and improve assessments over time, something that cannot be replicated with interviews. Much of this work is conducted by publishers before launch but can also be undertaken by the organisations using the tests for hiring. Consequently, pre-employment testing should be strongly considered by any organisation with ambitious diversity and inclusion goals that strives to ensure fairness and equality in its selection processes.