Answer Choice Randomization

The Problem: Position Affects Selection

Survey researchers have documented for decades that the physical position of a response option influences the probability that a respondent will select it, independent of its content. This phenomenon, known as positional response bias or order effect, occurs across a range of question formats and does not disappear when respondents are motivated or highly engaged. It is a structural feature of how people process lists under cognitive load.

Two patterns are most commonly observed. In verbal formats, where options are read sequentially, primacy effects lead respondents to give disproportionate weight to the first option encountered. In visual formats, where the list is fully visible, recency effects can produce the reverse, with the final option receiving elevated attention. Which pattern dominates depends on format, topic, and respondent characteristics, but both reflect the same underlying problem: selection is partly a function of position rather than purely a function of content (Krosnick & Alwin, 1987).¹

For behavioral assessments specifically, this creates a measurable distortion. If the highest-scoring option consistently appears at the top, a meaningful share of respondents will select it not because it accurately describes them but because it appeared first. If the lowest-scoring option consistently appears at the bottom, some respondents will avoid it even when it is the honest choice. The result is scores that are systematically biased in directions determined by design decisions rather than by the respondent's actual behavior.

How Randomization Addresses It

The Evans Learning Labs tools randomize the display order of the five answer choices in a different order every time the tool is accessed. What the respondent sees is a different positional arrangement each time.

Critically, the scoring architecture is unaffected. Each answer choice carries a fixed point value between 1 and 5, based on the behavioral level it describes. When a respondent selects an option, the system records the point value of that choice, not its position. Randomization only changes which choice appears first, second, third, and so on. It does not change which choice scores which value.

The practical effect is that respondents cannot use position as a shortcut. The highest-scoring description might appear anywhere in the list. The only reliable path to selecting the right option is to read and evaluate each one, which is exactly what the instrument requires.

Three Specific Benefits

What Does Not Change

Randomization applies only to the display sequence of the five options. Several things are deliberately held constant:

Point values. The behavioral level represented by each option, and the score it produces, is fixed. Option A always produces the same score regardless of whether it appears first, third, or fifth.

Option text. The wording of each choice is not altered by randomization. The content of each description is exactly what it was written to be. Randomization changes only where it appears in the list, not what it says.

Scoring logic. Domain scores, overall scores, and all interpretive outputs are calculated from the point values of selected options. The calculation is identical regardless of what order those options were presented in.