Running User Testing With People With Disabilities

1. Why automated testing does not replace genuine user feedback

Automated tools check structural and technical rules: whether an alt attribute exists, whether contrast is sufficient, whether a label exists for a form field. None of these tools can judge whether a person with low vision actually understands a product page, whether a person with a motor impairment can complete checkout in a reasonable amount of time, or whether a person with a cognitive disability can find their way through the navigation. Only observing real people doing real tasks can answer those questions.

On top of that, people with disabilities often use assistive technology in ways that differ substantially from a developer's default configuration, for example a screen reader running at a drastically increased speech rate, a customized switch mapping for switch access, or an individually tuned speech recognition setup. A team that only tests against the default configuration systematically misses exactly the cases where real usage diverges most from the lab assumption.

2. Recruiting diverse test groups

A useful test group does not cover a single disability category, it deliberately spans several: blind and severely low-vision people who primarily work with screen readers, people with motor impairments who navigate exclusively by keyboard, switch access, or voice control, people with cognitive disabilities for whom complexity and consistency matter most, and hard of hearing or deaf people for whom captioned video content and visual alternatives to audio cues are decisive.

Specialized agencies and panels that already have verified, experienced test participants across different disabilities are a solid recruiting channel, as is direct collaboration with disability advocacy organizations or self-help groups, which often have a genuine interest in helping shape digital accessibility. Participants should always be paid fairly and appropriately, regardless of whether recruitment happens through an agency or directly, because professional feedback deserves professional compensation.

3. Test setup: remote testing versus in-person testing

Remote testing over video conference with screen sharing has a decisive advantage: participants use their own, familiar technical environment, meaning the exact hardware and software combination they actually navigate the web with every day, instead of having to adapt to a lab-provided, possibly unfamiliar configuration. For screen reader users this matters a great deal, because individual settings like speech rate or preferred verbosity level significantly shape the interaction.

In-person testing offers the advantage of more direct observation, for example the exact hand movement of someone with a motor impairment, or nonverbal cues like visible frustration that are easier to lose over a video connection. In practice, a hybrid approach works well, where the bulk of sessions run remotely, complemented by a handful of in-person sessions for particularly complex interactions such as a full checkout run.

4. Participants' own assistive technology instead of a default setup

A common methodological mistake is handing test participants a laptop prepared by the team with a freshly installed, unconfigured copy of NVDA or JAWS, instead of letting them use their own device with the highly optimized configuration they have refined over years. That individual configuration, for example a speech rate above four hundred words per minute for an experienced screen reader user, is part of the real usage reality and needs to stay intact during the test.

The same holds for browser extensions, customized keyboard shortcuts, or specialized input devices like a trackball or an adaptive switch that a participant has set up specifically for their own needs. Imposing a sterile test environment instead means the team, in the worst case, measures how usable an unfamiliar configuration is that was never actually used before, rather than the target audience's real daily use.

5. Designing realistic tasks and scenarios

Test tasks should represent concrete actions typical for the specific store, such as finding a particular product through the filter, adding it to the cart, and completing checkout through to the order confirmation, instead of abstractly worded instructions like explore the page that offer no concrete success criterion. Concrete tasks also allow an objective measurement of whether and how long a task takes to complete successfully.

It also matters not to give participants hints or help during the task, even when a barrier visibly causes frustration, because exactly that failure delivers the most valuable finding of the entire test. A moderating team that steps in too early distorts the result and hides a barrier that, in real everyday use without support, would lead to the exact same abandonment.

6. Moderation technique: building trust instead of judging

Moderating a test with people with disabilities differs from classic usability testing on one decisive point: it is never about evaluating the participant's abilities, only ever about evaluating the application, and that needs to be communicated explicitly and unambiguously at the start of every session. A participant who feels they are being tested personally acts more cautiously, producing less authentic behavior.

It is also worth budgeting noticeably more time than for a test without a disability, because certain interactions with assistive technology inherently take longer, and time pressure during the session itself becomes a source of error that has nothing to do with the actual barrier in the application.

7. Evaluation: observation, not just raw screen reader output

During evaluation, logging just the output read aloud by the screen reader is not enough, because a technically correct announcement can still cause confusion, for example if the announcement exists but sits at a point in the flow that makes little sense. What matters is the combination of observed behavior, such as repeated jumping back in focus, verbal reactions expressed during think-aloud, and the actual success rate of the assigned task.

A structured table works well for evaluation, linking every observed barrier to the affected task, the affected user group, a concrete quote or observation, and an initial severity estimate, so raw data from multiple sessions can later be merged and compared instead of getting lost as scattered individual notes.

8. Prioritizing findings for development

Not every observed difficulty carries the same weight. A barrier that led to a full task abandonment independently across multiple participants, such as a checkout confirm button that cannot be focused, gets top priority, while a single, subjectively unpleasant but ultimately surmountable inconsistency can be prioritized lower. Frequency across multiple sessions is a far more reliable indicator than the first impression from a single session.

These prioritized findings then feed into the same ticketing process used for other accessibility findings, though a finding from real user testing should typically get treated with higher priority than a purely automated violation, because it has demonstrably caused an actual abandonment for a real person.

9. Establishing user testing as a recurring process

A single test round, no matter how carefully run, only covers the state of the application at that point in time and loses relevance with every major redesign. That is why it is worth establishing user testing with people with disabilities as a recurring event, for example twice a year or specifically before every major relaunch of a central page like checkout, rather than treating it as a one-off project.

Over time this also builds a stable, recurring group of familiar participants who already know the store and can therefore respond more precisely to concrete changes between two test rounds, which further increases how meaningful each individual round is.

10. Summary

11. FAQ: User Testing With Disabilities: Key Takeaways