Supercharge your agile UX research with informal cognitive walkthroughs

In the second phase of the ICW, we gain real-world context through an External User Review: a method that combines multiple parallel one-on-one observation sessions with focus group discussions. For the External User Review, we typically bring in five users per session and set up two back-to-back lab sessions — allowing us to talk to 10 customers in a single day.

To define the External User Review method, members of the Microsoft Intune team and I modified the Pluralistic Walkthrough, a usability testing method that involves adding representative users as participants in the cognitive walkthrough. Our approach differs from the traditional method most notably in the roles the product team members and usability experts play:

1. Because we always precede this session with an Internal Product Team Review, the feedback from the product team members and usability experts has already been integrated in the artifacts presented to the users. Thus, the External User Review focuses solely on learning from our study participants.

2. To run 5 one-on-one sessions in parallel, each participant is paired up with a product team member. Each pair runs through the major sections in the study at their own pace. This allows them to traverse the designs as they wish, as opposed to working from printed task scenarios (as is common in the Pluralistic Walkthrough) — which provides more natural feedback.

3. Our note-taking structure and group discussion method (described below) eliminate the need for post-session data analysis. They also guard against stakeholders overgeneralizing one or two points of view.

How to conduct the External User Review

By this point, you’ve identified the five product team members who will be pairing up with participants. The rest will sit behind the one-way mirror (or watch remotely).

In the second phase of the ICW, we gain real-world context through an External User Review: a method that combines multiple parallel one-on-one observation sessions with focus group discussions. For the External User Review, we typically bring in five users per session and set up two back-to-back lab sessions — allowing us to talk to 10 customers in a single day.

To define the External User Review method, members of the Microsoft Intune team and I modified the Pluralistic Walkthrough, a usability testing method that involves adding representative users as participants in the cognitive walkthrough.

In the second phase of the ICW, we gain real-world context through an External User Review: a method that combines multiple parallel one-on-one observation sessions with focus group discussions. For the External User Review, we typically bring in five users per session and set up two back-to-back lab sessions — allowing us to talk to 10 customers in a single day.

To define the External User Review method, members of the Microsoft Intune team and I modified the Pluralistic Walkthrough, a usability testing method that involves adding representative users as participants in the cognitive walkthrough.

In the second phase of the ICW, we gain real-world context through an External User Review: a method that combines multiple parallel one-on-one observation sessions with focus group discussions. For the External User Review, we typically bring in five users per session and set up two back-to-back lab sessions — allowing us to talk to 10 customers in a single day.

To define the External User Review method, members of the Microsoft Intune team and I modified the Pluralistic Walkthrough, a usability testing method that involves adding representative users as participants in the cognitive walkthrough.