Guerrilla Testing, Useful or Not?

What does science and professional say about it.

A soldier walking on war field
Photo by Stijn Swinnen on Unsplash

While user testing, card sorting, and many more tools have been scientifically proven to end up increasing the usability of an interface, the need to always tend to more agility give birth to new tools, too young to be back up by scientific studies.

Ux designers must be open to new ways to work and analyze their interfaces, but they must also be careful about what we are promised for using new tools.

Guerilla testing was a trend in 2017 and is still used by many Ux design companies. Two years later, what do we know about this tool?

A soldier pointing his gun to the right
Photo by Specna Arms on Unsplash

The promise

I got one blog post from a company to summarize what is said about guerilla testing on the web.

Guerilla is presented as a cheap method that allows getting a lot of feedback, quickly iterate, and get a better design for an interface.

You just have to ask random people to do a micro-test of your app and hear their first impressions.

The methodology presented is the following :

  • #1: Come up with a list of tasks
  • #2: Prioritize tasks
  • #3: Turn your tasks into scenarios
  • #4: Start guerilla testing
  • #5: Capture testing insights
  • #6: Fix your usability problems
  • #7: Test again, validate, make it a habit

These seven points are very similar to the preparatory work for any classical user testing.

What differentiates it from classical user testing is the targetted users and time.

Anybody can test the app, you stop somebody in the street or hallway and present your scenario.
Your scenario shouldn’t exceed 15 minutes top.

Prepare multiple user testing scenarios and have random people do the test for 15min.

This seven-point methodology could be the basis for a reproducible experiment. With a clear definition of how to do guerilla testing, it is would be possible to see if this tool brings value or not to designing interfaces.

A marine writing a letter, smoking a pipe. Sepia effect.
Photo by Museums Victoria on Unsplash

Two sentences are bothering me in this article.

“Because you’re collecting qualitative data during the tests, you just need 3 to 5 people to spot the biggest usability issues”

As we already cover, using only 5 users to spot usability problems is only partially true. Five random users might give you insight and spot usability problems but can’t be enough.

Guerrilla research, therefore, is the perfect starter drug for stakeholders who struggle to acknowledge the value of usability testing.”

This sentence enlights a major issue and might be the true origin of this new tool.

Is guerrilla testing a tool aiming to convince stakeholders?

If so, we are not in Ux anymore but in the domain of internal marketing.

I understand the need to convince that Ux research brings real value to products. The impact made by a good design is invisible. It prevents losses because users are retained, and less likely to go to a rival company if their experience is great. the value is hard to measure.

But the goal of Ux designers isn’t to convince stakeholders, it’s to create easy to use interfaces.

Three high ranking soldiers walking and smiling.
Photo by Museums Victoria on Unsplash

Scientific literature

The tools used by designers are studied by ergonomists and psychologist to scientifically prove the benefits bring by their use. This is the case for heuristic evaluation, usability testing, or card sorting.

Guerrilla testing has no scientific study to back it up, to my knowledge.

Two factors can explain why.

First, it’s a new tool.
Studies that back up other tools are 20 to 40 years old. Back in the day Ux designers didn’t exist or were rare.

People that conduct these studies were either developers, psychologist, or ergonomists that wanted to find how to create better software or analyze the mechanisms behind creativity.

Second, it might originate from marketing.
I didn’t find the origin of guerilla testing. It wasn’t created by a big name like Norman or Nielsen, it seems to have just popped up.

My theory is that having the first person available test an unfinished app is just a common practice for any designer. I like to have my girlfriend give me feedback, she always has great ideas and positive feedback motivates me to continue.

This common practice may have been given a name by the marketing department to give it some credibility.

Like the article used before said, “it’s great to convince stakeholders”.
If it is a creation from the marketing department, this would explain why no scientist studied it as it didn’t originate from scientific theories.

(and it would explain the badass name)
An old soldier conducing a truck in a parade with a young woman laughing in the passenger seat.
Photo by Vitor Pinto on Unsplash

Guerrilla is still present in scientific papers but in the study methodology, not as the main subject.

One example can help us identify flaws in the use of guerrilla: researchers used guerilla testing to compare the Google website ranking to the perceived utility of each site about breast cancer.

In this study, guerrilla testing was used to select the participant and create a small sample. People found in the hallway were graduated or PhD students. They were asked to read websites about breast cancer and say how useful the websites were. The 200 first results of Google search were used.

The conclusion is that even a website in the low ranking of Google could be useful and even more useful than some of the first results.

This study can give us some ideas about what are the pro and cons of guerilla teasing.

People didn’t evaluate the website but its efficiency. Guerrilla testing could be used to evaluate how user perceive an app’s functionality performance.

Is the efficiency of web search perceived better on Google or Bing? Is it perceived easier to send a direct message to someone on Discord, Slack, or Messenger?

The hallway creates a bias. There might be a limit to the “random” factor. In this study, every tester was highly educated, the hallway was clearly inside a university. If the goal to have randomly selected people to test an interface is to get diversity, the places you go to recruit can break the random factor as similar people go to similar places.

A girl alone in a destroyed city.
Photo by Jordy Meow on Unsplash

New ways to use guerrilla testing

Guerrilla testing is a new way to analyze an interface and to get data to iterate. We can’t tell yet if it is better than classical user testing or heuristic analysis.

It is said to help detect major flaws, but as an expert aren’t you suited enough to detect them by yourself?

I think this tool has real potential but might not be used to its full extent.

It could be linked to an emotion-maps by getting the first impressions of anybody that open the app. Doing this could help Ui designers achieve a wow effect or get users in a good mood.

A user pleased by the look of an interface will be in a better state of mind to use it and more tolerant of imperfections encountered.

If the goal is to have a friction less workflow, having a lot of random people use your app without context can give you clues about what kind of users are at ease with your app and what kind isn’t.

This could even be linked to marketing by helping define the “ideal customer persona”.

Lastly, it could be used as preparatory work for bigger scale user testing. Maybe having first reactions from random people can help better define scenarios for classical user testing.

In the near future, scientists will conduct experiments to define the real value of guerrilla testing in a reproducible way, with data backing it up. Meanwhile, Ux designers will continue to use this method because it’s easy and cheap, but we must keep in mind that it can’t replace traditional tools.

“Make salsa, Nor war” pink neon.
Photo by Victor Garcia on Unsplash

Guerrilla testing, useful or not? was originally published in UX Planet on Medium, where people are continuing the conversation by highlighting and responding to this story.