Guerrilla testing

What is guerrilla testing?

Do UX and UI design have something in common with warfare tactics? Surprisingly, yes. It is a guerrilla strategy applied to user testing.

Guerrilla is a powerful word. It also brings powerful meaning to the things with which it is connected nowadays. Originally the term "guerrilla" comes from the Spanish word that means "little war" and has been used to describe a form of asymmetric warfare. It's the practice of fighting a war in which one side has fewer resources than the other and often uses unorthodox methods to achieve victory.

This strategy nowadays often gets rid of its military connections and changes its meaning to strategic thinking. Guerrilla word accompanies many different fields of human activity and is connected, for instance, with marketing, gardening, art, and testing. The common ground in all those diverse areas is achieving results with fewer resources, unconventional methods, sometimes unofficial (with no support from local authorities), quickly, and with a lower budget. This out-of-ordinary approach can still show its power and sometimes can have unexpected results and even more impact than a conventional solution.

How is guerrilla applied to the type of testing in UX and UI design and research?

What is the definition of guerrilla testing (also called hallway usability testing)? It is a technique used in UX design and research to test your product with real people who give you a raw perspective on what they think of your design. Guerrilla testing is a type of usability testing where the tester tries to observe and evaluate the users' behavior in their natural habitat. It's sometimes called "on-the-street" testing because it lets you watch users in their natural environment while on the way to work, doing their favorite activities, or waiting for somebody or something else.

The most significant benefit is that you can catch up with potential users for a short period and still get valuable insights. It's a way of getting feedback on your product without having to make them come into the office to test it.

It's important to note that guerrilla testing isn't just about observing but also understanding why certain behaviors happen and what you could do to improve upon them.

It can be used by (almost) any company or business that wants to see how the end user uses their products. Some companies use it regularly, while others only resort to it when they need the most accurate feedback on their product or service.

So what does it look like? What characterizes guerrilla testing:

• Usually, one or two researchers representing the design team enter a chosen public place - they find themselves in a cafe, park, airport, gym, and so on - somewhere where they can meet potential users.
• Researchers are equipped with smartphones or laptops on which they have a prototype of the website or app
• They ask passers-by, people visiting public spaces, if they would have 10-15 minutes to test the new website or app in exchange for offering some coffee, sweets, or vouchers.

You can also organize a more simple guerrilla testing "on the fly" with almost no preparation. You can consider people around you who would be interested in a given topic of the designed website or app. Those can be your friends, colleagues from the company, or family members. You can ask them to check your digital product's chosen feature or process and share an honest opinion.

What is the difference between guerrilla testing and usability testing?

The difference between guerrilla and usability testing is that the second is carefully planned and prepared. In official usability testing, participants are precisely recruited and selected, and the whole process is designed with all details. Of course, it doesn’t mean that guerilla testing is conducted without a plan. It just means less preparation and the possibility of meeting and talking to people, possibly potential users and customers, who would have never taken part in a lab or online user testing.

Is guerrilla testing right for you?

When to use guerrilla testing? Are you considering guerrilla testing? Here are some reasons why you should conduct it and why it is crucial in your case:

• You are at an early stage of your product design and want to check its prototype and what users think about its features, functionality, and visual appearance.
• You feel stuck in some parts of your design and unsure about some functionalities.
• You are into Agile methodology - you want to revise the design process throughout the whole project, believe in constant testing to make improvements, and focus on user feedback at all levels.
• You are in the Design Sprint process and want to test early prototypes quickly to get quick responses and feedback.
• You want to quickly check your assumptions on the product's features and flow.
• Guerrilla testing is proper when you want to get unbiased feedback on your product and ensure that it meets your audience's needs.
• You want to identify problems with your design before you launch it publicly so that you can fix them before any damage is done.
• You want to improve the product before you conduct more formal usability testing.
• You don't have any budget for formal usability testing - it is always better to conduct guerrilla testing than not to do any usability testing at all.
• You are a small start-up or even a one-man-army where independently of a small team, you work on every part of product design, from the ideation and research phase, through design to development.
• You want to compare your product with another from competitors.

How to conduct guerrilla testing?

It is worth remembering that the whole organization of the guerrilla test can take several minutes to hours (the longer one is associated with the need to arrange a suitable place and equipment for tests and prepare scenarios). You can consider several steps to get the best results in the most extended version.

1. Set the goal

Before you start planning, ask yourself - what your goal is and what you want to achieve. You can do it by brainstorming with the team and stating some hypothesis research questions. You also have to prioritize those goals - what is the most crucial for you to know – because you will probably be unable to check and revise all your assumptions. For the first guerrilla testing, choose the essential goals regarding the functionality and UX of your app or website. Bear in mind that you will have just 10 minutes to test.

Your goals in the form of questions can sound, for instance:

• Can people log in to my app without any trouble?
• What do people think about the checkout process on my e-commerce website?
• Do users know what to expect when they reach a particular part of the app or website?

2. Plan the scenario

Success lies in planning. So think carefully about every aspect of your guerrilla research, starting from the place, the time, used devices, through your image and attitude, and finishing on properly asked questions.

3. Place / Location

Choose carefully a place where you will invite people for testing. Crowded streets where people are in a rush because of everyday duties are not a good idea. Think about your final users and where they usually go. Visit places where people take breaks or are taking some time off. If they spend their time in a coffee shop - you’re lucky. If they don’t take out coffee, usually people stop there and plan to spend at least a dozen or so minutes, and you can propose to them a coffee or some cake (or both) - so kind of incentive, something that they anyway wanted to buy. It also can be a food court where you can propose to some people to buy them lunch.

Think also about other places where you can find your potential customers and where they wouldn’t be in a rush. It can be a cafe or a quiet location in the gym, park benches, bus or train station, an airport where people wait for their departure, or some long lines in particular national departments, where people sometimes spend hours waiting for their turn. While you are planning to do some guerrilla research, plan a few different locations to find other potential users of your digital product.

Always ask the owners or manager from the chosen place if it is ok for them to conduct a guerrilla test. Explain to them carefully what the purpose of it is. You want to stay in good relation with them, especially since you will meet their customers and visitors, but also use their infrastructures like electricity, toilets, wifi, and furniture.

4. Time

Think cautiously about the time when you can conduct your research. If your target group is working adults, you would have to be ready to organize a few sessions later in the evening, after usual working hours, to find the right participants. You would probably have to plan not only different locations but different hours of the day to recruit suitable participants. Time is also connected to the place. Perhaps locations which are good in the morning will not be so good later in the day. Search for the site (e.g., cafe) you are considering on Google. You'll find there an indication of the busiest times in the business' summary box that helps you to plan the best time to find people, but still have not a too busy and noisy place to conduct it.

5. The team

Find a partner "in crime." The perfect number of people involved in guerrilla testing research is two. If there were more people, participants could feel like they were in a zoo, overwhelmed and intimidated by the number of people watching them. Usually, the best is when one person is the leading researcher responsible for recruiting people, introducing the whole of the research, conducting the test, and taking notes. At the same time, the other arranges the environment - taking care of devices, buying a coffee, cake, lunch, or other incentives, grabbing a camera (if needed), or just helping to keep on track with the scenario and time.

6. Create smart tasks - scenario and discussion guide

As you have 10 or maximum 15 minutes for the testing, revise your goals and plan strategic tasks carefully. Identify the critical tasks people must be able to accomplish using your product and write them down. For example, if you're building a mobile app for ordering food, you should test how people find meals, place orders, and add meals to their carts. If you create a website for SAAS, you will check how people create their accounts and decide which offer they choose.

Make a list of your tasks, prioritize them and decide what to test exactly. Make a scenario based on the top three tasks, and ensure it is easy for users to understand. Test all the scenarios and tasks during a dry run with somebody else (a family member or a friend) before the actual test to be sure that all tasks are easy to understand and that it is possible to do them in a time of 10-15 minutes.

When writing usability tasks, it is recommended that you keep the list of tasks short and simple, avoid leading questions, and use as few steps as possible. For guerrilla testing, simplicity is vital since participants won't have the same context around your product as they would in a conventional, prearranged test.

Thus, ensure that your research objectives are not reliant upon previous knowledge of your product's features.

EXAMPLE

You can set your tasks in a broader scenario. This way, they will know not only that they have to do appropriate action (click, sign in, create, order, invite, subscribe, download) but also the purpose of the action, for instance:

• It’s Friday evening. You invited 3 of your best friends to your home. Order enough pizza for you all and choose the delivery time.
• Your vacation time is coming. Choose Greece as a destination and book a room in a hotel for two people.
• You want to get a movie recommendation for the evening using the XYZ App. Create a profile with your preferences to get relevant recommendations.
• You would like to learn more about the impact of soya-based food on a diet. Go to the blog section and use a search engine to find the topic you are interested in.

Remember! Don’t group actions together (like creating the account and ordering food); give one task at a time! It helps you avoid the participant feeling overwhelmed by too many actions.

Prepare also follow-up open-ended questions that can help you deepen the reflection from the participants, for instance:

• “What do you think about this process?”
• “Is there anything you would improve in this?”
• ​​”What is your impression of this feature?”
• “Did anything surprise you or fail to meet your expectations?”

7. Prepare yourself, the prototype, and the place.

Image

You probably don’t care about your image in the case of guerrilla testing because you think your preparation and approach are the most important. And that’s true! However, you can help smooth the process and conduct recruitment efficiently when you care about more details. Your image can help you with that. How? Imagine that you test a bank app or other that requires the highest level of reliability. You don’t have to wear a suit, but it can be helpful to look more formal than usual. But when you test the new sports card website, it would be better to look more informal, even “sporty,” to identify more with participants and encourage them to test it.

Mentally and physically preparation

Don’t do the test on the fly before one another task from a different field. Prepare yourself mentally and be fully engaged in guerrilla testing. Be ready to listen actively. It will help you to follow the scenario, and the people, be a good listener, and remember all details that can help you later improve the product.

Loudness

It is good to visit places where you want to conduct guerrilla testing and check its loudness. If it is too loud, you should rethink changing the site.

Wifi

If you are planning to use the wifi from the place you will test your product, check before if its connection is stable and robust enough to conduct it. Think as well about plan B - using, for instance, your data transfer as a hotspot or even using the internet from the participant, if they agree.

Prototype or MVP

A prototype does not need to be a finished product, but it should be functional. If you decide to test a low-fidelity prototype, don’t forget to explain to the participant that the product still doesn’t have any visually attractive parts, and you will test its raw components with no visual interface. You can also test MVP this way.

Devices

Think about every device and software you need to conduct the test in a live setting. It can be a laptop, smartphone, tablet, charger, video camera, etc. Install the software before to have time to test it and check if everything works well and speaks to your needs.

Be familiar with the scenario and questions

In the case of guerrilla testing, we recommend a semi-structured form. It means that you have a set of questions and tasks, a prepared scenario with every part of it, but you still know them perfectly, learn them even by heart, and freely modify them to sound more natural and follow the flow of the meeting.

Have a plan B

• What if the chosen place will be an electric breakdown or an unsuspected, loud birthday party?
• What if you couldn’t find any eager participants in the desired spot?
• What if your laptop turns off for no apparent reason?

In every case, think about what could go wrong, be smart, and prepare a plan B to not be suspected and unready for those “crisis situations.” Have some extra devices with you, batteries, memory cards, or even printed screens of your product (when everything fails, you can simply talk about it). 

8. Choose the participants

Remove biases

You should beware of biases in choosing participants and confirmation biases in your research questions. Even if the dominant target group for your website about parenting is mothers, it is still worth asking fathers about their opinion. Also, don't try to find confirmation about your assumptions and feelings about the design or functionality that is especially important to you. You wouldn't like to influence the participants, but you want to collect honest feedback. Therefore rethink carefully prepared open-ended questions, and ensure that they

Try to find the most diverse group

Different places are linked to people representing different approaches, target groups, and so on. It is hardly possible that you will find every right person in one coffee shop. Therefore, diverse places as much as possible and also be open for people representing different sex, race, cultures, age, and lifestyles. It is crucial to get a diverse point of view.

Respect participants' time

Respect people's time. In guerrilla testing, they are not usability testing participants who are earning a set remuneration for testing in the lab environment. They are mainly on the way somewhere; just stop for a moment. If you are telling them that doing tasks will take a maximum of 10 minutes, don't exceed the time.

9. Follow the plan and be open to changes

Even if guerrilla sounds like something informal, it doesn't mean that it should be unprepared. Quite the opposite! The key to the success of this kind of testing is a well-thought-out plan and scenario. Also, prepare for the changes. Have a plan B in case of unpredicted situations, or follow the needs of your participants.

Introduce yourself

It can be challenging to accost strangers on the street or cafe. Prepare your perfect "sales pitch" for about 20-30 seconds, where you, shortly and sweetly, explain who you are, why you are bothering them, and what they can receive after completing tasks (coffee, cake, lunch, voucher).

EXAMPLE

"Hi! I'm Michael! We are testing some features of our new app. We would love to offer you a coffee or a cake if you would find 10 minutes to give us feedback about some of its parts".

If you need additionally qualify people for your testing, you can add additional questions in this pitch, for instance:

"Do you have a smartphone and you booked accommodation somewhere in the past through the website or mobile app?"

Ask the participant for a time

Ensure even twice that the participant has time; they are not in a rush and must be somewhere soon. Show them that you treat this time seriously, and it will be only those 10 or 15 minutes, nothing more.

Explain the aim of the testing

Be extra clear about what you need from participants. Tell them that you want them to look at the prototype of your website or app, do the following task, and talk out loud about this: what are they doing and what do they think while doing it, and what are their feelings. Emphasize that you're not testing the participant but the product. So there is no right or wrong way to do the task, or it is not even possible to fail because it is all about the product. So you will not help them if something is going wrong or they have some doubts because it is also important feedback for them about the product's usability.

It is crucial to avoid giving precise instructions that could impact the flow of the session.

Sometimes it is a good idea to omit redundant information (from the participant's perspective), for example, that you were involved in designing it, or even use a little white lie that you are not associated with the design. Ensure that you will be happy to get honest feedback, no matter what.

Ask if you can record the meeting

Ask for consent to record the meeting. Ensure that you will use it only to review parts that would be unclear. Make clear that it wouldn't be published anywhere; you want to record not a person but what she or he is talking and doing on the screen. And finally, explain that it will serve only for small team purposes.

You can have with you printed consent that someone can sign down before you start recording.

Use checkboxes on your consent form to ensure your participants understand each part. Giving them a consent form and expecting them to complete it and comprehend it is not enough. You will need to provide them with an explanation of the project.

After you have informed them of what you have told them, they will sign the consent form to confirm they understand:

• You may record them.
• You will not share these recordings outside the project team.
• The possibility of withdrawing at any time.
• After the testing is completed, you will not retain their contact information and personal details.
• You will follow data protection guidelines for the storage of their data.

Ask participants to introduce themselves

Before giving detailed tasks, ask participants to introduce themselves a bit to you.

Describe all the information you need about them; for example, their age, gender, job, where they're from, where they live, etc.

Have prepared a sheet for notes

Even if you are going to record the meeting, writing notes from the testing is valuable. Possibly you wouldn't have time to go back to records, and you will base your recommendation on the most important insights you will catch on the paper.

The table can consist of three columns. In the first column, you can write the scenario that can keep you on track and help you to remember every part of the session and the information you have to give to the participants. In the second column, you can write down questions with tasks you will pass the participant. Between them, prepare a chunk of space to write what people are doing and their feelings while completing tasks. In the last column, leave a space for additional observations and notes. For every participant, you should have a separate sheet.

Thanking participants

After completing tasks:

• Thank participants for their time.
• Buy them a coffee, a cake, or lunch.
• Or give them the voucher you mentioned in the beginning.

If they agree, you can also take a selfie with them to remember with whom you were talking.

NDA

Sometimes during the testing, you show essential information from the product that has not launched yet. Therefore you can even consider preparing a Non-Disclosure Agreement to be sure that participants won't tell anybody about your new product and its features before it is published.

Conduct the session with minimum 5 people (ideally 8-10 people)

There is a saying that after you have interviewed five participants successfully, you will start getting diminishing outcomes. However, in this type of testing, where you recruit people on the fly, you should consider including a few off-key people in the group (people who aren't your ideal users). Ideally, we want to get five strong candidates out of eight to ten successful interviews.

10. Analyze the results

After completing the guerrilla testing, you will stay with about 8-10 sheets of notes and the same number of recordings. It is good practice to go back into them shortly after the testing when it is still fresh in your mind. Then note down every insight on the post-it and, in the end, try to cluster them in repeatable motifs. If you have any doubts, watch the video records again to ensure you don’t miss anything.

Based on those repeatable patterns, prepare concrete recommendations for improvement for the next design iteration or even create a report you can present to the stakeholders.

Remote guerrilla testing

You can also try to conduct your guerrilla testing remotely. You can test remotely by exploring public forums such as Reddit, Quora, LinkedIn Groups, or other social media. You can write a post there where you describe the intent and the related incentive. Send them a link to an unmoderated test with written instructions if you find interested people. You can also use online tools, which we mentioned in the "Useful tools for guerrilla testing" section.

If you can spend more time on remote testing, you can also prepare complete remote usability testing, where you more carefully recruit people to test your product.

What are the benefits of guerrilla testing?

Guerrilla testing is an effective method for getting real-world feedback from users. It allows you to observe real-life scenarios and see how your product performs in those situations. What are the benefits of guerrilla testing?

• You can meet with real-life potential users.

You can do guerilla testing by asking passing-by people, not prepared for the test in a lab environment, what they think of your product or service.

• It's cheap

You can do guerrilla testing on any budget! It's low-cost and easy for anyone who wants to learn more about how their users think and behave when interacting with their products or services. If you don't have any budget for usability testing, it is better to conduct guerrilla testing than nothing.

• It's fast

It saves users' and recruiters' time by eliminating the need to find people 'on spec' and the cost of traveling to meet them. You can also get feedback from potential users. Instead of waiting weeks or months to receive answers from lab usability studies, you can use guerrilla testing to find the results within a few hours or days.

• No specific research skills are required.

Almost anyone from the team can conduct guerrilla testing with little preparation and an appropriate attitude. A moderator should be just open to accosting strangers and be aware of the importance of open-ended questions and not asking leading questions

• You are ready to get enough valuable insights

Guerrilla testing provides enough insight to inform strategic design decisions quickly and effectively.

• Demonstrate the value of user testing for stakeholders

Guerrilla testing can demonstrate the importance of user testing and UX research for stakeholders who struggle to acknowledge the value of usability testing.

• Test the product in the early stages of design

Testing and identifying usability issues early in the design process can help you avoid costly mistakes. It is a perfect way to diagnose areas for more profound research.

• You can find out more about your competitors.

People are usually eager to compare your solutions to those they already use, so you can get valuable information about your competitors and how they solve similar problems.

• You can test hypotheses or assumptions during design sprints.

Tests such as guerrilla testing can be helpful tools for validating hypotheses and creating validation checkpoints.

What are the drawbacks of guerrilla testing?

Even if there are many unquestionable benefits of guerrilla testing, this method also has some drawbacks:

• Possibility of not finding proper participants from the target audience

It may be that people recruited on the fly are not end users that will use it after the launch.

• You can have no participants at all

You can't be sure that you will recruit anyone to have time and answer your questions. People usually don't like to be grabbed on the street, and even for the offer of incentive, they could not be interested in spending 10 minutes testing your product.

• Possibility of missing essential issues

You are not spending enough time with participants in guerrilla testing, so there can't be enough time to deepen some reflections and opinions. It's more likely that their results aren't statistically significant because their sample size is usually smaller than in usability testing.

• The results may not be convincing enough

As this type of testing is not conducted by professional researchers with long preparation and recruitment of the participants, the results of it may not convince stakeholders.

• It is shallower in in-depth insights than usability testing

When we want to uncover nuanced and rich insights, follow more complex user journeys and learn about users' behavior, we should conduct in-depth listening sessions and usability testing. It will need a dedicated period for recruitment, but potential applicants suit our audience criteria better.

• It doesn't allow testing every product

Guerrilla testing doesn't fit every situation. You can't expect people to be competent enough to use your product.For example, if you require specific knowledge linked to financial or medical apps, it will be almost impossible to find the right users "on the street." Furthermore, it would not be easy to test the product if it requires a specific environment for it (e.g., a certain location or action like driving a car).

• It never replaces the form of usability testing

With its low resources and preparation, guerrilla testing would never replace more formal usability testing. With this second test type, you can get valuable insights from the right people, which will be precisely analyzed and show proper directions for improvement.

Useful tools for guerrilla testing

Display recorder

If you don't want to use the external video camera on the tripod for recording the meeting, it is good to consider some software that helps you record the screen. Here is a few chosen software that can help you to organize guerrilla testing.

Screen Recorder

Google Play or AppStore

You can record for free the view from your smartphone, tablet, or laptop. The only drawback is you have to consider the additional recording of the participant's voice because it records only what happens on the screen.

Bandicam Screen Recorder

It is a free screen recorder for Windows that enables you to capture any area of your screen as a screenshot or screencast video file. Furthermore, you can also record the participant's face and voice together on video with a webcam and microphone.

OBS Studio

It is a free video recorder for Windows, macOS, and Linux users. It also allows recording the person's voice in front of the computer. Moreover, you can adjust video and sound properties to your needs.

Voice Memos

With this free voice recorder, achievable for iOS and Android, you can easily record voice and insights from your participants of guerrilla testing.

CaptureUX

CaptureUX is PC-based software that you can use to record a usability test of any product. It can help you organize guerrilla testing on-site by using your own computer (only with Windows system), a participant computer with HDMI, and two cameras that record the smartphone screen. You can also conduct it remotely, thanks to recording online meetings like Webex or Zoom. Unfortunately, this software is quite expensive. It costs $900.00 per year plus tax for a Single-Seat License.

Helio

This web app can help you conduct UX research remotely - both test some prototypes, and early design, create a task for users and ask them some questions. With Helio, you can even specify your target audience. The drawback is that you have to pay for every option, and you can't even know the cost of it before you will share your email with the Helio company.

Five second test

It is a perfect tool to examine your users' first impressions of your design. Five-second tests are a method of user research that help you measure what information users take away and what impression they get within the first five seconds of viewing a design. Participants are given five seconds to view a design, after which they answer some simple questions. In the free plan, you can test for up to 2 minutes. The prices vary between 89 and 199$ per month.

Figma

You can prepare a working and testable prototype in Figma, with which participants can test your flows, design, functionalities, and UX.

Paper, pen

Old and foolproof tools like paper and pen can always help you to note down every most significant insight from the guerrilla testing participants.

Analysis

You should also consider where you will work on analyzing gathered insights. You can create, of course, a physical board with a post-it, but you can also use an online version. You can do it, for instance, through FigJam, Miro, or Google JamBoard, creating online post-its and clustering them in repetitive motifs and topics that can help you conclude actionable recommendations.

Conclusion

Guerrilla testing is ideal for when you're working with a small budget or limited time frame. You can use guerrilla testing to test out an idea before you commit to developing it fully or as part of your ongoing design process.

Guerrilla tests can be done at any point in the design process, whether you're creating a new feature from scratch or iterating on an existing one. The goal is always to get as much information as possible from each session to make informed decisions about your product's usability and design.

You'll find that guerrilla testing helps you discover things about your product that no other research method could uncover—things like people's emotional responses or willingness to pay for a feature. You'll also learn what features are important enough for users to pay attention to when using your product (or not).

So, if you're unsure if time-consuming and expensive usability testing is for you, you are afraid of meeting the final user and not sure about the importance of showing them the prototype, start with testing more informally, and go guerrilla!