The biggest problem with beta testing that mobile app developers and product managers report is not getting enough useful feedback. We define a beta test as the last testing stage before your app's launch that targets real users.
Making the most of your beta tests requires expertly coordinating a cluster of moving parts, including selecting relevant testers, coaxing them to participate up to their potential, and extracting precise and valuable feedback from them. Reaching these objectives often means asking your beta testers the right questions.
When designing the survey questions you'll ask your beta testers, follow the best practices outlined below to encourage a higher response rate and more accurate answers.
If you choose to use a third-party tool to gather technical information like user steps and bug reports, you'll save 45% of your time and make it significantly easier for your testers to provide more substantive feedback in other areas.
Quantifying the answers allows testers to give you more precise feedback and allows you to compare between answers and track changes over time in an absolute way. A 10-point scale can get too complex; with a one-to-five scale, you offer just enough wiggle room for manageable variance. Be sure to define your high and low, if not each value.
"What was the speed of the installation process like?"
- It was far slower than most apps I use.
- It was slower than most apps I use.
- It took about the same amount of time as most apps I use.
- It was faster than most apps I use.
- It was way faster than most apps I use.
When it comes to user feedback, yes or no questions are basically useless, so avoid them. On the other hand, open-ended questions are often more miss than hit -- most testers won't even bother to respond. With multiple-choice questions, you encourage participation by making it easier for your testers to do so. You clearly define their options and all they have to do is click -- they don't even have to type. With multiple-choice questions, you can also weight the answers in order to do more complex data analysis.
"How much would you pay for this app with all of its features (full version)?"
- I would prefer to pay just for certain features of this app.
- I wouldn't pay for this app at all.
For those who respond with 4, follow up with a direct conversation.
Some of the information you seek can't be quantified. Start with a general aspect you want to know more about, break it down into identifiable parts, and ask about those parts.
Don't ask, "Was the app useful?"
- Do ask, "When was the app most useful to you? What triggers would prompt you to use it? Which features were least useful?"
Give your testers the opportunity to leave any other comments they might have in an unfiltered way. Frame it around the app's objectives. Place it at the end after they've addressed your more high priority, specific queries.
- "Given the app's goal to help you manage your time, why wouldn't you use this app again?"
In beta testing, you want testers to try and break your app, identify corner-case bugs, and probe edge cases. Your questions should also seek to elicit answers that point to areas where your app needs improvement -- its weak spots.
Don't ask, "Did you have fun playing the game?"
- Do ask, "At which points did you feel bored with the game? Which parts felt unnecessarily complicated? Which elements felt irrelevant? What was missing about your game character?"
Depending on the category of your app and the objectives of your beta test, your survey could vary widely. Here are a few more general questions that could apply to any app:
- Which functions didn't work as expected?
- What features would you add?
- How likely are you to recommend this app to a friend?
Use a 0-10 scale for this question to calculate the user's Net Promoter Score (NPS). Those who respond with a 0 to 6 are referred to as "Detractors", while people who reply with 7 or 8 are called "Passives", and 9 or 10 are "Promoters". Having this information gives you a benchmark for further iterations and allows you to identify and target each segment for marketing campaigns. This metric is also commonly used throughout the market, which means you might be able to find the NPS of your competitors in order to see where your app stands in comparison.
Did the app help solve your problem/achieve your goal?
- It didn't help at all.
- It helped, but I could have solved my problem just as easily without this app.
- It helped, but I could have solved my problem just as easily or easier with another app.
- It made me more likely to achieve my goal.
- I wouldn't be able to achieve my goal without this app.
How easy was the app to use?
- I need someone to show me how to use the app.
- I need someone to show me how to use certain features.
- It will take me a few uses to master the app.
- Next time I use it, I'll know exactly what to do.
- It felt like I've been using it all my life.
- How to Improve Mobile App Performance
- Instabug vs Firebase
- Instabug vs New Relic
- Mobile App Quality: An Essential Guide