Quantitative testing tells us what's happening (or not), but it can't tell us why, and what to do to correct the situation. That's why we do qualitative testing. If users and customers are not responding to a product the way we had hoped, we need to figure out why that's the case.
As a reminder, qualitative testing is not about proving anything. That's what quantitative testing is for. Qualitative testing is about rapid learning and big insights.
I argue that qualitative testing of your product ideas with real users and customers is probably the single most important discovery activity for you and your product team.
When you do this type of qualitative user testing, you don't get your answer from any one user, but every user you test with is like another piece of the puzzle. Eventually, you see enough of the puzzle that you can understand where you've gone wrong.
I know this is a big claim, but I argue that qualitative testing of your product ideas with real users and customers is probably the single most important discovery activity for you and your product team. It is so important and helpful that I push product teams to do at least two or three qualitative value tests every single week. Here's how to do it:
We generally begin the user test with a short user interview where we try to make sure our user has the problems we think she has, how she solves these problems today, and what it would take for her to switch (see Customer Interview Technique).
We have many good techniques for testing value qualitatively, but they all depend on the user first understanding what your product is and how it works. This is why a value test is always preceded by a usability test.
During the usability test, we test to see whether the user can figure out how to operate our product. But, even more important, after a usability test the user knows what your product is all about and how it's meant to be used. Only then can we have a useful conversation with the user about value (or lack thereof).
Preparing a value test therefore includes preparing a usability test. I described how to prepare for and run a usability test in the last chapter, so for now let me again emphasize that it's important to conduct the usability test before the value test, and to do one immediately after the other.
If you try to do a value test without giving the user or customer the opportunity to learn how to use the product, then the value test becomes more like a focus group where people talk hypothetically about your product, and try to imagine how it might work. To be clear: focus groups might be helpful for gaining market insights, but they are not helpful in discovering the product we need to deliver (see Product Discovery Principle #1).
This testing involves at least you as product manager and your product designer, but I am constantly amazed at how often the magic happens when one of your engineers is right there watching the qualitative testing with you. So, it's worth your pushing to make this happen as much as possible.
To test usability and value, the user needs to be able to use one of the prototypes we described earlier. When we're focused on testing value, we usually utilize high‐fidelity user prototypes.
High‐fidelity means it feels very realistic, which turns out to be especially important for value testing. You can also use a live‐data prototype or a hybrid prototype.
The main challenge in testing value when you're sitting face to face with actual users and customers is that people are generally nice—and not willing to tell you what they really think. So, all of our tests for value are designed to make sure the person is not just being nice to you.
One technique I like for gauging value is to see if the user would be willing to pay for it, even if you have no intention of charging them for it. We're looking for the user to pull out his or her credit card right then and there and ask to buy the product (but we don't really want the card information).
If it's an expensive product for businesses—beyond what someone would put on a credit card—you can ask people if they will sign a “non‐binding letter of intent to buy” which is a good indicator that people are serious.
But there are other ways a user can “pay” for a product. You can see if they would be willing to pay with their reputation. You can ask them how likely they'd be to recommend the product to their friends or co‐workers or boss (typically on a scale of 0–10). You can ask them to share on social media. You can ask them to enter the e‐mail of their boss or their friends for a recommendation (even though we don't save the e‐mails, it's very meaningful if people are willing to provide them).
Especially with businesses, you can also ask the person if they'd be willing to schedule some significant time with you to work on this (even if we don't need it). This is another way people pay for value.
You can also ask people to provide the login credentials for whatever product they would be switching from (because you tell them there's a migration utility or something). Again, we don't really want their login and password—we just want to know if they value our product highly enough that they're truly willing to switch right then and there.
Remember, this is not about proving anything. It's about rapid learning. As soon as you believe you have a problem, or you want to try a different approach, you try it.
For example, if you show your prototype to two different people and the response you get is substantially different, your job is to try to figure out why. Maybe you have two different types of customers, with different kinds of problems. Maybe you have different types of users, with different skill sets or domain knowledge. Maybe they are running different solutions today, and one is happy with their current solution and one is not.
You might determine that you just aren't able to get people interested in this problem, or you can't figure out a way to make this usable enough that your target users can realize this value. In that case, you may decide to stop right there and put the idea on the shelf. Some product managers consider this a big failure. I view it as saving the company the wasted cost of building and shipping a product your customers don't value (and won't buy), plus the opportunity cost of what your engineering team could be building instead.
As product manager, you need to make sure you are at every single qualitative value test. Do not delegate this.
The remarkable thing about this kind of qualitative testing is just how easy and effective it is. The best way to prove this to yourself is to take your laptop or mobile device with your product or prototype on it to someone who hasn't seen it yet, and just give it a try.
One important note. As product manager, you need to make sure you are at every single qualitative value test. Do not delegate this, and certainly don't try to hire a firm to do this for you. Your contribution to the team comes from experiencing as many users as possible, first hand, interacting with and responding to your team's ideas. If you worked for me, the continuation of your monthly salary would depend on this.