While qualitative testing is all about fast learning and big insights, quantitative techniques are all about collecting evidence.
We will sometimes collect enough data that we have statistically significant results (especially with consumer services with a lot of daily traffic), and other times we'll set the bar lower and just collect actual usage data that we consider useful evidence—along with other factors—to make an informed decision.
This is the main purpose of the live‐data prototype we discussed earlier. As a reminder, a live‐data prototype is one of the forms of prototype created in product discovery intended to expose certain use cases to a limited group of users to collect some actual usage data.
We have a few key ways to collect this data, and the technique we select depends on the amount of traffic we have, the amount of time we have, and our tolerance for risk.
While qualitative testing is all about fast learning and big insights, quantitative techniques are all about collecting evidence.
In a true startup environment, we don't have much traffic and we also don't have much time, but we're usually fine with risk (we don't have much to lose yet).
In a more established company, we often have a lot of traffic, we have some amount of time (mostly we're worried about management losing patience), and the company is usually more averse to risk.
The gold standard for this type of testing is an A/B test. The reason we love A/B tests is because the user doesn't know which version of the product she is seeing. This yields data that is very predictive, which is what we ideally want.
Keep in mind that this is a slightly different type of A/B test than optimization A/B testing. Optimization testing is where we experiment with different calls to action, different color treatments on a button, and so forth. Conceptually they are the same, but in practice there are some differences. Optimization testing is normally surface‐level, low‐risk changes, which we often test in a split test (50:50).
In discovery A/B testing, we usually have the current product showing to 99 percent of our users, and the live‐data prototype showing to only 1 percent of our users or less. We monitor the A/B test more closely.
If your company is more risk averse, or if you just don't have enough traffic to be able to show to 1 percent—or even 10 percent—and get useful results anytime soon, then another effective way to collect evidence is the invite‐only test. This is where you identify a set of users or customers that you contact and invite to try the new version. You tell them that it is an experimental version, so they are effectively opting in if they choose to run it.
The data that this group generates is not as predictive as from a true, blind, A/B test. We realize that those who opt in are generally more early adopter types; nevertheless, we are getting a set of actual users doing their work with our live‐data prototype, and we are collecting really interesting data.
I can't tell you how often we think we have something they'll love, and then we make it available to a limited group like this and we find that they are just not feeling it. Unfortunately, with a quantitative test like this all we know for sure is that they're not using it—we can't know why. That's when we'll follow up with a qualitative test to try and quickly learn why they're not as into it as we had hoped.
A variation of the invite‐only test is to use the members of the customer discovery program we discussed in the section on ideation techniques. These companies have already opted in to testing new versions, and you already have a close relationship with them so you can follow up with them easily.
For products for businesses, I typically use this as my primary technique for collecting actual usage data. We have the customer discovery program customers getting frequent updates to the live‐data prototype, and we compare their usage data to that of our broader customers.
One of the most significant changes in how we do product today is our use of analytics. Any capable product manager today is expected to be comfortable with data and understand how to leverage analytics to learn and improve quickly.
I attribute this change to several factors.
First, as the market for our products has expanded dramatically due to access globally—and also by way of connected devices—the sheer volume of data has dramatically increased, which gives us interesting and statistically significantly results much faster.
Any capable product manager today is expected to be comfortable with data and understand how to leverage analytics to learn and improve quickly.
Second, the tools for accessing and learning from this data have improved significantly. Mostly, however, I see an increased awareness of the role that data can play in helping you learn and adapt quickly.
There are five main uses of analytics in strong product teams. Let's take a close look at each of these uses:
When most people think of analytics, they think of user analytics. That is, however, but one type of analytic. The idea is to understand how our users and customers are using our products (remember, there can be many users at a single customer—at least in the B2B context). We may do this to identify features that are not being used, or to confirm that features are being used as we expect, or simply to gain a better understanding of the difference between what people say and what they actually do.
This type of analytic has been collected and used for this purpose by good product teams for at least 30 years. A solid decade before the emergence of the web, desktops and servers were able to call home and upload behavior analytics, which were then used by the product team to make improvements. This to me is one of the very few non‐negotiables in product. My view is that, if you're going to put a feature in, you need to put in at least the basic usage analytics for that feature. Otherwise, how will you know if it's working as it needs to?
I have long been a strong advocate of using data to drive product teams. Rather than provide the team an old‐style roadmap listing someone's best guess of what features may or may not work, I strongly prefer to provide the product team with a set of business objectives—with measurable goals—and then the team makes the calls as to what are the best ways to achieve those goals. It's part of the larger trend in product to focus on outcome and not output.
Today, especially for consumer companies, we can isolate the contribution of new features, new versions of workflows, or new designs by running A/B tests and then comparing the results. This lets us prove which of our ideas work. We don't have to do this with everything, but with things that have high risk or high deployment costs, or that require changes in user behavior, this can be a tremendously powerful tool. Even where the volume of traffic is such that collecting statistically significant results is difficult or time consuming, we can still collect actual data from our live‐data prototypes to make decisions that are much better informed.
In my experience, the worst thing about product in the past was its reliance on opinions. And, usually, the higher up in the organization the person was who voiced it, the more that opinion counted.
Today, in the spirit of data beats opinions, we have the option of simply running a test, collecting some data, and then using that data to inform our decisions. The data is not everything, and we are not slaves to it, but I find countless examples today in the best product teams of decisions informed by test results. I hear constantly from teams how often they are surprised by the data, and how minds are changed by it.
While I am personally hooked on each of the above roles of analytics, I must admit that my personal favorite is this last point. The data we aggregate (from all sources) can be a gold mine. It often boils down to asking the right questions. But by exploring the data, we can find some very powerful product opportunities. Some of the best product work I see going on right now was inspired by the data. Yes, we often get great ideas by observing our customers, and we do often get great ideas by applying new technology. But studying the data itself can provide insights that lead to breakthrough product ideas.
Largely, this is because the data often catches us off guard. We have a set of assumptions about how the product is used—most of which we are not even conscious of—and when we see the data, we're surprised that it doesn't track with those assumptions. It's these surprises that lead to real progress.
It's also important for tech product managers to have a broad understanding of the types of analytics that are important to your product. Many have too narrow of a view. Here is the core set for most tech products:
Hopefully, you can see the power of analytics for product teams. However, as powerful as the role of data is for us, the most important thing to keep in mind about analytics is that the data will shine a light on what is happening, but it won't explain why. We need our qualitative techniques to explain the quantitative results.
________________
Note that we often refer to analytics as key performance indicators (KPIs).
Remarkably, I still encounter too many product teams that either aren't instrumenting their product to collect analytics, or they do it at such a minor level that they don't know if and how their product is being used.
My own teams—and every team I can think of that I've ever worked with—have been doing this for so long now that it's hard to imagine not having this information. It's hard for me to even remember what it was like to have no real idea how the product was used, or what features were really helping the customer, versus which ones we thought had to be there just to help close a sale.
This is easiest to do with cloud‐based products and services, and most of us use web analytics tools, but sometimes we use homegrown tools for this too.
Good product teams have been doing this for years. And, not just with cloud‐based sites, but also with installed mobile or desktop applications—on‐premise software, hardware, and devices that call home periodically and send the usage data back to the teams. A few companies are very conservative and ask permission before sending the data, but mostly this just happens silently.
We should all be anonymizing and aggregating the data so there's nothing personally identifiable in there. Occasionally, however, we see in the news that another company is in trouble for sending raw data in the rush to market. It seems the press thinks we're tracking these data for nefarious purposes, but at least with the companies I know and work with, they're simply trying to make the products better—more valuable and more usable. This has long been one of our most important tools for doing so.
The way the overall process works is that we first ask ourselves what we need to know about how our products are used, then we instrument the product to collect this information (the particular techniques depend on the tool you're using and what you want to collect). Finally, we generate various forms of online reports to view and interpret these data.
I still encounter too many product teams that either aren't instrumenting their product to collect analytics, or they do it at such a minor level that they don't know if and how their product is being used.
For everything new we add, we ensure we have the necessary instrumentation in place to know immediately if it is working as we expect, and if there are significant unintended consequences. Frankly, without that instrumentation, I wouldn't bother to roll out the feature. How would you know if it was working?
For most product managers, the first thing they do in the morning is to check the analytics to see what happened during the preceding night. They're usually running some form of test almost all the time, so they're very interested in what's happened.
There are of course some extreme environments where everything lives behind very strict firewalls, but even then, the products can generate periodic usage reports to be reviewed and approved before being forwarded (via electronic or printed reports, if necessary) back to the teams.
I'm very big on radically simplifying products by removing features that don't carry their own weight. But, without knowing what is being used, and how it's being used, it's a very painful process to do this when you don't know what's really going on. We don't have the data to back up our theories or decisions, so management (rightfully) balks.
My personal view is that you should start from the position that you simply must have this data, and then work backward from there to figure out the best way to get it.