Sometimes, in order to address a major risk identified in discovery, we need to be able to collect some actual usage data. But we need to collect this evidence while in discovery, well before taking the time and expense of building an actual scalable and shippable product.
Some of my favorite examples of this are when applying game dynamics, search result relevance, many social features, and product funnel work.
This is the purpose of a live‐data prototype.
A live‐data prototype is a very limited implementation. It typically has none of the productization that's normally required, such as the full set of use cases, automated tests, full analytics instrumentation, internationalization and localization, performance and scalability, SEO work, and so forth.
The live‐data prototype is substantially smaller than the eventual product, and the bar is dramatically lower in terms of quality, performance, and functionality. It needs to run well enough to collect data for some very specific use cases, and that's about it.
The key is to be able to send some limited amount of traffic, and to collect analytics on how this live‐data prototype is being used.
When creating a live‐data prototype, our engineers don't handle all the use cases. They don't address internationalization and localization work, they don't tackle performance or scalability, they don't create the automated tests, and they only include instrumentation for the specific use cases we're testing.
A live‐data prototype is just a small fraction of the productization effort (in my experience, somewhere between 5 and 10 percent of the eventual delivery productization work), but you get big value from it. There are two big limitations you do have to keep in mind, however:
Today, the technology for creating live‐data prototypes is so good that we can often get what we need in just a couple days to a week. And once we have it we can iterate very quickly.
Later, we'll discuss the quantitative‐validation techniques and you'll see the different ways we can utilize this live‐data prototype. But for now, know that the key is to be able to send some limited amount of traffic, and to collect analytics on how this live‐data prototype is being used.
What's important is that actual users will use the live‐data prototype for real work, and this will generate real data (analytics) that we can compare to our current product—or to our expectations—to see if this new approach performs better.