Editor’s note: We’ve asked a handful of design leaders to respond to prompts each week. This week’s prompt was “What key metrics do you use to track your design’s effectiveness?” Want to read other design leaders’ responses to the prompt? Check out answers from Chris Thelwell and Jeff Gothelf here.
The greatest consumer technology companies of the past 25 years have one thing in common: they created habits. This is what separates world-changing businesses from the rest. Apple, Facebook, Amazon, Google, Microsoft, and Twitter are used daily by a high proportion of their users and their products are so compelling that many of us struggle to imagine life before they existed.
But creating habits is easier said than done. Though I’ve written extensively about behavior engineering and the importance of habits to the future of the web, few resources give entrepreneurs the tools they need to design and measure user habits. It’s not that these techniques don’t exist—in fact, they’re quite familiar to people in all the companies named above. However, to the new entrepreneur, they largely remain a mystery.
“The greatest consumer technology companies of the past 25 years all created habits.”
I’ve learned these methods from some of the best in the business and put together an amalgamation of them that I call “Habit Testing.” It can be used by consumer web companies to build products that users not only love, but are hooked to.
Habit Testing fits hand-in-glove with the build, measure, learn methodology espoused by the lean startup movement and offers a new way to make data actionable. Habit Testing helps clarify 3 things:
- Who your devotees are
- What part of your product is habit forming, if any
- Why those aspects of your product are habit forming
A prerequisite to Habit Testing is having some kind of product up and running. Of course, before launching even a minimal viable product, it’s a good idea to take a stab at your business model hypotheses and how your product will create user desire.
Once you have a site or app live, you can begin collecting data. Habit Testing does not necessitate collecting data about everything—just the right things—so setting up the appropriate analytics is critical. In order for Habit Testing to be successful, you need to date stamp the path users take while using your site.
Step 1: Identify
Now that you have the requisite site and stats, you need to answer the first question of Habit Testing: “Who are the habitual users?” First, define what it means to be a devoted user. Ask yourself how often a user “should” use the site. That is to say, assuming that some day all the bugs are worked out and the product is perfectly “lickable,” how often would you expect a habitual user to be on the site?
Be realistic and honest. If your company builds a mobile social networking app like Foursquare or Instagram, you’d expect habitual users to be on the app several times per day. However, if you’re building a movie recommendation site, a la Flixster, you wouldn’t expect users to be on the site more than once or twice a week. Don’t come up with an overly aggressive prediction that only accounts for uber-addicts; you’re just looking for a realistic guess to calibrate how often users will interact with your site.
A good short-cut might be to take an average of how often you and the people in your office use your own product. Of course, more is better. Twitter was born within Odeo, the company Biz Stone and Jack Dorsey originally founded, because the engineers couldn’t stop playing with it.
One thing to note: the more frequently your product is used, the more likely it is to form a user habit. That’s not to say that web products that are used rarely can’t be good business, they just aren’t habit forming and thus have different characteristics. Viable, though non-habit forming businesses tend to be more transactional and require constant outreach to customers to stay top-of-mind.
For example, think of the travel industry’s relentless war to convince us to use one site over another. Expedia, Travelocity, and the rest are used too infrequently by their average customers to form a habit, so they constantly compete for attention. These are viable, even profitable companies, but since they are non-habit forming products they are open to greater competitive threat. Products used daily naturally create barriers to entry in their markets.
“Products used daily naturally create barriers to entry in their markets.”
Who’s got the habit?
Now that you know how often a user “should” be using the site, it’s time to crunch through the numbers and identify how many of your users actually meet that bar. This is where hiring a stats wiz can prove exceedingly helpful. Instead of pulling your engineers away from their crucial jobs building the product or even worse, getting your business people to do it, consider hiring a grad student fluent in statistics to help you quantify how many of your users are hooked. The best practice here is to create a cohort analysis to provide a baseline by which to measure future product iterations.
“Consider hiring someone to help quantify how many of your users are hooked.”
Step 2: Codify
Hopefully, you’ll have at least a few users who interact frequently enough for you to call them devotees. But how many devotees is enough? My rule of thumb is 5%. Though your rate of active users will need to be much higher to sustain your business, 5% is a good benchmark to being Habit Testing.
However, if at least 5% of your users don’t find your product valuable enough to use as much as you predicted they should, you have a problem. It may be time to go back to the drawing board and rework your vision. But assuming you’ve exceeded that bar and you’ve identified your habitual users, the next step is to codify the steps they took using your product so you can understand what hooked them.
Each user interacts with your product in a slightly different way. Even if you have a standard user flow, how users engage with your site creates a unique data fingerprint that can be analyzed to find patterns. Sift through the data to determine if there are similar behaviors that emerge. What you’ll hopefully discover is a “Habit Path,” a series of similar behaviors shared by your most loyal users.
For example, in its early days, Twitter discovered that once new users followed enough other members, they hit a tipping point which dramatically increased the odds they would keep using the site. Every company has a different set of actions that devoted users take. The goal of finding the Habit Path is to determine which of those steps were critical for creating devoted users.
Get in their heads
Now that you know the Habit Path, the next step is to create hypotheses about what it was along that path that tipped users from passers-bye to devotees. Granted, this step can look a little like assuming causation from correlation; but in the murky fog of launching a new product, it’s often the best thing we’ve got.
This phase is also a good time to talk to users in person to learn more about why and how they use the product. Habit Testing is meant to illuminate what is unique about these “earlyvangelists” and find insights that can be generalized to the rest of your users.
Step 3: Modify
With new hypotheses in mind, it’s time to get back inside the build, measure, learn loop and take new users down the same Habit Path the devotees took. For example, leveraging their Habit Path, Twitter’s onboarding process now guides new users to start following others immediately.
“If a product doesn’t create a habit, it may as well not exist.”
Habit Testing is a continual process companies can implement with every new feature and product iteration. Tracking users by cohort and comparing their activity to habitual users should guide how products evolve, improve, and foster habit formation.
Too often tech entrepreneurs find themselves alone with their vision because they fail to realize the importance of creating user habits. And unfortunately, when it comes to consumer web and the ever-increasing distractions we all face daily, if the product doesn’t create a habit, it may as well not exist. By using Habit Testing to determine what is most valuable and habit-forming about a product, entrepreneurs can better serve their users and increase the odds of creating world-changing companies.
Join the conversation
Write your own response to the prompt “What key metrics do you use to track your design’s effectiveness?” on Medium, and submit it to our publication.