January 8th, 2009 Systems posted by dmerritts View Comments

Instrument. Measure. Test. Optimize.

Web 2.0 is a big experiment. The corpus of knowledge is under developed. Some argue that the current web world is really only 3 or 4 years old. I agree that web as a platform is young and still acts like a child finding its way in the world. Best practices are still being developed. There is certainly no clear clear path to success. Granted, there are guidelines and companies that one can emulate. But, every business is a little different. Success first requires failing quickly. I’ve spent a lot of time talking to and listening to people who’ve created successful web companies and services and it really comes down to four activities beyond ideation and initial product: (1) Instrumenting User Interactions / Paths; (2) Measuring Drop-offs / Abandonment; (3) A/B & Multivariate Testing; (4) Optimize and Change Rapidly.

Unlike other industries, the data from web businesses is abundant and easy to collect. Tracking user interaction and engagement is trivial. Performing tests of new appflows is cheap and easy. One must be constantly hypothesizing and testing variations. Rigor in this testing is what drives success.


Since I come for the systems and application management world, I understand the complexity and need for deep and meaningful user interaction instrumentation. There are a lot of companies out that that help make this a development after-thought (e.g., Omniture, Tealeaf). Yet, much of the cost can be eliminated if you make instrumentation an initial development requirement. The key is ensuring that elements are properly tagged and easily traced and groked from logs. If instrumentation is a core building block from the beginning, the rest of this process becomes easier.


Identifying what’s important depends on the web application. However, the most important metric is funnel drop-off based on the goal of the business. If you’re an retailer, then shopping cart abandonment is paramount. If you’re a social site, then sign-up is paramount. If you’re a freemium based service company, then conversion rate (or drop-off) is paramount. If you’re a word-of-mouth site, then invite quantity and funnel is paramount. If you’re a lead generation site, then form complete is paramount. Identifying the most critical flows to one’s success, adoption and user engagement will drive these measurements. Not only is it important to measure drop-off rates, but it’s also important to begin to understand why certain user behavior is occurring. Each of these measurements should become a daily dashboard metric that product teams live by. Subsequent experiments should be developed and continually tuned.


Experimentation is what drives success. Initial educated guesses are typically wrong. And, web product teams that are not driven by statistics will likely fail to hit key milestones. Statistically driven experiments is what makes for better web applications and user adoption. One must define experiments in A/B and Multivariate terms. Doing so ensures that one can eliminate user drags and optimize user lifts. Simple things such as: color, button placement, number of fields, placement of field text, select alls can drastically change adoption. There is a lot of literature on these topics but the key to success is making this a priority often over new feature development. Remember, if you don’t have the users or a clear adoption or conversion rate, then why does the next new feature matter? So, develop intelligent and data guided experiments that guide statistically significant change.


Iteration is the name of the game. Taking small steps often is key. The faster the release process the better on the Web. Releases don’t have to be large, but they should be focused with clear goals for measurement and testing. Having scientific discipline around this process from a product and development standpoint will help make one’s web products successful.

At the end of the day, there are 150 Million websites all vying for users. If you’re not constantly innovating and testing, the site is likely to fail.

http://blog.danmerritts.com/wp-content/plugins/sociofluid/images/digg_48.png http://blog.danmerritts.com/wp-content/plugins/sociofluid/images/reddit_48.png http://blog.danmerritts.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png http://blog.danmerritts.com/wp-content/plugins/sociofluid/images/delicious_48.png http://blog.danmerritts.com/wp-content/plugins/sociofluid/images/facebook_48.png
blog comments powered by Disqus