How to measure anything finding the value of intangibles in business, second edition
HOW TO M E A SUR E A N Y TH I NG F I N D I N G T H E VA L U E O F “ I N TA N G I B L E S ” I N B U S I N E S S
2nd Edition REVISED, EXPANDED & SIMPLIFIED
DOUGLAS W. HUBBARD
How to Measure Anything Finding the Value of “Intangibles” in Business
DOUGLAS W. HUBBARD
John Wiley & Sons, Inc.
2010 by Douglas W. Hubbard. All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. For more information about Wiley products, visit our web site at www.wiley.com. Library of Congress Cataloging-in-Publication Data:
Hubbard, Douglas W., 1962How to measure anything : finding the value of “intangibles” in business / Douglas W. Hubbard. – 2nd ed. p. cm. Includes index. ISBN 978-0-470-53939-2 (cloth) 1. Intangible property–Valuation. I. Title. HF5681.I55H83 2010 657 .7–dc22 2009051051
Printed in the United States of America. 10
I dedicate this book to the people who are my inspirations for so many things: to my wife, Janet, and to our children, Evan, Madeleine, and Steven, who show every potential for being Renaissance people. I also would like to dedicate this book to the military men and women of the United States, so many of whom I know personally. I’ve been out of the Army National Guard for many years, but I hope my efforts at improving battlefield logistics for the U.S. Marines by using better measurements have improved their effectiveness and safety.
MEASUREMENT: THE SOLUTION EXISTS
Intangibles and the Challenge
Yes, I Mean Anything The Proposal
An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily
How an Ancient Greek Measured the Size of Earth Estimating: Be Like Fermi Experiments: Not Just for Adults Notes on What to Learn from Eratosthenes, Enrico, and Emily CHAPTER 3
The Illusion of Intangibles: Why Immeasurables Aren’t The Concept of Measurement The Object of Measurement The Methods of Measurement Economic Objections to Measurement The Broader Objection to the Usefulness of “Statistics”
10 11 13 18
21 22 26 28 35 37
Ethical Objections to Measurement Toward a Universal Approach to Measurement
BEFORE YOU MEASURE
Clarifying the Measurement Problem
Getting the Language Right: What “Uncertainty” and “Risk” Really Mean Examples of Clarification: Lessons for Business from, of All Places, Government CHAPTER 5
Calibrated Estimates: How Much Do You Know Now?
Calibration Exercise Further Improvements on Calibration Conceptual Obstacles to Calibration The Effects of Calibration
59 64 65 71
Measuring Risk through Modeling
How Not to Measure Risk Real Risk Analysis: The Monte Carlo An Example of the Monte Carlo Method and Risk Tools and Other Resources for Monte Carlo Simulations The Risk Paradox and the Need for Better Risk Analysis
79 81 82
Measuring the Value of Information
The Chance of Being Wrong and the Cost of Being Wrong: Expected Opportunity Loss The Value of Information for Ranges The Imperfect World: The Value of Partial Uncertainty Reduction The Epiphany Equation: How the Value of Information Changes Everything Summarizing Uncertainty, Risk, and Information Value: The First Measurements
100 103 107 110 114
The Transition: From What to Measure to How to Measure
Tools of Observation: Introduction to the Instrument of Measurement Decomposition Secondary Research: Assuming You Weren’t the First to Measure It The Basic Methods of Observation: If One Doesn’t Work, Try the Next Measure Just Enough Consider the Error Choose and Design the Instrument CHAPTER 9
Sampling Reality: How Observing Some Things Tells Us about All Things
120 124 127 128 131 132 136
Building an Intuition for Random Sampling: The Jelly Bean Example A Little about Little Samples: A Beer Brewer’s Approach Statistical Significance: A Matter of Degree When Outliers Matter Most The Easiest Sample Statistics Ever A Biased Sample of Sampling Methods Measure to the Threshold Experiment Seeing Relationships in the Data: An Introduction to Regression Modeling One Thing We Haven’t Discussed—and Why
142 145 148 150 153 162 165
Bayes: Adding to What You Know Now
Simple Bayesian Statistics Using Your Natural Bayesian Instinct Heterogeneous Benchmarking: A “Brand Damage” Application Bayesian Inversion for Ranges: An Overview
Bayesian Inversion for Ranges: The Details The Lessons of Bayes
BEYOND THE BASICS
Preference and Attitudes: The Softer Side of Measurement
Observing Opinions, Values, and the Pursuit of Happiness A Willingness to Pay: Measuring Value via Trade-offs Putting It All on the Line: Quantifying Risk Tolerance Quantifying Subjective Trade-offs: Dealing with Multiple Conflicting Preferences Keeping the Big Picture in Mind: Profit Maximization versus Purely Subjective Trade-offs
The Ultimate Measurement Instrument: Human Judges
203 207 211 214
Homo absurdus: The Weird Reasons behind Our Decisions Getting Organized: A Performance Evaluation Example Surprisingly Simple Linear Models How to Standardize Any Evaluation: Rasch Models Removing Human Inconsistency: The Lens Model Panacea or Placebo?: Questionable Methods of Measurement Comparing the Methods
227 228 230 234
New Measurement Instruments for Management
The Twenty-First-Century Tracker: Keeping Tabs with Technology Measuring the World: The Internet as an Instrument Prediction Markets: A Dynamic Aggregation of Opinions
251 254 257
A Universal Measurement Method: Applied Information Economics
Bringing the Pieces Together Case: The Value of the System that Monitors Your Drinking Water Case: Forecasting Fuel for the Marine Corps Ideas for Getting Started: A Few Final Examples Summarizing the Philosophy
Calibration Tests (and Their Answers)
270 275 281 287
lot has happened since the first edition of this book was released in 2007. First, my publisher and I found out that a book with the title How to Measure Anything apparently sparks interest. For three years, the book has consistently been the single best seller in Amazon’s math for business category. Interest shows no sign of slowing and, in fact, registrations on the book’s supplementary Web site (www.howtomeasureanything.com) show that the interest is growing across many industries and countries. It was successful enough that I could pitch my second book idea to my editor. The 2008 financial crisis occurred just as I was finishing my second book, The Failure of Risk Management: Why It’s Broken and How to Fix It. I started writing that book because I felt that the topic of risk, which I could spend only one chapter on in this book, merited much more space. I argued that a lot of the most popular methods used in risk assessments and risk management don’t stand up to the bright light of scientific scrutiny. And I wasn’t just talking about the financial industry. I started writing the book well before the financial crisis started. I wanted to make it just as relevant to another Katrina or 9/11 as to a financial crisis. I’ve also written several more articles, and the combined research from them, my second book, and comments from readers on the book’s Web site gave me plenty of new material to add to this second edition. But the basic message is still the same. I wrote this book to correct a costly myth that permeates many organizations today: that certain things can’t be measured. This widely held belief is a significant drain on the economy, public welfare, the environment, and even national security. “Intangibles” such as the value of quality, employee morale, or even the economic impact of cleaner water are frequently part of some critical business or government policy decision. Often an important decision requires better knowledge of the alleged intangible, but when an executive believes something to be immeasurable, attempts to measure it will not even be considered. As a result, decisions are less informed than they could be. The chance of error increases. Resources are misallocated, good ideas are rejected, and bad ideas are accepted. Money is wasted. In some cases life and health are
put in jeopardy. The belief that some things—even very important things— might be impossible to measure is sand in the gears of the entire economy. All important decision makers could benefit from learning that anything they really need to know is measurable. However, in a democracy and a free enterprise economy, voters and consumers count among these “important decision makers.” Chances are your decisions in some part of your life or your professional responsibilities would be improved by better measurement. And it’s virtually certain that your life has already been affected—negatively—by the lack of measurement in someone else’s decisions. I’ve made a career out of measuring the sorts of things many thought were immeasurable. I first started to notice the need for better measurement in 1988, shortly after I started working for Coopers & Lybrand as a brand-new MBA in the management consulting practice. I was surprised at how often clients dismissed a critical quantity—something that would affect a major new investment or policy decision—as completely beyond measurement. Statistics and quantitative methods courses were still fresh in my mind. In some cases, when someone called something “immeasurable,” I would remember a specific example where it was actually measured. I began to suspect any claim of immeasurability as possibly premature, and I would do research to confirm or refute the claim. Time after time, I kept finding that the allegedly immeasurable thing was already measured by an academic or perhaps professionals in another industry. At the same time, I was noticing that books about quantitative methods didn’t focus on making the case that everything is measurable. They also did not focus on making the material accessible to the people who really needed it. They start with the assumption that the reader already believes something to be measurable, and it is just a matter of executing the appropriate algorithm. And these books tended to assume that the reader’s objective was a level of rigor that would suffice for publication in a scientific journal—not merely a decrease in uncertainty about some critical decision with a method a nonstatistician could understand. In 1995, after years of these observations, I decided that a market existed for better measurements for managers. I pulled together methods from several fields to create a solution. The wide variety of measurement-related projects I had since 1995 allowed me to fine-tune this method. Not only was every alleged immeasurable turning out not to be so, the most intractable “intangibles” were often being measured by surprisingly simple methods. It was time to challenge the persistent belief that important quantities were beyond measurement. In the course of writing this book, I felt as if I were exposing a big secret and that once the secret was out, perhaps a lot of things would be different. I even imagined it would be a small “scientific revolution” of sorts
for managers—a distant cousin of the methods of “scientific management” introduced a century ago by Frederick Taylor. This material should be even more relevant than Taylor’s methods turned out to be for twenty-first-century managers. Whereas scientific management originally focused on optimizing labor processes, we now need to optimize measurements for management decisions. Formal methods for measuring those things management usually ignores have barely reached the level of alchemy. We need to move from alchemy to the equivalent of chemistry and physics. The publisher and I considered several titles. All the titles considered started with “How to Measure Anything” but weren’t always followed by “Finding the Value of Intangibles in Business.” I give a seminar called “How to Measure Anything, But Only What You Need To.” Since the methods in this book include computing the economic value of measurement (so that we know where to spend our measurement efforts), it seemed particularly appropriate. We also considered “How to Measure Anything: Valuing Intangibles in Business, Government, and Technology” since there are so many technology and government examples in this book alongside the general business examples. But the title chosen, How to Measure Anything: Finding the Value of “Intangibles” in Business, seemed to grab the right audience and convey the point of the book without necessarily excluding much of what the book is about. The book is organized into four sections. The chapters and sections should be read in order because the first three sections rely on instructions from the earlier sections. Section One makes the case that everything is measurable and offers some examples that should inspire readers to attempt measurements even when it seems impossible. It contains the basic philosophy of the entire book, so, if you don’t read anything else, read this section. In particular, the specific definition of measurement discussed in this section is critical to correctly understand the rest of the book. Section Two begins to get into more specific substance about how to measure things—specifically uncertainty, risk, and the value of information. These are not only measurements in their own right but, in the approach I’m proposing, prerequisites to all measurements. Readers will learn how to measure their own subjective uncertainty with “calibrated probability assessments” and how to use that information to compute risk and the value of additional measurements. It is critical to understand these concepts before moving on to the next section. Section Three deals with how to reduce uncertainty by various methods of observation, including random sampling and controlled experiments. It provides some shortcuts for quick approximations when possible. It also discusses methods to improve measurements by treating each observation as updating and marginally reducing a previous state of uncertainty. It reviews some material that readers may have seen in first-semester statistics
courses, but it is written specifically to build on the methods discussed in Section Two. Some of the more elaborate discussions on regression modeling and controlled experiments could be skimmed over or studied in detail, depending on the needs of the reader. Section Four is an eclectic collection of interesting measurement solutions and case examples. It discusses methods for measuring such things as preferences, values, flexibility, and quality. It covers some new or obscure measurement instruments, including calibrated human judges or even the Internet. It summarizes and pulls together the approaches covered in the rest of the book with detailed discussions of two case studies and other examples. In Chapter 1, I suggest a challenge for readers, and I will reinforce that challenge by mentioning it here. Write down one or more measurement challenges you have in home life or work, then read this book with the specific objective of finding a way to measure them. If those measurements influence a decision of any significance, then the cost of the book and the time to study it will be paid back manyfold.
o many contributed to the content of this book through their suggestions, reviews, and as sources of information about interesting measurement solutions. In no particular order, I would like to thank these people:
Freeman Dyson Peter Tippett Barry Nussbaum Skip Bailey James Randi Chuck McKay Ray Gilbert Henry Schaffer Leo Champion Tom Bakewell Bill Beaver Julianna Hale James Hammitt Rob Donat Michael Brown Sebastian Gheorghiu Jim Flyzik
Pat Plunkett Art Koines Terry Kunneman Luis Torres Mark Day Ray Epich Dominic Schilt Jeff Bryan Peter Schay Betty Koleson Arkalgud Ramaprasad Harry Epstein Rick Melberth Sam Savage Gunther Eyesenbach Johan Braet Jack Stenner
Robyn Dawes Jay Edward Russo Reed Augliere Linda Rosa Mike McShea Robin Hansen Mary Lunz Andrew Oswald George Eberstadt Grether David Todd Wilson Emile Servan-Schreiber Bruce Law Bob Clemen Michael Hodgson Moshe Kravitz Michael Gordon-Smith
Special thanks to Dominic Schilt at Riverpoint Group LLC, who saw the opportunities with this approach back in 1995 and has given so much support since then. And thanks to all of my blog readers who have contributed ideas for this second edition.
Measurement: The Solution Exists
Intangibles and the Challenge When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science. —Lord Kelvin, British physicist and member of the House of Lords, 1824–1907
nything can be measured. If a thing can be observed in any way at all, it lends itself to some type of measurement method. No matter how “fuzzy” the measurement is, it’s still a measurement if it tells you more than you knew before. And those very things most likely to be seen as immeasurable are, virtually always, solved by relatively simple measurement methods. As the title of this book indicates, we will discuss how to find the value of those things often called “intangibles” in business. There are two common understandings of the word “intangible.” It is routinely applied to things that are literally not tangible (i.e., not touchable, solid objects) yet are widely considered to be measurable. Things like time, budget, patent ownership, and so on are good examples of things that you cannot touch but yet are measured. In fact, there is a well-established industry around measuring so-called intangibles such as copyright and trademark valuation. But the word “intangible” has also come to mean utterly immeasurable in any way at all, directly or indirectly. It is in this context that I argue that intangibles do not exist. You’ve heard of “intangibles” in your own organization—things that presumably defy measurement of any type. The presumption of immeasurability is, in fact, so strong that no attempt is even made to make any observations that might tell you something—anything—about the alleged
Measurement: The Solution Exists
immeasurable that you might be surprised to learn. You may have run into one or more of these real-life examples of so-called intangibles: Management effectiveness The forecasted revenues of a new product The public health impact of a new government environmental policy The productivity of research The “flexibility” to create new products The value of information The risk of bankruptcy The chance of a given political party winning the White House The risk of failure of an information technology (IT) project Quality Public image Each of these examples can very well be relevant to some major decision an organization must make. It could even be the single most important impact of an expensive new initiative in either business or government policy. Yet in most organizations, because the specific “intangible” was assumed to be immeasurable, the decision was not nearly as informed as it could have been. One place I’ve seen this many times is in the “steering committees” that review proposed investments and decide which to accept or reject. The proposed investments may be related to IT, new product research and development, major real estate development, or advertising campaigns. In some cases, the committees were categorically rejecting any investment where the benefits were primarily “soft” ones. Important factors with names like “improved word-of-mouth advertising,” “reduced strategic risk,” or “premium brand positioning” were being ignored in the evaluation process because they were considered immeasurable. It’s not as if the idea was being rejected simply because the person proposing it hadn’t measured the benefit (a valid objection to a proposal); rather it was believed that the benefit couldn’t possibly be measured—ever. Consequently, some of the most important strategic proposals were being overlooked in favor of minor cost-savings ideas simply because everyone knew how to measure some things and didn’t know how to measure others. Equally disturbing, many major investments were approved with no basis for measuring whether they ever worked at all. The fact of the matter is that some organizations have succeeded in analyzing and measuring all of the previously listed items, using methods that are probably less complicated than you would think. The purpose of this book is to show organizations two things: 1. Intangibles that appear to be completely intractable can be measured. 2. This measurement can be done in a way that is economically justified.
Intangibles and the Challenge
To accomplish these goals, this book will address some common misconceptions about intangibles, describe a “universal approach” to show how to go about measuring an “intangible,” and provide some interesting methods for particular problems. Throughout, I have attempted to include some examples (some of which I hope the reader finds inspirational) of how people have tackled some of the most difficult measurements there are. Without compromising substance, this book also attempts to make some of the more seemingly esoteric statistics around measurement as simple as they can be. Whenever possible, math is converted into simpler charts, tables, and procedures. Some of the methods are so much simpler than what is taught in the typical introductory statistics courses that we might be able to overcome many phobias about the use of quantitative measurement methods. Readers do not need any advanced training in any mathematical methods at all. They just need some aptitude for clearly defining problems. Readers are encouraged to use this book’s Web site at www. howtomeasureanything.com. The site offers a library of downloadable spreadsheets for many of the more detailed calculations shown in this book. There also are additional learning aids, examples, and a discussion board for questions about the book or measurement challenges in general. The site also provides a way for me to discuss new technologies or techniques that were not available when this book was printed.
Yes, I Mean Anything I have one recommendation for a useful exercise to try. When reading through the chapters, write down those things you believe are immeasurable or, at least, you are not sure how to measure. After reading this book, my goal is that you are able to identify methods for measuring each and every one of them. And don’t hold back. We will be talking about measuring such seemingly immeasurable things as the number of fish in the ocean, the value of a happy marriage, and even the value of a human life. Whether you want to measure phenomena related to business, government, education, art, or anything else, the methods herein apply. With a title like How to Measure Anything, anything less than a multivolume text would be sure to leave out something. My objective does not include every area of physical science or economics, especially where measurements are well developed. Those disciplines have measurement methods for a variety of interesting problems, and the professionals in those disciplines are already much less inclined even to apply the label “intangible” to something they are curious about. The focus here is on measurements that are relevant—even critical—to major organizational decisions and yet don’t seem to lend themselves to an obvious and practical measurement solution.
Measurement: The Solution Exists
If I do not mention your specific measurement problem by name, don’t conclude that methods relevant to that issue aren’t being covered. The approach I will talk about applies to any uncertainty that has some relevance to your firm, your community, even your personal life. This extrapolation should not be difficult. When you studied arithmetic in elementary school, you may not have covered the solution to 347 times 79 in particular but you knew that the same procedures applied to any combination of numbers and operations. So, if your problem happens to be something that isn’t specifically analyzed in this book—such as measuring the value of better product labeling laws, the quality of a movie script, or effectiveness of motivational seminars—don’t be dismayed. Just read the entire book and apply the steps described. Your immeasurable will turn out to be entirely measurable.
The Proposal Let me begin by stating the three propositions as a way to define and approach the problem of measurement in business: 1. Management cares about measurements because measurements inform uncertain decisions. 2. For any decision or set of decisions, there are a large combination of things to measure and ways to measure them—but perfect certainty is rarely a realistic option. 3. Therefore, management needs a method to analyze options for reducing uncertainty about decisions. Perhaps you think the first two points are too obvious to make. But while it may seem obvious, few management consultants, performance metrics experts, or even statisticians approach the problem with the explicit purpose of supporting defined decisions. Even if they had that squarely in mind, the last point, at a minimum, is where a lot of business measurement methods fall short. It is very useful to see measurement as a type of optimization problem for reducing uncertainty. Upon reading the first edition of this book, a business school professor remarked that he thought I had written a book about the somewhat esoteric field called “decision analysis” and disguised it under a title about measurement so that people from business and government would read it. That wasn’t my intention when I set out, but I think he hit the nail on the head. Measurement is about supporting decisions, and there are even several decisions to make within measurements themselves. If the decision in question is highly uncertain and has significant consequences if it turns out wrong, then measurements that reduce uncertainty
Intangibles and the Challenge
about it have a high value. Nobody should care about measuring something if it doesn’t inform a significant bet of some kind. Likewise, if measurements were free, obvious, and instantaneous, we would have no dilemma about what, how, or even whether to measure. Granted, a measurement might also be taken because it has its own market value (e.g., results of a consumer survey) or because it is simply satisfying a curiosity or will be entertaining (e.g., academic research about the evolution of clay pottery). But the methods we discuss in the decisionfocused approach to measurement should be useful on those occasions, too. If a measurement is not informing your decisions, it could still be informing the decisions of others who are willing to pay for the information. And if you are an academic curious about what really happened to the wooly mammoth, then, again, I believe this book will have some bearing on how you set up the problem. From here on out, this book addresses three broad issues: why nothing is really immeasurable, how to set up and define any measurement problem, and how to use powerful and practical measurement methods to resolve the problem. The next two chapters of this book build the argument for the first point: that you can really measure anything. Chapters 4 through 7 set up the measurement problem by answering questions from the point of view of supporting specific decisions. We have to answer the question “What is the real problem/decision/dilemma?” underlying the desired measurement. We also have to answer the question “What about that problem really needs to be measured and by how much (to what degree of accuracy/precision)?” These questions frame the problem in terms of the primary decision the measurement is meant to resolve and the “microdecisions” that need to be made within the measurement process itself. The remainder of the book combines this approach with powerful and practical empirical methods to reduce uncertainty—some basic, some more advanced. The final chapter pulls it all together into a solution and describes how that solution has been applied to real-world problems. Since this approach can apply to anything, the details might sometimes get complicated. But it is much less complicated than many other initiatives organizations routinely commit to doing. I know, because I’ve helped many organizations apply these methods to the really complicated problems: venture capital, IT portfolios, measuring training, improving homeland security, and more. In fact, measurements that are useful are often much simpler than people first suspect. I make this point in Chapter 2 by showing how three clever individuals measured things that were previously thought to be difficult or impossible to measure.