Do Rising Expectations Distort Our View of Satisfaction?
The idea of rising expectations is not new and almost always in play. We do not generally make our evaluations between real world elements and ideal forms in the Platonic sense. Rather we make more empirical comparisons. Even the most objective and concrete forms of measurement are fundamentally comparisons. When the referents of those comparisons are changing, the nature of the comparison and its manifestation in measurement changes as well. At some point high scores are not only limited by upper bounds but their meaning begins to change as well. When the meaning changes the operational value and informational carried change as well.
While the scores are still high, how they relate to the core concepts of interest, sales, retention, etc. change because the relationships based on one range of the scores being related are different from other ranges. This is exacerbated because the range for satisfaction does not change with the change in theoretical content. Whereas before comparison was between apples and oranges, now the comparison is between Chihuahuas that look like apples and oranges.
There are other reasons to be concerned about the demand for high or higher satisfaction scores. As aggregate scores based on a bounded scale get higher they must become more homogeneous. Think about grade point average. It takes only one “B” to lower your perfect GPA and no matter how many “A’s” you get after that you can never reach a “4.0”. This insight points to two limitations of conventional satisfaction measurement and scores. The higher the score the harder it becomes to move the needle higher in both absolute and relative terms. While large absolute increases may always be difficult to achieve, it is always easier to achieve larger absolute and relative increases from lower scores than from higher scores.
When you have a “good” high scoring product or service it will be hard to use those scores to keep it good or make it better. If it is psychologically unlikely that measurement will continue to describe the same thing as scores and true satisfaction get higher and higher, it is also true that the measurement itself will become more volatile. The best known example of this is “regression to the mean”. It is a mathematical necessity that when two variables, like the orientation of satisfaction and the measurement of satisfaction, are less than perfectly correlated, which they always will be in the real world (measurement error of all kinds is always present), cases that are at one extreme on one of the variables will, on the average, be less extreme on the other. This means that there is a perhaps unknown, but real, error intrinsic in the measurement of satisfaction and as satisfaction scores get higher they become more volatile downward. Together these two effects make such scores less operationally useful. You want high scores, work hard to get them but recognize that you cannot maintain them, there meaning will inevitably be changing making them un-comparable, and the higher they are the less usable operational information they can provide.
The homogeneity of higher scores intrinsically and artificially limits variance. This in turn truncates maximum observable relationships. When there is little or no variance there is less or nothing to compare and little to be learned. Measurement that limits variance misses the point of measuring. We can measure age as old or young OR years since birth. It should be obvious which measurement strategy captures the most information and is likely to have the greatest operational value. Even if the two distributions bi-modal and continuous seem to be perfectly correlated, they are not. The latter carries for more information than the former. Do a quick and dirty spreadsheet exercise correlating older (6-10)/younger(1-5) with age, use ten records ages 1-10 and the r=.87 not one.
Higher Scores Can Be Difficult to Maintain
Contractual “obligations” for high aggregate satisfaction scores can be obtained, but the higher the aggregate score required the more difficult it will be to obtain as a mathematical necessity. Also, the less operationally useful the scores will be to either party to the contract. They are difficult to maintain and carry less actionable operational information than scores designed to capture satisfaction at middle ranges of a scale. Tying absolute, or changes, in satisfaction scores to compensation or other incentives may be valid in the short term but is ultimately doomed to failure when the observed scores become higher and more homogeneous. At some point, they begin failing to differentiate, providing less and less operational usefulness. Higher scores are more difficult to maintain and most likely to decline making them unsatisfactory for the intended purpose. Higher scores no longer measure what they did in earlier measurement periods making them more difficult to interpret or understand.
How should orientations like satisfaction be measured and why? First, opt for a measurement strategy that is continuous and unbounded. This maximizes observed and recorded variance and minimizes artificial truncation of observed empirical relationships necessary for understanding why the scores take the values that they do. Because it is difficult to provide such a measurement strategy practice use the best approximation of continuous and unbounded that is practical. Then, do not get hung up on the values. They are just placeholders for the information you really want. Focus attention instead on how likely the score is to change and whether what you are doing operationally changes the score. Use the score to model the psychological state it represents and then use the parameters to guide your business decisions.
Companies Need to be Cautious of Their Satisfaction Education
When managers and executives focus on higher scores they tend to misbehave in order to obtain them. AT&T provided a crib sheet to customers on how to answer its after service satisfaction survey. A Pensacola, FL hospital printed an image of a correctly completed (all top boxes) satisfaction survey on its emergency department discharge envelope. A major financial institution’s CEO emailed its employees instructing them to provide different (read higher) responses to loyalty and satisfaction questions on its annual employee survey. None of this is new. For years we have rightly decried grade inflation and teaching the test in schools to no avail. These practices are no different. The business school adage needs to be corrected. You cannot manage what you do not measure validly and measurement validity inevitably suffers when the goal is simply high scores.
TOG has demonstrated that we can change the scale score by changing the response options. But more on that later.
Dennis Gleiber, Ph.D., Chief Research Scientist