• Home
  • What Could we do Better?
  • Instituting a Management Improvement Culture in Your Organization
  • Find the Root Cause Instead of the Person to Blame
  • Good Process Improvement Practices
  • Management is Prediction
  • The Purpose of an Organization
  • Performance Without Appraisal
  • Manufacturing and the Economy
  • Practical Ways to Respect People
  • 10 stocks for 10 years
  • Deming and Toyota
  • Curious Cat Management Improvement Articles
  • John Hunter
  • Institute for Healthcare Improvement
  • Superfactory
  • Management Improvement Jobs
  • Deming on Management
  • Management and Leadership Quotes

    I am now using this blog to re-post some comments I make other blogs. For my full management blog see the Curious Cat Management Blog

    Sunday, August 29, 2004

    Dangers of forgetting proxy nature of data

    Topic: Management - Data
    Written in relation to a thread (Staff Attitude) on the Deming Electronic Network (but the DEN archive seems to indicate I never posted this message).

    We use data to act as a proxy for some results of the system. Often people forget that the end result is not for the number to be improved but for the situation to be improved. We hope, if the measure improves the situation will have improved. But there are many reasons this may not be the case (one number improving at the expense of other parts of the system, the failure of the number to accurately serve as a proxy, distorting numbers, etc.).

    I find something I learned from Brian Joiner an excellent summary - which I remember as:

    Data (measuring a system) can be improved by

    1) distorting the system

    2) distorting the data

    or 3) improving the system (which tends to be more difficult though likely what is desired)
    Brian Joiner's book, 4th Generation Management is a great book for managers.

    It seems to me another danger to examine when looking at "results" is whether the proxy nature of the measure is deminishing. It seems to me the situation where turnover increases because people are "growing" (and therefore the assumed value as measure of employee dissatisfaction is diminished) is a situation where the system changes and reliance on the proxy of turnover to measure dissatisfaction breaks down.

    This, to me, relates to the concept that all models are wrong but some are useful (which I learned from George Box) and leads me to believe that measures such as turnover and sick leave use can be valuable. And can be helpful in measuring the results of attempting to improve the system.

    Another version of that concept states that all models are wrong but in a given world a model may be useful. In a certain state of equilibrium it may be that you could track increases in turnover (and use it as a proxy for employee satisfaction) even if that was not perfect measure for why all turnover happened. But then once the system shifted to a new state the turnover rate no longer served as an effective proxy. Reality is always more complex than the measures we use to try and understand it so even while the model is useful it is not a completely accurate representation of reality.

    It seems to me that once we begin collecting data we often put very little thought into whether the data continue to be valuable (assuming we did so to begin with) and continue to serve the purpose they once did. And we often fail to explore whether changes in the numbers (which we call results) are representative of the "true results" of the system or they fail to do so.


    Blogger David Ranney said...

    This phenomenon appears in many different ways. Companies rise and fall based on "numbers." It's not uncommon for companies to hire contractors over regular employees in order to increase their profit to employee ratio, even if contractors cost more money. A contractor is not technically an employee, it makes the statistic look better, and that's what's seen as important.

    Theoretically you could establish well thought out metrics for a company's performance which resist that kind of tampering, but nobody within the financial community seems terribly interested in doing so.

    3:35 PM  

    Post a Comment

    << Home