Dangers of forgetting proxy nature of data

Topic: Management - Data
Written in relation to a thread (Staff Attitude) on the Deming Electronic Network (but the DEN archive seems to indicate I never posted this message).

We use data to act as a proxy for some results of the system. Often people forget that the end result is not for the number to be improved but for the situation to be improved. We hope, if the measure improves the situation will have improved. But there are many reasons this may not be the case (one number improving at the expense of other parts of the system, the failure of the number to accurately serve as a proxy, distorting numbers, etc.).

I find something I learned from Brian Joiner an excellent summary - which I remember as:

Data (measuring a system) can be improved by

1) distorting the system

2) distorting the data

or 3) improving the system (which tends to be more difficult though likely what is desired)
Brian Joiner's book, 4th Generation Management is a great book for managers.

It seems to me another danger to examine when looking at "results" is whether the proxy nature of the measure is deminishing. It seems to me the situation where turnover increases because people are "growing" (and therefore the assumed value as measure of employee dissatisfaction is diminished) is a situation where the system changes and reliance on the proxy of turnover to measure dissatisfaction breaks down.

This, to me, relates to the concept that all models are wrong but some are useful (which I learned from George Box) and leads me to believe that measures such as turnover and sick leave use can be valuable. And can be helpful in measuring the results of attempting to improve the system.

Another version of that concept states that all models are wrong but in a given world a model may be useful. In a certain state of equilibrium it may be that you could track increases in turnover (and use it as a proxy for employee satisfaction) even if that was not perfect measure for why all turnover happened. But then once the system shifted to a new state the turnover rate no longer served as an effective proxy. Reality is always more complex than the measures we use to try and understand it so even while the model is useful it is not a completely accurate representation of reality.

It seems to me that once we begin collecting data we often put very little thought into whether the data continue to be valuable (assuming we did so to begin with) and continue to serve the purpose they once did. And we often fail to explore whether changes in the numbers (which we call results) are representative of the "true results" of the system or they fail to do so.