Experimenting to Improve Sleep Quality

comments on: Can a Humidifier Help You Sleep Better and Snore Less?

After doing some research I learned that humidifiers have helped folks snore less. So, after some more research, I picked up a slick little ultrasonic humidifier and gave it a try. Now, it’s been less than a week which I know isn’t enough to get too excited about statistically speaking. But one thing is becoming crystal clear…it’s most definitely helping me sleep better.

Interesting post, which includes control charts showing the impressive progress.

"I’m also still trying to figure out what caused the three special cause signals in January." One nice aspect of improvement is sometimes you can make a system improvement that even without knowing the causes of previous problems, the new improvement stops those from happening again. Maybe that won't be the case this time but maybe it will. Health related issues are so touchy that I could imagine it is something like a couple bad factors stacked on top just push things over the limit. So being a bit tired and say too low humidity and you didn't drink quite enough liquid and sleep quality is bad but just 1 or 2 of those and it might be a bit worse but not horrible.

Special cause signals will be more frequent if several factors together amplify each other (and they rarely happen together so those amplified results are rare). What happens is those rare amplified events will be special, outside of the system that generates that regular variation when they act alone but when all that variation lines up just right the result will be outside what is normal (due to the very large change in the result for that special case where the individual factors acting together (amplifying) create a very large change in the result.

Related: Gadgets to Mask Noise and Help You Sleep or Concentrate - Apply Management Improvement Principles to Your Situation - Zeo Personal Sleep Manager - Using Control Chart to Understand Free Throw Shooting Results

Shared Principles for Managing People Engaged in Diverse Tasks

This post expands on my response to Michael Sweeney's (Cloud Infrastructure Engineering for Salesforce) comments on Linked In:

This is a great blog to follow [Curious Cat Management Improvement Blog] if you aren’t aware of it. I agree with John, in general, but in this post I felt the missed one basic thing that I’ve seen in my own career- software is not physical and people who write great software and people who make great physical things MUST think (and therefore be managed) differently. The crossover between Agile and Lean thinking, to me, is the ability to identify non-value added activity (waste) in the Toyota sense, and empower small teams to make decisions and charge forward in the Agile sense. Getting a combination of software and hardware thinking together will be the key to winning the Cloud Wars and moving into the Fourth Industrial Revolution.

Thanks for your comments. I do agree that the system within which people are operating determines how they must be managed. There are definitely features of software development that are significantly different than manufacturing scalpels or basketballs or tables. As there is a difference between a surgical team in an operating room, road construction, mining, editing books, investment banking, manufacturing industrial robots, researching new drugs, manufacturing drugs, teaching in a university, maintaining plane engines, coaching an athletic team...

I see universal principles of management (respect for people, customer focus, continual improvement...) that cross all different human enterprises. How those principles should be manifest in a particular situations depend on the work being done, the management system that is in place, the individual people involved, the specific focus of the effort right now... The way those principles are manifest will look very different in all the varied types of organizations we create and the different work and processes used within those organizations.

John Hunter, presenting at a Deming management seminar in Hong Kong

It is interesting (on the software v not software divide) to note that 100 years ago what was manufactured didn't contain software elements. And the manufacturing process also didn't involve software. That isn't very often the case today. Think of all the manufactured things you use and a high percentage (measured by the cost of the manufactured goods) have software components (cars, phones, appliances, speakers...) and they are built with a great deal of software involved in the manufacturing process.

In addition, the sales process and other processes involved in the organization doing the manufacturing rely heavily on software. As you say "Getting a combination of software and hardware thinking together" is indeed key today and will be continue to be in the future. While relying on software as part of the manufacturing process (and in the supporting processes) isn't the same as developing software the thought process on how to use software within manufacturing systems and how that software should work, be adjusted... is very different from the work of manufacturing tires 100 years ago.

I also discuss related ideas in: Deming and Software Development.

Related posts: How to Manage What You Can’t Measure - Create a System That Lets People Take Pride in Their Work - Good Process Improvement Practices - The Importance of Management Improvement - Thinking Required, No Simple Management Recipe to Follow -Unpacking the Components of Hard Work to Design Better Work Conditions - Do We Need to Find Management Ideas from Our Industry? (No) - Avoiding Difficult Problems

Avoiding Difficult Problems

Response to: Toyota is Admired for Good Reason… But About Those Rotating Job Shifts…

Thanks for another good post. It points out that while Toyota does many things very well they have opportunities to improve. And I am sure they agree they have a huge number of things to improve.

I agree, this rotating shifts seems like something important to try and improve. One of the things that happens in many organizations is that working on things that are going to make lots of people mad are often avoided. This rotating shift work seems like something that is likely to make people upset.

Even if you worked on improving it, likely during those PDSA cycles many people would be annoyed. And if you were working on it, you could get blamed, you could be tarnished as someone people didn't like it and didn't appreciate "you doing this to them." If, on the other hand, no one is making significant efforts to improve, even if people are annoyed it is at some amorphous policy and usually doesn't stick to 1 person.

If the dis-satisfaction does accumulate toward 1 person (say the plant manager) that person then often will push through and deal with it - or assign someone to deal with it and stay on them to make sure they do the difficult work (even if doing so will make that person's life much more difficult).

Now I would hope Toyota is better managed than that and doesn't shy away from important work just because the people that would take the lead have strong reasons to avoid working on that task.

There are often so many things to improve that it isn't too hard to shift the focus away from something that no one wants to take on. Sometimes though things are so important they must be taken on. Usually those would be more direct and obvious problems. Systemic problems that cause great damage but in a less visible way (such as rotating job shifts) are less likely to be addressed.

Coping with this issue (of avoiding unpleasant, systemic and long term rather than acute problems) is one of the things that separates great corporate culture from decent or bad corporate culture.

If there are fairly obvious or fairly easy improvements those would likely be acted on. There are, rarely, but still sometimes, instances where those vocal or politically powerful individuals who would lose out in a fairly obvious improvement will prevent action.

I am pretty sure Toyota doesn't have fairly easy improvements on this issue, that they must realize is important, that they are just not getting to. I am a bit less confident that Toyota isn't avoiding trying to find solutions using a PDSA type effort because it would be really hard and risky work for whoever takes it on. Also Toyota can try to rely on the good will built up through their many efforts to expect enough employees to say that they will accept the suffering caused by this policy for the good of the everyone.

I am not convinced there are not ways to improve the situation. And I am pretty confident it is important enough to try. And I believe (though I might be wrong) with a concerted effort of knowledgable people improvements that would make a big difference in the quality of life could be achieved. I am not so certain those people involved in leading the effort would be seen in great lights though even if they "succeed." People are much more likely to remember negative consequences to them personally, even if they gain much more than they lost overall. And most will remember the effort negatively if they lost only a little but overall the gains (to the overall organization) were much bigger: in such cases many will hold onto feelings that they were harmed by those that took action.

Related: Respect for People Doesn’t Mean Avoiding Any Hint of Criticism - The Importance of Critical Thinking and Challenging Assumptions - Unpacking the Components of Hard Work to Design Better Work Conditions

Code Software to be Robust and Easy to Update Quickly

Comment on: Correction vs Prevention in Software Development

I think both prevention and designing the software and management system so that rapid correction is possible are important. While rare events may be difficult to prevent when looking at each instance there are styles of coding that make more "edge case" failures more likely. Coding so that the system is as robust as possible is wise but you should also realize those efforts will likely not be perfect and so designing in visible notifications of failure and coding so rapid correction is possible is necessary.

In addition to the need to update quickly for bugs, software should be easy to update due to changing requirements and to aid in continual improvement efforts.

Related: Improving Software Development with Automated Tests - Software Supporting Processes Not the Other Way Around - Building a System to Reduce Interruptions for Software Developers
- Use Urls: Don’t Use Click x, Then Click y, Then Click z Instructions

The Sociology of Organizational Change

Comments on: Researching Laggards

The Late Majority is the stabilizing force, the repository of institutional knowledge that slowly absorbs and productionizes the ideas proven to best serve the organization. They aren’t as eager for change as the Early Majority, but they’re happy to adopt proven practices.

The Laggards provide challenge the Instigators most directly, questioning or outright denying the value of a new idea, and provide the most vocal and active resistance. However, their direct criticism may inspire the Instigators to find unexpected common ground and more effective solutions than they otherwise might.

Yes, I think laggards really are common. The grey area between laggards and late majority may be pretty large. Many are swayed by the critical mass of opinion. At first they seem like laggards because they side with them, as the momentum grows they side with late majority...

True active laggards fighting well after the critical mass makes it obvious the culture expects the "new" behavior" isn't a huge group I don't believe. But getting the point where the those siding with laggards switch to siding with late majority is a very challenging point to reach for most significant changes.

How you help change the culture of an organization requires understanding the inertia against change in most organizations and the strategies that are useful in creating the critical mass to accept new ideas and cultural attributes as the new normal.

Related: Why Do People Fail to Adopt Better Management Methods? - Podcast: Building Organizational Capability - Culture Change Requires That Leaders Change Their Behavior - Transforming a Management System – A Case Study From the Madison Wisconsin Police Department - Change Management: Create a Culture Seeking Continual Improvement or Use Band-Aids? - Communicating Change - Building Adoption of Management Improvement Ideas in Your Organization - Grow Your Circle of Influence

The Problem Is Exacerbated by Fear of the Word Problem

Comments on What’s Another Word for “Problem”?

I think this is a wise recognition: "may need help in more areas than process improvement."

Fear is likely a part of the problem (yes problem). Such a desire to ignore problems and the word problem can also be greatly enhanced with performance appraisals systems that create a mindset that is focused on hiding potential issues that may reflect poorly on those appraisals...

The problem with the word problem is often not as simple as it may seem at first. Changing the word used may do a tiny bit of good but not much. The underlying issues that cause people to think problems are something to not acknowledge is not something solved by avoiding the word.

Related: If Your Staff Doesn’t Bring You Problems That is a Bad Sign - The Problem is Likely Not the Person Pointing Out The Problem - Is Using the Words Resources or Assets When Talking About People the Problem? - The Importance of Making Problems Visible

Should I be in the Check Phase of PDCA Daily?

Below is my response on closed forum about whether doing the "check" phase of PDCA daily was too often. I expanded on my comments there a bit in this post.

The check/study phase should be reviewing the results of the experiment done in the Do the experiment phase. "Checking" how things are going during the experiment makes sense but that isn't the check/study phase of PDSA .

For example, you don't want to pay no attention during the experiment and then look at the data and discover the data shows obvious signs the operational definitions were not clear, or the process is providing very bad results. So you need to have those doing the experiment paying attention daily.

Remember one key to using the PDSA cycle is to turn through the whole cycle quickly. Daily would be exceptionally quick. Moving through the whole cycle in 2-6 weeks is more normal. Organizations successful using PDSA will quickly turn the cycle 4+ times for a specific effort (often the 2nd, 3rd... times through are much faster than the first time through).

More on how to use the PDSA well:

• Resources for Using the PDSA Cycle to Improve Results
• The Improvement Guide is a book anyone using PDSA should read and consult frequently as a resource (blog post on the book)
• Keys to the Effective Use of the PDSA Improvement Cycle

How to Improve at Understanding Variation and Using Data to Improve

My comments based on a question on, How to Use Data and Avoid Being Mislead by Data:

Thanks for this post John. This is the part of Deming’s teaching that I often struggle with (understanding variation). I read Wheeler’s book Understanding Variation and it helped me with the concept, but I am challenged trying to apply it where I work. I often am not sure what to measure and if I do, I’m not sure how to measure it. Folks appreciate my burn down charts showing trends, but this is about the best I’ve been able to do. Do you have any recommendations on where I can look to help me get better at this?

Getting better at using data is a bit tricky, so struggling is fairly common.
Probably the easiest thing to do is to stop reacting to normal variation (caused by the system) as if it were special. This isn’t super easy but it is the easiest step. And it does make a big difference even if it doesn’t seem very exciting.

The idea of actually using data properly provides big benefit but it much trickier. Don Wheeler’s book is a great start. Making predictions and evaluating how those predictions turn out is also valuable. And in doing so often (though not always) it will also spur you to collect data. This process of predicting, figuring out what data to use to help do so (and to evaluate the results) and considering the result of the prediction and how well the predictions overall are working can help.

You learn what data is often useful, you experiment with real data and real processes and you learn what needs to improve. If you are at least somewhat close to using data well then just doing it and learning from your experience is very useful. If you are really far off the experience might not help any 🙁
The links in the post above I think provide some useful tips (and the links within the posts they link to…).

More: Measurement and Data Collection

If you don’t have an answer for how you will use the data, once you get it, then you probably shouldn’t waste resources collecting it (and I find there is frequently no plan for using the results).

It isn’t uncommon that the measures you would like to have are just not realistically available or are hard to determine. How to get started in this is one of the tricker pieces in my experience. It is a place where consultants may be very helpful. If that isn’t an option another possibility is just to ask others at your workplace for ideas for metrics (there are issues with this and a big one is that many metrics will more likely to lead you astray than actually help).

This can also be an area where seeing what others are using can be helpful. Because it is hard to think up what are great metric seeing what others are doing may provide insight. Of course, the ideas must be evaluate for whether they would work for you (even if they are right for others they may not be right for you – and many are not really right for others it is just a thing they measure and while they have associated it with good things maybe they are wrong (correlation but not causation]).

Don't Use Targets as a Management Tool

comments on: Deconstructing Deming XI B – Eliminate numerical goals for management


I agree with the comments that targets are unwise. There is one sense in which I think the idea of (but not actual) targets can be useful and that is in setting the scope.

If we want to find an improvement that is immense (versus small continual improvement) that can set the expectation of how we approach improvement, including an understanding that we are going to have to really make big changes in how things are done.

I have written more about this, here:

Deming on Targets

Basically I don't see that scoping "target" as really a target but it is similar so if you want to see it that way, then in that sense I can see a "target" as useful.

Related: Innovation at Toyota - Targets Distorting the System - Righter Incentivization

The Importance of a Work Culture That Values and Supports Critical Thinking

Critical thinking is tremendously important. I have come to think it might be the most important precursor to management improvement (evidence based management, continual improvement...).

One of the big issues is for people to understand thinking critically about ideas isn't an insult to whoever came up with the idea being discussed. This isn't something I would have thought of as important until seeing so many cases where people are not comfortable discussing ideas (and weaknesses in those ideas) in the workplace.

Comments prompted by: thinking critically

Related: A Good Management Culture Encourages the Debate of Ideas - Respect for People Doesn’t Mean Avoiding Any Hint of Criticism - Dangers of Forgetting the Proxy Nature of Data

Root Cause - Addressing Systemic Causes Not Symptoms

Comment on Just Ask Why Five Times? Effective Problem Solving for #Lean or #LeanStartup Doesn’t Start or End There

> Would you agree that complex systems rarely have a single root cause?

Yes. And also "root cause" is a neat concept but in reality it is not usually "true" but an a sensible acceptance of a cause that is systemic enough and addressable enough to consider "root." It isn't that there is this "true root cause" that created the current problem. There is a way to look at the issue and find a deeper cause that will allow you to address it and improve the future performance of the system.

Depending on how you look at the problem there can be many different "root causes" that are sensible from their different perspectives. The important thing is by aiming to fix root/systemic problems you will not just treat the current symptom you are dealing with today but eliminate future problems from occurring. If you are doing that, you are doing well.

If you start noticing that you are addressing problems that could have been addressed in previous attempts to address root causes, you can exploring whether going further in each attempt makes sense. It isn't as simple as this but if you notice you addressed a systemic problem at the "branch" level effectively for example. So if you had 3 fixes that did stop future problems on each of the branches but on the fourth fix you looked and said hey all 4 of these connect to this larger branch (or tree trunk - which is connected to a real "root") I would say asking if you should have addressed the "next why" may well make sense.

Related: Root Cause, Interactions, Robustness and Design of Experiments - Address the Root Cause Instead of Finding the Person to Blame - Poor Results Should be Addressed by Improving the System Not Blaming Individuals - Firing Workers Isn’t Fixing Problems - Examine the System, Don't Look to Blame a Person - Why Do You Ask Why?

Spend More Time Doing What You Do Well

Comment on: Is it Better to Work on Strengths or Weaknesses?

As you say to the extent your weaknesses are things you have to do spending time improving them usually makes sense. I think often the most productive thing is to spend time working on the system to maximize the use of people's strengths and minimize the use of their weaknesses. This often has a big impact without much effort.

And when you do that it is often the magnitude of strengths that makes a big difference. So you can avoid dealing with much of the weaknesses in the team and focus most effort on the strengths. And when you do that my getting even better at x allows the improvement not to just be x * 1.1 but (x * 1.1) + (y * 1.3) + (z * 1.4) - (if say I am now 30% "better" than y at the task and 40% better than z. Obviously it doesn't work so cleanly in the real world but that concept that you can get way more improvement normally by adjusting the way work is done than just by having everyone get less bad at the stuff they really should avoid doing most of the time.

You do also have to pay attention to the long term, so if someone wants to move into supervision but has some weaknesses they need to address and strengths to improve working on that makes sense.

Related: Take Advantage of the Strengths Each Person Brings to Work - Helping Employees Improve
- Many Good Employees Want to Continue to Do Their Current Job Well - Lessons for Managers from Wisconsin and Duke Basketball

Change and the Management System

comments on: Better Change Leadership as a Countermeasure to “Resistance to Lean”

I agree, the problem isn't change but the process for change. There are many smart things to do to help the process work better.

The most important thing though is the entire management culture. Tactics can help change efforts be more successful. But if the culture is hostile to continual improvement (fear based, performance appraisal based, target based, blame based, imposing from on high...) the tactics are working in a difficult situation. Still a good idea, but no matter what tactics are used it will be a challenge.

When the culture has the right environment (PDSA, seek to continually improve the system, respect for people, support for innovation, understanding of variation in results, seek process weakness to improve not people to blame, provide training...) change is set in a system where resistance is much lower (and in very highly functioning systems it is encouraged not resisted).

Change tactics are still sensible but often they are baked into how things are done. As you grow more toward becoming such an organization change tactics fade into the normal process and the resistance fades too.

Related: Communicating Change - People Take Time to Believe Claims of Changed Management Practices - Encourage Improvement Action by Everyone

Learning From Process Improvement Efforts

Comment on: How to Improve (at just about anything)

1. The classic way:

Do – make an improvement
Do – change a process
Do – implement some training
Do – install a system

When you have been through the 4 do’s keep right on doing.
2. The recommended way:

Plan – develop an idea or innovation, work out how you will implement it.
Do – carry out the plan on a small-scale, test it to see if it works.
Check – study what happened, did the plan work? If not why not? What can you change?
Act – adopt the change and roll it out, abandon it or learn from it and adapt it.

Another huge benefit to the PDSA cycle in my experience is to learn. I can't remember how many times I would see in the do-do-do-do organization that

do#1 was x
do#2 was y
do#3 was x again
do#4 was z
do#5 was y again

Um, ok, yeah why are we trying things we already know don't work (they are presented as fixes not, as well this old way wasn't great but jeez it was much less bad than the mess we have now so lets go back). Why are we thinking x is going to work when we just dumped x because it wasn't working? PDSA makes you think about the process, study the historical data and document your predictions. The learning will

Related: How to Improve - Document Your Decisions to Learn More Effectively - Learn by Seeking Knowledge, Not Just from Mistakes - Write it Down

Deming Wanted Managers to Understand the Systems They Managed and to Visit Where the Work was Done

Comments on Deming wasn’t a fan of Management by Walking Around

Deming’s view is entrenched in Lean management practice in the form of “Genchi Genbutsu”, literally “go and see” at the “real place”. Where practitioners of Management by Walking Around merely visit the workers for a chat, practitioners of Genchi Genbutsu stay with the workers to understanding what is going on.

True, and the details you provide are important. It wasn't managers going to the gemba Deming was against. What mattered is the system. Some people did Management by Walking Around (MBWA), even decades ago, in a useful way - with understanding. But most did not.

Gemba walks are much more likely to be useful (because the expectations are for a more engaged leader) but there are plenty of times those are not done well, and are no better than bad MBWA. Jim Womack has a book out on Gemba Walks which provides good details to managers on what they should do (and what Deming wanted them to do).

Related: Out of Touch Executives Damage Companies, Go to the Gemba - Leadership and Management - Jeff Bezos Spends a Week Working in Amazon’s Kentucky Distribution Center (2009) - Customer Gemba

Building a System to Reduce Interruptions for Software Developers

I feel strongly about the damage done by interruption to maker's focus. My appreciation for this damage to system performance was greatly enhanced as I became a software developer.

In my work prior to software development I could see that interruptions were not ideal but they were not so damaging to my work and they do also have value. But work that requires extra focus, and software development sure did for me, the damage increasing greatly.

Over 20 years ago at the Madison Area Quality Improvement Network they had a simple visual management sign at everyones office and cube. You could dial between various settings - GREEN available (free to interrupt)... RED busy, don't interrupt... There were 3 to 5 settings. It worked very well for them.

Of course such a system is highly dependent on the overall management system. Just putting that up in most offices would likely fail. MAQIN was an organization that embodied modern management practices (Deming, lean, respect for people, organization as system, etc.). It worked very well for them.

When building a software development team I tried to protect developers from interruption while having very open communication between developers and program managers (product owners). As is often the case, the real roles didn't line up exactly. In our organization, the people that needed us to develop software (to varying levels) didn't want to take on the responsibilities that agile would suggest (setting priorities etc.). Nor did the that organization want to deal with the conflicting priorities of the various people in their organization had for software development. As part of making things work, I took on a bit of the priority setting roles.

One of the things I did was explain to those that had us do software development was why it was critical to allow software developers to have uninterrupted time to focus. I shared various articles and reports over time and would talk about this as I interacted with them. I encouraged people to talk to me first (at that time I had transitioned from developer/software-development-program-manager to nearly entirely a program manager role).

We also had weekly meetings (when it made sense) with the "product owners", me and the developers working on the software. We scheduled additional meetings when necessary.

I also encouraged people to use email when possible to just check on things, ask simple questions, ask to meet. Again I explained how just going up to talk to the developer could impose significantly higher costs to them being able to focus of their work than there were for interrupting my work or others that had work that could more easily be interrupted with lower consequences.

When there were very important deadlines I would increase these instructions not to interrupt the software developers. And I would talk to the developers about how things were going and if the developers wanted help pushing back a bit I would then do so - mainly by trying to work on the system (provide a set time to allow needed discussions, coach the "product owners" on the difficulty caused by interruptions etc.).

As you might expect this worked better with some people than others. It made a significant difference though. And in our organization it was actually a bit less effective because the software developers were all so nice. They could say how interruptions damaged what they could do systemically but were always super friendly whenever they were interrupted. I was by far the most willing to actually disappoint people face to face in the name of improving overall performance.

It is very hard to overestimate the systemic nature of all of this. By delivering great results we were able to reinforce that if we followed the modified Deming/agile methods we were using were important. Those needing work from us ranged from very skeptical to accepting of those ideas but our performance (both in meeting their desires for software and for interpersonal interactions) made them willing to go along with the methods we said were important.

Related: Deming and Software Development - Mistake Proofing Deployment of Software Code - Creating a Culture that Values Continual Improvement

The Value of Putting Pen to Paper

Comments on Learning by Writing… by Hand

The psychology behind the learning advantage of handwriting is starting to be understood... [Carol Holstead] "It turned out my theory was right and now is supported by research. A study published last year in Psychological Science showed that students who write out notes longhand remember conceptual information better than those who take notes on a computer."

I am also a fan of technology. And also a fan of learning and paying attention to research. Pen on paper has advantages for learning that technology has yet to equal. At the same time technology has many advantages also.

We seem to understand the advantages of using technology fairly well but under-appreciate the advantages of pen on paper. To make sure we don't lose out due to this bias we should think before we accept that pen on paper isn't worthwhile.

From a post I wrote in 2005, Measurement and Data Collection

I believe, it is better to focus on less data, really focus on it. My father, Bill Hunter, and Brain Joiner, believed in the value of actually plotting the data yourself by hand. In this day and age that is almost never done (especially in an office environment). I think doing so does add value. For one thing, it makes you select the vital few important measures to your job.

Lots of data will be kept in computers and that makes sense. But putting pen to paper has value that we too quickly dismiss.

Related: Experience Teaches Nothing Without Theory - The Illusion of Knowledge - Write it Down to Improve Learning (Ackoff)

Learning Can't Take Place Without Theory

Response to: The Secrets of Lean

I think you make good points, but I think you make a mistake stating:

"This system of learning has come from experience, not theory."

For some reason, culturally, we created this idea that theory was about disengaged people (away from the gemba) thinking in a way disconnected from practice. But this is not what theory is.

Learning can't take place without theory. Experience doesn't lead to learning. Experience with a theory and an informed observer that questions what they see can lead to learning.

It seems to me accept theory that is separated from the gemba as what theory is and then say theory is not useful, experience is. But we are making a mistake when we think this way. The problem we see in theory being used disconnected from feedback from the gemba is a bad use of theory. But the problem is not that theory needs to be eliminated, but that theory disconnected from the gemba is not useful in learning about systems and improving our organizations.

Related: Experimentation is an iterative process - Effort Without the Right Knowledge and Strategy is Often Wasted

It isn't Fair to be Judged for Performance Outside Your Control

My response to Simon Guilfoyle post

It’s also necessary to acknowledge that multiple variables affect crime rates; factors such as economic cycles, substance abuse, the weather, societal influences, changes in legislation, and so on. None of these are directly within the gift of the police to influence. Also, what about where the police cause an increase in reported crime by having the temerity to find someone carrying a weapon? Surely proactive problem-solving should not be discouraged on the basis that finding hitherto unreported criminal offences is incongruous with an over-simplified crime reduction narrative...

I would say many of the examples are outcome measures of the system - which is a better measure than is often used.

However, they are often beyond the control of individuals and even police departments overall (many other factors play a part - economic development, social services system, education system...).

Likely more directly relevant the measurement error is often so high that the figures have more to do with measurement than the actual outcome. And when the figures are being used to blame then it dramatically increases the likelihood the figures will be a poor representation of outcomes.

I would rather see more focus on outcome measures. We should also reduce incentives to misreport data (often blame related).

I think the issues you raise about the system being larger than the police can tackle alone should be a reason to INCREASE the view of the SYSTEM. The important system is LARGER than the police department. When we have institutional walls that break up the system we need to find ways to knock down those walls so the system can work together. Granted this seems nearly impossible given how much difficulty we have even just breaking down barriers inside tiny pieces of our organizations.

Nevertheless that is where the focus should be. We shouldn't decide the outcome measures are not fair given where we put organizational barriers. We should decide that we need to realize when we cut funding for x it drives bad results in section a. And when we allow y to retain fitfully outdated management practices that doesn't just impact their ability to succeed it likely creates lots of other problems all over the place.

Related: Be Careful What You Measure - Millennium Development Goals

Data Must be Understood to Intelligently Use Evidence Based Thinking

All metrics are wrong, but some are useful

Metrics might tell you something about the world in a quantified way, but for the how and why we need models and theories … metrics are generated must be open and transparent to make gaming of the system more difficult, and to expose the biases that are inherent in humanly created data

True, understanding the proxy nature of data (and how well or questionably the proxy fits) is important.

Data can't lie but we often make it easy for others to mislead us when we don't understand (or question) what the data really means (what operational definitions were used in the collection, etc.).

Related: Operational Definitions and Data Collection - Actionable Metrics