Experimenting to Discover

Causal Reasoning in Science: Don’t Dismiss Correlations (the broken link was removed)

Box, Hunter, and Hunter were/are theorists, in the sense that they don’t do experiments (or even collect data) themselves.
...
Science is about increasing certainty — about learning. You can learn from any observation, as distasteful as that may be to evidence snobs. By saying that experiments are “necessary” to find out something, Box et al. said the opposite of you can learn from any observation.

William Hunter was my father. He did many experiments. George Box did many experiments. You are entitled to your opinions obviously but the claim that they only dealt with other people's data is not accurate. It is true they were world renowned experts on experimenting and had many people consult them about their experiments, for help: designing them, analyzing them, what to do next, how to improve the process of experimentation in their organization, etc.. While it seems to be implied in the post that such consultation was a reason to distrust their thoughts on experimentation I hardly think that is a sensible conclusion to draw. Most of those they helped were running experiments in industry, to improve results (not to publish papers).

They were, and are, applied statisticians (and though I am obviously biased, I think many would agree, 2 of the most accomplished in that field in the 20th century). What experiments need to be done is critical for an applied statistician. What matters is making improvement in real world processes. If you don't run the right experiments, you won't learn things to help you improve.

They worked on the problem of where to focus, in order to learn, quite a bit. One significant part of there belief was to have those involved in the work do the thinking about what needed to be improved. This isn't tremendously radical today but in the past you had many people that thought "workers" should do what the college graduates in their office at headquarters tell them to do. Here is one of many such example, from Managing Our Way to Economic Success by William Hunter:

The key is that employees at all levels must have appropriate technical tools so that they can do the following things:

- recognize when a problem has arisen or an opportunity for improvement exists,
- collect relevant data,
- analyze the situation,
- determine whose responsibility it is to take further action,
- solve the problem or refer it to someone more appropriate...

I don't have the book in front of me, but doesn't it start with an example on learning where you can use inductive reasoning and from the facts that you see you can draw conclusions and construct a theory that fits the facts. If so, it seems to call into question the idea that they claimed "[the] opposite of you can learn from any observation." is not actually accurate. They understood you can use inductive reasoning to create theories. You then use experiments to test theories.

The books is called Statistics for Experimenters, right? Not statistics for drawing conclusions when not doing experiments. When you are experimenting you can test whether beliefs you have are accurate and you can learn about things you try. Smart people can make guesses what will happen and be right. I know the authors would believe those knowledgable about the system in question are well suited to determine what variables to test. It is that knowledge that will lead to experiments that are likely to be effective.

The authors of the book were trying to help those that often failed to learn as much from experiments as they could. Far too many people still don't use the most effective statistical tools when experimenting.

They emphasized, consistently, the need for those doing the work to involved in the experiments. The job of statisticians was to help in the cases where advanced statistical tools and knowledge would be useful. The reason for those who do the work (are familiar with the process) is because they have knowledge to bring to what should be tried in experiments.

When I read through The Scientific Context of Quality Improvement, 1987 by George Box and Soren Bisgaard it seems to me it discusses the types of issues you raise: how do we learn without experimenting? I am not sure if it is just me, or if it clearly addresses that issue. Here is another, Statistics as a Catalyst to Learning by Scientific Method by George E. P. Box. And another, Statistics for Discovery.

There are many other sources, I am sure. They understood the importance of learning as much as you could from available sources. They just also understood the importance of experiments and learning the most you could from experiments. And the book, Statistics for Experimenters, was focused on the most effective ways to improve using statistics to learn from experiments..

Here is what Box, said in his own words about the objective (and it isn't proving the hypothesis):

[too many people ]"can’t really get the fact that it’s not about proving a theorem, it’s about being curious about things. There aren’t enough people who will apply [DOE] as a way of finding things out"

Statistics for Experimenters: Design, Innovation, and Discovery shows that the goal of design of experiments is to learn and refine your experiment based on the knowledge you gain and experiment again. It is a process of discovery. That discovery is useful when it allows you to make improvement in real world outcomes. That is the objective.