Scientists Rise Up Against Statistical Significance
I have often felt that I could "see" a difference in a load, yet I could not prove a statistical difference. Maybe I could prove it if I fired a thousand shots, or a million shots, but most of us don't have time to fire a million shots. There's nothing wrong with saying "I think this load might be a little bit better, but I can't prove it mathematically, at least not without more data."We should never conclude there is ‘no difference’ or ‘no association’ just because a P value is larger than a threshold such as 0.05 or, equivalently, because a confidence interval includes zero. Neither should we conclude that two studies conflict because one had a statistically significant result and the other did not.
We ... call for the entire concept of statistical significance to be abandoned ... we should not treat them categorically ... We must learn to embrace uncertainty.
And of course some people will disagree with you and say "you don't have enough data to prove your load is better," and that's fine, too. We can all agree that it would be great to have more data. But while we are waiting for more data, it's OK to make cautious observations based on the data that we do have.