Thoughts on the ASA P Values Statement

“When a measure becomes a target, it ceases to be a good measure”

The ASA’s recent statement on the issue of p-values in research is a timely and important contribution to what has been a long-running and oft-contentious debate.

Creators of statistical software for both advanced and general users like Statistical Solutions have always been very aware of the need to balance the needs of researchers in terms of speed and efficiency with the need to educate and encourage end users to use proper statistical practice.

The p-value has become the prototypical example of how researchers’ understandable need for definitive answers and measures from the statistics community lead to a statistic becoming overused, abused and stretched beyond its, very obvious, limits. However, the overwhelming evidence of practices such as p-hacking show the folly of trying to reduce the scientific process to a single measure or, worse still, a single declaration of “significance” based on a tenuous cut-off. Research conducted in such a way leads only to frustration for researchers attempting statistical alchemy with little understanding or guidance on the purpose of the methods and to statisticians asked to bring only a tiny fraction of their knowledge to the fore in collaborations and also seeing their subject brought into disrepute. 

The long-term solution to this problem will only come about if the interests of researchers and statisticians can be aligned with best practices of study design and statistical analysis. In both the ASA’s statement and in the excellent supplementary responses we see that the solution is not simply to put forward some better alternative statistic (though these will be of some use) but that we need reform how study design and statistical methods are conducted and how they are taught. Better statistical reporting using methods such as confidence intervals or Bayes factors form part of the solution but the moves towards greater access to data, protocols and code, mandating the creation of pre-trial study design justifications and hypotheses and a greater emphasis on practical guidance and alternative methods in statistics and trial design classes are also needed. The ultimate objectives of this reform may even require a more fundamental reimagining of how we reward and incentivise researchers to ensure that “publish or perish” does not continue to mutate into “p-hack or perish”.

In the interim, all moves towards greater transparency and reproducibility are to be welcomed both in academia and in industry. Here at Statistical Solutions we continue to strive to create software which dovetails both with these wider philosophical and regulatory changes and with the continued innovations we see in statistical practice and methods. As the developer of the sample size justification software nQuery Advisor, we are well aware of the need to help make better research practices available to researchers in an intuitive, easy-to-use way while also ensuring the highest standards of research ethics and education. We hope the ASA’s statement will provide a launching pad for what will be fascinating shifts in our field and we look forward to continuing to collaborate with statisticians and researchers to provide the solutions which can hopefully make those changes that bit more realizable.