Power Combined with Precision Helps Improve Sample Size Justification

This month the ASA’s statement of p-values restated what statisticians have been saying for a long time...

That an overreliance on a single measure or cut-off will likely leave you ignoring real data and alternative statistical approaches which could ensure proper inference and decision-making.

In sample size justification, the power analysis approach is the effective twin of the null hypothesis significance testing (NHST) approach. Power analysis has been used with great success, over many decades, to address the long-standing and still far too prevalent occurrence of studies having insufficient sample size and power to make claims commensurate with a study’s aims and objectives. Overall, power analysis should continue to be strongly encouraged in study design as this has ameliorated many of the concerns researchers most fear from p-values in areas such as clinical trials.

However, while the power approach has been very successful, there is also no denying that standard power analysis methods are intrinsically tied to the NHST framework and its usage of p-values and significance levels. Moreover, it has also spawned its own issues of “standard effect sizes”, the continued usage of post-hoc power analysis and arbitrary cut-offs for acceptable power.

One solution to ameliorating some of these issues mirrors the approach for helping soften the detrimental impact of p-values, namely that we should consider not only the likelihood of Type I and II errors but also look at the expected precision for a parameter estimate. The most commonly used approach in research to doing so is the confidence interval; a useful tool though not without its own idiosyncrasies. Increasingly, regulatory agencies and journals are asking for more than the mere p-values for an analysis but also the actual parameter estimate and its associated confidence interval.

To reflect this, it is perhaps time that researchers providing sample size justification mirror this approach and give not only the power associated with a given sample size and expected effect size but also to give the expected confidence interval width based on these assumptions. The additional context provided by this information may make researchers more cognizant of what they are considering a “significant” effect for their proposed study and help highlight the differences between practical and statistical success early on.  Knowing that your high power study is actually associated with a confidence interval close to the null is as vital before a study as afterwards.  In early phase clinical trials and survey methodology, precision based sample size is already well entrenched so these methods are easily available.

This precision based approach is only one possible approach to reforming or buttressing the power approach to sample size justification. There is a growing literature on sample size justification taking into account cost considerations, Bayesian approaches (both pure and frequentist hybrids) and information theoretic methods. Additionally, continued increases in more complex design considerations such as group sequential methods, sample size re-estimation, simulation approaches and multiple endpoints mean that sample size justification continues to evolve as quickly as statistics as a whole.

Here at Statistical Solutions we are always looking for additional input, collaborations and advice on suggested sample size methods and approaches so if you have anything which is of particular interest to you, be sure to get in touch with us. Our user input has and will continue to drive the evolution our sample size solution nQuery.