Wider Integration of Cost Considerations in Sample Size

In part 3/3 of this mini series, Head of Statistics at Statsols - Ronan Fitzpatrick - examines key trends in sample size determination which are being felt in trial design.

Wider Integration of Cost Considerations
in Sample Size

While sample size and power analysis are an integral part of trial design in clinical trials, it continues to be less widely used in other research contexts. The most likely reason is that in clinical trials a significant amount of resources are invested per trial due to regulatory oversight, the development costs of new treatments and that an incorrect decision in drug development or medical devices has such a prohibitive cost both economically and ethically. Thus a strict focus on sample size, power and other trial considerations is necessary and appropriate.

However, the most commonly used and taught methods for sample size estimation are ported almost wholesale from the clinical trials environment without consideration of the effect of cost, which is often understandably a primary driver of trial design decisions for most ordinary researchers. In practical terms this tension has led to a number of problematic
outcomes ranging from studies not justifying their sample size at all to widespread usage of useless post-hoc power calculations to research proposals being rejected or abandoned due to perceived low power.

While there is a healthy corpus of academic research into how to integrate cost considerations in sample size estimation, the adoption of these methods in widely used sample size routines and software has been slower. With the recent emphasis on the issue of research reproducibility, there should now be an increased emphasis on the design decisions which undermine scientific enquiry and credibility. It’s important to note that sample size and power reflect the low-hanging fruit of creating better research but need to be modulated to reflect the practical constraints researchers face in academia and industry.

With this in mind, the expectation is that cost based sample size approaches will quickly move from theory to practical implementation in sample size packages and software in the near future. This does not imply that concepts like power or precision will or should be ignored but simply that there will be a continued evolution and integration between these considerations and cost. This will allow researchers to see explicitly the trade-offs available in their study. In conjunction with methods and trial designs which can help alleviate excessive costs, the hope is that more robust research can emerge as a natural consequence of a more real-world interpretation of sample size determination.

References

Below you will find a selection of interesting papers and resources on the topic discussed above. These are only meant as an illustrative set of papers in the relevant fields and should not be considered a comprehensive summary of these areas.

Allison, David B., et al. "Power and money: Designing statistically powerful studies while minimizing financial costs." Psychological Methods 2.1 (1997): 20.
Bacchetti, Peter, Charles E. McCulloch, and Mark R. Segal. "Simple, defensible sample sizes based on cost efficiency." Biometrics 64.2 (2008): 577-585.
Bacchetti, Peter. "Current sample size conventions: flaws, harms, and alternatives." BMC medicine 8.1 (2010): 1.
Dattalo, Patrick. Determining sample size: Balancing power, precision, and practicality. Oxford University Press, 2008.
Nam, Jun-Mo. "Optimum sample sizes for the comparison of the control and treatment." Biometrics (1973): 101-108.