To increase power:
- Increase alpha
- Conduct a one-tailed test
- Increase the effect size
- Decrease random error
- Increase sample size
Sound so simple, right? The reality is that although these 5 ways all work theoretically, you might have trouble with some in practice.
1. Alpha is pretty well set at .05 for most scientific studies (there are rare exceptions). So you’re not going to get away with this one.
2. One-tailed tests have nothing wrong with them theoretically. However, there are only a few tests in which they’re even possible–namely t- and z-tests. So they’re not used much and have therefore appear dubious. Most reviewers won’t believe you that you really were hypothesizing that direction (even if it’s obvious). They will assume you’re trying to artificially get that p-value lower (it has been done).
So, once again, unless you’re in an enlightened field, or one in which one sided tests are commonly done, you can forget this one too.
3. Unless you’re coming up with a more precise way to measure your constructs, it’s likely that the effect size is a big as it’s going to get. Keep going.
4. Aha, something we can work with. There are two great ways to reduce random error. One is to make it not random. Explain it with a control variable, turning into explained variation.
The other related way is to use some sort of repeated measures design. Because we have multiple measurements on a subject, we can now separate the error variance from the subject variance.
5. Finally, the crux of the matter. If #4 doesn’t work, and it won’t always, your only option is to increase sample size. (But you knew that one, right?)
It is my understanding that you can sometimes increase power by switching to more advanced statistical testing. For example, is it true that when you use a general linear mixed model (versus rm-anova) you can increase power…I assume by increase sample size using long data formatting (versus wide format).
Can you point to some webinar or article or training which would address this? I’m also interested in a step-by-step explanation about how to determine sample size using GLMM (possibly with samplesizeshop).
Thanks
Hi Jeff,
In that example, yes, you’ll gain power using a mixed model over rm-anova if there is any missing data. RM-anova will drop anyone with any missing data, and mixed won’t.
Mixed is also more flexible, so allows a more precise model, and this can help power as well as model fit. For example rm-anova assumes all time-varying predictors are categorical.
The best way I know of to estimate sample size for a GLMM is with simulations.
How high effect size increase power of the study? it is obvious that as effect size like OR increase, sample size decreases. so these confused me, please clarify it.
Thanks for this great article, I have shared itt on Twitter.