A Note From Karen
I hope your October is off to a good start. Here in New York we reached fall weather... and then went back to summer heat. Now the crisp air has finally cooled off for the beautiful season we have ahead!
This is an exciting quarter for The Analysis Factor. We are currently running two workshops, but then we are taking a break from workshops for the rest of the year. Instead, we will be working on putting together some new offerings for you. A big one involves our free Craft of Statistical Analysis webinar program... We'll tell you more once everything is ready.
We're also busy planning the winter 2018 calendar for webinars in our Statistically Speaking membership program. If you are a member and have suggestions for topics you'd like to see covered, please share them with us in the "Suggestions" section of the forum. Or give a thumbs-up to another member's suggestion with our new rating feature.
To answer a very common question about how to check assumptions, I wrote the article below. I hope you enjoy it!
Happy analyzing,
Karen
The Problem with Using Tests for Statistical Assumptions
Every statistical model and hypothesis test has assumptions.
And yes, if you’re going to use a statistical test, you need to check whether those assumptions are reasonable to whatever extent you can.
Some assumptions are easier to check than others. Some are so obviously reasonable that you don’t need to do much to check them most of the time. And some have no good way of being checked directly, so you have to use situational clues.
There are so many nuances with assumptions as well: depending on a lot of the details of your particular study and data set, violations of some assumptions may be more or less serious. It depends on a lot of details: sample sizes, imbalance in the data across groups, whether the study is exploratory or confirmatory, etc.
And here’s the kicker -- the simple rules your stats professor told you to use to test assumptions were absolutely sufficient when you were learning about tests and assumptions. But now that you’re doing real data analysis? It’s time to dig into the details.
Before You Begin
1. Make sure you understand what the assumptions are for the statistical test you're using and what they mean. There is a lot of misinformation and vague information out there about assumptions.
2. Don’t forget that when assumptions violations are serious, it will call into question all your results. This really is important.
Don’t rely on a single statistical test to decide if another test’s assumptions have been met.
There are many tests -- Levene’s test for homogeneity of variance, the Kolmogorov-Smirnov test for normality, the Bartlett’s test for sphericity -- whose main usage is to test the assumptions of another test.
I suspect they have other uses, but this is how I’ve generally seen them used.
These tests provide useful information about whether an assumption is being met. But each test is just one indication of whether an assumption is reasonably being met by your data.
Because, nuances.
Let’s use the example of Levene’s test of homogeneity of variance. It’s often used in ANOVA as a sole decision criterion. And software makes it so easy.
But basing your decision on one test is too simplistic for real research. Why?
1. It relies too much on p-values, and therefore, sample sizes. Levene’s will have a smaller p-value for large samples than for small samples, given the same population variances.
So it’s very likely that you’re overstating a problem with the assumption in large samples and understating it in small samples. You can’t ignore the actual size difference in the sample variances when making this decision.
So sure, look at the Levene's p-value, but also look at the actual variances and how much bigger some are than others. (In other words, actually look at the effect size, not just the p-value).
2. The ANOVA is generally considered robust to violations of this assumption when sample sizes across groups are equal. So even if Levene’s is significant, moderately different variances may not be a problem in balanced data sets.
Keppel (p. 98 of the 1992 edition) suggests that a good rule of thumb is that if sample sizes are equal, robustness should hold until the largest variance is more than 9 times the smallest variance.
3. This robustness goes away the more unbalanced the samples are. So you need to use judgment here, taking into account both the imbalance and the actual difference in variances.
Gather Evidence Instead
The Levene’s test is one piece of evidence you’ll use to make a decision.
In addition to that and Keppel's ratio rule of thumb, the other pieces of info could potentially include:
A graph of your data. Is there an obvious difference in spread across groups?
A different test of the same assumption, if it exists. (Brown-Forsythe is another one for equal variances, but there are others.) See if the results match.
Then consider each piece of evidence within the wider data context.
Now make a judgment, taking into account the sample sizes when interpreting p-values from any tests.
I know it’s not comfortable to make judgments based on uncertain information.
(Experience helps here.)
But remember, in data analysis, it’s impossible not to.
Just be transparent in your results section of your report about what you did and on what evidence you based your decision.
References and Further Reading:
Keppel, Geoffrey: Design and Analysis, 3rd Edition. Prentice-Hall.
Neter, Kutner, Nachtsheim, Wasserman: Applied Linear Regression Models. Irwin.
When Assumptions of ANCOVA Are Irrelevant
The Assumptions of Linear Models: Explicit and Implicit
|
