5 Practical Issues to Consider in Choosing a Statistical Analysis

There are 4 questions you must answer to choose an appropriate statistical analysis.

1. What is your Research Question?
2. What is the scale of measurement of the variables used to answer the research question?
3. What is the Design? (between subjects, within subjects, etc.)
4. Are there any data issues? (missing, censored, truncated, etc.)

If you have not already, read about these in more detail.

Answering those four questions well will often give you a range of statistical analyses that could all give you accurate results. So you will nearly always still be left with options.

For example, it may be that two different dependent variables both answer your research question well, but require different analyses because they are measured on different scales.  One is numerical and therefore can be included in a relatively simple model, like a linear model, and the other is ordered categories and will need a more sophisticated model like a logistic regression.

I am not advocating data fishing or ignoring real issues like assumptions. But I am a  pragmatist.  The reality is that statistics is filled with grey areas, and there is often good justification for using a variety of statistical approaches to answer the same research question with the same set of variables.

This article outlines 5 practical issues to consider in choosing an analysis when you have options.  These are all issues I take into account with my statistical consulting clients.

Which approach to pursue depends on these practical issues:

1. The purpose of the research and where it will be published

An unpublished honors or masters thesis has the primary goal of teaching the author, not obtaining scientific results (although that would be nice).  If no one is ever going to read it, less elegant approaches are fine.

On the other extreme, if you’re publishing in Science, you’d better have every i dotted and every missing value multiply imputed even if you do get the same results from listwise deletion.

2. The resources you have to conduct the analysis, including software, time, and money.

Clearly, if you need to turn in the dissertation in a week, you may need to use a simpler analysis than you otherwise would. If you are over-simplifying just to meet a university deadline, I strongly suggest you reanalyze the data before publication.

Also remember that there are limits to how much you can simplify an analysis. Some analyses are simply wrong. Using the analysis you know because that’s faster isn’t okay if it’s actually giving you inaccurate results.

3. Your capacity to correctly run and interpret the analysis

Someone who is just beginning statistical analysis and has never done a regression will not be able to handle a multilevel model.

End of story.

If you don’t have the resources (question 2) to hire someone to do it, another approach will need to be considered. (And forget publishing in Science.)

And if this is how you’re going with it, be very transparent about it in your paper.

4. Whether you ever have to do this analysis again

Some masters and PhD students are not planning research careers.  If this is the one and only statistical analysis you’ll ever do, it is less important for you to put the time and resources into mastering complicated statistical analysis.

This brings us back to the purpose. If you won’t have a research career, publishing isn’t important, and please don’t publish if it’s not accurate.

On the other hand, if you do need to publish, journal editors are getting more sophisticated in the types of analyses they’re expecting.  If you’re going to have to do it again and again, making the time to learn it now may well save much time and frustration over the years.

5. The expectations of the people who will be evaluating it—reviewers, committee, management

This is especially important if the real purpose of this research is to finish your dissertation.  If your stickler committee member wants it done a certain way and it has no real effect on the conclusions, just do it.  You can always redo your analyses once it’s time to publish.

Reviewers can be trickier.  Decide how much you’re willing to fight.

You may have to educate your reviewers (and editor) about a more appropriate technique that they don’t know about.  Be prepared with lots of references and possibly include some educational text in the paper or cover letter.

That said, I’ve witnessed researchers with PhDs in statistics jump through silly and unnecessary statistical hoops to please reviewers who thought they understood it better. At some point it might be more efficient to fight back.

Conclusion

What it comes down to, you want to keep your analysis as simple as you can without oversimplifying it so much that it’s no longer answering your research question, failing assumptions, or ignoring data issues.