Karen Grace-Martin

Covariance Matrices, Covariance Structures, and Bears, Oh My!

September 16th, 2011 by

Of all the concepts I see researchers struggle with as they start to learn high-level statistics, the one that seems to most often elicit the blank stare of incomprehension is the Covariance Matrix, and its friend, the Covariance Structure.

And since understanding them is fundamental to a number of statistical analyses, particularly Mixed Models and Structural Equation Modeling, it’s an incomprehension you can’t afford.

So I’m going to explain what they are and how they’re not so different from what you’re used to.  I hope you’ll see that once you get to know them, they aren’t so scary after all.

What is a Covariance Matrix?

There are two concepts inherent in a covariance matrix–covariance and matrix.  Either one can throw you off.

Let’s start with matrix.  If you never took linear algebra, the idea of matrices can be frightening.  (And if you still are in school, I highly recommend you take it.  Highly).  And there are a lot of very complicated, mathematical things you can do with matrices.

But you, a researcher and data analyst, don’t need to be able to do all those complicated processes to your matrices.  You do need to understand what a matrix is, be able to follow the notation, and understand a few simple matrix processes, like multiplication of a matrix by a constant.

The thing to keep in mind when it all gets overwhelming is a matrix is just a table.  That’s it.

A Covariance Matrix, like many matrices used in statistics, is symmetric.  That means that the table has the same headings across the top as it does along the side.

Start with a Correlation Matrix

The simplest example, and a cousin of a covariance matrix, is a correlation matrix.  It’s just a table in which each variable is listed in both the column headings and row headings, and each cell of the table (i.e. matrix) is the correlation between the variables that make up the column and row headings.  Here is a simple example from a data set on 62 species of mammal:

From this table, you can see that the correlation between Weight in kg and Hours of Sleep, highlighted in purple, is -.307. Smaller mammals tend to sleep more.

You’ll notice that this is the same above and below the diagonal. The correlation of Hours of Sleep with Weight in kg is the same as the correlation between Weight in kg and Hours of Sleep.

Likewise, all correlations on the diagonal equal 1, because they’re the correlation of each variable with itself.

If this table were written as a matrix, you’d only see the numbers, without the column headings.

Now, the Covariance Matrix

A Covariance Matrix is very similar. There are really two differences between it and the Correlation Matrix. It has this form:

First, we have substituted the correlation values with covariances.

Covariance is just an unstandardized version of correlation.  To compute any correlation, we divide the covariance by the standard deviation of both variables to remove units of measurement.  So a covariance is just a correlation measured in the units of the original variables.

Covariance, unlike  correlation, is not constrained to being between -1 and 1. But the covariance’s sign will always be the same as the corresponding correlation’s. And a covariance=0 has the exact same meaning as a correlation=0: no linear relationship.

Because covariance is in the original units of the variables, variables on scales with bigger numbers and with wider distributions will necessarily have bigger covariances. So for example, Life Span has similar correlations to Weight and Exposure while sleeping, both around .3.

But values of Weight vary a lot (this data set contains both Elephants and Shrews), whereas Exposure is an index variable that ranges from only 1 to 5. So Life Span’s covariance with Weight (5113.27) is much larger than than with Exposure (10.66).

Second, the diagonal cells of the matrix contain the variances of each variable. A covariance of a variable with itself is simply the variance. So you have a context for interpreting these covariance values.

Once again, a covariance matrix is just the table without the row and column headings.

What about Covariance Structures?

Covariance Structures are just patterns in covariance matrices.  Some of these patterns occur often enough in some statistical procedures that they have names.

You may have heard of some of these names–Compound Symmetry, Variance Components, Unstructured, for example.  They sound strange because they’re often thrown about without any explanation.

But they’re just descriptions of patterns.

For example, the Compound Symmetry structure just means that all the variances are equal to each other and all the covariances are equal to each other. That’s it.

It wouldn’t make sense with our animal data set because each variable is measured on a different scale. But if all four variables were measured on the same scale, or better yet, if they were all the same variable measured under four experimental conditions, it’s a very plausible pattern.

Variance Components just means that each variance is different, and all covariances=0. So if all four variables were completely independent of each other and measured on different scales, that would be a reasonable pattern.

Unstructured just means there is no pattern at all.  Each variance and each covariance is completely different and has no relation to the others.

There are many, many covariance structures.  And each one makes sense in certain statistical situations.  Until you’ve encountered those situations, they look crazy.  But each one is just describing a pattern that makes sense in some situations.

 


5 Reasons to Run Sample Size Calculations Before Collecting Data

September 9th, 2011 by

Most of us run sample size calculations when a granting agency or committee requires it.  That’s reason 1.

That is a very good reason.  But there are others, and it can be helpful to keep these in mind when you’re tempted to skip this step or are grumbling through the calculations you’re required to do.

It’s easy to base your sample size on what is customary in your field (“I’ll use 20 subjects per condition”) or to just use the number of subjects in a similar study (“They used 150, so I will too”).

Sometimes you can get away with doing that.

However, there really are some good reasons beyond funding to do some sample size estimates. And since they’re not especially time-consuming, it’s worth doing them. (more…)


How to Combine Complicated Models with Tricky Effects

July 22nd, 2011 by

Need to dummy code in a Cox regression model?

Interpret interactions in a logistic regression?

Add a quadratic term to a multilevel model?

quadratic interaction plotThis is where statistical analysis starts to feel really hard. You’re combining two difficult issues into one.

You’re dealing with both a complicated modeling technique at Stage 3 (survival analysis, logistic regression, multilevel modeling) and tricky effects in the model (dummy coding, interactions, and quadratic terms).

The only way to figure it all out in a situation like that is to break it down into parts.  (more…)


Dummy Code Software Defaults Mess With All of Us

July 15th, 2011 by

In my last blog post, I wrote about a mistake I once made when I didn’t realize the defaults for dummy coding were different in two SPSS procedures (Binary Logistic and GEE).

Ironically, about the same time I wrote it, I was having a conversation with Ann Maria de Mars on Twitter.  She was trying to figure out why her logistic regression model fit results were identical in SAS Proc Logistic and SPSS Binary Logistic, but the coefficients in SAS were half those of SPSS.

It was ironic because I, of course, didn’t recognize it as the same issue and wasn’t much help.

But Ann Maria investigated and discovered that it came down to differences in the defaults for coding categorical predictors in SAS and SPSS that did it.  Her detailed and humorous explanation is here.

Some takeaways for you, the researcher and data analyst:

1. Give yourself a break if you hit a snag.  Even very experienced data analysts, statisticians who understand what they’re doing, get stumped sometimes.  Don’t ever think that performing data analysis is an IQ test.  You’re bringing together many skills and complex tools.

2. Learn thy software.  In my last post, I phrased it “Know thy software”, but this is where you get to know it.  Snags are good opportunities to investigate the details of your software, just like Ann Maria did.  If you can think of it as a challenge to figure out–a puzzle–it can actually be fun.

Make friends with your syntax manuals.

3. Get help when you need it. Statistical software packages *are* complex tools. You don’t have to know everything to use them

Ask colleagues.  Call customer support. Call a stat consultant.  That’s what they’re there for.

4. A great way to check your work is to run your test two different ways.  It’s another reason to be able to use at least two stat software packages.  I’m not suggesting you have to run every analysis twice.  But when a result looks strange, or you want to double-check a specific important model, this can be a good strategy for testing things out.

It may be that your results aren’t telling you what you think they are.

 

[Logistic_Regression_Workshop]


When Dummy Codes are Backwards, Your Stat Software may be Messing With You

July 8th, 2011 by

One of the tricky parts about dummy coded (0/1) variables is keeping track of what’s a 0 and what’s a 1.

This is made particularly tricky because sometimes your software switches them on you.

Here’s one example in a question I received recently.  The context was a Linear Mixed Model, but this can happen in other procedures as well.

I dummy code my categorical variables “0” or “1” but for some reason in the (more…)


7 Practical Guidelines for Accurate Statistical Model Building

June 24th, 2011 by

Stage 2Model  Building–choosing predictors–is one of those skills in statistics that is difficult to teach.   It’s hard to lay out the steps, because at each step, you have to evaluate the situation and make decisions on the next step.

If you’re running purely predictive models, and the relationships among the variables aren’t the focus, it’s much easier.  Go ahead and run a stepwise regression model.  Let the data give you the best prediction.

But if the point is to answer a research question that describes relationships, you’re going to have to get your hands dirty.

It’s easy to say “use theory” or “test your research question” but that ignores a lot of practical issues.  Like the fact that you may have 10 different variables that all measure the same theoretical construct, and it’s not clear which one to use. (more…)