Inter Rater Reliability is one of those statistics I seem to need just seldom enough that I forget all the details and have to look it up every time.
Luckily, there are a few really great web sites by experts that explain it (and related concepts) really well, in language that is accessible to non-statisticians.
So rather than reinvent the wheel and write about it, I’m going to refer you to these really great sites:
- Computing Intraclass Correlations (ICC) as Estimates of Interrater Reliability in SPSS by Richard Landers
- Statistical Methods for Rater and Diagnostic Agreement by John Uebersax
If you know of any others, please share in the comments. I’ll be happy to add to the list.