Designing experiments would always be simple if we could just randomly assign subjects to different treatment conditions with no other restrictions. Unfortunately, that doesn’t always work.
For example, there are many experimental situations where the subjects aren’t independent of each other. The subjects that are related to each other are combined into clusters called “blocks.” It can happen due to practicalities of running an experiment efficiently or you can intentionally plan it as a way to reduce random variance.
In either case, this is a randomized complete block design. It’s a great design to become familiar with because it will greatly expand your ability to create and analyze experiments.
How It Works
When you have subjects that share characteristics with one another, it can sometimes be difficult to isolate those characteristics directly. This makes it hard to record them as additional variables. By identifying the subjects that are similar, you can still capture how those characteristics affect the outcome. Subjects that are similar are grouped into “blocks.”
From there, you can make treatment assignments so that you put subjects from the same block into different treatment groups.
Why different treatment groups? Suppose subjects from the same block were assigned to the same treatment group. (more…)
Post-hoc tests, pairwise or other linear contrasts, are typical in an analysis of variance (ANOVA) setting to understand which group means differ. They incorporate p-value adjustments to avoid concluding that group means differ when they actually do not. There are several adjustments that can be considered for conducting multiple post-hoc tests, including single-step and stepwise adjustments. (more…)
One component often overlooked in the ‘Define & Design’ phase of a study, is writing the analysis plan. The statistical analysis plan integrates a lot of information about the study including the research question, study design, variables and data used, and the type of statistical analysis that will be conducted.
(more…)
Before you can write a data analysis plan, you have to choose the best statistical test or model. You have to integrate a lot of information about your research question, your design, your variables, and the data itself.
(more…)
At The Analysis Factor, we are on a mission to help researchers improve their statistical skills so they can do amazing research.
We all tend to think of “Statistical Analysis” as one big skill, but it’s not.
Over the years of training, coaching, and mentoring data analysts at all stages, I’ve realized there are four fundamental stages of statistical skill:
Stage 1: The Fundamentals
Stage 2: Linear Models
Stage 3: Extensions of Linear Models
Stage 4: Advanced Models
There is also a stage beyond these where the mathematical statisticians dwell. But that stage is required for such a tiny fraction of data analysis projects, we’re going to ignore that one for now.
If you try to master the skill of “statistical analysis” as a whole, it’s going to be overwhelming.
And honestly, you’ll never finish. It’s too big of a field.
But if you can work through these stages, you’ll find you can learn and do just about any statistical analysis you need to. (more…)
A few years back the winning t-shirt design in a contest for the American Association of Public Opinion Research read “Weighting is the Hardest Part.” And I don’t think the t-shirt was referring to 
anything about patience!
Most statistical methods assume that every individual in the sample has the same chance of selection.
Complex Sample Surveys are different. They use multistage sampling designs that include stratification and cluster sampling. As a result, the assumption that every selected unit has the same chance of selection is not true.
To get statistical estimates that accurately reflect the population, cases in these samples need to be weighted. If not, all statistical estimates and their standard errors will be biased.
But selection probabilities are only part of weighting. (more…)
