One method for determining correct denominators in analysis of variance is the Cornfield-Tukey method. This FAQ presents a modified version of the Cornfield-Tukey method for manually deriving the symbolic values for the expected mean squares. It is from these expected mean squares that one can determine appropriate error terms.
Please note that this approach to deriving expected mean squares assumes that the interaction of the fixed and random effects sum to zero over the fixed effect levels. This approach can be found in a number of classical ANOVA textbooks, such as, Kirk, Winer and Keppel. This assumption, however, is not universal and is not used in most mixed programs (proc mixed, xtmixed, etc).
If you would like to try a program that automates much of the computation for this algorithm, go to How can I determine the correct term in an anova using Stata?.
Steps in deriving expected mean squares
Step 1 - Write the linear model for the design.
Step 2 - Construct a table with three parts.
Step 3 - The row headings in part 1 contain each of the terms from the linear
model including their subscripts but leaving out μ.
Step 4 - The column heading in part 2 contain the subscripts from the linear
model, the symbol for the number of levels along with the sampling
coefficient. Sampling coefficients are coded 1 for random variables
and 0 for fixed.
Step 5 - If a column heading appears as a row subscript in parentheses
enter a 1 in part 2.
Step 6 - If a column heading appears as a row subscript, not in parentheses,
enter the appropriate sampling coefficient (0 or 1).
Step 7 - If a column heading does not appear as a row subscript enter the
letter for the number of levels
Step 8 - In part 3 list a variance for each term in the linear model that
contains all the row subscripts.
Step 9 - Coefficients for variances are obtained by covering the column headed
by subscripts that appear in the row but not including subscripts in
parentheses. Obviously, terms with zero coefficients drop out.
Example three-way factorial design
In this example, A & C are fixed and B is random. The subscript for ε is i(jkl) because the subjects are nested in the A*B*C cells. The subjects themselves are also random. The term, εi(jkl), is known as error, within cell or residual.
Step 1 - Yijkl = μ + αj + βk + γl + αβjk + αγjl + βγkl + αβγjkl + εi(jkl)
Part 1 Part 2 Part 3
subscript i j k l
levels n p q m
sampling coef 1 0 1 0
-------------------------------------------------------------------
αj n 0 q m σ2ε + 0σ2αβγ + 0σ2αγ + nmσ2αβ + nqmσ2α
σ2ε + nmσ2αβ + nqmσ2α
βk n p 1 m σ2ε + 0σ2αβγ + 0σ2βγ + 0σ2αβ + npmσ2β
σ2ε + npmσ2β
γl n p q 0 σ2ε + 0σ2αβγ + npσ2βγ + 0σ2αγ + npqσ2γ
σ2ε + npσ2βγ + npqσ2γ
αβjk n 0 1 m σ2ε + 0σ2αβγ + nmσ2αβ
σ2ε + nmσ2αβ
αγjl n 0 q 0 σ2ε + nσ2αβγ + nqσ2αγ
βγkl n p 1 0 σ2εσ2ε + 0σ2αβγ + npσ2βγ
σ2ε + npσ2βγ
αβγjkl n 0 1 0 σ2ε + nσ2αβγ
εi(jkl) 1 1 1 1 σ2ε
A correctly formed F-ratio will have one more term in the numerator than in the denominator. The additional
term in the numerator is the effect of interest. Thus, the F-ration for A main effect would
look something like this:
σ2ε + nmσ2αβ + nqmσ2α MS(A)
F(A) = ------------------------ = ---------
σ2ε + nmσ2αβ MS(A*B)
Here are the terms that go into each of the F-ratios for the above model:
Effect Error Term numerator denominator MS(A) MS(A*B) MS(B) MS(residual) MS(C) MS(B*C) MS(A*B) MS(residual) MS(A*C) MS(A*B*C) MS(B*C) MS(residual) MS(A*B*C) MS(residual)
Reference
Kirk, Roger E. (1998) Experimental Design: Procedures for the Behavioral Sciences, Third Edition. Monterey, California: Brooks/Cole Publishing. ISBN 0-534-25092-0