The anova and mixed commands present their results rather differently. The anova command displays a single test for each factor in the model including factors that have more than one degree of freedom. The mixed command displays estimate for each degree of freedom. Even when you follow the mixed command with test the results often don’t agree with anova except for the highest order interaction. This page will show you how you can get anova type results from mixed by using the margins command.
Example 1
Our first example will use split-plot factorial model with one between subject factor a (2 levels) and one within subject factor b (4 levels). We will begin by loading the data and running the anova model for comparison purposes.
use https://stats.idre.ucla.edu/stat/data/spf24, clear anova y a / s|a b a#b / Number of obs = 32 R-squared = 0.9613 Root MSE = .712 Adj R-squared = 0.9333 Source | Partial SS df MS F Prob>F -----------+---------------------------------------------------- Model | 226.375 13 17.413462 34.35 0.0000 | a | 3.125 1 3.125 2.00 0.2070 s|a | 9.375 6 1.5625 -----------+---------------------------------------------------- b | 194.5 3 64.833333 127.89 0.0000 a#b | 19.375 3 6.4583333 12.74 0.0001 | Residual | 9.125 18 .50694444 -----------+---------------------------------------------------- Total | 235.5 31 7.5967742
This model has two error terms s and the residual (which is technically the same as the b#s|a interaction). One advantage to using mixed for these mixed model ANOVAs is that we do not have to specify which error term is used to test which effect. We do, however, have to be sure to include all of the necessary random effects in our mixed model. There is an additional advantage in that you get better estimates using mixed when there is missing data for some subjects.
We will now run the mixed command followed by tests for each of the effects. The setup for mixed has the fixed effects to the left of the double pipe (||) and the random effects to the right. For each test with more than one degree of freedom we will also show the chi-square value divided by the degrees of freedom so that they are scaled the same as the F-ratio in the above ANOVA. The reml option must be used to replicate the results of the ANOVA.
mixed y a##b || s: , reml Performing EM optimization: Performing gradient-based optimization: Iteration 0: log restricted-likelihood = -34.824381 Iteration 1: log restricted-likelihood = -34.824379 Computing standard errors: Mixed-effects REML regression Number of obs = 32 Group variable: s Number of groups = 8 Obs per group: min = 4 avg = 4.0 max = 4 Wald chi2(7) = 423.89 Log restricted-likelihood = -34.824379 Prob > chi2 = 0.0000 ------------------------------------------------------------------------------ y | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- 2.a | -2 .6208193 -3.22 0.001 -3.216783 -.7832165 | b | 2 | .25 .5034603 0.50 0.619 -.736764 1.236764 3 | 3.25 .5034603 6.46 0.000 2.263236 4.236764 4 | 4.25 .5034603 8.44 0.000 3.263236 5.236764 | a#b | 2 2 | 1 .7120004 1.40 0.160 -.3954951 2.395495 2 3 | .5 .7120004 0.70 0.483 -.8954951 1.895495 2 4 | 4 .7120004 5.62 0.000 2.604505 5.395495 | _cons | 3.75 .4389855 8.54 0.000 2.889604 4.610396 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval] -----------------------------+------------------------------------------------ s: Identity | var(_cons) | .2638887 .2294499 .0480071 1.450562 -----------------------------+------------------------------------------------ var(Residual) | .5069445 .1689815 .2637707 .9743036 ------------------------------------------------------------------------------ LR test vs. linear model: chibar2(01) = 3.30 Prob >= chibar2 = 0.0346 estimates store m1 /* save estimate for use later on */ testparm a#b ( 1) [y]2.a#2.b = 0 ( 2) [y]2.a#3.b = 0 ( 3) [y]2.a#4.b = 0 chi2( 3) = 38.22 Prob > chi2 = 0.0000 display "display as F " r(chi2)/r(df) display as F 12.739724 testparm i.b ( 1) [y]2.b = 0 ( 2) [y]3.b = 0 ( 3) [y]4.b = 0 chi2( 3) = 107.88 Prob > chi2 = 0.0000 display "display as F " r(chi2)/r(df) display as F 35.958898 testparm i.a ( 1) [y]2.a = 0 chi2( 1) = 10.38 Prob > chi2 = 0.0013
As you can see, the only test that has the same value in both the anova and the mixed is the one for the a#b interaction. The reason for this because the factor-variables used in the mixed command use indicator (dummy) coding. The tests of the main effects for a and b are really test of simple effects for the reference level of the categorical variable.
We can easily get the “correct” tests for the main effects using the margins command.
margins b, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- b | 1 | 2.75 .3104096 8.86 0.000 2.141608 3.358392 2 | 3.5 .3104096 11.28 0.000 2.891608 4.108392 3 | 6.25 .3104096 20.13 0.000 5.641608 6.858392 4 | 9 .3104096 28.99 0.000 8.391608 9.608392 ------------------------------------------------------------------------------ test (1.b=2.b)(1.b=3.b)(1.b=4.b) ( 1) 1bn.b - 2.b = 0 ( 2) 1bn.b - 3.b = 0 ( 3) 1bn.b - 4.b = 0 chi2( 3) = 383.67 Prob > chi2 = 0.0000 display "display as F " r(chi2)/r(df) display as F 127.89039 estimates restore m1 (results m1 are active now) margins a, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- a | 1 | 5.6875 .3124999 18.20 0.000 5.075011 6.299989 2 | 5.0625 .3124999 16.20 0.000 4.450011 5.674989 ------------------------------------------------------------------------------ test (1.a=2.a) ( 1) 1bn.a - 2.a = 0 chi2( 1) = 2.00 Prob > chi2 = 0.1573
Note that the tests for a and b main effects now agree with the anova results in the first part of this example.
For our second example, let’s try a model that is a bit more complex.
Example 2
The second example is a split-plot factorial with one between subject factor a (2 levels) and two within subject factors b and c (2 levels each). The model has four separate error terms but we won’t have to divide any of the chi-square values from the mixed model by their degrees of freedom because all of the tests have just a single degree of freedom.
use https://stats.idre.ucla.edu/stat/data/spf2-22, clear anova y a / s|a b a#b / b#s|a c a#c / c#s|a b#c a#b#c / Number of obs = 32 R-squared = 0.9920 Root MSE = .559017 Adj R-squared = 0.9589 Source | Partial SS df MS F Prob > F -----------+---------------------------------------------------- Model | 233.625 25 9.345 29.90 0.0002 | a | 3.125 1 3.125 2.00 0.2070 s|a | 9.375 6 1.5625 -----------+---------------------------------------------------- b | 162 1 162 199.38 0.0000 a#b | 6.125 1 6.125 7.54 0.0335 b#s|a | 4.875 6 .8125 -----------+---------------------------------------------------- c | 24.5 1 24.5 61.89 0.0002 a#c | 10.125 1 10.125 25.58 0.0023 c#s|a | 2.375 6 .395833333 -----------+---------------------------------------------------- b#c | 8 1 8 25.60 0.0023 a#b#c | 3.125 1 3.125 10.00 0.0195 | Residual | 1.875 6 .3125 -----------+---------------------------------------------------- Total | 235.5 31 7.59677419
We begin this next section by using the egen group command to create combinations of variables to act as our random effects. We will need b by s, c by s and b by c by s along with s for our random effects. We will then run the mixed command followed by test of the a#b#c interaction to show that the results for the highest order interaction is the same as the anova results.
egen bs = group(b s) egen cs = group(c s) egen bcs= group(b c s) mixed y a##b##c || _all: R.bs || _all: R.cs || _all: R.bcs || s:, reml Performing EM optimization: Performing gradient-based optimization: Iteration 0: log restricted-likelihood = -34.084157 (not concave) Iteration 1: log restricted-likelihood = -34.049357 (not concave) Iteration 2: log restricted-likelihood = -34.046954 Iteration 3: log restricted-likelihood = -34.046003 (not concave) Iteration 4: log restricted-likelihood = -34.045909 Iteration 5: log restricted-likelihood = -34.045907 Computing standard errors: Mixed-effects REML regression Number of obs = 32 ------------------------------------------------------------- | No. of Observations per Group Group Variable | Groups Minimum Average Maximum ----------------+-------------------------------------------- _all | 1 32 32.0 32 s | 8 4 4.0 4 ------------------------------------------------------------- Wald chi2(7) = 332.00 Log restricted-likelihood = -34.045907 Prob > chi2 = 0.0000 ------------------------------------------------------------------------------ y | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- 2.a | -2 .6208194 -3.22 0.001 -3.216784 -.7832163 2.b | 3.25 .5303299 6.13 0.000 2.210572 4.289428 | a#b | 2 2 | .5 .7499998 0.67 0.505 -.9699725 1.969973 | 2.c | .25 .4208128 0.59 0.552 -.5747779 1.074778 | a#c | 2 2 | 1 .5951191 1.68 0.093 -.1664121 2.166412 | b#c | 2 2 | .75 .5590141 1.34 0.180 -.3456476 1.845648 | a#b#c | 2 2 2 | 2.5 .7905654 3.16 0.002 .9505203 4.04948 | _cons | 3.75 .4389856 8.54 0.000 2.889604 4.610396 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval] -----------------------------+------------------------------------------------ _all: Identity | var(R.bs) | .2500028 .2512985 .0348603 1.792908 -----------------------------+------------------------------------------------ _all: Identity | var(R.cs) | .04167 .1455805 .0000443 39.23096 -----------------------------+------------------------------------------------ _all: Identity | var(R.bcs) | .2498766 .180416 .0606933 1.028752 -----------------------------+------------------------------------------------ s: Identity | var(_cons) | .1666639 .2644133 .0074371 3.734895 -----------------------------+------------------------------------------------ var(Residual) | .0626203 . . . ------------------------------------------------------------------------------ LR test vs. linear model: chi2(4) = 4.86 Prob > chi2 = 0.3019 Note: LR test is conservative and provided only for reference. estimates store m1 /* save estimate for use later on */ testparm a#b#c ( 1) [y]2.a#2.b#2.c = 0 chi2( 1) = 10.00 Prob > chi2 = 0.0016
Yes indeed, the result for the a#b#c interaction is the same in both the mixed and the anova. Next, we will test each of the main effects and two-way interactions to show that these results are not equivalent to those in the anova above.
testparm i.a ( 1) [y]2.a = 0 chi2( 1) = 10.38 Prob > chi2 = 0.0013 testparm i.b ( 1) [y]2.b = 0 chi2( 1) = 37.56 Prob > chi2 = 0.0000 testparm a#b ( 1) [y]2.a#2.b = 0 chi2( 1) = 0.44 Prob > chi2 = 0.5050 testparm i.c ( 1) [y]2.c = 0 chi2( 1) = 0.35 Prob > chi2 = 0.5525 testparm a#c ( 1) [y]2.a#2.c = 0 chi2( 1) = 2.82 Prob > chi2 = 0.0929 testparm b#c ( 1) [y]2.b#2.c = 0 chi2( 1) = 1.80 Prob > chi2 = 0.1797
Having verified that the tests for the main effects and two-way interactions differ between mixed and anova, we will finish up by running a series of margin commands testing the main effects and two-way interactions correctly.
margins a, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) c (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- a | 1 | 5.6875 .3124997 18.20 0.000 5.075012 6.299988 2 | 5.0625 .3124997 16.20 0.000 4.450012 5.674988 ------------------------------------------------------------------------------ test (1.a = 2.a) /* a main effect */ ( 1) 1bn.a - 2.a = 0 chi2( 1) = 2.00 Prob > chi2 = 0.1573 estimates restore m1 margins b, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) c (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- b | 1 | 3.125 .2724312 11.47 0.000 2.591045 3.658955 2 | 7.625 .2724312 27.99 0.000 7.091045 8.158955 ------------------------------------------------------------------------------ test (1.b = 2.b) /* b main effect */ ( 1) 1bn.b - 2.b = 0 chi2( 1) = 199.38 Prob > chi2 = 0.0000 estimates restore m1 margins c, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) c (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- c | 1 | 4.5 .2473822 18.19 0.000 4.01514 4.98486 2 | 6.25 .2473822 25.26 0.000 5.76514 6.73486 ------------------------------------------------------------------------------ test (1.c = 2.c) /* c main effect */ ( 1) 1bn.c - 2.c = 0 chi2( 1) = 61.89 Prob > chi2 = 0.0000 estimates restore m1 margins a#b, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) c (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- a#b | 1 1 | 3.875 .3852758 10.06 0.000 3.119873 4.630127 1 2 | 7.5 .3852758 19.47 0.000 6.744873 8.255127 2 1 | 2.375 .3852758 6.16 0.000 1.619873 3.130127 2 2 | 7.75 .3852758 20.12 0.000 6.994873 8.505127 ------------------------------------------------------------------------------ test (1.a#1.b - 1.a#2.b - 2.a#1.b + 2.a#2.b = 0) /* a#b interaction */ ( 1) 1bn.a#1bn.b - 1bn.a#2.b - 2.a#1bn.b + 2.a#2.b = 0 chi2( 1) = 7.54 Prob > chi2 = 0.0060 estimates restore m1 margins a#c, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) c (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- a#c | 1 1 | 5.375 .3498513 15.36 0.000 4.689304 6.060696 1 2 | 6 .3498513 17.15 0.000 5.314304 6.685696 2 1 | 3.625 .3498513 10.36 0.000 2.939304 4.310696 2 2 | 6.5 .3498513 18.58 0.000 5.814304 7.185696 ------------------------------------------------------------------------------ test (1.a#1.c - 1.a#2.c - 2.a#1.c + 2.a#2.c = 0) /* a#c interaction */ ( 1) 1bn.a#1bn.c - 1bn.a#2.c - 2.a#1bn.c + 2.a#2.c = 0 chi2( 1) = 25.58 Prob > chi2 = 0.0000 estimates restore m1 margins b#c, post asbalanced Adjusted predictions Number of obs = 32 Expression : Linear prediction, fixed portion, predict() at : a (asbalanced) b (asbalanced) c (asbalanced) ------------------------------------------------------------------------------ | Delta-method | Margin Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- b#c | 1 1 | 2.75 .3104097 8.86 0.000 2.141608 3.358392 1 2 | 3.5 .3104097 11.28 0.000 2.891608 4.108392 2 1 | 6.25 .3104097 20.13 0.000 5.641608 6.858392 2 2 | 9 .3104097 28.99 0.000 8.391608 9.608392 ------------------------------------------------------------------------------ test (1.b#1.c - 1.b#2.c - 2.b#1.c + 2.b#2.c = 0) /* b#c interaction */ ( 1) 1bn.b#1bn.c - 1bn.b#2.c - 2.b#1bn.c + 2.b#2.c = 0 chi2( 1) = 25.60 Prob > chi2 = 0.0000
These two examples should give you a good feel for how to use the margins command to get anova type results from mixed.