Chi-square difference tests are frequently used to test differences between nested models in confirmatory factor analysis, path analysis and structural equation modeling. Nested models are two models (or more if one is fitting a series of models) that are identical except that one of the models constrains some of the parameters (the null model) and one does not have those constraints (the alternative model). Examples of this include the introduction of a set of dichotomous predictors representing a single nominal (categorical) variable to the model, or a test for differences across groups in a multiple group model. Typically a chi-square difference test involves calculating the difference between the chi-square statistic for the null and alternative models, the resulting statistic is distributed chi-square with degrees of freedom equal to the difference in the degrees of freedom between the two models. However, when a model is run in Mplus using the MLM or MLR estimators, the following warning message is displayed, warning the user that the standard chi-square difference test is not valid:
* The chi-square value for MLM, MLMV, MLR, ULSMV, WLSM and WLSMV cannot be used
for chi-square difference tests. MLM, MLR and WLSM chi-square difference
testing is described in the Mplus Technical Appendices at www.statmodel.com.
See chi-square difference testing in the index of the Mplus User's Guide.
This page shows how to calculate the correct chi-square difference test statistic for models estimated with the MLM or MLR estimator as described on the Mplus website (see http://www.statmodel.com/chidiff.shtml for additional information). For both MLM and MLR the Satorra-Bentler scaled chi-square difference test can be used. An additional test is available to test for differences in nested models that use the MLR estimator; this test is based on the log-likelihoods from nested models. Information on how to calculate the Satorra-Benter scaled chi-square test appears first, followed by information on using the log-likelihood to calculate a difference test. Each section includes both a discussion of the formulae and an example.
Cautions
While the above message appears for a number of estimators, the procedure described below is for use with the MLM and MLR estimators only. If the model uses the MLMV or WLSMV estimator, the difftest command can be used to test for differences across nested models. A Wald test can also be used to test nested models using the model test command. See the Mplus manual for more information on these commands. If you are unsure of what estimator is being used in your model, you can find the estimator listed towards the top of the output file.
Also keep in mind that this type of test is only valid if the models are nested; that is, the models must be the same except that one of the models includes additional constraints on the parameters.
The Satorra-Bentler scaled chi-square difference test
In order to calculate the Satorra-Bentler scaled chi-square difference test, we will need a number of pieces of information. Below is a list of the information needed, along with the symbol (i.e., letter and number) used to represent each value.
c0 = scaling correction factor for the null model. c1 = scaling correction factor for the alternative model. d0 = degrees of freedom for the null model. d1 = degrees of freedom for the alternative model. SB0 = Satorra-Bentler scaled chi-square value for the null model. SB1 = Satorra-Bentler scaled chi-square value for the alternative model.
In order to calculate the test statistic, T, we first need to calculate the value cd:
cd = (d0 * c0 - d1 * c1)/(d0 - d1)
Once we have calculated cd, we can compute:
T = (SB0 * c0 - SB1 * c1)/cd
T is distributed chi-square with degrees of freedom:
df = d0 - d1
Example
Below are two Mplus input files. On the left is the null model, which constrains the paths from x2 to f1 and from x3 to f1 to be zero (using x2@0 x3@0 respectively). On the right is the alternative model which estimates the regression path from x2 to f1 and from x3 to f1.
| The null model | The alternative model |
|
|
Once we have run the model, we can find all the information we need to compute the test statistic in the section of the output labeled "Chi-Square Test of Model Fit." This section of the output appears below (all other output is omitted). Output for the null model appears on the left, and output for the alternative model appears on the right. The information we need appears in bold. Below the output, each of the values highlighted appears on the appropriate line. If your output does not include the scaling correction factors (c0 and c1), instructions for calculating them from other output appears below.
| The null model | The alternative model |
Chi-Square Test of Model Fit Value 178.097* Degrees of Freedom 8 P-Value 0.0000 Scaling Correction Factor 1.018 for MLR |
Chi-Square Test of Model Fit
Value 35.122*
Degrees of Freedom 6
P-Value 0.0000
Scaling Correction Factor 0.958
for MLR
|
c0 = 1.018 (scaling correction factor for the null model) c1 = 0.958 (scaling correction factor for the alternative model) d0 = 8 (degrees of freedom for the null model) d1 = 6 (degrees of freedom for the alternative model) SB0 = 178.097 (the Satorra-Bentler adjusted chi-square value for the null model) SB1 = 35.122 (the Satorra-Bentler adjusted chi-square value for the alternative model)
Once we have identified and noted all the necessary values in our output, the test statistic (T) is computed in two steps.
cd = (d0 * c0 - d1 * c1)/(d0 - d1) = (8 * 1.018 - 6 * 0.958)/(8 - 6) = 1.198 T = (SB0 * c0 - SB1 * c1)/cd = (178.097 * 1.018 - 35.122 * 0.958) / 1.198 = 123.25198
The test statistic T is distributed chi-square with df = d0-d1. We can look up the p-value for a chi-square statistic of 123.25, with two degrees of freedom using a table or some other method (chi2(2) = 123.25, p < 0.01).
Calculating the scaling correction factor
If your output does not include the scaling correction factor, you can calculate this value from the unadjusted chi-square and the Satorra-Bentler scaled chi-square statistic. So, we are going to pretend that we have output that contains only the unadjusted chi-square, the Satorra-Bentler scaled chi-square statistic and the degrees of freedom.
T0 = Unadjusted chi-square value for the null model. T1 = Unadjusted chi-square value for the alternative model. SB0 = The Satorra-Bentler scaled chi-square for the null model. SB1 = The Satorra-Bentler scaled chi-square for the alternative model.
Given those values, the scaling correction factors (c0 and c1) can be calculated as shown below. c0 is the scaling correction factor for the null model, and c1 is the scaling correction factor for the alternative model.
c0 = T0/SB0 c1 = T1/SB1
For example, taking the appropriate values from the output:
T0 = 181.303 T1 = 33.647 SB0 = 178.097 SB1 = 35.122
The scaling correction factors can be calculated:
c0 = 181.303/178.097 = 1.018 c1 = 33.647/35.122 = 0.958
A test using the log-likelihood
For the MLR estimator there is an additional test for nested models. This test compares the log-likelihoods for the null and alternative models rather than the chi-square values. Below is a list of the information needed, along with the symbol (i.e., letter and number) used to represent each value.
L0 = log-likelihood for the null model. L1 = log-likelihood for the alternative model. c0 = scaling correction factor for the null model. c1 = scaling correction factor for the alternative model. p0 = number of parameters estimated in the null model. p1 = number of parameters estimated in the alternative model.
From this information the tests statistic TRd can be calculated (note that cd is calculated first then used to calculate TRd), along with the degrees of freedom:
cd = (p0*c0-p1*c1)/(p0-p1) TRd = -2*(L0-L1)/cd df = p1-p0
Example
Below is an example of output from two nested models. The information necessary to calculate this test statistic appears in the sections of output labeled “Loglikelihood” and the section immediately after it labeled “Information Criteria” (all other output has been omitted). The information needed for the calculations appears in bold. Note that the log likelihoods and scaling correction factors identified as H0 should be used, not the values labeled H1. Below the output, each of the values highlighted appears on the appropriate line.
| The null model | The alternative model |
Loglikelihood
H0 Value -3063.145
H0 Scaling Correction Factor 0.942
for MLR
H1 Value -2972.450
H1 Scaling Correction Factor 0.976
for MLR
Information Criteria
Number of Free Parameters 10
Akaike (AIC) 6146.290
Bayesian (BIC) 6179.273
Sample-Size Adjusted BIC 6147.592
(n* = (n + 2) / 24)
|
Loglikelihood
H0 Value -2989.266
H0 Scaling Correction Factor 0.985
for MLR
H1 Value -2972.450
H1 Scaling Correction Factor 0.976
for MLR
Information Criteria
Number of Free Parameters 12
Akaike (AIC) 6002.533
Bayesian (BIC) 6042.112
Sample-Size Adjusted BIC 6004.095
(n* = (n + 2) / 24)
|