This page was adapted from a web page at the SPSS web page.
We thank SPSS for their permission to adapt and distribute this page via our web site.
This page is composed of 5 articles from SPSS Keywords
exploring issues in the understanding and interpretation of parameter estimates in
regression models and anova models. The figures have been renumbered consecutively.
You can download the data file ( https://stats.idre.ucla.edu/wp-content/uploads/2016/02/catreg.sav
) used in this article so you can reproduce the results for yourself.
USING CATEGORICAL VARIABLES IN REGRESSION
David P. Nichols
Senior Support Statistician
SPSS, Inc.
From SPSS Keywords, Number 56, 1995
When we polled Keywords readers to find out what kinds of topics they most wanted to
see covered in future Statistically Speaking articles, we found that many SPSS users are
concerned about the proper use of categorical predictor variables in regression models.
Since the interpretation of the estimated coefficients is a major part of the analysis of
a regression model, and since this interpretation depends upon how the predictors have
been coded (or in technical terms, how the model has been parameterized), this is indeed
an important topic.
To begin with, we will assume that the model under consideration involves only first
order or main effects of predictor variables. That is, no higher order polynomial terms
such as squares or cubes are used, and no interactions between predictors are involved.
Such higher order or product terms introduce complexities beyond those introduced by the
presence of main effects involving categorical variables. We will avoid these complexities
for the time being. We will further assume that we have complete data; that is, no missing
values on any predictor or dependent variables. We begin with a brief review of the
interpretation of estimated regression coefficients.
As you may remember, in a linear regression model the estimated raw or unstandardized
regression coefficient for a predictor variable (referred to as B on the SPSS REGRESSION
output) is interpreted as the change in the predicted value of the dependent variable for
a one unit increase in the predictor variable. Thus a B coefficient of 1.0 would indicate
that for every unit increase in the predictor, the predicted value of the dependent
variable also increases by one unit. In the common case where there are two or more
correlated predictors in the model, the B coefficient is known as a partial regression
coefficient, and it represents the predicted change in the dependent variable when that
predictor is increased by one unit while holding all other predictors constant. The
intercept or constant term gives the predicted value of the dependent variable when all
predictors are set to 0.
For our purposes the important distinction between different types of predictor
variables is between those measured on at least an interval scale, where a change of one
unit in the predictor has a constant meaning across the entire scale, and those where such
consistency of unit differences is not assumed. Though these are theoretically distinct,
in practice it often happens that the terms interval and subinterval are replaced by
continuous and categorical. The interpretation of estimated regression coefficients given
above applies in a fairly straightforward manner to interval predictors, continuous or
not, and their use in procedures like REGRESSION is quite simple as a practical matter:
just name them as independent variables and specify when you want them used. For
subinterval variables, which is the assumption in SPSS for categorical variables, things
are more complicated. Despite the fact that equating continuous with interval and
categorical with subinterval is an abuse of language, we’ll proceed to do just that, to
avoid confusion related to use of SPSS procedures.
One reason that the handling of categorical predictors is so important is that by the
time one gets to the actual computation of the regression equation, no distinction is made
between subinterval and interval variables. To put it another way, a matrix algebra
routine knows nothing about different types of numbers; they’re all just numbers. Some
SPSS procedures used to analyze linear and generalized linear regression models are
designed to handle the translation from categorical to interval representations with only
minimal guidance from the user. These include the T-TEST procedure, the analysis of
variance procedures ONEWAY, ANOVA and MANOVA, and the newer nonlinear regression
procedures LOGISTIC REGRESSION and COX REGRESSION. However, even when such automatic
handling of categorical predictors is available, it is still incumbent upon the user to
make sure that he or she understands categorical variable representations well enough to
produce useful results and to be able to interpret these results.
The simplest possible regression involving categorical predictors is one with a single
dichotomous (two level) independent variable. An example of such a regression model would
be the prediction of 1990 murder rates in each of the 50 states in the U.S.A. based upon
whether or not each state had a death penalty statute in force just prior to and during
that time. The data are compiled from almanac sources; murder rates are measured in number
per 100,000 population. The variable of interest, denoted MURDER90, has a mean value of
about 4.97 for the fourteen states without a death penalty statute, and about 7.86 for the
36 states with the death penalty.
Figure 1 presents the results of a dummy variable regression of MURDER90 on DEATHPEN, a
categorical variable taking on a value of 0 for the no death penalty states and 1 for the
death penalty states. 0-1 coding, known as dummy or indicator coding, is quite popular, as
it often lends itself to the simplest possible interpretation.
Figure 1
---------------------------------------------------------------------------
Multiple R .33556
R Square .11260
Adjusted R Square .09411
Standard Error 3.72103
Analysis of Variance
DF Sum of Squares Mean Square
Regression 1 84.33257 84.33257
Residual 48 664.61163 13.84608
F = 6.09072 Signif F = .0172
------------------ Variables in the Equation ------------------
Variable B SE B Beta T Sig T
DEATHPEN 2.892460 1.172015 .335562 2.468 .0172
(Constant) 4.971429 .994488 4.999 .0000
---------------------------------------------------------------------------
Here we have two coefficients, a constant or intercept term, and a "slope"
coefficient for the DEATHPEN variable. Recall that the interpretation is that the constant
is the predicted value when all predictors are set to 0, which here simply represents
those states with no death penalty. Thus the constant coefficient is equal to the mean
murder rate for this group. The DEATHPEN coefficient is the predicted increase in murder
rate for a unit increase in the DEATHPEN variable. Since those states with a DEATHPEN
value of 1 are those states with a death penalty statute, this coefficient represents the
change in estimated or predicted murder rate for these states relative to those without
the death penalty. The 2.89 value is exactly the difference between the two means, so that
adding it to the constant produces the mean for the death penalty states. Since we are
considering the entire population of states, the significance level is not necessary of
particular interest, though if we were to conceptualize the current situation as resulting
from a sampling from some hypothetical populations, the p-value of .0172 would indicate
that so large a coefficient is unlikely to result from chance were random samples of this
size drawn from hypothetical populations with equal means.
Other results of note are that the p-value for the t-test for the MURDER90 coefficient
is the same as that for the overall regression F-test. This is due to the fact that the
t-test tests the null hypothesis that this coefficient is 0 in the population, while the
F-test tests the null hypothesis that all coefficients other than intercept are 0 in the
population, and with only one predictor, these hypotheses are the same. The F-value is
precisely the square of the t-value. This holds only for a simple regression involving one
predictor. Also of note is the fact that the Multiple R, which reduces to the absolute
value of the correlation between the predictor and the dependent variable in a simple
regression, is equal to the standardized regression coefficient (Beta). In a simple
regression, the standardized coefficient is the correlation between the predictor and
dependent variables, and is thus constrained to be between -1 and +1. Note that this
generally holds true only for a simple regression, and that with correlated predictor
variables, the standardized coefficients may be larger than 1 in absolute value.
This correlation between a dichotomous variable and a continuous variable is sometimes
known as a point-biserial correlation. No special formula is required; special
computational formulas in texts are simply special cases of the general Pearson product
moment correlation coefficient formula applied to this combination of variable types. If
both variables are dichotomous, the standard formula reduces further to that for a phi
coefficient.
Finally, note that there are a number of ways in SPSS to achieve the same results we
obtained from REGRESSION, if our purpose were to test the null hypothesis of equality of
means between the two groups of states drawn from our hypothetical populations. Precisely
the same t-statistic (or the negative of the value from REGRESSION, which means the same
thing, given the variable codings) could be obtained from T-TEST, the CONTRAST option in
ONEWAY or parameter estimate output in MANOVA, and the F-statistic could be duplicated in
ONEWAY, ANOVA or MANOVA. In ONEWAY or ANOVA we would have to use the dummy variable for
DEATHPEN as a two level factor, while in MANOVA we could either specify it as a factor or
as a covariate. The results in any case would be the same in terms of test statistics and
p-values. One example is given in Figure 2, using default DEVIATION contrasts in MANOVA:
Figure 2
---------------------------------------------------------------------------
Tests of Significance for MURDER90 using UNIQUE sums of squares
Source of Variation SS DF MS F Sig of F
WITHIN+RESIDUAL 664.61 48 13.85
DEATHPEN 84.33 1 84.33 6.09 .017
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Estimates for MURDER90
--- Individual univariate .9500 confidence intervals
CONSTANT
Parameter Coeff. Std. Err. t-Value Sig. t Lower -95% CL- Upper
1 6.41765873 .58601 10.95150 .00000 5.23941 7.59591
DEATHPEN
Parameter Coeff. Std. Err. t-Value Sig. t Lower -95% CL- Upper
2 -1.4462302 .58601 -2.46794 .01721 -2.62448 -.26798
---------------------------------------------------------------------------
We see that the significance level for the t-test for the DEATHPEN parameter estimate
is the same as we obtained in REGRESSION. However, it is opposite in sign to our earlier
coefficient, and only half the size. Our constant has also changed; note that it’s value
is now halfway between the means of the two groups. The differences we see here are due to
the use of a different set of predictor codings being used internally by MANOVA. That is,
MANOVA has parameterized the model somewhat differently than we did earlier. The default
DEVIATION contrasts in MANOVA are designed to compare each level of a factor to the mean
of all levels. In this case the DEATHPEN coefficient compares the no death penalty mean to
the simple average of the two group means. The CONSTANT coefficient is this simple mean of
group means. The F-statistic remains the same, the square of the t-value, as was the case
in REGRESSION.
These results point to three important features of the regression model. One is that
the interpretation of the estimated model coefficients is dependent upon the
parameterization of the model; in order to know how to interpret the coefficients, we must
be aware of how the predictor values have been coded. Second, despite having used two
different parameterizations, we obtained the same results in terms of the test statistics
for DEATHPEN. This result would occur regardless of the two numerical values used to
represent the groups; all we could do by changing these values would be to flip the sign
of the coefficient and inflate or deflate the coefficient and it’s standard error by an
equal factor, so that the absolute value of the ratio remains the same. This is true
because the existence of only two groups means that a numerical representation of a
comparison between them can only be done in one way; any differences in practical results
are due to scaling considerations. Another way of saying this is to note that since we
have only two groups, there can be only one degree of freedom in any test used to compare
them, and the results must therefore always be the same. Finally, though the identical
error sums of squares only intimate this and do not necessarily prove it, it is the case
that the predicted values produced by the two approaches are identical. In other words, we
have really fitted the same overall model in two slightly different ways.
We have yet to identify the codings given to the two levels of DEATHPEN that resulted
in the MANOVA parameter estimates. In MANOVA we specified DEATHPEN as a categorical factor
variable with codes of 0 and 1, and had the procedure internally create the design or
basis matrix required for the model fitting. In REGRESSION, only the constant or intercept
column of 1’s is provided automatically by the procedure; the other columns are provided
by the user in the form of the predictor variables specified. In MANOVA, the procedure
automatically creates a set of predictor variables to represent a factor instead of
requiring the user to do so. In the case of a dichotomous factor, MANOVA creates only one
predictor in addition to the constant term, and by default it gives this variable values
of 1 and -1, respectively. In our example, the states without the death penalty are the
first group (having factor variable value 0), and are coded 1, while states with the death
penalty receive a value of -1.
If we recall the interpretation of the regression coefficient as the increase in the
predicted value of the dependent variable for a unit increase in the predictor, we can see
why the DEATHPEN coefficient in MANOVA is -1/2 that of the one in REGRESSION. First, the
directionality has been changed. That is, an increase in the predictor means moving from
the death penalty group toward the no death penalty group. Thus the change in sign.
Second, in order to compare the two groups in this parameterization, we must move two
units, from -1 to 1, rather than from 0 to 1. Thus the two parameterizations are really
telling us exactly the same thing. This is further illustrated by using the MANOVA results
to predict the murder rates of the two groups. For the states with no death penalty, we
add the CONSTANT and DEATHPEN coefficients, giving us a predicted value of about 4.97. For
the death penalty group, we subtract the DEATHPEN coefficient from the CONSTANT, and
obtain a predicted value of about 7.86. These are of course the same values obtained using
REGRESSION.
What if we wanted to produce the same estimates in MANOVA that we obtained in
REGRESSION? The only straightforward way to produce exactly the same estimates would be to
enter the DEATHPEN predictor as a covariate coded 0-1. (There is a way to trick MANOVA
into providing the same coefficients as REGRESSION even with DEATHPEN as a factor, but
we’ll ignore that here.) The reason for this is that in it’s automatic reparameterization
or internal recoding of the factor(s), MANOVA enforces a sum to 0 restriction on the
values of the category codings. Thus 0-1 coding is not available. We can still obtain the
same parameter value for the difference between the two groups of states however. This can
be obtained by using SIMPLE contrasts with the first category as the reference category.
This uses category codes of -1/2 and 1/2, so that an increase of one unit in the predictor
would mean a change from no death penalty to the death penalty, and the resulting
coefficient would be the same in both magnitude and sign as that given in REGRESSION.
However, the constant or intercept term would still be the unweighted mean of the two
group means. Using the CONSTANT coefficient from the MANOVA output plus or minus 1/2 times
the DEATHPEN coefficient from the REGRESSION output, you can verify that this
parameterization again produces exactly the same predicted values as our earlier
approaches.
So much for the simple situation of a dichotomous predictor. As we have seen, in this
situation the coding of the variable is important in interpreting the value of the
regression coefficient, but not when we want to test whether the predictor has a nonzero
population relationship with the dependent variable. One way to think about this fact is
that when there are only two values of a predictor, there is only one interval between
those values, so the assumption of equal meanings of intervals is automatically satisfied.
However, once we move to predictors with more than two levels, things become more
complicated. We’ll save those complications for the next issue.
USING POLYTOMOUS PREDICTORS IN REGRESSION
David P. Nichols
Senior Support Statistician
SPSS, Inc.
From SPSS Keywords, Number 57, 1995
Here we extend our discussion begun last issue on categorical predictors in linear
regression to the case of a three level predictor variable. One way to do this in our
example is to distinguish between states with a death penalty statute but no executions in
a relevant time period and those where one or more executions took place during that time.
Here we will use a new variable called STATUS89, which takes on a value of 0 for those
states with no death penalty, 1 for the 28 states with death penalty statutes in force but
no executions taking place in 1989, and 2 for the eight states where executions occurred
during 1989. The no death penalty states are the same 14 states from last time, with a
mean 1990 murder rate of about 4.97. The 28 states with statutes but no 1989 executions
had a rate of about 6.98, and the eight executing states had a rate of about 10.96.
How should we represent our new three level predictor? The dummy variable approach is
again probably the most popular one among applied researchers. Since there are three
groups, we have two degrees of freedom for making comparisons. Thus we need two dummy
variables to represent STATUS89. If we make STATUS1 a dummy variable with a value of 1 for
the states with a value of 1 for STATUS89 and 0 otherwise, and make STATUS2 a dummy
variable with a value of 1 for the states with a value of 2 for STATUS89 and 0 otherwise,
then the STATUS1 coefficient will compare states with a value of 1 on STATUS89 with those
having a value of 0, and the STATUS2 coefficient will compare states with a value of 2 on
STATUS89 with those having a 0 value. One useful way to look at what we are doing is to
lay out the design matrix we are using at the cell level; that is, to list out the values
of the new predictor variables once for each different level of the original predictor.
This listing is given in Figure 3:
Figure 3
---------------------------------------------------------------------------
STATUS89 CONSTANT STATUS1 STATUS2
0 1 0 0
1 1 1 0
2 1 0 1
---------------------------------------------------------------------------
One question that may come to mind here is how does STATUS1 compare level 1 of STATUS89
to level 0 and not to both levels 0 and 2? Similarly, how does STATUS2 represent a
comparison between level 2 of STATUS89 and level 0 and not between level 2 and the other
two levels? To see why this is the case, note that cases at level 1 of STATUS89 differ
from those at level 0 only on STATUS1, while cases at level 2 differ from those at level 0
only on STATUS2. From this coding layout, we can see that the states at level 0 of
STATUS89 are predicted by the CONSTANT, while states at level 1 are predicted by the sum
of the CONSTANT and STATUS1 coefficients, and the states at level 2 of STATUS89 are
predicted by the sum of the CONSTANT and STATUS2 coefficients. Thus our regression
coefficients should yield the mean for the no death penalty states as the CONSTANT, the
difference between the non-executing death penalty states and the no death penalty states
as the STATUS1 coefficient, and the difference between the executing states and the no
death penalty states for the STATUS2 coefficient. As the output from REGRESSION in Figure
4 shows, this in indeed the case:
Figure 4
---------------------------------------------------------------------------
Multiple R .49443
R Square .24446
Adjusted R Square .21231
Standard Error 3.46979
Analysis of Variance
DF Sum of Squares Mean Square
Regression 2 183.08974 91.54487
Residual 47 565.85446 12.03946
F = 7.60374 Signif F = .0014
------------------ Variables in the Equation ------------------
Variable B SE B Beta T Sig T
STATUS1 2.007143 1.135756 .257430 1.767 .0837
STATUS2 5.991071 1.537821 .567498 3.896 .0003
(Constant) 4.971429 .927341 5.361 .0000
---------------------------------------------------------------------------
As noted earlier, most of the identities between various numbers on the output
disappear when multiple predictors are involved. The t-tests for the individual STATUS
coefficients indicate that the hypothetical population means for no death penalty and
death penalty but no executions groups may not be different, while the hypothetical
population mean for the executing states would seem to be higher than that for the no
death penalty states. The F-test in the analysis of variance table tests the null
hypothesis that both STATUS coefficients are 0 in the population, which means logically
that all three population means are equal. Note that despite the logical relationships
between hypotheses tested by the two types of tests (one or more nonzero coefficients
implies rejection of the omnibus null hypothesis, and vice versa), when making inferences
about population values based on sample data, it is possible to obtain logical
contradictions. That is, one or more individual t-tests may be "significant"
when the overall F-test is not, or the overall F-test may be "significant" when
no individual t-tests are. If the overall F-test is "significant" then a
paramterization can be found that will produce at least one "significant"
t-test, but in some cases this parameterization many not provide any useful interpretation
in terms of differences among group means.
Figure 5 shows what the default MANOVA parameterization produces for this analysis. As
can be seen in the ANOVA table, the overall F-value for STATUS89 is identical to that
given for the overall regression in REGRESSION. This is true because even though the
different parameterizations produce different coefficients, we are still using two
predictors to differentiate among three groups. Any two nonredundant predictors accurately
representing the differences among our three groups would produce the same results. This
is formally stated by saying that the overall test is invariant under different
parameterizations. That is, we are testing the same omnibus null hypothesis of equality
among the three population means, regardless of the specific contrast codings we use to
compare groups. As with the earlier dichotomous predictor, the CONSTANT coefficient in
this analysis represents the simple unweighted mean of the sample group means.
Figure 5
---------------------------------------------------------------------------
Tests of Significance for MURDER90 using UNIQUE sums of squares
Source of Variation SS DF MS F Sig of F
WITHIN+RESIDUAL 565.85 47 12.04
STATUS89 183.09 2 91.54 7.60 .001
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Estimates for MURDER90
--- Individual univariate .9500 confidence intervals
CONSTANT
Parameter Coeff. Std. Err. t-Value Sig. t Lower -95% CL- Upper
1 7.63750000 .55726 13.70539 .00000 6.51643 8.75857
STATUS89
Parameter Coeff. Std. Err. t-Value Sig. t Lower -95% CL- Upper
2 -2.6660714 .77278 -3.44996 .00119 -4.22071 -1.11143
3 -.65892857 .67370 -.97808 .33304 -2.01423 .69638
---------------------------------------------------------------------------
The two deviation coefficients for STATUS89 give the means of each of the first two
levels minus the unweighted grand mean of all three levels. This type of parameterization
is often referred to as "effect" coding, because the parameter for a level of a
factor variable is interpreted as the effect of being at that level of the predictor as
opposed to being at the overall average. This terminology is probably more familiar to
users whose linear models work has been handled in an analysis of variance framework, as
opposed to a multiple regression approach. What is important to note here is that
notwithstanding certain terminological preferences, both approaches are doing exactly the
same thing.
Since we’re fitting the same overall model in MANOVA that we did in REGRESSION, we must
be able to predict the same values for each state that were predicted earlier. In order to
be able to see exactly how this is done, we need to look at the design or basis matrix
that MANOVA used. As you can see by looking at the coefficients in matrix in Figure 6, the
predicted value for the no death penalty group is derived by summing the first and second
parameter estimates, the value for the death penalty but no executions group is obtained
by summing the first and third estimates, and the death penalty group value is obtained by
subtracting the second and third estimates from the first. By doing the arithmetic here
yourself, you can verify the identical predictions resulting from the two different
parameterizations.
Figure 6
---------------------------------------------------------------------------
STATUS89 CONSTANT STATUS89 STATUS90
(1) (2) (3)
0 1 1 0
1 1 0 1
2 1 -1 -1
---------------------------------------------------------------------------
INTERACTIONS AMONG DICHOTOMOUS PREDICTORS IN REGRESSION
David P. Nichols
Senior Support Statistician
SPSS, Inc.
August 1995
Here we continue the discussion of parameterization of linear regression models
involving categorical predictor variables from Keywords issues 56 and 57. In this article
we will deal with the problem of interactions among categorical predictors. We will
continue to use the United States data with 1990 murder rate as our dependent variable,
and also to suppose that the states might be viewed as a random sample from some
theoretical population of interest, in order to motivate attention to test statistics.
We’ll deal with the simplest case of interaction, that between two binary or
dichotomous predictor variables. DEATHPEN, introduced in issue 56, is a 0-1 variable
indicating absence or presence of a death penalty statute in 1989-90. CULTURE is a new
variable, again 0-1, indicating absence or presence of a certain level of influence of a
set of cultural characteristics thought by some social scientists to be involved in the
production of high rates of certain types of violence. Though both predictor variables are
measured prior to the dependent variable and are potentially causal contributants, there
are undoubtedly a number of other factors left out of our simple model. Thus, the
relationships shown here should be taken only as an illustration of basic regression
methods and not as a rigorous analysis of the contributants to murder rates.
Figure 7 contains the means, standard deviations and cell counts for the four
combinations of the two predictors. We can see that the means for states without the
cultural factor are lower than those with it, and that on average the states without the
death penalty have lower means. However, we also see that the mean for the CULTURE 0
states is higher for the states without the death penalty. There is thus evidence of a
differential impact of DEATHPEN depending on what level of CULTURE one considers. In other
words, it would appear that DEATHPEN interacts with CULTURE in it’s effects on MURDER90.
Formally, an interaction means that the effect of a predictor on the dependent variable is
dependent upon the level of the other predictor considered. (Incidentally, the 13 states
in the CULTURE 0-DEATHPEN 1 cell do not include any of the states carrying out executions
in 1989. Though a 2×3 analysis using the three level STATUS89 variable from issue 57 is
perhaps a more theoretically satisfactory one, it results in an empty combination of
predictors, which produces issues too complicated to be dealt with in this brief article.)
Figure 7: Descriptive Statistics
---------------------------------------------------------------------------
Variable .. MURDER90
FACTOR CODE Mean Std. Dev. N
CULTURE 0
DEATHPEN 0 4.840 4.307 10
DEATHPEN 1 4.092 1.506 13
CULTURE 1
DEATHPEN 0 5.300 1.671 4
DEATHPEN 1 9.996 2.796 23
---------------------------------------------------------------------------
We will first analyze the data using the REGRESSION procedure, entering the two dummy
variables CULTURE and DEATHPEN, and a product variable computed by multiplying the two
variables. This INTERACT product variable, when entered along with CULTURE and DEATHPEN,
represents the interaction of CULTURE and DEATHPEN. The results of the regression are
given in Figure 8.
Figure 8: REGRESSION results with dummy coding
---------------------------------------------------------------------------
Multiple R .70702
R Square .49988
Adjusted R Square .46726
Standard Error 2.85354
Analysis of Variance
DF Sum of Squares Mean Square
Regression 3 374.38140 124.79380
Residual 46 374.56280 8.14267
F = 15.32591 Signif F = .0000
------------------ Variables in the Equation ------------------
Variable B SE B Beta T Sig T
CULTURE .460000 1.688175 .059237 .272 .7865
DEATHPEN -.747692 1.200261 -.086742 -.623 .5364
INTERACT 5.443344 1.957121 .700974 2.781 .0078
(Constant) 4.840000 .902367 5.364 .0000
---------------------------------------------------------------------------
The most important thing to notice here is that the INTERACT variable has a
significance level of .0078, indicating that the interaction term should remain in the
model. Some people might look at the significance levels for the CULTURE and DEATHPEN
"main effects," which are both well above .05, and conclude that we have a
situation where this is an interaction but no main effects. This evinces a
misunderstanding of the meaning of an interaction. An interaction means that the effects
of a variable differ across the levels of another variable. In order for the effects of
one variable to differ at different levels of another variable, some of these effects must
be nonzero. Logically then, an interaction implies that all involved main effects are
present as well.
An important feature of the interaction model is brought out by comparing these results
with those from the MANOVA procedure, where we’ve fitted the same model, but used a
somewhat different parameterization. Recall that in our previous analyses involving only
main effects, the parameterization did not change the overall main effects F-test (only
the constant was affected). We see here that this no longer holds once an interaction is
introduced into the model. Figure 9 presents the (edited) MANOVA results using SIMPLE(1)
contrasts, which request comparison of the second category of each factor to the first,
just as is produced by dummy coding when we fit only main effects.
Figure 9: MANOVA results with SIMPLE(1) contrasts
--------------------------------------------------------------------------- Parameter Coeff. Std. Err. t-Value Sig. t Lower -95% CL- Upper CONSTANT 6.05698997 .48928 12.37939 .00000 5.07212 7.04186 CULTURE 3.18167224 .97856 3.25138 .00215 1.21193 5.15141 DEATHPEN 1.97397993 .97856 2.01723 .04953 .00424 3.94372 CULTURE BY 5.44334448 1.95712 2.78130 .00782 1.50386 9.38282 DEATHPEN ---------------------------------------------------------------------------
Again, the important thing to look at is the interaction term, which is identical to
that given by REGRESSION. Note that changing the parameterization might change the scaling
of the parameter here, but would not change the value of the t-statistic or it’s
significance. Relationships among individual parameters are more complicated when factors
have more than two levels, but the overall F-statistics remain the same regardless of
parameterization when the highest order term is being considered. Since all terms here
have one degree of freedom, the F-tests test the same thing as the t-tests, and have been
omitted to save space.
The next thing we might notice is that according to the MANOVA results, both of the
"main effects" are significant, unlike in the REGRESSION results. This is where
understanding how the model has been parameterized is crucial. Let’s look at the
"parameter codings" or basis matrices used in the two analyses to see how we can
square the two sets of findings. Figure 10 gives the values of the codings for the dummy
approach. The 4×4 matrix on the right is the basis or design matrix (X) used in the linear
model. The contrast matrix (C) produced by this coding scheme is given in Figure 11. The
relationship between the two matrices can be verified by evaluating the following
equation:
C = (X'X)-1X'
Figure 10: Parameterization using dummy codings
--------------------------------------------------------------------------- CULTURE DEATHPEN | CONSTANT CULTURE DEATHPEN INTERACT 0 0 | 1 0 0 0 0 1 | 1 0 1 0 1 0 | 1 1 0 0 1 1 | 1 1 1 1 ---------------------------------------------------------------------------
Reading across the first row of Figure 10, we can see that the CULTURE 0, DEATHPEN 0
states are represented completely by the CONSTANT parameter. This is borne out by the
values in the first row of the contrast matrix in Figure 11, where the CONSTANT is seen to
be simply the mean of the CULTURE 0, DEATHPEN 0 group, and by noting that the 4.84 value
for the CONSTANT in the REGRESSION output is the mean for those states. The second row of
the basis shows that the CULTURE 0, DEATHPEN 1 mean is modeled by summing the CONSTANT and
DEATHPEN parameters, which implies that the DEATHPEN parameter compares this group to the
CULTURE 0, DEATHPEN 0 group. The third line of the contrast matrix shows that the DEATHPEN
parameter is indeed comparing these two groups. Again, you should be able to derive the
parameter estimate value from the appropriate means (within printed levels of precision).
Thus the DEATHPEN effect here is really the simple main effect of DEATHPEN at the 0 level
of CULTURE (4.092-4.84=-.748), which according to the significance level printed is quite
possibly chance variation. The third row of the basis in Figure 10 and the second row of
the contrast matrix in Figure 11 show that the CULTURE parameter is assessing the simple
main effect of culture for the DEATHPEN 0 states (5.3-4.84=.46), which is also easily
attributable to chance. Finally, the INTERACT parameter estimates the difference between
the simple main effects of CULTURE at the two levels of DEATHPEN and vice versa:
5.44 = (9.996-4.092) - (5.3-4.84) = (9.996-5.3) - (4.092-4.84).
Thinking about the interaction parameter in this way illustrates how an interaction
implies main effects: interactions are differences among differences, and if all
differences are 0, then by definition the differences among the differences must also be
0.
Figure 11: Contrasts estimated by dummy codings
---------------------------------------------------------------------------
CULTURE0 CULTURE0 CULTURE1 CULTURE1
DEATHPEN0 DEATHPEN1 DEATHPEN0 DEATHPEN1
CONSTANT 1 0 0 0
CULTURE -1 0 1 0
DEATHPEN -1 1 0 0
INTERACT 1 -1 -1 1
---------------------------------------------------------------------------
Now let’s look at the basis and contrast matrices for the SIMPLE(1) coding used in
MANOVA (Figure 12). The last line of Figure 13 shows that the contrast estimated by the
interaction parameter is the same as with dummy coding, which fits with our earlier
observation that the parameter estimates for the interaction terms were the same in both
analyses. As we noted earlier though, none of the other parameters represent the same
things. The similarity of the basis and contrast matrices for the SIMPLE contrasts is an
artifact of the 2×2 design, and does not hold for designs involving larger numbers of
levels. In general, it is difficult to use the basis matrix with SIMPLE contrasts to see
what is being estimated.
Figure 12: Parameterization using SIMPLE(1) codings
--------------------------------------------------------------------------- CULTURE DEATHPEN | CONSTANT CULTURE DEATHPEN INTERACT 0 0 | 1 -.5 -.5 .25 0 1 | 1 -.5 .5 -.25 1 0 | 1 .5 -.5 -.25 1 1 | 1 .5 .5 .25 ---------------------------------------------------------------------------
The contrast matrix shows us that the CONSTANT parameter is simply estimating the
(unweighted) average of the four means. The CULTURE parameter estimates the average of the
two CULTURE 1 cells minus the average of the two CULTURE 0 cells, while the DEATHPEN
parameter estimates the average of the two DEATHPEN 1 cells minus the average of the two
DEATHPEN 0 cells. You should be able to use the cell means to reproduce the parameter
estimate values. Note that what are labeled as main effects here are averages of the
simple main effects of each factor across the levels of the other factor. Thus the 1.974
coefficient for DEATHPEN is the average of the 9.996-5.3=4.696 difference at level 1 of
CULTURE and the 4.092-4.84=-.748 difference at level 0 of CULTURE. Though such "main
effects" have become the norm for computer output from ANOVA/linear models
procedures, largely due to the simplicity of the hypotheses tested, the danger in taking
averages of different effects as representative of the whole is well illustrated by this
example. If DEATHPEN were a treatment, CULTURE denoted two types of patients and the
dependent variable were a measure of health, for example, we might conclude based on the
averaged effect that the treatment is good for everyone, when the results are really
telling us that it has no effect or is perhaps harmful for one group of patients.
Figure 13: Contrasts estimated by SIMPLE(1) codings
---------------------------------------------------------------------------
CULTURE0 CULTURE0 CULTURE1 CULTURE1
DEATHPEN0 DEATHPEN1 DEATHPEN0 DEATHPEN1
CONSTANT .25 .25 .25 .25
CULTURE -.50 -.50 .50 .50
DEATHPEN -.50 .50 -.50 .50
INTERACT 1.00 -1.00 -1.00 1.00
---------------------------------------------------------------------------
In addition to using the means to verify the interpretations of the parameter
estimates, you should be able to use the basis matrices and parameter estimate values to
reproduce the predicted values for each cell for both analyses. If you do this, you will
see that the predictions produced in each case are identical. This is true because have
fitted exactly the same model in both cases. The choice of parameterization strategy
affects only the interpretation of the individual parameters, not the overall model. The
most important thing to note here is that the interaction is the only term that is
independent of the choice of parameterizations. The presence of an interaction means that
there is no single main effect for a factor involved in that interaction, so different
choices of parameterization lead to different interpretations for the "main
effects."
CONTINUOUS BY CATEGORICAL INTERACTIONS IN REGRESSION
David P. Nichols
Senior Support Statistician
SPSS, Inc.
From SPSS Keywords, Number 61, 1996
Continuing the topic of using categorical variables in linear regression, in this issue
we will briefly demonstrate some of the issues involved in modeling interactions between
categorical and continuous predictors. As in previous issues, we will be modeling 1990
murder rates in the 50 states of the U. S. Our predictors will be the previously used 0-1
culture dummy variable, along with a new variable: state 1990 per capita income, expressed
as a percentage deviation from the national average (i.e., a value of 10 indicates that a
state’s per capita income was 10% above the national mean).
For most people, the parameterization of choice in this situation is to code the
culture dummy variable so that 0 means no and 1 means yes (that state is deemed to be
affected by the cultural factor of interest). The interaction variable is created by
multiplying the dummy variable by the income variable. Results of this REGRESSION are
given in Figure 14. As always, the constant is the predicted value for the dependent
variable when all predictors are 0. In this case, it represents the predicted 1990 murder
rate for a state without the cultural characteristic of interest, with 1990 per capita
income equal to the national average. The culture parameter gives the change in predicted
value for the affected states relative to the others when income is 0 (at the national
average). Affected states have a much larger predicted rate (almost twice as high). The
income parameter gives the predicted slope for the unaffected states; increases in per
capita income are associated with higher predicted murder rates for these states. The
interaction parameter estimates the difference in predicted slope for the affected states;
the -.13 value here can be added to the .08 value for income to obtain the predicted slope
for these states, which is about -.047. In other words, for these states, increases in
income are associated with decreases in predicted murder rates.
Figure 14: REGRESSION results with dummy coding (1=YES)
---------------------------------------------------------------------------- Variable B SE B Beta T Sig T CONSTANT 4.530989 .620501 7.302 .0000 CULTURE 4.377789 .909829 .563756 4.812 .0000 INCOME .082682 .040147 .318512 2.059 .0451 CUL_INC -.130178 .057808 -.366026 -2.252 .0291 ----------------------------------------------------------------------------
Note that the interpretation of the "main effects" is conditional upon the
level of the other variable, due to the presence of the interaction in the model. An
alternative parameterization is presented in Figure 15. Here, the dummy variable for
culture has been reverse coded, 0 for affected, 1 for not. Here, the constant now gives
the affected group’s predicted value at 0 or mean income. Culture again compares the two
groups at mean income, but this time subtracts affected from unaffected and thus has a
negative sign. Income gives the -.047 value we calculated above: the predicted slope for
the affected states. The interaction coefficient is the same but with an opposite sign.
Again, adding this to income produces the predicted slope for the group coded 1.
Figure 15: REGRESSION results with dummy coding (1=NO)
---------------------------------------------------------------------------- Variable B SE B Beta T Sig T CONSTANT 8.908778 .665407 13.388 .0000 CULTURE -4.377789 .909829 -.563756 -4.812 .0000 INCOME -.047495 .041593 -.182964 -1.142 .2594 CUL_INC .130178 .057808 .351942 2.252 .0291 ----------------------------------------------------------------------------
Note that the t-statistics for the constant and income parameters differ between the
two tables. This is because these parameters are estimates of different things under the
alternative parameterizations. You should be able to reproduce exactly the same predicted
value for a given combination of culture and income from the two parameterizations in
order to verify the fact that they are estimating the same overall model. For example, for
an affected state with an income value of 10, parameterization 1 would predict
4.530989 + 4.377789 * 1 + .082682 * 10 - .130178 * 1 * 10 = 8.4338
while parameterization 2 would predict
8.908778 - 4.377789 * 0 - .047495 * 10 + .130178 * 0 * 10 = 8.4338.
While the intercept and income terms here varied with the parameterization of the
model, the culture term did not. There are ways of parameterizing the model that would
produce different results for culture. Can you think of what some of these might be? We’ll
talk about this in the next issue.
FURTHER INTERACTIONS WITH CATEGORICAL VARIABLES IN REGRESSION
David P. Nichols
Senior Support Statistician
SPSS, Inc.
From SPSS Keywords, Number 62, 1996
In the last issue, we talked about interpretation of the parameters of a regression
model that included one dichotomous and one continuous predictor, plus their interaction.
As has been the case throughout this series, the point was to illustrate the dependence of
parameter interpretations on the way the predictor variables were coded; that is, on how
the model was parameterized. We saw how the interpretation of the "main effect"
of the continuous income variable was conditional upon the way the dichotomous culture
variable was coded. We also saw that in the two parameterizations we used (reversing the
0-1 dummy coding for culture), the culture "main effect" parameter had the same
absolute value, producing the same t-statistic and significance level. At the end of that
article, we alluded to the fact that this would not necessarily be the case under
alternative parameterizations, and promised to show why.
Figure 16 gives the same numbers as Figure 14 (from the above article). Recall that the
culture dummy variable was coded 0 for no and 1 for yes, and that the income variable was
expressed as a percentage deviation from the national mean. Though it was not specified at
the time, the national mean was based on an unweighted mean of individuals or a weighted
mean of states, so that states with higher populations contributed more heavily. When we
use such a variable in an analysis with each state treated as a single unit, the mean of
the income variable is not 0; in this case, it’s -5.08 (states with larger populations
tend to have higher relative incomes).
Figure 16: Original REGRESSION results (from previous issue)
------------------------------------------------------------------------------- Variable B SE B Beta T Sig T CONSTANT 4.530989 .620501 7.302 .0000 CULTURE 4.377789 .909829 .563756 4.812 .0000 INCOME .082682 .040147 .318512 2.059 .0451 CUL_INC -.130178 .057808 -.366026 -2.252 .0291 -------------------------------------------------------------------------------
Suppose we now recompute the income variable by centering it; that is, we make it so
that it has a mean of 0 in our sample (we do this by adding 5.08 to the earlier income
variable). What happens now when we recompute the interaction product variable for culture
and income and run a regression using the same dummy coding for culture? As you can see
from Figure 17, we now get different results for the constant term and for the culture
"main effect." The constant is of course the predicted value when all predictors
are set to 0. In both cases, this means culture=0. However, in Figure 16, it means that
income is set to 100% of the national mean, while in Figure 17, income is set to 94.92% of
the national mean. Thus the constant for Figure 17 is equal to the original constant minus
5.08 times the income coefficient:
4.110964 = 4.530989 - 5.08 * .082682.
Figure 17: Centered REGRESSION results
------------------------------------------------------------------------------- Variable B SE B Beta T Sig T CONSTANT 4.110964 .635702 6.467 .0000 CULTURE 5.039091 .864147 .648916 5.831 .0000 INCOME .082682 .040147 .318512 2.059 .0451 CUL_INC -.130178 .057808 -.343087 -2.252 .0291 -------------------------------------------------------------------------------
Notice also that the culture coefficient has changed, from 4.377789 to 5.039091, as has
it’s t-value and significance. This is because we are now estimating something different:
the difference in predicted value for a state with culture=1 compared with a state with
culture=0, but this time at income equal to -5.08 on the old scale (which is 0 on the new
scale). The culture coefficient in Figure 17 is (to within rounding error) the original
culture coefficient minus 5.08 times the interaction coefficient:
5.039091 = 4.377789 - 5.08 * (-.130178).
The primary implication of course is that the interpretation of the "main
effect" of the culture variable, like that of the income variable, depends on how the
model has been parameterized. There is no single interpretation of this effect available.
Another implication is that we can make this coefficient estimate the predicted difference
between groups at any fixed value of the income variable we choose, simply by subtracting
that value from the original variable. A common usage of this property is the centering of
variables so that comparisons are made at the mean of a variable rather than at the 0
point of the original continuous predictor, which often isn’t of interest.
Finally, you may have noticed that the interaction coefficient remained the same in
absolute value throughout our variable transformations. It is possible to change this
value by rescaling the predictors. If we change the distance between the two groups on the
dichotomous culture predictor, while keeping the continuous income predictor the same, the
result is to multiply the interaction coefficient by the reciprocal of the change in
distance (e.g., doubling the distance between the two group codes, to say -1 and 1 rather
than 0 and 1, results in a halving of the interaction coefficient). Similarly, keeping the
unit distance between the culture codings and multiplying the continuous income predictor
by a constant produces a reciprocal change in the interaction coefficient (e.g.,
multiplying the income variable by two produces an interaction coefficient half the size
of the original one). Note in each case that the interaction product variable must be
recomputed from the transformed original variables.
While linear transformations (multiplication by a constant and addition of another
constant, or new=a+b*old) of original variables will rescale the interaction term, the
standard error will also be rescaled, resulting in the same t-statistic and significance
level. The interaction term in this model is the highest order term in a hierarchical
model, and is thus _invariant_ under such transformations. It is the only term in this
model for which this is true. All lower order terms are "contained within" or
"marginal to" this interaction effect, and are thus dependent upon the specific
model parameterization for their meaning. In a nutshell, this is the lesson of this
series.
This page was adapted from a web page at the SPSS web page.
We thank SPSS for their permission to adapt and distribute this page via our web site.