Version info: Code for this page was tested in
SAS 9.3.
Multinomial logistic regression is for modeling nominal
outcome variables, in which the log odds of the outcomes are modeled as a linear
combination of the predictor variables.
Please Note: The purpose of this page is to show how to use various data analysis commands.
It does not cover all aspects of the research process which researchers are expected to do. In
particular, it does not cover data cleaning and checking, verification of assumptions, model
diagnostics and potential follow-up analyses.
Examples of multinomial logistic regression
Example 1. People’s occupational choices might be influenced
by their parents’ occupations and their own education level. We can study the
relationship of one’s occupation choice with education level and father’s
occupation. The occupational choices will be the outcome variable which
consists of categories of occupations.
Example 2. A biologist may be interested in food choices that alligators make. Adult alligators might have
difference preference than young ones. The outcome variable here will be the
types of food, and the predictor variables might be the length of the alligators
and other environmental variables.
Example 3. Entering high school students make program choices among general program,
vocational program and academic program. Their choice might be modeled using
their writing score and their social economic status.
Description of the data
For our data analysis example, we will expand the third example using the
hsbdemo data set. You can download the data
here .
proc contents data = "c:\hsbdemo"; run;The CONTENTS Procedure Data Set Name c:\datahsbdemo Observations 200 Member Type DATA Variables 13 Engine V9 Indexes 0 Created Thursday, August 29, 2013 09:42:59 AM Observation Length 40 Last Modified Thursday, August 29, 2013 09:42:59 AM Deleted Observations 0 Protection Compressed NO Data Set Type Sorted YES Label Written by SAS Data Representation WINDOWS_64 Encoding wlatin1 Western (Windows) Engine/Host Dependent Information Data Set Page Size 4096 Number of Data Set Pages 3 First Data Page 1 Max Obs per Page 101 Obs in First Data Page 42 Number of Data Set Repairs 0 Filename c:\datahsbdemo.sas7bdat Release Created 9.0301M1 Host Created X64_7PRO Alphabetic List of Variables and Attributes # Variable Type Len Label 12 AWARDS Num 3 13 CID Num 3 2 FEMALE Num 3 11 HONORS Num 3 honores eng 1 ID Num 4 8 MATH Num 3 math score 5 PROG Num 3 type of program 6 READ Num 3 reading score 4 SCHTYP Num 3 type of school 9 SCIENCE Num 3 science score 3 SES Num 3 10 SOCST Num 3 social studies score 7 WRITE Num 3 writing score Sort Information Sortedby PROG Validated YES Character Set ANSI
The data set contains variables on 200 students. The outcome variable is prog, program type. The predictor variables
are social economic status, ses, a three-level categorical variable
and writing score, write, a continuous variable. Let’s start with
getting some descriptive statistics of the
variables of interest.
proc freq data = "c:\hsbdemo"; tables prog*ses / chisq norow nocol nofreq; run;The FREQ Procedure Table of PROG by SES PROG(type of program) SES Percent | 1| 2| 3| Total --------+--------+--------+--------+ 1 | 8.00 | 10.00 | 4.50 | 22.50 --------+--------+--------+--------+ 2 | 9.50 | 22.00 | 21.00 | 52.50 --------+--------+--------+--------+ 3 | 6.00 | 15.50 | 3.50 | 25.00 --------+--------+--------+--------+ Total 47 95 58 200 23.50 47.50 29.00 100.00 Statistics for Table of PROG by SES Statistic DF Value Prob ------------------------------------------------------ Chi-Square 4 16.6044 0.0023 Likelihood Ratio Chi-Square 4 16.7830 0.0021 Mantel-Haenszel Chi-Square 1 0.0598 0.8068 Phi Coefficient 0.2881 Contingency Coefficient 0.2769 Cramer's V 0.2037 Sample Size = 200proc sort data = "c:\hsbdemo"; by prog; run; proc means data = "c:\hsbdemo"; var write; by prog; run;type of program=1 The MEANS Procedure Analysis Variable : WRITE writing score N Mean Std Dev Minimum Maximum ------------------------------------------------------------------- 45 51.3333333 9.3977754 31.0000000 67.0000000 ------------------------------------------------------------------- type of program=2 Analysis Variable : WRITE writing score N Mean Std Dev Minimum Maximum ------------------------------------------------------------------- 105 56.2571429 7.9433433 33.0000000 67.0000000 ------------------------------------------------------------------- type of program=3 Analysis Variable : WRITE writing score N Mean Std Dev Minimum Maximum ------------------------------------------------------------------- 50 46.7600000 9.3187544 31.0000000 67.0000000 -------------------------------------------------------------------
Analysis methods you might consider
- Multinomial logistic regression: the focus of this page.
- Multinomial probit regression: similar to multinomial logistic
regression but with independent normal error terms. - Multiple-group discriminant function analysis: A multivariate method for
multinomial outcome variables - Multiple logistic regression analyses, one for each pair of outcomes:
One problem with this approach is that each analysis is potentially run on a different
sample. The other problem is that without constraining the logistic models,
we can end up with the probability of choosing all possible outcome categories
greater than 1. - Collapsing number of categories to two and then doing a logistic regression: This approach
suffers from loss of information and changes the original research questions to
very different ones. - Ordinal logistic regression: If the outcome variable is truly ordered
and if it also satisfies the assumption of proportional
odds, then switching to ordinal logistic regression will make the model more
parsimonious. - Alternative-specific multinomial probit regression: allows
different error structures therefore allows to relax the independence of
irrelevant alternatives (IIA, see below “Things to Consider”) assumption.
This requires that the data structure be choice-specific. - Nested logit model: also relaxes the IIA assumption, also
requires the data structure be choice-specific.
Multinomial logistic regression
Below we use proc logistic to estimate a multinomial logistic
regression model. The outcome prog and the predictor ses are both
categorical variables and should be indicated as such on the class statement. We
can specify the baseline category for prog using (ref = “2”) and
the reference group for ses using (ref = “1”). The param=ref option
on
the class statement tells SAS to use dummy coding rather than effect coding
for the variable ses. Note that the levels of prog are defined as:
1=general
2=academic (reference group)
3=vocational
proc logistic data = "c:\hsbdemo"; class prog (ref = "2") ses (ref = "1") / param = ref; model prog = ses write / link = glogit; run; The LOGISTIC Procedure Model Information Data Set c:\datahsbdemo Written by SAS Response Variable PROG type of program Number of Response Levels 3 Model generalized logit Optimization Technique Newton-Raphson Number of Observations Read 200 Number of Observations Used 200 Response Profile Ordered Total Value PROG Frequency 1 1 45 2 2 105 3 3 50 Logits modeled use PROG=2 as the reference category. Class Level Information Design Class Value Variables SES 1 0 0 2 1 0 3 0 1 Model Convergence Status Convergence criterion (GCONV=1E-8) satisfied. Model Fit Statistics Intercept Intercept and Criterion Only Covariates AIC 412.193 375.963 SC 418.790 402.350 -2 Log L 408.193 359.963Testing Global Null Hypothesis: BETA=0 Test Chi-Square DF Pr > ChiSq Likelihood Ratio 48.2299 6 <.0001 Score 45.1588 6 <.0001 Wald 37.2946 6 <.0001 Type 3 Analysis of Effects Wald Effect DF Chi-Square Pr > ChiSq SES 4 10.8162 0.0287 WRITE 2 26.4633 <.0001 Analysis of Maximum Likelihood Estimates Standard Wald Parameter PROG DF Estimate Error Chi-Square Pr > ChiSq Intercept 1 1 2.8522 1.1664 5.9790 0.0145 Intercept 3 1 5.2182 1.1635 20.1128 <.0001 SES 2 1 1 -0.5333 0.4437 1.4444 0.2294 SES 2 3 1 0.2914 0.4764 0.3742 0.5407 SES 3 1 1 -1.1628 0.5142 5.1137 0.0237 SES 3 3 1 -0.9827 0.5956 2.7224 0.0989 WRITE 1 1 -0.0579 0.0214 7.3200 0.0068 WRITE 3 1 -0.1136 0.0222 26.1392 <.0001 Odds Ratio Estimates Point 95% Wald Effect PROG Estimate Confidence Limits SES 2 vs 1 1 0.587 0.246 1.400 SES 2 vs 1 3 1.338 0.526 3.404 SES 3 vs 1 1 0.313 0.114 0.856 SES 3 vs 1 3 0.374 0.116 1.203 WRITE 1 0.944 0.905 0.984 WRITE 3 0.893 0.855 0.932
-
- In the output above, the likelihood ratio chi-square of48.23 with a p-value < 0.0001 tells us that our model as a whole fits
significantly better than an empty model (i.e., a model with no
predictors)- Several model fit measures such as the AIC are listed under
Model Fit Statistics
- Several model fit measures such as the AIC are listed under
- In the output above, the likelihood ratio chi-square of48.23 with a p-value < 0.0001 tells us that our model as a whole fits
Two models are tested in this multinomial regression, one comparing
membership to general versus academic program and one comparing membership to
vocational versus academic program. They correspond to the two equations below:
$$ln\left(\frac{P(prog=general)}{P(prog=academic)}\right) = b_{10} + b_{11}(ses=2) + b_{12}(ses=3) + b_{13}write$$
$$ln\left(\frac{P(prog=vocation)}{P(prog=academic)}\right) = b_{20} + b_{21}(ses=2) + b_{22}(ses=3) + b_{23}write$$
where \(b\)s are the regression coefficients.
-
-
- A one-unit increase in the variable write is associated with a
.058 decrease in the relative log odds of being in general program vs.
academic program . - A one-unit increase in the variable write is associated with a
.1136 decrease in the relative log odds of being in vocation program vs.
academic program. - The relative log odds of being in general program vs. in academic program will
decrease by 1.163 if moving from the lowest level of ses (ses==1) to the
highest level of ses (ses==3). - The overall effects of ses and write are listed under
“Type 3 Analysis of Effects”, and both are significant.
- A one-unit increase in the variable write is associated with a
-
-
-
- The ratio of the probability of choosing one outcome category over the
probability of choosing the baseline category is often referred to as relative risk
(and it is also sometimes referred to as odds as we have just used to described the
regression parameters above). Relative risk can be obtained by
exponentiating the linear equations above, yielding regression coefficients that
are relative risk ratios for a unit change in the predictor variable.
In the case of two categories, relative risk ratios are equivalent to
odds ratios, which are listed in the output as well.
- The ratio of the probability of choosing one outcome category over the
-
-
-
- The odds ratio for a one-unit increase in the variable write
is .944 (exp(-.0579) from the regression coefficients above the odds
ratios) for being in general program vs. academic program. - The odds ratio of switching from ses = 1 to 3 is .313 for being
in general program vs. academic program. In other words, the expected risk
of staying in the general program is lower for subjects who are high in ses.
- The odds ratio for a one-unit increase in the variable write
-
Using the test statement, we can also test specific hypotheses within
or even across logits, such as if the effect of ses=3 in
predicting general versus academic equals the effect of ses = 3 in
predicting vocational versus academic. Use of the test statement requires the
unique names SAS assigns each parameter in the model. The option outest
on the proc logistic statement produces an output dataset with
the parameter names and values. We can get these names by printing them,
and we transpose them to be more readable. The noobs option on the proc print
statement suppresses observation numbers, since they are meaningless in the parameter dataset.
proc logistic data = "c:\hsbdemo" outest = mlogit_param; class prog (ref = "2") ses (ref = "1") / param = ref; model prog = ses write / link = glogit; run; proc transpose data = mlogit_param; run; proc print noobs; run; _NAME_ _LABEL_ PROG Intercept_1 Intercept: PROG=1 2.852 Intercept_3 Intercept: PROG=3 5.218 SES2_1 SES 2: PROG=1 -0.533 SES2_3 SES 2: PROG=3 0.291 SES3_1 SES 3: PROG=1 -1.163 SES3_3 SES 3: PROG=3 -0.983 WRITE_1 writing score: PROG=1 -0.058 WRITE_3 writing score: PROG=3 -0.114 _LNLIKE_ Model Log Likelihood -179.982
Here we see the same parameters as in the output above, but with their unique SAS-given names.
We are interested in testing whether SES3_general is equal to SES3_vocational,
which we can now do with the test statement. The code preceding the “:”
on the test statement is a label identifying the test in the output, and it must
conform to SAS variable-naming rules (i.e., 32 characters in length or less, letters,
numerals, and underscore).
proc logistic data = "c:\hsbdemo" outest = mlogit_param; class prog (ref = "2") ses (ref = "1") / param = ref; model prog = ses write / link = glogit; SES3_general_vs_SES3_vocational: test SES3_1 - SES3_3; run; ***SOME OUTPUT OMITTED***Linear Hypotheses Testing Results Wald Label Chi-Square DF Pr > ChiSq SES3_general_vs_SES3_vocational 0.0772 1 0.7811
The effect of ses=3 for predicting general versus academic is not different from the effect of
ses=3 for predicting vocational versus academic.
You can also use predicted probabilities to help you understand the model.
You can calculate predicted probabilities using the lsmeans statement and
the ilink option. For multinomial data, lsmeans requires glm
rather than reference (dummy) coding, even though they are essentially
the same, so be sure to respecify the coding on the class statement.
However, glm coding only allows the last category to be the reference
group (prog = vocational and ses = 3)and will ignore any other
reference group specifications. Below we use lsmeans to
calculate the predicted probability of choosing program type academic or general at each level
of ses, holding write at its means.
proc logistic data = "c:\hsbdemo" outest = mlogit_param; class prog ses / param = glm; model prog = ses write / link = glogit; lsmeans ses / e ilink cl; run; ***SOME OUTPUT OMITTED*** Coefficients for SES Least Squares Means type of Parameter program SES Row1 Row2 Row3 Row4 Row5 Row6 Intercept 1 1 1 1 Intercept 2 1 1 1 SES 1 1 1 1 SES 1 2 1 1 SES 2 1 2 1 SES 2 2 2 1 SES 3 1 3 1 SES 3 2 3 1 writing score 1 52.775 52.775 52.775 writing score 2 52.775 52.775 52.775***SOME OUTPUT OMITTED***SES Least Squares Means Standard type of Error of Lower Upper program SES Mean Mean Mean Mean 1 1 0.3582 0.07264 0.2158 0.5006 1 2 0.2283 0.04512 0.1399 0.3168 1 3 0.1785 0.05405 0.07256 0.2844 2 1 0.4397 0.07799 0.2868 0.5925 2 2 0.4777 0.05526 0.3694 0.5861 2 3 0.7009 0.06630 0.5709 0.8309
The predicted probabilities are in the “Mean” column. Thus, for ses
= 3 and write = 52.775, we see that the probability of being the academic
program (program type 2) is 0.7009; for the general program (program type 1),
the probability is 0.1785.
To obtain predicted probabilities for the program type vocational, we can reverse the ordering of the categories
using the descending option on the proc logistic statement.
This will make academic the reference group for prog and 3 the reference
group for ses.
proc logistic data = "c:\hsbdemo" outest = mlogit_param descending;
class prog ses / param = glm;
model prog = ses write / link = glogit;
lsmeans ses / e ilink cl;
run;
***SOME OUTPUT OMITTED***
Coefficients for SES Least Squares Means
type of
Parameter program SES Row1 Row2 Row3 Row4 Row5 Row6
Intercept 3 1 1 1
Intercept 2 1 1 1
SES 1 3 1 1
SES 1 2 1 1
SES 2 3 2 1
SES 2 2 2 1
SES 3 3 3 1
SES 3 2 3 1
writing score 3 52.775 52.775 52.775
writing score 2 52.775 52.775 52.775
***SOME OUTPUT OMITTED***
SES Least Squares Means
Standard
type of Error of Lower Upper
program SES Mean Mean Mean Mean
3 1 0.2021 0.05996 0.08459 0.3197
3 2 0.2939 0.05036 0.1952 0.3926
3 3 0.1206 0.04643 0.02960 0.2116
2 1 0.4397 0.07799 0.2868 0.5925
2 2 0.4777 0.05526 0.3694 0.5861
2 3 0.7009 0.06630 0.5709 0.8309
Here we see the probability of being in the vocational program when ses = 3 and
write = 52.775 is 0.1206, which is what we would have expected since (1 –
0.7009 – 0.1785) = 0.1206, where 0.7009 and 0.1785 are the probabilities of
being in the academic and general programs under the same conditions.
Things to consider
-
-
- The Independence of Irrelevant Alternatives (IIA) assumption: Roughly,
the IIA assumption means that adding or deleting alternative outcome
categories does not affect the odds among the remaining outcomes. - Diagnostics and model fit: Unlike logistic regression where there are
many statistics for performing model diagnostics, it is not as
straightforward to do diagnostics with multinomial logistic regression
models. Some model fit statistics are listed in the output. - Pseudo-R-Squared: The R-squared offered in the output is basically the
change in terms of log-likelihood from the intercept-only model to the
current model. It does not convey the same information as the R-square for
linear regression, even though it is still “the higher, the better”. - Sample size: Multinomial regression uses a maximum likelihood estimation
method. Therefore, it requires a large sample size. It also uses multiple
equations. Therefore, it requires an even larger sample size than ordinal or
binary logistic regression. - Complete or quasi-complete separation: Complete separation implies that only one value of a predictor variable is
associated with only one value of the response variable. You can tell from the output of the
regression coefficients that something is wrong. You can then do a two-way tabulation of the outcome
variable with the problematic variable to confirm this and then rerun the model
without the problematic variable. - Empty cells or small cells: You should check for empty or small
cells by doing a crosstab between categorical predictors and
the outcome variable. If a cell has very few cases (a small cell), the
model may become unstable or it might not run at all. - Sometimes observations are clustered into groups (e.g., people within
families, students within classrooms). In such cases, you may want to see
our page on non-independence within clusters.
- The Independence of Irrelevant Alternatives (IIA) assumption: Roughly,
-
See Also
References
-
-
- Hosmer, D. and Lemeshow, S. (2000) Applied Logistic Regression (Second
Edition). New York: John Wiley & Sons, Inc.. - Agresti, A. (1996) An Introduction to Categorical Data
Analysis. New York: John Wiley & Sons, Inc.
- Hosmer, D. and Lemeshow, S. (2000) Applied Logistic Regression (Second
-