When modeling a count variable, there are several available models to choose
from. Stata supports Poisson, negative binomial, zero-inflated Poisson,
zero-inflated negative binomial models, zero-truncated Poisson, and
zero-truncated negative binomial models. Among these, you can likely
narrow your data down to one or two models based on how the data were collected
and the distribution of your outcome variable.
Some important questions to ask yourself before you start building a count model
are:
- Does my outcome variable contain zeroes?
- If yes, what does a zero mean?
- If no, why not?
- How is my outcome variable distributed?
- What is the variance of my outcome variable?
- How does the variance compare to the mean?
The answers to the first set of questions can indicate whether your data are
zero-truncated or zero-inflated.
If your count variable does not contain zeroes…
ask yourself why. Given what you know about the variable, would you have
expected to see counts of zero? In some instances, a count of zero will
never appear in the dataset. For example, if your count variable is the number
of visits to a given coffee shop in a week and your data are collected among people
in the coffee shop,
then all the people for whom you collected data have a count of at least one. A
person who never visits the coffee shop will not be included in the dataset. This
is an example of zero-truncated data and would most appropriately be
modeled using a zero-truncated model. However, just because a variable
does not contain zeroes does not mean the variable is zero-truncated.
Truncation occurs because of the way in which the data are collected and cannot
be assumed just from looking at the data.
If your count variable does contain zeroes…
again,
ask yourself why.
What does a count of zero mean, given what you know about the variable? In some
instances, records in a dataset could ONLY have counts of zero. For
examples, if you are collecting data on the number of fish caught by a person in
a given weekend, a person who did not go fishing could only have a count of
zero. A person who did go fishing could also have a count of zero, but these two
zeroes would have very different meanings. This is an example of
zero-inflated data and may be most appropriately modeled with a
zero-inflated model.
The answers to the second set of questions can indicate whether a Poisson or negative
binomial distribution is more appropriate for your data. The Poisson
distribution is characterized by equal mean and variance. If the sample mean and
sample variance of your outcome variable are in the same neighborhood, a Poisson
model may be appropriate. If the variance exceeds the mean by a great deal, a
negative binomial model may be appropriate. Keep in mind that zero-truncation
and zero-inflation can impact these statistics and there is not a magical
cut-off determining when one model is better than the other. These are simply
aspects of your data worth considering.
Deciding on a final model
You may find that, after making all of the above considerations and deciding
which predictors should be included in your model, you are still torn. You have
two count models that both seem substantively reasonable, but that are clearly
mathematically different. At this point, the countfit function in Stata
(written by Long and Freese) can be a helpful tool. You can download it by
typing search countfit (see How can I
use the search command to search for programs and get additional help? for
more information about using search) and following the appropriate links.
Countfit runs user-specified count models (Poisson, zero-inflated
Poisson, negative binomial, and zero-inflated negative binomial) with
user-specified variables and compares the model residuals. If you do not
indicate which models you wish to compare (nbreg, zinp, prm,
or zip), countfit will default to
running and comparing all four. One if the quirks of countfit is that it displays results for a given predictor next to the predictor’s
label rather than name. So
an unlabeled predictor will lead to both empty space in the output and possible
confusion. This can be prevented by labeling ALL predictors before running
countfit.
In this example, we will be looking at academic information on
316 students. The response variable is days absent during the school year (daysabs).
We have narrowed our model choices down to two negative binomial models, one
with zero-inflation and one without. We believe that the daysabs can best
be predicted with math standardized tests score (mathnce),
language standardized tests score (langnce) and gender (female).
We suspect certain zeroes can be predicted with bilingual status (biling)
and language score (langnce). We will run the countfit command
indicating these predictors and the two models we wish to compare,
then discuss the output.
use https://stats.idre.ucla.edu/stat/stata/notes/lahigh, clear generate female = (gender == 1)label variable female `"female"'countfit daysabs mathnce langnce female, inf(biling langnce) nbreg zinb
---------------------------------------------------------- Variable | NBRM ZINB ---------------------------------+------------------------ daysabs | ctbs math nce | 0.998 0.998 | -0.33 -0.34 ctbs lang nce | 0.986 0.987 | -2.57 -2.38 female | 1.539 1.502 | 3.09 2.94 Constant | 9.825 9.795 | 10.89 10.94 ---------------------------------+------------------------ lnalpha | Constant | 1.288 1.191 | 2.65 1.62 ---------------------------------+------------------------ inflate | bilingual status | 10.132 | 1.53 ctbs lang nce | 1.094 | 1.92 Constant | 0.000 | -2.16 ---------------------------------+------------------------ Statistics | alpha | 1.288 N | 316.000 316.000 ll | -880.873 -879.131 bic | 1790.525 1804.307 aic | 1771.746 1774.262 ---------------------------------------------------------- legend: b/t Comparison of Mean Observed and Predicted Count Maximum At Mean Model Difference Value |Diff| --------------------------------------------- NBRM 0.011 1 0.004 ZINB 0.018 1 0.005 NBRM: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.201 0.004 0.031 1 0.146 0.135 0.011 0.278 2 0.098 0.104 0.006 0.093 3 0.082 0.083 0.001 0.004 4 0.070 0.068 0.001 0.008 5 0.063 0.057 0.006 0.230 6 0.047 0.048 0.000 0.001 7 0.038 0.040 0.002 0.046 8 0.028 0.034 0.006 0.318 9 0.035 0.029 0.005 0.319 ------------------------------------------------ Sum 0.804 0.799 0.044 1.327 ZINB: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.207 0.011 0.176 1 0.146 0.127 0.018 0.850 2 0.098 0.101 0.003 0.022 3 0.082 0.082 0.000 0.000 4 0.070 0.068 0.001 0.007 5 0.063 0.057 0.006 0.194 6 0.047 0.048 0.001 0.006 7 0.038 0.041 0.003 0.077 8 0.028 0.035 0.007 0.393 9 0.035 0.030 0.005 0.240 ------------------------------------------------ Sum 0.804 0.798 0.055 1.965 Tests and Fit Statistics ------------------------------------------------------------------------- NBRM BIC= -28.290 AIC= 5.607 Prefer Over Evidence ------------------------------------------------------------------------- vs ZINB BIC= -14.507 dif= -13.783 NBRM ZINB Very strong AIC= 5.615 dif= -0.008 NBRM ZINB Vuong= 0.858 prob= 0.195 ZINB NBRM p=0.195
Making sense of the output
First, let’s discuss the graph. Countfit produces a graph that plots the residuals from the tested models. Small residuals are indicative of good-fitting
models, so the models with lines closest to zero should be considered for our
data. This may be useful in eliminating a model, but be careful
about deciding on one model over all the others based solely on this graph.
In the graph above, we see that our two models perform very similarly for counts
greater than two, and that they both differ most from the actual values and each other at the
zero and one counts. At the zero and one counts, the negative binomial
model appears slightly better than the zero-inflated negative binomial model.
Model parameters and fit
The first table in the output summarizes the parameter estimates from each of
the tested models. For each of the models, we see the exponentiated
coefficients and their t-statistics in the first block of the table.
Then, for any negative binomial models, we will see the estimated dispersion parameters.
Next, for any zero-inflated models, we see the estimates from the logistic model
predicting the certain zeroes. In the last block of the table, a set of fit
statistics is provided for each of the four models. This includes the
log-likelihood, BIC, and AIC.
----------------------------------------------------------
Variable | NBRM ZINB
---------------------------------+------------------------
daysabs |
ctbs math nce | 0.998 0.998
| -0.33 -0.34
ctbs lang nce | 0.986 0.987
| -2.57 -2.38
female | 1.539 1.502
| 3.09 2.94
Constant | 9.825 9.795
| 10.89 10.94
---------------------------------+------------------------
lnalpha |
Constant | 1.288 1.191
| 2.65 1.62
---------------------------------+------------------------
inflate |
bilingual status | 10.132
| 1.53
ctbs lang nce | 1.094
| 1.92
Constant | 0.000
| -2.16
---------------------------------+------------------------
Statistics |
alpha | 1.288
N | 316.000 316.000
ll | -880.873 -879.131
bic | 1790.525 1804.307
aic | 1771.746 1774.262
----------------------------------------------------------
legend: b/t
From the last block, we can see that the two models are extremely
close. The parameter estimates are nearly identical. We can continue to look at
the rest of the output.
Residuals by count
Next, we see a table with one line per model showing the maximum and mean
differences in observed versus predicted counts.
Comparison of Mean Observed and Predicted Count
Maximum At Mean
Model Difference Value |Diff|
---------------------------------------------
NBRM 0.011 1 0.004
ZINB 0.018 1 0.005
This confirms what we observed in the graph: both models performed worst when
predicting a count of 1. Between these two, we see that the negative binomial
did better at this prediction and, overall, had a lower mean absolute difference
between predicted and observed values. At this point, the negative binomial
model is looking more appropriate than the zero-inflated negative binomial
model. Next, we have one table for each of the models containing count-by-count
information.
NBRM: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.201 0.004 0.031 1 0.146 0.135 0.011 0.278 2 0.098 0.104 0.006 0.093 3 0.082 0.083 0.001 0.004 4 0.070 0.068 0.001 0.008 5 0.063 0.057 0.006 0.230 6 0.047 0.048 0.000 0.001 7 0.038 0.040 0.002 0.046 8 0.028 0.034 0.006 0.318 9 0.035 0.029 0.005 0.319 ------------------------------------------------ Sum 0.804 0.799 0.044 1.327 ZINB: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.207 0.011 0.176 1 0.146 0.127 0.018 0.850 2 0.098 0.101 0.003 0.022 3 0.082 0.082 0.000 0.000 4 0.070 0.068 0.001 0.007 5 0.063 0.057 0.006 0.194 6 0.047 0.048 0.001 0.006 7 0.038 0.041 0.003 0.077 8 0.028 0.035 0.007 0.393 9 0.035 0.030 0.005 0.240 ------------------------------------------------ Sum 0.804 0.798 0.055 1.965
In these two tables, we are able to see, for counts 0-9, the actual
proportion of our data records with the given count and the predicted proportion
from each models. The absolute
difference is included, as is the given count’s contribution to a Pearson
Chi-Square statistic comparing the actual distribution of the data and the
distribution proposed by the model. For a given row, the Pearson statistic can
be calculated as N(|Diff|2)/Predicted, where N is the number of
observations in the dataset. Looking at the sum of the Pearson column
gives us a sense of how close the predicted proportions were to the actual
proportions. Using this method to compare, the negative binomial appears better
than the zero-inflated negative binomial.
Summary comparisons
Tests and Fit Statistics
-------------------------------------------------------------------------
NBRM BIC= -28.290 AIC= 5.607 Prefer Over Evidence
-------------------------------------------------------------------------
vs ZINB BIC= -14.507 dif= -13.783 NBRM ZINB Very strong
AIC= 5.615 dif= -0.008 NBRM ZINB
Vuong= 0.858 prob= 0.195 ZINB NBRM p=0.195
In this table, the tested models are compared to each other head-to-head using
the tests appropriate to each comparison. Each line can be boiled down to
the last three columns. They suggest which model is preferred by the given
comparison and the strength of the evidence supporting this preference.
When we compare our two models using the BIC and AIC,
the negative binomial is preferred over zero-inflated negative
binomial. The Vuong test prefers zero-inflated negative binomial model over the
negative binomial model, but not at a statistically significant level.
Thus, these model fit statistics support what we have seen in the model
residuals.
Full model output
There may be times when you wish to see the full model output for each of the
four models. This can be especially helpful in choosing between two models
because it allows you to see the significance levels of the parameters, which
are not included in the countfit output. For instance, in this
example, we might be interested in the significance levels of our zero-inflated
variables. To do
this, you can add the noisily option to the countfit command:
countfit daysabs mathnce langnce female, inf(biling langnce) nbreg zinb noisily
This will print the full output from each of the four models, then the summary output provided without the
noisily option.
Replace option
When you run countfit, variables are generated
and added to your dataset. If you ran the same countfit command a
second time, you would encounter error messages telling you that variables have
already been defined. Thus, if you are running multiple countfit tests,
you can add the replace option to your command to replace the variables
generated by prior runs with the variables generated by the current run.
Final thoughts
Countfit presents
several nice ways for you to compare models that are not necessarily easy to
compare. It presents side-by-side fit statistics and parameter estimates, it
gives you a graphical representation of the model residuals, as well as a table
of the residuals by counts. All of these pieces of information are worth taking
into consideration when choosing a model. For your purposes, some of these
measures may be more relevant than the others. Think carefully about what you
are aiming to do in the model and which measures should be prioritized over
others. While countfit is very convenient and provides lots of
information, sorting through the information it provides must be done carefully.