Introduction
Power analysis is the name given to the process for determining the sample size for a research study. The technical definition of power is that it is the probability of detecting a “true” effect when it exists. Many students think that there is a simple formula for determining sample size for every research situation. However, the reality it that there are many research situations that are so complex that they almost defy rational power analysis. In most cases, power analysis involves a number of simplifying assumptions, in order to make the problem tractable, and running the power analysis numerous times with different variations to cover all of the contingencies.
In this unit we will try to illustrate the logit power analysis process using a simple logistic regression with a single continuous predictor. We will follow up this example with a multiple logistic regression model with five predictors.
Description of the Experiment
A small and very exclusive liberal arts college wishes to do a quantitative analysis of their admission process. Currently, the college uses an admissions committee made up of administrators, faculty and students to admit 70 freshmen each school year. The admissions committee looks at grades, test scores and essays in deciding which students to admit. They do not, however, use any quantitative methods to justify their academic judgment.
The registrar wishes to validate the admissions process using a logistic regression model. The first model the registrar wants to try uses the Verbal SAT to predict the admit/no admit (0/1) response variable. The registrar expects that only 8% of students (probability = .08) who score at the mean will be admitted. He would like to test whether students who score one standard deviation above the mean will be 15% (increase the probability of .08 by .15 to .23) more likely to be admitted.
The Power Analysis
We will make use of the Stata program powerlog (search powerlog) (see How can I use the search command to search for programs and get additional help? for more information about using search) to do the power analysis for simple logistic regression. The powerlog program needs the following information in order to do the power analysis: 1) the probability of being admitted when scoring at the mean of the Verbal SAT (p1 = .08), 2) the probability of being admitted when scoring one standard deviation above the mean on the Verbal SAT (p2 = .08 + .15 = .23), and 3) the alpha level (alpha = .05 for this example). Since there is only one predictor variable in the model, we will leave the rsq option at zero. We include the help option to display the explanation of the terms in the command.
powerlog, p1(.08) p2(.23) alpha(.05) help
Logistic regression power analysis
One-tailed test: alpha=.05 p1=.08 p2=.23 p2-p1=.15 rsq=0
power n
0.60 73
0.65 81
0.70 89
0.75 98
0.80 109
0.85 123
0.90 141
Explanation of terms
p1 -- the probability that the response variable equals 1
when the predictor is at the mean
p2 -- the probability that the response variable equals 1
when the predictor is one standard deviation above the mean
rsq -- the squared multiple correlation between the predictor
variable and all other variables in the model
The output indicates that 73 observations would be needed to have a power of .6 and that 109 observations are needed for a power of .8. These numbers are probably the bare minimum needed since logistic regression uses maximum likelihood estimation which many researchers believe needs fairly large sample sizes.
The registrar is in luck because he has data on 192 students (70 admitted and 122 not admitted). But what if the registrar wanted to use an alpha level of .01? How many observations would be needed?
powerlog, p1(.08) p2(.23) alpha(.01) Logistic regression power analysis One-tailed test: alpha=.01 p1=.08 p2=.23 p2-p1=.15 rsq=0 power n 0.60 139 0.65 149 0.70 160 0.75 172 0.80 187 0.85 204 0.90 228
The registrar’s luck is still holding since with alpha set to .01, the logistic regression would need 187 observation for a power of .8.
Let’s go back to alpha = .05 and see what happens if we increase the effect size. In t-tests and ANOVAs, effect size is given in terms of mean differences and standard deviations. In logistic regression effect size can be stated in terms of the probability at the mean of the predictor and the probability at the mean plus one standard deviation. In the first model the probability at the mean was .08 and at the mean plus one standard deviation was .23. To increase the effect size to .2 we leave p1 at .08 and increase p2 to .28.
powerlog, p1(.08) p2(.28) alpha(.05) Logistic regression power analysis One-tailed test: alpha=.05 p1=.08 p2=.28 p2-p1=.2 rsq=0 power n 0.60 117 0.65 127 0.70 138 0.75 151 0.80 165 0.85 183 0.90 206
These results may be different than you were expecting. The sample sizes have actually gone up. The reason for this is the nonlinear nature of logistic regression.
This next analysis demonstrates that it is important to realize that the sample size needed for an effect size of .2 depends upon the p1 value. Let’s shift p1 up to .4, keep the effect size at .2 for a p2 of .6 and rerun the previous analysis.
powerlog, p1(.4) p2(.6) alpha(.05) Logistic regression power analysis One-tailed test: alpha=.05 p1=.4 p2=.6 p2-p1=.2 rsq=0 power n 0.60 40 0.65 45 0.70 51 0.75 57 0.80 65 0.85 74 0.90 87
Since this effect size is centered about the probability of .5, i.e., it is in the middle of the probability distribution, a much small sample size is needed than detecting the same size change in the tails of the probability curve. Again, this due to the nonlinear nature of the model in terms of probability.
Now, let’s see what happens when we use multiple predictors in the model while still focusing of the effect of Verbal SAT. Let’s say that there are five predictors in the model; we will need to specify the squared multiple correlation of Verbal SAT with the other four predictors. We will run two two analyses setting rsq, the squared multiple correlation, to .2 and then to .4. We will stick with the original p1 = .08 and p2 = .23 and alpha of .05.
powerlog, p1(.08) p2(.23) alpha(.05) rsq(.2) Logistic regression power analysis One-tailed test: alpha=.05 p1=.08 p2=.23 p2-p1=.15 rsq=.2 power n 0.60 92 0.65 101 0.70 111 0.75 123 0.80 137 0.85 154 0.90 176 powerlog, p1(.08) p2(.23) alpha(.05) rsq(.4) Logistic regression power analysis One-tailed test: alpha=.05 p1=.08 p2=.23 p2-p1=.15 rsq=.4 power n 0.60 122 0.65 135 0.70 148 0.75 164 0.80 182 0.85 205 0.90 235
As you can see, as the R-squared with the other predictors goes up, the number of observations needed also goes up.
Discussion
The power analysis for logistic regression looks, on the surface, to be relatively straight forward. However, when you get into it, you might find that it can be difficult to come up with reasonable and meaningful estimates of the two probabilities that are needed. One of the best ways to come up with these probabilities is through a pilot study that closely mimics your research design. The standard suggestion to obtain values from the literature may not work as well with logistic regression because published articles do not always give enough information to determine the probability at both the mean and at one standard deviation above the mean. Many researchers think that all that is necessary is to know the change in probability. However, since logistic regression is a nonlinear model, knowledge of the first probability is necessary since the sample size for a .1 change in probability starting at .2 is larger than a .1 change starting at .5.
It is also necessary to reiterate that the sample sizes generated by powerlog should be considered to be a lower bound. Although the sample sizes provided are valid for hypotheses testing with a specified power, methodologist are not in complete agreement as to how big sample sizes need to be to obtain stable estimates. Long (1997) suggests that sample sizes of less than 100 should be avoided and that 500 observations should be adequate for almost any situation. However, this leaves a relatively large gap between 100 and 500. If powerlog gives a sample size of less than 100, you might want to increase it to at least 100, just to be safe. Even if powerlog suggests an N of say 110, you might want to use an larger sample if you believe that you data might be problematic in any way.
See Also
- References
-
Agresti, A. 2002. Categorical Data Analysis, Second Edition
Hoboken, NJ: Wiley.
Agresti, A. 1996. An Introduction to Categorical Data Analysis. New York: Wiley.
Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences, Second Edition. Mahwah, NJ: Lawrence Erlbaum Associates.
Long, J.S. 1997. Regression Models for Categorical and Limited Dependent Variables. Thousand Oaks, CA: SAGE Publications, Inc.