Introduction
Power analysis is the name given to the process for determining the sample
size for a research study. The technical definition of power is the
probability of detecting a "true" effect when it exists. Many students think
that there is a simple formula for determining sample size for every research
situation. However, the reality is that there are many research situations that
are so complex that they almost defy rational power analysis. In most cases,
power analysis involves making simplifying assumptions that make
the problem tractable and running the analyses numerous times with different
variations to cover all of the contingencies.
In this unit we will try to illustrate the power analysis process using a
simple four group design.
Description of the experiment
We wish to conduct a study in the area of mathematics education involving
different teaching methods to improve standardized math scores in local
classrooms. The study will include four different teaching methods and use
fourth grade students who are randomly sampled from a large urban school
district and are then random assigned to the four different teaching methods.
The four different teaching methods to be examined are 1) the
traditional teaching method where the classroom teacher explains the concepts
and assigns homework problems from the textbook; 2) the intensive practice
method, in which students fill out additional work sheets both before and after
school; 3) the computer assisted method, in which students learn math concepts
and skills from using various computer based math learning programs; and 4) the
peer assistance learning method, which pairs each fourth grader with a fifth
grader who helps them learn the concepts followed by the student teaching the
same material to another student in their group.
Students will stay in their math learning groups for an entire academic year.
At the end of the spring semester all students will take the Multiple Math
Proficiency Inventory (MMPI). This standardized test has a mean for fourth
graders of 550 with a standard deviation of 80.
The experiment is designed so that each of the four groups will have the same
sample size. One of the important questions we need to answer in designing the
study is how many students will be needed in each group?
The power analysis
In order to answer this question, we will need to make some assumptions and
some educated guesses about the data. First, we will assume that the four groups
will have the same standard deviation and that it will be equal to the national
value of 80. Further, after looking at prior research, we expect that the
traditional teaching group (Group 1) will have the lowest mean score and that
the peer assistance group (Group 4) will have the highest mean score on the MMPI.
In fact, we expect that Group 1 will have a mean of 550 and that Group 4 will
have mean that is greater by 1.2 standard deviations, i.e., the mean will equal
at least 646. For the sake of simplicity, we will assume that the means of the
other two groups will be equal to the grand mean.
To perform the power analysis, we will make use of Sample Power’s ‘Oneway
analysis of variance’ procedure, under the ‘ANOVA’ tab in the
Procedures catalog.
To proceed, we require 1) the number of levels (or groups), 2) the effect
size (called delta), and 3) the alpha level. As stated above, there are four
groups. We will set alpha = 0.05 and the standard deviation (SD within
cell) to 80 and we will then compute the effect size.
There are a number of ways to accomplish this, all of them accessible by
clicking on either the number of cases input cell or the effect size input cell.
A window is opened, requesting a variety of inputs.
First, we indicate how many groups we have in the box below "Number of categories in this factor"–we
set this to 4.
From here, clicking the ‘Enter Means’ tab, we can input the expected means of
the four groups. We already have the mean = 550 for the lowest group and
the mean = 646 for the highest group. We will first set the means for the two
middle groups to be the grand mean. Based on this setup and the assumption that
the common standard deviation is equal to 80, we can do some simply calculation
to see that the grand mean will be 598. Alternatively, we can use the
‘Enter Range’ tab, inputting the high and low means and selecting a centered
distribution (given our assumption that the means of the two middle groups will
be equal to the grand mean). Regardless of the method used, clicking
‘Compute f’ followed by ‘Register f’ will calculate the effect size and transfer
it to the main window.
Let’s now set the power to be .8 and calculate the corresponding sample size
by clicking the ‘Find N for any power’ button.
This means we need total of 17*4 = 68 subjects for the power of .8.
If we want to see how sample size affects power, we can close the ‘Find
N for any power’ window, adjust the number in ‘N of cases per cell’ (in the
lower right-hand corner), and observe the change in achieved power.
So we see that when we have 25 subjects in each group, we will have power of
.95.
In the setup above, we worked under the assumption that the two middle groups
will have means equal to the grand mean. Now in general, the means for the two
middle groups can be anything in between. If you have a good idea on what these
means should be, you might want to make use of this piece of information in your
power analysis. For example, let’s say the means for the two middle groups
should be 575 and 635. We will set the power to .8. Note that for this
calculation, the ‘Enter Means’ tab in the Effect Size window must be used.
So we see that for power of .8 we need fewer subjects than before when the
two middle groups have the mean as the grand mean. This should be expected since
the power calculated here corresponds to the overall F test for the ANOVA and
when the means
are further away from the grand mean, it is easier to detect the
group effect.
Effect size
The difference of the means between the lowest group and the highest group
over the common standard deviation is a measure of effect size (note that this
measure of effect size (delta) is in fact different from the one used and
displayed in Sample Power (f)). In the calculation above, we have used 550 and
646 with common standard deviation of 80. This gives effect size of (646-550)/80
= 1.2. This is considered to be a large effect size. Let’s say now we have a
medium effect size of .75. What does this translate into in terms of groups
means? Well, we can always use 550 for the lowest group. The mean for the
highest group will be .75*80 + 550 = 610. Let’s assume the means of the two middle groups
are the grand mean, say g. Then we have (550 + g + g + 610) / 4 = g.
This gives us g = (550 + 610)/2 = 580. Let’s now redo our sample size
calculation with this set of means.
So we see that at size of 40 for each group, we have power of .8.
What about a small effect size, say, .25? We can do the same calculation as
we did previously. The mean for each of the groups will be 550, 560, 560 and
570.
When the effect size is small, the required sample size increases
drastically.
Discussion
The sample size calculation is based a number of assumptions. One of these
is the normality assumption for each group. We also assume that the groups have
the same common variance. As our power analysis calculation is rooted in these
assumptions, it is important to remain aware of them.
We have also assumed that we have some knowledge of the magnitude of effect we are
going to detect which is described in terms of group means. As we saw in
the last example with the small effect size, this can have an enormous impact on
the required sample size. When we are unsure
about the groups means, we should use more conservative estimates. For example,
we might not have a good idea on the two means for the two middle groups, then
setting them to be the grand mean is more conservative than setting them to be
something arbitrary.
Here are the sample sizes per group that we have come up with in our power
analysis: 17 (best case scenario), 40 (medium effect size), and 350 (almost the
worst case scenario). Even though we expect a large effect, we will shoot for a
sample size of between 40 and 50. This will help ensure that we have enough
power in case some of the assumptions mentioned above are not met or in case we
have some incomplete cases (i.e., missing data).
For more information on power analysis, please visit our
Introduction to Power Analysis
seminar.