When it comes to sample sizes, how big is big enough? Here’s a quick introduction to planning out the optimal sample size to study the effects of your program.
The number of participants you need in your sample depends first and foremost on the size of the effect you anticipate. The smaller the effect, the more people (or classrooms, or schools) you need to find it. If that sounds a little abstract, here’s a metaphor I like to use to keep it straight.
Imagine that you’re on a game show and you’re challenged to find an object in a haystack within a set time limit. You can get as many people to help you as you want, all of whom will split the prize money. Let’s say you’re told that there’s a bicycle hidden in the haystack. Shouldn’t be too hard to find, right? You pick three people for your team. What if you were instead looking for a needle in that same haystack? You would substantially increase the number of searchers.
Studying a program’s effects can often feel like looking for a needle in a haystack of data. If you expect the program will have a large effect on your outcome of choice, it won’t be too hard to find. If the effect is smaller, you’ll need to study many more people to detect it. Thus, the initial question to design your sample size is: how big do you expect the effect of a program to be on your outcome of interest?
Second, what is the likelihood of attrition from your study? Several factors can affect study attrition, including the length of the study and the amount of effort that the study requires. The longer and more effortful a study is, the more likely people will be to drop out, making it harder to “find” the effect.
Third, what additional data can be collected to better understand the “true” effect of a program? You can decrease the number of people you’ll need in your sample size by collecting additional information, like demographics or pre-tests. You can use these data as covariates in your analysis, enabling you to see the “true” effect of a program with a smaller sample. In other words, co-variates can shrink the size of the haystack.
With an understanding of the relationship between sample size and effect size, and an idea of the effect size, attrition, and coviariates in your study, you are now ready to begin working with several programs to help you estimate the number of people you’ll need to study the effectiveness of a curriculum or program. Below are additional resources to determine your sample size.
Happy sampling!
Sample Size and Effect Size Articles
Lipsey M. W., Puzio, K., Yun, C., Hebert M. A., Steinka-Fry, K., Cole M. W., Roberts M., Anthony K. S., & Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. (NCSER 2013-3000). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. Available online: http://ies.ed.gov/ncser/pubs/20133000/
Middlemis Maher, J., Markey, J. C. & Ebert-May, D. (2013). The other half of the story: Effect size analysis in quantitative research. CBE-Life Sciences Education, 12(3), 345-351. Available online: http://tinyurl.com/CBEEffectSize
Sample Size Calculators
G*Power: Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160. Available online: http://www.gpower.hhu.de/en.html
Optimal Design: Raudenbush, S.W., Spybrook, J., Congdon, R., Liu, X., Martinez, A., Bloom, H., & Hill, C. (2011). Optimal Design Plus Empirical Evidence (Version 3.0). Available online: http://www.wtgrantfoundation.org/resources/optimal-design