Challenges and Opportunities of Meta-Analysis in Education Research
Abstract
Meta-analyses are systematic summaries of research that use quantitative methods to find the mean effect size (standardized mean difference) for interventions. Critics of meta-analysis point out that such analyses can conflate the results of low- and high-quality studies, make improper comparisons and result in statistical noise. All these criticisms are valid for low-quality meta-analyses. However, high-quality meta-analyses correct all these problems. Critics of meta-analysis often suggest that selecting high-quality RCTs is a more valid methodology. However, education RCTs do not show consistent findings, even when all factors are controlled. Education is a social science, and variability is inevitable. Scholars who try to select the best RCTs will likely select RCTs that confirm their bias. High-quality meta-analyses offer a more transparent and rigorous model for determining best practices in education. While meta-analyses are not without limitations, they are the best tool for evaluating educational pedagogies and programs.
Full Text:
PDFReferences
Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2011). Pedagogical strategies for teaching literacy to ESL immigrant students: a meta-analysis. The British journal of educational psychology, 81(Pt 4), 629–653. https://doi.org/10.1111/j.2044-8279.2010.02015.x
Bratsch-Hines, M., Vernon-Feagans, L., Pedonti, S., & Varghese, C. (2020). Differential effects of the Targeted Reading Intervention for students with low phonological awareness and/or vocabulary. Learning Disability Quarterly, 43(4), 214-226. https://doi.org/10.1177/0731948719858683
Camilli, G., Vargas, S., & Yurecko, M. (2003). Teaching Children to Read: The Fragile Link Between Science & Federal Education Policy. Education Policy Analysis Archives, 11, 15. https://doi.org/10.14507/epaa.v11n15.2003
Center for Research in Education and Social Policy. (2022). Reading Recovery: Long-term effects and cost-effectiveness. Under the investing in innovation (i3) scale-up. University of Delaware. https://www.cresp.udel.edu/research-project/efficacy-follow-study-long-term-effects-reading-recovery-i3-scale/
Clougherty, L. (2019). Emergent bilinguals and academic language acquisition through the use of sentence frames. Cal Poly Humboldt theses and projects, 247. https://digitalcommons.humboldt.edu/cgi/viewcontent.cgi?article=1304&context=etd
Fitton, L., McIlraith, A. L., & Wood, C. L. (2018). Shared book reading interventions with English learners: A meta-analysis. Review of Educational Research, 88(5), 712-751. https://doi.org/10.3102/0034654318790909
Ehri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M. (2001). Systematic phonics instruction helps students learn to read: Evidence from the National Reading Panel's meta-analysis. Review of Educational Research, 71(3), 393-447. https://doi.org/10.3102/00346543071003393
Farokhbakht, L. (n.d.). The effect of using multisensory-based phonics in teaching literacy on EFL young female/male learners' early reading motivation. University of Isfahan. https://jolly2.s3.amazonaws.com/Research/The%20Effect%20of%20Using%20Multisensorybased%20Phonics%20in%20Teaching%20Literacy%20on%20EFL%20Young%20FemaleMale%20Learners'%20Early%20Reading%20Motivation.pdf
Filderman, M. J., Austin, C. R., Boucher, A. N., O'Donnell, K., & Swanson, E. A. (2022). A meta-analysis of the effects of reading comprehension interventions on the reading comprehension outcomes of struggling readers in third through 12th grades. Exceptional Children, 88(2), 163-184. https://doi.org/10.1177/00144029211050860
Fitzgerald, R., & Hartry, A. (2008). What works in afterschool programs: The impact of a reading intervention on student achievement in the Brockton Public Schools (Phase II). Berkeley, CA: MPR Associates, Inc. and the National Partnership for Quality Afterschool Learning at SEDL. https://ies.ed.gov/ncee/wwc/Study/82599
Hansford, N & King, J. (2022). A meta-analysis and literature review of language programs. Teaching by Science. https://www.teachingbyscience.com/a-meta-analysis-of-language-programs
Hansford, N & McGlynn, S. (2022). Read 180. Teaching by Science. https://www.pedagogynongrata.com/read-180
Hechinger Report. (2022). Opinion: A call for rejecting the newest reading wars. https://hechingerreport.org/opinion-a-call-for-rejecting-the-newest-reading-wars/?hsCtaTracking=87e509b0-21a9-40e1-aa17-2d072a8d37f6%7C4aa6fa0e-fb2a-45c3-866c-05165077f3fd
Holliman, A., & Hurry, J. (2013). The effects of Reading Recovery on children's literacy progress and special educational needs status: A three-year follow-up study. Educational Psychology, 33, 10.1080/01443410.2013.785048 https://www.tandfonline.com/doi/abs/10.1080/01443410.2013.785048
Interactive, Inc. (2002). An efficacy study of READ 180: A print and electronic adaptive intervention program, grades 4 and above. Scholastic. http://teacher.scholastic.com/products/research/pdfs/ER_Council_Great_Schools.pdf
IntHout, J., Ioannidis, J., Borm, G., & Goeman, J. (2015). Small studies are more heterogeneous than large ones: A meta-meta-analysis. Journal of Clinical Epidemiology, 68(8), 860-869. https://doi.org/10.1016/j.jclinepi.2015.03.017
Jia. (2021). Toward a set of design principles for decoding training: A systematic review of studies of English as a foreign/second language listening education. Educational Research Review, 33, N.PAG. https://www.sciencedirect.com/science/article/abs/pii/S1747938X21000154
Lang, L., Torgesen, J., Vogel, W., Chanter, C., Lefsky, E., & Petscher, Y. (2009). Exploring the relative effectiveness of reading interventions for high school students. Journal of Research on Educational Effectiveness, 2, 149-175. https://doi.org/10.1080/19345740802641535
Lee, W., & Hotopf, W. (2012). Funnel plot. Science Direct. https://www.sciencedirect.com/topics/medicine-and-dentistry/funnel-plot#:~:text=The%20funnel%20plot%20is%20a,this%20difference%20which%20is%20detectable
Li, R. (2022). Effects of blended language learning on EFL learners' language performance: An activity theory approach. Journal of Computer Assisted Learning, 38(5), 1273-1285. https://doi.org/10.1111/jcal.12697
Lv, X., Ren, W., & Xie, Y. (2021). The effects of online feedback on ESL/EFL writing: A meta-analysis. Asia-Pacific Education Researcher, 30(6), 643-653. https://doi.org/10.1007/s40299-021-00594-6
Meisch, A., Hamilton, J., Chen, E., Quintanilla, P., Fong, P., Gray-Adams, K., & Thornton, N. (2011). Striving Readers study: Targeted and whole-school interventions-year 5. Rockville, MD: Westat. https://eric.ed.gov/?id=ED601086
NRP. (2001). Teaching children to read: An evidence-based assessment of the scientific literature on reading instruction. United States Government. Retrieved from https://www.nichd.nih.gov/sites/default/files/publications/pubs/nrp/Documents/report.pdf
Piasta, S. B., & Wagner, R. K. (2010). Developing early literacy skills: A meta-analysis of alphabet learning and instruction. Reading Research Quarterly, 45(1), 8-38. https://doi.org/10.1598/RRQ.45.1.2
Roessingh, H. (2004). Effective high school ESL programs: A synthesis and meta-analysis. Canadian Modern Language Review, 60(5), 611-636. https://doi.org/10.3138/cmlr.60.5.611
Rui Li. (2022). Effects of mobile-assisted language learning on EFL/ESL reading comprehension. Journal of Educational Technology & Society, 25(3), 15-29. https://www.jstor.org/stable/48673721
Schenck, A. (2020). Using meta-analysis of technique and timing to optimize corrective feedback for specific grammatical features. Asian-Pacific Journal of Second and Foreign Language Education, 5, 1-20. https://doi.org/10.1186/s40862-020-00097-9
Sprague, K., Zaller, C., Kite, A., & Hussar, K. (2012). Springfield-Chicopee School Districts Striving Readers program final report Years 1-5: Evaluation of implementation and impact. Providence, RI: The Education Alliance at Brown University.
Stuebing, K. K., Barth, A. E., Cirino, P. T., Francis, D. J., & Fletcher, J. M. (2008). A response to recent reanalyses of the National Reading Panel report: Effects of systematic phonics instruction are practically significant. Journal of Educational Psychology, 100(1), 123-134. https://doi.org/10.1037/0022-0663.100.1.123
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113-2126. https://doi.org/10.1002/sim.1461
Thompson, C. (2020). Video-game-based instruction for vocabulary acquisition with English language learners: A Bayesian meta-analysis. Educational Research Review, 30, N.PAG. https://www.researchgate.net/publication/340640024_Video-game_based_instruction_for_vocabulary_acquisition_with_English_language_learners_A_Bayesian_meta-analysis
Unkyoung Maeng. (2014). The effectiveness of reading strategy instruction: A meta-analysis. English Teaching, 69(3), 105-127. https://doi.org/10.15858/engtea.69.3.201409.105
Vaden-Kiernan, M., Borman, G., Caverly, S., Bell, N., Sullivan, K., Ruiz de Castilla, V., Fleming, G., Rodriguez, D., Henry, C., Long, T., & Hughes Jones, D. (2018). Findings from a multiyear scale-up effectiveness trial of Open Court Reading. Journal of Research on Educational Effectiveness, 11(1), 109-132. https://doi.org/10.1080/19345747.2017.1342886
DOI: https://doi.org/10.51383/ijonmes.2023.313
Refbacks
- There are currently no refbacks.
Copyright (c) 2023 http://creativecommons.org/licenses/by/4.0
This work is licensed under a Creative Commons Attribution 4.0 International License.