Statistical Power Analysis for Univariate Meta-Analysis: A Three-Level Model
Statistical power analysis has long been a requirement for researchers seeking funding from the Institute of Education Sciences (IES). As in all individual studies, power analysis is also important when conducting meta-analytic review studies to ensure that the study has sufficient ability to detect an overall treatment effect of interest across a large group of related studies. For example, suppose a meta-analytic review determines that a school intervention significantly improves student performance and has good power to detect that effect, then, researchers will have more confidence in further investing in, developing, and recommending the specific intervention for extensive usage. Calculating statistical power can also inform researchers as they design studies. For instance, power analysis can inform the necessary number of studies needed in their sample to detect an effect across all of those studies in a meta-analysis. This study extends prior research on power analysis for univariate meta-analysis and adds new aspects that facilitate the calculations of statistical power.
A three-level model in meta-analysis considers heterogeneity across research groups
There are a few common approaches to conduct meta-analysis. However, recent realizations suggest that the same authors often publish several studies in a certain topic, and thus may be represented many times in the meta-analysis. To address this issue, approaches to calculating statistical power in these studies should account for the repeated representation of the same study teams. Thus, in our study, we formally introduce methodology that adds third level units in the meta-analysis.
In the proposed three-level meta-analysis, the effect sizes are nested within studies, which in turn are nested within research groups of investigators (see the illustrative figure). Specifically, in this illustration, one effect size (e.g., ES 1) is extracted from each study (e.g., Study 1) and several studies (e.g., Study 1 to Study i) are linked to a research group (e.g., Research Group 1) because they are conducted by the same authors. The variance between these third level units (i.e., research groups) may influence the power of the meta-analysis. Consequently, the proposed three-level model takes into account the between-study (second level) and the between-research group (third level) variances and produces more accurate power estimates.
How to Read this Chart . This is an illustration of a three-level structure of effect sizes.
Power analysis in the three-level meta-analytic model
In this study, we first provide closed-form equations for computing prospective power in three-level meta-analysis. Then, we provide examples to illustrate how power calculations are carried out.
Second, this study extends the univariate two-level meta-analytic model to a three-level model, which considers heterogeneity across research groups in the third level. The study provides general procedures to construct the variance-covariance structure of the residuals in the three-level model and offers equations to calculate statistical power in three-level univariate meta-analysis.
Moreover, a simulation study is conducted to demonstrate how prospective statistical power is modified by changes of the values of the design parameters (e.g., effect size, number of research groups, within-study variance, level-2 and level-3 heterogeneity information) that impact power within the context of the three-level model.
Full Article Citation:
B. Zhang., S. Konstantopoulos (2023). Statistical power analysis for univariate meta-analysis: A three-level model. Journal of Research on Educational Effectiveness. https://doi.org/10.1080/19345747.2023.2290544