Advantages of Meta-Analysis Over Literature Reviews

Compared to traditional literature reviews, meta-analysis has four major advantages:

  • The conclusions of literature reviews are drawn without using formal rules. The authors of literature reviews are free to decide which studies to emphasize, and which to downplay, in the results. This introduces an undesirable element of subjectivity into the analysis, allowing the results to be affected by the author's ideology and personal convictions. By contrast, meta-analysis offers a rule-based way of drawing conclusions, reducing the subjectivity of the results.

  • Literature reviews offer qualitative, not quantitative, conclusions. At best, the author can say that the effects are small or large, without being able to specify exactly how small or large they are. By reporting the mean effect size, a meta-analysis can provide more precise estimates of the impact of policy.

  • Literature reviews offer no estimate of the statistical significance of the results. In addition to knowing the size of an effect, policymakers also need to its likelihood – the degree of confidence we can place in the results. In other words, they need to know how likely it is that the results occurred by random chance. Meta-analysis provides an estimate of the joint significance of all the studies in the field taken together. Specifically, the analysis provides an answer to this question: how likely is it that we could have encountered all the results we see in the field, if the policy actually had no effect?

  • Literature reviews offer no systematic method for determining what factors influence the outcomes. Policy interventions may have different outcomes depending on the context of the intervention. What works in inner-city Chicago may not work in suburban Boston, and vice-versa. In scientific parlance, these context conditions are called “moderator variables.” Literature reviews examine these moderator variables only in a limited and qualitative fashion, but meta-analysis offers a rigorous way to correlate moderator variables with the size of policy effects.

Terms that you may encounter in the papers on this web-site:

  1. Effect size: a common measure used to compare results from multiple studies in the same field. Since studies do not all report their results in the same manner, some conversion is often necessary in order to put the results on the same scale. Each of the papers on this website uses a different effect size measure.

  2. Moderator variables: context variables which alter the effectiveness (or perceived effectiveness) of policy interventions. Common examples include location, age, gender, and features of the methodology of individual studies.

  3. Statistical significance: a measure of the our confidence in the reliability of the results. More formally, a measure of the likelihood that the observed results occurred by random chance through sampling error, rather than through the effects of a policy intervention.

  4. Inverse chi-squared (Fisher) test: a statistical test to determine the joint statistical significance of a group of studies taken together.

  5. Codebook: the set of rules used to extract data from individual studies. The codebook answers questions like: Which regression will be used from each study? Which coefficient will be taken from each regression? In some studies, the word “codebook” also is extended to mean the data so extracted.
  7. Click here to go to our links page for more information about meta-analysis.