The Effect Size The effect size (ES) makes metaanalysis possible The ES encodes the selected research findings on a numeric scale There are many different types of ES measures, each suited to different research situations Each ES type may also have multiple methods of computation Practical Meta-Analysis -- D. B. Wilson Examples of Different Types of Effect Sizes (ES) Standardized mean difference Group contrast research Treatment groups Naturally occurring groups Inherently continuous construct

Odds-ratio (OR) Group contrast research Treatment groups Naturally occurring groups Inherently dichotomous construct Correlation coefficient Association between variables research Practical Meta-Analysis -- D. B. Wilson Examples of Different Types of Effect Sizes Risk-ratio or Relative Risk (RR) Group differences research (naturally occurring groups) Commonly used by epidemiologist and medical meta-analyses

Inherently dichotomous construct Easier to interpret than the odds-ratio (OR) Does not overstate ES like OR does. Practical Meta-Analysis -- D. B. Wilson Examples of Different Types of Effect Sizes Proportion Central tendency research HIV/AIDS prevalence rates Proportion of homeless persons found to be alcohol abusers Standardized gain score Gain or change between two measurement points on the same variable Cholesterol level before and after completing a therapy

Others? Practical Meta-Analysis -- D. B. Wilson What Makes Something an Effect Size for Meta-analytic Purposes The type of ES must be comparable across the collection of studies of interest This is generally accomplished through standardization Must be able to calculate a standard error for that type of ES The standard error is needed to calculate the ES weights, called inverse variance weights (more on this later) All meta-analytic analyses are weighted averages (simple example on next slide)

Practical Meta-Analysis -- D. B. Wilson Weighted Averages Example: Calculating GPAs Practical Meta-Analysis -- D. B. Wilson Weighted Averages Example: Calculating GPAs Practical Meta-Analysis -- D. B. Wilson Weighted Averages Example: Calculating GPAs Practical Meta-Analysis -- D. B. Wilson Weighted Averages Example: Calculating GPAs

Practical Meta-Analysis -- D. B. Wilson The Standardized Mean Difference X X ES s G1 G2 s pooled s12 n1 1 s22 n2 1 n1 n2 2 pooled

Represents a standardized group contrast on an inherently continuous measure Uses the pooled standard deviation (some situations use control group standard deviation) Commonly called d or occasionally g Cohens d (see separate short lecture on Cohens d) Practical Meta-Analysis -- D. B. Wilson The Correlation Coefficient (r) ES r Represents the strength of linear association between two inherently continuous measures Generally reported directly as r (the Pearson product moment

correlation) Practical Meta-Analysis -- D. B. Wilson The Odds-Ratio (OR) Recall, the odds-ratio is based on a 2 by 2 contingency table, such as the one below Frequencies o o Success Failure Treatment Group a

b Control Group c d ad ES bc The Odds Ratio (OR) is the odds for success in the treatment group relative to the odds for success in the control group. ORs can also come from results of logistic regression analysis, but these would difficult to use in a meta-analysis due to model

differences. Practical Meta-Analysis -- D. B. Wilson Relative Risk (RR) The relative risk (RR) is also based on data from a 2 by 2 contingency table, and is the ratio of the probability of success (or failure) for each group a / (a b ) ES c / (c d ) Practical Meta-Analysis -- D. B. Wilson Unstandardized Effect Size Metric If you are synthesizing a research domain that using a common measure across studies, you may wish to use an effect size that is

unstandardized, such as a simple mean difference. Multi-site evaluations or evaluation contracted by a single granting agency. Practical Meta-Analysis -- D. B. Wilson Effect Size Decision Tree for Group Differences Research (from Wilson & Lipsey) All dependent variables are inherently dichotomous Difference between proportions as ES [see Note 1] Group contrast on dependent variable Odds ratio; Log of the odds ratio

as ES All dependent variables are inherently continuous All dependent variables measured on a continuous scale All studies involve same measure/scale Unstandardized mean difference ES Studies use different measures/scales Standardized mean difference ES

Some dependent variables measured on a continuous scale, some artificially dichotomized Some dependent variables are inherently dichotomous, some are inherently continuous Standardized mean difference ES; those involving dichotomies computed using probit [see Note 2] or arcsine [see Note 3] Do separate meta-analyses for dichotomous and continuous variables Practical Meta-Analysis - Wilson &

Lipsey Methods of Calculating the Standardized Mean Difference The standardized mean difference probably has more methods of calculation than any other effect size type. Practical Meta-Analysis -- D. B. Wilson Poor Good Great Degrees of Approximation to the ES

Value Depending of Method of Computation Direct calculation based on means and standard deviations Algebraically equivalent formulas (t-test) Exact probability value for a t-test Approximations based on continuous data (correlation coefficient) Estimates of the mean difference (adjusted means, regression b weight, gain score means) Estimates of the pooled standard deviation (gain score standard deviation, one-way ANOVA with 3 or more groups, ANCOVA) Approximations based on dichotomous data Practical Meta-Analysis -- D. B. Wilson Methods of Calculating the Standardized Mean Difference (Independent

Samples) Direction Calculation Method X1 X 2 X1 X 2 ES s pooled s12 (n1 1) s22 (n2 1) n1 n2 2 Practical Meta-Analysis -- D. B. Wilson Methods of Calculating the Standardized Mean Difference (Independent Samples)

Algebraically Equivalent Formulas: n1 n2 ES t n1n2 ES independent samples t-test F (n1 n2 ) two-group one-way ANOVA n1n2 Exact p-values from a t-test or F-ratio can be convert into t-value and the above formula applied. Practical Meta-Analysis -- D. B. Wilson

Methods of Calculating the Standardized Mean Difference A study may report a grouped frequency distribution from which you can calculate means and standard deviations and apply to direct calculation method. = =1

=1 2 = =1

( )2 =1 Practical Meta-Analysis -- D. B. Wilson 1

Methods of Calculating the Standardized Mean Difference (Independent Samples) Close Approximation Based on Continuous Data: Point-Biserial Correlation - For example, the correlation between treatment/no treatment and 2 r outcome measured ES 2 on a continuous scale. 1 r

Point-Biserial Correlation: Pearsons Product Moment Correlation (r) between the response (Y) and group indicator (X) coded as: Group 1 = 0, Group 2 = 1 and treated as a numeric variable. Practical Meta-Analysis -- D. B. Wilson Methods of Calculating the Standardized Mean Difference (Independent Samples) Estimates of the Numerator of ES The Mean Difference difference between covariance adjusted means unstandardized regression coefficient (b) for group membership Practical Meta-Analysis -- D. B. Wilson

Methods of Calculating the Standardized Mean Difference (Independent Samples, more than two groups) Estimates of the Denominator of ES Pooled Standard Deviation s MS F between pooled MS between

MS 2 ( X n ) j j 2 X n j j nj k1

error one-way ANOVA more than 2 groups should be found in an ANOVA table in the paper Methods of Calculating the Standardized Mean Difference Estimates of the Denominator of ES - Standard Deviation of the Paired Differences (Gain Scores) s SE n 1 d SE = standard error of the mean

the paired differences Paired difference between scores = gain scores Gain = pre-test post-test (or vise versa) Practical Meta-Analysis -- D. B. Wilson Methods of Calculating the Standardized Mean Difference Estimates of the Denominator of ES -Pooled Standard Deviation s pooled s gain 2(1 r ) standard deviation of gain scores, where r is the

correlation between pretest and posttest scores Practical Meta-Analysis -- D. B. Wilson Methods of Calculating the Standardized Mean Difference Estimates of the Denominator of ES -Pooled Standard Deviation s pooled MS error df error 1 2 1 r df error 2 ANCOVA, where r is the

correlation between the covariate and the dependent variable. Practical Meta-Analysis -- D. B. Wilson Methods of Calculating the Standardized Mean Difference Estimates of the Denominator of ES -Pooled Standard Deviation s pooled SS B SS AB SSW df B df AB dfW A two-way factorial ANOVA where B is the irrelevant factor

and AB is the interaction between the irrelevant factor and group membership (factor A). Practical Meta-Analysis -- D. B. Wilson Methods of Calculating the Standardized Difference between Two Proportions Approximations Based on Dichotomous Data ES probit ( p group1 ) probit ( p group2 ) - the difference between the probits transformation of the proportion successful in each group - converts proportion into a z-value Practical Meta-Analysis -- D. B. Wilson

Methods of Calculating the Standardized Mean Difference Approximations Based on Dichotomous Data ES 3 log Odds ratio this represents the rescaling of the logged odds-ratio (see Sanchez-Meca et al 2004 Psychological Methods article) Practical Meta-Analysis -- D. B. Wilson 29 Methods of Calculating the Standardized Mean Difference

Approximations Based on Dichotomous Data 2 ES 2 N 2 2r ES 1 r chi-square must be based on a 2 by 2 contingency table (i.e., have only 1 df) phi coefficient 2 Practical Meta-Analysis -- D. B. Wilson

Practical Meta-Analysis -- D. B. Wilson LINKED TO LECTURE SECTION OF COURSE WEBSITE Practical Meta-Analysis -- D. B. Wilson Formulas for the Correlation Coefficient (r) Results typically reported directly as a correlation. Any data for which you can calculate a standardized mean difference effect size, you can also calculate a correlation type effect size. Practical Meta-Analysis -- D. B. Wilson Formulas for the Odds Ratio Results typically reported in one of three forms:

Frequency of successes in each group Proportion of successes in each group 2 by 2 contingency table Practical Meta-Analysis -- D. B. Wilson Data to Code Along With the ES The effect size (ES)

May want to code the statistics from which the ES is calculated Confidence in ES calculation Method of calculation Any additional data needed for calculation of the inverse variance weight Sample size ES specific attrition Construct measured Point in time when variable measured Reliability of measure Type of statistical test used Practical Meta-Analysis -- D. B. Wilson Issues in Coding Effect Sizes Which formula to use when summary statistics are available for multiple

formulas Multiple documents/publications reporting the same data (not always in agreement) How much guessing should be allowed sample size is important but may not be presented for both groups some numbers matter more than others Practical Meta-Analysis -- D. B. Wilson