# Likelihood Ratio, Wald, and Lagrange Multiplier (Score) Tests Likelihood Ratio, Wald, and Lagrange Multiplier (Score) Tests Soccer Goals in European Premier Leagues - 2004 Statistical Testing Principles Goal: Test a Hypothesis concerning parameter value(s) in a larger population (or nature), based on observed sample data Data Identified with respect to a (possibly hypothesized) probability distribution that is indexed by one or more unknown parameters Notation: Data: y1 ,..., yn Parameter(s): 1 ,..., k Joint Density Function: f y1 ,..., yn 1 ,..., k Example English League Total Goals/Match Suppose we wish to test whether the mean number of goals (in a hypothetically infinite population) of games is equal to 3. Note: all games of equal length (no overtime in regular season games) Data: Y=Total # of goals in a randomly selected game Distribution: Assume Poisson with parameter Null Hypothesis: H0: = 3 Alternative Hypothesis: HA: 3 Joint Probability Density Function: n n yi n yi e e i1 f y1 ,..., yn n yi ! i 1 yi ! i 1 yi 0,1, 2,... Likelihood Function Another term for joint probability density/mass function. Common Notation: L() or L(,yy) or L(|yy) Considered as a function of both the (observed) data and the (unknown) parameter values Used in estimation and testing parameter value(s) Goal is to choose parameter value(s) that maximize likelihood function given the observed data. Typically work with the log of the likelihood, as it is often easier to differentiate to solve for maximum likelihood (ML) estimators for many families of probability distributions ML Estimation of Poisson Mean n yi

n L, y e i1 n y ! i i 1 n n l ln L , y n yi ln ln yi ! i 1 i 1 Taking derivative (wrt ) and setting to zero for maximum: n n y dl n i 1 d i set 0 0 n y i n i 1 ^ 0 ^ y i 1 n i y Goals 0 1 2

3 4 5 6 7 8 9 10+ Total Frequency 30 79 99 67 61 24 11 6 2 1 0 380 380 ^ y i i 1 380 975 2.57 380 Total Goals Data Likelihood Ratio Test Identify the parameter space: Identify the parameter space under H0: Evaluate the maximum log-Likelihood Evaluate the log-Likelihood under H0 Any terms not involving parameter can be ignored Take -2 times difference (H0 maximum) Under null hypothesis (and large samples), statistic is approximately chi-square with 1 degree of freedom (number of constraints under H0) X 2 LR

^ 2 ln L 0 , y ln L , y Soccer Goals Example 380 380 ln L , y 380 yi ln ln yi ! i 1 i 1 380 Under H 0 : 3 (Ignoring ln yi ! ) : i 1 ln L 3, y 380(3) 975 ln(3) 1140 1071.15 68.85 ^ Maximum Value @ 2.57 : ^ ln L , y 380(2.57) 975 ln( 2.57) 976.6 920.31 56.29 Test Statistic: X 2 LR ^ 2 2 ln L 3, y ln L , y 2 68.85 ( 56.29) 25.12 .05,1 3.84 We have strong evidence to conclude the true mean total number of goals is below 3. Wald Test - I By Central Limit Theorem arguments, many estimators have sampling distributions that are approximately normal in large samples Then, if we have an estimate of the variance of the estimator, we can obtain a chi-square statistic by taking the square of the distance between the ML estimate and the value under H0 divided by the estimated variance The estimated variance can be obtained from the second derivative of the log-Likelihood Wald Test - II ^ V I 1 2 ln( L) where: I E 2 2

Wald Chi-Square Statistic: X W2 ln L , y Poisson Model: ^ 2 0 ^ ^ ^ ^ I 0 V n n n yi ln ln yi ! i 1 i 1 n ln L , y n y i n i 1 2 ln L , y 2 y i

i 1 2 2 ln( L) n n I E 2 2 2 X W2 ^ ^ n 2 0 0 ^ ^ ^ ^ I 0 ^ V 2 2 380 2.57 3 27.34 2.57 Lagrange Multiplier (Score) Test Obtain the first derivative of the log-Likelihood evaluated at the parameter under H0 (This is the slope of the log-Likelihood, evaluated at 0 and is called the score) Multiply the square of the score by the variance of the ML estimate, evaluated at 0 . This is the inverse

of the variance of the score. Then chi-square test statistic is computed as follows: X 2 LM s 0 , y I 0 2 where s , y ln L , y Soccer Goals Example n n ln L , y n yi ln ln yi ! i 1 i 1 n s , y n I 2 X LM ln L , y y i n i 1 s 0 , y 380 n 385 I 0 0 3 s , y 0 I 0

2 2 55 23.57 385 1 3 2 2 Note that: X W2 27.34 X LR 25.12 X LM 23.57 975 55 3 Generalization to Tests of Multiple Parameters 1 Parameter Vector: k H 0 : R r R11 R1k R Rg1 Rgk r1 r rg ^ Maximum Likelihood Estimator over entire parameter space: ~ Maximum Likelihood Estimator over constraint under H 0 : Likelihood Ratio Statistic: X ^ Wald statistic: X n R 2 W 2 LR ~ ^ 2 ln L , y ln L , y 1

T 1 ^ T ^ r RI R R Lagrange Multiplier (Score) Statistic: X 2 LM 1 n 1 2 where: I ij E ln L , y n i j r T 1 ~ ~ ~ s , y I s , y si , y ln L , y i rank R g Soccer Goals Example Premier League Games in 2004 for k=5 European Countries: England France n1 = 380, Y1 = 975 n2 = 380, Y2 = 826 Germany n3 = 306, Y3 = 890 Italy n4 = 380, Y4 = 960 Spain n5 = 380, Y5 = 980 5 5 yi exp nii i i 1 i 1 L , y ni 5

yij ! i 1 j 1 ni yi yij j 1 Testing Equality of Mean Goals Among Countries - I 1 1 0 0 0 1 0 1 0 0 H 0 : 1 2 3 4 5 R r R 1 0 0 1 0 1 0 0 0 1 5 5 5 ni ln L , y ni i yi ln i ln yij ! i 1 i 1 i 1 j 1 5 ni Under H 0 : ln L , y n y ln ln yij ! i 1 j 1 ^ ln L , y yi y ni i i y i i i ni Under H 0 : ln L , y n y ~ 0 0

r 0 0 y y n ^ ^ ^ ^ 975 826 890 960 980 1 2.57 2 2.17 3 2.91 4 2.53 5 2.58 380 380 306 380 380 ~ 975 826 890 960 980 4631 2.54 380 380 306 380 380 1826 2 ln L , y 2 ln L , y 2 ln L , y yi nii ni 0 E i2 i2 i j i2 i2 i ^ Testing Equality of Mean Goals Among Countries - II n1 n2 s , y n3

n4 n5 y1 1 y 2 2 y3 3 y 4 4 y5 5 380 1826 1 0 I , y 0 0 0 0 0 0 380 1826 2 0 0 0 306 1826 3 0 0 0 380 1826 4 0

0 0 0 0 0 380 18265 0 Likelihood Ratio Test 5 5 ^ 5 ni ^ ^ ln L , y ni i yi ln i ln yij ! i 1 i 1 i 1 j 1 5 5 ni y yi ln y i ln yij ! i 1 i 1 j 1 5 ni 4631 918.71 641.33 950.20 889.69 928.43 ln yij ! i 1 j 1 5 ni 5 ni 4631 4328.36 ln yij ! 302.64 ln yij ! i 1 j 1 i 1 j 1 5 ni ~ ~ ln L , y y ln y ln yij !

i 1 j 1 5 ni 5 ni 4631 4309.82 ln yij ! 321.18 ln yij ! i 1 j 1 i 1 j 1 2 2 X LR 2 321.18 ( 302.64) 37.08 4,.05 9.49 Evidence that the true population means differ (in particular: France lower, Germany higher than the others) Wald Test T 1 ^ ^ ^ Wald statistic: X W2 n R r RI 1 RT R r 2.57 0 0 1 1 0 0 2.57 2.17 0 0.40 2.17 ^ 1 0 1 0 0 0 2.57 2.91 0 0.34 2.91 R r 1 0 0 1 0 0 2.57 2.53 0 0.04 2.53 0 0 1 0 2.57 2.58 0 0.01 1 0 2.58

1826(2.57) 0 0 0 0 380 1826(2.17) 1 1 1 1 0 0 0 0 0 0 1 1 0 380 1 0 0 0 1 0 1 0 0 1826(2.91) ^ 0 1 0 RI 1 R T 0 0 0 0 0 1 0 0 1 0 306 0 0 1 0 1826(2.53) 0 0 1 1 0 0 0 0

0 0 0 0 1 380 1826(2.58) 0 0 0 0 380 1 1 1 1 0057 0 0 0 .0068 .0125 .0068 .0068 .0068 1 0 0 0 .0068 .0068 .0163 .0068 .0068 0 .0095 0 0 0 1 0 1826 0 1826 .0068 0 0 .0067 0 .0068 .0068 .0135 .0068 0 1 0 0 0 0 0 .0068 .0068 .0068 .0068 .0136 .0068

0 0 0 1 132.72 25.34 36.23 35.49 1 1 25.34 89.96 21.80 21.36 1 ^ T RI R X W2 38.33 36.23 21.80 119.25 30.53 1826 35.49 21.36 30.53 117.44 Lagrange Multiplier (Score) Test T Lagrange Multiplier (Score) Statistic: X 2 LM ~ ~ 1 s , y I n 1 ~ s , y ~ 4631 y 2.5361 1826 975 380 380 0 0 0

~ ~ 1826 826 380 0 0 380 ~ 4.42 0 ~ 1826 54.31 ~ 306 ~ 306 890 0 0 s , y 44.93 I , y 0 ~ ~ 1826 1.47 380 380 960 6.41 0 0 0 ~ ~ 1826

980 380 0 0 0 ~ 0 0 0 0 0 4.42 12.19 54.31 0 12.19 0 0 0 1 ~ ~ s , y 44.93 I , y 0 0 15.13 0 0 0 0 12.19 0 1.47 0 6.41 0 0 0 12.19 0 2 X LM 36.83

0 0 0 380 ~ 1826 0 Testing Goodness of Fit to Poisson Distribution All estimation and testing has assumed that number of goals follow Poisson distributions To test whether that assumption is reasonable, we compare the observed distributions of goals with what we would expect under the Poisson model We can check whether the observed mean and variance are similar (under Poisson model they are equal) We can also obtain a chi-square statistic by summing over range of goals: (observed#-expected#)2/expected# which under hypothesis of model fits is approximately chi-square with (# in range)-1 degrees of freedom Distributions of Goals Goals 0 1 2 3 4 5 6 7 8 9 10 Total Games Total Goals Average Observed England France 30 54 79 82 99 110 67 57 61 51 24 15 11 4 6 6 2 1 1 0

0 0 380 380 975 826 2.5658 2.1737 X Germany 18 43 66 77 54 29 13 4 1 1 0 306 890 2.9085 2 obs Italy 36 85 85 78 49 20 20 4 1 1 1 380 960 2.5263 7 i 0 Spain 29 73 96 79 60 28 8 6 1 0 0 380 980 2.5789 Expected (Truncated at 7)

England France Germany 29.2062 43.2279 16.6947 74.9370 93.9639 48.5563 96.1363 102.1239 70.6130 82.2218 73.9950 68.4592 52.7410 40.2105 49.7783 27.0645 17.4810 28.9560 11.5736 6.3330 14.0364 6.1196 2.6648 8.9060 #N/A #N/A #N/A #N/A #N/A #N/A #N/A #N/A #N/A obsi expi expi Italy 30.3822 76.7549 96.9536 81.6451 51.5653 26.0541 10.9701 5.6747 #N/A #N/A #N/A 2 Spain 28.8244 74.3367 95.8553 82.4019 53.1275 27.4026 11.7783 6.2732 #N/A #N/A #N/A Chi-square CritVal P-Value approx Chi-Square Statistic England France 0.0216 2.6843 0.2203 1.5233 0.0853 0.6074 2.8180 3.9034 1.2933

2.8951 0.3470 0.3521 0.0284 0.8595 1.3558 7.0529 6.1697 14.0671 0.5201 ~ 19.8780 14.0671 0.0058 Germany 0.1021 0.6358 0.3014 1.0655 0.3580 0.0001 0.0765 0.9482 Italy 1.0388 0.8857 1.4738 0.1627 0.1276 1.4068 7.4328 0.3095 Spain 0.0011 0.0240 0.0002 0.1404 0.8890 0.0130 1.2120 0.0842 3.4876 14.0671 0.8365 12.8377 14.0671 0.0762 2.3640 14.0671 0.9370 2 7 All leagues, except France, appear to be well described by the Poisson distribution. Especially England, Germany, and Spain