Factorial design II & III: Factorial ANOVA Week 7.1 and 7.2 Outline We learnt what a factorial design is conceptually. This week, we will learn how to analyze a factorial design Todays plan First 40 mins: Theory/Revision Next 30 mins: Halo effect dataset Remaining 10 mins + home time: Memory dataset WHAT EXACTLY IS HAPPENING IN AN ANOVA? 3 Basic logic Forget statistics. Lets play billiards. Q: After Black is hit, what will happen? Back to ANOVA Participant before manipulation Experimental manipulation

Participant after manipulation (treatment effect) Energy loss through sound (random variation) When you have a treatment, some effects is passed onto the participant, some effects are dissipated. A treatment works when: ____________ > ____________ Misconception alert: This between and within has nothing to do with whether your design is between- or within-subjects Whats the logic? This is a large effect: This is a small effect: Whats the logic? This is no effect (notice the overlap)

Aka. error, residuals, etc. # correct reconstructions ANOVA summary table & writing results Source df Expertise Sum of squares 23.56 F p 1 Mean Square 23.56

4.56 .04 Position type 11.33 1 11.33 2.00 .17 Expertise Position type 29.45 1 29.45 5.87

.03 Residual (Error) 41.33 36 5.17 Total 94.67 39 59.51 Real 20 10 0 Fake 20

8 Expert 8 8 Novice Expertise Main effect of expertise Main effect of position Interaction DV: memory Write-up A 2 (Expertise: Experts vs. Novices) 2 (Position type: Real vs. Fake) fully between-subjects ANOVA revealed a main effect of Expertise, F(1, 36) = 4.56, p = .04; no main effect of Position type, F(1, 36) = 2.00, p = .17. Notably, these effects were qualified by a significant interaction between Expertise and Position type, F(1, 36) = 5.87, p = .03 8 Why is it important to mention the df (degrees of freedom)? Source

df Expertise Sum of squares 23.56 F p 1 Mean Square 23.56 4.56 .04 Position type 11.33

1 11.33 2.00 .17 Expertise Position type 29.45 1 29.45 5.87 .03 Residual (Error) 41.33 36

5.17 Total 94.67 39 59.51 DV: memory Side note: Sums of squares = sums of the squared values of the difference between predicted and actual It is functionally equivalent to variance Consider this portion: revealed a main effect of Expertise, F(1, 36) = 4.56, p = .04 Numerator df indicates the number of groups that you are comparing (i.e., df = k - 1)

Denominator df indicates approx. number of observations (subjects) you have Reporting these dfs tells reader quickly the number of groups you have, and approx. the number of observations, for the particular analysis. What the factorial ANOVA does It partitions the variance into components 29 41 11 23 Residual Expertise The pie chart area is derived from SS(component)/ SStotal Each component has its F, df, and p-values Position Interaction 10

Good reporting practices A 2 (Expertise: Experts vs. Novices) 2 (Position type: Real vs. Fake) fully between-subjects ANOVA revealed a main effect of Expertise, F(1, 36) = 4.56, p = .04; no main effect of Position type, F(1, 36) = 2.00, p = .17. Notably, these effects were qualified by a significant interaction between Expertise and Position type, F(1, 36) = 5.87, p = .03. 1. Always mention the specific analyses done What type of ANOVA? Which is the between or within-subject factor, if applicable? 2. Phrase it in a meaningful way. Sometimes people report the interaction first, then main effects; sometimes its the other way round. Think of how you want readers to process the information. 11 What exactly is a significant interaction? 25 20 15 10 5 0 Fake

20 8 Expert Expertise 8 8 Novice Real 25 20 15 10 5 0 Fake 20 20 8 Expert Expertise 8

Novice # correct reconstructions Real # correct reconstructions # correct reconstructions So far, all we know is the interaction is significant. A significant interaction = at least one mean is different from the rest (i.e., non-parallel lines) This is meaningless by itself. Because a significant interaction could be any of the following (and many more not drawn!): Real 25 20 15 10 5 0 Fake 20 20

20 8 Expert Novice Expertise Reporting that an interaction is significant is useless, because where does the difference in means lie? You need to breakdown the interaction (we will practice this) 12 # correct reconstructions What if there are no significant interactions? Real 25 20 15 10 5 0 20

Fake 20 8 Expert 8 Novice Expertise Simply report main effects is enough. If there are main effects, describe what the main effects are. Steps to doing factorial ANOVA 1. Put all IVs and DVs into the correct boxes (SPSS/JASP) 2. Request a plot of results 3. Output: See if there are significant interactions and main effects 4. If there is a significant interaction, run simple effects analyses 5. If there is no significant interaction, no need to run simple effects analyses (you may still run it if you are

curious) For now, learn the steps. When you have mastered them, you may deviate a little. Steps to simple effects analyses Copy your plot to paper If 2 x 2 design, report all simple effects Anything beyond, select the simple effects that Are meaningful Answer the research question Whenever you find your simple effects, go back to your plot, circle the comparisons, and write down the t-, F-, p-values. For now, learn the steps. When you have mastered them, you may deviate a little. Different designs will require data to be entered in different ways *Note to R users: R requires a long format for within-subjects analyses. DATAFILE STRUCTURE* 16 #1 rule for data entry in SPSS 1 row = 1 subject This is known as the wide format way of data entry.

17 Hence, this is WRONG! Subj DV 1 2 3 4 5 6 7 N (score) (score) (score) (score) (score) (score) (score) (score) (score) No matter what software you are using.

18 Fully between-subjects Subject Condition Position Type Memory 1 Expert Real (score) 2 Expert Real (score)

3 Expert Fake (score) 4 Expert Fake (score) 5 Novice Real (score) 6

Novice Real (score) 7 Novice Fake (score) 8 Novice Fake (score) 19 Fully within-subjects Subject

Expert.Real Expert.Fake Novice.Real Novice.Fake 1 (score) (score) (score) (score) 2 (score) (score) (score) (score)

3 (score) (score) (score) (score) 4 (score) (score) (score) (score) 5 (score) (score)

(score) (score) 6 (score) (score) (score) (score) 7 (score) (score) (score) (score) 8 (score)

(score) (score) (score) 20 Mixed-subjects Betweensubjects factor Within-subjects factor Subject Condition Position.Real Position.Fake 1 Expert (score)

(score) 2 Expert (score) (score) 3 Expert (score) (score) 4 Expert (score) (score)

5 Novice (score) (score) 6 Novice (score) (score) 7 Novice (score) (score) 8 Novice

(score) (score) 21 Summary: Data entry You must be able to identify whether a design is between-subjects, within-subjects, or mixed simply from looking at the dataset. Why? In professional settings, your collaborators are going to send you data with so many variables that it will be impossible for them to tell you which variables are between or within. 22 Mixed designs The SPSS output is similar, but useful information is located at different parts of the output But conceptually, there is little difference in interpreting mixed designs from between- or within-subjects designs The main difference is mathematical: how variance is partitioned (dont need to know) 23

Simple effects in mixed ANOVA In fully between- or within-subjects ANOVA, simple effects are straightforward (it is a matter of being observant how many levels each factor has) In mixed ANOVA, you need to be absolutely clear which factor is between or within, and how many levels the factor has, because these determines which type of simple effects tests you run Question: How do I interpret the ANOVA summary table for different designs? Various headers of each ANOVA tables will tell you what the between or within-subjects effects are. For example, in a mixed-subjects design, you will get a table of Between-subjects effects, and another table of Withinsubjects effects. The way you then interpret the ANOVA table will be the same. We will practice this in Fridays class. 25 OTHER QUESTIONS YOU MAY HAVE 26 FAQ 1. Can I get a significant interaction without any significant

simple effects? Yes. 2. Can I get significant main effects without a significant interaction? Yes. 3. Why dont we just perform four t-tests for a 22 design? You can, but that will inflate your Type I error. 27 # c o r r e c t r e c o n s t r u c ti o n s FAQ 5. What if I have a 2 3 design? Analysis remains similar. However, graph the one with more levels on a y-axis it usually makes it easier to understand Which is easier to understand? Real 25 20 15 10 5 0 Real

Fake 25 20 15 10 5 0 Expert Advanced Intermediate Expertise Novice Fake Expert Intermediate Advanced Novice 28 FAQ 5. For a 2 4 design, do I need to compare 8 simple effects? What does your theory predict?

6. How many factors should I have in my study? What does your theory predict? 7. What if I want a 2 2 2 2 2 design? What are you even thinking?!?! Anything more than a three factor design is not advisable. 29 Want to read more about the math behind ANOVA? Roberts & Russo (1999). A students guide to analysis of variance. London, UK: Routledge. Do you need to read it? No. 30 Last words on factorial ANOVA Once you master factorial ANOVA, you appreciating the complexities of the real world, and understanding why simple explanations are sometimes inadequate at best, misleading at worse. Understanding statistics changes your worldview; you start seeing the world in conditionals