Research Methods for the Learning Sciences

Research Methods for the Learning Sciences

Special Topics in Educational Data Mining HUDK5199 Spring, 2013 April 3, 2013 Todays Class Factor Analysis Goal 1 of Factor Analysis You have a lot of quantitative* variables, e.g. high dimensionality You want to reduce the dimensionality into a smaller number of factors Goal 1 of Factor Analysis You have a lot of quantitative* variables, e.g. high dimensionality You want to reduce the dimensionality into a smaller number of factors * -- there is also a variant for categorical and binary data, Latent Class Factor Analysis (LCFA -Magidson & Vermunt, 2001; Vermunt & Magidson, 2004), as well as a variant for mixed data types, Exponential Family Principal Component Analysis (EPCA Collins et al., 2001) Goal 2 of Factor Analysis You have a lot of quantitative* variables, e.g. high dimensionality

You want to understand the structure that unifies these variables Classic Example You have a questionnaire with 100 items Do the 100 items group into a smaller number of factors E.g. Do the 100 items actually tap only 6 deeper constructs? Can the 100 items be divided into 6 scales? Which items fit poorly in their scales? Common in attempting to design questionnaire with scales and sub-scales Another Example You have a set of 600 features of student behavior You want to reduce the data space before running a classification algorithm Do the 600 features group into a smaller number of factors? E.g. Do the 600 features actually tap only 15 deeper constructs? Example from my work (Baker et al., 2009) We developed a taxonomy of 79 design features that a Cognitive Tutor lesson could possess We wanted to reduce the data space before running statistical significance tests

Do the 79 design features group into a smaller number of factors? E.g. Do the 79 features actually group into a set major dimensions of tutor design? The answer was yes they group into 6 factors Factors were then used In relationship mining analyses To study which features of the design of intelligent tutors are associated with Gaming the system (Baker et al., 2009) Off-task behavior (Baker, 2009) Affective states (Doddannarra et al., accepted) Two types of Factor Analysis Experimental Determine variable groupings in bottom-up fashion More common in EDM/DM Confirmatory Take existing structure, verify its goodness More common in Psychometrics Mathematical Assumption in most Factor Analysis Each variable loads onto every factor, but with different strengths

And some strengths are infinitesimally small Example F1 F2 F3 V1 0.01 -0.7 -0.03 V2 -0.62 0.1 -0.05 V3 0.003

-0.14 0.82 V4 0.04 0.03 -0.02 V5 0.05 0.73 -0.11 V6 -0.66 0.02 0.07

V7 0.04 -0.03 0.59 V8 0.02 -0.01 -0.56 V9 0.32 -0.34 0.02 V10 0.01

-0.02 -0.07 V11 -0.03 -0.02 0.64 V12 0.55 -0.32 0.02 Computing a Factor Score Can we write an equation for F1? F1 F2 F3

V1 0.01 -0.7 -0.03 V2 -0.62 0.1 -0.05 V3 0.003 -0.14 0.82 V4 0.04

0.03 -0.02 V5 0.05 0.73 -0.11 V6 -0.66 0.02 0.07 V7 0.04 -0.03 0.59

V8 0.02 -0.01 -0.56 V9 0.32 -0.34 0.02 V10 0.01 -0.02 -0.07 V11 -0.03

-0.02 0.64 V12 0.55 -0.32 0.02 Can we write an equation for F1? (Its just a straight-up linear equation, like in linear regression! Cazart!) F1 F2 F3 V1 0.01 -0.7 -0.03

V2 -0.62 0.1 -0.05 V3 0.003 -0.14 0.82 V4 0.04 0.03 -0.02 V5 0.05

0.73 -0.11 V6 -0.66 0.02 0.07 V7 0.04 -0.03 0.59 V8 0.02 -0.01 -0.56

V9 0.32 -0.34 0.02 V10 0.01 -0.02 -0.07 V11 -0.03 -0.02 0.64 V12 0.55

-0.32 0.02 Which variables load strongly on F1? F1 F2 F3 V1 0.01 -0.7 -0.03 V2 -0.62 0.1 -0.05

V3 0.003 -0.14 0.82 V4 0.04 0.03 -0.02 V5 0.05 0.73 -0.11 V6 -0.66

0.02 0.07 V7 0.04 -0.03 0.59 V8 0.02 -0.01 -0.56 V9 0.32 -0.34 0.02

V10 0.01 -0.02 -0.07 V11 -0.03 -0.02 0.64 V12 0.55 -0.32 0.02 Wait whats a strong loading? One common guideline: > 0.4 or < -0.4 Comrey & Lee (1992) 0.70 excellent (or -0.70)

0.63 very good 0.55 good 0.45 fair 0.32 poor One of those arbitrary things that people seem to take exceedingly seriously Another approach is to look for a gap in the loadings in your actual data Which variables load strongly on F2? F1 F2 F3 V1 0.01 -0.7 -0.03 V2 -0.62

0.1 -0.05 V3 0.003 -0.14 0.82 V4 0.04 0.03 -0.02 V5 0.05 0.73 -0.11

V6 -0.66 0.02 0.07 V7 0.04 -0.03 0.59 V8 0.02 -0.01 -0.56 V9 0.32

-0.34 0.02 V10 0.01 -0.02 -0.07 V11 -0.03 -0.02 0.64 V12 0.55 -0.32 0.02

Which variables load strongly on F3? F1 F2 F3 V1 0.01 -0.7 -0.03 V2 -0.62 0.1 -0.05 V3 0.003

-0.14 0.82 V4 0.04 0.03 -0.02 V5 0.05 0.73 -0.11 V6 -0.66 0.02 0.07

V7 0.04 -0.03 0.59 V8 0.02 -0.01 -0.56 V9 0.32 -0.34 0.02 V10 0.01

-0.02 -0.07 V11 -0.03 -0.02 0.64 V12 0.55 -0.32 0.02 Which variables dont fit this scheme? F1 F2 F3 V1

0.01 -0.7 -0.03 V2 -0.62 0.1 -0.05 V3 0.003 -0.14 0.82 V4 0.04 0.03

-0.02 V5 0.05 0.73 -0.11 V6 -0.66 0.02 0.07 V7 0.04 -0.03 0.59 V8

0.02 -0.01 -0.56 V9 0.32 -0.34 0.02 V10 0.01 -0.02 -0.07 V11 -0.03 -0.02

0.64 V12 0.55 -0.32 0.02 Assigning items to factors to create scales After loading is created, you can create onefactor-per-variable models (scales) by iteratively assigning each item to one factor dropping the one item that loads most poorly in its factor, if it has no strong loading re-fitting factors Lets try that algorithm F1 F2 F3 V1

0.01 -0.7 -0.03 V2 -0.62 0.1 -0.05 V3 0.003 -0.14 0.82 V4 0.04 0.03

-0.02 V5 0.05 0.73 -0.11 V6 -0.66 0.02 0.07 V7 0.04 -0.03 0.59 V8

0.02 -0.01 -0.56 V9 0.32 -0.34 0.02 V10 0.01 -0.02 -0.07 V11 -0.03 -0.02

0.64 V12 0.55 -0.32 0.02 Item Selection Some researchers recommend conducting item selection based on face validity e.g. if it doesnt look like it should fit, dont include it What do you think about this? How does it work mathematically? Two algorithms (Ferguson, 1971) Principal axis factoring (PAF) Fits to shared variance between variables Principal components analysis (PCA) Fits to all variance between variables, including variance unique to specific variables PCA is more common these days

Very similar, especially as number of variables increases How does it work mathematically? First factor tries to find a combination of variable-weightings that gets the best fit to the data Second factor tries to find a combination of variable-weightings that best fits the remaining unexplained variance Third factor tries to find a combination of variable-weightings that best fits the remaining unexplained variance How does it work mathematically? Factors are then made orthogonal (e.g. uncorrelated to each other) Uses statistical process called factor rotation, which takes a set of factors and re-fits to maintain equal fit while minimizing factor correlation Essentially, there is a large equivalence class of possible solutions; factor rotation tries to find the solution that minimizes between-factor correlation Looking at this another way This approach tries to find lines, planes, and hyperplanes in the K-dimensional space (K variables)

Which best fit the data This may remind you of support vector machines Goodness What proportion of the variance in the original variables is explained by the factoring? (e.g. r2 called in Factor Analysis land the estimate of the communality) Better to use cross-validated r2 Still not standard How many factors? Best approach: decide using cross-validated r2 Alternate approach: drop any factor with fewer than 3 strong loadings Alternate approach: add factors until you get an incomprehensible factor But one persons incomprehensible factor is another persons research finding! Relatively robust to violations of assumptions Non-linearity of relationships between variables Leads to weaker associations Outliers

Leads to weaker associations Low correlations between variables Leads to weaker associations Desired Amount of Data At least 5 data points per variable (Gorsuch, 1983) At least 3-6 data points per variable (Cattell, 1978) At least 100 total data points (Gorsuch, 1983) Comrey and Lee (1992) guidelines for total sample size 100= poor 200 = fair 300 = good 500 = very good 1,000 or more = excellent Desired Amount of Data At least 5 data points per variable (Gorsuch, 1983) At least 3-6 data points per variable (Cattell, 1978) At least 100 total data points (Gorsuch, 1983) Comrey and Lee (1992) guidelines for total sample size 100= poor 200 = fair 300 = good 500 = very good 1,000 or more = excellent My opinion: use cross-validation and see empirically

OK youve done a factor analysis, and youve got scales One more thing to do before you publish OK youve done a factor analysis, and youve got scales One more thing to do before you publish Check internal reliability of scales Cronbachs a Cronbachs a N = number of items C = average inter-item covariance (averaged at subject level) V = average variance (averaged at subject level) Cronbachs a: magic numbers (George & Mallory, 2003) > 0.9 Excellent

0.8-0.9 Good 0.7-0.8 Acceptable 0.6-0.7 Questionable 0.5-0.6 Poor < 0.5 Unacceptable Related Topic Clustering Not the same as factor analysis Factor analysis finds how data features/variables/items group together Clustering finds how data points/students group together In many cases, one problem can be transformed into the other But conceptually still not the same thing Next class! Curious Question Factor Analysis is not very frequently used in EDM Why not? Asgn. 7 Questions? Comments? Next Class

Monday, March 15 NO CLASS NEXT WEEK! Clustering Readings Witten, I.H., Frank, E. (2011) Data Mining: Practical Machine Learning Tools and Techniques. Ch. 4.8, 6.6 Amershi, S. Conati, C. (2009) Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments. Journal of Educational Data Mining, 1 (1), 18-71. Assignments Due: 7. Clustering The End

Recently Viewed Presentations

  • NHS Finance Language

    NHS Finance Language

    Cuckoo Lane Practice The CQC completed a comprehensive inspection of the Cuckoo Lane practice on the 28/01/2015, and rated the practice as outstanding It was commended for being well-led and providing responsive and effective services
  • 2013 Tax Changes - NCACC

    2013 Tax Changes - NCACC

    The FUTA tax credit was reduced by 0.3% for 2012; and will be reduced by an additional 0.3% for 2013. Those reductions mean the effective FUTA tax rate for NC employers for 2013 will be 1.2%, or $84 per employee....
  • Final Verdict of Supreme Court of Vodafone Case.

    Final Verdict of Supreme Court of Vodafone Case.

    Vodafone Group Plc makes a final binding offer of US $ 11.076 billion, based on an enterprise value of US $ 18.800 billion of HEL. Hutch board accepts Vodafone Offer. ... Thus, Supreme Court has laid down several important and...
  • Plants: Part 1 - PC&#92;|MAC

    Plants: Part 1 - PC\|MAC

    Plants. Plants are thought to have evolved from ancient photosynthetic green algae that formed a plant group called charophyceans. These plants had to overcome some challenges to start growing on land; 1. Prevention of dehydration. Cuticle: a waxy, thick, waterproof...
  • What is it - Nuffield Foundation

    What is it - Nuffield Foundation

    Matching graphs and scenarios Measurements from real-life situations give different shapes of graph. Using the cards, match each scenario with its graph. Measuring pulse rate Conversion graph (miles against kilometres) 0 160 100 kilometres miles 0 2 100 Graph showing...
  • Discrete Mathematics Math 6A

    Discrete Mathematics Math 6A

    Discrete Mathematics Math 6A ... (b+1), and a+2 2(b+1). 6.1-11 a) Since the number of bacteria triples every hour, the recurrence relation should say that the number bacteria after n hours is 3 times the number of bacteria after n-1...
  • Introduction to Psychology

    Introduction to Psychology

    William Sheldon Somatotyping (body typing) 1. Endomorph— plump, relaxed, jolly (Santa Claus) 2. Ectomorph— high strung and solitary (Sherlock Holmes) 3. Mesomorph— bold and physically active (Superman) Evaluation of Trait Perspective Doesn't really explain personality, simply describe ...
  • From Adam & Eve, to Cain, to Josephs

    From Adam & Eve, to Cain, to Josephs

    From Adam & Eve, to Cain, to Joseph's brothers…The Bible's first stories are filled with people who sin and then attempt . to . cover up their sin by . lying. about it