Research Methods for the Learning Sciences

Research Methods for the Learning Sciences

Core Methods in Educational Data Mining EDUC691 Spring 2019 We lost a class session

Lets see if we can find a day and time for a substitute session Would adding a class May 8 at the regular class time work? Or we can run through all possible times

Alternative: we drop causal mining and merge correlation mining into network analysis session Diagnostic Metrics -- HW Any questions about any metrics?

Does anyone want to discuss any of the problems? Diagnostic Metrics -- HW When do you want to use fail-soft

interventions? Diagnostic Metrics -- HW When do you not want to use fail-soft interventions?

Textbook/Readings Detector Confidence Any questions about detector confidence?

Detector Confidence What is a detector confidence value? Detector Confidence

What are the pluses and minuses of making sharp distinctions at 50% confidence? Detector Confidence Is it any better to have two cut-offs?

Detector Confidence How would you determine where to place the two cut-offs? Cost-Benefit Analysis

Why dont more people do cost-benefit analysis of automated detectors? Detector Confidence Is there any way around having intervention

cut-offs somewhere? Goodness Metrics Exercise

Detector Academic Suspension Detector No Academic Suspension

Data Suspension 2

3 Data No Suspension

5 140 What is accuracy?

Exercise Detector Academic Suspension

Detector No Academic Suspension Data Suspension

2 3

Data No Suspension 5

140 What is kappa? Accuracy

Why is it bad? Kappa What are its pluses and minuses?

ROC Curve Is this a good model or a bad model? Is this a good model or a bad model?

Is this a good model or a bad model? Is this a good model or a bad model? Is this a good model or a bad model?

ROC Curve What are its pluses and minuses? AUC ROC

What are its pluses and minuses? Any questions about AUC ROC? Precision and Recall

Precision = TP TP + FP

Recall = TP TP + FN

Precision and Recall What do they mean? What do these mean? Precision = The probability that a data point

classified as true is actually true Recall = The probability that a data point that is actually true is classified as true Precision and Recall What are their pluses and minuses?

Correlation vs RMSE What is the difference between correlation and RMSE? What are their relative merits?

What does it mean? 1. 2. 3.

4. High correlation, low RMSE Low correlation, high RMSE High correlation, high RMSE

Low correlation, low RMSE AIC/BIC vs Cross-Validation AIC is asymptotically equivalent to LOOCV BIC is asymptotically equivalent to k-fold cv

Why might you still want to use crossvalidation instead of AIC/BIC? Why might you still want to use AIC/BIC instead of cross-validation? AIC vs BIC

Any comments or questions? LOOCV vs k-fold CV Any comments or questions?

Other questions, comments, concerns about textbook? Thoughts on the Knowles reading? Thoughts on the Jeni reading?

Thoughts on the Kitto reading? Kitto et al.s warnings Warning 1. For some educational scenarios, reporting improvement in

algorithmic performance is insufficient as a form of validation. Warning 2. Being able to report upon a metric does not mean that you should use it, either in the tool, or in reporting its worth. Warning 3. Feedback should not necessarily be set at the same resolution that the analytics make possible.

Warning 4. Overemphasising computational accuracy is likely to delay the adoption of LA tools that could already be used productively. Please explain why Kitto says these; do you agree?

Warning 1. For some educational scenarios, reporting improvement in algorithmic performance is insufficient as a form of validation. Warning 2. Being able to report upon a metric does not mean that you should use it, either in the tool, or in reporting its worth. Warning 3. Feedback should not necessarily be set at the same

resolution that the analytics make possible. Warning 4. Overemphasising computational accuracy is likely to delay the adoption of LA tools that could already be used productively. Kitto et al.s suggestion

Once the analytics is embedded in an appropriate learning design we can see that its purpose is to provide enough scaffolding to start a conversation between the student and the analytics-driven feedback, or between

peers. What are the pluses and minuses of this framing? Other questions or comments?

Creative Assignment 2 Feature Engineering Next Class Wednesday, February 27

Feature Engineering Baker, R.S. (2014) Big Data and Education. Ch. 3, V3, V4, V5 Sao Pedro, M., Baker, R.S.J.d., Gobert, J. (2012) Improving Construct Validity Yields Better Models of Systematic Inquiry, Even with Less Information. Proceedings of the 20th International Conference on

User Modeling, Adaptation and Personalization (UMAP 2012),249260. Creative: Feature Engineering

Recently Viewed Presentations

  • Managing Your Small Woodland: Energize Your Woodland Investment

    Managing Your Small Woodland: Energize Your Woodland Investment

    Managing Your Small Woodland: Energize Your Woodland Investment. Indiana Watershed Leadership Academy. Phil Cox and Larry Owen. May 20, 2015. This template can be used as a starter file for presenting training materials in a group setting.
  • Introduction to Cryptography

    Introduction to Cryptography

    Introduction to Cryptography Matt Mahoney Florida Tech. Definitions Cryptography = the science (art) of encryption Cryptanalysis = the science (art) of breaking encryption Cryptology = cryptography + cryptanalysis Cryptography Goals Encryption - Prevent Eve from intercepting message Authentication - Prevent...
  • Kindergarten Back to School Night - Jenks Public Schools

    Kindergarten Back to School Night - Jenks Public Schools

    Kindergarten Back to School Night Ms. Pitezel Introduction Single mom of 7th grader, Andrew Have taught Kindergarten for 22 years, all here in Jenks Received my Bachelor of Science in Elementary and Early Childhood Education from Northeastern State University in...
  • 14 15 The Constitutional Convention Lesson 27 An

    14 15 The Constitutional Convention Lesson 27 An

    The Assembly Room is set up for all thirteen states, ... Rhode Island did not send delegates becauseit feared that a strong national government would force people to repay the war debts on difficult terms. ... The Dream Team concept...
  • ETHICAL ISSUES IN POMPEII AND HERCULANEUM QUESTIONS OF

    ETHICAL ISSUES IN POMPEII AND HERCULANEUM QUESTIONS OF

    VERMILLION ACCORD ON HUMAN REMAINS. The Vermillion Accord was adopted in 1989 at the South Dakota WAC Inter-Congress. 1. Respect for the mortal remains of the dead shall be accorded to all, irrespective of origin, race, religion, nationality, custom and...
  • Comverge Technologies, Inc.

    Comverge Technologies, Inc.

    Demand Response Programs & Technologies Frank A. Magnotti President, Global Sales & Marketing President, Comverge Enterprises November 17, 2004 The "Peak Demand" Problem "Old-Think" "New-Think" Question! Are Demand Response and Load Management programs technologically, environmentally and economically viable?
  • How to apply the theory in the exam

    How to apply the theory in the exam

    Media Theory Revision This contains all the theory we have gone through over the past year.
  • THE NEW FRONTIER AND THE GREAT SOCIETY CHAPTER

    THE NEW FRONTIER AND THE GREAT SOCIETY CHAPTER

    the new frontier and the great society chapter 27