Core Methods in Educational Data Mining EDUC691 Spring 2019 We lost a class session
Lets see if we can find a day and time for a substitute session Would adding a class May 8 at the regular class time work? Or we can run through all possible times
Alternative: we drop causal mining and merge correlation mining into network analysis session Diagnostic Metrics -- HW Any questions about any metrics?
Does anyone want to discuss any of the problems? Diagnostic Metrics -- HW When do you want to use fail-soft
interventions? Diagnostic Metrics -- HW When do you not want to use fail-soft interventions?
Textbook/Readings Detector Confidence Any questions about detector confidence?
Detector Confidence What is a detector confidence value? Detector Confidence
What are the pluses and minuses of making sharp distinctions at 50% confidence? Detector Confidence Is it any better to have two cut-offs?
Detector Confidence How would you determine where to place the two cut-offs? Cost-Benefit Analysis
Why dont more people do cost-benefit analysis of automated detectors? Detector Confidence Is there any way around having intervention
cut-offs somewhere? Goodness Metrics Exercise
Detector Academic Suspension Detector No Academic Suspension
Data Suspension 2
3 Data No Suspension
5 140 What is accuracy?
Exercise Detector Academic Suspension
Detector No Academic Suspension Data Suspension
2 3
Data No Suspension 5
140 What is kappa? Accuracy
Why is it bad? Kappa What are its pluses and minuses?
ROC Curve Is this a good model or a bad model? Is this a good model or a bad model?
Is this a good model or a bad model? Is this a good model or a bad model? Is this a good model or a bad model?
ROC Curve What are its pluses and minuses? AUC ROC
What are its pluses and minuses? Any questions about AUC ROC? Precision and Recall
Precision = TP TP + FP
Recall = TP TP + FN
Precision and Recall What do they mean? What do these mean? Precision = The probability that a data point
classified as true is actually true Recall = The probability that a data point that is actually true is classified as true Precision and Recall What are their pluses and minuses?
Correlation vs RMSE What is the difference between correlation and RMSE? What are their relative merits?
What does it mean? 1. 2. 3.
4. High correlation, low RMSE Low correlation, high RMSE High correlation, high RMSE
Low correlation, low RMSE AIC/BIC vs Cross-Validation AIC is asymptotically equivalent to LOOCV BIC is asymptotically equivalent to k-fold cv
Why might you still want to use crossvalidation instead of AIC/BIC? Why might you still want to use AIC/BIC instead of cross-validation? AIC vs BIC
Any comments or questions? LOOCV vs k-fold CV Any comments or questions?
Other questions, comments, concerns about textbook? Thoughts on the Knowles reading? Thoughts on the Jeni reading?
Thoughts on the Kitto reading? Kitto et al.s warnings Warning 1. For some educational scenarios, reporting improvement in
algorithmic performance is insufficient as a form of validation. Warning 2. Being able to report upon a metric does not mean that you should use it, either in the tool, or in reporting its worth. Warning 3. Feedback should not necessarily be set at the same resolution that the analytics make possible.
Warning 4. Overemphasising computational accuracy is likely to delay the adoption of LA tools that could already be used productively. Please explain why Kitto says these; do you agree?
Warning 1. For some educational scenarios, reporting improvement in algorithmic performance is insufficient as a form of validation. Warning 2. Being able to report upon a metric does not mean that you should use it, either in the tool, or in reporting its worth. Warning 3. Feedback should not necessarily be set at the same
resolution that the analytics make possible. Warning 4. Overemphasising computational accuracy is likely to delay the adoption of LA tools that could already be used productively. Kitto et al.s suggestion
Once the analytics is embedded in an appropriate learning design we can see that its purpose is to provide enough scaffolding to start a conversation between the student and the analytics-driven feedback, or between
peers. What are the pluses and minuses of this framing? Other questions or comments?
Creative Assignment 2 Feature Engineering Next Class Wednesday, February 27
Feature Engineering Baker, R.S. (2014) Big Data and Education. Ch. 3, V3, V4, V5 Sao Pedro, M., Baker, R.S.J.d., Gobert, J. (2012) Improving Construct Validity Yields Better Models of Systematic Inquiry, Even with Less Information. Proceedings of the 20th International Conference on
User Modeling, Adaptation and Personalization (UMAP 2012),249260. Creative: Feature Engineering
