Building Bridges: Teaching Resources for Research Scientists

Building Bridges: Teaching Resources for Research Scientists

Assessment: The Hidden Variable of Achievement A Field-tested Learning Assessment Guide (FLAG) for STEM Instructors Michael Zeilik University of New Mexico (www.flaguide.org) Pop Quiz! In terms of learning gains, which instructor attribute enhances achievement the most? A. Teaching experience B. Clarity of presentations C. Energy and enthusiasm D. Deep knowledge of subject E. None of the above What Works?

20th Century Gain Results Active Learning, Mastery Learning (SD 0.5, 0.5 PSI, discussion, debates, games, role playing, controversy) Cooperative Learning (SD 0.5, 0.5 cognitive and affective; century of research, all disciplines) One-on-One Tutoring (SD = 2, with trained tutors) What Does Not Work? (Small gains SD < 0.3) EVERYTHING ELSE! Lectures (standard model), reinforce memorization! Unstructured discussion, supervised independent study, autonomous small groups, self study Audio-tutorials, programmed instruction, computer-based instruction, instructional television, Web-based instruction

What Works in Context: Physics & Astronomy Disciplinary education research, uses tools of the trade to conduct experiments Create a mostly empirical robust knowledge base about learning in physics & astronomy Probe initial state (prior knowledge), final state (learning outcomes), and student thinking Measurement: Assessment! Galileo Galilei: Measure what is measurable, and make measurable what is not so. But: Measure what you value, and value what you measure. (M. Zeilik) FLAG Features

Assessment Basics What is this assessment business all about? Making Goals What do you want to measure? Classroom Assessment Techniques (CATs) How do you measure it? Searchable DatabaseToolbox What are good tools for measurement? All peer reviewed and evidence based FLAG CATs Attitudinal Surveys: E. Seymour, E. Lewis Concept Tests: A. Ellis Concept Maps: M. Zeilik

Conceptual Diagnostic Tests: M. Zeilik Interviews: M. Smith, S. A. Southerland Performance Assessments: T. Slater Portfolios: T. Slater Scoring Rubrics: D. Ebert-May Student Assessment of Learning Gains: E. Seymour Weekly Reports: E. Etkina Mathematical Thinking: M. Swan, J. Rideway Multiple-Choice Tests: J. Parkes Minute Papers: M. Zeilik

Minute Paper Take a few minutes at the end of class and ask for a written response to: What was the most important concept you learned in class? What important question remains unanswered? What was the muddiest point of this class? Few Minute Paper: Teams reach consensus, submit written report Analysis: Sort into themes (cards) Weekly Report: Extended minute paper Concept Tests Quick feedback on conceptual (not factual) understanding Instructor gives conceptual question with choices (common misconceptions) After a minute, whole class responds (hands, flash cards, class polling system) Instructor assesses responses: If most incorrect, pair up do discuss

(peer teaching) Class response again to gauge mastery; instructor adapts in real time Earth asteroid The magnitude of the force exerted by the asteroid on the Earth is a) larger than the magnitude of the force exerted by the Earth on the asteroid b) the same as the magnitude of the force exerted by the Earth on the asteroid c) smaller than the magnitude of the force exerted by the Earth on the asteroid d) zero. (the asteroid exerts no force on the Earth) Results: Midwestern Universities (Dostal) (P111, conceptual; P221 calc; P112 algebra) Larger than: Same as:

Smaller than: Zero: N P111 P221 P112 2% 2% 5% 13% 9% 24% 62% 84% 67% 23% 101

5% 91 5% 101 Student Assessments Attitude Surveys: Perceptions about course, discipline; seem easy-not!, best based on a robust, field-tested model Minute Paper: Given limited resources, what one change would you make to improve this course?; sort by themes Student Assessment of Learning Gains (SALG): Probes learning gains that students perceive; avoids performance critiques; easily customized (15 min); available on-line UNM P102 SALG Spring 2003 (Mean = 3.38)

Error Bars are Standard Deviations 3.99 Videos 3.75 CT/CRS 4.15 Coop quizzes 3.71 Activities 3.64 Team work 3.52 Discussions

3.68 Instructor talk 1.00 2.00 3.00 4.00 Rating (scale 1 - 5) 5.00 P102 UNM Spring 2003 Gender MattersSometimes! Female Male Posttest

Confidence All p < 0.002 Pretest confidence p < 0.0001 Last math p < 0.002 Science SelfAptitude NS! Math SelfAptitude NS!

1.00 2.00 3.00 4.00 Score (1 = very poor, 5 = very good) 5.00 Cooperative Quiz Gains by Gender P102 UNM Spring 2003 Quizzes Female Effect size (F) Male Effect size (M)

1.00 0.80 0.60 0.40 0.20 0.00 Practice Quiz #1 Quiz #2 Quiz #3 Mean Does it stick? (Coop quizzes/Test) A101 UNM Fall 2000 Identical Items (p = 0.96) 120 100

80 60 40 20 Percent correct 0 Q1-12 Q2-4 Q2-9 Q2-8 Q2-7 Q3-6 Q3-7 Item Pre % Post % Test#1 Attitude Results: Intro Astro & Physics Measure (50%= neutral) UNMA101 (n = 1 31) CMUA111 ( n =224)

UNMA271 (n = 8 2) UNMP102 (n = 31) Pretest (%) Posttest Gain and (%) Effect Size 62 10 63 57 55 78

82 60 60 Not significant Not significant Not significant Not significant Conceptual Diagnostic Tests Ideally research-based on misconceptions, revealed by student think aloud interviews Measures pre/post conceptual gains as a summative assessment Force Concept Inventory (FCI; 1985);

Astronomy Diagnostic Test (ADT; 1999) version 2; national baselines; large data sample (about 5,000) Follow protocol! ADT 2 UNM Fall 2000 vs. National F) = 0.84 => 80% of postscores above mean of pres M) = 0.53 => 70% of postscores above mean of pres 30 25 30 25 20 26 28 22 20

Females 15 Males 10 All 5 0 Gain (National) (Pre = 5346; Post = 3842) Gain (UNM) Pre/post: ADT National Project/UNM = (post% - pre%)/(100% - pre%) 60 50

Standard errors plotted 38 40 30 32.4 Score (%) 20 10 53.7 47.3 56 41.5 34 27

U N M U N M 0 Pre-course (N=5346) Overall: = 0.22 Men: = 0.25 Post-course (N=3842) Women: = 0.20 UNM: = 0.31 P102 UNM Spring 2003 Pre/Post Female

Male 70% 60% 50% 40% 30% 20% Percentage 10% 0% FCI Pre% FCI Post% FCI GRC FCI+GRC Pre(FCI Post(FCI + GRC) + GRC)

Classroom Assessment: Good News! Well-done formative assessment results in a pre/post gain of about 0.5 standard deviation (70% rather than 50% on a standardized test)

Recently Viewed Presentations

  • Issue Y2K The Great War for Talent!

    Issue Y2K The Great War for Talent!

    Tom Peters' Re-Imagine! Business Excellence in a Disruptive Age REI.500/19May2005
  • Fertility in organic systems - Sustainable Agriculture

    Fertility in organic systems - Sustainable Agriculture

    Transplant production. Media is key. Usually a peat-based mix with vermiculite, perlite and nutrients (compost, worm castings, bone meal, etc) All ingredients must be allowable by OMRI
  • Bellwork - Ms. Kim&#x27;s 8th Grade Science - Home Page

    Bellwork - Ms. Kim's 8th Grade Science - Home Page

    Bellwork. Please walk in SILENTLY. Stop by the PICK UP table. Get started on your bell work. Describe Aristotle's method of classification. Describe CarolousLinneaus' method of classification.
  • Poetry Stuff - Mrs. Lundgren&#x27;s Website

    Poetry Stuff - Mrs. Lundgren's Website

    Metaphor. The term metaphor has two meanings, a broad, more general meaning and a concise, specific meaning. All figures of speech which use association, comparison, or resemblance can generally be called types of metaphor, or metaphorical. One specific figure of...
  • Oceanic and International Operations

    Oceanic and International Operations

    Procedures for Air Navigation Services Air Traffic Management (ICAO 4444 PANS ATM), Chapter 16.5.1 (as amended) North Atlantic Operations and Airspace Manual (NAT Doc 007), Chapter 8.5.2 - 8.5.5 (as amended) SS: links are not working. SS: pdf: PANS ATM...
  • Performance Assessment Assurances, Day 2, Elementary Science ...

    Performance Assessment Assurances, Day 2, Elementary Science ...

    Module 4: Introduction to Common Rubrics. ... new in that English language arts has always have a state writing rubric that is used on the state tests, and that has been adopted as part of assessment practice in ELA classes....
  • Signing transactions anonymously with Identity Mixer in Hyperledger

    Signing transactions anonymously with Identity Mixer in Hyperledger

    Signing transactions anonymously with Identity Mixer in Hyperledger . Idemix team: Jan Camenisch, Manu Drijvers, Maria Dubovitskaya. Blockchain team: Elli Androulaki, Angelo De Caro, Andreas Kind, Alessandro Sorniotti
  • From God to Us:

    From God to Us:

    Does it come with the power of God to change lives?—Pastoral concern Circulation and Collection problems None of the NT writers had a New Testament—circulation Rev. 1-3 Circulation problems: Ephesus had it, Jerusalem didn't, sub-collections forming Collection processes taking time....