The Big Heist: Internet Braindump sites David Foster, Ph.D ...

The Big Heist: Internet Braindump sites David Foster, Ph.D ...

Overview of New State Data Forensics Analysis March 2011 Data Forensics (DF) Process FDOE DF Goals DF Tools & Methods Spring 2011 DF Program Conservative thresholds Students & Schools Summary Q&A Analyses of test data First building a model of typical

question responses Identify unusual behaviors with potential of unfair advantage Examples of Unusual Behavior Very high agreement among pairs or groups of test takers Very unusual number of erasures, particularly wrong to right Very substantial gains or losses from one occasion to another Many high-stakes testing programs now using Data Forensics Standards for testing, e.g., CCSSOs Operational Best Practices for State Assessment Programs Essential to act on the results

Uphold fairness and validity of test results Identify risks and irregularities Take action based on data and analysis Measure and Manage Communicate zero tolerance for cheating Ensure (and then certify) the test administration is fair and proper Declare scores invalid when fairness and validity are negatively impacted Absolute due diligence when proctoring a test Administering or proctoring a test is not a passive activity!

Similarity: answer-copying, collusion Erasures: tampering Gains: pre-knowledge, coaching Aberrance: tampering, exposure Identical tests: collusion Perfect tests: answer key loss

Our most powerful & credible statistic Measures degree of similarity between 2 or more test instances Analyze each test instance against all other test instances in the test Probable causes of extremely high similarity: Answer copying Test coaching Proxy test taking Collusion Based on estimated answer changing rates from: Wrong-to-Right Anything-to-Wrong

Find answer sheets with unusual WtR answers Extreme statistical outliers could involve tampering, panic cheating, etc. Predict score using prior year information Measure large score increases/decreases against predicted score Which score truly reflects the students actual ability or competence? Extreme gains/losses may result from: Pre-knowledge Coaching Student developmentvisual acuity

Focus on two groups Student-level School-level Utilize VERY conservative thresholds Chance of being hit by lightning = 1 in a million Chance of winning the lottery = 1 in 10 million

Chance of DNA false-positive = 1 in 30 million Chance of tests being flagged and taken independently = 1 in a TRILLION Similarity Analysis only Most credible Chance of tests being so similar, and taken independently = 1 in a trillion Invalidate test scores beyond 1012 Fairness and validity of test instance must be questioned Appeals process to be implemented

Identifies apparent student collusion Definitions Dominant = same answer selected by majority of group members non-Dominant = different answer selected by majority of group members Example of 2 students that passed, but not independently i.e., they didnt do their own work Grade N Flagged Tests 3rd

408,317 144 4th 394,039 103 5th 390,714 92 6th 387,502

224 7th 393,401 245 8th 387,190 69 9th 401,046 622 10th

360,176 57 Totals 3,122,385 1,556 Similarity, gains, and erasures Flagged schools conduct local review Extreme instances may prompt formal investigations and sanctions M4 Similarity Rate Index

Erasures Rate Index Gains Rate Index 286.20 306.49 300.62 310.77 376.53 292.00 280.79 344.23 299.73 303.05 265.98 Incident Rate 0.0 0.0

0.0 0.2 45.0 0.0 0.0 39.6 0.0 0.0 0.0 Overall Index 0.43 0.37 0.60 0.67 0.97 0.52 0.18 0.87 0.32

0.35 0.28 Mean Score Pass Index 338 672 532 664 364 512 338 830 534 458 197 Pass Rate

Subject M R M M M M R M R R M Number of Tests District-School xxxx yyyy zzzzz aaaa bbbb

cccc dddd eeee ffff gggg hhhh 32.7 21.7 19.8 15.5 12.5 12.5 11.7 11.3 10.8 10.4 10.3 0.45 0.32

0.36 0.29 0.15 0.33 0.36 0.13 0.32 0.29 0.46 34.9 24.3 21.3 17.5 0.0 14.8 13.5 0.0 13.1 1.5 12.5

0.6 0.1 0.3 0.1 0.0 0.1 0.3 0.0 0.0 12.7 2.4 9.1 0.1 4.5 3.5 6.1 5.3 1.0 2.2

0.2 0.0 10.3 Focus on most egregious instances Provides results that are Explainable Defensible Can move later to different thresholds Easier to manage Walk before we run 0.07 Proportion of Detected Tests Monitored

behavior improves Invalidations deter cheating 0.06 0.05 0.04 0.03 0.02 0.01 0 Spring 2006 Spring 2007 Spring

2008 Spring 2009 Spring 2010 Goal: Fair and valid testing for all students FDOE will conduct Data Forensics on FCAT/FCAT 2.0/EOC test data Focus on Individual studentsextremely similar tests Schoolssimilarity, gains, and erasures

Recently Viewed Presentations

  • Properties of Water Review - Mr. Dudley's Website

    Properties of Water Review - Mr. Dudley's Website

    This property of water makes it difficult to remove a wet microscope slide from a table. Answer 6. Adhesion. Question 7. The following pictures show a phenomena known as a meniscus. What property/properties of water causes this? Answer 7.
  • Budget Hearings = Appropriation Bills

    Budget Hearings = Appropriation Bills

    Budget Hearings = Appropriation Bills. Appropriation bills for each State Agency and Institution of Higher Education are drafted as approved in Budget Hearings. Arkansas Legislative Council & Joint Budget Committee (ALC/JBC) conducts budget hearings for Regular Session and Joint Budget...
  • Clean Sky contribution Session: Strategy for effective noise

    Clean Sky contribution Session: Strategy for effective noise

    Aviation targets in reduced emissions and noise. Vision 2020 targets are for new aircraft technology relative to 2000 performance. Reduce fuel consumption and CO. 2. emissions by. 50% by 2020. Reduce . NO. X . emissions by. 80 % by...
  • School Year Calendar and minutes reporting

    School Year Calendar and minutes reporting

    SCHOOL-DAY-WAIVER-MINUTES indicates the waiver minutes for the campus and calendar date. For the . 2017-2018 school year, it was determined that the Domain of Values needed to be updated from 0-480 to 0-600 in order to allow schools to report...
  • Sensation and Perception

    Sensation and Perception

    Sensation Sensation= 1) activation of our senses (eyes, ears etc) 2) Physical energy encoded in neural signals 3) handout Perception How is the information organized? Defined= 1) Process of understanding sensation 2) Perception enables us to derive meaning from sensory...
  • Natural Sciences Grade 7 Term 1: Life and

    Natural Sciences Grade 7 Term 1: Life and

    3 examples of classes of arthropods are insects, arachnids and crustaceans. Arthropods: Insects. Natural Sciences - Grade . 7. This group is very diverse - more than a million different species have been identified. Examples: beetles, locusts, mosquitoes, bees and...
  • Science Jeopardy -

    Science Jeopardy -

    Final Jeopardy Question Answer * * * Help (1) Save a duplicate of this template. (2) Enter all answers and questions in the outline view. (view/outline) (3) Change the category headings in the outline view (view/outline) (4) View as a...
  • The National Health Service

    The National Health Service

    GP Appraisal in the UK's National Health Service Michael Harris Bath, UK