Review of AERA/APA/NCME Test Standards Revision and Students ...

Review of AERA/APA/NCME Test Standards Revision and Students ...

Update on the Revisions to the Standards for Educational and Psychological Testing Antonio E. Puente, Ph.D. University of North Carolina Wilmington Joint Committee for the Revision to the Standards for Educational and Psychological Testing www.apa.org/science/programs/testing/standards.aspx Mission & Brief History The purpose for publishing the standards is to provide criteria for the evaluation tests, testing practices, and the effects of tests use. (1999) Established in 1954; revised in 1955, 1966, 1999 and currently (2009-2011) American Psychological Association, American Educational Research Association, National

Council on Measurement in Education 02/08/2020 Update on Revisions to the Test Standards 2 Update on the Revisions to the Standards for Educational and Psychological Testing: Overview of the Revision Process Scope of the Revision Based on comments each organization received from invitation to comment Summarized by the Management Committee in consultation with the CoChairs

Wayne Camara, Chair, APA Suzanne Lane, AERA David Frisbie, NCME 02/08/2020 Update on Revisions to the Test Standards 4 Joint Committee Members Lauress Wise, Co Chair, HumRRO Barbara Plake, Co Chair, University of Nebraska Linda Cook, ETS Fritz Drasgow, University of Illinois Brian Gong, NCIEA Laura Hamilton, Rand Corporation Jo-Ida Hansen, University on MN Joan Herman, UCLA

02/08/2020 Update on Revisions to the Test Standards 5 Joint Committee Members Michael Kane, ETS Michael Kolen, University of Iowa Antonio E. Puente, UNC-Wilmington Paul Sackett, University of MN Nancy Tippins, Valtera Corporation Walter (Denny) Way, Pearson Frank Worrell, Univ of CA- Berkeley 02/08/2020 Update on Revisions to the Test Standards

6 Five Identified Areas for the Revisions Access/Fairness Accountability Technology Workplace Format issues 02/08/2020 Update on Revisions to the Test Standards 7 Timeline First meeting January, 2009

Three year process for completing text of revision Release of draft revision following December 2010 JC meeting Open comment/Organization reviews Projected publication Summer, 2012 02/08/2020 Update on Revisions to the Test Standards 8 Anticipated Review Process Draft to be released December, 2010

90 Day (+/-) Review Period Public Portal for Submitting Comments Sponsoring Organizations with Publicize Opportunity for Comments Committee will Summarize Key Issues from Reviews and Provide Response(s) 02/08/2020 Update on Revisions to the Test Standards 9 Publication Options Management Committee responsibility Goal is for electronic access Pursuing options for Kindle, etc.

Concerns about retaining integrity and financial support for future revision efforts 02/08/2020 May 1, 2010 Update on Revisions to the Test Standards 10 Update on the Revisions to the Standards for Educational and Psychological Testing: Fairness for Diverse Populations 1999 Approach Standards related to fairness appear

throughout many chapters Concentrated attention in: Chapter 7: Fairness in Testing and Test Use Chapter 8: Rights and Responsibilities of Test Takers Chapter 9: Testing Individuals of Diverse Linguistic Backgrounds Chapter 10: Testing Individuals with Disabilities 02/08/2020 Update on Revisions to the Test Standards 12 Committee Charge

Five elements of the charge focused on accommodations/modifications Impact/differentiation of accommodation and modification Appropriate selection/use for English learners (ELs) and examinees with disabilities (EwD) Attention to other groups, e.g., pre-K, older populations Flagging

Comparability/validity of accommodated scores One element focused on adequacy and comparability of translations One element focused on Universal Design 02/08/2020 Update on Revisions to the Test Standards 13 Revision Response

Fairness is fundamental to test validity: include as foundation chapter Fairness and access are inseparable Same principles of fairness and access apply to all individuals and regardless of specific subgroup From three chapters to a single chapter that describes core principles and standards 02/08/2020 Examples drawn from ELs, EwD, and other groups (young children, aging adults adults, etc) Comments point to applications for specific groups More particular standards retained where appropriate (e.g., test translations) Update on Revisions to the Test Standards

14 Overview to Fairness Chapter Section I: General Views of Fairness Section II: Threats to the Fair and Valid Interpretations of Test Scores Section III: Minimizing Construct Irrelevant Components Through the Use of Test Design and Testing Adaptations Section IV: The Standards 02/08/2020 Update on Revisions to the Test Standards 15 Four Clusters of Standards 1.

2. 3. 4. Use test design, development administration and scoring procedures that minimize barriers to valid test interpretations for all individuals. Conduct studies to examine the validity of test score inferences for the intended examinee population. Provide appropriate accommodations to remove barriers to the accessibility of the construct measured by the assessment and to the valid interpretation of the assessment scores. Guard against inappropriate interpretations, use, and/or unintended consequences of test results for individuals or subgroups. 02/08/2020

Update on Revisions to the Test Standards 16 Update on the Revisions to the Standards for Educational and Psychological Testing: Other Issues Other Issues in Committee Charge Increased use of tests for accountability Impact of technology on testing Format and style issues 02/08/2020 Update on Revisions to the Test Standards 18

Increased Use of Testing for Accountability Use of tests for accountability has expanded Most notably in education but also in other areas such as behavioral health Facilitated by increasing availability of data and analysis tools Recent and impending federal and state initiatives will likely lead to further expansion Use of test scores in policy and program evaluations continues to be widespread

02/08/2020 May 1, 2010 Reinforced by groups that fund and evaluate research (e.g., IES, What Works Clearinghouse) Update on Revisions to the Test Standards 19 Organization of Accountability Material Chapter on policy uses of tests focuses on use of aggregate scores for accountability and policy

Chapter on educational testing addresses student-level accountability (e.g., promotional gates, high school exit exams) and interim assessment Validity, reliability, and fairness standards in earlier chapters apply to accountability testing as well 02/08/2020 May 1, 2010 Update on Revisions to the Test Standards 20

Impact of Technology Technological advances are changing the way tests are delivered, scored, interpreted and in some cases, the nature of the tests themselves The Joint Committee has been charged with considering how technological advances should impact revisions to the Standards As with the other themes, comments on the standards that related to technology were compiled by the Management Committee and summarized in their charge to the Joint Committee 02/08/2020 Update on Revisions to the Test Standards

21 Key Technology Issues Included in our Charge Reliability & validity of innovative item formats Validity issues associated with the use of: Automated scoring algorithms Automated score reports and interpretations Security issues for tests delivered over the

internet Issues with web-accessible data, including data warehousing 02/08/2020 Update on Revisions to the Test Standards 22 Reliability & Validity of Innovative Item Formats What special issues exist for innovative items with

respect to access and elimination of bias against particular groups? How might the standards reflect these issues? What steps should the standards suggest with regards to usability of innovative items? What issues will emerge over the next five years related to innovative items/test formats that need to be addressed by the standards? 02/08/2020 Update on Revisions to the Test Standards 23 Automated Scoring Algorithms What level of documentation/disclosure is appropriate and tolerable for automated scoring

developers/vendors? What sorts of evidence seem most important for demonstrating the validity and reliability of automated scoring systems? What issues will emerge over the next five years related to automated scoring systems that need to be addressed by the standards? 02/08/2020 Update on Revisions to the Test Standards 24

Automated Score Reports and Interpretation Use of computer for score interpretation Actionable reports (e.g., routing students and teachers to instructional materials and lesson plans based on test results) Documentation of rationale Supporting validity evidence 02/08/2020 Update on Revisions to the Test Standards 25

Format Issues Consideration of ways to identify of Priority Standards More parallelism between chapter Tone Complexity Technical language Organization of Chapters 02/08/2020 May 1, 2010 Update on Revisions to the Test Standards

26 Possible Ways to Identify Priority Standards Clustering of Standards into thematic topics Over-arching Standards/ Guiding Principles Application Chapters Connection of standards to previous standards 02/08/2020 May 1, 2010

Update on Revisions to the Test Standards 27 Organization of Chapters 1999 Testing Standards Three sections Foundation: Validity, Reliability, Test Development, 02/08/2020 May 1, 2010 Scaling & Equating, Administration & Scoring, Documentation Fairness: Fairness, Test Takers Rights and

Responsibilities, Disabilities, Linguistic Minorities Applications: Test Users, Psychological, Educational, Workplace, Policy Update on Revisions to the Test Standards 28 Possible New Organization Section 1: Core Principles Validity, Reliability, Fairness Testing Operations Test Design and Development, Scaling & Equating, Test Administration & Scoring, Documentation, Test Takers, Test Users Section 2 : Applications Psychological, Educational, Workplace, Policy and Accountability

Update on Revisions to the Test 02/08/2020 May 1, 2010 Standards 29 Additional Important Information Documentation Third Party Observers Standardized Administration/Scoring Norms Technicians Supervision Diverse Populations 02/08/2020 Update on Revisions to the Test

Standards 30 Feedback Requested Feedback Information Web-based Necessary due to its impact Feedback Timetable Comment period- Early 2011 Revision period- Mid to late 2011 Publication expected- 2012 (APA) Next revision probably in 15 or so years 02/08/2020 Update on Revisions to the Test Standards 31

Recently Viewed Presentations

  • OLD BYLER ROAD PROJECT - Walker County History

    OLD BYLER ROAD PROJECT - Walker County History

    John BylerRoad Builder 1781 - 1824. John Byler was married to Elizabeth Walker, born in 1776. Their daughter, Elizabeth Catherine, married Eldridge T. Mallard in 1819. Mallard was the tollgate manager at Eldridge on the Byler Road, and the town...
  • Poetry: Sound & Sense

    Poetry: Sound & Sense

    POETRY: SOUND & SENSE Free Verse Free verse poetry does not follow a regular rhyme scheme or meter. Poets rely on poetic devices instead. There are several specific types of free verse poetry.
  • Chapter 3

    Chapter 3

    Carolyn Doering, HHSS ILLUSTRATION 3-8 SUMMARY OF ADJUSTING ENTRIES 1.Prepaid Assets and Assets overstated Dr. Expenses expenses expenses Expenses understated Cr. ... WRDSB Document presentation format: Custom Company: Sheridan College Other titles:
  • Transistors and Logic Gates - CS/ECE 252

    Transistors and Logic Gates - CS/ECE 252

    N-type: connect to GND, turn on (with 1) to pull down to 0 P-type: connect to +2.9V, turn on (with 0) to pull up to 1 Basic gates: NOT, NOR, NAND Logic functions are usually expressed with AND, OR, and...
  • Medical Assistance in Dying Community of Practice September

    Medical Assistance in Dying Community of Practice September

    Having said that there may be occasional times when the team member will be unable to meet the time frame (travelling in a car, managing another case). ... Physicians are to make no reference to medical assistance in dying, or...
  • Anne Bradstreet - PC\|MAC

    Anne Bradstreet - PC\|MAC

    Bradstreet's Use of Plain Style Although Anne Bradstreet's "Upon the Burning of Our House" contains some figurative language, it is a good example of the plain style. Plain Style vs. Ornate Style Inversion Bradstreet also used inversion. Inversion is the...
  • Prezentacja programu PowerPoint - TERENA

    Prezentacja programu PowerPoint - TERENA

    The IP phone receives a Directory Number (DN) the IP phone is ready to make and receive calls VoIP signalling Chapter 3 H.323 is an ITU-T recommendiation umbrella set of standards that defines components, protocols, and procedures necessary to provide...
  • [Business Communication] - UWI St. Augustine

    [Business Communication] - UWI St. Augustine

    In the Western Hemisphere, there are an estimated 200 million business units, 120 million of which are micro, small and medium enterprises (MSMEs). Of these MSMEs, which generate between 50 and 70 percent of jobsin the region, more than a...