Copyright 2012 Pearson Education, Inc. or its affiliate(s).

Copyright  2012 Pearson Education, Inc. or its affiliate(s).

Copyright 2012 Pearson Education, Inc. or its affiliate(s). All rights reserved. 800 837 8969 Admissions testing: A journey in finding successful candidates Belinda Brunner Test Development Strategist Pearson VUE 2 The why, what and how of admissions testing 3 Why use standardised testing in selection? The purpose of an admissions test

Underlying premise: The higher education admissions selection process should be meritocratic and fair. MERITOCRACY. a system in which the talented are chosen and moved ahead on the basis of their achievement; leadership selected on the basis of intellectual criteria. meritocratic \mer--t-kra-tik\ adjective. FAIRNESS. Having or exhibiting a disposition that is free from favouritism or bias; impartial. fr \ adjective. The role for standardised testing What is a standardised test? An assessment in which the administration and

scoring procedures are consistent: Test takers receive same set of questions or questions are selected for administration based upon a set of balancing rules Administration conditions are consistent and standardised Scoring procedures enable comparisons across time and administrations 6 The goal of an admission selection process is to capture as clear a picture of the applicant as possible. Other factors in admissions decisions

Previous grades Personal statements Writing samples Interviews Recommendations Source: NACAC Admissions Trends Survey, 2013. As reported in State of College Admissions Report (http://www.nxtbook.com/ygsreprints/NACAC/2014SoCA_nxtbk/#/28) 9 Trends in admissions decisions

http://www.pbs.org/newshour/updates/new-toolcolleges-using-admissions-decisions-big-data/ http://www.onlineschools.org/beyond-the-sat/ Case study (Hypothetical) ABC University Large university offering a range of undergraduate degree programmes in the arts and sciences International students Current admissions process: Secondary school grades Application with personal statement Interview

11 Case study ABC University Why introduce an admissions test to the undergraduate admissions process? Common, standardised assessment Broaden access to higher education Controlled environment to complete the writing sample Targeted interviewing

12 Predictors of success: Aptitude vs achievement measures Aptitude: the degree of readiness to learn and to perform well in a particular situation or in a fixed domain ~ Corno, L., Cronbach, L.J., Kupermintz, H., Lohman, D.F., Mandinach, E.B., Porteus, A.W., & Talbert, J.E. (2002, p. 3). Remaking the concept of aptitude: Extending the legacy of Richard E. Snow. Mahwah, NJ: Lawrence Erlbaum Associates. Achievement: planned changes in cognitive behavior of students resulting from instructions or trainingdefined in terms of content ~ Haladyna, T.M. & Rodriguez, M.C. (2013, p. 5) Developing and validating test items. New

York: Routledge. 13 Predictors of success: Aptitude vs achievement measures Aptitude Reasoning tests Achievement Grades averaged across subjects

Course grades Subject-specific knowledge tests Adapted from Lohman (2004). Aptitude for college: The important of reasoning tests for minority admissions. In Zwick, R. (Ed.), Rethinking the SAT: The future of standardized testing in university admissions (pp. 41-54). New York: Routledge Palmer. 14 A comparison of medical school admissions tests Test Country of Origin Aptitude/

Reasoning Achievement/ Knowledge Writing Task(s) Website Target Population Secondary School Students Undergraduate Students BMAT UKCAT UMAT MCAT GAMSAT United United Australia

United States Australia Kingdom Kingdom Components www.admissionst estingservice.org

www.ukcat.ac.uk www.umat.acer.edu.a u www.aamc.org www.gamsat.acer.ed u.au Case study ABC University Process for determining the test constructs 16

Case study ABC University Admissions test components: University-wide reasoning test Verbal reasoning Quantitative reasoning Spatial ability Subject tests for selected degree programs Writing sample 17 Summary Why use a standardised test? Standardised admissions tests are particularly useful when there

is a lack of a standardised curriculum in the pre-university academic settings. Reasoning tests can be useful in identifying applicants who have not had the same opportunities to learn, but who have the ability to perform well if given the opportunity. Reasoning tests can also be useful in identifying students with a particular talent. 18 Reasoning tests thus have a place at the admissions table. It is not at the head of the table, as some once thought; rather, such tests provide a way to supplement grades and other measures of past achievement In other words, prior achievement is often an important aptitude for future learning. But it is never the only

aptitude, and sometimes not even the most important aptitude. ~ Lohman, D.F. (2004, p. 54). Aptitude for college: The importance of reasoning tests for minority admissions. In Zwick, R. (Ed.), Rethinking the SAT: The future of standardized testing in university admissions (pp.41-54). New York: Routledge Palmer. 19 What role should standardised testing play in selection? Using test scores Using admissions test scores 21

What part do admissions tests play in the selection process? 22 How are admissions test scores considered? Profiles of US students, grades 3-12, on the Verbal, Quantitative & Nonverbal batteries of the Cognitive Abilities Test, Form 6 (Lohman & Hagen, 2001) Lohman, D.F., Gambrell, J., & Lakin, J. (2008). The commonality of extreme discrepancies in the ability profiles of academically gifted students. Psychological Science, 50, 269-282. 23 What does this mean for how admissions test

scores are considered in the selection process? Consider 3 hypothetical students on a hypothetical admissions test: 3 sectional scores reported Verbal Quantitative Spatial reasoning Scores reported on a 300-900 scale 24 Options for how scores are considered (Lohman, 2012) And rule: Must score above the cut score on all sections to be considered for admission Avg rule: Average score must be above the cut

score to be considered for admission Or rule: Must score above the cut score on one of the sections to be considered for admission Lohman, D.F. (3 Dec 2012). An aptitude framework for college admissions testing. Paper presented at the First International Conference on Assessment and Evaluation. National Center for Assessment in Higher Education, Riyadh, Saudi Arabia. 25 Using admissions test scores Admitted No No And rule: Must score above the cut score on all sections

26 Using admissions test scores Admitted No Admitted Avg rule: Average score must be above the cut score 27 Using admissions test scores Admitted Admitted

Admitted Or rule: Must score above the cut score on one of the sections 28 Using admission test scores Student Major Scores Decision rule

Verbal Quant Spatial Avg And Avg Or 1 History

703 645 635 661 Y Y Y 2

Maths 440 675 640 585 N N Y 3

English 835 620 650 702 N Y Y

29 Using admission test scores Student Major Scores Decision rule Verbal Quant Spatial

Avg And Avg Or 1 History 703 645 635

661 Y Y Y 2 Maths 440 675

640 585 N N Y 3 English 835 620

650 702 N Y Y 30 Using admission test scores Student Major

Scores Decision rule Verbal Quant Spatial Avg And Avg

Or 1 History 703 645 635 661 Y Y

Y 2 Maths 440 675 640 585 N

N Y 3 English 835 620 650 702 N

Y Y 31 Using admission test scores (Lohman, 2012) The And rule The most restrictive Administratively convenient The same effect occurs if both admissions test scores and grades in previous coursework must exceed a minimum value. The Avg rule Separate scores can receive different weights Higher score in one area compensates for lower scores in others

Ignores distinct information if different scores reflect disparate test constructs The Or rule Most defensible of the three methods if the sectional test scores measure disparate test constructs May assist in identifying talent in a particular area 32 Case study ABC University Liberally-set minimum acceptable score or Minimum cut score;

scores used for placement 33 Case study ABC University or 34 Case study ABC University Common, standardised assessment All applicants complete the reasoning tests Broaden access to higher education

Or rule for secondary grades/reasoning test scores Controlled environment to complete the writing sample Writing sample completed as part of the standardised assessment Targeted interviewing Interviewing occurs after standardised testing 35 Summary What role should standardised testing play in the selection process? How test scores should be used depends upon: The purpose for testing

The context in which further learning occurs The other factors considered in the selection process Different ways of incorporating multiple measures into selection can lead to different decisions as to who is admitted and who is not. 36 How is a fit-for-purpose admissions test created? Test development and administration Meritocracy in testing 38 Validity: Predictive

Key question: How well does the admissions test predict subsequent academic performance? Most commonly studied using first-year grades Generally speaking, studies support the incremental validity of standardised admissions tests* Complicating factors: Restriction of range Criterion inaccuracies *See Zwick, R. (2006). Higher education admissions testing. In Brennan, R.L. (Ed.), Educational measurement (pp. 647-680). Westport, CT: Praeger and American Council on Education. 39 Validity: Construct/content

Research/expert opinion to support the included constructs Creation of a test plan/blueprint Linking of test items to the test plan SMEs involved in the item writing & review processes Review of items include content & statistical review Item bias and sensitivity review 40 Test factors influencing reliability The test itself Test length Item format & quality

Test administration Standardised & consistent testing experience Test construction & equating procedures Homogeneity in the test taker population a reliability coefficient tells us about a quality of a test (and one that we usually value) but not about the quality of a test ~ Traub, R.E. & Rowley, G.L. (1991, p. 44). Understanding reliability. Educational measurement: Issues and practice, 10(1), pp. 37-45. Retrieved 15 July 2013 from http://ncme.org/linkservid/65F3B451-1320-5CAE-6E5A1C4257CFDA23/showMeta/0/ 41 Fairness in testing The Standards for Educational and Psychological Testing (AERA, APA

& NCME, 2014) view fairness according to various perspectives: In treatment during the test process As a lack of measurement bias In access to the construct(s) measured As validity of individual test score interpretations for the intended uses 42 Use of computer-based testing (CBT) for admissions testing security Professional, standardised, consistent Increased testing environment Increased test centre coverage &

accessibility More frequent testing opportunities: windows or on-demand Immediate results reporting Advanced item types Measurement accuracy and efficiency Data-rich results Brand extension & customisation 43 Case study ABC University Undergraduate admissions test Test administration Computer-based test administered in a 4-month test window

Test design 3 sections: Verbal reasoning, quantitative reasoning, spatial ability Subsections (specific item types) defined for each section Test structure Designed to promote fairness and prevent item overexposure 44 Case study ABC University Test structure Verbal Quantitative Spatial

45 Case study ABC University Test structure Verbal Quantitative Spatial Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet 1 2 3 1 2 3

1 2 3 Testlet: A group of items that will be administered together All testlets within a section are comparable according to content and statistical properties Items within a testlet are administered in randomised order 46 Case study ABC University Test structure Verbal Quantitative

Spatial Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet 1 2 3 1 2 3 1 2 3 Pre-test pool (Select n from n) Pre-test pool

(Select n from n) Pre-test pool (Select n from n) Testlet: A group of items that will be administered together All testlets within a section are comparable according to content and statistical properties Items within a testlet are administered in randomised order Pre-test pool: Pre-test items administered for trialing/field testing purposes; they do not count toward the test-takers scores Items randomly selected from the pool for administration Each test taker receives the same number of pre-test items 47

Case study ABC University Scoring & results reporting IRT equating Scaled scores on a 300-900 scale Post scoring for early test takers, immediate scoring thereafter Test structure allows for building an item bank of pre-tested items to allow for immediate scoring in subsequent years 48 Summary Standardised tests can provide a reliable

measure to be considered in selection decisions. Test design, content, development, administration and use (and how these relate to the tests purpose) are all validity considerations. To be fair, testing procedures should treat all equitably, safeguard against bias, and the results should reflect testtaker ability on the desired test construct. 49 Summary Computer-based testing is state-of-the-art admissions testing. Uses computer technology that

applicants use in their private lives and for study Greater access and flexibility Supports a wider variety of question formats Supports the use of modern psychometric methods which strengthen the credibility and defensibility of the test 50 A selection system should always be considered a work in progress. ~ Lohman, D.F. (3 Dec 2012, p. 16). An aptitude framework for college admissions testing. Paper presented at the First International Conference on Assessment and Evaluation. National Center for Assessment in Higher Education, Riyadh, Saudi Arabia.

51 Thank you Belinda Brunner [email protected] 52

Recently Viewed Presentations

  • Folie 1 - UKE

    Folie 1 - UKE

    TRIALFINDER Offene Studien Solide Tumore I Solide Tumore II Hämatologische Erkrankungen Bronchial Mamma-Ca ALL Kopf-Hals-Tumor Ovarial-Ca AML Ösophaguskarzinom
  • Zombie-nets, Pop-ups, and Spam

    Zombie-nets, Pop-ups, and Spam

    Imagine a tax preparation program that erases your disk on April 14 cookies Data stored on your computer by a web server, and returned to that server on future connections Used to track you and your activities Not always a...
  • Abstraction Transformation & Heuristics Md Modasshir Sharmin Rahman

    Abstraction Transformation & Heuristics Md Modasshir Sharmin Rahman

    Consistency. A consistent (or monotone) heuristic function is a function that estimates the distance of a given state to a goal state, and that is always at most equal to the estimated distance from any neighboring vertex plus the step...
  • Honey, I Shrunk My Backup/Recovery Windows, You Can Too!

    Honey, I Shrunk My Backup/Recovery Windows, You Can Too!

    Integrated File System (IFS) Performance V5R1 and V5R2 Type2 Directories Save/Restore Performance Improvement Many Directories Containing Few Files For Example - OnDemand Product 1 File per Directory 50 Percent Improvement Low Ratio of Directories to Files, No Change Integrated File...
  • Harestock Primary School E-Safety Session  Parents Sue Savory

    Harestock Primary School E-Safety Session Parents Sue Savory

    There are a range of things online that might upset children and affect what should be a healthy online experience. It's important to remember that inappropriate content online includes pornographic content, but could also include other content such as race...
  • Productions/Operations Management

    Productions/Operations Management

    In the lean philosophy, the ideal lot size is one. Benefits of small lot size. Reduced in-process inventory. Lower carrying costs. Less storage space is necessary. Inspection and rework costs are less when problems with quality do occur. Permits greater...
  • Personification

    Personification

    Circle the Personified Object. 4. The brisk wind whispered secrets about faraway places. 5. The tornado attacked the corn field with great wrath. 6. The children built a tree house in the old elm on the hill, and now the...
  • PLANTS

    PLANTS

    Plants need water for growth. LIGHT LIGHT Green plants need sunlight in to make their own food. NUTRIENTS NUTRIENTS Plants need the minerals found in soil for healthy growth. WARMTH Plants only grow well in the right conditions. Temperatures that...