Observations on Developing a Postsecondary Institution Rating System

Observations on Developing a Postsecondary Institution Rating System

Observations on Developing a Postsecondary Institution Rating System (PIRS) Dr. Braden J. Hosch Asst. Vice President for Institutional Research, Planning & Effectiveness Stony Brook University January 22, 2014 CONTEXT (1) Current role: Asst. VP for Institutional Research, Planning & Effectiveness at Stony Brook University (NY) IPEDS National Trainer, Advisory Board Member Former role: Director of Policy & Research, Connecticut Board of Regents for Higher Ed 2 CONTEXT (2) Connecticut changed its approach to higher education accountability in 2011 Broad stakeholder involvement

5 goals with 23 indicators, disaggregated Used data publicly available through IPEDS Institutional performance measured over time compared to similar institutions Accountability more than consumer information Work in progress See Higher Education Coordinating Council. (2012). Public Policy Framework for Higher Education. http://www.ct.edu/files/opr/A_CTFramework.pdf 3 CHALLENGES Comparison Groups Graduation Rates Learning Outcomes Affordability Employment Outcomes 4 COMPARISON GROUPS Exclusively mathematical approach to identify institutions

for comparison groups was elusive A balance of data, statistics, and judgment was needed Data from IPEDS based on about 10 institutional characteristics generated lists of 20-50 institutions in most cases Outliers remained a problem, likely due to data quality / unique state practices Campuses asked to add or remove institutions from the comparison group based on judgment to arrive at 10-15 comparison institutions 5 GRADUATION RATES Controls for student inputs are essential to the interpretation of graduation rates; individual factors matter much more than institutional factors Data from HERI illustrate this relationship, independent of institutional effects 6 INSTITUTION-BASED 6-YEAR GRAD RATES BY SAT COMPOSITE SCORE

Source: IPEDS Data Center Public Institutions Private Not-for-Profit Institutions 7 Graduation Rate (percent) INDIVIDUAL-BASED GRADUATION RATES BY HIGH SCHOOL GPA 100 90 80 70 60 50 40 30 20

10 0 79.3 70.6 59.8 48.7 36.6 21.2 6.3 C or less 27.7 9.8 C+ 58.3 47.8 35.9 25.2

15.5 B- B B+ A- A+, A HS GPA as Reported on the CIRP Freshman Survey 4-Years 6-Years Source: Higher Education Research Institute (UCLA) 8 Graduation Rate (percent)

STONY BROOK UNIV. GRADUATION RATES BY HIGH SCHOOL GPA 100 90 80 70 60 50 40 30 20 10 0 79.12 58.85 33.93 Below 87

62.61 40.21 87-89.9 67.26 46.12 90-92.9 71.93 60.6 50.31 93-95.9 96+ HS GPA 4-Years 6-Years

Source: SBU Office of Institutional Research, Planning & Effectiveness 9 Graduation Rate (percent) STONY BROOK UNIV. GRADUATION RATES BY HIGH SCHOOL GPA AFRICAN AMERICAN ONLY 100 90 80 70 60 50 40 30 20 10 0 75.21

74.48 66.33 51.65 52.3 80.39 71.93 56.86 49.12 39.65 Below 87 87-89.9 90-92.9 93-95.9

96+ HS GPA 4-Years 6-Years Source: SBU Office of Institutional Research, Planning & Effectiveness 10 COMPLETIONS PER 100 FTE ENROLLMENT Connecticut adopted a completions per FTE metric intended to supplant graduation rate Statistical adjustments were necessary Lag enrollment by 2 years Weight certificates by 1/3 Degree-seeking UG enrollment only Data adjustments were necessary Results not comparable across sectors

Cannot be interpreted without reference groups 11 LEARNING OUTCOMES Deemed very important for quality assurance but measurement deferred pending further research Research had demonstrated Testing is sensitive to recruitment practices and testing conditions Student motivation affected institutional results State pursuing external validation of learning using authentic assessment through a nine state multi-state collaborative 12 AFFORDABILITY AND NET PRICE Net price involves reasonable components, but wide variation exists in how institutions determine costs associated with room & board and other

expenses for students living off-campus. Regulatory or legislative action needed to define calculation method 13 EXAMPLE NET PRICE ANOMALIES Expenses Living Off-Campus, Not with Family:4 Institutions within 1.6 miles $25,000 $20,000 other expenses room and board $8,528 $15,000 $10,000 $5,000 $7,410 $2,146

$6,840 $5,823 0 0.4 $13,117 $3,450 $4,900 $0 1.5 1.6 Distance (miles) Source: IPEDS Data Center, components are for 2011-12

14 EMPLOYMENT OUTCOMES Available employment data have severe limitations Alumni survey response rates for public institutions typically range 20-30%, with likely high levels of non-respondent bias State unemployment insurance (UI) data do not cross state lines or identify full-time/part-time employment or partial/full quarter earnings IRS data are problematic for joint returns Age and prior employment confound central tendencies more research is needed 15 EXAMPLE WAGE DATA ANOMALIES Avg. Weekly Wages & Change in Weekly Wages for Undergraduate Completers $1,200 $1,076 $1,000 $800

$600 $540 $473 $400 $254 $451 $507 $472 $270 $200 $0 CT Community Colleges

Charter Oak State College (Public online college) Weekly Wages Upon Entering Employment CT State Universities (Public Regional) Univ. of Connecticut (Public Flagship) Change in Wages from Start of Program Data gathered from unemployment insurance records through CT Dept. of Labor Source: CT Employment & Training Commission Legislative Report Card (2013) 16 RECOMMENDATIONS Consider institutional input for comparison groups Adopt individualized graduation rate calculator

Define cost of attendance methodology for offcampus arrangements Defer inclusion of and support research on: Learning outcomes Employment outcomes 17 REFERENCES Connecticut Dept. of Labor and Connecticut Dept. of Higher Education (2010). Building Connecticuts Workforce: Report on 2007-08 Graduates. Retrieved January 13, 2014 from http:// www.ctdhe.org/info/pdfs/2010/HigherEdReport-2008grads.pdf Connecticut Higher Education Coordinating Council. (2012). Public Policy Framework for Higher Education. Retrieved January 10, 2014 from http://www.ct.edu/files/opr/A_CTFramework.pdf. DeAngelo, L., Franke, R., Hurtado, S., Pryor, J. H., & Tran, S. (2011). Completing college: Assessing graduation rates at four-year institutions. Los Angeles: Higher Education Research Institute, UCLA. Hosch, B. (2008). Institutional and student characteristics that predict retention and graduation rates. Annual Conference of the North East Assoc. for Institutional Research, Providence, RI. Hosch, B. (2012). Time on test, student motivation, and performance on the Collegiate Learning Assessment: Implications for institutional accountability. Journal for Assessment and Institutional Effectiveness 2(1): 55-76.

National Governors Association. (2010). Complete to Compete: From Information to Action: Revamping Higher Education Accountability Systems 18

Recently Viewed Presentations

  • Randomized Phase III Trial of Regorafenib in Patients (pts ...

    Randomized Phase III Trial of Regorafenib in Patients (pts ...

    Radioactive Iodine (RAI)-Refractory Differentiated Thyroid Cancer (DTC) It is estimated1 that in the USA in 2013 there were: >60 000 new cases of thyroid cancer, and
  • Diesel fumes  are they the next asbestos? What

    Diesel fumes are they the next asbestos? What

    Polycyclic aromatic hydrocarbons PAHs, ie semi-volatile organic compounds, including unburned diesel. Unburnt carbon particulates (with adsorbed organic compounds), sulphates, nitrates and other trace elements, not to mention other mining extraction-related pollutants such as radon gas and asbestos fibres.
  • 48x36 poster template - University of Cincinnati

    48x36 poster template - University of Cincinnati

    This lesson is intended to be a fun way to show how trigonometry can be used in a real-world situation. The students are given a playbook in which some of the route information is missing. They are asked to determine...
  • Innovative Product Development Class Spring 2016 MGMT /

    Innovative Product Development Class Spring 2016 MGMT /

    MGMT / ENGR / CARC / SPH/LA 489 & 689. This course is multi-disciplinary taught with faculty from several colleges that will provide real world, hands-on learning about starting a company and launching an innovative product. You will talk to...
  • A Linear Space Algorithm for Computing Maximal Common ...

    A Linear Space Algorithm for Computing Maximal Common ...

    This means that the number of unique next hop information is greater than the number of output ports. (a) Multibit trie for Figure 1(a) with strides {3, 2, 2}. (b) Its leaf-pushed trie. -2 : with child link -1 :...
  • Standard EPS Shell Presentation

    Standard EPS Shell Presentation

    Many kinds of friction exist. 5.2 Friction 5.2 Friction and two surfaces Friction depends on both of the surfaces in contact. When the hockey puck slides on ice, a thin layer of water between the rubber and the ice allows...
  • Configuration Management - Winthrop

    Configuration Management - Winthrop

    Use Good CM Tools Have a Sound CM Procedure CM Repository - Common Jargon CM Objects CM probably organized as object-oriented database Basic Objects Aggregate Objects Configuration Item standard term for entities in the CMR CM Repository - Features Supports...
  • Preschool Curriculum

    Preschool Curriculum

    Example: The Literacy foundation block contains Phonological Awareness, including identification of beginning word sounds. This is an essential foundation of learning to read. Foundation blocks