# Assessing Quantitative Reasoning in Student Writing: A QuIRKy ...

Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience Nathan D. Grawe Carleton College Serc.carleton.edu/quirk With support from the US Department of Educations Fund for the Improvement of PostSecondary Education, the National Science Foundation, and the WM Keck Foundation. What is QR?

What is QR? The habit of mind to consider the power and limitations of quantitative evidence in the evaluation, construction, and communication of arguments in public, professional, and personal life.

What is QR? Four facets of QR: 1) QR requires a basic skill set What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context "The test of numeracy, as of any literacy, is

whether a person naturally uses appropriate skills in many different contexts" -National Council on Education and the Disciplines (2001) What is QR? Four facets of QR: 1) QR requires a basic skill set

2) QR demands application in context 3) QR involves argument What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves argument Deploying numbers skillfully is as

important to communication as deploying verbs. -Max Frankel, The New York Times Magazine What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context

3) QR involves argument Numbers [are] the principal language of public argument. -BBC Program More or Less What is QR? Four facets of QR: 1) QR requires a basic skill set

2) QR demands application in context 3) QR involves argument 4) QR is a habit of mind [QR] is not a discipline but a way of thinking. -Lynn Steen Achieving Quantitative Literacy What do the numbers show?

What do the numbers show? How representative is that? Compared to what? How was the variable operationalized?

Who is in the measurement sample? Registered Voters Eligible Voters 2006 Midterm Elections

Is the outcome statistically significant? Whats the effect size? Whats the effect size? Consider two statements: A) With a p<.001 we can conclude that

whites in MN, WI, and IA were more likely to vote for Obama than whites in LA, MS, and AL B) The fraction of whites who voted for Obama in MN, WI, and IA was at least 35 percentage points higher than that in LA, MS, and AL (<15% vs. >50%) forget the p value! Whats the research design

(correllational or experimental)? Fact: Those who work with computers earn 15-20% more than others. Thus, computer training may, at least in the short run, be a profitable investment for public and private job training programs.

Whats the research design (correllational or experimental)? Other interesting returns: Calculator = 12.8% Telephone = 11.4% Pencil/Pen = 11.2% Work while sitting = 10.1%

Controlling for what? Controlling for what? About 50% of the differential has to do with different career choices.

25% percent involves greater time women spend on care-taking. The other 25 percent is due to bias and prejudice. What is QR? sophisticated reasoning with

elementary mathematics more than elementary reasoning with sophisticated mathematics. -Lynn Steen Achieving Quantitative Literacy Implications for Assessment Four facets of QR: 1) QR requires a basic skill set

2) QR demands application in context 3) QR involves communication 4) QR is a habit of mind Implications for Assessment [QR] is largely absent from our current systems of assessment and accountability. -NCED (2001)

Implications for Assessment Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves communication 4) QR is a habit of mind Implications for Assessment

Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves communication 4) QR is a habit of mind Implications for Assessment Application in context: Standardized conditions are

decontextualized by design" -Grant Wiggins 'Get Real!': Assessing for Quantitative Literacy Implications for Assessment Communication: While traditional assessment tools

effectively measure comprehension, the ability to read others QR exposition does not guarantee the ability to engage in the creation of QR arguments. Implications for Assessment Habit of mind: As in book literacy, evidence of students ability to

play the messy game of the [QR] discipline depends on seeing whether they can handle tasks without specific cues, prompts, or simplifying scaffolds from the teacher-coach or test designer. -Grant Wiggins 'Get Real!': Assessing for Quantitative Literacy Implications for Assessment

"[QR] requires creativity in assessment, since neither course grades nor test scores provide a reliable surrogate." -Lynn Steen Achieving Quantitative Literacy Implications for Assessment

"The interdisciplinary and contextual nature of [QR] cries out for a cross cutting approach." -Lynn Steen Achieving Quantitative Literacy We want to regularly assess student work with numbers and numerical ideas in the field. -Grant Wiggins 'Get Real!': Assessing for Quantitative Literacy

The QuIRK Rubric QuIRKs idea: Measure QR in the natural context of papers written in courses across the curriculumpapers written for authentic purposes. The QuIRK Rubric

Sophomore Writing Portfolio 3-5 papers plus reflective essay written in 2 of the 4 college divisions -observation -analysis -interpretation -documented sources -thesis-driven argument

The QuIRK Rubric Writing requirement assessment: 30 faculty members, 3 days, 450 portfolios QR assessment: 6-8 faculty members, 3 days, random sample of portfolios, 1 paper each drawn from analysis, interpretation, or observation categories

The QuIRK Rubric 1. A little book-keeping The QuIRK Rubric 2. Did the student take the paper in a QR-relevant direction? The QuIRK Rubric

Central Use: Use of numbers to address a central question, issue, or theme Peripheral Use: Use of numbers to provide useful detail, enrich descriptions, present background, or establish frames of reference

The QuIRK Rubric The importance of the periphery: Even for works that are not inherently quantitative, one or two numeric facts can help convey the importance or context of your topic. -Jane Miller The Chicago Guide to Writing About Numbers

The QuIRK Rubric 3. If relevant, to what extent did the student actually use QR? The QuIRK Rubric 4. If present, rate the holistic quality of the QR.

The QuIRK Rubric 5. Spotlight on several recurring problems: The QuIRK Rubric 6. Does assignment explicitly call for QR? The QuIRK Rubric

Inter-Rater Agreement Statistics QR Relevance: 75% QR Extent: 82% QR Quality: 67% Problem Characteristics: 68%

Examples from Student Work Example of QR-irrelevant paper: The Maiden who Needs No Savingan analysis of Keats treatment of helplessness and power in La Belle Dame Sans Merci Examples from Student Work

Example of ineffective peripheral QR paper: Title: Untitled Thesis: A synthesis of 3 alternative views on poverty provides a more complete picture than any 1 perspective alone Examples from Student Writing

\$192 billion 1/65th of GNP \$60 billion 1/230th of GNP Examples from Student Work Example of effective peripheral QR

paper: Title: Les Banlieues dIslam Thesis: Examples from Student Work Example of ineffective centrally QR paper: Title: Untitled Thesis: Based on data from a time

diary and other sources, the American quality of life is better than that in lessdeveloped societies. Examples from Student Work Example of ineffective centrally QR paper: Day Care and Development Topic: Literature review of effects of day care participation on child

development Examples from Student Work Even though quantitative evidence is referenced extensively, student doesnt appreciate the real power of the actual numbers: Ex: The amount of time that children spent in day-care was positively

correlated with reports of how many friends they had. Examples from Student Work Example of effective centrally QR paper: Title: Modeling the Implicit Learning of Language Acquisition Thesis: Even if people arent aware of the systems undergirding language,

they implicitly learn the system through experience. Examples from Student Work In phase two, the mean number of correct string classifications the subjects gave was 17.3 out of 24; all but two participants correctly identified more than 12 strings, and the mode number of correct responses was 18.

[Discussion of statistical test.] It was expected that the participants would be able to correctly classify more than 12 of the 24 strings in phase two, so the result is not especially surprising. Still, the data provide a striking example of implicit learning. Infusing QR Across the Curriculum numeracy is not something mastered in a

single course.Thus quantitative material needs to permeate the curriculum, so that students have opportunities to practice their skills and see how useful they can be in understanding a wide range of problems. -Derek Bok (2006) authentic and enduring learningcan rarely succeed one course at a time. -Lee Shulman (1997)

Infusing QR Across the Curriculum If [QR] remains the responsibility solely of mathematics departments especially if it is caged into a single course such as Math for Liberal Artsstudents will continue to see [QR] as something that happens only

in the mathematics classroom. -Lynn Steen Achieving Quantitative Literacy Infusing QR Across the Curriculum Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are centrally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum Where are peripherally relevant papers written?

(Population: All papers submitted to writing portfolio) Legend Green = Math & Natural Sciences Blue = Social Sciences Red = Other Dark = Centrally Relevant

Medium = Peripherally Relevant Light = Irrelevant Infusing QR Across the Curriculum 30% of gen ed papers are centrally QR relevant Of these, nearly 50% are outside natural

sciences Another 25% are peripherally so Of these, 90+% are outside natural sciences and more than half are in Arts, Lit, & Humanities Infusing QR Across the Curriculum

By QR relevance, what fraction of papers use QR little/none, some, and extensively? Centrally Relevant Peripherally Relevant

Either Little/ None 19.5% 94.7%

52.1% Some 17.1% 5.3% 12.0%

Extensive 63.4% 0.0% 35.9%

Infusing QR Across the Curriculum Weasely examples: A) The downtown areas of large cities are often home to copious franchises. B) Recall Daycare Paper Our Curricular Response

Our Curricular Response Our Curricular Response Our Curricular Response Our Curricular Response Disciplines Currently Represented:

Our Curricular Response Coming Soon!: Our Curricular Response Example 1: An Ill-Structured Physics Lab Example 2: Intro to Lintina/o Studies Paper Example 3: History Paper on Indias Colonial Census

Example 4: Exploring and Architectural Remodel Our Curricular Response Example 5: Environmental History Paper Example 6: Development Economics Paper Example 7: Native-American Literature Analysis

Our Curricular Response

## Recently Viewed Presentations

• EU Regulation (EC) No 1272/2008 on Classification, Labelling and Packaging entered into force on 20 January 2009. It replaces the Dangerous Substances Directive and the Dangerous Preparations Directive. There is a transitional period 2010 -2015 during which time both classification...
• Block off any railing a child could climb over. Fix loose floor coverings & replace flammable rugs. ... Third Party Fundraising Opportunities such as the Brian M. Anselmo Foundation's Dancing with the Stars, KC CAN! Auction and Dinner, and Laura...
• 无线网、邮件、vpn使用介绍 信息化办公室 2015年1月 主要内容 无线网连接设置 1 邮件的多种登录方式 2 校外通过vpn方式访问校内图书馆资源 3 1.1 无线校园网的ssid 我校无线校园网接入点为： ccnu-auto(适合各类移动终端，有wifi信号即自动联网) ccnu(适合笔记本电脑及访客用户，使用网页登录) ccnu(小写 ...
• Methods and Tools for the Human Health Sector Kristie L. Ebi, Ph.D., MPH ... Menne B. Methods of Assessing Human Health Vulnerability and Public Health Adaptation to Climate Change. ... and on precipitation constraints on the survival and breeding capacity...
• Strwythur y Rhaglen. ... Mae'r her newydd ym Magloriaeth Cymru yn cynnwys her "Dinesydd Byd-eang" felly bydd angen i ddisgyblion i gyd gael eu cyflwyno ac ymgyfarwyddo â'r cysyniadau trwy gyfnodau cynharach eu haddysg os ydynt am lwyddo yn CA4...
• David Squires, Tompkins County, Director of Finance, and Community Foundation Board. Frost Travis, Travis & Travis Development and Ithaca Rentals and Renovations. Bryan Warren, Warren Realty. The Survey of Consumer Finances (SCF) is a triennial survey of the balance sheet,...
• Funding opportunities for JETSET Marie Curie fellows and other Early Career Researchers JETSET winter school, Galway, 8 January 2008 ... outgoing phase (1 - 2 years) for training and career development in any third country institution mandatory return phase (1...
• Needs-based education and skills learning architecture strengthened. Output 3: Protection and policy environment for self-reliance strengthened. Output 1 - ILO. Capacity building of local authorities, contractors, and training institutions to develop EIIP, promote job matching, promotion of access to procurement...