Maine 2003 - University of North Texas

Conducting Scientifically-Based Research in Teaching with Technology, Part I SITE Annual Meeting Symposium Atlanta, Georgia Gerald Knezek & Rhonda Christensen University of North Texas Charlotte Owens & Dale Magoun University of Louisiana at Monroe March 2, 2004 Our History of Scientifically-Based Research Foundation: More than ten years of instrumentation development/validation Research based on dissertation criteria Large data sets analyzed (replication of findings) Quantitative to tell us what is

happening; Qualitative to tell us why it is happening Components for Evaluation with a Research Agenda Plan for Evaluation (when writing the grant - not after you get it) Use reliable/valid instruments and/or Work on developing instruments the first year Get baseline data - how can you know how far you have come if you dont know where you started Use comparison groups such as other PT3 grantees Common Instruments Stages of Adoption of Technology

CBAM Levels of Use Technology Proficiency Self Assessment Teachers Attitudes Toward Computers (TAC) Online Data Acquisition System Provided by UNT Unix/Linux Based Stores Data in Files Data Shared with Contributors Why are we gathering this data? Campbell, D. T. & Stanley, J. C. (1966). Experimental and Quasi-Experimental Designs for Research on Teaching. From Gage, N. L. (Ed.) Handbook of Research on Teaching. Boston: Rand McNally, 1963.

Frequently references: McCall, W. A. (1923). How to Experiment in Education. Adding Research Agendas to Evaluation By experiment we refer to that portion of research in which variables are manipulated and their effects upon other variables are observed. (Campbell & Stanley, 1963, p. 1) Dependent = outcome variable; predicted or measured; we hope this depends on something Independent = predictor variable; one manipulated to make, or believed to make a difference Did changing x influence/impact/improve y? Y = f(x) Longitudinal Designs

PT3/Univ. of North Texas: 1999-2003 Baseline data year 1 Pre-post course measures over multiple years Trends in exit student survey data PT3/University of Nevada/Reno: 2003-2006 Best features of UNT plus comparisons w/UNT Added random selection of 30-60 teachers to track retention through end of induction year Stages of Adoption of Technology Fall 1998 6 5 5.2 4.13

3.9 4 4.1 4.4 3.1 3 2 1 0 Typical CECS 1100 CECS 3440 CECS 4100 Teacher Students

Students Students (n=1141) Pretest Post Test CECS 4100 Technology Skills Pre and Post - Spring 1999 6 5.6 5.84 5.39 5.5 5

5.56 4.93 4.5 4.54 4.5 4.03 4 4.72 3.96 3.51

3.5 2.91 3 2.5 2 1.5 1 E-mail WWW Integrated App. Tech. in Teaching

Multimedia Web skills Skills pre-test post-test What is the Experiment here? Dependent variables: Email, WWW, Integrated Applications, Teaching with Technology Competencies Independent Variable: completion of content of course (CECS 4100, Computers in Education) Longitudinal Trends in Integration Abilities (Research Item)

Stages of Adoption: CECS 4100 (Computers in Education) Univ. of North Texas 6 5 4 3 2 1 0 Pre Post

Fall 2001 ES Pre Post ES Spring 2002 Pre Post Fall 2002 ES

Pre Post ES Spring 2003 Stage 1: Aware ness I am aware that technology exists but have not used it - perhaps I'm even avoiding it. Stage 2: Learning the process I am currently trying to learn the basics. I am often frustrated using computers. I lack confidence when using computers. Stage 3: Understanding and application of the process I am beginning to understand the process of using technology and can think of specific tasks in which it might be useful.

Stage 4: Familiarity and confidence I am gaining a sense of confidence in using the computer for specific tasks. I am starting to feel comfortable using the computer. Stage 5: Adaptation to other contexts I think about the computer as a tool to help me and am no longer concerned about it as technology. I can use it in many applications and as an instructional aid. Stage 6: Creative application to new contexts I can apply what I know about technology in the classroom. I am able to use it as an instructional tool and integrate it into the curriculum. From: Christensen, R. (1997). Effect of technology integration education on the attitudes of teachers and their students. Doctoral dissertation, University of North Texas. Based on Russell, A. L. (1995) Stages in learning new technology. Computers in Education 25(4), 173-178. Growth in Technology Integration Course at Univ. of North Texas (Typical PT3 Evaluation Item) CECS 4100 Enrollment Spring '99 - Spring '03

Enrollment Numbers 200 154 150 Spring Sum I Sum II Fall 129 102 100 50

167 52 46 21 20 52 47 16 12 0 99 00 47

37 21 0 0 01 02 Year 03 Data Sharing with PT3 Projects Control groups are difficult

Comparisons within CE clusters is easy! Similar trends are positive confirmations for each other Spring 2002: Snapshot Data Univ. North Texas Texas A&M Univ. St. Thomas of Miami Univ. Nevada at Reno Northwestern Oklahoma State Univ. Wichita State University (Kansas) Demographics Spring 2002 481 subjects from 5 schools for pretest

UNT = 179 TAMU = 65 Miami = 14 Nevada = 91 Oklahoma = 95 Wichita St. = 37 157 subjects from 3 schools for post test UNT, TAMU, St. Thomas (2 times) Demographics Spring 2002 (cont.) Age: Wichita State students are older Mean = 28 years

Gender: UNT & TAMU have more females 85% and 97% Graduation: UNT, Nevada, Oklahoma students expect to graduate later Teaching Level: TAMU students Elem. Educational Technology Preservice Courses Fall 2002 Pre and Post - UNT and UF 6 5 UNT Pre

4 UF Pre UNT Post UF Post 3 2 1 0 Stages CBAM Tpemail

TPWWW TPIntegrated Apps TP Teach with Tech Educational Technology Preservice Courses Spring 2003 Pre and Post - UNT and UF 6 5 4 UNT Pre UF Pre

3 UNT Post UF Post 2 1 0 Stages CBAM Tpemail TPWWW

TPIntegrated TP Teach with Apps Tech What is the Experiment here? Dependent Variable: Gains in technology integration proficiency Independent Variables: Completion of course content (as before) Comparisons/contrasts among different environments/curricular models (value added) General Findings Reliability of Stages is High (r = .88 test-retest) Reliability of Skill Self-Efficacy Data is

High (Alpha = .77 to .88 for 4 TPSA scales) Gender: Females are higher in Web Access, Home Computer Use, and WWW Skills Spring 2002 Pretest - Six PT3 Sites 5.5 5 4.5 UNT-Pre UNT-Post TAMU St. Thomas UNReno Wichita State NWOSU

4 3.5 3 2.5 2 1.5 1 Stages TP-email TP-WWW TP-IA TP-TT

Pre-Post Trends for TAMU: Two Teacher Preparation Courses 6 5 4 TAMU22Pre TAMU22PST 3 TAMU21Pre TAMU21Pst 2 1 0

CBAM Stages Stages2 TPSA-IA TPSAEmail TPSA-TT TPSAWWW Impact Across 2 Schools (PrePost, UNT & TAMU) Stages: ES = .42 to .76 CBAM LOU: ES = .73 to 1.15 TPSA-IA: ES = .18 to .82 TPSA-TT: ES = .33 to 1.12 TPSA-WWW: ES = .05 to .49

How to Interpret Effect Size Cohens d vs. other Small (.2), medium (.5) vs. large (.8) Compare to other common effect sizes As a quick rule of thumb, an effect size of 0.30 or greater is considered to be important in studies of educational programs. (NCREL) For example .1 is one month learning (NCREL) others SRI International. http://www.ncrel.org/tech/claims/measure.html APA Guidelines for Effect Size The Publication Manual of the American Psychological Association (APA, 2001) strongly suggests that effect size statistics be reported in addition to the usual statistical

tests. To quote from this venerable guide, "For the reader to fully understand the importance of your findings, it is almost always necessary to include Some index of effect size or strength of relationship in your Results section" (APA, 2001, p. 25). This certainly sounds like reasonable advice, but authors have been reluctant to follow this advice and include the suggested effect sizes in their submissions. So, following the lead of several other journals, effect size statistics are now required for the primary findings presented in a manuscript. UNR Collaborative Exchange New PT3 Project Univ. of Nevada - Reno is lead and IITTL at UNT as outside evaluator One component - following teachers after they graduate from the teacher

ed. Program Randomly select from a pool of 2004 graduates and contact them prior to graduation; pay a stipend to continue in the project by providing yearly data Procedure for Unbiased Selection Locate prospective graduates to be certified to teach during spring 2004 Number consecutively Use random number table to select a preservice candidate from the list Verify student completed technology integration course with B or better Invite preservice candidate to participate during induction year and possibly beyond Repeat process until 60 agree to participate

From Edwards, A. L. (1954). Statistical Methods for the Behavioral Sciences. NY: Rinehart. Maine 2003 Maine Learning Technology Initiative (MLTI) 2001-2002 Laptops for all 7th graders 2002-2003 Laptops for all 7th and 8th graders in the whole state of Maine Maine Learns is About Curriculum Interesting Aspects of Research Sample or Population (all 17,000 students in the state) Selection of Exploratory Schools (if wished to participate, one from each region) Statistical measures of significance

Strong reliance on Effect Size Research Design 9 Exploration schools (1 per region) Compared with 214 others Used 8th grade state-wide achievement Examined trend over 3 years in math, science, social studies, and visual/performing arts Intervention Extensive teacher preparation Laptop and software for every 7th-8th teacher/student Some permitted to take home, others not

2003 Findings Evaluators reports Achievement Effect Sizes Student self reports on Attitudes toward school Self Concept Serendipitous findings are the sometimes the most valuable Home Access Gender Equity MEA 2000- 2001: Group 1 = 9 Exploration Schools, G roup 2 = All Others Effect Size Group Statistics (Cohen's D)

GROUP N Mean Std. Dev Science 1 9 529.11 3.95 0.05 2 204 528.90 4.25 SocStud 1 9 531.33 4.39 -0.12

2 204 531.89 4.54 Math 1 9 527.78 3.87 0.03 2 204 527.61 5.03 VPArts 1 9 531.00 5.59 0.06

2 204 530.65 5.52 MEA 2001- 2002: Group 1 = 9 Exploration Schools, G roup 2 = All Others Group Statistics Effect Size (Cohen's D) GROUP N Mean Std. Dev Science 1 9 529.56 3.84 0.44 2 214

527.67 4.27 SocStud 1 9 529.44 4.36 -0.06 2 214 529.76 5.20 Math 1 9 527.78 6.61 0.21 2 214

526.59 5.72 VPArts 1 9 530.33 4.72 0.11 2 213 529.67 6.10 MEA 2002- 2003: Group 1 = 9 Exploration Schools, G roup 2 = All Others Group Statistics Effect Size (Cohen's D) GROUP N Mean

Std. Dev Science 1 9 529.00 3.43 0.22 2 211 528.03 4.52 SocStud 1 9 531.44 3.32 0.02 2 211 531.35

5.41 Math 1 9 528.44 3.88 0.22 2 211 527.37 4.94 VPArts 1 9 531.67 4.50 0.22 2 211 530.37 6.08

0.50 0.40 0.30 0.20 2000-2001 2001-2002 2002-2003 0.10 ci al S

So -0.20 VP A M at h -0.10 tu di es 0.00

Sc ie nc e MLTI 9 Project School Scores vs. 200 Other Maine Middle Schools, in Standard Deviation Units Effect of Maine Learning Technology I nitiative 2000 - 2003 Would Cohen Have Predicted This Effect? "Small Effect Size: d = .2. In new areas of research inquiry, effect sizes are likely to be small (when they are not zero!). This is because the phenomena under study are typically not under good experimental or measurement control or both. When phenomena are studied which cannot be

brought into the laboratory, the influence of uncontrollable extraneous variables ("noise') makes the size of the effect small relative to these (makes the 'signal' difficult to detect). Cohen, J. (1977), p. 25. Exploratory - as Illustrated by: Impact of Computer Access Restricted to School - Maine 7th Graders June 2003 5 4.5 4 3.5 3 2.5 2 1.5 1

0.5 0 No Access Outside School Take Home Laptop and/or Other Home Access Effect Size CAQ Attitude Toward School CAQ Self Concept CAQ Email Skill

CAQ Total Skill Contrast with Louisiana Confidence Intervals (Teacher Perceptions of Impact) 4.0 3.5 3.0 2.5 Mean 2.0

95%CIUpper 95%CILower 1.5 MEAN ListngSkill MusicInterst MathSkills AREA PostiveLrng PosEdEffect

ReadngSkill Teachers' Perception of Usefulness of ARTS to the Delta for Math vs. Fostering Interest in Music, Learning, or Education in General N 22 22 22 22 22 22 22 22 22 22 Math Mean Math SD Music SD

t 2.41 1.05 3.09 2.41 1.87 Postive Learning Experience 2.41 1.05 3.05 1.33 1.77 Positive Effect on Education 2.41 1.05 2.95 1.4 1.45

Signif 0.068 not quite significan 0.0837 not quite significan 0.1552 not statistically sign Reading Reading SD Music SD t 2.32

1 3.09 1.34 2.16 Postive Learning Experience 2.32 1 3.05 1.33 2.06 Positive Effect on Education 2.32 1 2.95 1.4 1.72 Signif

0.0365 statistically signific 0.0459 statistically signific 0.0932 not quite significan Math Skills vs. Music Interest P value and statistical significance: The two-tailed P value equals 0.0680

By conventional criteria, this difference is considered to be not quite statistically significant. Confidence interval: The mean of Group One minus Group Two equals -0.6800 9 5% confidence interval of this difference: From -1.4125 to 0.0525 Intermediate values used in calculations: t = 1.8735 df = 42 standard error of difference = 0.363 Source: Graphpad Quickcalcs. Free Online Calculators for Scientists. Graphpad.com. Retrieved February 27, 2004.

Its all About Confidence As shown in Figure 1, three of the measures 95% confidence intervals are roughly 3/4 of a confidence interval band above that is, no more than 1/4 of the 95% confidence interval range overlaps from the upper to the lower group. Differences in this range are as a rule-ofthumb meaningful according to emerging APA guidelines, and roughly comparable to a p = .05 level of significance (Cumming, 2003). The effect size for the combined upper three versus the lower two is approximately [((3.09+3.05+2.95)/3) ((2.32+2.41)/2]/ ((1.34+1.33+1.40+1.00+1.05)/5) = (3.03 2.37) / 1.22 = .66 / 1.22 = .54, considerably larger than the .30 cutoff beyond which technology interventions are considered meaningful (Bialo & SivinKachala, 1996). Teachers rated the ARTS to the Delta class as much more useful for promoting interest in music and creating a positive effect on students overall education experience that for improving reading and math skills.

Recently Viewed Presentations

  • Anatomy & Physiology Lecture 24: The Digestive System

    Anatomy & Physiology Lecture 24: The Digestive System

    Lecture 24: The Digestive System Lecturer: Dr. Barjis Room P307 Phone: (718) 260-5285 E-Mail: [email protected] * * * * * * * * * * * * * * * * * * * * * * * * *...
  • Chapter 13

    Chapter 13

    Case involved a challenge to a New Jersey law allowing the state to bus students to parochial schools. Since the . Everson . ruling in 1947, the Court has ruled some forms of state aid to parochial schools constitutional but...
  • PowerPoint Presentation

    PowerPoint Presentation

    Demerit Goods. In economics, a demerit good is a good or service whose consumption is considered unhealthy, degrading, or otherwise socially undesirable due to the perceived negative effects on the consumers themselves. It is over-consumed if left to market forces.
  • MEF Global Interconnect Briefing

    MEF Global Interconnect Briefing

    The introductions of all IP technologies like EVDO, WCDMA R.5, LTE, and WiMAX require IP backhaul in addition to the legacy TDM/ATM. Ethernet is the right choice for growing data services. The introductions of All-IP technologies like EVDO, WCDMA R.5,...
  • PRIMARY HEALTH CARE - University of Pittsburgh

    PRIMARY HEALTH CARE - University of Pittsburgh

    WHAT IS PRIMARY HEALTH CARE Contd. ELEMENTS OF PRIMARY HEATH CARE Contd. PRINCIPLES OF PRIMARY HEALTH CARE GOALS TO BE ACHIEVED BY 2000 The Basic Requirements for Sound PHC (the 8 A's and the 3 C's) Strategies of PHC Strategies...
  • Similarity - Andrews University

    Similarity - Andrews University

    Congruent polygons have a scale factor of 1:1. 6.3 Use Similar Polygons. If two polygons are similar, then the ratio of their perimeters is equal to the ratios of their corresponding side lengths. Perimeters of Similar Polygons
  • GEOS Underground Storage Tank Annual Tank Registration

    GEOS Underground Storage Tank Annual Tank Registration

    An RO can request additional Account Types through the GEOS system. Certify and submit a data entry form in GEOS. Associate a consultant to their account to prepare applications. View and submit Online Payment. Track account transactions. View and prepare...
  • Ieee 802.11

    Ieee 802.11

    Wireless Medium Access Control (MAC)(Refer Section 7.3.1 and 7.3.2 in textbook)Slides Adopted from:Romit Roy Choudhury Wireless Networking Lectures University of Illinois at Urbana Champaign