Conducting Scientifically-Based Research in Teaching with Technology, Part I SITE Annual Meeting Symposium Atlanta, Georgia Gerald Knezek & Rhonda Christensen University of North Texas Charlotte Owens & Dale Magoun University of Louisiana at Monroe March 2, 2004 Our History of Scientifically-Based Research Foundation: More than ten years of instrumentation development/validation Research based on dissertation criteria Large data sets analyzed (replication of findings) Quantitative to tell us what is
happening; Qualitative to tell us why it is happening Components for Evaluation with a Research Agenda Plan for Evaluation (when writing the grant - not after you get it) Use reliable/valid instruments and/or Work on developing instruments the first year Get baseline data - how can you know how far you have come if you dont know where you started Use comparison groups such as other PT3 grantees Common Instruments Stages of Adoption of Technology
CBAM Levels of Use Technology Proficiency Self Assessment Teachers Attitudes Toward Computers (TAC) Online Data Acquisition System Provided by UNT Unix/Linux Based Stores Data in Files Data Shared with Contributors Why are we gathering this data? Campbell, D. T. & Stanley, J. C. (1966). Experimental and Quasi-Experimental Designs for Research on Teaching. From Gage, N. L. (Ed.) Handbook of Research on Teaching. Boston: Rand McNally, 1963.
Frequently references: McCall, W. A. (1923). How to Experiment in Education. Adding Research Agendas to Evaluation By experiment we refer to that portion of research in which variables are manipulated and their effects upon other variables are observed. (Campbell & Stanley, 1963, p. 1) Dependent = outcome variable; predicted or measured; we hope this depends on something Independent = predictor variable; one manipulated to make, or believed to make a difference Did changing x influence/impact/improve y? Y = f(x) Longitudinal Designs
PT3/Univ. of North Texas: 1999-2003 Baseline data year 1 Pre-post course measures over multiple years Trends in exit student survey data PT3/University of Nevada/Reno: 2003-2006 Best features of UNT plus comparisons w/UNT Added random selection of 30-60 teachers to track retention through end of induction year Stages of Adoption of Technology Fall 1998 6 5 5.2 4.13
Multimedia Web skills Skills pre-test post-test What is the Experiment here? Dependent variables: Email, WWW, Integrated Applications, Teaching with Technology Competencies Independent Variable: completion of content of course (CECS 4100, Computers in Education) Longitudinal Trends in Integration Abilities (Research Item)
Stages of Adoption: CECS 4100 (Computers in Education) Univ. of North Texas 6 5 4 3 2 1 0 Pre Post
Fall 2001 ES Pre Post ES Spring 2002 Pre Post Fall 2002 ES
Pre Post ES Spring 2003 Stage 1: Aware ness I am aware that technology exists but have not used it - perhaps I'm even avoiding it. Stage 2: Learning the process I am currently trying to learn the basics. I am often frustrated using computers. I lack confidence when using computers. Stage 3: Understanding and application of the process I am beginning to understand the process of using technology and can think of specific tasks in which it might be useful.
Stage 4: Familiarity and confidence I am gaining a sense of confidence in using the computer for specific tasks. I am starting to feel comfortable using the computer. Stage 5: Adaptation to other contexts I think about the computer as a tool to help me and am no longer concerned about it as technology. I can use it in many applications and as an instructional aid. Stage 6: Creative application to new contexts I can apply what I know about technology in the classroom. I am able to use it as an instructional tool and integrate it into the curriculum. From: Christensen, R. (1997). Effect of technology integration education on the attitudes of teachers and their students. Doctoral dissertation, University of North Texas. Based on Russell, A. L. (1995) Stages in learning new technology. Computers in Education 25(4), 173-178. Growth in Technology Integration Course at Univ. of North Texas (Typical PT3 Evaluation Item) CECS 4100 Enrollment Spring '99 - Spring '03
Enrollment Numbers 200 154 150 Spring Sum I Sum II Fall 129 102 100 50
167 52 46 21 20 52 47 16 12 0 99 00 47
37 21 0 0 01 02 Year 03 Data Sharing with PT3 Projects Control groups are difficult
Comparisons within CE clusters is easy! Similar trends are positive confirmations for each other Spring 2002: Snapshot Data Univ. North Texas Texas A&M Univ. St. Thomas of Miami Univ. Nevada at Reno Northwestern Oklahoma State Univ. Wichita State University (Kansas) Demographics Spring 2002 481 subjects from 5 schools for pretest
UNT = 179 TAMU = 65 Miami = 14 Nevada = 91 Oklahoma = 95 Wichita St. = 37 157 subjects from 3 schools for post test UNT, TAMU, St. Thomas (2 times) Demographics Spring 2002 (cont.) Age: Wichita State students are older Mean = 28 years
Gender: UNT & TAMU have more females 85% and 97% Graduation: UNT, Nevada, Oklahoma students expect to graduate later Teaching Level: TAMU students Elem. Educational Technology Preservice Courses Fall 2002 Pre and Post - UNT and UF 6 5 UNT Pre
4 UF Pre UNT Post UF Post 3 2 1 0 Stages CBAM Tpemail
TPWWW TPIntegrated Apps TP Teach with Tech Educational Technology Preservice Courses Spring 2003 Pre and Post - UNT and UF 6 5 4 UNT Pre UF Pre
3 UNT Post UF Post 2 1 0 Stages CBAM Tpemail TPWWW
TPIntegrated TP Teach with Apps Tech What is the Experiment here? Dependent Variable: Gains in technology integration proficiency Independent Variables: Completion of course content (as before) Comparisons/contrasts among different environments/curricular models (value added) General Findings Reliability of Stages is High (r = .88 test-retest) Reliability of Skill Self-Efficacy Data is
High (Alpha = .77 to .88 for 4 TPSA scales) Gender: Females are higher in Web Access, Home Computer Use, and WWW Skills Spring 2002 Pretest - Six PT3 Sites 5.5 5 4.5 UNT-Pre UNT-Post TAMU St. Thomas UNReno Wichita State NWOSU
Pre-Post Trends for TAMU: Two Teacher Preparation Courses 6 5 4 TAMU22Pre TAMU22PST 3 TAMU21Pre TAMU21Pst 2 1 0
CBAM Stages Stages2 TPSA-IA TPSAEmail TPSA-TT TPSAWWW Impact Across 2 Schools (PrePost, UNT & TAMU) Stages: ES = .42 to .76 CBAM LOU: ES = .73 to 1.15 TPSA-IA: ES = .18 to .82 TPSA-TT: ES = .33 to 1.12 TPSA-WWW: ES = .05 to .49
How to Interpret Effect Size Cohens d vs. other Small (.2), medium (.5) vs. large (.8) Compare to other common effect sizes As a quick rule of thumb, an effect size of 0.30 or greater is considered to be important in studies of educational programs. (NCREL) For example .1 is one month learning (NCREL) others SRI International. http://www.ncrel.org/tech/claims/measure.html APA Guidelines for Effect Size The Publication Manual of the American Psychological Association (APA, 2001) strongly suggests that effect size statistics be reported in addition to the usual statistical
tests. To quote from this venerable guide, "For the reader to fully understand the importance of your findings, it is almost always necessary to include Some index of effect size or strength of relationship in your Results section" (APA, 2001, p. 25). This certainly sounds like reasonable advice, but authors have been reluctant to follow this advice and include the suggested effect sizes in their submissions. So, following the lead of several other journals, effect size statistics are now required for the primary findings presented in a manuscript. UNR Collaborative Exchange New PT3 Project Univ. of Nevada - Reno is lead and IITTL at UNT as outside evaluator One component - following teachers after they graduate from the teacher
ed. Program Randomly select from a pool of 2004 graduates and contact them prior to graduation; pay a stipend to continue in the project by providing yearly data Procedure for Unbiased Selection Locate prospective graduates to be certified to teach during spring 2004 Number consecutively Use random number table to select a preservice candidate from the list Verify student completed technology integration course with B or better Invite preservice candidate to participate during induction year and possibly beyond Repeat process until 60 agree to participate
From Edwards, A. L. (1954). Statistical Methods for the Behavioral Sciences. NY: Rinehart. Maine 2003 Maine Learning Technology Initiative (MLTI) 2001-2002 Laptops for all 7th graders 2002-2003 Laptops for all 7th and 8th graders in the whole state of Maine Maine Learns is About Curriculum Interesting Aspects of Research Sample or Population (all 17,000 students in the state) Selection of Exploratory Schools (if wished to participate, one from each region) Statistical measures of significance
Strong reliance on Effect Size Research Design 9 Exploration schools (1 per region) Compared with 214 others Used 8th grade state-wide achievement Examined trend over 3 years in math, science, social studies, and visual/performing arts Intervention Extensive teacher preparation Laptop and software for every 7th-8th teacher/student Some permitted to take home, others not
2003 Findings Evaluators reports Achievement Effect Sizes Student self reports on Attitudes toward school Self Concept Serendipitous findings are the sometimes the most valuable Home Access Gender Equity MEA 2000- 2001: Group 1 = 9 Exploration Schools, G roup 2 = All Others Effect Size Group Statistics (Cohen's D)
GROUP N Mean Std. Dev Science 1 9 529.11 3.95 0.05 2 204 528.90 4.25 SocStud 1 9 531.33 4.39 -0.12
0.50 0.40 0.30 0.20 2000-2001 2001-2002 2002-2003 0.10 ci al S
So -0.20 VP A M at h -0.10 tu di es 0.00
Sc ie nc e MLTI 9 Project School Scores vs. 200 Other Maine Middle Schools, in Standard Deviation Units Effect of Maine Learning Technology I nitiative 2000 - 2003 Would Cohen Have Predicted This Effect? "Small Effect Size: d = .2. In new areas of research inquiry, effect sizes are likely to be small (when they are not zero!). This is because the phenomena under study are typically not under good experimental or measurement control or both. When phenomena are studied which cannot be
brought into the laboratory, the influence of uncontrollable extraneous variables ("noise') makes the size of the effect small relative to these (makes the 'signal' difficult to detect). Cohen, J. (1977), p. 25. Exploratory - as Illustrated by: Impact of Computer Access Restricted to School - Maine 7th Graders June 2003 5 4.5 4 3.5 3 2.5 2 1.5 1
0.5 0 No Access Outside School Take Home Laptop and/or Other Home Access Effect Size CAQ Attitude Toward School CAQ Self Concept CAQ Email Skill
CAQ Total Skill Contrast with Louisiana Confidence Intervals (Teacher Perceptions of Impact) 4.0 3.5 3.0 2.5 Mean 2.0
95%CIUpper 95%CILower 1.5 MEAN ListngSkill MusicInterst MathSkills AREA PostiveLrng PosEdEffect
ReadngSkill Teachers' Perception of Usefulness of ARTS to the Delta for Math vs. Fostering Interest in Music, Learning, or Education in General N 22 22 22 22 22 22 22 22 22 22 Math Mean Math SD Music SD
0.0365 statistically signific 0.0459 statistically signific 0.0932 not quite significan Math Skills vs. Music Interest P value and statistical significance: The two-tailed P value equals 0.0680
By conventional criteria, this difference is considered to be not quite statistically significant. Confidence interval: The mean of Group One minus Group Two equals -0.6800 9 5% confidence interval of this difference: From -1.4125 to 0.0525 Intermediate values used in calculations: t = 1.8735 df = 42 standard error of difference = 0.363 Source: Graphpad Quickcalcs. Free Online Calculators for Scientists. Graphpad.com. Retrieved February 27, 2004.
Its all About Confidence As shown in Figure 1, three of the measures 95% confidence intervals are roughly 3/4 of a confidence interval band above that is, no more than 1/4 of the 95% confidence interval range overlaps from the upper to the lower group. Differences in this range are as a rule-ofthumb meaningful according to emerging APA guidelines, and roughly comparable to a p = .05 level of significance (Cumming, 2003). The effect size for the combined upper three versus the lower two is approximately [((3.09+3.05+2.95)/3) ((2.32+2.41)/2]/ ((1.34+1.33+1.40+1.00+1.05)/5) = (3.03 2.37) / 1.22 = .66 / 1.22 = .54, considerably larger than the .30 cutoff beyond which technology interventions are considered meaningful (Bialo & SivinKachala, 1996). Teachers rated the ARTS to the Delta class as much more useful for promoting interest in music and creating a positive effect on students overall education experience that for improving reading and math skills.
Case involved a challenge to a New Jersey law allowing the state to bus students to parochial schools. Since the . Everson . ruling in 1947, the Court has ruled some forms of state aid to parochial schools constitutional but...
Demerit Goods. In economics, a demerit good is a good or service whose consumption is considered unhealthy, degrading, or otherwise socially undesirable due to the perceived negative effects on the consumers themselves. It is over-consumed if left to market forces.
The introductions of all IP technologies like EVDO, WCDMA R.5, LTE, and WiMAX require IP backhaul in addition to the legacy TDM/ATM. Ethernet is the right choice for growing data services. The introductions of All-IP technologies like EVDO, WCDMA R.5,...
WHAT IS PRIMARY HEALTH CARE Contd. ELEMENTS OF PRIMARY HEATH CARE Contd. PRINCIPLES OF PRIMARY HEALTH CARE GOALS TO BE ACHIEVED BY 2000 The Basic Requirements for Sound PHC (the 8 A's and the 3 C's) Strategies of PHC Strategies...
Congruent polygons have a scale factor of 1:1. 6.3 Use Similar Polygons. If two polygons are similar, then the ratio of their perimeters is equal to the ratios of their corresponding side lengths. Perimeters of Similar Polygons
An RO can request additional Account Types through the GEOS system. Certify and submit a data entry form in GEOS. Associate a consultant to their account to prepare applications. View and submit Online Payment. Track account transactions. View and prepare...