ESRA 2013 - European Survey Research Association

ESRA 2013 - European Survey Research Association

General population surveys on the web: new findings from the UK Joel Williams ESRA 2013 TNS 19 July 2013 Context for the experiment? Community Life survey Covers volunteering, charity donation, civic participation, neighbourhood perspectives, attitudes to diversity etc. Revival of classically designed face-to-face interview probability sample survey but client has digital default philosophy Commissioned an experiment to gather evidence for an online survey alternative Recruited through postal address sample frame rather than access panel Similar to many of Dillmans experiments in US but larger scale Parallel face-to-face interview sample for comparison ESRA 2013 TNS 19 July 2013

2 What did we want to find out from the experiment? What response rate can be achieved with this method of recruitment? What difference do incentives make to the response rate? What is the data quality like? What is the profile of an online sample compared to a face-to-face interview sample? Do the two samples give similar estimates? What value is added by the inclusion of a postal questionnaire with reminder letter? ESRA 2013 TNS 19 July 2013 3 What did the experiment look like? 3 contacts

OR Conditional incentive of 0,5 or 10 2 contacts +1 contact Unconditional incentive of 0 or 5 ESRA 2013 TNS 19 July 2013 4 Incentive increases response rate but not to level achieved with

face-to-face interviews WEB ONLY No incentive WEB ONLY C5 incentive WEB ONLY C10 incentive WEB ONLY U5 incentive FACE-TO-FACE C5 incentive ESRA 2013 TNS 19 July 2013 5

Adding a postal questionnaire to 2nd reminder substantially increases the response rate (but shorter questionnaire) WEB +POSTAL No incentive WEB +POSTAL C5 incentive 31 WEB +POSTAL C10 incentive 35 WEB +POSTAL U5 incentive FACE-TO-FACE

C5 incentive 39 ESRA 2013 TNS 19 July 2013 6 What is the quality of the data using web designs? Some positive results: Little evidence of respondents speeding through the online questionnaire Same level of care taken to complete the questions Similar number of items selected from multi-code questions Similar differentiation between questions using same response scale No (additional) bias towards items at top of response list Some less positive results: One quarter of respondents appear to be the wrong individual Higher dropout than with F2F interviews especially at first few screens

More dont know and refusal answers (partly due to inevitably imperfect adaptation of interview questions) ESRA 2013 TNS 19 July 2013 7 What do we know about the demographic profile? No gender bias in the web samples (Expected) bias towards individuals who are... middle aged white degree educated working home-owning married Addition of postal questionnaire only slightly improved sample profile and not in all respects

ESRA 2013 TNS 19 July 2013 8 Despite increase in response rate when adding postal questionnaire, the age profile gets worse 16-24 25-34 35-49 50-64 65-74 75+

ESRA 2013 TNS 19 July 2013 9 How do the findings compare with findings from F2F interviews? Small sample sizes for each design so large margins of error around findings, even when borrowing power through use of main effects rather than saturated model Inclusion of a postal questionnaire with the second reminder does not make estimates more aligned despite much higher response rate Minimal impact of demographic weighting of data suggests fairly weak relationship between response propensity and key measures Differences may remain due to (i) different modes of data collection or (ii) insufficient accounting for sample differences ESRA 2013 TNS 19 July 2013

10 Incentive and demographic weighting improves alignment of estimates but many remain statistically significant [mean t scores] WEB ONLY No incentive WEB ONLY C10 incentive WEB ONLY No incentive DEMOG WTS WEB ONLY C10 incentive DEMOG WTS 2.0

2.0 4 MEAN T SCORE (difference from interview estimate) 3 2 2.7 1 2.4 0 ESRA 2013 TNS 19 July 2013

11 In summary... What response rate can be achieved with online-only data collection? A little over 15% unless an incentive is used What difference do incentives make to the response rate? A conditional incentive of 10 raises the response rate to 22% An unconditional incentive can achieve a slightly higher rate but cost is prohibitive What is the data quality like? Generally good although some doubt over random selection within household What is the profile of an online sample compared to a F2F interview sample? More likely to be middle-aged, white, well educated, working, home-owning and married Do the two samples give similar estimates? Expect approximately 40% to be significantly different (some large, most small) What value is added by the inclusion of a postal questionnaire? Very little; higher response rate but estimates no more aligned with F2F estimates

ESRA 2013 TNS 19 July 2013 12 What now? Client has commissioned a much larger test of the web-only method A number of questions still to answer: What is the most effective weighting strategy for an online-only survey? Can we get a better understanding of the impact of data collection mode on the results? What can we do to ensure random sampling of individuals at sampled addresses? How should we help data users account for a change in design? ESRA 2013

TNS 19 July 2013 13 Thank you ESRA 2013 TNS 19 July 2013 14

Recently Viewed Presentations

  • Special Education Needs and Disabilities (SEND) Reforms SEN

    Special Education Needs and Disabilities (SEND) Reforms SEN

    , co-ordinated across education, health and care, and involving children and young people and their families throughout. A new 0-25 Education, Health and Care plan . to replace the current system of Statements and Learning Difficulty Assessments, which reflect the...
  • Chapter 1 Introduction - Stellenbosch University

    Chapter 1 Introduction - Stellenbosch University

    AFM diagram showing the distinction between selected tholeiitic rocks from Iceland, the Mid-Atlantic Ridge, the Columbia River Basalts, and Hawaii (solid circles) plus the calc-alkaline rocks of the Cascade volcanics (open circles). From Irving and Baragar (1971). After Irvine and...
  • U.S. Department of Commerce Office of Security NOAA

    U.S. Department of Commerce Office of Security NOAA

    Security is Everyone's Responsibility . The Sponsor/COR submits: 1) Completed Security Coversheet (Revised 06/06/2014) 2) Fingerprint card (FD-258) / unless the subject is able to go to the NOAA Pass & I D Offices located
  • Multimedia - web.cs.wpi.edu

    Multimedia - web.cs.wpi.edu

    Multimedia CS-3013 & CS-502 Operating Systems
  • Index Laws Objectives: B Grade Use Index Laws

    Index Laws Objectives: B Grade Use Index Laws

    In full this looks like: x × x x × x × x × x x × x x × x × x × x Simplify = x-2 = 1 x2 So x-2 = 1 x2 This result can be...
  • What is a trade union? Trade unions are

    What is a trade union? Trade unions are

    What is a trade union? "Trade unions are organised groups of workers who come together to support each other in the workplace, negotiating with employers to improve pay, terms and conditions and ensure fair and equal treatment.. Unions provide members...
  • Topographic Maps - Weebly

    Topographic Maps - Weebly

    (3) They are closely spaced on steep slopes. (4) Evenly spaced contour lines indicate a uniform slope. (5) Contour lines do not cross or intersect each other (6) All contour lines eventually close on the map or beyond its margins....
  • Mr. Richard S. Page Chief Operating Officer 1

    Mr. Richard S. Page Chief Operating Officer 1

    All managers and employees will be transitioning into a new multi tasked PD. At each commissary and comprehensive two week training program takes place that includes training on the transition to a WOF store, lessons learned, team building, change management...