Mixed Methods in Program Evaluation Presented by Tom Chapel Thomas J. Chapel, MA, MBA Chief Evaluation Officer Centers for Disease Control and Prevention [email protected] 404-639-2116 Agenda 1. The why and how of mixed methods:
Rationale Options Challenges Criteria for making choices 2. Apply points to some simple examples
CDCs Evaluation Framework STEPS Engage stakeholders Ensure use and share lessons learned Standards Utility
Feasibility Propriety Accuracy Justify conclusions Gather credible evidence Describe the
program Focus the evaluation design The Standards apply especially when were trying to make data collection choices. CDCs Evaluation Standards
Standards Utility Feasibility Propriety Accuracy The Standards provide a quick and easy way to identify the 2 or 3 best data collection choices for this
evaluation. CDCs Evaluation Framework STEPS Not Collect data Not Analyze data Rather Engage stakeholders Ensure use
Gather credible evidence Gather credible evidence Describe the program Focus the
evaluation design Steps 1-3 Help You Focus Design And Data Collection Choices After the first 3 steps of the Evaluation Framework, we know which evidence will work for these stakeholders in this situation. Qualitative data? Quantitative data?
The Evaluation Standards help us narrow down our data collection choices to the handful of methods that will work for this evaluation at this time. Mixed Methods Data collection methods that will work for this evaluation at this time sometimes means surveys or
focus groups. But sometimes there is no one best way. The best choice would be a combination of methods or mixed methods. Six (Most) Common Ways to Collect Data
How Standards Inform the Choice of Methods Consider the context : Standards Utility Feasibility Propriety Accuracy How soon do I need the
results? What resources can I use? Are there any ethical issues to consider? How Standards Inform the Choice of Methods Also consider the content : Standards
Utility Feasibility Propriety Accuracy Sensitivity of the issue How Standards Inform the Choice of Methods Also consider the content : Standards Utility
Feasibility Propriety Accuracy The Hawthorne Effect Will the act of being observed cause someone to distort their response? How Standards Inform the Choice of Methods Also consider the content :
Standards Utility Feasibility Propriety Accuracy Validity Reliability Mixed Methods Address Concerns Key Concept: Regardless of the method, when there are
validity and reliability concerns, often using more than one method-- i.e., mixed methods-will help. Mixed Methods: Definition The combination of at least one qualitative and at least one quantitative component in a single research project or program. (Bergman 2008) Use Complementary Methods Mixed methods is: A combination of methods
that has complementary strengths and nonoverlapping weaknesses. The purpose is to supplement or complement the validity and reliability of the information. Strengths of Quantitative Methods Strengths of quantitative methods:
Require less time than qualitative methods Cost less Permit researcher control Quantitative data is considered to be scientific Easier to explain validity and reliability
Easily amenable to statistical analyses Strengths of Qualitative Methods Choose qualitative methods when you are trying to: Explore or describe a phenomenon
Strengths of Qualitative Methods Choose qualitative methods when you are trying to: Look for induction (i.e., surprise) Strengths of Qualitative Methods Choose qualitative methods
when you are trying to: Identify patterns Strengths of Qualitative Methods Qualitative data can help you understand not just what but WHY. When to Use Mixed Methods 1. Corroboration
better understanding; more credibility triangulation measuring the same thing from several different viewpoints 2. Clarification trying to understand why we got this result When to Use Mixed Methods Mixed Methods are most commonly used for: 3. Explanation similar to clarification want to know the why or what behind the situation 4. Exploration similar to explanation
charting new territory trying to observe patterns examine different situations and varying results to induce patterns Number of Project Facets Reported via Each Data Collection Method Source: Gregory Guest, PhD Number of Project Facets Reported via Each Data Collection Method
This is an example of using a qualitative method (site visits) to corroborate a quantitative method (surveys). The result was increased validity of the data. Source: Gregory Guest, PhD Which to Choose? How do you choose which methods
to use? Which method comes first, the quantitative or the qualitative? You have a lot of flexibility in these decisions. Parallel or Concurrent Mixed Methods QUANTITATIVE QUALITATIVE For parallel or concurrent mixed methods,
quantitative and qualitative data collection happen at the same time. Sequential Mixed Methods QUANTITATIVE QUALITATIVE OR QUALITATIVE
QUANTITATIVE For sequential mixed methods, either quantitative or qualitative data collection can happen first. Example of Sequential Mixed Methods to Corroborate Data QUALITATIVE In this case, the qualitative method (site visits) was used to corroborate the
quantitative (survey) method and the results were different. QUANTITATIVE Mixed Methods Is Your Choice You are never required to use mixed methods. However, you may choose to use mixed methods when: you have some indication that a single method may give you incorrect data.
a single method may give you an incorrect perception of reality. Mixing Methods During Data Analysis Qualitative data (focus groups, observations, secondary data, etc.) can be converted to numbers via quantitative techniques like content analysis.
This is also a mixed method design approach. Mixing Methods During Data Analysis Qualitative data can be very complex. Examining qualitative data with quantitative techniques helps to identify or validate patterns or themes.
Deciding When To Use Mixed Methods and How Key Concept: Using mixed methods is a deliberate design decision. You use it when you dont trust the data from any single method. The reason for your uncertainty determines the methods you choose to mix and the order in which you use them. Example 1 Concurrent Design
Do people give similar responses on Problem surveysor as Purpose: well as in focus groups? Survey (quantitative) and Example focus groups
(qualitative) are Example 2 Explanatory Sequential Design Use a qualitative method to explain Problem or Purpose: blindside results from a quantitative
method. Survey Example (quantitative) Example 3 Exploratory Sequential Design Problem or Purpose: Explore Verify suspected patterns
Focus Example groups Design Options Summary You mix quantitative and qualitative methods in a different order depending on the presenting problem: Validate results
Design Options Summary You mix quantitative and qualitative methods in a different order depending on the presenting problem: Validate results Explain the unexpected Design Options Summary You mix quantitative and qualitative methods in a different order depending on the presenting
problem: Validate results Explain the unexpected Explore new themes Selected Resources (Page 1 of 2) Caracelli, V. and J. Greene (eds.). 1997. Advances in MixedMethod Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms. San Francisco, CA: Jossey-Bass. Creswell, J. and V. Plano Clark. 2010. Designing and Conducting Mixed Methods Research, 2nd edition.
Thousand Oaks, CA. Sage Publications. Morse, J. and L. Niehaus. 2009. Mixed Method Design: Principles and Procedures. Walnut Creek, CA. Left Coast Press. Selected Resources (Page 2 of 2) Johnson, R. Burke, and L. Christensen. Evaluation Methods. 2008. www.southalabama.edu/coe/bset/johnson/ Plano Clark, V. and J. Creswell. 2008. The Mixed Methods Reader. Thousand Oaks, CA: Sage Publications. Teddlie, C. and Tashakkori, A. 2009. Foundations of Mixed
Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA. Sage Publications. Recommended Resource Creswell, J. and V. Plano Clark. 2010. Designing and Conducting Mixed Methods Research, 2nd edition. Thousand Oaks, CA. Sage Publications. The Community Tool Box
Community Tool Box http://ctb.ku.edu Chapter 37, Section 5. Collecting and Analyzing Data End Mixed Methods Return to Webinar 4: Gathering Data, Developing Conclusions, and Putting Your Findings to Use Return to Evaluation Webinars home page
Hammett plots - the most common linear free energy relationship. It is important to know effects of substituents on chemical properties. In particular, this contributes to understanding reaction mechanisms and to predicting rate constants and equilibrium constants.
What you see on the screen is a partial sample observation checklist. The checklists are designed to provide descriptive language teachers can use to capture quickly the nature of the evidence of student work in relation to the standards.
Beach Byelaws. Fingal County Council in exercise of the power conferred on the Council by Part VII of the Local Government Act, 1994 and in the interest of the common good of the local community make the following Bye-Laws for...
"Original Sin"(Handbook, pages 83-85) Adam and Eve, the first humans, had but one command from God to follow, and they still disobeyed God. Original Sin . is the term used to describe the condition of humanity's separation from God due...