If there's anything we know for sure about a project going in, it is that the unexpected will occur. While you must track the outcomes you expect (i.e., what you told your funders and clients to expect), it's also important that you watch for and document unintended results. To accomplish all that, is it better to use qualitative or quantitative methods in your evaluation?
For us, there is no question. Qualitative and quantitative methods are different tools in the same box. You match the tool to the need. Where quantitative methods capture the numbers to support your accounts of effectiveness; qualitative methods help you articulate your story. The usefulness of qualitative methods is often less well-understood than quantitative. Read on for some qualitative data guidelines and tips.
What you gain with qualitative methods
Qualitative methods are those data collection techniques that do not rely on pre-determined response choices. The aim is not to measure the quantity of people who have a particular experience with your program but rather to understand how people actually experience it. These methods allow you to gain a deeper understanding of what people think, believe and experience. Each technique allows you to gather slightly different information. In-depth Interviews allow individuals to express what is important about their experience with your program during a confidential phone or in-person meeting
Focus Groups help you learn about the experiences of several stakeholders at once and explore what is common or not about those experiences
Direct Observations add environmental context and a first-hand understanding of how a program is being delivered
Document Reviews help reveal the history and evolution of a program through the words already recorded by those involved at different stages
Case Studies provide rich detail about how a program works and achieves its goals by looking in-depth at one specific example - an individual person, site or organization
With each of the above, you do not go in with fixed expectations about what you will learn. Instead, you go in with a guide that includes well thought-out, open-ended questions. Rather than ask, "On a scale from 1 to 5, how much did the program help improve student attendance?" the question is, "In what ways did the program help students this year?"
Information considerations for deciding to use qualitative methods Before you pick up that qualitative tool, it is important to clarify your information needs and make sure those needs match the data approach. Some considerations for when to choose qualitative methods...
Your program is in its early stages, or next stages, of development. Qualitative methods (e.g., focus groups, interviews) help you uncover the needs of your target population, gather information to define strategies that will work for them, and explore what outcomes actually result from your work. Among other things, the findings can help you to shape your program as it's being implemented to ensure it fits the people it was designed to assist.
You are not required to deliver numbers. There is sometimes a discrepancy between what you want to know and what funders want to know. You may want and need to know the whole gamut of outcomes that result from your efforts. Methods such as in-depth interviews will deliver that for you. But, if you still need to deliver concrete numbers, as in 75% of students improved attendance, quantitative methods need to stay in the mix.
You find that the numbers aren't enough. Knowing that a majority of your clients improved in ways you expected and planned for is very powerful information. That percentage tells you the result, but does not help you understand why it happened or its significance in your client's lives. Methods such as interviews and case studies help you gather testimonials of what those changes mean for individuals and can help you examine the links between your results and services.
You want to make improvements. There is no better way to know what does not work about your program in the ways you had hoped but to ask the very open-ended question, "What can we improve?"
Practical considerations for deciding to use qualitative methods It is equally important to assess your resources, internal and external, for effectively carrying out qualitative data collection techniques.
Time Because qualitative methods usually involve face-to-face interaction, they often take more time than quantitative methods such as an online survey. Once data are collected, time is also needed for the coding process in which themes and trends unfold through review and discussion.
Research knowledge To make sure the data are collected appropriately, the methods need to follow the principles of good research. Issues such as confidentiality, appropriateness of questions, how you choose your sample and coding reliability need to be addressed for your findings to be solid.
For examples on how to handle such issues, check out the Interview Summary Appendix from our recent work for the Ventura County Behavioral Health Department's Prevention and Early Intervention planning process.
A strong team Every player on the data collection team must be on the same page about the information that is needed and the techniques being used. Qualitative requires a high level of training, organization, and coordination.
Summing up - it's a balancing act Before your next evaluation, the question to ask then is not "To use qualitative or quantitative?" In less Shakespearian, but more useful terms, the question to ask is, "What are our information needs and resources?"
There really is no one right choice - qualitative or quantitative. We believe the most powerful evaluation designs rely on a complementary mix of numbers, information from key stakeholders, objective observations and the expertise of project staff.
|
