Are the Best Surveys a.) Standardized b.) Customized or c.) Both?
|
|
When choosing the best approach for designing a survey, you have a few options. You can choose a standardized, off-the-shelf one that fits with your program goals, or you can create one that is unique and specific to your organization. Or you can do as EVALCORP does for our clients - use a combination of both standardized and customized! One of our client's experiences helps show the benefits of integrating the two and how it can give you the best of both worlds.
Murrieta Valley Unified School District's (MVUSD) Breakthrough Student Assistance Program* addresses the need to reduce adolescent alcohol use, binge drinking and violence. With the purpose of measuring resiliency and AOD-related outcomes, the Breakthrough staff piloted use of the standardized survey - "California Healthy Kids Survey (CHKS), Modules B and C".
The outcomes for Breakthrough participants are clearly defined: increase resiliency and assets, reduce alcohol & other drug (AOD) use and associated high risk behaviors, and improve school connectedness, academic achievement and parent/child relationships. So the choice of this standardized survey is great for a few reasons:
- Items on the CHKS survey are already established for reliability and validity, and it is widely known and accepted among schools, funders and others.
- Districts throughout California use this survey so results can be compared to a general sample of youth in Murrieta and/or California as a whole.
- The survey includes many questions that directly relate to two of the Breakthrough outcome areas - AOD use and resiliency.
However, this survey does not address how students benefit specifically from MVUSD's program. Working together with Breakthrough staff, EVALCORP developed an additional survey customized to the program, the "Student Evaluation Survey (SES)." This survey asks students to describe their program experiences. For example: "Was the Family Conference plan helpful to you and your family?" and "What did you like best about the Breakthrough program?".
The two tools, CHKS Modules B & C and the Breakthrough SES, provided us and the program staff with two sets of necessary and complementary information.
The CHKS revealed that 20% more of the Breakthrough six-month participants were alcohol-free than were students who had not yet participated. The SES results substantiated this finding, with participants responding that one of the primary program impacts is that they are "less likely to drink alcohol or use other drugs." Further, the SES complemented the results of the CHKS by providing student perspectives about how the program helped them change their risk behaviors. See the MVUSD EVALBRIEF for more data results.
The year during which we first implemented both surveys was a pilot year for outcome evaluation. It provided an opportunity to see how the surveys worked and identify challenges. After this pilot phase, EVALCORP and Breakthrough came together again to refine the surveys and the administration process for the following year. We made several key changes:
- Reduced the CHKS Modules from 4 pages to 2 by including only those items most relevant to the Breakthrough program and MVUSD students.
- Added ID numbers to the surveys so we could connect students' CHKS and SES survey responses.
- Created an administration guide that clearly defines which participants to survey and when and provides sample "scripts" for staff to use to explain the survey purposes.
The amount of usable data that Breakthrough received this year, even from a pilot administration, is testament to the value of using both standardized and customized surveys. So next time, we suggest that you choose option c - both!
* Funded through the Riverside County Department of Mental Health Substance Abuse Program Prevention Services, as funded by the California Alcohol and Drug Programs, Safe and Drug-Free Schools and Communities (SDFSC) Governor's Program.
|
