The Best Laid Plans Evolve
"The best laid plans of mice and men oft go awry," is one of the most-often quoted lines of English poetry.  But is it true?  As evaluators, we observe the outcomes of well-constructed plans every day.  According to what we've seen, a more accurate, if less poetic version of the saying is: "The best laid plans of mice and men oft change and evolve."  That is exactly what happened as EVALCORP worked with Solutions for Treatment Expansion Project (STEP).  To best support the STEP initiative, we used a "developmental evaluation" approach - so when change happened, the measurement plan evolved.  Not only did the program not "go awry," it grew stronger than ever.

An article by Michael Quinn Patton "Evaluation for the Way We Work," originally published in The Nonprofit Quarterly, explains the differences between traditional evaluation and the practice of "developmental evaluation" that we increasingly employ at EVALCORP.  Our STEP experience demonstrates what distinguishes developmental evaluation from other forms and its value for innovative and emerging projects. 

Developmental evaluation supports learning and growth, versus assessing end results
We began working with the STEP initiative, based in San Diego County, in 2006.  With two years of funding from The California Endowment, their purpose was to break down obstacles to the establishment of residential drug and alcohol rehabilitation sites in the area.  Originally, their plan of action focused on countering negative perceptions about residential programs and advocating for modifications to the application process for opening or expanding a site, known as the Conditional Use Permit (CUP).  Their objectives and activities were built around advancing discussions with local decision makers and CUP gate-keepers by presenting data on the benefits of treatment sites for whole communities.

But as they implemented the plan, STEP staff learned that local government is at risk of violating fair housing laws when they deny use permits for the rehabilitation sites based on residents' Not In My Backyard (NIMBY) concerns.  So, while the fears that drive NIMBY could be addressed with the benefits of data, the law provided an even more powerful force for breaking down the permit obstacles.
With this knowledge came a significant shift in STEP's approach -- from education and discussion on the benefits of treatment sites to awareness of and compliance with legal protections for these homes.  This shift required a re-definition of the project's goals, strategies and evaluation activities.  To facilitate the evolving plan, EVALCORP adapted its approach from a more traditional formative evaluation to a developmental one.  Plans to assess the implementation were broadened to include a focus on using the evaluation to help shape the project's development and strengthen it.  This was a highly participatory process.  Staff from STEP and EVALCORP worked closely together at every stage, with both having an active role in shaping data collection instruments, identifying key informants, and interpreting evaluation findings.

A developmental evaluation provides a highly effective means of determining the changes a program must make in response to emerging outcomes as the plan unfolds, while keeping it all on track with the mission.  It was exactly what STEP needed.  As STEP Project Manager, Deborah Parker, said: "Our project was so rapidly and successfully evolving beyond our original objectives - in ways none of us could have anticipated - that we ended up in previously unexplored territory.  EVALCORP helped us both map and navigate that territory.  They shifted the evaluation to meet the new plans and helped us to develop strategies that moved us forward effectively."

Needed measures emerge during the process, versus being predetermined
During the second year of implementation, STEP integrated fair housing and other legal protections information into its purpose, objectives, strategies, materials, and media and advocacy efforts.  EVALCORP customized the evaluation to support this evolution.  It relied primarily on qualitative data collection methods, including document reviews, observational methods, and structured interviews with key informants and staff.  Such methods are more useful than quantitative for allowing the story of a project to un-fold from the viewpoint of those involved.  There are times when a project needs to be assessed according to specific criteria; and developmental evaluation allows for that to emerge during the process as well.  

The findings helped STEP to understand and adapt to priority needs as they existed, rather than as they were initially conceived to be.  The evaluation also helped point out the most effective activities and successful outcomes.  Project achievements at this stage included:
  • an increase in understanding at the local, state, and national levels about the application of fair housing laws and other legal protections to residential providers for persons with disabilities;
  • growth in positive public perceptions about rehabilitation centers;
  • an increase in providers seeking permits through CUP; and
  • emerging dialogues and relationships between providers, advocates, local governments, and county and statewide organizations.  
STEP then received funding for an additional two years from The California Endowment.

The evaluator collaborates within the organization, rather than observes from outside
The project's experiences in San Diego soon led to even more initiative changes.  STEP expanded its education and advocacy efforts to four other counties:  Los Angeles, Orange, Riverside, and San Bernardino.  Because land use regulations for residential alcohol and other drug treatment providers are under the umbrella of residential service providers for persons with disabilities, the advocacy also expanded to encompass land use issues for providers for persons with disabilities of any type.  Collaborating with STEP staff, EVALCORP again customized evaluation methods that matched the program's evolving activities in the second phase.  

With a collaborative approach, the evaluation and the program work together in dynamic reciprocity, improving outcomes even as they are measured.  For example, project staff identified a need to document zoning and land use violations against residential treatment and housing providers for persons with disabilities in the five counties.  STEP and EVALCORP worked together to develop a ground-breaking online survey that could be quickly analyzed.  STEP staff presented these findings at the state assembly, using this "evaluation" data immediately in their advocacy activities.

Rather than standing outside the process as the disengaged observer, the developmental evaluator is a member of the organization's team, collaborating with staff in ongoing analysis, innovation, and enactment of the program's directives.  With this approach, EVALCORP and STEP created a potent combination of in-depth qualitative interviews, observational research, document review, and training survey findings that produced a clear narrative about STEP as it occurred.  

With the help of developmental evaluation, STEP has gone far beyond the CUP application process in San Diego County.  Today, the organization tackles land use issues for residential service providers of all kinds at the local, state and national level, truly demonstrating that the best laid plans evolve.