Evaluation is the cornerstone of public health programming.  Without evaluation data we would not know what programs actually work.  Evaluation is simply the practice of examining your project and your activities to ensure you are doing what you said you were going to do and if what you are doing is really working.  We do this by collecting information on the project before, during and after implementation.  Evaluation and project implementation should run parallel, but oftentimes are so far removed from each other that the task of evaluation seems daunting.  It doesn't have to be so ominous.

The specialized language and vocabulary can also be confusing and new to many people.  In essence, there are three kinds of evaluation that community and Tribally-based programming should be doing.

• Formative Evaluation: the process of collecting information on what the prevalent needs of the community are, who is at risk for specific conditions, and what public health programming may best need these needs.  In more familiar terms, we call formative evaluation a community assessment (or gap assessment, needs assessment, etc.). We use this information to create programs.

• Process-based Evaluation: this is the process of collecting information about how a program is running, what services are being delivered, to whom, and what resources are being used to deliver the services.  Having people sign-in when they attend an educational session so you know how many people attended is process-based evaluation.  You are simply collecting information about the process used to deliver a service and asking yourself, "Did this process do what we intended it to do?"

• Outcome-based Evaluation: this is the process of collecting information about what changes occurred because you implemented a program.  For example, if people learned something, then that is a change in knowledge.  If people quit smoking, then that is a change in behavior.  These changes are called outcomes and reporting on our outcomes tells us if the programs/services were effective or not.  And sometimes knowing if something is not working is just as helpful as knowing if something is working.

Knowing what you want to learn from your program implementation can tell you right up front what kind of data you will need to collect throughout your project.  For example, if you want to know if the community likes the program, then you will need to collect satisfaction-based information from people who attend your activities as well as community members who may not have attended an activity but are still impacting your work.  If you want to know if your collaboration with another department or Tribe has been successful, then you will have to collect information from employees and staff as well.  Laying out what you want to know, how you will get the information, and when is called evaluation planning.  Creating an evaluation plan should be done at the same time as program activities are planned-ideally when writing a funding proposal. 

Evaluation serves many different purposes, but for MSPI projects there are some key reasons why evaluation is vital to the success of MSPI.  Of course, at the Tribal level, it tells project managers and directors how well the project is performing.  That information is wonderful to ensure implementation of success projects, and to explore opportunities for improvement.  It also creates an evidence-base for best practices in Indian Country.  For the Indian Health Service, they can take information on project performance and use it to justify the budgetary expenditure needed to sustain the Methamphetamine and Suicide Prevention Initiative.  So, in essence, a good evaluation at the Tribal level can help to secure the future of the project in the coming years. 

Evaluation is not a fruitless waste of time. By understanding why we do it, the needs for it and how to perform evaluation more efficiently, we can strengthen our case for funding, improve our projects, and share with other communities what is working - all of which are solid contributions to the public health of Indian Country.

Working with an Evaluator

Ideally, Tribes should seek to raise their own capacity to conduct their own evaluation.  However, if that is not an option, then there are a variety of resources available for Tribes to explore to secure evaluation services.  Tribes may consider contacting their area Tribal Epidemiology Centers or Area Indian Health Boards to see if contract evaluation services are available.  Tribal Colleges are also a good place to search for evaluation services (from faculty or students); the same is true for non-Tribal institutions as well (even community colleges).  There are a large number of consulting firms that specialize in public health programming.  These firms are willing to assign a staff member to a specific project for the length of the project, and can provide a sense of continuity of the length of the project. There is no single listing of all of the evaluation firms in the country, but as word of mouth is their primary form of marketing, asking colleagues or even funders will generally provide a healthy list from which to start an inquiry. 

Contract evaluators will work with one Tribe or a number of smaller projects at the same time.  The scope of work will impact the consultant fees requested, but smaller projects coming together can pool funds to hire a single contractor.  A contract with an evaluator can be expensive, and every contract evaluator will have differing rates.  Most contractors will charge an hourly rate.  Tribes should plan on meeting with contract evaluators up front to discuss expectations and tasks, and then it is up to the contractor to prepare a proposed scope of work and total fees requested.  Tribes should feel negotiated this to align with their budgets.  And when considering a fee schedule, Tribes should consider creating performance-based contracts - where contractors are only paid upon completion and submission of specific deliverables (not on a regular pay basis). 

An evaluator (whether staff or contract evaluator) can and should do many things for a project - design an evaluation plan, create instruments, test instruments, conduct evaluation activities (hold focus groups, do phone interviews, etc.), analyze data, and/or prepare evaluation reports.  A contract evaluator can also be a source of capacity building assistance for Tribal programs.  An evaluator can provide training and one on one assistance to Tribal programs so that they understand the evaluation processes and can, in the future, conduct their own evaluation activities. 

Additional Resources

For more help with conducting and using evaluation, feel free to contact NIHB for technical assistance.  And please feel free to look at these additional resources:

Community Toolkit

CDC Program Evaluation Resources