Creating Beautiful Evaluation Results



Maybe you have a few hundred public opinion polls, some store surveys, or observations from parks or apartment complexes.  It was hard work to collect that data, and now you want to share the results.  But how do you do that?  More precisely, how do you present the evaluation results in ways that are interesting and engaging?  


Why Make Tables and Charts?



Beautiful Tables

• Always include the total number of people or things you counted.  For each category, list the number and percent so the reader can compare across categories.

• Avoid clutter.  Footnotes, double lines or images that don't relate to the material distract the reader from the main point of the table. 

Beautiful Graphs and Charts

•  As with tables, state the total number of people or things you are counting, and the number and/or percent in each category.

For More Information



Beautiful graphics created by WestEd
back to top

The Tobacco Control Evaluation Center will be hitting the road to offer trainings around the state on evaluation report writing.

Open to project directors, evaluators and staff of all TCP-funded programs in CA, these interactive trainings will cover the components of a final evaluation report, how your reports are scored, tips on writing the results section, and ways you can repackage pieces of the report for use with different audiences.

Trainings will be offered in the following locations:

The sessions will run from 10 a.m. to 3:30 p.m., with an hour afterward for "office hours" with TCEC staff if you would like individualized consultation.  Registration is free, and lunch is included.

Travel reimbursements are available for CTCP grantees traveling 100+ miles one-way to the conference location. Contact Diana Dmitrevsky at ddmitrevsky@ucdavis.edu or (530) 752-9951 for more information.   Prior approval for travel reimbursement is required.

Seating is limited, so register now by clicking this link!  Once you register, we'll send you a confirmation with the date and specific location, along with a map and agenda.

Please help us build our image archive of tobacco control activities by sharing your photos with us.  We are looking for shots of people conducting surveys, interviews, observations or evaluation trainings.  We also could use photos of apartment complexes, parks, outdoor dining, rodeos, tobacco advertising, smoking areas, signage, parks, beaches, casinos, etc. in your community.   For more submission details, click here.
 

Stay tuned for the next issue of our quarterly newsletter coming in May which will cover topics like:

More than Luck: Producing Usable Data
for a Smoke-free Casino Objective

 
Stay tuned on PARTNERS to hear how the casino chooses to move forward!

Photos by Jim G and twoGiraffe

back to top

    Analyze This!  Doing Quantitative Analysis


You have piles of surveys on your desk, and your computer practically spills over with data.  Now what?  To make sense of all of those responses and figure out what they can tell you will require quantitative analysis.  Quantitative refers to measurable numbers and percentages, including relationships between variables and statistical significance.
 
Let's look at why the numbers are important.  An example of a quantitative evaluation activity often used in process evaluation is a public opinion survey.  The results will be used to help convince policy makers, so the main purpose is to gather answers to the question "Are you in favor of XYZ policy?"  Numbers matter in this case because the more people voice that they are in favor of a policy, the stronger the campaign becomes.
 
But chances are you will also ask additional questions that can inform your campaign -- questions that measure awareness of the issue or explore various options for the way the policy could be framed, for example.  What's key is that you can quantify the answers and show how many people answered one way or another.  If you are also interested in showing how people with different characteristics answered the questions, you can add some demographic questions and analyze results by grouping responses according to demographic characteristics.
 
In outcome evaluation, quantitative data can show how much change occurred because of an intervention.  Is there less cigarette litter now than before the non-smoking policy?  Are fewer participants smoking than before they took a cessation class?  Are more apartments designated as non-smoking than before the housing campaign?  And did the difference occur across the board or only for segments of the population or the group of interest?
 
Although statistical analysis of quantitative data can get rather complicated, most Tobacco Control evaluation activities require only the most basic statistical concepts, which are percentages, means, and median.  These calculations can be fairly easily arrived at, especially with the help of software like Excel or an online program like SurveyMonkey which does the analysis for you.
 
Percentages
Once data are entered, you can start calculating.  If calculating percentages, make sure you look at how many people answered each of the questions.  For instance, "30% of respondents said 'yes' when asked if they smoked."  Note that the percentage is of respondents rather than the total number of people asked.  If your sample size was 120 but only 100 of those approached actually answered, then the percentages are calculated using the respondents only (n=100).
 
Means and Medians
Let's say you did a survey of bar night participants where the average age was important to know.  Using the data from the chart below, the mean (or average) age of respondents was 28 years.  The mean is easily calculated by adding all the ages and dividing by the number of respondents. But looking at this scatter plot graph which shows us each of the data points, we can easily see that a few respondents were much older than the rest.  This shifted the average age higher than the age of most participants. The actual midpoint, or median, of the ages of all participants is 26 years. This difference between the 26 years of the median and the 28 years average shows us what effect a few outliers, or atypical cases, can have on the mean.  For this reason, it is a good idea to report both the mean and median and/or standard deviation (a measure of the spread of data points from the mean) because a sophisticated critic might challenge results that show only means.

Relationships between Variables
If you are interested in showing relationships between variables (the things that were measured) or showing statistical significance (which can be useful if only a small percentage of difference is expected between categories or in change over time), a statistical program like SPSS is best suited for calculation.
 
Let's look at an example: Casino employees and patrons answered the same questions about smoking preference in the casino.  When analyzing results, you want to see if employees answered a particular question differently from patrons.  A statistical program lets you compare the variables and will tell you whether or not the difference is "statistically significant," meaning that it is not only accidentally different, but that the difference is big enough given the sample size that it can be considered to have something to do with the variable (e.g., the status of the person answering -- employee or patron) and not by chance.
 
Statistical significance is especially relevant for comparisons over time or between groups; for instance you may want to show whether or not and to what extent a difference occurred after your intervention.  When the difference is very obvious, it is not necessary to figure out statistical significance.  If before a smoke-free parks policy was passed 200 pieces of cigarette litter were collected compared to only 10 pieces after its passage, the difference is obvious.  But what if smoking incidences among students at a school declined by 1.4 percent?  Is that statistically significant or just a matter of chance?  Would another round of the intervention yield the same results? A statistical program can easily tell us whether mere chance was at work or if we are actually on to something.
 
Be aware, though, that statistical significance only shows that there is a difference.  It cannot tell us whether or not the difference occurred because of the intervention.
 
Using Results to Guide Program Efforts
So what if a person's status (employee or patron) predicts their smoking preference?  What does this mean for the intervention?  Let's say most patrons want to smoke but the employees don't. The purpose of pursuing a smoke-free casino is to protect employees and patrons from secondhand smoke, but casinos are concerned about losing patrons whose desire to smoke is not accommodated.  One recommendation could be to launch an education campaign directed towards patrons who may become more supportive if they understand that a smoke-free casino policy is an attempt to protect a group of people rather than to take freedoms away from smokers.
 
If even the patrons remain non-supportive, the campaign could take other tacks like collecting information about the experience of casinos that have gone smoke-free, exploring the financial benefit of reduced employee sick time or cleaning costs, or touting the likely economic advantage of marketing to patrons who would be attracted to a smoke-free venue.
 
So whatever your quantitative data reveals -- whether support or opposition for your campaign -- it can be useful to your program in deciding what needs to be done next.
 
For more ideas on how to analyze quantitative data with descriptive statistics, check out Tips & Tools #10 on the TCEC website.

Photo by Robyn Lee

back to top

 


The Tobacco Control Evaluation Center would like to introduce the newest member of our team, evaluation associate Travis Satterlund.  Travis comes to us from UC Berkeley, where he recently completed a two-year post-doctoral fellowship in the School of Public Health. 

As a research scientist, he combined his interests in qualitative research methods, public policy and law by conducting hundreds of hours of observations and interviews in Bay Area bars and taverns to evaluate California's smoke-free workplace law.  (If you ever need a recommendation for a good pub in the City, he's your man!)  As part of these studies, Travis was able to see firsthand the tobacco-related norm changes that have taken place in bars due to the hard work of people like you. 

Travis is excited be a part of the TCEC team and looks forward to assisting all of our Tobacco Control partners.  For more info on our newest evaluator, check out the TCEC website!    

back to top

  
  Wading through the Words:
  Deriving Meaning with Qualitative Analysis

How can you synthesize a lengthy conversation into usable data?  With qualitative information gathered from key informant interviews, focus groups or some record reviews, the aim is not to just tally the number of similar responses like with a survey.  Instead, you look for themes and patterns in order to understand what is meaningful and what is extraneous.  This is what thematic or content analysis consists of.  These patterns and themes tell a story, and this story should be used to inform your program efforts. 
 
So how do you go about analyzing qualitative data?  Typically, you develop what is called a "coding manual" based on the main topics that are central to the interview questions or the objective.  Once a preliminary list of themes is created, actual data (quotes from interviews or notes from observations, etc.) can be coded.  Quite often you will have data relating to something you set out to evaluate as well as data you were not necessarily expecting to find.   
 
For instance, if you were evaluating the implementation and enforcement of a smoke-free park  policy, your preliminary list of broad themes might include:  barriers to implementation, public viewpoints, policy makers' rationale, factors related to smoking in parks, reasons for successful implementation, and so on.  Because of the emergent nature of analyzing and organizing the data, your code list will evolve over time as you find more sub-themes or categories in the data.
 
The next step is the coding.  There are many ways to sort and code the data, although the idea remains in the same regardless of the method used.  Some people use qualitative statistical software programs like ATLAS.ti, The Ethnograph, and Nud*ist.  However, by no means are such software packages necessary.  In fact, unless you have conducted hundreds of interviews, coding the data can be done by simply sorting the data into separate Word documents based on the respective codes, or even by cutting out the quotations and creating separate piles based on the codes.  What is key is that you read the transcripts, notes or documents several times, allowing the information to sink in and percolate.  
 
Regardless of how you decide to code the data,  during this stage the data get separated into categories.  For example, here is an excerpt of a transcript of a county health administrator:  "Yeah, last year our city passed a law regarding smoke-free parks.  I thought it was a great law, and much needed.  I know for a fact that one of the children's parks was a frequent hangout for homeless and vagrant folk, and they all seemed to smoke and drink there.  Anyway, the law was passed and went into effect earlier this year.  While I think the community supports it, I don't think the law really gets enforced.  I've gotten phone calls from citizens who report smoking.  So, I write a report and send it to the local police department, which is supposed to be in charge of enforcing the law.  However, I'm pretty sure they don't care; or, they certainly don't make it a priority.  I never hear from them, and people are still smoking in some of those parks."   
 
In coding this paragraph, you might first label it based on the demographics of the person interviewed.  In this case: white, male, county health administrator, etc.  The demographics provide context and may, in fact, play a major part of the overall story.  Next you would code according to theme.  You might code this entire paragraph under the theme of "Smoke-free Park Law" since this was the topic of the quote.  However, this same quote, or parts of it, may also be placed in other category codes as well.  For example, "homeless," "park smoking," "children's park," "alcohol," "enforcement," "written report," and/or "police."

Once all the data are coded, the real analysis begins.  In reviewing the coded data, decide whether the original themes and categories you started with were salient to the story that has emerged from the data.  Some people make tallies of how often a code or theme might come up in the data, while others prefer to use of method of weighting based on theory or relevance.  How you decide to do this is up to you, but you'll need to be consistent and transparent by describing the process you used in the write up of your results. 

You will also most likely collapse many of the codes or categories into new broader themes.  This is where it gets tricky for most newcomers, because there is no set way to do this.  But, it's helpful to think of the data as main points in a story about this issue.  What do the data say that will enable you to communicate this story?  Again, this is not easy, and it can be a painstakingly slow process, even for experts in the field (and why many people prefer quantitative data collection activities).  Nonetheless, the more you become intimately familiar with your data, the more a pronounced story will emerge.  It simply takes time. 
 
So how would you write up this data?  If a pattern emerged across several interviews, you could write about how one of the barriers to implementing a successful smoke-free park law or policy is the lack of enforcement.  You could further say that it was suggested that law enforcement officials do not view this policy or law as a priority to enforce.  Add a few direct quotes in the write up to document these findings.  And voila, you then have one part of the story!  
 
The most important part of any story, though, is the ending where the reader is told what it all means.  The real heart of your write up of any evaluation activity is where you interpret the findings and make recommendations for next steps.  So in this case, the data indicated that your project  may need to do more education and collaboration with local law enforcement. 
 
Ultimately, qualitative analysis is all about making sense of the various pieces of data so that your project has a clear picture about what needs to happen next (or what has happened and why). 
 
For more ideas about how to analyze qualitative data, check out Tips & Tools #7 and Tips & Tools #4 on the TCEC website.

Photos by: Robin Kipke, The Fluid Project

back to top