Most educational program developers are very interested in how engaged participants or visitors are in their programs. Typical ways of determining engagement are through use of written surveys, intercept interviews, focus groups, and observations. Many of these data are more qualitative than quantitative. Unobtrusive observations using "tracking (what visitors saw) and timing (how long a visitor spends)" are often used by evaluators and researchers to gather quantitative audience data, as GRG did in a recent evaluation of the traveling museum exhibit on Black Holes. Rucha Londhe, GRG Research Associate and lead evaluator on the Black Holes evaluation, employed two additional methods developed by Serell, 1998:

1.  Sweep rate index (SRI): The exhibit's square footage is divided by the average total time spent there by a group of observed casual visitors. A lower sweep rate means that these visitors spent more time in the exhibit and presumably learned more from the exhibit.

2. Percentage of diligent visitors (%DV): The percentage of folks in the observed visitor group who stopped at more than half of the stations in the exhibit. Higher percentages of diligent visitors mean that more people were paying attention to more stations, and fewer exhibit elements were being ignored, skipped, or missed.

 
Based on analyses of data from numerous exhibits, the average value for SRI is 300 and the average % DV is 26. Thus, any exhibit with SRI lower than 300 (Black Holes was 100) and % DV higher than 26% (Black Holes was 40%) can claim to be an effective exhibit. The Black Holes exhibit far exceeded the average on both counts. In addition to these two analyses, Dr. Londhe found that the time spent at the exhibit was positively correlated with overall engagement; i.e., the longer the time spent at the exhibit, the higher the visitor engagement with the exhibit components, as measured by trained observers who rated how engaged they were. 

"We believe that these types of analysis have the potential to be used in settings beyond museums and in different fields; for instance, at a career fair, health expo or a field day program,"
Dr. Londhe stated; "We're eager to use them in some of our other evaluations."

GRG recently completed a rigorous evaluation of the Albert Einstein Distinguished Educator Fellowship Program, administered by Triangle Coalition of Arlington, VA.  Fellows are outstanding K-12 science, technology, engineering, and mathematics (STEM) teachers who spend a year in Washington, DC, bringing their expertise as classroom teachers to Congress and appropriate federal agencies while gaining a broader perspective on national educational issues.

GRG's multi-method evaluation plan included online surveys, phone interviews, focus groups, and site visits conducted over 18 months. Collecting data from three different stakeholder groups - Fellows and alumni, federal agency sponsors, and post-Fellowship supervisors - gave a comprehensive picture of the program from several viewpoints. The evaluation also included a comparison group of finalists, who were in the top 10% of applicants but were not offered an agency placement. Comparing Fellows to finalists helped disentangle program effects from the gains any highly qualified and motivated teachers might make over time.

The evaluation found that the Einstein Fellowship is a win-win-win program with benefits for all stakeholders. For Fellows, the program is a life-changing professional development experience providing knowledge and leadership growth. Federal agency sponsors and post-Fellowship supervisors agree with Fellow reports of their gains during the Fellowship year. Further, finalists do not make similar gains over the same time period. For agency sponsors, Fellows provide a very useful K-12 perspective on STEM education, and they assist with tasks that might not otherwise be accomplished. 

Project manager Karen Gareis said of the Fellows, "The Einstein Fellows are a really passionate, engaged group of STEM teachers to begin with, and it was amazing to see how much they made of their opportunities in Washington and how much they were transformed by the experience."  

Einstein Fellows Make Great Gains in their Ability to Understand and Track National and International Education Issues  

N = 120-122 alumni and 2009-2010 Fellows Note: The grey segment shows "before" scores, while the purple segment shows statistically significant gains (p < .05) from before to after program participation. "After" scores are significantly higher than those of finalists.

Ford Family Foundation Scholar Program

The Ford Family Foundation has contracted GRG to evaluate its Ford Scholar Program, which offers college scholarships to graduating high school seniors and community college students in Oregon and Northern California. In addition to the scholarship funds, Ford Scholars receive support through conferences, counseling, and on-campus resources. GRG is assessing the impact of the program through an online survey of alumni and fellows and focus groups with current scholars and their parents. The goal is to capture the benefits of the program, explore potential improvements, and contribute to the literature on college access.

WGBH Vietnam Open Vault

GRG Facilitated Discussion at Public Science Events Conference

GRG staff members Colleen Manning and Molly Priedeman, along with GRG consultant Karen Peterman, led discussions at the first ever International Public Science Events Conference, which took place the week of February 14th in Washington, DC. The first session, Knowing What Works: Festival Evaluation, was led by Ms. Manning, Ms. Priedeman, and Dr. Peterman. The second session, Strategies for Evaluating Science Cafes with Adult and Teen Audiences, was led by Ms. Manning, Dr. Peterman, and Susan Foutz of the Institute for Learning Innovation. In addition, Ms. Manning assisted NOVA/WGBH clients Rachel Connelly and Jennifer Larese in conducting a town hall meeting with the science café community focused on the current needs and future vision of the movement.

The Philadelphia Science Festival

GRG is actively involved in public science events and is currently conducting a multi-method process and summative evaluation to demonstrate the success of the Science Festival Alliance project. Science festivals were recently held in San Diego, CA; Philadelphia, PA; and Cambridge, MA. In the fall, there will be a science festival in Bay Area, CA.

Click here to read more about our evaluation here and the International Public Science Events Conference here.