P E R S P E C T I V E S:
Evaluation & Research News
In This Issue
Using Tracking and Timing for Museum Exhibit Evaluation
Einstein Fellows Evaluation
Evaluation Spotlight
In Other News...
In Brief

Helena Pylvainen, Senior Research Assistant, in April, presented at the American Educational Research Association (AERA) Conference on our US Department of Education, BioBridge project. Read more about project.

Irene Goodman and Elizabeth Bachrach traveled to Israel as part of our evaluation of the Jim Joseph Foundation Bar Ilan University Fellows Program. Read more about project.

Rucha Londhe, Colleen Manning, and Irene Goodman were published in Autonomous Robots. Read more.

GRG is gearing up for data collection for the Corporation for Public Broadcasting-funded SID the Science Kid 2 evaluation. Read more about project.

GRG recently completed evaluation
of Reading Rockets Grant 9 for WETA-TV.  


Tina Lagerstedt


Want to learn more? Check us out online!


  Spring 2011  
Welcome from the President.....


In each of my annual State of the Company presentations to the GRG staff, I have highlighted the successes and challenges of the previous year. This past March, we had a number of accomplishments to review from 2010. We:

  • Pushed the envelope with our data collection and analysis methods
  • Played a greater role in facilitating presentations at professional meetings and conferences
  • Published articles in journals and newsletters
  • Displayed data in a more interesting, reader-friendly, and elegant way in our reports. 

A cadre of in-house staff and field researchers has been extremely busy this spring:

  • Administering student surveys and interviews and parent and teacher surveys for our own National Science Foundation (NSF) funded research study, the Massachusetts Linking Experiences and Pathways (M-LEAP) Project, in eight school districts across the state
  • Fanning out across three cities (San Diego, Philadelphia, and Cambridge) to conduct intercept interviews with and observations of festival attendees for the Science Festival Alliance.
  • For numerous other projects, we're collecting data from doctors and nurses, early childhood providers, parents of adolescents, opera aficionados, climate scientists, reading specialists, history teachers, Scholars, Fellows, and underserved minorities in science, technology, engineering, and mathematics (STEM) fields. And the list goes on. Find out more on our website, our Facebook page, or - even more radical - give us a call if there's anything you'd like to hear more about.
We have also been fortunate this year to have help at GRG from Jessica Parsons, a Master's candidate from Harvard Graduate School of Education, who served as an intern here this spring, and from Dreolin Fleischer, a former RA who has been working part-time with me on business development while making progress toward her doctorate in evaluation at Claremont Graduate School of Education. Plus, we've had a whole slew of students from local colleges helping us with the masses of data that we obtain from our evaluations.

Spring and summer do tend to bring visits from former staff, sometimes with babies in tow. Maria Fusaro, Ed.D., a former GRG research assistant who is now doing a post-doc at UC Davis, came to visit our offices recently. She and Dreolin started as RAs at GRG on the same day back in 2002, and they hired Laura Houseman as a student data entry clerk that year. Laura is now senior research assistant here at GRG!




(Pictured from left to right): Dreolin Fleischer, Irene Goodman, Maria Fusaro and son Leo, GRG's Director of Research Colleen Manning, and Laura Houseman.





Best wishes for the summer,

'Irene' signature
Irene F. Goodman, Ed.D.
Founder and President

Using Tracking and Timing Data for Museum Exhibit Evaluation

Most educational program developers are very interested in how engaged participants or visitors are in their programs. Typical ways of determining engagement are through use of written surveys, intercept interviews, focus groups, and observations. Many of these data are more qualitative than quantitative. Unobtrusive observations using "tracking (what visitors saw) and timing (how long a visitor spends)" are often used by evaluators and researchers to gather quantitative audience data, as GRG did in a recent evaluation of the traveling museum exhibit on Black Holes. Rucha Londhe, GRG Research Associate and lead evaluator on the Black Holes evaluation, employed two additional methods developed by Serell, 1998:


1.  Sweep rate index (SRI): The exhibit's square footage is divided by the average total time spent there by a group of observed casual visitors. A lower sweep rate means that these visitors spent more time in the exhibit and presumably learned more from the exhibit.


2. Percentage of diligent visitors (%DV): The percentage of folks in the observed visitor group who stopped at more than half of the stations in the exhibit. Higher percentages of diligent visitors mean that more people were paying attention to more stations, and fewer exhibit elements were being ignored, skipped, or missed.

Based on analyses of data from numerous exhibits, the average value for SRI is 300 and the average % DV is 26. Thus, any exhibit with SRI lower than 300 (Black Holes was 100) and % DV higher than 26% (Black Holes was 40%) can claim to be an effective exhibit. The Black Holes exhibit far exceeded the average on both counts. In addition to these two analyses, Dr. Londhe found that the time spent at the exhibit was positively correlated with overall engagement; i.e., the longer the time spent at the exhibit, the higher the visitor engagement with the exhibit components, as measured by trained observers who rated how engaged they were. 


"We believe that these types of analysis have the potential to be used in settings beyond museums and in different fields; for instance, at a career fair, health expo or a field day program,"
Dr. Londhe stated;
"We're eager to use them in some of our other evaluations."


GRG Completes Comprehensive Evaluation of Albert Einstein Distinguished Educator Fellowship Program

GRG recently completed a rigorous evaluation of the Albert Einstein Distinguished Educator Fellowship Program, administered by Triangle Coalition of Arlington, VA.  Fellows are outstanding K-12 science, technology, engineering, and mathematics (STEM) teachers who spend a year in Washington, DC, bringing their expertise as classroom teachers to Congress and appropriate federal agencies while gaining a broader perspective on national educational issues.


GRG's multi-method evaluation plan included online surveys, phone interviews, focus groups, and site visits conducted over 18 months. Collecting data from three different stakeholder groups - Fellows and alumni, federal agency sponsors, and post-Fellowship supervisors - gave a comprehensive picture of the program from several viewpoints. The evaluation also included a comparison group of finalists, who were in the top 10% of applicants but were not offered an agency placement. Comparing Fellows to finalists helped disentangle program effects from the gains any highly qualified and motivated teachers might make over time.


The evaluation found that the Einstein Fellowship is a win-win-win program with benefits for all stakeholders. For Fellows, the program is a life-changing professional development experience providing knowledge and leadership growth. Federal agency sponsors and post-Fellowship supervisors agree with Fellow reports of their gains during the Fellowship year. Further, finalists do not make similar gains over the same time period. For agency sponsors, Fellows provide a very useful K-12 perspective on STEM education, and they assist with tasks that might not otherwise be accomplished. 


Project manager Karen Gareis said of the Fellows, "The Einstein Fellows are a really passionate, engaged group of STEM teachers to begin with, and it was amazing to see how much they made of their opportunities in Washington and how much they were transformed by the experience."  



Einstein Fellows Make Great Gains in their Ability to Understand and Track National and International Education Issues  

Einstein Graph

N = 120-122 alumni and 2009-2010 Fellows

Note: The grey segment shows "before" scores, while the purple segment shows statistically significant gains (p < .05) from before to after program participation. "After" scores are significantly higher than those of finalists.  

Evaluation Spotlight

Ford Family Foundation Scholar Programspotlight

The Ford Family Foundation has contracted GRG to evaluate its Ford Scholar Program, which offers college scholarships to graduating high school seniors and community college students in Oregon and Northern California. In addition to the scholarship funds, Ford Scholars receive support through conferences, counseling, and on-campus resources. GRG is assessing the impact of the program through an online survey of alumni and fellows and focus groups with current scholars and their parents. The goal is to capture the benefits of the program, explore potential improvements, and contribute to the literature on college access.


WGBH Vietnam Open Vault

GRG has teamed up with WGBH's Media Library and Archives (MLA) department to conduct an evaluation of their Vietnam Collection on their Open Vault website. The online collection contains images, interviews, original footage, and stock footage from the WGBH 1983 landmark series; Vietnam: A Television History. Through surveys and interviews with college professors around the country, GRG is gathering overall impressions of the online collection and feedback about how the resources could be used in their teaching and research.


In Other News...

GRG Facilitated Discussion at Public Science Events Conference

GRG staff members Colleen Manning and Molly Priedeman, along with GRG consultant Karen Peterman, led discussions at the first ever International Public Science Events Conference, which took place the week of February 14th in Washington, DC. The first session, Knowing What Works: Festival Evaluation, was led by Ms. Manning, Ms. Priedeman, and Dr. Peterman. The second session, Strategies for Evaluating Science Cafes with Adult and Teen Audiences, was led by Ms. Manning, Dr. Peterman, and Susan Foutz of the Institute for Learning Innovation. In addition, Ms. Manning assisted NOVA/WGBH clients Rachel Connelly and Jennifer Larese in conducting a town hall meeting with the science café community focused on the current needs and future vision of the movement.

The Philadelphia Science Festival

GRG is actively involved in public science events and is currently conducting a multi-method process and summative evaluation to demonstrate the success of the Science Festival Alliance project. Science festivals were recently held in San Diego, CA; Philadelphia, PA; and Cambridge, MA. In the fall, there will be a science festival in Bay Area, CA.

Click here to read more about our evaluation here and the International Public Science Events Conference here

Thank you for reading our newsletter! For more information about our exciting work, check out our website,

Find us on Facebook  View our profile on LinkedIn