Banner600
In this issue...
  • Cuts Proposed
  • Full-Text Access
  • Silent Auction
  • A View of Denver
  • Overheard
  • Get Involved

  • AEA Newsletter
    March 2008

    Dear AEA Colleagues,

    I wanted to give you an update on the work of our Evaluation Policy Task Force, which is helping AEA to develop an ongoing capability to promote sound federal evaluation policy. What exactly is evaluation policy? For the EPTF, of which I am the chairman, our working definition is: "Any rule or principle that a group or organization uses to guide decisions about evaluation." These decisions include, but aren't limited to, how evaluation is defined, when and how it is employed, who participates in the process, and what resources are made available.

    In recent years, the federal government has launched several programs that seek to formally incorporate evaluation into its decision-making processes. Examples include the Government Performance and Results Act (GPRA), the Program Assessment Rating Tool (PART), the U.S. Department of Health and Human Services' 1 percent budget set-aside for many of its programs, and the U.S. Department of Education's program evaluation requirements. Even so, federal evaluation policy decisions are still made without systematic input and advice from the evaluation community. AEA is well positioned to be the source of such advice, and the EPTF's goal is to help us become precisely that.

    We will begin by talking with policymakers in both the executive branch and in Congress, and with non-governmental organizations that share our concerns. And we will scan legislative, regulatory and administration policies to identify opportunities for action as well as examples of successful high-impact evaluations that can serve as models for evaluation policy. Recently, EPTF members met with a group of

    The EPTF welcomes your thoughts and suggestions. You can e-mail the task force at [email protected]. More Information is available on-line at http://www.eval.org/EPTF.asp. We'll keep AEA members informed about new developments in federal evaluation policy. Below you'll find an article with two examples of how policy is being implemented.

    Sincerely,


    William Trochim,
    AEA President

    Cuts Proposed
    National Research Council notes inconsistencies in PART use

    The Bush administration recently proposed eliminating or decreasing the budgets of 151 federal programs in fiscal 2009, citing poor performance and redundancy, and basing its recommendations in part on results from its Program Assessment Rating Tool (PART). President Bush said the proposed cuts would save an estimated $18 billion and Jim Nussle, Director of the Office of Management and Budget (OMB), was quoted by GovExec.com as saying the targeted programs "are frankly just not achieving the results that they need to achieve." Congress still must weigh in on the recommendations. Of 141 cuts proposed for fiscal 2008, lawmakers enacted just 29. About 70 percent of this year's proposed changes were also part of fiscal 2008's budget recommendations.

    Meanwhile, a recent report from the National Research Council (NRC) criticized how OMB applies PART. In a review focusing on the use of PART to evaluate research programs at the Environmental Protection Agency (EPA), an NRC committee said OMB places too much emphasis upon "ultimate outcomes" and applies the tool inconsistently across agencies. The NRC report indicated that OMB's metrics to evaluate federal research programs were inadequate, focusing primarily on process efficiency. "Ultimate outcome-based efficiency metrics are neither achievable nor valid," and several other metrics could be used to measure the process efficiency of research programs, noted NRC. The report said OMB doesn't focus on investment efficiency metrics and warned that evaluation of research "should not over-emphasize efficiency," because it is only one part of a proper evaluation and because the primary goal of research is the development of knowledge.

    The NRC report indicated that OMB had rejected metrics proposed by EPA that were similar to those it had approved for use by other agencies, and had encouraged EPA to apply earned value management (EVM) to its programs when no other agency has used it to measure basic research. NRC urged OMB to apply the same efficiency standards to all agencies and to conduct oversight and training programs to make sure budget examiners apply PART consistently and equitably.

    "Both of these stories underscore the importance of government evaluation policies and the care with which they must be formulated and applied," says George Grob, President of the Center for Public Program Evaluation and a consultant to AEA's Evaluation Policy Task Force (EPTF). "AEA needs to be at the table when such policies are made." AEA President and EPTF Chair William Trochim agrees. "All evaluators should be concerned when respected organizations like the NRC raise critical questions about how evaluation is being accomplished and utilized." said Trochim. "Members of AEA's EPTF will be meeting with officials at OMB and others to discuss the concerns raised by the NRC and other organizations and consider how AEA might enhance the quality and use of evaluation in the federal government."

    Read the NRC report

    AJEcover Full-Text Access
    Check AJE references - Get free access to many SAGE resources

    One benefit of your AEA membership is online access to the American Journal of Evaluation (AJE). But did you know you can also access much of the research referenced in AJE? By clicking on the links in the reference section of each AJE article, you can follow the reference trail back to cited material. This includes full-text articles, government publications, and important databases - all without leaving the original article. This is made possible through a partnership between SAGE (which publishes the journal for AEA) and HighWire Press, a division of the Stanford University Libraries that hosts 1,095 journals from more than 130 major scholarly publishers - including 485 journals published by SAGE. Organizations that partner with HighWire Press agree to offer linking between their articles and related articles on the platform.

    The December 2007 issue of AJE, for example, included an article entitled Evaluations That Consider the Cost of Educational Programs by John A. Ross, Khaled Barkaoui, and Garth Scott from the University of Toronto. This article contains 61 references, 48 hosted on the HighWire platform. When you go to the article online and click on "references," you'll find several types of links. Some go to different databases such as Cross Ref, ISI, and Medline, while others are links to government reports or other freely available "non-published" documents such as conference proceedings. The ones marked "Free Full Text" in red type are articles accessible at no charge. In the article mentioned above, there are six references with "Free Full Text" links.

    Different publishers have different policies. Journals published by SAGE always offer free full-text online access through reference links to AJE members. AEA members also have free online access to two SAGE journals not sponsored by AEA - Evaluation Review and Evaluation and the Health Professions.

    To access AJE online, sign in to the AEA website at http://www.eval.org/ using your AEA username and password.

    • Your username is: Custom Field 1
    • Your password is: Custom Field 2


    auctionphoto Silent Auction
    AEA's successful fundraiser benefits international presenters

    What do a carved necklace from South Korea, barbecue sauce from Kansas City, and batik from Niger have in common? They were all among the 100-plus items donated for a silent auction held during Evaluation 2007 in Baltimore, Maryland. Contributions included a colorful assortment of autographed books, hand-knit scarves, hometown-favorite specialty items, handmade jewelry, artwork, and handicrafts from Sri Lanka and Indonesia. More than $6,000 was raised at the auction through the sale of goods from around the world and cash contributions. The event is coordinated by the International and Cross-Cultural Evaluation Topical Interest Group and proceeds fund AEA's International Travel Awards, which help subsidize travel to the conference by presenters from developing countries and countries in transition. Last year, awards went to three participants:

    • Natalia Kosheleva of Russia's Community Service School Programme (CSSP), offered a poster presentation on how evaluation of CSSP, which was implemented in six regions of Russia in 2004-2006, was used to enhance participants' learning from their experience.
    • Mohammed Javad Ahmadi of Afghanistan's Creative Associate International Inc. (CAII), took part in two presentations. One explored attention to variations in how local conditions, perspectives and resources contribute to the design of promising education decentralization initiatives in Afghanistan. Another focused on the country's new national in-service teacher training initiative.
    • Carolina Castrillo with Catholic Relief Services (CRS), made a presentation on the organization's efforts to institutionalize learning in El Salvador.

    If you know of an evaluator in a developing country, or a country in transition, who would benefit from attending this year's conference in Denver, please encourage him or her to review the 2008 AEA International Travel Awards Call for Proposals. And, by the way, it's not too early to start thinking about what you can bring to the 2008 auction!

    Go to the International Travel Awards details on the AEA website

    A View of Denver
    The Mile-High City as seen on the big screen!

    It's eight months away from Evaluation 2008 and you likely still have proposals to write and plans to make. But you can whet your appetite and get a glimpse of your destination through an assortment of movies that spotlight Denver, Colorado. Some are notable for story line; others for breath-taking scenery. If you want a better feel for what the city -and beyond - has to offer, it's as easy as a movie night at home!

    Filmed in and around Denver

    • About Schmidt (2002) is set against Denver and Boulder
    • Die Hard 2 (1990) shows Stapleton International Airport
    • Cliffhanger (1993) features aerial views of the Rockie Mountains, around Durango
    • Sleeper (1973) utilizes Denver's Church of the Risen Christ, Currigan Exhibition Hall, Denver Botanical Gardens, Genesee Park just west of Denver, Greenwood Village , Lakewood and the National Center for Atmospheric Research in Boulder
    • Things to Do in Denver When You're Dead (1995) Five Points in Denver, Rossonian Hotel, Casino Cabaret now Cervante's Masterpiece Ballroom

    New & Notable

    • Skills Like This (2007) The romantic comedy took home a Best of the Fest award at the 2007 Edinburgh International Film Festival and an Audience Award for Best Narrative Feature at the South by Southwest Film Festival. Directed by Denver resident Monty Miranda and produced by Academy Award-winner Donna Dewey for her locally-based production company, Dewey-Obenchain Films
    • Nowhereland (2008) Starring Eddie Murphy and to be released in September 2008

    Old Time Favorites/Award Winners

    • Around the World in Eighty Days (1956) Denver and Rio Grande Western Railroad in Durango
    • How the West Was Won (1962) Bent's Old Fort, Durango, Uncompaghre National Forest
    • Butch Cassidy and the Sundance Kid (1969) Silverton and Durango
    • True Grit (1969) Buckskin Joe Frontier Town & Railway in Canon City, Castle Rock, Gunnison, Montrose, Ouray, Owl Creek Pass, Ridgeway
    • Indiana Jones and the Last Crusade (1989) Cumbres & Toltec Railroad in Antonito
    • City Slickers (1991) Durango
    • Nurse Betty (2001) Durango


    Overheard

    Periodically, we check in on discussion at AEA's listserv, EVALTALK.

    Lori Friedman wrote: Are there any websites or other publicly available resources which contain listings or registries about tools? The only one I'm aware of is the FRIENDS compendium.
    http://www.friendsnrc.org/outcome/toolkit /annot.htm

    Dave Colton at the Commonwealth Center for Children and Adolescents suggested: It depends on the purpose of the instrument and field of study. Here are some examples:
    http://buros.unl.edu/buros/jsp/search.jsp
    http://www.instrumentwizard.com/
    http://libraries.uta.edu/helen/Test&meas/testmainfram e.htm

    Tom Kelly of the Annie E Casey Foundation noted: While developing a cross-site survey for our Making Connections initiative, Chapin Hall Center for Children in Chicago compiled a data base of items/questions used in other related survey instruments (focused on children, families, community).
    http://tarc.aecf.org/initiatives/mc/mcid/index.php

    Jan Hense at Ludwig-Maximilians University Munich shared: These are the directories I know of. Most of them are confined to a specific domain though:
    http://oerl.sri.com/instruments/instruments.html
    http://www.tmg- web.com/modules/eval_mods_main.htm
    http://www.lib.vt.edu/help/instruct/evaluate/evalbiblio.ht ml#forms

    Roger Phillips, Independent Consultant added: You might want to check "Early Childhood Measures Profiles" from Child Trends (Berry, Bridges & Zaslow, 2004). Be advised: it's an enormous document. http://aspe.hhs.gov/hsp/ECMeasures04/report.pdf

    We have compiled and annotated the listings, along with a few others, on the AEA website at http://www.eval.org/Resources/instruments.asp. Please take a look at this new resource and recommend any updates, changes, or additions to our website coordinator at [email protected]. EVALTALK is a vibrant community of more than 2000 subscribers. If you have not already, we invite you to join.

    Go to the Evaltalk information page to subscribe

    Get Involved
    Get the most out of your membership

    A short list of the many things to do right now to participate in the life of the association. Please click through to the appropriate item below.

    We'll have more to share over the coming months.

    AEA Site Links
  • AEA Home
  • TIGs
  • Affiliates
  • Board
  • About Us

    The American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of evaluation in all its forms.

    The American Evaluation Association's mission is to:

    • Improve evaluation practices and methods
    • Increase evaluation use
    • Promote evaluation as a profession and
    • Support the contribution of evaluation to the generation of theory and knowledge about effective human action.

    phone: 1-508-748-3326 or 1-888-232-2275