When President Obama took office a little more than a year ago, we noticed his interest in program evaluation. Now we have fresh evidence of it in his 2011 budget. AEA's Executive Director, Susan Kistler, has prepared an analysis of that budget, which we are sharing with you here. Evaluation is woven throughout the document. Here are the highlights.

 

Evaluation of specific Federal programs. This includes proposals to increase the number of effective teachers and principals; boost development of clean energy on tribal lands; improve services for seniors and people with disabilities; and promote wellness initiatives in the Federal workplace. Especially noteworthy is a request for $500 million to expand the Investing in Innovation Fund, to evaluate and expand proven models for achieving student success.

 

Terminations, reductions, and savings. The budget cites evaluations to support budget reductions in several programs, such as certain children and families services' job demonstration programs, rural health care services programs, and watershed and flood prevention programs.

 

Sweeping evaluation policy. Far and away, the most impressive evaluation message is a set of overarching policies found in an annex to the budget called Analytical Perspectives. The parts that are relevant to evaluation are the first two sections (labeled 7 and 8) in a chapter called "Performance and Management."

 

Section 7, Delivering High-Performance Government, provides a new look at performance measurement and substitutes a new performance management system that builds on but also replaces the PART system used by the previous Administration.

 

Section 8, Program Evaluation, provides detailed steps and explanations to implement the policies enunciated by OMB Director Peter Orzag in his October 7 memorandum and discussed in our October column. The introductory paragraph puts evaluation in the context of performance management. "Performance measurement is a critical tool managers use to improve performance, but often cannot conclusively answer questions about how outcomes would differ in the absence of a program or if a program had been administered in a different way. That is where pro­gram evaluations play a critical role."

 

The second paragraph is equally telling. "A central pillar of good government is a culture where answering . . . questions [about how well programs work] is a fundamental part of program design and where agencies have the capacity to use evidence to invest more in what works and less in what does not."

 

The budget allocates approximately $100 million to 17 agencies to conduct new evaluations with strong study designs that address important, actionable questions or to strengthen agency capacity to support such strong evaluations.

 

We cannot do justice to these new policies in this short article. I urge evaluators to read these two sections themselves. For the evaluators involved in the work of the 17 agencies named in section 8, my take on this is that the ball is now in your court. Actually, I think it is in all of our courts.

 

Go to the EPTF web page

 

 

"Their voices, shared with scholars who work as their partners to support the increase of social justice and human rights, are reflected in this shift to transformative beliefs to guide researchers and evaluators. The transformative approach to evaluation makes explicit the use of evaluation for the purpose of furthering social justice," says Mertens. "This focus in our theory and practice is consistent with the ethical codes that guide our profession. I am thrilled to be honored in this way and humbled to think of my small part in contributing to this critical aspect of theory in evaluation."

 

A prolific writer and author, Mertens served as AEA president in 1998 and helped found AEA's Diversity Initiative, its Graduate Education Diversity Internship Program (GEDIP) as well as the International Organization for Cooperation in Evaluation (IOCE). She has made lifelong contributions to evaluation theory through teaching, presenting and practice and has authored/edited several books, most recently Research and Evaluation in Education & Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods (SAGE, 2010), Transformative Research and Evaluation (Guilford Press, 2009) and Handbook of Social Research Ethics (with Pauline Ginsberg, co-editor, SAGE, 2009).

What a great kickoff for the 2010 Thought Leaders Discussion Series. We heard from Jane Davidson in January and Ricardo Millet in February. And for March? Rodney Hopson will be joining us in the Thought Leader's Forum the week of March 7-13, 2010.

 

Rodney Hopson has served on the AEA Board of Directors and was the founding chair of the Graduate Education Diversity Internship Program. He has helped the association and the field to explore issues related to i) research on evaluation, especially in a recent National Science Foundation study on logic model use (with Rosalie Torres), ii) study and application of program evaluation standards for the third edition of Program Evaluation Standards (with Don Yarbrough, Lyn Shula, and Flora Caruthers), and iii) understanding the role of culture and context within evaluation and related disciplines. Hopson is the Hillman Distinguished Professor in Sociolinguistics, Ethnography, and Evaluation, Department of Educational Foundations and Leadership in the School of Education, and faculty member in the Center for Interpretive and Qualitative Research, Duquesne University.
 
For those new to this program, the Thought Leader Discussion Series brings together members for an informal week-long asynchronous discussion around issues raised by a guest thought leader. The full exchange takes place electronically via the AEA Thought Leaders Forum (formerly the President's Forum) and postings may be checked at your convenience each day throughout the week.
 
How does one sign up for the AEA Discussion Series? This is a members-only opportunity and it is free to all members who wish to participate. To join the forum, sign on to the AEA website using your username and password, navigate over to the forums, and click on "Add/Change Subscriptions" near the top right of the forum page. 

Login at: http://www.eval.org

Nick York, head of evaluation for the United Kingdom Department for International Development (DFID) and chair of the Organization for Economic Co-operation and Development's Development Assistance Committee's evaluation network, studied economics at Cambridge and joined the UK government economic service in 1994. He spent 10 years on health systems reform in the UK, helping to develop the international evidence base for quality in health care. He joined DFID in 2004 where he has played a leading role in developing the UK government's evaluation policy stance and has helped to deliver major evaluations in areas such as budget support, gender, country programs, HIV and health. York has a special interest in developing international systems for rigorous impact evaluation and helped lead the Network of Networks on Impact Evaluation (NONIE) as well as 3ie, a new organization based in Delhi that funds new impact evaluations to help promote development effectiveness. He recently agreed to an interview as part of AEA's series of profiles of leaders in international evaluation.

York shares that he got into evaluation "quite by accident," that he currently is engaged in capacity building, and is intrigued with the real-world impact of evaluation.

"I moved into it from health systems reform and evidence-based health care. The UK Department for International Development (DFID) urgently needed to build up their evaluation capacity and hired me.

"Helping some of the poorest people in the world is pretty motivating, but the bit I would highlight is evaluation partnerships, across many countries and organizations. I am involved in the Organization for Economic Co-operation and Development's Development Assistance Committee's evaluation network (OECD DAC) and it is very satisfying to see things moving forward through a collective effort from many people."

York's work entails complex evaluations with multiple partners. While immensely gratifying, collaboration can also be extremely challenging - and insightful.

"Getting people from different backgrounds and disciplines to value each other's contributions can be challenging. People who work in evaluation, particularly in international development, are really passionate about their work and develop their expertise over many years.

Next steps, he says, are broadening the circle of engagement and the sphere of influence.

"I am most interested in how we can learn from others. For example, what do the experiences from evidence-based health care and medicine have to tell us in international development? How have others managed to measure policy dialogue and influencing? How can we promote the impact of evaluation evidence on decision making?"

This is one in a series of interviews AEA is conducting with leading figures in the international evaluation community. Many thanks to AEA's International Committee for its help with these interviews.

Go to the full interview

 

Over the past several months, AEA has made great strides in keeping up with new technological developments and enhancing our online community and overall member connectivity. To name a few, this includes our aea365 tip-a-day blog (discussed last month), webinar demonstration series, thought leader discussion series, evaluation headlines, and our always-expanding online message forums.

 

To improve accessibility to our expanded content offerings, I am happy to say that AEA has recently upgraded the navigation of our website to make site-browsing more user-friendly. Particularly, the new layout's headings are now topically-based, which makes finding information much more intuitive. For example, we've changed the "Publications" heading to "Reading" and include more of our community-driven content (e.g., public eLibrary, aea365, etc.). Secondly, we've implemented a similar modification, changing the "Training" heading to "Learning", which now encompasses not only our physical learning venues, but our virtual ones as well (e.g., webinar demonstration series, thought leader discussion series, etc.). Thirdly, all of our general online community information (e.g., EVALTALK discussion list, groups and forums subscriptions, LinkedIn group, etc.) can be found within the "Community" header. This makes finding and connecting to our networking communities much easier. Fourth, in case you're starting your career as an evaluator, or wishing to find one, our "Career" and "Find an Evaluator" tabs are more straight-forward for your respective needs. Lastly, our new "Members Only" heading is now essentially a one-stop-shop for all things membership-related. This includes not only membership management information, but also gives direct access to our exclusive members-only benefits.

 

While these may be minor changes, I'm sure they will be significant in enhancing your overall browsing experience. Please stay tuned as we progressively improve our website and membership offerings. In the meantime, if you have any questions, suggestions, or concerns, please do not hesitate to contact me at marcus@eval.org.

Senior AEA member Sandra Mathison and her colleague E. Wayne Ross are founding co-editors of an online journal, Critical Education. AEA members Melissa Freeman, Saville Kushner and Linda Mabry are part of the journal's editorial team. Critical Education is an international peer-reviewed journal focusing on critical examinations of contemporary education contexts and practices. Critical Education is interested in theoretical and empirical research as well as articles that advance educational practices that challenge the existing state of affairs in society, schools, and informal education.

 

AEA members may be particularly interested in the journal's very first article, as it focuses on High Stakes Testing, an issue tackled by AEA via the association's public statements on Educational Accountability (2006) and High Stakes Testing (2002). The premier article in Critical Education is "The Idiocy of Policy: The Anti-Democratic Curriculum of High-stakes Testing" by Wayne Au. Au is assistant professor of education at Cal State University, Fullerton and author of Unequal By Design: High-Stakes Testing and the Standardization of Inequality (Routledge, 2009).

 

Notes Mathison, "We saw a need for an open access journal that encourages a wide range of critical research perspectives in education and while this isn't an evaluation journal, I believe many of the topics explored will be of interest to evaluators working in educational contexts."

 

To receive notification of new content in Critical Education, sign up as a journal user on their website.

 

Go to the Critical Education website

The Eastern Evaluation Research Society (EERS) will hold its 33^rd annual conference April 18-20, at the Seaview Resort and Spa in Galloway, NJ (just north of Atlantic City). This year's theme is Expanding the Evaluator's Toolkit. The keynote speaker will be Leslie J. Cooksy, AEA's 2010 President and Founder and Director of the Interdisciplinary Graduate Program in Evaluation at the University of Delaware. Pre-conference workshops will be led by Susan Berkowitz, Westat, Qualitative Data Analysis with NVIVO and Joy Quill, CJ Quill & Associates, Effective Writing for Evaluators. A Sunday evening interactive and networking session will be led by John Kelley, Villanova University.

"This year's conference will be the best ever," says Jill Feldman, current EERS co-president, "for gaining new skills and knowledge in an intimate and welcoming atmosphere, engaging in cross-disciplinary dialogue, and rubbing elbows with national evaluation experts in an informal setting."

For best rates, register by March 1. For more information, visit the EERS website.

Go to the EERS Website

Looking for ways to get involved in the life of the association? AEA's new Member Involvement Initiative (MII) has the following updates related to new volunteer opportunities within AEA: 

Evaluation FAQs Working Group: AEA would like to develop short, straightforward answers to Frequently Asked Questions such as "What is evaluation, What does an evaluator do, and Why is evaluation useful?" This is intended for use on the AEA website, in press releases and in materials used internally but also externally with non-practitioners who might be potential users, partners or simply have an interest in better understanding evaluation and evaluators. Our working group will meet monthly by phone May-October with ongoing email exchange around draft language. We envision the participation of a small group of seasoned veterans, as well as newcomers to the fold. Those with expertise in communications are especially valuable in understanding the goals of the initiative, translating complex information for public consumption, and understanding and appreciating the nuances of the field. If you are interested in serving in this capacity, please contact AEA Communications Director Gwen Newman at gwen@eval.org. (Deadline: March 15)

Evaluation 2010 Proposal Reviewers: We'll receive over 1000 proposals to present at Evaluation 2010! The proposals are reviewed by teams from AEA's 40+ Topical Interest Groups (TIGs). If you would like to be considered to be a conference proposal reviewer, please do either (but not both) of the following:

1. If you are submitting a proposal, there is a space right on the proposal submission form to indicate that you would like to be considered to be a reviewer and to indicate the TIGs for which you are interest in reviewing; OR
2. New! If you are not submitting a proposal, then send an email to Heidi in the AEA office at heidi@eval.org with the subject line "AEA Proposal Reviewer" noting that you are interested in reviewing and identifying up to two TIGs for which you would like to be considered as a reviewer.

The list of TIGs may be found online here. Please note that taking either approach will result in your name being added to a list of potential reviewers that is supplied to the TIG Program Chairs. The TIG Program Chairs will contact potential reviewers at their discretion. Each TIG's review process is slightly different, but all conduct their review during the month of April. When/if a Program Chair reaches out to you, she or he will include information about the expectations and timeline and will reconfirm your interest and availability. (Deadline: March 19)

International Silect Auction Coordinator: Each year at the annual conference, the International and Cross-Cultural (ICCE) TIG hosts a silent auction, the revenue from which benefits a fund to assist evaluators from developing countries and countries in transition to attend the annual meeting. After many years of dedicated service, our previous auction coordinator is stepping down and we're seeking a new coordinator to fill her shoes. The coordinator (a) solicits donations prior to the event through postings to discussion lists, outreach to TIG members, and her or his own professional network, (b) maintains a list of committed donors, (c) sets up and oversees the auction on site at the conference with the help of volunteers, and (d) manages the checkout on site and hands off the total collected to AEA staff. The total time commitment is approximately 5 hours during the fall before the event and an additional 7 hours on site at the conference. Please email ICCE TIG Chair Tessie Catsambas at tcatsambas@encompassworld.com, if you are interested in serving in this capacity. (Deadline: April 1)

What's new this month in the AEA Online Career Center? The following positions and Requests for Proposals (RFPs) have been added recently:

Descriptions for each of these positions, and many others, are available in the AEA Online Career Center. According to Google analytics, the Career Center received close to 4000 unique visitors in the past month. It is an outstanding resource for posting your resume or position, or for finding your next employer, contractor or employee.
 
Job hunting? You can also sign up to receive notifications of new position postings via email or RSS feed.
 
Go to the AEA Online Career Center

 

The American Evaluation Association's mission is to:

email: info@eval.org

phone: 1-508-748-3326 or 1-888-232-2275

web: http://www.eval.org