News from The American Evaluation Association

Newsletter: April 2014

Vol 14, Issue 4

Message from the President:
Networking for the Common Good

Dear AEA Colleagues,

Last month, I drew attention to AEA's Topical Interest Groups (TIGs) and how they are an important way for members to connect and learn. This month, I'd like to draw your attention to another valuable avenue for building connections: Local Affiliates.

Local Affiliate organizations have no legal or financial ties to AEA, but they are aligned with the mission of AEA, and the relationship between Local Affiliates and AEA is mutually beneficial.

Some of the Local Affiliates predate AEA itself. For example, the Eastern Evaluation Research Society (EERS), an AEA Local Affiliate, is the oldest professional society for program evaluators in the United States.

Earlier this month, I had the privilege of participating at the annual conference of EERS. In my comments at the conference, I reflected on the connection between the EERS theme of Balancing Rigor and Resources and the AEA 2014 conference theme of Visionary Evaluation for a Sustainable, Equitable Future. This is one example of the opportunities we have to build connections between Local Affiliates and AEA and look at how we are individually and collectively shaping the evaluation field.

Local Affiliates are 100 percent standalone organizations. They establish their own criteria for membership, and determine their own dues and procedures for maintaining their finances and their data. Affiliates differ from each other, not only in size, but in focus, target audience, and main activities. The number of affiliates has increased from about a dozen in the mid-to-late 1990s to 28 today.

Did you know that there is also a Local Affiliate Collaborative (LAC)? Celebrating its 10th anniversary this year, the LAC is the most visible and lasting product of a 2004 grant from the W.K. Kellogg Foundation to help AEA and Local Affiliates do joint projects, share information, and support one another. Tom Chapel, a longtime AEA member and convener of the Atlanta-Area Evaluation Association (AaEA), led this grant and continues to be a leader in the LAC Steering Committee.

The LAC Steering Committee — a core of about 10 LAC zealots — meets by phone most months to work on products and events to support affiliates. The LAC is in the midst of producing a set of short "Tip Sheets," covering topics from recruitment to strategic planning. They have also produced tool kits on organizing conferences and starting new affiliates.

You don't need to be an AEA member to be in a Local Affiliate, and the Local Affiliates do not need to be composed of AEA members in order to connect with AEA. Groups can petition to become a Local Affiliate by submitting a proposal to AEA's executive director. Please contact her for details on the information she needs.

I'd love to hear from you about how you view the value of the Local Affiliate-AEA connection.

Warm regards,

Beverly Parsons

AEA 2014 President

In This Issue

2014 Summer Institute

Quick Links

Topical Interest Groups

Affiliates

Board

Important Note

To ensure this newsletter reaches you every month, add info@eval.org to your email contacts!

Join our Mailing List!

Register Now: 2014 Summer Institute, June 1-4 in Atlanta

Registration for the 2014 Summer Institute is open! Join the American Evaluation Association June 1-4, 2014, in Atlanta for this year's Summer Institute. Evaluators, applied researchers, grantmakers, foundation program officers, nonprofit administrators, and social science students are all invited to attend. Sessions are filled on a first-come, first-served basis. View session availability.

The Summer Institute will include three keynote addresses, five rotations of three-hour and 40-minute training sessions, plus two group lunches to allow for networking among conference attendees. View the agenda-at-a-glance.

Presenters include experts who have conducted evaluations in a variety of settings, nationally known authors, those working on the cutting edge, evaluation experts, and outstanding trainers. View the workshop and course descriptions.

AEA Values - Walking the Talk with Leah Neubauer

Are you familiar with AEA's values statement? What do these values mean to you in your service to AEA and in your own professional work? Each month, we'll be asking a member of the AEA community to contribute her or his own reflections on the association's values.

AEA's Values Statement

The American Evaluation Association values excellence in evaluation practice, utilization of evaluation findings, and inclusion and diversity in the evaluation community.

i. We value high quality, ethically defensible, culturally responsive evaluation practices that lead to effective and humane organizations and ultimately to the enhancement of the public good.

ii. We value high quality, ethically defensible, culturally responsive evaluation practices that contribute to decision-making processes, program improvement, and policy formulation.

iii. We value a global and international evaluation community and understanding of evaluation practices.

iv. We value the continual development of evaluation professionals and the development of evaluators from under-represented groups.

v. We value inclusiveness and diversity, welcoming members at any point in their career, from any context, and representing a range of thought and approaches.

vi. We value efficient, effective, responsive, transparent, and socially responsible association operations.

See AEA's Mission, Vision, Values

I am Leah C. Neubauer, program manager and instructor in the Master of Public Health Program at DePaul University. I am the president of AEA's Chicagoland Local Affiliate and an active member of the Local Affiliates Collaborative (LAC) Steering Committee.

AEA's vision and values ground and shape my evaluation practice as an evaluator and educator. From my adult education doctoral training, I offer up the notion that our philosophies of evaluation practice are influenced and shaped by our personal and professional values. As such, my insights are anchored around my youth, my first evaluation job, and my AEA experiences.

While my 1980s childhood lacked fantasies of an evaluation profession future, my family upbringing — oldest child in a large multigenerational, multiethnic, social-change-minded family — had me in a constant state of assessment and evaluation. My family's desire to discuss opinions on everything provided foundational training for many useful evaluation skills and guiding values, such as: active learning, project management, credible summary statements (at the dinner table), robust stakeholder engagement, and, quite specifically, inclusion of multiple voices in everything (all the time!).

From these childhood moments, I formally found evaluation through urban HIV/AIDS activism and community-based programming in the late 1990s. I moved to Chicago for university and volunteered with a Latino-serving community-based HIV/AIDS organization. Knowing that data and outcomes were often the deciding factor between being funded or not, I first witnessed the power of evaluation. I was doing evaluation by day and spending my evenings as an HIV/AIDS outreach educator focused on behavior change. I started thinking about the role of learning and evaluation; I became quite interested in evaluative learning and capacity building. So while working with evaluation team members on "high-quality evaluation," I became quite serious about contributing to evaluative learning, organizational capacity-building, and community enrichment in the same places we conducted evaluation. I believed that evaluation efforts could help keep doors open, while also aiming to enrich the lives of all individuals, organizations, and communities involved.

From these community-based roots, I found AEA via Portland in 2006. My mentor encouraged me to attend given my growing interest in HIV/AIDS and evaluation. I assumed that being an evaluator meant going to AEA, and I also knew that "the famous community psychologist Robin Miller" was a part of the AEA. I had to be there. I journeyed to my first conference. I was hooked in large part because of vision and personal/professional values alignment. It just felt right being at AEA. Folks at AEA cared about the same issues I did. Also, the people proved to be quite fun!

Since my initial AEA experience, I have witnessed values in action through local affiliate work, international evaluation efforts, education in the public health classroom, and AEA's Cultural Competence Dissemination Working Group. I find this to be a very exciting and evolving time in AEA, particularly for fellow self-identified evaluators for social change. I welcome my seat at the AEA member table, enjoying and learning from the AEA company who clearly embodies a lifelong commitment to the vision and values being fully realized.

Face of AEA - Meet Michele Tarsilla

AEA's more than 7,800 members worldwide represent a range of backgrounds, specialties, and interest areas. Join us as we profile a different member each month via a short question-and-answer exchange. This month's profile spotlights Michele Tarsilla.

Name: Michele Tarsilla

Affiliation: Chair of the International and Cross-Cultural Evaluation TIG; Vice President of the Evaluation Capacity Development Group (ECDG)

Degrees: Ph.D., Interdisciplinary Evaluation, Western Michigan; Graduate Diploma, Public Policy and Program Evaluation, Carleton University; M.S., Foreign Service, Georgetown University School of Foreign Service; B.A., International Political Science, Catholic University

Years in Evaluation Field: 12

Joined AEA: 2008

Why do you belong to AEA?

I decided to join AEA after working in evaluation in Africa and Latin America for a number of years. Becoming part of a professional evaluation association proved quite beneficial to me from the very beginning. On the one hand, I learned about cutting-edge evaluation theories and methods that rapidly helped me refine my own evaluation practice. On the other hand, I had the unique opportunity to exchange with and learn from seasoned evaluation professionals conducting evaluations in international and cross-cultural settings.

Over the years, I got increasingly involved in AEA activities and, by so doing, I became aware of two of the association's distinct features. First, the incredibly diversity of its membership: I learned about theories and methods of direct applicability to my international development evaluation assignments from AEA colleagues whose work was primarily domestic. Second, the continued encouragement for members to develop and disseminate innovative approaches and tools aimed at enhancing the contemporary evaluation practice. I myself had a chance to use the annual conferences and the AEA365Blog as a privileged platform from where to launch my Evaluation Capacity Development framework, discuss my book chapter with Donna Mertens on new trends in mixed methods evaluation, and present the findings of my research with Michael Bamberger on the measurement of unintended and unexpected outcomes in performance and impact evaluations.

What's the most memorable or meaningful evaluation that you have been part of?

This is a particularly challenging question, as every evaluation is unique and represents an enriching experience, both from a professional and personal standpoint. That said, of all the evaluations that I have worked on to this date, there is one that I still remember quite vividly: the summative evaluation of a micro-loan program implemented by a credit union in Ontario, Canada, and primarily aimed at marginalized population groups.

Based on the findings of this four-month evaluation, small-size loans granted to individuals (with either no or very low credit score) to start up a small business were effective in improving the economic conditions and social integration of both the borrowers and their respective households. This evaluation is particularly dear to me for two different reasons. First, it prompted me to develop and exert tactful responsiveness and culturally competence toward an unusually diverse sample of respondents in a very short period of time. Having to interview individuals with many different backgrounds, often with very limited language skills or low Internet savvy, and with little willingness to talk about their income and personal well-being with a stranger (me), forced me to think on my feet at all time and adapt my evaluation strategy to unforeseen circumstances throughout the evaluation. Second, the evaluation findings were successfully used to lobby the provincial government of Ontario as well other local foundations to sponsor similar micro-finance programs in the future, by also making additional resources available to make up for the risk of loan delinquency.

What advice would you give to those new to the field?

My first piece of advice is to look for as many and diverse opportunities as possible to practice evaluation. Some of my colleagues are of the opinion that, in order to succeed in getting evaluation jobs, you need to evaluate programs in only one area over and over so as to become a recognized expert in a specific field. I personally believe that content expertise needs to be matched with methodological versatility and that, therefore, being open to work in a variety of areas will equip you with a stronger methodological toolbox and allow you to land a larger number of assignments in the longer term. I myself have developed my evaluation practice across a large variety of settings over the last 10 years (from evaluating school programs in refugee camps in northwestern Kenya to assessing the gender-responsiveness of HIV and AIDS programs in the favelas of Rio de Janeiro, from developing a M&E framework for a new Decent Work Program in Agriculture in Malawi and Tanzania to delivering of workshops for commissioners, mangers and practitioners in over 25 countries).

My second piece of advice is to constantly look beyond your own area of practice and never stop searching for innovative evaluation methods and approaches used in other fields that might prove particularly adequate to address your evaluation questions. Such "evaluative curiosity," combined with the timely review of specialized literature (including open-source peer-reviewed journals) and periodic perusal of online evaluation exchanges sponsored by a variety of evaluation communities of practice, is likely to strengthen your professional skills. I myself am part of several online evaluation communities and, besides downloading those reports and tools advertised by them which I find the most suitable to my interests and needs, I actively contribute to some of their sponsored evaluation discussions from time to time.

My third piece of advice is to get used to filtering the incredibly large volume of evaluation-related information that one might get exposed to on a daily basis. In particular, it is critical to be able to assess both the strengths and limitations of those evaluation tools and methods which, despite being advertised as "hot" or "must-haves," might not be appropriate to address your specific evaluation needs.

Policy Watch - FY 2015 Guidance on Promoting Data for Evaluation

From Cheryl Oros, Consultant to the Evaluation Policy Task Force (EPTF)

The Obama administration has issued further guidance on developing and using data to improve government programs via evidence-based decisions. Two recent documents explain the importance and improvement of data use: the 2015 Budget fact sheet on building and using evidence and the Office of Management and Budget Memo (M-14-06) "Guidance for Providing and Using Administrative Data for Statistical Purposes."

Previously, we described the many investments in the administration's budget. These total billions in pilot and demonstration programs, outcome-focused grant reforms, and new strategies that pay only for empirically proven approaches. The budget also lists examples of programs to invest in, scale up, or change on the basis of strong evidence.

The OMB memo is designed to promote opportunities and tools for addressing barriers to agencies' access to and use of administrative data for evidence-building. The memo notes that high-quality and reliable statistics provide the foundation for the research, evaluation, and analysis that help the federal government understand how public needs are changing, how well federal policy and programs are addressing those needs, and where greater progress can be achieved. Such statistical information can be created more efficiently through greater use of information that the government has already collected for programmatic and regulatory purposes. Information of this kind is often called "administrative data."

The OMB memo encourages agencies to keep possible statistical purposes in mind in managing their administrative data. OMB further notes that agencies sometimes do not make full, appropriate use of non-public administrative data for statistical purposes, because they perceive the requirements and protections that apply to non-public data as being too complicated and burdensome to navigate.

The memo requests that agencies:

Foster greater collaboration between program and statistical offices; develop strong data stewardship policies and practices around the statistical use of administrative data; require the documentation of quality control measures and key attributes of important datasets; and require the designation of responsibilities and practices through the use of agreements among these offices.
Promote the use of administrative data for statistical purposes, in ways consistent with legal and policy requirements for such uses, including the need to continue to fully protect the privacy and confidentiality afforded to those providing data.
Use suggested "best practice" tools, including guidance on the interaction of Privacy Act requirements and use of data, as well as model interagency agreements for developing their policies and practices for sharing data with other agencies.
Report to OMB progress in implementing the Memo to identify any barriers to moving forward.

Also, as part of the president's Management Agenda and Open Data Initiative, agencies are encouraged to seek ways to open up federal data for private sector innovation and public use. The OMB guidance aims to help agencies access and utilize existing federal data to answer important questions about program outcomes. Many of these efforts are ongoing and involve leveraging existing data, including linking data on program participants to administrative data on earnings, health, or other outcomes.

Diversity - Applications for the 2014-2015 GEDI Cohort Now Open

From Zachary Grays, AEA Headquarters

The American Evaluation Association is proud to announce that we are now accepting applications for the 2014-2015 Graduate Education Diversity Internship! We had an overwhelming response from sites anxious to host interns, and we are looking forward to matching them with the next group of future evaluators. It has been a little more than 10 years since the GEDI program set out to provide a unique internship and training opportunity for graduate students studying evaluation. Over 50 interns later, and thanks to the support of countless host sites and volunteer program chairs, the GEDI program has become a pillar of diversity at AEA. Our current interns are coming to the end of their journey, excited to leave their mark on the evaluation discipline and deepen the profession's capacity to work in racially, ethnically, and culturally diverse settings.

The Graduate Education Diversity Internship, commonly referred to as GEDI, aims to expand the pool of graduate students of color and from other under-represented groups who have extended their research capacities to evaluation. Interns participate actively at their host sites during the academic year by applying skills acquired during classroom time to real-world evaluations. In addition to this invaluable experience on-site, interns meet both online and in person to build on their existing evaluation knowledge and skill set. Under the direction of Directors Dr. Stewart Donaldson and Dr. Ashaki Jackson and Program Liaison John LaVelle, this year's impressive interns — Crystal Coker, Shipi Kankane, Bailey Murph, and Anael Ngando — have spent the last nine months completing the rigorous body of work necessary to be recognized as successful GEDI graduates.

"What I like a lot about the GEDI program is that it forces you to think about issues of culture in evaluation," said Anael Ngando, GEDI intern with Education Development Center Inc. "All the aspects of the program (assignments, workshops, meetings, webinars, etc.) contribute to improving interns' understanding of the relevance of culture in evaluation. The GEDI program gave me real-life, hands-on experience on the process, challenges, and rewards of conducting evaluation."

In less than two months, our GEDI interns will conduct their final presentations and graduate from interns to GEDI alumni at the Summer Institute in Atlanta. It has been an absolute pleasure working with these very talented students this year, and we have no doubt they will make remarkable evaluators.

Applications are now available for the next GEDI cohort. I encourage you all to circulate the call to those who would not only make an excellent GEDI but have a passion for championing diversity and cultural competence in evaluation.