AEA's International Listening Project

Request for Proposals

Proposal Due Date: Monday, May 2, 2011

Introduction

In November of 2010, a task force was established by the AEA Board of Directors to develop strategies for learning what appropriate roles might be for AEA in the international community. The AEA International Listening Project Task Force seeks to learn how AEA can engage in ways that are mutually beneficial to the association and entities internationally. AEA currently designates a representative to the International Organization for Cooperation in Evaluation (IOCE), provides joint membership with the Canadian Evaluation Society (CES), and supports an International and Cross-Cultural Topical Interest Group. In 2010, about 15%, or about 900 AEA members, resided outside the United States, and in the same year, AEA welcomed about 8% or 220 international conference registrants to its annual meeting. The goal of the International Listening Project is to develop AEA policies that clearly articulate the role the association wants to play in the international community of evaluators and evaluation users in accordance with AEA's Goals Policies (see Appendix A: 2010 Goals Policies Section 1.4).

Purposes

This activity, AEA's International Listening Project, is aligned with AEA's mission. Specifically, the activity reflects AEA's commitment to valuing a global and international evaluation community and understanding of evaluation practices (Value iii). The project also builds on previous work conducted by AEA's International Committee (see Appendix B: Proposed Relationships and Responsibilities - Goals and Policies Specific to International Issues, 2009). The goal of this project is to advance AEA's development of policies that are in accordance with AEA's mission and values.

This project should embody AEA's intention to "listen" to others among international communities. AEA values its relationship with partners in a global community and believes that AEA will only benefit by listening well to the perspectives, experiences, and needs of its partners. Finally, we anticipate that in the process of developing policy guidance, the project will also identify specific activities. These should be captured and will be shared with AEA's Executive Director. The policy guidance should be in the form of key principles, themes, and criteria for future internationally-focused work.

What follows are suggestions for the shape and content of this project, based on current understandings of the international evaluation domain by the AEA Board. Other approaches to this project are welcomed.

Participants

The project will solicit the perspectives and advice of a wide range of AEA stakeholders. The individuals and organizations to be consulted include, in priority order:

1.     Leaders of other evaluation associations

2.     Members of AEA who live outside the United States

3.     Members of AEA who live in the United States but do international work

4.     Leaders responsible for evaluation in multilateral and bilateral agencies

5.     Representatives of foundations interested in evaluation, evaluation capacity building, and
      evaluation in international contexts

6.     Members of AEA who only work within the United States

A preliminary list of likely organizations to engage is listed below. Additional associations, organizations, and funding agencies should be identified and included.

• African Evaluation Association (AfrEA)
• Australasian Evaluation Society (AES)
• Canadian Evaluation Society (CES)
• Community of Evaluators (COE) (South Asia)
• European Evaluation Society (EES)
• International Development Evaluation Association (IDEAS)
• International Organization for Cooperation in Evaluation (IOCE)
• International Program Evaluation Network (IPEN) (Russia, NIS)
• Network of Networks of International Evaluators (NONIE)
• Latin American Evaluation Network (EvalNet)

• Organization for Economic Co-operation and Development (OECD)/Development Co-operation Directorate (DAC)

• Red de Seguimiento, Evaluacion y Sistematizaci�n en America Latina y el Caribe (ReLAC)
• United Nationas Evaluation Group (UNEG)
• United States Agency for International Development (USAID)
• World Bank Evaluation Cooperation Group (ECG)
• The Ford Foundation
• The Bill & Melinda Gates Foundation
• The David & Lucile Packard Foundation
• The Rockefeller Foundation

Activities 

1.   Planned interviews with specific individuals

Specific individuals will be selected to provide a range of views and to ensure relevant organizations are engaged. These individuals will include representatives of other associations, organizations, and funding agencies. Interviewees may include follow up with individuals who post insights or comments on the AEA websites (below). A set of semi-structured interview questions should be developed based on a list of possible AEA goals. The interviews would be conducted by phone, virtually, or in person where possible.

2.   Web-based feedback

The project should facilitate generating feedback through a web-based platform; feedback should also be made transparent as much as possible. There are at least two options: 1) Visibly sharing the feedback, as occurs in comments on a blog that allows synergies to emerge between ideas and issues. (This option could be facilitated with the assistance of AEA's staff via a blog.) 2) Soliciting feedback which is only shared when it has all been received (similar with approaches to reviewing AEA policy statements).

3.   Other

Respondents to the RFP may offer suggestions of other effective and economical ways to solicit feedback in accordance with the overall purposes of the project.

The information gathering and analysis processes should accommodate other languages (French, Spanish) besides English.

Analysis

Basic descriptive analysis of interview data and documentation may be conducted. The goals listed in previous documentation (see Appendixes) may provide an initial set of categories for clustering the data, and should be revised as needed. Data in these categories could then be summarized, with analysis of any patterns in the responses in terms of sources of comments or positions on particular issues, and illustrative comments.

Product

The project will result in a recommended set of principles, strategic directions, and goals for AEA's international presence and roles that AEA could pursue. One principle, for example, could be "AEA should partner with at least one other organization in all of its international activities." And one strategic direction could be "AEA should enhance the professional development opportunities for evaluators around the globe, especially in areas under-served by extant opportunities." The recommendations should represent highest priorities, rather than an exhaustive catalogue of possibilities, and may reflect the standpoints of targeted individuals, organizations, or other entities. All recommendations should be congruent with AEA's mission, vision, and values. Advantages and limitations of each recommended principle, strategic direction, and goal should be identified, including benefits to the AEA as an association, benefits to partnering entities, political consequences, industry and peer trends, technology, costs, and financial sustainability. Ideas for specific activities may emerge during the data collection process, and these can usefully be summarized; however, operational-level activities are not the focus of this project.

The final report from this project will be used primarily by the AEA Board of Directors to determine the association's future priorities in the international domain. A written narrative report (20 pages maximum) will be the major expected product. Presentation slides or some alternative format for communicating findings are the second product.

Who will conduct this "listening project"?

An evaluation researcher will be contracted to conduct the project. The researcher should be well-positioned to represent AEA, should have knowledge of and contacts with relevant societies and associations, and should have access to a wide variety of partners. Experience in interviewing and qualitative analysis is also required.

The International Listening Task Force (Patricia Rogers, chair, Tristi Nichols, and Victor Kuo) will appoint an International Listening Project Oversight Task Force from the membership of AEA (3-4 persons). The Oversight Task Force will be responsible for reviewing proposals and recommending one to the board, for direct communications with the contracted researcher during the project, and for reviewing a draft of the final report. One or more members of the Board's International Listening Task Force will also serve on the Oversight Task Force to ensure common understandings are enacted and to facilitate communications. The International Listening Task Force more broadly will serve as the liaison between the contracted researcher and the Board. The AEA Board is ultimately responsible for approving both the contracted researcher and the final report.

Timeline

• May 2, 2011: Proposals Due
• May 25, 2011: Consultants reviewed and selected (Board approval required)
• August 15, 2011: Draft report and presentation materials due to Board 
• September 1, 2011: Board review and approval of draft materials due back to Consultant 
• September 30, 2011: Project Completion
• November 2-5, 2011: AEA Think Tank Session (proposed by AEA Task Force) 

Budget

The maximum expendable under this contract is $8,000.

Instructions for Submitting Proposals

Proposals are due by Monday, May 2, 2011. Submit an electronic version of the proposal, as a single file including CV(s), to Heidi Nye at the American Evaluation Association: heidi@eval.org

Proposals should be no longer than 5 pages and include:

• Brief narrative describing approach and methods
• Task plan and timeline
• Budget with budget narrative

CVs of key consultants may be included as appendices in addition to the 5 page-narrative. No other appendices or approaches to extending the length of the proposal will be accepted.  

Proposals that do not follow the submission instructions and length restrictions will not be reviewed.

Questions

Questions regarding this the substance of this proposal should be directed to AEA President Jennifer Greene at jcgreene@illinois.edu; questions regarding the submission process should be addressed to Heidi Nye in the AEA office at heidi@eval.org.

Appendix A: AEA Goals Policies 2010-2011, Section 1

I. GOALS POLICIES - AS APPROVED JULY 2010:

1. Evaluators: AEA will provide and increase access to resources, and contribute to communities, that enable and support evaluators to:

A. become knowledgeable, effective, culturally competent, and ethical professionals;

i. provide resources that support rigorous education regarding history, methods, and theories of

evaluation

ii. provide resources for teaching evaluation standards, ethics and culturally responsive practices

B. use a multicultural lens to inspire excellence and rigor in evaluation theories, methods, applications,

and practices, specifically to:

i. expand our understanding of multiculturalism and enhance our ability to confront oppression in

various forms.

ii. promote international solidarity among evaluators and openness to additional diverse social,

cognitive, and political perspectives that can influence how we think about and practice

evaluation.

C. develop, disseminate, and transfer knowledge about evaluation;

D. engage diverse communities in evaluation practice;

E. use culturally responsive evaluation models to contribute to inclusiveness in society and to enhance

social justice and equity for persons of color and others from underrepresented groups;

F. contribute to building evaluation capacity within the communities and organizations in which they

work;

G. facilitate meaningful feedback mechanisms from evaluation consumers/public at all stages of

evaluation;

H. engage in the field and profession of evaluation and in the life of the association;

I. engage in other fields and associations that are related to or aligned with the field of evaluation;

Specifically, AEA has taken steps towards achieving the above Goals Policies through the following

activities and programs:

1.1. Annual Conference: AEA will have an Annual Conference

1.1.1 Content for the Annual Conference is primarily identified through a member-driven peer review process

1.1.2 Content for the Annual Conference is also identified by the President or her or his Designee for a thematically-focused Presidential Strand

1.2. Awards: AEA will have an Awards Program

1.2.1 The AEA Awards Program will recognize excellence in the field of evaluation.

1.3. Diversity Focused Programs: AEA will offer diversity-focused programs and services

1.3.2 AEA will offer a Graduate Education Diversity Internship Program

1.3.1 AEA's diversity-focused programs will be aimed at expanding the diversity of the membership and those practicing and teaching in the field. 

1.3.2 AEA will offer a Graduate Education Diversity Internship Program 

1.3.3 AEA will offer a range of other diversity-focused programs for students and professionals

1.4 Internationally Focused Programs: AEA will internationally-focused programs and services

1.4.1 AEA's internationally-focused programs will be aimed at supporting our international members, evaluators working in international contexts, AEA's networking with other associations, and the field as it expands internationally

1.4.2 AEA will designate a representative to a major international organization

1.4.3 AEA will offer a joint membership program with the Canadian Evaluation Society

1.4.4 AEA will offer a range of other internationally-focused programs

1.5 Journals and Peer-Reviewed Content: AEA will offer peer-reviewed content

1.5.1 AEA will offer New Directions for Evaluation as part of membership

1.5.2 AEA will offer the American Journal of Evaluation as part of membership

1.5.3 AEA will take a variety of approaches to increasing access to other peer-reviewed content.

1.6 Practitioner-focused content: AEA will offer practitioner-focused content

1.6.1 AEA will offer association-sponsored publications with practitioner-focused content

1.6.2 AEA will increase access to other materials including electronic resources for evaluation practitioners

1.7 Professional Development: AEA will offer professional development

1.7.1 AEA will offer professional development workshops as part of the annual conference,

1.7.2 AEA will offer an Evaluation Institute

1.7.3 AEA will offer a range of other professional development opportunities.

1.8 Standards and Principles for the Field: AEA will serve as a leader in the setting and vetting of Standards and Principles for the field

1.8.1 AEA will develop and maintain Guiding Principles for Evaluators

1.8.2 AEA will send a Representative to the Joint Committee on Standards for Educational Evaluation (JCES)

1.8.3 AEA will be involved in the vetting process for new standards in the field

1.8.4 AEA will also support other programs that promote and advance ethical practice in the field.

Appendix B: AEA International Committee Report

AEA International Committee's proposed statement for consideration by new AEA committees related to RELATIONSHIPS AND RESPONSIBILITIES - GOALS AND POLICIES SPECIFIC TO INTERNATIONAL ISSUES[1]

INTERNATIONAL GOALS

AEA commits to promoting and supporting, in the United States and internationally:

1.     recognized evaluation principles, concepts, methods, and techniques;

2.     improvements and refinements of existing and emerging evaluation concepts, methods, and techniques

3.     mutual learning related to the theory and practice of evaluation for multiple purposes in a diversity of contexts.

4.     strong international evaluation organizations and networks

5.     the needs of the AEA's international membership

DESIRED RESULTS

These goals are intended to ensure:

• mutual learning and sharing of knowledge and experiences;
• professional and personal development;
• evaluation capacity building and training;
• business opportunities and consulting;
• fostering and developing relationships;
• networking and collaborating;
• the needs of AEA's international members are considered and reflected in AEA's policies and practices
• international organizations' growth and development; and
• fun!

These goals will be demonstrated by significant progress in the following areas:

1.      Professional Development of Evaluators:

AEA members within the United States who:    

a)      are more knowledgeable about evaluation principles and practices outside the United States;

b)      create and participate in opportunities for mutual learning and sharing of knowledge with evaluators and evaluation organizations globally;

c)      support the creation or strengthening of evaluation organizations or groups in other countries; and

d)      are aware of individuals and organizations outside the United States with evaluation expertise that can be useful to their professional needs and facilitate communication with such individuals/organizations.   

AEA members from other countries and other evaluators throughout the world who:   

a)      are even more competent, effective, and ethical;

b)      develop, disseminate, and transfer knowledge about evaluation with evaluators in the United States and other countries;

c)      engage with and contribute to inclusive, diverse, and international networks and communities of evaluation practice;

d)      understand their role in building evaluation capacity in their nations, communities, and the organizations within which they work; and

e)      access and use high-quality resources, including those of AEA.

The AEA's contribution will be to:

maintain, develop, facilitate and support relationships among members of the AEA, in the United States and elsewhere, that promote collaboration, partnership, experience and understanding of evaluation at a global level.

2.      Institutional Capacity

Decision-makers and policy-makers in the United States and elsewhere will:
have a basic knowledge about multiple ways of doing and using evaluation;

a)      hold positive views about the role evaluation can play;

b)      have evaluation policies in place;

c)      use well-known, respected resources, including those of AEA, to support their efforts; and

d)      make evaluation a standard business practice in their organization's operations.    

The AEA's contribution will be to:

i.    maintain, develop, facilitate, and support relationships with and among a variety of organizations (international, national and regional membership associations, consortia of organizations) that promote high-quality evaluation standards and practices; and

ii.    assist these organizations, as appropriate, to develop or promote effective evaluation and its continual improvement within their spheres or sectors.

3.      Public Awareness:

The general public in the United States and elsewhere are:   

a)      aware of the concept of evaluation;

b)      understand the value of evaluation;

c)      understand that evaluation can improve programs and policies in a variety of settings around the world;

d)      expect/demand that programs be evaluated; and

e)      be better consumers of evaluation information.

The AEA's contribution will be to:

i.   provide access to educational materials and in other ways make available to the public opportunities to learn about sound evaluation principles, concepts, methods, and techniques; and

ii.  offer access to individuals and organizations available for networking.