AEA's Evaluation of Transition to Policy-Based Governance

Request for Proposals 

Proposal Due Date: Friday, May 6

Background

In January 2009, the Board of Directors of the American Evaluation Association (AEA) began operating under a policy-based governance approach based on the Policy Governance model developed by John Carver (Carver, 2006). As described in materials developed by the Board Leadership at the time (such as Frequently Asked Questions:  The Move to Policy Governance, May 2009), the model was chosen among other reasons, because it was considered one of the more explicit models of governance, and had a strong emphasis on written and publicly available policies.

The need for a new model, as expressed in Board documents, was spurred by challenges created by the Association's fairly rapid growth in membership, broadening of programs and services offered, and nascent role in national policy and beyond. The challenges included increasingly more time spent by the Board and its committees on operational rather than strategic planning issues, concerns for the lack of explicit description of priorities and thus continuity in direction, lack of written policies and strategic plans, volunteer opportunities largely restricted to standing committees, and a sense that the committees and overall committee structure were not operating smoothly.

The adoption of the model was therefore intended to:

• provide a structure that increased the Board's focus on policy versus operations,
• support continuity of strategic direction and initiatives by the Board by providing a written basis for guiding the work of the Association,
• increase the transparency of Board work to the membership,
• increase the efficiency of the Board and its committees,
• provide greater opportunities for volunteer involvement and member engagement, and
• create a more nimble structure for Board support.

The AEA is seeking to engage an external evaluator to undertake an evaluation of this recent transition in governance. The evaluation is to focus on the effects to date of the transition throughout the Association, including the extent to which it is achieving the outcomes intended from adopting a policy-based approach to governance. The external evaluation will draw on available documentation related to the transition, to be provided by the AEA, that documents the reasons for the change, the overall design, and current status in implementation. Primary data collection will focus on assessing the outcomes to date from the new governance model.

Purposes of the Evaluation

The evaluation is primarily intended to inform ongoing decisions about governance structures and processes. Given concerns that have been identified by some members about the move to policy governance, the evaluation is also intended to ensure that members have an opportunity to raise their concerns and that these concerns will be examined and addressed, as fully as possible.

Key Evaluation Questions

Question 1: What have been the effects to date of the transition?  In particular, to what extent are the desired outcomes being achieved? These outcomes include (1) less Board time on operational issues and more Board time on policy and strategic planning; (2) more continuity in strategic planning and direction of Board initiatives; (3) more accountability of the Board; (4) greater transparency of Board actions; (5) wider range of opportunities for member engagement in AEA activities; (6) improved functioning of committees/Priority Area Teams (PATs), task forces, working groups; and (7) decreased AMC resources being spent to support these various groups. Furthermore, what unintended outcomes are resulting - both positive and negative?

Question 2: How well does the model of policy governance adopted by the AEA fit the current needs of the Association? How does the model compare to the models of governance being used by other professional associations?

Question 3: What are the possible implications of these evaluation findings for AEA?

Implications may include options presented for consideration by the Board, President, leaders of PATs and TIGs, and Executive Director that would enhance effective, efficient, transparent, and accountable governance of AEA. Implications could also include aspects of the governance model that might be discontinued, changed, increased or added.

________________________________________
Scope of Work

 
Evaluation Design
The external evaluator will develop an evaluation design for answering the key evaluation questions within the available budget. The design will outline the methods of data collection, strategies for selecting respondents, and measures (including any additional documents to review) that will be used to address the main evaluation questions. Data collection protocols, including any interview guides, questionnaires, and abstraction tools, should be included in the design. Once approved by the Transition Evaluation Oversight Task Force, and after any necessary revisions, the external evaluator should implement the evaluation design and produce an evaluation report that addresses all evaluation questions, and identifies implications and possible options for the Board.

A copy of the final transition evaluation report will be made available to all AEA members.
 
Data Collection
The evaluation is expected to draw on multiple sources of data to address the key evaluation questions. A range of key informants should be interviewed or surveyed. Possibly important key informants[1] include:  

• Members of the Board during the three-year period just prior to the selection of the policy governance model (2005-2008)
• Board members newly elected and with no prior involvement or commitment to the new governance model,
• Representatives from TIGs and local affiliates (and possibly, editors, vendors, publishers),
• AEA Presidents from 2005 to present
• AEA's Executive Director
• Members of the current Priority Action Teams (PATS) as well as members of the Board Committees prior to the PATs
• Purposeful samples of persons known to be critical of the transition and persons known to be supporters of the transition.

The key informant sampling should strive to represent a range of knowledgeable views and involvement, within the available budget.  In addition, an open call for input from all AEA members should be conducted and the data included in the evaluation. It should be noted, however, that many AEA members will have little knowledge of the association's transition to policy governance that is the focus of this evaluation.

To assist in designing and implementing the evaluation, AEA will provide the evaluator with all readily-available documentation of the impetus, development, and operation of policy governance, including Board correspondence to the membership on the policy governance model and Board minutes that record its development and implementation. It is expected that the evaluator will gather additional descriptive information on AEA's transition in governance as part of the evaluation process.

 
Evaluative Criteria
The evaluation should be explicit about the evaluative criteria to be used to judge the quality of the transition, with priority attention to outcomes being achieved (intended and unintended), and appropriate attention to processes and the distribution of costs and benefits. In particular the criteria for judging quality should explicitly include:

• To what extent the intended outcomes of the transition are being achieved -  These outcomes include (1) less Board time on operational issues and more Board time on policy and strategic planning; (2) more continuity in strategic planning and direction of Board initiatives; (3) more accountability of the Board; (4) greater transparency of Board actions; (5) wider range of opportunities for member engagement in AEA activities; (6) improved functioning of committees/Priority Area Teams (PATs), task forces, working groups; and (7) decreased AMC resources being spent to support these various groups.  
• The magnitude and import of unintended outcomes (positive and negative
• To what extent increases in transparency and efficiency are being achieved
• Equity in the distribution of impacts - whether certain groups of members have been particularly affected (positively or negatively) by the transition

Where possible, the evaluator is encouraged to contribute to the development of indicators and report on data that can provide baselines for ongoing monitoring of the effectiveness of AEA's policy governance model.

Report
A draft of the evaluation report should be submitted by September 30, 2011 for review by the Transition Evaluation Oversight Task Force and then by the Board at its November 2011 meeting. The report should outline the methodology used and answer each of the key evaluation questions.

________________________________________
Award Criteria: Evaluator Qualifications
Applicants should demonstrate a background in evaluation and experience in conducting evaluations, especially formative evaluations that involve key informant data, document review, and analysis of existing data. Experience in conducting qualitative interviews with senior professionals is critical. Strong experience and skills in writing evaluation reports required. Familiarity with board governance models is highly desirable.

The proposal should demonstrate an understanding of the purpose of the evaluation and the scope of work. The proposal should display knowledge and understanding of the kinds of issues that can arise in collecting these types of data, such as areas of disagreement among informants or among other data sources, and strategies for clarifying or reconciling the differences.

Applicants should provide information about the daily rates of all members of the evaluation team (as relevant) and an indication of the number of days of effort that could be covered by the available budget for the external evaluation.

The evaluator must not have been a member of the AEA Board or a Board committee/Priority Action Team during the period 2008-2011 and otherwise be free from a conflict of interest or perception of conflict of interest. 

Timeline

• May 6, 2011: Proposals Due
• May 25, 2011: Contractor reviewed and selected (Board approval required)
• By June 1, 2011: AEA provides available background information to the evaluator
• By June 15, 2011: Evaluator meets via telephone with Transition Evaluation Oversight Task Force to review the scope of the evaluation, identify additional documentation needs, discuss plans for data collection, and other items as needed.
• September 15, 2011: Draft final evaluation report due to Transition Evaluation Oversight Task Force
• September 30, 2011: Task Force review of draft report due back to Evaluator
• October 17, 2011: Second draft report due back to Task Force for submission to Board
• Note: The evaluator may be asked to present the report orally at the November 2011 Board meeting.
• November 15, 2011: Board review of second draft report due back to Evaluator
• November 30, 2011: Final report due/project completion

Budget

The maximum expendable under this contract is $23,000.

Evaluation Management
The evaluation will be managed by a Transition Evaluation Oversight Task Force, comprised of AEA members appointed by the AEA Board. The Transition Evaluation Oversight Task Force will review RFPs and recommend the selection of one evaluator (or evaluation team). The Task Force will further monitor the progress of the evaluation, review draft instruments and sampling plans as appropriate, review the draft final report and then the revised final report, and make recommendations to the Board. The Board will select the evaluator and approve the final report.

The AEA Executive Director will manage the legal and financial aspects of the contract for the evaluation, including advertising the RFP and arranging contracts and payments.

Instructions for Submitting Proposals
Proposals are due by Friday, May 6, 2011. Submit an electronic version of the proposal, as a single file including CV(s), to Heidi Nye at the American Evaluation Association: heidi@eval.org
 
Proposals should be no longer than 5 pages and include:

• Brief narrative describing approach and methods
• Task plan and timeline
• Budget with budget narrative

CVs of key consultants may be included as appendices in addition to the 5 page-narrative. No other appendices or approaches to extending the length of the proposal will be accepted. 
 
Proposals that do not follow the submission instructions and length restrictions will not be reviewed.

Questions
Questions regarding this the substance of this proposal should be directed to AEA President Jennifer Greene at jcgreene@illinois.edu; questions regarding the submission process should be addressed to Heidi Nye in the AEA office at heidi@eval.org.

 
References
AEA Frequently Asked Questions:  The Move to Policy Governance, May, 2009.

Carver, J. (2006). Boards that Make a Difference: A New Design for Leadership in Nonprofit and Public Organizations. San Francisco: Jossey-Bass.

[1] A significant number of key informants will participate in a retreat directly related to next steps in the policy governance transition in June 2011 in Atlanta GA. The evaluator may wish to have an observer present at this retreat. Just prior to the retreat, a number of key informants will also be participating in the AEA-CDC Institute (professional development for evaluators), also in Atlanta. The evaluator may wish to take advantage of this gathering to conduct some key informant interviews efficiently.