INCEPTION REPORT October 2014 Evaluation of the CGIAR Research Program on Aquatic Agricultural Systems Image: Photo by Stevie Mann, 2007.From WorldFish Blog. Regina Birner(Team leader) Jim Sumberg (Team leader) Sirkka Immonen (IEA) Sophie Zimm (IEA) Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Table of contents List of Abbreviations ............................................................................................................................4 1. Introduction .............................................................................................................................5 Rationale for the Evaluation ............................................................................................................5 Purpose of the Evaluation ................................................................................................................5 2. Background ..............................................................................................................................6 Program context ..............................................................................................................................6 Program background .......................................................................................................................8 Introduction .....................................................................................................................................8 AAS Structure and organisation .................................................................................................... 11 Theory of change .......................................................................................................................... 11 Current state of play ..................................................................................................................... 12 AAS Governance and management .............................................................................................. 12 AAS funding and expenditures ...................................................................................................... 13 AAS current portfolio .................................................................................................................... 14 3. Scope of the evaluation ........................................................................................................ 16 4. Evaluation Criteria and Questions ........................................................................................ 17 AAS-specific overarching questions .............................................................................................. 17 Research/Program Performance .................................................................................................. 18 Relevance .................................................................................................................................. 18 Quality of Science...................................................................................................................... 19 Likely Effectiveness ................................................................................................................... 19 Sustainability ............................................................................................................................. 19 Impact ....................................................................................................................................... 19 Partnerships .............................................................................................................................. 20 Gender ...................................................................................................................................... 20 Capacity building ....................................................................................................................... 20 Monitoring and Evaluation ....................................................................................................... 20 Governance and management ..................................................................................................... 21 5. Evaluation approach and methodology ................................................................................ 21 Evaluation approach and rationale ............................................................................................... 21 Overview of the methodology ...................................................................................................... 22 Methods of data collection and analysis ...................................................................................... 23 Component 1: Qualitative case studies .................................................................................... 23 Component 2: Publications review ........................................................................................... 27 Component 3: Research staff survey ........................................................................................ 28 Component 4: Interviews AAS members, partners and experts .............................................. 28 Methodological frameworks for selected topics .......................................................................... 28 Quality of science assessment .................................................................................................. 29 Impact assessment .................................................................................................................... 30 2 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Gender assessment ................................................................................................................... 30 Governance and management assessment .............................................................................. 31 6. Limitations of the Evaluation ................................................................................................ 32 7. Organization and Timing of the Evaluation .......................................................................... 32 Team Composition and Responsibilities ....................................................................................... 32 Evaluation governance – roles and responsibilities ...................................................................... 33 Stakeholder Involvement .............................................................................................................. 33 Timeline and key deliverables ....................................................................................................... 34 ANNEX 1: References ........................................................................................................................ 35 ANNEX 2: EVALUATION MATRIX ....................................................................................................... 36 ANNEX 3: LIST OF PERSONS CONSULTED IN INCEPTION PERIOD ..................................................... 41 ANNEX 4: EVALUATION TEAM PROFILES .......................................................................................... 42 ANNEX 5: members of EVALUATION reference group ..................................................................... 44 ANNEX 6: field visits .......................................................................................................................... 45 Tables Table 1: AAS Objectives and Research Themes .................................................................................... 10 Table 2: AAS Intermediate Development Outcomes (IDOs) ................................................................. 10 Table 3: Overview of AAS Flagship Projects Portfolio (figures in USD thousands) ............................... 15 Table 4: Bilateral projects larger that USD 500,000 (figures in USD thousands) .................................. 15 Table 5: Planned field visits ................................................................................................................... 24 Table 6: Team composition and primary responsibilities ..................................................................... 32 Table 7: Evaluation schedule ................................................................................................................ 34 Figures Figure 1: AAS Scaling Pathways ............................................................................................................ 11 Figure 2: AAS Expenditures (2011-2013) and expected budget (2014-2016) in USD thousands ......... 13 Figure 3: Expenditure (from mid-2011 to 2013) per Theme ................................................................ 14 Figure 4: AAS Portfolio budget per Flagship and cluster of activities/research theme (in USD million).16 3 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 LIST OF ABBREVIATIONS AAS CGIAR Research Program on Aquatic Agricultural Systems ACIAR Australian Centre for International Agricultural Research CO Consortium Office CRP CGIAR Research Program CRS Catholic Relief Services CSISA Cereal Systems Initiative for South Asia EPMR External Program and Management Review (CGIAR) FP Flagship Project IDO Intermediate Development Outcome IEA Independent Evaluation Arrangement (Rome) ISPC Independent Science and Partnership Council IWMI International Water Management Institute NGO Non-governmental organization (general) PLT Program Leadership Team (AAS) POP Program Oversight Panel (AAS) POWB Program of Work and Budget PSU Program Support Unit (AAS) RinD Research in Development SRF Strategy and Results Framework (CGIAR) SLO System-Level Outcome (CGIAR) W1 Window 1 funding type (CGIAR) W2 Window 2 funding type (CGIAR) W3 Window 3 funding type (CGIAR) 4 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 1. INTRODUCTION Rationale for the Evaluation In 2008, the CGIAR underwent a major reform with the goal to become a coherent, demand-driven research partnership that tackles global development challenges. The underlying vision is of a "food- secure future".1 Research in the reformed CGIAR is guided by an overall Strategy and Results Framework (SRF), which was first approved in 2011. The SRF sets forth the System’s common goals in terms of development impact (System-Level Outcomes [SLOs])2, strategic objectives and results. A major element of the CGIAR reform was the establishment of the CGIAR Research Programs (CRPs), through which the 15 CGIAR Centres and their partners jointly implement the SRF. The CRPs are funded through a mechanism by which the donors pool their funds (the CGIAR Fund3) as well as through bilateral funds to individual Centres. In the CGIAR, the Independent Evaluation Arrangement (IEA) Office is responsible for System-level external evaluations. The IEA is mandated to develop a coordinated, harmonized and cost-effective evaluation system. Its main responsibility is to lead the implementation of the CGIAR Policy for Independent External Evaluations4, through the conduct of strategic evaluations of the CRPs and institutional elements of the CGIAR. The IEA’s first four-year Rolling Evaluation Work Plan (2014-17), approved in November 2013 by the Fund Council, foresees the evaluation of up to 10 CRPs over the 2013-2015 period. The order in which the CRPs will be evaluated was established on basis of multiple criteria including the starting date of the CRP, and donor feed-back. One of the CRPs to be evaluated in 2014 is Aquatic Agricultural Systems (AAS). Among the 15 CRPs, AAS is one of three that focus on improving the productivity, profitability, sustainability, and resilience of a "system", rather than focusing on single commodities.5 In the original proposal, aquatic agriculture systems were defined as “agricultural systems in which the annual production dynamics of freshwater and/or saline or brackish coastal systems contribute significantly to total household income.”6 Purpose of the Evaluation The primary purpose of this evaluation is to enhance the contribution that AAS is likely to make towards the SRF. The evaluation should provide essential evaluative information for decision making 1 See http://www.cgiar.org/who-we-are/history-of-cgiar/cgiar-reform/ 2 Defined as four System-Level Outcomes: reduction of poverty, improvement pf food security, increasing nutrition and health; and more sustainable management of natural resources. 3 The CGIAR Fund is a multi-donor, multi-year funding mechanism that provides funding to (i) CRPs through two “Windows”; Window 1 across CRPs as per Consortium decision and Window 2 to donor-specified CRP; and to (ii) donor-specified Centers through Window 3. 4 http://www.cgiarfund.org/sites/cgiarfund.org/files/Documents/PDF/CGIAR_evaluation_policy_jan2012.pdf 5 See http://www.cgiar.org/our-research/cgiar-research-programs/ for a classification of the CRPs. 6 See original AAS proposal, page 1, footnote a 5 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 by AAS management, the Consortium Office and funders on issues such as modification, extension, expansion and the structure of the program. The CGIAR is in the process of updating the SRF. Subsequently, a second call for CRPs will be launched. Following a Fund Council agreement, all current CRPs will undergo some form of evaluation before the initiation of full second phase proposal development. In this context, the evaluation of AAS should feed directly into decision-making processes of the CRP management, CO and Fund Council. Taking into account the stage of the program and given its nature and timelines for results, the evaluation aims to provide an overview and a critical analysis of its relevance, achievements to date and likely impacts. The evaluation is guided by the Terms of Reference (TOR).7 This inception report builds on the TOR and elaborates on the approach adopted for this evaluation and the methods to be used to address specific questions and issues. The remainder of this Inception Report is structured as follows. The next section provides background on the AAS and sets it in the context of the CGIAR reform process. AAS structure, governance and funding are described. In Section 3, the scope of the evaluation is briefly circumscribed, and Section 4 elaborates on the evaluation issues and criteria. Section 5 outlines the proposed evaluation approach and methodology. The limitations of the evaluation are discussed in Section 6, while Section 7 outlines the organization and timetable for the evaluation. 2. BACKGROUND Program context The latest round of CGIAR reform was set in motion in 2008. The CGIAR donors, in a Joint Declaration,8 agreed on the following main principles for the reform: 1) “To harmonize our approach to funding and implementing international agricultural research for development through the CGIAR Fund (the Fund), The Strategy and Results Framework and the consortium established by the Centers (the Consortium), respectively; 2) To manage for results in accordance with the agreed Strategy and Results Framework (SRF) and the Mega Programs9 that derive from the SRF; 3) To ensure effective governance and efficient operations in the provision and use of our resources; and 4) To collaborate and partner with and among funders, implementers, and users of SRF research, as well as other external partners supporting the SRF.” The SRF was approved in 2011 at a time when the Center-led CRPs had already been developed and two of them (on rice and climate change) had been approved. Thus, the current CRPs did not emerge in direct response to the SRF, although the SRF is intended to provide the broad rationale and context for the development, implementation and evaluation of all CRPs. In the current SRF, one of the areas for developing or strengthening competence was research on production systems. The concept of innovation systems was recognized as important in the changing institutional landscape of agricultural 7 http://iea.cgiar.org/publication/tors-aquatic-agricultural-systems 8 http://library.cgiar.org/bitstream/handle/10947/2551/jointdeclar_final_jan2010.pdf?sequence=1 9 Mega programs were later termed CGIAR Research Programs, CRPs 6 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 research.10 Subsequently three "systems" CRPs were approved: drylands, humid tropics and aquatic agricultural systems. The CRPs were developed and appraised following a set of common criteria that addressed the (i) strategic program coherence; (ii) focus on delivering outcomes and impacts towards the SLOs; (iii) quality of science; (iv), management of partnerships, including both research and development partners; (v) efficiency of program management; and (vi) accountability, sound financial planning and efficiency of governance. Under Consortium Office coordination and instructions, since 2012 a set of Intermediate Development Outcomes (IDOs) has been developed. The IDOs link the CGIAR research to the SLOs and should facilitate priority setting, again both at the CGIAR and CRP levels. Simultaneously, CRPs have been instructed to use the notion of Flagship Projects (FPs) to restructure their programs, with clusters of activities being set within each FP. In principle, each FP should contribute to one or more IDOs, and thereby to the SLOs. The articulation of theories of change and impact pathways – leading from research activities to the achievement of the IDOs – was also required. Specifically the CRPs were instructed to define the IDOs in terms of clear target domains (agroecologies and end user groups) and measurable results at the outcome level.11 One needs to note, however, that the CGIAR (and CRP) IDOs are still a work in progress. The major purpose of the work done in 2013 and 2014 was to get the CRPs focused on outcomes, to identify how to measure progress against these outcomes, and to better understand what is required to achieve these. These insights will inform the design the next phase of CRPs (2017 onwards) and to develop a Results-based Management System. The funding sources available to CRPs in the reformed CGIAR are shown in Box 1. Box 1: Major Sources of Funding in the CGIAR System To maximize coordination and harmonization of funding, donors to CGIAR are strongly encouraged to channel their resources through the CGIAR Fund. Donors to the Fund may designate their contributions to one or more of three funding “windows”: • Contributions to Window 1 (W1) are the least restricted, leaving to the Fund Council how these funds are allocated to CGIAR Research Programs, used to pay system costs or otherwise applied to achieving the CGIAR mission. • Contributions to Window 2 (W2) are designated by Fund donors to specific CGIAR Research Programs. • Contributions to Window 3 (W3) are allocated by Fund donors to specific CGIAR Centers. Participating Centers also mobilize financial resources for specific activities directly from donors as bilateral funding and negotiate agreements with their respective donors for the use of these resources. Source: CGIAR website: http://www.cgiar.org/who-we-are/cgiar-fund/ The W1/W2 components of the budget are the least restricted. Their initial level was set on the basis of the core funding in the period preceding the CRP (i.e. 2010). 10 See, e.g., http://www.cgiar.org/consortium-news/with-an-innovative-systems-approach-cgiar-adapts-science- to-meet-smallholders-needs/ 11 SRF management update (approved at FC10 in Mexico). Link: https://library.cgiar.org/bitstream/handle/10947/2895/2014%20- %20SRF%20Management%20Update_Final_2013_12_20.pdf?sequence=1 7 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 The internal reform context has also involved development of guidelines and templates for annual reporting to the Consortium regarding all sources of funding. In parallel, bilateral funders have their own specific reporting requirements. Given that bilateral funding remains a significant proportion of all funding, the reform has not yet resulted in the hoped for reduction in the reporting burden. Most CRPs were initially approved for a three-year period to run in parallel to the SRF. As the evaluation of AAS is beginning, an updated SRF is being prepared by the Consortium Office to include system-level research funding priorities. At the same time, AAS, as all CRPs, has applied for extension funding for 2015-16, and in August, at the request of the Consortium, the Program submitted a revised proposal. Finally, a process for the 2nd call of CRPs is in preparation, and a CGIAR Mid-Term Review is being completed to provide assessment of and guidance for the reform. This rapid schedule in the implementation of the reform is putting pressure on the CRPs and their partners. It also has implications for the AAS evaluation in terms of the evolving CGIAR and CRP context. Program background Introduction The AAS Proposal Document was approved by the Fund Council in July 2011 and the program started operations in the same month.12 In deciding to fund AAS the FC noted that the program proposal required further development, which was to be undertaken during the first year. AAS has as its overall goal “to improve the well-being of AAS-dependent people”. This goal is pursued through six objectives: increased productivity of aquatic agricultural systems; improved access to services and markets; improved resilience of marginalized people; reduced gender disparities; better policies and institutions; and improved knowledge sharing and learning (Table 1). These objectives form the basis of the six research themes and also underpin the seven AAS-specific IDOs (Figure 1).13 In line with CGIAR requirements, AAS is meant to deliver both international public goods and a series of place-based development outcomes. AAS set out to distinguish itself from what the proposal framed as "business as usual" approaches to agricultural research, which, the proposal suggests, have had limited successes in benefiting poor and marginal people dependent on aquatic agricultural systems. As the proposal states, “the central hypothesis driving the approach” of AAS is that "the CGIAR can have greater impact on AAS by moving beyond the linear production model that has dominated much agricultural research and embracing a more integrated, innovative view of how to achieve development in agricultural systems.” The proposal further states that “we will do this through an action research and partnership-driven approach to development that moves far beyond the view of development as a purely technical process, as well as the persistent views of development as charity.”14 AAS also emphasizes that it does not have a commodity focus ("AAS is not about fish"). Rather, it is a systems research program that is framed in terms of livelihoods, systems change, transformational social change, sustainability and resilience. AAS management has also emphasized the need for the program to evolve, not only in 12 See CGIAR 5th Fund Council Meeting Summary, July 6-8, 2011 13 The Extension Proposal already outlines quantifiable IDO indicators (for 2019, 2022 and 2025). 14 See original AAS proposal, p. 7. 8 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 response to the changing CGIAR environment, but also due to the program’s commitment to be a learning organization. In the original proposal, the AAS team indicated an orientation toward action research. This was defined in the proposal based on Reason and Bradbury's (2008) action research handbook, which highlights the typical elements of active participation of those who would otherwise be considered as research subjects, and systematic cycles of action and reflection. The proposal also identifies a continuum between “technical action research" on the one hand, and “emancipatory action research” on the other.15 In subsequent documents, the program uses the term “participatory action research.”16 The related terms “systems research” and “participatory research” are also used in AAS documents. AAS builds on past productivity-oriented research, particularly of WorldFish, and a major emphasis is on harnessing the full potential of aquatic agricultural systems for increased productivity. AAS is also supposed to draw on knowledge and technology generated by the other CRPs. The program uses the term “Research in development” (RinD) to describe its approach.17 The suggestion is that through this approach, agricultural research is embedded in on-going development actions and processes, and adds value by leveraging potential for innovation. Its implementation focuses on participatory action research within a set of geographical defined “learning hubs”. The action research approach promoted by AAS emphasizes engagement with farmers, fishers, NGOs and government institutions in each hub to identify the main challenges and thereby the research priorities. It involves the potential beneficiaries in the action research process. A key feature of AAS is its focus on "transformational change" with a very strong focus on transforming gender relations. 15 The proposal does not include a review of the various earlier approaches to promote action research in the CGIAR system, even those existed since the 1980s (see, e.g., Becker, 2000). The AAS proposal does, however, emphasize its distinction from what is considered to be typical CGIAR research. As the proposal states: “We recognize that full immersion into action research will require a major change in the way most CGIAR scientists work.” (CRP proposal, p. 8). 16 See AAS Working Paper “Research in Development: The Approach of AAS” by Dugan, Apgar and Douthwaite. 17 We note that the term "research in development" is used by other CGIAR centres, but in different ways (see e.g.: Coe, R., Sinclair, F. and Barrios, E. (2014) Scaling up agroforestry requires research ‘in’ rather than ‘for’ development, Current Opinion in Environmental Sustainability 6:73–77. 9 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Table 1: AAS Objectives and Research Themes No Objective Research Theme 1 Increased benefits to AAS-dependent households from Sustainable increases in system productivity: environmentally sustainable increases in productivity. participatory production, adaptation of appropriate technologies 2 Improved markets and services available to poor and Equitable access to markets: improve benefits vulnerable AAS households. to low-income actors along agricultural and natural resource value chains 3 Social-ecological resilience and adaptive Strengthened resilience and adaptive capacity in poor, capacity: social systems research and action for vulnerable and marginalized groups and households. social change to support improved capacity and resilience 4 Reduced gender disparities in access to and control of Gender equality: integrates gender within all resources and decision making through beneficial research themes, studies gender norms and changes in gender norms and roles. roles 5 Improved policy and formal and informal institutional Policies and institutions to empower AAS structures and processes implemented to support pro- users: supporting improved arrangements; poor, gender-equitable and sustainable development. action research involving key stakeholders 6 Productive relationships, partnerships and networks Knowledge sharing, learning, and innovation: capable of achieving research and development participatory action research to foster program outcomes sustained through effective knowledge learning and support social learning with sharing and learning. stakeholders Source: AAS Proposal Document March 2011 Table 2: AAS Intermediate Development Outcomes (IDOs) Material 1. Income: Increased and more equitable income from agricultural and natural resource management and environmental services earned by low income value chain actors in aquatic agricultural systems 2. Nutrition: Improved diet quality of low income households in aquatic agricultural systems, especially by nutritionally vulnerable women and children 3. Future Options: Greater resilience in aquatic agricultural systems for enhanced ecosystem services Enabling 4. Productivity: Improved productivity in aquatic agricultural systems (water and total factor productivity) 5. Gender & Empowerment: Increased control of assets, inputs, decision-making and benefits by women and other marginalized groups in aquatic agricultural systems 6. Capacity to innovate: Increased capacity to innovate within low income and vulnerable rural communities in aquatic agricultural systems 7. Capacity to adapt: Increased capacity to adapt to environmental and economic variability, shocks and longer term changes in low income communities in aquatic agricultural systems Source: AAS Extension Proposal. 10 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 AAS Structure and organisation The AAS is led by WorldFish Center, with Bioversity International and IWMI as main CGIAR partners. The proposal emphasizes partnerships with other CG centers and a wide range of other partners. AAS was originally structured along the lines of six research themes18. In 2014 it was restructured along so-called geographical flagships/hubs and one additional Flagship for global science and scaling.19 In the first version of the 2015-2016 extension proposal, three of the four flagships are defined as major AAS ecologies and the fourth remains as Global Science and Scaling. Furthermore, the extension proposal identified two additional hubs. Accordingly, AAS is structured in relation to (i) Flagship projects (= 6); (ii) Countries (= 5); (iii) Hubs (= 5); (iv) Research themes or clusters (= 6); and (v) Bilateral projects (= approximately 50). Theory of change The AAS Theory of Change articulates three impact pathways, which are now also referred to as scaling pathways (Figure 1). The first pathway focuses on direct engagement with communities and partners in the hubs, and the second on transforming the enabling conditions at hub-level. The third pathway involves scaling up what has been learnt at the hub-level to the national, regional and global level. Through these three pathways AAS expects to deliver both place-based "development outcomes" (Pathways 1 and 2) as well as international public goods (Pathway 3). Figure 1: AAS Scaling Pathways Source: AAS Working Paper. Using theory of change to achieve impact in AAS. 18 System productivity, access to markets, resilience and adaptive capacity, gender equity, policies and institutions and knowledge sharing 19 Program of Work and Budget 9POWB), 2014 11 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Current state of play AAS completed its 3rd year of operations in July 2014. In order to synchronize those CRPs that were launched in mid-2011 with those that started later, AAS was granted an extension until the end of 2014 based on the submission of its Program of Work and Budget 2014.20 An extension proposal for 2015 and 2016 was submitted to the Consortium Board in April 2014 and was reviewed by the Independent Science and Partnership Council (June 2014) and the Consortium Office (July 2014). As a result AAS has been asked to submit a revised proposal by the end of August 2014. The on-going extension process is an “intermediate solution” in anticipation of a call for second phase CRP proposals. AAS Governance and management The AAS proposal specified the following Governance and Management structure: the Consortium Board; a Program Oversight Panel; a Program Leader; a Program Leadership Team; a Program Support Unit; a Program Forum; Country Program Committees; Country Program Teams; and Country Program Managers. This structure is being implemented through a number of formal agreements. A Joint Agreement between Fund Council and the Consortium in 2011 sets out the umbrella terms governing the submission and approval of CRP proposals and the transfer to and use of W1/W2 funds by CRPs. In the Consortium Performance Agreement the Consortium assumes overall financial and programmatic responsibility for the implementation of AAS. In a Program Implementation Agreement between Consortium and WorldFish, the latter assumes responsibility for the use of W1/W2 funds transferred to it by the Consortium, and for the satisfactory performance of AAS. WorldFish signed a Program Participant Agreement with IWMI and Bioversity in which the latter two centers accept responsibility for the use of W1/W2 funds transferred to them by WorldFish, and for the satisfactory performance of AAS activities. The AAS Director (Patrick Dugan, also Deputy Director General of WorldFish) has overall responsibility for the implementation of AAS. The Program Oversight Panel (POP) is the main oversight body for AAS. It was established to provide “objective and flexible oversight of the program” and consists of eight internationally recognized scientists and development professionals. The POP reports to the WorldFish Board of Trustees (including an annual report on the implementation of the Program Implementation Agreement).21 The Program Leadership Team (PLT) provides “collective leadership of the program” and supports the Program Director and the POP in providing strategic direction and coherence, ensuring science quality, and ensuring operational effectiveness. It consists of representatives of participating centers (Bioversity, IWMI, WorldFish), representatives from two international NGO partners (CRS, CARE), AAS Country Leaders (or their delegates), the head of the Program Support Unit (PSU), and lead scientists for the Program’s six research themes. The PLT is chaired by the Program Director. He informed the evaluation team that the leadership arrangements for the program have been in transition in 2014 and that an expanded Senior Leadership Group will be implemented in 2015. 20 See CGIAR 11th Fund Council Meeting Summary, May 7-8, 2014 21 See Terms of Reference for the Program Oversight Panel of CRP AAS, Approved by the WorldFish Board of Trustees 11th August 2011 12 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 AAS funding and expenditures AAS is one of the smallest CRPs with a three year (2011-2013) proposed budget of USD 59 million (USD 17.3, 19.8 and 22.4 in Years 1, 2 and 3 respectively). About 20% was projected to come from the Fund (W1/W2) and 45% from bilateral sources: the source of the remaining 35% was not specified. The original budget projection for 2014-2017 was for USD 85 million. In the first 2.5 years of implementation (June 2011 to December 2013), AAS received and spent around USD 54 million, somewhat above what was projected in the proposal. Most of the total expenditure (96%) has been through WorldFish, while IWMI and Bioversity spent only 3 and 2%, respectively. Figure 2 shows the expenditures for 2011 to 2013 and the budget estimates for 2014 to 2016. W1/W2 funding has been proportionately more than projected in the original proposal (in the first 2.5 years 42%). Figure 2: AAS Expenditures (2011-2013) and expected budget (2014-2016) in USD thousands 50,000 45,000 40,000 35,000 21,430 30,000 19,530 Center funds 25,000 15,588 W3/Bilateral 20,000 13,271 15,000 W1/2 12,544 10,000 5,000 12,289 15,997 20,040 22,540 3,754 4,176 7,543 0 2011 2012 2013 2014 2015 2016 Source: Financial Reports 2012 and 2013 (L101), POWB 2014, Extension Proposal 2015-2016. The largest bilateral donors have been USAID (project "Aquaculture for Income and Nutrition in Bangladesh" and others); IRRI (money passed to WorldFish for the "Cereal Systems Initiative for South Asia" project); IDRC ("Economy and Environment Program for Southeast Asia"); and ACIAR (the "Pacific Fisheries" project and others). CRP Management and Coordination accounts for 33% of the total expenditures. It should be noted that in addition to governance and management, expenditure for partnership, communications, science leadership and support also falls under this heading. The research themes within AAS are funded to different levels (Figure 3), with Theme 1 (Systems Productivity) funded at almost twice the level of the second most funded theme. 13 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Figure 3: Expenditure (from mid-2011 to 2013) per Theme22 Theme 6: Theme 1: Knowledge Sustainable sharing, learning increase in and innovation, system 5,951 , 16% productivity, Theme 5: Policies 11,430 , 31% and institutions to empower AAS users, 4,521 , 12% Theme 4: Gender and equity, 3,161 , 9% Theme 3: Social- ecological resilience and Theme 2: adaptive Equitable access capacity, 5,052 , to markets, 14% 6,360 , 18% Source: AAS Financial Reports 2012 (L101) and 2013 (L131) AAS current portfolio Work in the different Flagships and themes is presently at different stages, and is funded from different sources. Funding under W1/W2 should give AAS the best opportunity to implement its envisaged approach, while bilaterally funded projects are associated with specific interests of donor agencies. It is essentially the W1/W2 funding which allows the "roll-out" of the AAS approach in the hubs and the initiation of the "integrated research initiatives" designed in response to the roll-out exercises. AAS activities funded through W1/W2 are detailed in 78 Activity Plans and mid-year activity reports, which will be an important resource for the evaluation. There are a total of 53 bilateral projects, all of which are led by WorldFish. In contrast to other CRPs, the participating centers have not included funding from any of their bilateral projects within the AAS portfolio. There are a number of “legacy projects” which are a continuation of, or which build on, research undertaken by World Fish in the pre-AAS period. Many of the current bilateral projects were designed before the formal start of AAS. Work under each geographical Flagship and the global Flagship consists of clusters of activities organized under the respective research themes (Table 3). It needs to be noted that several of the bilateral projects are spread over different clusters of activities within the same Flagship or in two cases even over different Flagships.23 22 This figure does not include management and coordination expenditures. 23 The EC funded project "Implementing an ecosystem approach to fisheries (EAF) in small scale tropical marine fisheries" (DCI-ENV/2011/221-352) as well as the ACIAR "Pacific Fisheries Project" are part of both the Solomon Islands and Global Scaling Flagships. 14 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Work under each geographical Flagship and the global Flagship consists of clusters of activities organized under the research themes and funded through W1/W2 and bilateral projects (Table 3). Table 3: Overview of AAS Flagship Projects Portfolio (figures in USD thousands)24 Number of projects Budget Flagship project W1/W2 Bilateral Total W1/2 Bilateral Total % 1 – BAN 7 13 28 1.417 6.593 8.010 25 2 – CAM 6 6 12 1.029 1.126 2.155 7 3 – PHI 5 5 10 1.052 1.999 3.052 10 4 – SOL 5 6 11 1.099 1.250 2.350 7 5 – ZAM 6 2 10 4.080 0.800 4.880 15 6 – GLO 7 21 30 7.572 3.838 11.410 36 TOTAL 36 53 100 16.252 15.607 31.860 100 BAN – Bangladesh; CAM = Cambodia; PHI – Philippines; SOL – Solomon Islands; ZAM – Zambia; GLO = Global Science and Scaling. Source: AAS Project database, as of Aug 2014. The largest Flagship overall is Global Science and Scaling which accounts for about 46% of W1/W2 funds. According to AAS management, a substantial share of the funding under this Flagship is spent on design, implementation and analysis of research in the geographical flagships. Flagship 1 (Bangladesh) is the largest of the geographical Flagships and also has the largest number of bilateral projects. Almost 90% of bilateral projects have budgets less than USD 250,000: there are only seven projects with budgets of more than UDS 500,000 (Table 4). Budget by Flagship project and research theme is shown in Figure 4. Table 4: Bilateral projects larger that USD 500,000 (figures in USD thousands) Research theme Project Title Budget Start date End date Funder Expansion of Cereal Systems Initiative for South Asia (CSISA) in Bangladesh 0.556 1-Oct-10 30-Sep-15 IRRI FTF: Aquaculture for Income and Nutrition Productivity (AIN) (some part of budget is in Value 4.750 1-Oct-11 30-Sep-14 USAID chain Theme) Rice field fisheries improvement project 0.609 18-Apr-12 17-Apr-16 USAID Global and Pacific Fisheries 1.327 1-Jul-13 30-Jun-17 ACIAR regional Improving research and development of scaling Myanmar's inland and coastal fisheries 0.670 1-Sep-12 31-Aug-16 ACIAR Source: AAS Project database, as of Aug 2014. A small part of the Pacific Fisheries project is mapped to Solomon Islands 24 The two projects which are mapped to SOL and GLO Flagships are counted in SOL for this overview. 15 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Figure 4: AAS Portfolio budget per Flagship and cluster of activities/research theme (in USD million). Global and regional scaling Knowledge sharing, learning and innovation Policies and institutions to empower AAS… Gender and equity Social-ecological resilience and adaptive… Equitable access to markets Sustainable increase In system productivity Scaling AAS learning in Zambia Knowledge sharing and learning Gender transformative approach Flood risk management Improved agricultural and NRM value chains Increased farm productivity and… Climate change Gender transformative approach Knowledge sharing and learning Marine resource management and… Sustainable farming for nutrition and income Knowledge sharing and learning Gender transformative approach Enhanced and effective governance for… Equitable and fair market access Farm productivity and diversification Scaling AAS learning in Cambodia Knowledge sharing and learning Policy and institutions to empower AAS users Gender transformative approach Equitable access to markets Improved productivity In Country Scaling Knowledge, sharing and learning Gender transformative approach Adaptive capacity and resilience Improved value chains Improved nutrition Increased farm productivity and… 0.000 1.000 2.000 3.000 4.000 5.000 6.000 Source: AAS Project database, as of Aug 2014. 3. SCOPE OF THE EVALUATION The evaluation will cover all research and related activities and processes of AAS since its inception in 16 Independent iea.cgiar.org Evaluation Arrangement FP 1- BAN FP 2 - CAM FP 3 - PHI FP 4 - SOL FP 5 - ZAM FP 6 - GLO Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 July 2011. This includes also "transferred (or legacy) research" that was initiated before the start of AAS and is carried on within AAS. The evaluation will be both summative and formative. The summative element that looks at achievements and results will, to some extent, cover research done before the AAS was initiated. Given that AAS has been in operation for only three years, the major formative emphasis of the evaluation will be formative. Throughout the evaluation, particular emphasis will be placed on the "AAS approach" and the likelihood that it will lead to sustainable outcomes and impacts in aquatic agriculture systems on the one hand, and the production of international public goods on the other. 4. EVALUATION CRITERIA AND QUESTIONS AAS-specific overarching questions As indicated in the introduction above, AAS defines itself as using a “Research-In-Development” (RinD) approach and claims that this is fundamentally different from what the AAS proposal labels as “traditional” CGIAR research.25 Through the development and use of RinD AAS has the ambition to be more effective in achieving its objectives than (the AAS proposal suggests) has been the case with research approaches used by the CGIAR in aquatic agricultural systems in the past. It also seeks to become an “exemplary vehicle for implementing the fundamental changes in ways of working that the CGIAR reform process foreshadowed.”26 Although the CGIAR has in the past made important contributions in the realms of farming systems and participatory research,27 AAS sets itself apart from past CGIAR programs and the other CRPs by its encompassing and exclusive dedication to participatory action research. As articulated in the proposal and developed over the last three years, the "AAS approach" has been proclaimed as more innovative, and an alternative to what the AAS characterizes as conventional, "linear" approaches to commodity-focused agricultural research. To take the specific framing and ambition of AAS adequately into account, this evaluation will address the following questions that are specific to the AAS approach and go beyond the concerns of a classic research performance evaluation that are addressed in the subsequent sections: 25 As stated in the AAS Proposal (p. 1): “Pursuing our work in this way will challenge the CGIAR to move beyond traditional circles and change the way we do much of our research. By emphasizing approaches that call for research in development — rather than research and development or research for development — we will pursue a conscious change in emphasis and mind set, one that can help the CGIAR to conceive and deliver our research differently.” 26 AAS Proposal, p.1. 27 See, e.g., Becker (2000) for a review. Most notably, the CGIAR had a System-wide program for Participatory Research and Gender Analysis. See Anderson, Herdt & Scobie (1988, p. 41 ff) for an early review of Farming Systems Research in the CGIAR. 17 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 1. What in the AAS approach to research is new, innovative and/or unique in the context of agricultural research and the CGIAR? 2. How does the AAS approach to research draw on earlier and on-going work, especially participatory action research approaches, from inside the CG and elsewhere? 3. What progress has been made so far in developing and implementing the AAS approach to research? 4. What are the challenges faced while implementing the AAS approach and are there any missing links or actions, which could help to make the AAS approach more effective? 5. To what extent is the AAS approach better suited than other research approaches to meet the challenges faced by people living and working in AASs? To what extent is it better suited to reach marginalized groups and women and to address the inequities they face? 6. Is the AAS approach to research, as it is being implemented in the various hubs, likely to deliver both international public goods and the identified IDOs? 7. What are the comparative advantages and the added value of the CGIAR via AAS in developing and implementing this approach to research in aquatic agriculture systems? 8. To what extent is the AAS approach a model for other CGIAR research programs, especially with regard to the CGIAR's stated aim of moving from a “supply-driven” to a ”demand-driven” mode of research? 9. How do partners and networks understand the AAS approach and its implementation? The importance of these questions has been reinforced by the evaluation team's initial reading of AAS documentation and the first interaction with AAS management and staff, who emphasized the need to consider the unique nature of the AAS approach in this evaluation. These questions are also very much reflected in the ISPC and CO comments on the first version of the AAS 2015-2016 extension proposal. Over the course of this evaluation, the team will address these questions by examining in detail various aspects of AAS design, implementation, outputs and management and governance. The evaluation methodology, especially the case study component, has been specifically designed to address the nature of AAS. While paying special attention to the AAS approach, the evaluation will also apply established criteria for evaluating research and organizational performance, as further outlined in the following sections. Research/Program Performance Relevance The assessment focuses on the following main aspects of relevance: • The strategic coherence and consistency of AAS regarding its own objectives and the CGIAR’s System Level Outcomes; • The comparative advantage of the CGIAR and the participating centers developing and implementing AAS compared to other research and development organizations, given the CGIAR’s focus on generating international public goods; • The relevance of the program in responding to the needs and priorities of the intended beneficiaries and stakeholders and the national, regional and global priorities; • The relevance of program to more equitable gender and social relations; 18 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 • The relevance of the program in responding to emerging opportunities for generating new research-based knowledge. The evaluation will assess the formulation of the IDOs and their relevance against the program objectives and CGIAR SLOs. It will assess the integration of research within and among the FPs and the prioritization of activities for addressing the IDOs. Priority setting processes will be assessed, as will the use of W1/W2 funding, resource mobilization and strategic foresight. The evaluation will also assess the synergies among AAS partners, and opportunities for further enhancing the relevance of research results. Quality of Science The evaluation of science quality will look at several dimensions of quality including the make-up of the research teams and partnerships, research design, research management, quality assurance and research outputs. The evaluation will look at the processes and incentives in place for ensuring high quality research across program components and partners. It will assess the track record of research leaders and the competences of research staff. It will look at the program design in terms of problem setting, the use of state-of-the art research literature and methods, and novelty. Likely Effectiveness Effectiveness will be assessed primarily from the point of view of likely effectiveness of the current program, rather than past impact. The evaluation will look at the program design, and particularly the plausibility of the theories of change and impact pathways (both generic and specific). The assumptions underpinning the theories of change will be assessed as well as the Program’s use of the theories of change for informing the assumptions and monitoring changes towards outcomes. The evaluation will consider the extent to which risks and constraints influencing out-scaling, outcomes and impacts are being addressed in research design, partnerships and capacity building. It will look at the extent to which gender analysis and social analysis more broadly have informed the impact pathways. The evaluation will also consider the linkages that AAS has with other CRPs, and opportunities for further enhancing the likely effectiveness of the research they contribute to. The evaluation will assess progress towards milestones and outputs across the research portfolio. It will assess the M&E system and the extent to which it is used by management to adjust research plans and impact pathway designs, including learning from gender and policy analyses. Sustainability Given that AAS is aiming at transformative social changes and enhancing system sustainability, the evaluation will assess the value, in terms of increased sustainability and resilience, that the approach may add to productivity enhancing and other more technology-oriented research done by AAS or other CRPs collaborating with AAS in aquatic agricultural systems. The evaluation will also assess the extent to which the AAS approach is likely to be sustained and scaled over time. Impact As part of the summative component of the evaluation the extent to which past research has led to positive outcomes and impacts will be assessed. This analysis will likely be limited by the availability of evidence of impact. Furthermore, the evaluation of past research will include only research that has 19 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 continuing relevance to AAS (i.e. research transferred to AAS or relevant for the current program). Most of this research has been conducted by WorldFish. To the extent possible, the evaluation will assess emerging results, outcomes and influence of AAS since its beginning, and the approaches of documenting AAS results. Partnerships The evaluation will consider the partnerships among the implementing centers (WorldFish, IWMI and Bioversity International), linkages with other centers and CRPs, and with other research and development partners. It will look at co-researcher arrangements and how partner activities within AAS are funded and managed. The evaluation will consider issues such as coordination, decision- making, joint ownership of results and transaction costs, and assess equity, transparency, efficiency and effectiveness of partnerships. Gender As with all CRPs, AAS has a gender strategy that has been developed with the guidance of the Consortium Office. Furthermore, in AAS gender is one of the research themes. The evaluation will assess the implementation of the gender strategy and orientation and quality of gender-oriented research within the gender theme and across other themes and activities. Capacity building The evaluation will look at how capacity building is prioritized in order to address partners’ needs; the incorporation of capacity building into research activities for mentoring and enhancing the relevance and likely uptake of research results; the consideration of capacity issues among assumptions and risks related to the theories of change; and equity in targeting. Monitoring and Evaluation Monitoring and evaluation are part of the research implementation in AAS and thus the M&E design, indicators, and frequency and timing of use in adaptive management will be evaluated as part of the process. Novel aspects of M&E built within an action research paradigm and contribution to methodology literature will also be looked at in the evaluation. The methods used for monitoring and documenting AAS results will be assessed, including their compatibility with the action research approach being taken, the aspects of program design and implementation, for instance baseline studies that will enable impact assessment, and the resources allocated to documenting outcomes and impacts. 20 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Governance and management Governance and management – if functioning properly – lead to good organizational performance. This is one of the pre-conditions for the performance of AAS and its ability to produce international public goods and contribute to the IDOs. All CRPs including AAS function in the framework of the "new CGIAR“. This framework is of utmost importance for AAS and hence for this evaluation. As a first step the evaluation will accept this framework as given. In a second step however, it will – as part of the evaluation of the governance and management component – also include an analysis of the influence the framework and its CGIAR level implementation by the CGIAR Consortium and the Fund Council have on the likely success of the AAS. In order to facilitate the understanding and consistency across the CGIAR, this part of the evaluation will wherever possible and appropriate use the same terminology and criteria as the “Review of CGIAR Research Programs’ Governance and Management” (Final Report, March 2014). In line with this cross- CRP review, the following review criteria will be addressed: (i) legitimacy and participation, (ii) accountability, (iii) fairness and equity, (iv) transparency, (v) efficiency, (vi) effectiveness and (vii) independence. With these criteria in mind, the evaluation on governance aspects will focus on: (i) management oversight; (ii) stakeholder participation, (iii) risk management, (iv) conflict management and (v) audit and evaluation. In relation to management the evaluation will focus on: (i) priority setting and planning, (ii) regulatory compliance, (iii) reviewing and reporting, (iv) administrative efficiency, (v) internal and external communication and relationships, (vi) learning, (vii) financial management and (viii) human resource development and staff performance assessment. 5. EVALUATION APPROACH AND METHODOLOGY Evaluation approach and rationale As explained in Chapter 4, the evaluation approach was developed to address the specific nature of the AAS Research-in-Development approach, which describes itself in terms of "action research", "participatory action research", "participatory research" and "systems research". The evaluation approach also takes into account the fact that the program has a strong focus on qualitative research methods, which need to be evaluated differently from quantitative research approaches. It is also acknowledged that AAS is a CRP that is systems rather than commodity oriented, and that it focuses on system change, system productivity, sustainability and resilience combined with transformational social change. The evaluation approach addresses the fact that AAS aims to develop international public goods based on highly contextualized research activities taking place in specific localities. It also pays specific attention to the AAS principle of “commitment to place and people” which emphasizes the quality of the research process, in particular the quality of engagement with communities and partners (see AAS Approach Paper). The evaluation also takes into account the fact that AAS pursues a "gender transformative approach", which is meant to go beyond "mainstreaming" or simply considering men and women as stakeholder groups (see AAS Approach Paper, p.5). The evaluation approach reflects the intention to be consistent with (i) the principles for evaluation used in the systems, participatory and action research communities, and (ii) the principles established 21 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 for evaluating qualitative research methods (see, e.g. Bitsch, 2005). At the same time, it reflects the requirements of the Independent Evaluation Arrangement for the evaluation of CRPs (as stated in the Evaluation Policy, Guidelines and Standards28), which respond to the needs of the funding agencies and the principles of evaluation in international organizations (see, e.g. UNEG, 2005). In developing its evaluation approach, the team explicitly acknowledges that different disciplines and research communities have epistemological differences regarding what constitutes evidence, what methods are appropriate to generate and analyze evidence, and what inferences and insights can be drawn from different types of evidence and analysis. The approach to the evaluation outlined here takes into account the fact that, in principle, action research encompasses evaluation as an integral part of the research process. This is often described as the "observe" and "reflect" steps of the action research cycle. We recognize that AAS has put in place an M&E system and we assume that it will play an important role in this evaluation. The evaluation approach explicitly acknowledges the fact that AAS has operated for only three years and that for the scope of an external evaluation of the sort pursued here there are limitations in terms of time and resources. A major constraint in evaluating the action research of AAS is the limited time that the evaluation team can spend with communities and partners in the field. The evaluation has been designed taking into account the magnitude of the AAS program. As of July 2014, AAS has been in operation for three years, although some AAS research is a continuation of, or builds on, research undertaken in the pre-AAS period by the participating Centers, WorldFish in particular. The evaluation team acknowledges that, over the three years of operation, there have been some important changes to AAS, some resulting from the CGIAR requirements. These include the changing use of the term "Flagship", a shift from research themes to clusters, and a change in the way the term "hub" is used. Overview of the methodology The methodology developed for this evaluation can be described as “case-based, multi-level and mixed methods.” This consists of four main components: (1) A qualitative case study component, which focuses on an in-depth assessment of purposely selected AAS hub-level initiatives and research projects. These initiatives (referred to as “roll-out cases” or “roll-out activities”)29 and the (bilaterally funded) research projects will serve as cases to be studied using a mix of methods, including participatory methods (see below for more information). Special attention will be paid in the case studies to research methodology, results, dissemination techniques and interactions with communities and partners. The aim of the case studies is to provide insight into the overall RinD approach and into the Flagships and research themes. The cases will be carefully selected in collaboration with AAS Management Team to provide the best opportunities to learn how the AAS approach works in practice. (2) A document review component to provide evaluative information about the broader portfolio of 28 www.iea.cgiar.org 29 This terminology has been proposed by AAS Management with the aim to make it clear that the activities pursued according the AAS RinD approach are different from bi-lateral research projects. 22 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 AAS activities. This component will include a review of documents related to the overall AAS program, including governance and management; a sample of documents that relate to specific roll-out activities and projects, research outputs (e.g. peer reviewed journal articles, conference proceedings etc.), dissemination (e.g. success stories, policy papers, manuals, documentaries etc.) and impact. A combination of standardized format and qualitative evaluative assessment will be used. (3) A research staff survey, which will involve a questionnaire-based survey of the researchers involved in AAS, either through direct employment or through partner organizations that receive funding from AAS. The purpose of the survey is to offer a possibility to all researchers involved to reflect on their experience, and contribute their views and insights to the evaluation. (4) An interview component, which will be based on semi-structured interviews to provide insights into different areas of AAS complementing the case studies. The interviewees will include members of the governing bodies of and the management and administration of the AAS; members of the CGIAR governing bodies; partners and funders of AAS; leading academics with insights into AAS research; and long-term members and observes of the CGIAR system who are particularly familiar with the history of action research or comparative approaches in the system. This evaluation methodology relies on a combination of methods, which have been selected to address the specific nature of AAS. It will allow the evaluation team to strike an appropriate balance between (i) the goal of supporting AAS by creating opportunities for learning and for jointly identifying options for improvement, and (ii) the goal of providing guidance for the stakeholders in the CGIAR system in terms of accountability and funding decisions. Where possible and appropriate, the team will explore opportunities to link to other CRP evaluations so that a comparative dimension might be developed. For example, to the extent that it is appropriate, a common set of questions will be used in the staff survey. Methods of data collection and analysis Component 1: Qualitative case studies Rationale for the case study approach Qualitative case studies are a research design that is widely used in the social sciences, health sciences and other disciplines. The goal of the case study approach is to understand a phenomenon by conducting an in-depth and holistic study of one or more carefully selected cases. The case study approach involves more than just examining a single case or situation (cf. Baxter and Jack, 2008). In particular, it entails studying the selected cases (i) in their specific contexts, (ii) applying different disciplinary perspectives, and (iii) using different methods of inquiry. Both qualitative and quantitative data collection methods can be used within a case study methodology. Cases can be selected for a variety of reasons: because they provide for a particular contrast of interest; because they represent different stages; because they represent extremes etc. One of the most valuable aspects of case studies is that individual cases can be adequately contextualized. The case study approach has been acknowledged as being “useful for testing whether scientific theories and models actually work in the real world” (Shuttleworth, 2008). Definition of cases 23 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 For the AAS evaluation we will focus on two types of cases: "roll-out" cases and "research project" cases. These types of cases constitute the basic building blocks of the AAS program. We use the term roll-out case to refer to the hub-based efforts to formally implement the AAS approach. Primarily funded with W1/W2 resources, roll-out cases allow an exploration of the AAS approach in its purest form (i.e. not encumbered by the legacy of bilateral projects). There are three “first-generation” roll out cases (Bangladesh, Zambia, Solomon Islands) and two “second-generation” cases (Cambodia and Philippines). The basic documentation for the roll-out cases are activity plans which include detailed work plans that specify activities, deliverables and outcomes. There are currently a total of 78 activity plans. In addition there are a number of research reports that are relevant to the roll-out cases. We use the term research project case to refer to an individual research project undertaken by AAS with bilateral funds. These projects typically have specific documentation depending on the requirements of the funding agency (proposals, contracts, reports, etc., referring to objectives, methods, inputs and outputs). There are currently 53 active bi-lateral projects, the majority of which were designed before the initiation of AAS. The evaluation acknowledges that AAS may face limitations in implementing all elements of the AAS RinD Approach in all project cases. Yet, since they form a substantial share of AAS activities and funding, they must be a central element of the evaluation. Selection of cases In consideration of the AAS portfolio of activities and in consultation with AAS management, the countries where the evaluation will do field work and will collect information for the case studies are as follows: Table 5: Planned field visits30 Research hub Type Rationale Bangladesh First generation Hub with a lot of “legacy” bilateral projects, Cambodia Second generation New hub with high productivity related research share, observation of first phases of project implementation Solomon Islands First generation Hub with some “legacy” bilateral projects Zambia First generation Hub with the largest amount of W1/2 funding, particularly pronounced gender focus In addition to the roll-out cases the evaluation will look at a sample of 10 bilaterally funded research projects (i.e. research project cases; Table 6). Research project cases are selected for inclusion in the evaluation based on several criteria. The selection process was designed to: • Include projects identified by AAS management and staff as being particularly important, innovative and representative of the AAS approach; • Reflect the portfolio of different activities carried out in AAS with different funding sources; 30 In addition WorldFish headquarters in Penang, Malaysia, will be visited, primarily for purposes of the governance and management evaluation component). 24 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 • Include large projects that account for a substantial share of AAS funding sources; • Cover all research themes and the majority of flagships and hubs. Table 6: Bilateral projects suggested by AAS for review Research theme Project Title Budget Start date End date Funder Developing inland aquaculture in Solomon 0.279 1-Oct-11 30-Sep-15 Productivity, Islands SL3756ACI ACIAR Income and Myanmar fisheries CA4066ACI and 0.699 1-Sep-12 31-Aug-16 Nutrition BU10059 ACIAR Fisheries and nutrition BA3739IFA 0.500 n/a n/a IFAD Implementing an ecosystem approach to Resilience fisheries in small scale tropical fisheries 0.229 29-Dec-11 28-Dec-14 EC NR4046ECU Pacific fisheries (BU10275-278 0.145 1.182 1-Jul-13 30-Jun-17 ACIAR Scaling out community based marine resource governance in Solomon Islands, 0.324 17-Jun-11 16-Jun-15 ACIAR Kiribati and Vanuatu SL3766ACI Governance Strengthening aquatic resource 0.070 1-Apr-11 30-Jun-14 STARGO governance PE3708BMZ BMZ Wetlands Alliance SIDA Mekong 0.750 n/a n/a STARGO ME1625SID BMZ Implementation and analysis of roll-out cases The roll-out case studies will be conducted by at least two members of the evaluation team, who will visit the selected countries. The following methodological steps are envisaged (allowing for some case- specific modifications): • The case studies will start with an intensive review of the entire documentation available on the case (including activity plans, proposals, reports, outputs). Special attention will also be paid to carefully studying the data generated by the M&E systems and reviewing the outputs to date. • A Process Net-Map will be conducted with the AAS Country Manager and AAS Country Program Team as well as field staff. This mapping exercise, which is a participatory appraisal method, will help the team to better understand all steps involved in the selected roll-out or project case, to identify the stakeholders involved and to understand their role, and to identify issues to be further explored in the course of the case study. • Based on the Process Net-Map, interviewees will be identified, including researchers, partners, project beneficiaries and other stakeholders. The team will then interact with the identified interviewees using methods, such as focus group discussions, in-depth interviews and other participatory methods. The evaluators use instruments, such as interview guidelines, that will ensure comparability across cases, while allowing for sufficient flexibility in addressing case- specific issues. • The evaluators may also interact with organizations that are not part of AAS, but pursue similar activities in the respective case study areas. The objective of this interaction is to gain 25 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 comparative insights. These organizations may include national research organizations as well as NGOs operating in the same area and working on similar topics. • At the end of the field work, the evaluators will hold follow-up discussions with the project teams to discuss emerging findings and provide opportunity for feedback. The field visit will be approximately five days. The schedule for the field visits is presented in Annex 5. Team members will keep extensive notes of their interactions and observations and collect other type of documentation (e.g., maps from participatory exercises). Based on the field work, the team will first conduct within-case analysis. Each case study will be written up by the teams in a comparable format so that they can subsequently be used for a comparative cross-case analysis. The case study approach is open to discover aspects of AAS that were not anticipated in advance. At the same time, the analysis will seek answers to the following aspects of AAS research: • Integration of AAS approach with transferred research • Coherence of field activities with the principles of the AAS RinD approach • Knowledge gap(s) it seeks to address/formulation of research questions • Methodology and methods employed in the field (to be compared with the principles outlined in the AAS approach) • Inclusion of gender and equity approaches • Staffing and partnerships • Management processes in place • Progress to date and project outputs (to date and anticipated) • Outcomes (to date and anticipated) and their relationship with the AAS IDOs • Contribution to international public goods (to date and anticipated) Implementation and analysis of research project cases Each research project case studies will be conducted by two members of the evaluation team. The following methodological steps are envisaged (allowing for some case-specific modifications): • The case studies will start with an intensive review of the entire documentation available on the case (including proposals, reports, outputs). Special attention will also be paid to carefully studying the data generated by the M&E systems and reviewing the outputs to date. • As appropriate AAS research staff, partners and peers will be interviewed. Based on the documents and interviews the team will first conduct within-case analysis. Each case study will be written up in a comparable format so that they can subsequently be used for a comparative cross-case analysis. The case study approach is open to discover aspects of AAS that were not anticipated in advance. At the same time, the analysis will seek answers to the following aspects of AAS research: • Framing of the research • Knowledge gap(s) it seeks to address/formulation of research questions Choice of methodology and methods • Relevance of outputs • Quality of outputs 26 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 • Coherence within AAS and contribution to the AAS • Coherence of field activities with the principles of the AAS RinD approach • Contribution to international public goods (to date and anticipated) • Inclusion of gender and equity approaches • Staffing and partnerships Component 2: Publications review The AAS Annual Reports (2011-2013) include a list of publications or science outputs which are attributed to the program (Book chapters, Journal Articles, Manuscripts, Policy Briefs and reports). 31 The list of publications will be updated to the most recent point in time possible. Similar to the review of roll-out and project documents described above, the review of the publications will involve two steps: an assessment relating to all publications compiled in the above list, and a more in-depth assessment of sample of the publications. Assessment related to all publications This step will involve the following: • Classification of the publications by type (such as peer-reviewed journal articles, book chapters, books, manuals, guidelines policy briefs, etc.) • Classification of the publications by theme, flagship, country, hub and project; • In case of journal publications: Classification of the disciplinary or thematic orientation of the journal; distribution among journals indicating ranking among disciplinary category, article citations. This information will be analyzed with a view to providing an overview of the publication activity of AAS, to be disaggregated by flagships, themes and other criteria. In-depth assessment of a sample of publications A sample of the publications will be analyzed in more detail. Publications will be assigned to team members based on their area of expertise. This assessment will involve a scoring of the publications based on a set of criteria, which will be adjusted to the type of publication. As further outlined below, this component will provide an essential input in the evaluation of the quality of science. The total number of publications to be included in this sample will be decided after Step 1 is conducted and a better overview of the numbers of publications of different types is available. The sample will include: • All publications related to the case studies selected under Component 1 above. • All publications relating to the roll-out activities and projects that were randomly and purposely selected under Component 2 B above. • A random sample of additional publications, stratified by publication type. The justification for random sampling has been explained above. • Publications that are identified by AAS management as key publications of AAS and that have not been covered otherwise in the sample. Documents relating to the impact of AAS 31 CGIAR Research Output and Collaboration Study by Elsevier (2014) commissioned by the Consortium Office will provide an analysis of WorldFish publishing track record before AAS was established. 27 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 This evaluation acknowledges that a period of three years is too soon to assess impacts of a research program. However, to the extent possible on basis of available documentation, the evaluation will consider impacts from past research that has relevance to AAS (see section on Impact). The case studies (Component 1) will pay specific attention to assessing likely effectiveness based on the case study information. They should in particular allow for an assessment of the first two impact pathways specified by the project (see Figure 1 above). The review of the publications will contribute to an assessment of the third impact pathway. Regarding achievements to-date, AAS has already published documents with quantitative information on the effects of AAS activities claimed to be achieved. Taking this into account, the evaluation will include the following types of documents: • Narratives on impacts from WorldFish research relevant for AAS and from AAS activities (including early outcomes and influence); • Pre-AAS impact studies and AAS impact documentation as evidence supporting the narratives. The review of these documents will not only include an assessment of the type and magnitude of impact achieved or expected to be achieved, but also an assessment of the methods by which the impact assessments have been carried out. Component 3: Research staff survey The team will conduct a staff survey of all researchers employed by a WF, Bioversity and IWMI – or any other organization – who have some staff time paid for by AAS. The survey will be implemented electronically using an application such as the SurveyMonkey on-line tools, and its results will be presented I a way protecting the anonymity of the respondents. A special effort will be made to reach research staff from partner organizations who are involved in AAS research on the basis of collaborator contracts or other arrangements. We acknowledge that in an action research approach there is also a wide range of co-researchers at community level who cannot be reached through an electronic survey. Effort will be made to interview representative group of co-researchers during field visits. Component 4: Interviews AAS members, partners and experts This component includes interviews to be conducted as part of the case studies (Component 1), during field visits and through virtual means. Interviews will cover representatives of different stakeholder groups as described above in Section 5.2. The team will develop a list of the persons to be interviewed and interview guidelines for the different types of respondents. Detailed notes of every interview will be kept. The team will respect the right of any interviewee to remain anonymous. To the extent possible, personal interviews will be held during the visit to the AAS Headquarters in Penang, during the country visits or using other opportunities. Where this is not possible, interviews will be held by skype or phone, and these interviews will be timed mostly after the team has had the opportunity to observe AAS activities in the field and through the case studies. Methodological frameworks for selected topics The information and data collected through the four components outlined above will provide the basis for the evaluation of all evaluation criteria and topics. 28 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Below, the assessment frameworks are presented for four topics: quality of science, impact gender, and governance and management where consistency across CRP evaluations is being established. Quality of science assessment The framework for evaluating Quality of Science has four dimensions: (i) processes for assuring quality; (ii) input quality; (iii) output quality; and (iv) perceptions of quality. Processes for assuring quality This assessment will be done at the AAS program level with a focus on assessing how science quality is managed in AAS. The evaluation will look at all internal processes that are explicitly aimed at assuring quality. These may include: • Internal peer review processes; • Use of commissioned evaluations/reviews or external advisory groups; • Staff performance assessments (by participating centers); • Incentives and staff development aimed at enhancing science quality; • Mentoring and capacity development among co-researcher groups in the flagships; Input quality This assessment will be done at the level of the Flagships projects and cover both roll-out activities funded under the W1/W2 windows as well as bilaterally-funded projects. The case studies (Component 1), the sample-based document review (Component 2.2), and the staff assessment (Component 3) will play an important role in this regard. The assessment aims at identifying variability within the CRP, highlighting areas of excellence and identifying areas where improvements could be made. ISPC comments on science quality in relation to the original 2016-2016 extension proposals will be taken into account. The analysis will take the specific requirements of the AAS RinD approach into account and focus on the following aspects: • Track record and competence of team leaders (using, for example, h-index); • Composition and competence of teams; • Quality of research proposals; appropriateness and innovativeness of research designs; and • Quality of data collection and management. For research staff employed by the CGIAR centers who spend a substantial share of their time on AAS (to be specified, depending on the available information), information will also be collected from CVs. This information will include education (level, discipline) and length of professional experience. Together with information from the staff survey, this will contribute to assessing the human resource capacity available for AAS. Output quality This assessment will be done at the level of the Flagship projects and of the overall program. The evaluation will look at both the quantity and the quality of outputs, covering major types of outputs including those that resulted from research by participating Centers, when relevant. The review of outputs of the sampled projects and review of sample publications will contribute to this assessment. As in case of input quality, the assessment aims at identifying variability within CRP, highlighting areas of excellence and identifying areas where improvements could be made. 29 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Perceptions of quality The evaluation will also draw on perceptions of the quality of AAS research outputs, particularly among peers in both the social and technical sciences. Other aspects will include the overall reputation for science quality. Comparisons will be primarily internal, looking at AAS components; and, to the extent possible, also external, looking at suitable comparator organizations and against peer expectation of quality of international research of excellence, taking the specific nature of the AAS approach into account. Impact assessment Impact here is defined to include results along the impact pathway beyond delivery of outputs; e.g. adoption, influence, outcomes and longer-term impacts towards the CGIAR goals. These are mostly related to research done by the lead center prior to AAS on “transferred” research but may include early outcomes from AAS activities. AAS will be asked to develop a narrative around relevant "impacts" of past research that are relevant to AAS. The narrative should incorporate claims made regarding the volume, scale and level (along the impact pathway) of the results which can be supported by evidence. The cut-off date will be 2009, the date of the most recent External Program and Management Review (EPMR) of WorldFish. Different data collection methods will contribute to the impact assessment, as has been indicated above (see Component 2.4). These narratives will be assessed against the available evidence. The evaluation will also assess the effort of documenting outcomes and impact, and the overall scale of impacts from past research. Gender assessment AAS has made gender a theme in its own right (Theme 4: Gender Equality) and it is expected that gender will be integrated into all of the other themes. With the ultimate goal of gender transformation, AAS has aimed to undertake strategic research to facilitate change in gender norms, attitudes and practices. It has also supported innovation and experimentation to operationalize concepts and develop tools and systematically test interventions to understand what works. The evaluation of the gender component will focus on a study of methodologies and tools that have been used, results already obtained and potential for longer-term impact. The approach will include the following: i. Qualitative case studies - all case studies will include a gender component, examining the extent to which AAS has integrated gender concerns into research design and research outcomes. In addition, at least one case study will focus specifically on gender research, giving special attention to examples of innovation and experimentation. ii. Document review – all document review will take note of the extent to which gender concerns have been integrated into the work of AAS. Gender publications will be critically reviewed. iii. Staff assessment – staff competency in the area of gender research will be assessed. Special attention will be given to efforts to build staff competency in gender research. iv. Interviews – key researchers and partners, will be asked about the gender-related work that has been done by AAS and to give their own assessment of strengths and weaknesses. Questions on gender will be integrated in the interview templates. 30 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 v. Governance and management –attention will be given to the management of AAS and the extent to which both sexes are represented in management structures and the types of opportunities (e.g. for training or capacity-building, participation in meetings, etc.) given to both sexes. Governance and management assessment As specified in Section 4, this component will address both governance dimensions (including oversight, stakeholder participation and risk and conflict resolution) as well as management dimensions (including planning, reporting, and financial and human resource management). The governance focus will be on the internal and external bodies and functions (including their interactions) responsible for AAS strategic direction and oversight. The management focus will be on the organizational, financial and human resource aspects of AAS. The evaluation will first explore governance and management aspects in the design of AAS, including structures and processes, and secondly the actual governance and management in AAS, based on an analysis of meeting minutes, field visits, interviews and survey results. The data collection methods described in the four components above will contribute to the governance and management review, as follows: • Component 1: In the case studies, governance and management aspects will be examined as part of the case study approach. The team members will share a joint list of issues related to governance and management that are to be covered as part of the case studies. • Component 2: The documents referring to the AAS program in general (see Component 2.1 above) will be analysed with regard to the way in which governance and management aspects, including structures and processes, are designed in AAS (like ToR of governance and management bodies and meeting minutes). The documents to be reviewed regarding the roll- out activities and projects (see Component 2.2) will also be examined with regard to governance and management issues. In particular, the in-depth review of the project documents for the selected sample (see Component 2.2.2) will be analysed with regard to governance and management questions, including questions such as priority setting, regulatory compliance, reporting, etc. Financial reports and related information, such as audit reports, will also be reviewed. • Component 3: This component will provide important information on human resource management. A respective set of questions will be included in the staff survey (Component 3.1). The compilation of information from the CVs (Component 3.2) will also provide information on human resource management. • Component 4: Governance bodies, management and administration of AAS will be included in the interviews. The selection of the interviewees will ensure that all aspects, including financial management, are adequately captured. The team will also explore to what extent it will be possible to draw on two surveys that have already been conducted outside this evaluation: one with CRP governance bodies, and the other one with CRP management. For AAS, 10 persons have participated in the governance survey and probably more in the management survey. In case that it is technically possible and ethically acceptable (given the small numbers of respondents), the answers of the AAS respondents will be retrieved and analyzed to get AAS-specific results from those surveys, including open-ended answers. 31 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Beyond the CRP Governance and Management Review, this evaluation will draw from other documents including: (i) CGIAR Common Operational Framework; (ii) CGIAR Strategy and Results Framework (2011), (iii) CGIAR Governance Review (2013), (iv) The CGIAR’s Challenge Program Experiences: A Critical Analysis. A contribution to Consortium and Mega-program design (2009), and (v) IEG/World Bank: Sourcebook for Evaluating Global and Regional Partnership Programs (2007). 6. LIMITATIONS OF THE EVALUATION Due to the limited time that AAS has been in operation, the evaluation has only a relatively short time for assessing program performance and achievements. The ability to assess achievements and impact from transferred research relevant to the current CRP may be limited by the lack of evaluative information. The evaluation team also needs to judge the extent to which past success (or lack of it) is relevant to the likelihood of success of AAS. The geographic spread and heterogeneity of the geographical Flagship projects may limit the ability of the evaluation team to collect information at a desirable level of detail. 7. ORGANIZATION AND TIMING OF THE EVALUATION Team Composition and Responsibilities Team members, their primary area of responsibility and the research sites to be visited are given in (Table 6). As is evident from the biographical sketches in the next section, the team members bring complementary and to some degree overlapping backgrounds, skills and experience to the evaluation. It is expected therefore that in addition to their primary area of responsibility, team members will contribute to other areas of the evaluation as appropriate. In addition individual team members will be given responsibility for drafting specific sections of the report, in addition to contributing to the report more generally. Table 6: Team composition and primary responsibilities Team Member Primary responsibility for Sites to be visited • Bangladesh Regina Birner Co-leading evaluation team Research methodology & program coherence • Zambia • WF HQ Co-leading evaluation team • Cambodia James Sumberg • Solomon Islands AAS approach & science quality • WF HQ Ram Bhujel Research on technology & productivity enhancement • Bangladesh • Cambodia Nadarajah Sriskandarajah Action research & partner / stakeholder engagement • Bangladesh 32 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Eva M. Rathgeber Research on gender & social transformation • Bangladesh • Zambia Felix von Sury Governance & management • Bangladesh • WF HQ Team member profiles are given in Annex 4 Evaluation governance – roles and responsibilities The evaluation Co-Team Leaders have final responsibility for the Evaluation Report and all findings and recommendations therein, subject to adherence to IEA Evaluation Standards. They are responsible for submitting the report and any other deliverables, as outlined in the detailed Timeline later in this chapter. IEA is responsible for initial planning, initiating, and managing the evaluation. The IEA will also be responsible for the quality assurance of the evaluation process and outputs, and for the dissemination of the results. The IEA will take an active role in the preparatory phase of the evaluation by collecting background data and information and by carrying out preliminary analysis on AAS. An Evaluation Manager, supported by an Evaluation Analyst, will provide support to the team throughout the evaluation. AAS management is responsible for responding to the evaluation team’s informational needs. It provides documentation and data, information on all AAS activities, access to staff for engagement with the evaluators, and information on partners and stakeholders. It facilitates arrangement of site visits and appointments within the lead Center and other stakeholders. AAS management is also responsible for giving factual feedback on the Draft Report and for preparing the Management Response to the Final Report. It assists in dissemination of the report and its finding and lessons and it acts on the accepted recommendations. While the evaluation is coordinated with AAS management, WorldFish as the lead Center is a key stakeholder in the evaluation. It hosts visits to the Center and its leadership and board are expected to make themselves available for consultations during the evaluation process. Stakeholder Involvement As appropriate the evaluation will use participatory approaches involving a variety of stakeholders. Both internal stakeholders (including staff from AAS, World Fish and other CGIAR centers), external stakeholders (including AAS partners and beneficiaries) will be included in these exercises. A Reference Group has been set-up to work with the IEA Evaluation Manager to ensure good communication with, learning by, and appropriate accountability to primary evaluation clients and key stakeholders, while preserving the independence of evaluators. The Reference Group provides views and inputs at key decision stages in the evaluation design and implementation process, including for the Terms of Reference, the Inception Report and the Draft Report. The Reference Group may also play an important role in leading evaluators to key people and documents. Members of the Reference Group are shown in Annex 4. 33 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 Timeline and key deliverables The evaluation schedule is given in Table 6. This Inception Report builds on the original Evaluation Terms of Reference and constitutes the guide for conducting the evaluation. The Evaluation Report is the main output of this evaluation. It will describe the methods, findings, conclusions and recommendations. The recommendations will be evidence-based, relevant, focused, clearly formulated and actionable. They will be prioritized and addressed to the different stakeholders who are in a position to implement them. The main findings, conclusions and recommendations will be summarized in an executive summary. Table 7: Evaluation schedule Phase Period Main outputs Responsibility Preparatory Phase Feb 2013 – May 2014 Final ToR Evaluation team recruited IEA Inception Phase Jun 2014 – Aug 2014 Inception Report ET + IEA Inquiry phase Sep 2014 – Dec 2014 Various reports and analyses ET Presentation of preliminary Presentation of preliminary findings Dec 2014 findings ET Feedback from main IEA stakeholders Reporting phase Drafting of report Jan 2015– Feb 2015 Draft Evaluation Report ET Final Evaluation Report Mar 2015 Final Evaluation Report ET Management Response Apr 2015 Management Response CRP Management IEA Dissemination phase May 2015 Communications products Co-Team Leaders AAS Management (ET = Evaluation Team) 34 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 ANNEX 1: REFERENCES METHODOLOGY Anderson, J. R., Herdt, R. W., & Scobie, G. M. (1988). Science and Food: The CGIAR and Its Partners. Washington, DC: Published for the Consultative Group on International Agricultural Research; World Bank. Becker, T. (2000). Participatory Research in the CGIAR. Discussion Paper Prepared for the NGO- Workshop "Food for All - Farmer First in Research" accompanying the GFAR 200 in Dresden. URL: www.agrecol.de/?q=node/243 (Accessed March, 2014) Bitsch, V. (2005). Qualitative Research: A Grounded Theory Example and Evaluation Criteria. Journal of Agribusiness, 23(1), 75–91. Baxter, P and Jack, S. (2008) "Qualitative Case Study Methodology: Study design and implementation for novice researchers", in The Qualitative Report, 13(4): 544-559. UNEG (2005). Norms for Evaluation in the UN System. United Nations Evaluation Group (UNEG). New York. 35 Independent iea.cgiar.org Evaluation Arrangement Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 ANNEX 2: EVALUATION MATRIX Areas of interest Specific questions Sources of evidence Overarching 1. What in the AAS approach to research is new, innovative and/or unique in the These questions cut across the criteria and are questions context of agricultural research and the CGIAR? to be addressed through synthesis of the data 2. How does the AAS approach to research draw on earlier and on-going work, analysis shown below and the evaluation especially participatory action research approaches, from inside and outside findings and the CG? 3. What progress has been made so far in developing and implementing the AAS approach to research? 4. What are the challenges faced while implementing the AAS approach and are there any missing links or actions, which could help to make the AAS approach more effective? 5. To what extent is the AAS approach better suited than other research approaches to meet the challenges faced by people living and working in AASs? To what extent is it better suited to reach marginalized groups and women and to address the inequities they face? 6. Is the AAS approach to research, as it is being implemented in the various hubs, likely to deliver international public goods and the identified IDOs? 7. What are the comparative advantages and the added value of the CGIAR via AAS in developing and implementing this approach to research in aquatic agriculture systems? 8. To what extent is the AAS approach a model for other CGIAR research programs, especially with regard to the CGIAR reform goal to move from a “supply-driven” to a ”demand-driven” mode of research? 9. How do partners and networks understand, and are likely to make contributions to the implementation of the AAS approach for producing desired outcomes and achieving objectives? Research / Programme Performance 36 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 • Review of documents such as the Strategy 1. Is the AAS research programme internally coherent and relevant to the SRF? and Results Framework, AAS proposal and 2. Are the research problem identified and outputs relevant to the IDOs? planning documents, including on IDOs, 3. Are the research outputs and IDOs relevant to the development challenges facing Relevance ISPA appraisals people who live and work in aquatic agriculture systems? 4. Are the Theory of Change and impact (scaling) pathways relevant and convincing? • Review of a sample of roll-out activities 5. How much does the AAS research draw on research generated in the CGIAR system and projects and elsewhere? • Field visits • Mgt, staff & partner interviews • Output database (in-depth review of sample of outputs) • Review of documents, such as ISPC and CP 1. Does AAS create the conditions and incentives, and does it have the facilities and appraisals internal processes, to ensure high quality scientific output? • Review of a sample of roll-out activities ad 2. Does AAS have the number, quality and level of researchers and research leaders projects to deliver relevant, high quality scientific output? Quality of science • Bibliometric analyses 3. Are research issues, questions, hypotheses and designs clearly and appropriately articulated and informed by current scientific literature? • CGIAR Research Output and Collaboration 4. Do the research outputs reflect the level of expenditure, and are they of high Study 2014 by Elsevier quality? • Analysis of staff track record 5. What is the value added to AAS research outputs of the AAS research approach? • Field visits • Staff survey • Expert interviews • Mgt, staff & partner interviews 1. Is AAS likely to be more effective in developing and demonstrating an alternative • Output database (in-depth review of approach to development-oriented agricultural research? sample of outputs) 2. Is AAS likely to be more efficient in delivering international public goods (i.e. through Impact/Scaling pathway 3)? • Review of documents such as Annual Likely 3. Is AAS likely to be effective in contributing to the IDOs it has identified (i.e. through reports, progress reports, M&E effectiveness Impact/Scaling pathways 1 and 2)? documentation 4. Is AAS likely to be effective in fostering and contributing to "transformational • Review of impact pathways and theories change"? of change 5. Have the potential factors identified which may obstruct AAS to be effective? • Review of a sample of rill-out activities 6. What evidence or early indications are available to indicated potential or likely and projects 37 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 impact of AAS research? • Field visits • Staff survey • Expert interviews • Mgt, staff & partner interviews 1. To what extent has AAS been influential and generated impacts to-date? • Review of evidence of achievements and Impact impact 2. To what extent have impacts been demonstrated from "transferred research"? • M&E system • Mgt, staff & partner interviews Other areas • Review of AAS gender strategy and other 1. To what extent has strategic research in gender facilitated change in norms, documentation attitudes and practices? • Review of a sample of roll-out activities 2. To what extent is AAS operationalizing gender concepts and developing new tools? and projects Gender 3. Is AAS using innovative approaches and if so, are they systematically testing • Field visits innovations to understand what works? • Focus groups 4. What is the potential for replication and scaling-up of AAS’s gender research? • Staff survey/ interviews • Expert interviews • M&E system 1. Are the criteria or scrutinizing process used appropriate for selecting partners? • Documentation on partnerships 2. Has AAS identified new partners with whom WorldFish has not worked in the past? • Review of a sample of roll-out activities Partnership 3. Are the partnerships appropriate for the research? and projects 4. Are the partnerships effective in delivering the research? • Field visits 5. Are the partnerships satisfactory to the partners? • Focus groups 6. Are the levels of collaboration within AAS and with other CRPs appropriate? • Mgt, staff & partner interviews 38 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 1. To what extent do AAS capacity development strategy and activities address partners’ needs? 2. How effectively does AAS capacity development address the needs of different social and professional categories? 3. • Review of documents and records on Are capacity issues and constraints adequately considered in the impact pathway Capacity building capacity development analysis? 4. Are the regular needs assessment and capacity development processes in place? • Focus groups 5. Have the capacity development activities been relevant and effective? • Mgt, staff & partner interviews 6. How does AAS capacity development address the "capacity to innovate"? 7. Does AAS create opportunities for MA/MSc and PhD students? 8. Are capacity building opportunities equally available to men and women? Management & governance • Review of agreement documents, terms of reference and membership or different oversight and management bodies 1. Are the governance and management arrangements appropriate (e.g. in terms of • Review of meetings minutes, including the legitimacy, accountability, transparency)? lead Center Board, AAS governing bodies 2. Are the governance and management arrangements effective? etc. 3. Are the governance and management arrangements efficient? Management & 4. Do those involved in governance and/or management of AAS perceive that the • Review of financial documents, audit governance reformed CGIAR organizational structures and processes have facilitated reports, documents related to resource arrangements implementation? allocation 5. Do both men and women have a strong voice in management and governance of • Review of relevant policies AAS? • Staff survey 6. To what extent does AAS have good financial management, budgeting and • Interview including relected Fund Office reporting? and Consortium Office staff, members of AAS oversight and management bodies, staff in participating centers and partner orgnaizations, theme and Flagship leader 1. Is the M&E system in place and functional? Are there any unique or novel • Review of progress reports and other The M&E system aspects to the M &E systems in place within an action research approach? documentation 2. To what extent is the M&E system generating relevant information on • Mgt & staff interviews 39 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 progress and achievement and is that information effectively used? 3. What role does the M&E system play in validating impact claims? 4. What role does the M&E system play in extracting international public goods from the programme's place-based research? 40 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 ANNEX 3: LIST OF PERSONS CONSULTED IN INCEPTION PERIOD Name Organization Position Peter Gardiner ISPC Executive Director James Stevenson ISPC Agricultural Research Officer Patrick Dugan WorldFish AAS Director Charles Crissman WorldFish M&E Leader Boru Douthwaite WorldFish Program Leader, Knowledge sharing and Learning Bill Downing WorldFish Head, Operation and Program Support Unit Ranjitha Puskur WorldFish Program Leader, Gender & Equity Sonali Senaratna IWMI Organizational Representative Andrea Roderick CARE Organizational Representative Fabrice De Clerk Bioversity International Organizational Representative Rachel Sauvinet-Bedouin IEA Director Kwesi Atta-Krah HumidTropics CRP Director 41 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 ANNEX 4: EVALUATION TEAM PROFILES TEAM CO-LEADERS Regina Birner has been the Chair of Social and Institutional Change in Agricultural Development at the University of Hohenheim (Germany) since 2010. She has more than 20 years of experience in agricultural research for development and has conducted numerous research projects in Asia and Africa. She was the leader of IFPRI’s Research Program on “Governance for Agricultural and Rural Development”. Dr. Birner has acted as advisor to many international organizations, including the World Bank, FAO and USAID and participated in evaluations. In 2008 she served in the core author team for the World Development Report. Her research focuses on socio-economic issues in the context of agricultural development, including topics such as participatory research, institutions, knowledge and innovation, gender. Dr. Birner has a PHD in Rural Development from University of Göttingen. James Sumberg is a Research Fellow at the Knowledge, Technology and Society Team, Institute of Development Studies (IDS) at University of Sussex. He is an agriculturalist by training, with PhD from Cornell University on Plant Breeding and Animal Nutrition, and has over 25 years of experience of research on small-scale agriculture, natural resource management, agricultural research policy, and food and rural development in tropical regions, with a particular emphasis on sub-Saharan Africa. He has participated in evaluation of agriculture and natural resource management projects. His past work experience includes, among other, The New Economics Foundation in London, University of East Anglia (Senior Lecturer in NRM) and CARE, and brief periods in WARDA and CIAT. He has published on change in agricultural systems, innovation and policy. TEAM MEMBERS Ram Bhujel is currently Senior Scientist and Affiliated Faculty Coordinator for the Aqua-Internship and Training Program in Aquaculture and Aquatic Resources Management at Asian Institute of Technology (AIT). He has long standing experience in the field of small—scale and commercial aquaculture and integrated farming with vegetables and livestock. He has been working as project coordinator for several aquaculture development projects in Asia and is currently coordinating the Agricultural Learning Experience for Asian Regional Networking (AgLEARN) project funded by USAID/RDMA and the Development Rural Aquaculture through Entrepreneurship in Women in Myanmar (eWomen) funded by the Foreign Ministry of Italy. He also served in several community and industry engagement projects. Mr. Bhujel has a PhD in aquaculture. Nadarajah Sriskandarajah (Sri) has been Professor in Environmental Communication at the Swedish University of Agricultural Sciences in Uppsala, Department of Urban and Rural Development since March 2007. Sri has held university positions in Sri Lanka, Australia, Papua New Guinea and Denmark and worked on projects in a number of countries in Asia, Africa, Latin-America and Europe. His research deals with the challenge of complexity at the human-nature interface, within a variety of contexts ranging from farming and food systems to forestry and nature conservation. His 42 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 approach is essentially systemic, inter-disciplinary and learning-oriented with action research as an apt descriptor to much of his recent work. He has published widely in the field of action research, learning and farming systems. He has a PhD in Animal Husbandry from University of Sydney, Australia. Eva M. Rathgeber is a consultant in international development. From 2002-2006 she held the Joint Chair of Women’s Studies at the University of Ottawa/ Carleton University in Ottawa, Canada and she continues to serve as an adjunct professor at both universities. She spent many years with the International Development Research Centre (IDRC), starting as a program officer in science and technology, and later became founder and director of IDRC’s Gender and Development Program. From 1992-2001, she was IDRC Regional Director for Eastern and Southern Africa, based in Nairobi, Kenya. She continues to work closely with many international organizations, including FAO, IAEA, UNESCO, the African Virtual University, and the CGIAR system. She has published widely on science and technology policy, knowledge production, gender and development, and gender and natural resource management. She currently serves as chair of the Steering Committee of the Gender and Water Alliance. Felix von Sury has extensive experience in international and development cooperation. He served for 13 years in the SDC, Swiss Agency for Development and Cooperation, Swiss Foreign Ministry, where he was among others Country Director for Nepal and Division Head for Eastern Europe. From 2000 until 2011 he was Executive Director of Intercooperation, a major Swiss development NGO active mainly in the fields of renewable natural resources, agriculture, forestry and climate change. Long-term assignments have taken him to Peru, Australia, Nepal and India. Since 2012 von Sury has been a freelance consultant. He has a PhD in Agricultural Science from ETH Zurich. 43 Evaluation of CRP Aquatic Agriculture Systems, Inception Report, OCT 2014 ANNEX 5: MEMBERS OF EVALUATION REFERENCE GROUP Name Position/role Patrick Dugan AAS Director WorldFish Board Chair; former Deputy Director-General of the Swiss Remo Gautschi Agency for Development and Cooperation (SDC) of the Department of Foreign Affairs John Lynam POP member, ICRAF Board Chair; Independent consultant, agricultural economist POP member, IWMI Board member; Director, Water Resources & Barbara Schreiner Management Consultant Pegasys Strategy and Development South Africa John Mayne Independent advisor, evaluation Research Secretary Philip Chiverton Unit of Research Cooperation Swedish International Development Cooperation Agency (Sida) SE-105 25 STOCKHOLM, Sweden Ann Waters-Bayer Agricultural sociologist, ETC-International; research partner Jürgen Anthofer Executive Secretary, EIARD Rieky Stuart Executive Director, Oxfam, Canada; gender perspective 44 ANNEX 6: FIELD VISITS Schedule of field visits Evaluation team member Country Flagship project Sri Eva Ram Felix Regina Jim Asian Mega Deltas Bangladesh Southern Polder Early Early Early Early Early Early Zone Nov Nov Nov Nov Nov Nov Cambodia Tonle Sap Mid Nov Mid Nov Asia-Pacific Islands Solomon Malaita & Western Islands Provinces Mid Nov African Inland Zambia Barotse floodplain Early Oct Early Oct Global Global Penang Early Early Nov Nov