Evaluation of the “Strengthening Impact Assessment in CGIAR” (SIAC) Project Phase 1, 2013-16 Volume 1 -Evaluation Report December 2016 Julia Compton Timothy J. Dalton Sophie Zimm This evaluation has been commissioned by Standing Panel on Impact Assessment (of ISPC) and is managed by the Independent Evaluation Arrangement (IEA) of CGIAR. The Independent Evaluation Arrangement (IEA) of CGIAR encourages fair use of this material provided proper citation is made. Correct citation: CGIAR-IEA (2016), Evaluation of the “Strengthening Impact Assessment in CGIAR” (SIAC), Project Phase 1, 2013-16. Rome, Italy: Independent Evaluation Arrangement (IEA) of CGIAR. http://iea.cgiar.org/ Cover image credit: IRRI SIAC Evaluation - Report Table of Contents Acknowledgements .................................................................................................................................. iii Acronyms and Key Terms ......................................................................................................................... iv Executive Summary ................................................................................................................................... vi Background .......................................................................................................................................... vi Methodology and limitations of the evaluation ..................................................................................... vi Main findings and conclusions .............................................................................................................. vi Recommendations .............................................................................................................................. viii 1. Introduction ......................................................................................................................................... 1 1.1 Origins of the evaluation .......................................................................................................... 1 1.2 Structure of the report .............................................................................................................. 1 1.3 Evaluation Purpose, Objectives and Clients ............................................................................... 1 1.4 Scope of the Evaluation ............................................................................................................ 2 1.5 Evaluation approach and questions........................................................................................... 2 1.6 Evaluation approach, methods and analysis .............................................................................. 2 1.7 Main limitations of the evaluation and changes from the Inception Report ............................... 3 1.8 The CGIAR institutional context ................................................................................................ 4 1.9 Overview of SIAC...................................................................................................................... 5 1.10 Project objectives and intervention logic ................................................................................... 6 2. Main findings ....................................................................................................................................... 9 2.1 Relevance ................................................................................................................................. 9 2.2 Effectiveness ........................................................................................................................... 22 2.3 Quality Of Science .................................................................................................................. 29 2.4 Management and Governance ............................................................................................... 31 3. Conclusions ........................................................................................................................................ 42 4. Recommendations .............................................................................................................................. 43 5. References cited in the text................................................................................................................. 46 Tables Table 1: Summary findings on relevance of SIAC design to SPIA mandate .............................................. 21 Table 2: Summary ‘traffic light’ table of progress against SIAC outputs .................................................. 23 Table 3: Example of fragmentation: the full list of topics for funded proposals in the SIAC nutrition call . 27 i iea.cgiar.org SIAC Evaluation - Report Figures Figure 1: Organogram of SIAC and SPIA in the CGIAR (Phase I).............................................................. 5 Figure 2: SIAC has dramatically increased central resources for IA: SPIA funding, 2007-2016 .............. 6 Figure 3: SIAC initial and current budget by SIAC Objective (USD thousands) ....................................... 7 Figure 4: Summary logic model for the SIAC project .............................................................................. 8 Figure 5: SIAC and CGIAR Research Program Phase 2 average annual budgets for impact assessment ............................................................................................................................................ 12 Figure 6: SIAC project logic model from proposal document, with annotations by evaluation team . 18 Figure 7 Adoption rates are difficult to interpret from a snapshot survey: Use of leading rice varieties in Indonesia, 1985-2009 ....................................................................................................................... 19 Figure 8: SIAC outsources much of its research management to trusted partners: Distribution of budget across main partner types ........................................................................................................ 26 Figure 9: Percentage of research proposals examined with given characteristics ............................... 31 Figure 10: Timing for key stages of major SIAC research calls .............................................................. 34 Figure 11: Geographic distribution of lead partners in SIAC study contracts ....................................... 35 Figure 12: Average number of invitations, applications and funded proposals for 3 SIAC calls .......... 36 Boxes Box 1 The composition and ToR of the PSC ......................................................................................... 37 Annexes (in Volume 2) Annex 1: List of stakeholders consulted Annex 2: Evaluation team composition Annex 3: Timeline of key events relating to the SIAC project Annex 4: What is “impact assessment” and why is terminology important? Annex 5: What do donors want from impact assessment in the CGIAR? Annex 6: Brief findings on the organisation of IA in CRPs and Centers Annex 7: Independence, impartiality and overcoming bias in impact assessment Annex 8: Progress against SIAC outputs Annex 9: Science quality assessment ii iea.cgiar.org SIAC Evaluation - Report Acknowledgements We are very grateful to everyone who has taken the time to speak to or email us, share views and information, and comment on drafts of this report (Annex 1). In particular we would like to thank Douglas Gollin, SPIA Chair, and Tim Kelley, Lakshmi Krishnan, James Stevenson and Ira Vater at the SPIA secretariat for their enthusiastic collaboration and support for this evaluation. Special mention is also due to Birte Snilsveit of 3iE who went out of her way to look up information for us. We also thank Rachel Sauvinet-Bedouin, Evaluation Manager (and Head of IEA) for very helpful guidance and comments on the draft, and Deborah Basilici, IEA Administrator, and Jenin Assaf, Communications Manager, for their kind assistance. iii iea.cgiar.org SIAC Evaluation - Report Acronyms and Key Terms ARI Advanced Research Institution BMGF Bill and Melinda Gates Foundation CO CGIAR Consortium Office (replaced by the System Management Office on 1 July 2016) CGIAR No longer an acronym. Used to stand for Consultative Group for International Agricultural Research CoP Community of Practice CRP CGIAR Research Program DFID UK Department for International Development DIIVA Diffusion and Impact of Improved Varieties in Africa (BMGF funded) EIAC Evaluation and Impact Assessment Committee EoI Expression of Interest (for a competitive grant) epIA ex-post Impact Assessment FC Fund Council of the CGIAR (replaced by the System Council on 1 July 2016) FAO Food and Agriculture Organization of the United Nations IA Impact assessment IAFP Impact Assessment Focal Point IEA Independent Evaluation Arrangement of the CGIAR ISPC Independent Science and Partnership Council of the CGIAR LSMS-ISA Living Standards Measurement Survey – Integrated Surveys on Agriculture (of the World Bank) LTLS Long-term large scale ex post IA studies MSU Michigan State University, USA NARS National Agricultural Research System NRM(R) Natural Resources Management (Research) POR(IA) Policy-Oriented Research (Impact Assessment) R&D Research and Development RCT Randomized Control Trial (experimental method of impact assessment) RRO Regional Research Organization SC System Council of the CGIAR (replaced the Fund Council) SIAC Strengthening Impact Assessment in the CGIAR SMB System Management Board (SMB) of the CGIAR (replaced the Consortium Board) SPIA Standing Panel on Impact Assessment (of ISPC) STMS Short-term micro scale impact studies using experimental & quasi experimental methods Tbc to be confirmed ToR Terms of Reference VOIP Voice Over Internet Protocol (Skype or similar) iv iea.cgiar.org SIAC Evaluation - Report W1 ‘Window 1’ funding from Fund Council (not normally tied by donors to any research program) ARI Advanced Research Institution BMGF Bill and Melinda Gates Foundation CO CGIAR Consortium Office (replaced by the System Management Office on 1 July 2016) CGIAR No longer an acronym. Used to stand for Consultative Group for International Agricultural Research CoP Community of Practice CRP CGIAR Research Program DFID UK Department for International Development DIIVA Diffusion and Impact of Improved Varieties in Africa (BMGF funded) EIAC Evaluation and Impact Assessment Committee EoI Expression of Interest (for a competitive grant) epIA ex-post Impact Assessment FC Fund Council of the CGIAR (replaced by the System Council on 1 July 2016) FAO Food and Agriculture Organization of the United Nations IA Impact assessment IAFP Impact Assessment Focal Point IEA Independent Evaluation Arrangement of the CGIAR ISPC Independent Science and Partnership Council of the CGIAR LSMS-ISA Living Standards Measurement Survey – Integrated Surveys on Agriculture (of the World Bank) LTLS Long-term large scale ex post IA studies MSU Michigan State University, USA NARS National Agricultural Research System NRM(R) Natural Resources Management (Research) POR(IA) Policy-Oriented Research (Impact Assessment) R&D Research and Development RCT Randomized Control Trial (experimental method of impact assessment) RRO Regional Research Organization SC System Council of the CGIAR (replaced the Fund Council) SIAC Strengthening Impact Assessment in the CGIAR v iea.cgiar.org SIAC Evaluation - Report Executive Summary Background This is an evaluation of the first phase of SIAC: a project with the aim and title of Strengthening Impact Assessment in the CGIAR (a global consortium of international agricultural research centers). SIAC funds the work programme of an existing CGIAR institution: the Standing Panel for Impact Assessment (SPIA). The evaluation has two main objectives: for SPIA to demonstrate accountability to SIAC donors for Phase 1 (2013-2016); and to draw lessons and make recommendations that will inform the second phase of SIAC and future directions of SPIA. The primary users of this evaluation are SPIA and its parent body, the CGIAR Independent Science and Partnership Council (ISPC); the two donors supporting SIAC; and the CGIAR System Council. There are also other important stakeholders, including CGIAR research leaders in Centers and CGIAR Research Programs (CRPs) and CGIAR Impact Assessment Focal Points (IAFPs). Methodology and limitations of the evaluation The evaluation addresses four main areas: project relevance and design, effectiveness, quality of science, and management and governance. The main methods were document review and analysis, semi-structured interviews with 51 individual stakeholders, three IAFP discussion groups, and observation of two global SIAC meetings. We also asked the SIAC team to carry out some self-evaluation, with the results then triangulated by the external evaluation team. The Head of IEA provided oversight of the evaluation process and quality assurance of the draft report. The timeframe was constrained: the evaluation started in July 2016 and most data collection took place in August and early September. Emerging findings and recommendations were discussed with SIAC and the ISPC in September. This is the second draft of the report, which incorporates comments from SIAC and its Project Steering Committee. The main limitations of the evaluation were: the lack of a well-developed theory of change for SIAC; the short evaluation timeframe that reduced the depth and breadth of analysis; and the fact that the project is still young, given the time needed to plan, commission and carry out impact assessments, and at the time of our analysis had only a small handful of research outputs to consider. Main findings and conclusions The SIAC has given a major boost to central IA resources in the CGIAR. Through SIAC and its committed team, SPIA has undertaken an ambitious and wide-ranging programme of work. Regarding relevance: we found that the design of SIAC is broadly relevant to the mandate of SPIA and to the needs of CGIAR funders and Centers/CRPs. However, there are a number of areas for improvement, as detailed in the report. There are two overarching messages. First, the theory of change of SIAC needs revisiting in depth before taking major decisions on the scope and activities of vi iea.cgiar.org SIAC Evaluation - Report any further phase of SIAC – with a focus on how SIAC activities and outputs can be best designed to lead to institutional strengthening of impact assessment in the CGIAR. Second, more work needs to be done to agree on the comparative advantage of SIAC/SPIA and its priority activities vis-à-vis impact assessment conducted by Centers and CRPs. We believe this is a priority for the forthcoming evaluation of ISPC/SPIA. Regarding effectiveness: generally good progress has been made against planned outputs and productivity has been high. This has included inter alia some innovative and influential work, for example on methods for varietal adoption studies. However, the focus in SIAC Phase I on quickly delivering a large number of high quality studies has had some downsides, including fragmentation of the portfolio and loss of opportunities for longer term institutional learning. We believe that for the next phase of the work, more thought should be given to the theory of change, options for management of the research, and the institutional learning and communications investments needed to reach the wider outcomes desired. Regarding quality of science: it was too early to assess outputs, so our analysis focused on research proposals and SIAC processes. Generally, quality was high. Proposal documentation was weakest in the areas of research ethics, open data and distributional impacts. The project team has been gaining experience in how to manage research calls and improve quality, including calls managed by partners, and is already tackling these areas. Regarding management and governance: the management of the project has been efficient, within the institutional constraints it has faced, and there is evidence of improvement in efficiency over the lifetime of the project. However, the administrative systems of FAO have been a major constraint for the project, leading to delays and limitations on contracting partners, and the closure of the incipient SIAC small grant scheme. Another area for possible improvement is to broaden out external partnerships, in particular to get greater involvement of nationals of countries where the work is taking place. In contrast, governance of the project by the CGIAR system has been weak, partly because the CGIAR has been undergoing institutional reforms over the project period. Broader SPIA governance issues are outside the scope of this evaluation, but we note that the Fund Council has been unable to exercise effective governance of SPIA/SIAC to date, and that there may be an opportunity to address this in the new System Council through its ‘Strategic Impact, Monitoring and Evaluation Committee’ and/or through the ISPC. Furthermore, we concluded that despite the commitment and hard work of its members, the Project Steering Committee (PSC) is not appropriately composed and configured for either a management or a governance role, and suggest that this be revisited before a second phase. vii iea.cgiar.org SIAC Evaluation - Report Recommendations A summary of the recommendations follows. Please see the recommendations section of the report for detailed suggestions under each recommendation. R1 Revisit the theory of change of the SIAC project, and in particular the links between SIAC/SPIA and IA conducted elsewhere in the CGIAR. R2 Put in place a more systematic process for selection of IA topics and specific studies undertaken by SIAC. R3 Carry out systematic consultation of CGIAR research leaders on needs for IA, the proposed SIAC work programme, and the division of responsibilities for IA with other parts of the CGIAR. R4 Take steps to improve the utilisation of IA results in the prioritisation of CGIAR research. R5 Invest more strategically in helping to institutionalise IA across the CGIAR. R6 Revisit the management and governance of SIAC/SPIA. R7 (Recommendation for IEA): In the planned 2017 evaluation of ISPC and SPIA, include an analysis of impact assessment roles and responsibilities across the CGIAR. viii iea.cgiar.org SIAC Evaluation - Report 1. Introduction 1.1 Origins of the evaluation This is an evaluation of the first phase of SIAC: a major project to strengthen impact assessment in the CGIAR1. The evaluation is being commissioned by SPIA and financed through the project itself (as planned in the original project proposal). In order to ensure evaluation independence, SPIA requested the Independent Evaluation Arrangement (IEA) of the CGIAR to contract independent evaluators and manage the evaluation. 1.2 Structure of the report This report starts with an overview of the evaluation and of SIAC and its context. It then covers the main Evaluation Questions, grouped under the headings of Relevance, Effectiveness, Efficiency and Governance and Management before moving to the conclusions and recommendations. Of the annexes, we would particularly like to call the reader’s attention to Annex 4: ‘What is “impact assessment” and why is terminology important?’ and Annex 7: ‘Independence, impartiality and overcoming bias in impact assessment’. These raise key conceptual issues for the evaluation, but have been moved into annexes to shorten and improve the flow of the main report. 1.3 Evaluation Purpose, Objectives and Clients The evaluation has two main objectives according to the Terms of Reference (ToR)2: • for SPIA to demonstrate accountability to SIAC donors (primary purpose); • to contribute to a better understanding of SPIA’s contribution to the CGIAR. Specifically: “the evaluation will draw lessons and make recommendations that will inform the second phase of SIAC and, more generally, advise on future directions of SPIA”. We have defined3 the primary users for this evaluation to be: SPIA and the Independent Science and Partnership Council (ISPC), the two donors supporting SIAC, and finally the CGIAR System Council4. There are a number of other important stakeholders, including CGIAR Center Directorates and CRP leaders, and their Impact Assessment Focal Points, but the primary users will be our main target audience. 1 Throughout this report, we refer to SIAC when the issue is project specific, SPIA when it is a longer term institutional issue, and SIAC/SPIA when they overlap. 2 The ToR can be found online at:http://iea.cgiar.org/sites/default/files/SIAC%20Evaluation_TORs_Jun2016.pdf 3 See Inception Report for discussion on the importance and selection of primary stakeholders 4 See section on the CGIAR Institutional Context (paragraph 0). 1 iea.cgiar.org SIAC Evaluation - Report 1.4 Scope of the Evaluation The main timeframe considered is SIAC Phase I (2012-16). SIAC is still young, with some first phase activities still having received less than two years of full funding. The evaluation is thus predominantly formative (learning) although there is a summative (accountability) element. Because SIAC and SPIA are closely interlinked (indeed, indistinguishable as regards their workplan and budget), the scope of the evaluation goes beyond SIAC-funded activities to consider broader questions about the role and comparative advantage of SPIA. However, this is an evaluation of SIAC and not of SPIA/ISPC, so the evaluation will not be able to fully address these broader questions or make recommendations. In particular, this evaluation does not make recommendations on the governance structure for SPIA. 1.5 Evaluation approach and questions The Terms of Reference sets out evaluation questions under four areas, as follows: Relevance and project design: How relevant is SIAC for SPIA’s mandate, ISPC mandate and the goals of the CGIAR? To what extent does the design of SIAC address the demand for reliable information on impact from donors and other key stakeholders? To what extent does the design of SIAC address the objective of developing a strong impact assessment culture in the CGIAR? Effectiveness: To what extent have outputs been produced as planned under each Objective? To what extent has there been progress towards meeting the four Objectives of SIAC and which activities have contributed most? What are the main enabling as well as constraining factors which explain the project’s achievements (or lack of)? Has SIAC made appropriate adjustments (in terms of activities and management) in response to changed circumstances? Quality of science: Do the IA methods developed under SIAC (Objective 1) reflect state of the art quality of science? Do the processes of designing, selecting and managing the impact assessments and technical studies being carried out under SIAC promote high quality? Management and governance: Are the human and financial resources of the project adequate, and used efficiently? To what extent has the PSC been effective as a mechanism for guidance and oversight? To what extent have the partnership and contractual arrangements with regard to project components been efficient and effective? 1.6 Evaluation approach, methods and analysis A brief summary is provided here, and more details can be found in the Inception Report5. Data collection instruments can be provided on request. The (virtual) evaluation team comprised two independent external evaluators (total budgeted 48 person days) and an Evaluation Analyst employed by IEA. Julia Compton (UK), as team leader and lead report author, has had primary responsibility for managing the process and conducted most of the 5 The Inception report can be found online at: http://iea.cgiar.org/sites/default/files/Inception%20Report- SIAC%20FINAL.pdf 2 iea.cgiar.org SIAC Evaluation - Report interviews. Timothy Dalton (USA), as impact assessment expert, was primarily responsible for assessing science quality and authoring that section of the report. He has also contributed his expertise to all aspects of the evaluation, in particular on project resources and their use. Sophie Zimm (IEA, Rome) worked primarily on data analysis and presentation, but also reviewed the full report and contributed her expertise to all aspects of the evaluation. Annex 2 gives more details on the evaluation team. The evaluation approach is 'utilization-focused' (Patton and Horton, 2009; Quinn Patton, 2008), that is, it aims to be useful to decision-makers through a joint learning process. An Evaluation Matrix (see Inception Report) was used as a primary tool for managing the evaluation: first to agree the approach and basis of evaluative judgements; and then as a planning tool for data collection. The main sources of information included: semi-structured interviews with 51 stakeholders and also three focus group discussions with Impact Assessment Focal Points (IAFPs) (full list in Annex 1); observation of two key SIAC meetings (an IAFP meeting and a research partners meeting); and document review and analysis, including an analysis of the quality of approved research proposal documents. We also asked the SIAC team to carry out some self-evaluation, for example of progress with outputs (Annex 8), with the results then triangulated by the external evaluation team. All data, notes and documents collected, were systematically filed by the evaluation team and held securely in an on-line Dropbox folder. Bibliographic references were uploaded to a shared Zotero group, and all non-confidential references are available to share with SPIA. The time available for the evaluation was constrained6: the evaluation team was selected in June 2016 and started work at the beginning of July 2016, the inception report was finally approved in early August; and most data collection took place in August and early September. Emerging findings and recommendations were discussed with the SIAC in early September and presented to the ISPC bi-annual meeting in Hyderabad in mid September. The first draft evaluation report was submitted to SIAC in early October. Following discussion at the Project Steering Committee in November and incorporating written comments from SIAC and the PSC, this second draft is being circulated widely for comments. The final report will be submitted to SIAC management, i.e., the PSC, for management response, and published. The IEA evaluation manager (Head of IEA) provided oversight and internal quality assurance at all phases of the evaluation. 1.7 Main limitations of the evaluation and changes from the Inception Report The main limitations were, as identified in the Inception Report: • the lack of a well-developed theory of change for SIAC; • the constrained timeline reduced the number of people consulted and the depth of analysis that could be carried out; 6 The reason for the short timeframe was that the evaluation was intended to inform the proposal for a second phase of SIAC which originally was to be submitted in draft in October; however this has now been postponed. 3 iea.cgiar.org SIAC Evaluation - Report • the project is still in its early stages and has only a small handful of research outputs to consider. (It is worth noting that some have come out since this report was drafted, but unfortunately not in time to be considered in the evaluation); • the most significant change from the Inception Report7 is that we were unable to work with SIAC on the project theory of change. Instead, we have made developing the theory of change a high- priority recommendation for SIAC/SPIA for its next phase of work. 1.8 The CGIAR institutional context According to its website8, the Standing Panel on Impact Assessment (SPIA) is "a sub-group of the CGIAR Independent Science and Partnership Council (ISPC), which has an advisory role, primarily to CGIAR members (a group of 68 countries and funders9) through the Fund Council10, on issues relating to the quality, relevance and impact of CGIAR research activity". SPIA is managed by a part-time Chair and two other members, who are all academics independent of the CGIAR. The SPIA secretariat is also part-time and shared with ISPC, based in the offices of FAO in Rome11. SPIA's mandate is: To provide CGIAR members with timely, objective and credible information on the impacts at the system level of past CGIAR investments and outputs in terms of the CGIAR goals of enhanced food security, poverty alleviation and sustained natural resources To provide support to and complement the centres in their ex post impact assessment activities; (this includes facilitating inter-centre impact assessment efforts and providing a forum for exchange of experience from impact studies) To provide feedback to CGIAR priority setting and create synergies by developing links to ex ante assessment and overall planning, monitoring and evaluation functions in the CGIAR Figure 1 depicts our understanding of how SPIA and SIAC fit into the CGIAR system, with the main lines of funding and reporting shown. SIAC reports directly to BMGF for the use of its project funds, and through ISPC for the use of CGIAR ‘Window 1’ funding (W1)12, in particular the DFID funding for SIAC. Administratively, W1 money is channeled to SPIA/SIAC through FAO and BMGF money was channeled through the Consortium Office. 7 http://iea.cgiar.org/sites/default/files/Inception%20Report-SIAC%20FINAL.pdf 8 http://impact.cgiar.org/about accessed 7/7/16 9 We could not locate an up to date list of members. CGIAR (2011) lists 64 members in 2009, including 25 developing countries. More recent documents refer to funders rather than members. 10 Replaced by the CGIAR System Council on 1 July 2016 11 For details and identities, please see the SPIA website, http://impact.cgiar.org/about 12For an explanation of CGIAR funding windows, see http://www.cgiar.org/about-us/governing-2010-june-2016/cgiar- fund/ 4 iea.cgiar.org SIAC Evaluation - Report Figure 1: Organogram of SIAC and SPIA in the CGIAR (Phase I) Source: Evaluation team, based on discussions with SPIA. PSC = Project Steering Committee; EIAC = Evaluation and Impact Assessment Committee of Fund Council; MSU = Michigan State University. For others see List of Acronyms. 1.9 Overview of SIAC The timeline in Annex 3 gives details of the context and key milestones for the SIAC project. In brief: The project was set up in response to perceptions expressed by donors that the data on CGIAR impact was inadequate both in quality and quantity. Prior to this, funding to SPIA had been relatively low; SPIA's budget was under $250,000 a year until 2009, and was dominated by a single large project (DIIVA, with over two thirds of the SPIA budget) from 2010-1313. The SIAC project started in November 2012 with funding from the Bill and Melinda Gates Foundation, BMGF (to date totalling US$5.2M), channelled through the CGIAR Consortium. Additional funding from DFID (to date totalling US$ 4.5M) started in 2014, channelled through the Fund Council (as a ‘Window 1 special project’) and managed through FAO. SIAC has made a considerable difference to overall resources available centrally for IA in the CGIAR (see Figure 2 below). 13 The Diffusion and Impact of Improved Varieties in Africa (DIIVA) project together with its sister project Tracking Improved Varieties in South Asia (TRIVSA) were initiated and funded by BMGF to address the lack of recent evidence on diffusion of improved crop varieties. Their success led to the formulation of SIAC. 5 iea.cgiar.org SIAC Evaluation - Report • Since the SIAC project started, SPIA has allocated its total budget from all sources, including ISPC, to the SIAC work plan14. SIAC funds virtually all the SPIA programme of work, plus consultants and temporary staff (this excludes the SPIA Chair, Members and three management staff). • SIAC is managed by the SPIA Secretariat (three staff members with total of 2.25 FTE as well as consultant contributions). Michigan State University is subcontracted to lead many of the activities under SIAC objectives 1 and 2. • A Project Steering Committee (PSC) oversees SIAC. It is chaired by the SPIA Chair and includes the SPIA secretary (secretary to the PSC), a representative of the Fund Council (now System Council) and from the Consortium Office (now System Management Office), an external independent expert, an observer representative from the grant recipient institution or its designate and the Head of the IEA (also as observer). Figure 2: SIAC has dramatically increased central resources for IA: SPIA funding, 2007-2016 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 ISPC DIIVA project SIAC BMGF SIAC W1 Source: Evaluation team graph of SPIA data 1.10 Project objectives and intervention logic The overarching (long-term) project goal is “to contribute to poverty reduction, food security, nutrition and health, and sustainable natural resource use by improving knowledge and understanding concerning the impacts of international agricultural research” (SIAC 2012, p.2). More specifically, the project proposal states (SIAC 2012, p.2): “...success may be judged by two key indicators: 14 ISPC contributes in the region of half a million dollars a year. This includes also the time of SPIA Chair, SPIA Members, SPIA Secretariat staff. The main personnel costs charged to the project in 2015 were for the Financial and Administrative Coordinator for the project and two SIAC Research Associates. 6 iea.cgiar.org Annual SPIA budget (US$ million) SIAC Evaluation - Report • an expansion of the available set of impact studies, providing useful and credible information to guide future investments in the CGIAR; • CRPs and Centers of the CGIAR have institutionalized impact assessment such that ex post impact assessment is regarded as an essential part of prudent research management for accountability purposes and as an input to ex ante strategic planning.” The SIAC Project proposal (SIAC 2012, p.3) further argues that SPIA has three major areas of comparative advantage, which SIAC will work to strengthen. These are: • “Public goods for the impact assessment community of the CGIAR, for example new metrics and measures; open access databases; quality assurance and training impact assessment specialists. • Coordination: identifying gaps in the research base, promoting harmonisation of methods and definitions so research is comparable; central point of information exchange for studies. • Synthesis and Overview studies such as meta-analyses and less formal reviews” • The project is structured around four objectives, each with numerous activities: • Objective 1 (Methods): Develop, pilot and verify innovative methods for collection and assembly of diffusion data • Objective 2 (Outcomes): Institutionalize the collection of the diffusion data needed to conduct critical CGIAR impact evaluations. • Objective 3 (Impacts): Assess the full range of impacts from CGIAR research • Objective 4 (Building a community of practice): Support the development of communities of practice for ex post impact assessment within the CGIAR and between the CGIAR and the development community more broadly. Figure 3 shows project expenditure to date by main objective. As might be expected, the bulk of the expenditure is on studies, under objectives 1-3. Figure 3: SIAC initial and current budget by SIAC Objective (USD thousands) 7,000 6,009 6,000 5,000 4,207 4,000 3,000 2,000 1,021 1,000 628 0 1 and 2 3 4 Oversight, Management and M&E Original Budget ($000) Current budget 7 iea.cgiar.org SIAC Evaluation - Report Source: Evaluation team graph drawing on SIAC Progress Narrative to BMGF (2014) and SIAC monitoring of budget by activity (Aug 2016). The logic model from the project proposal is depicted in Figure 4 below (see paragraph 2.1.5). For the BMGF, the project has produced a detailed results framework for reporting against selected outputs, outcomes and milestones, used in our outputs analysis (Annex 8). Figure 4: Summary logic model for the SIAC project Source: SIAC Proposal (Word version) p. 24 8 iea.cgiar.org SIAC Evaluation - Report 2. Main findings 2.1 Relevance Subquestions addressed15: How relevant is SIAC for SPIA’s mandate and the goals of the CGIAR? To what extent does the design of SIAC address the demand for reliable information on impact from donors and other stakeholders? To what extent does the design of SIAC contribute to the development of a strong impact assessment culture in the CGIAR16? Main evidence sources: Document review; Review of 2014 SIAC donor survey; Semi-structured interviews with CGIAR funders (11 individuals representing 5 major donors), research leaders and managers, and group discussions / individual interviews with Impact Assessment Focal Points (IAFP); Observation of IAFP and research meetings. For further details see: Annexes 4, 5, 6 and 7. 2.1.1. Introduction This section is structured by the three main areas of SPIA’s mandate: to provide information on IA at the system level; to support Centers/CRPs in their IA work; and to provide feedback to CGIAR priority- setting; and this is followed by a discussion of the SIAC theory of change. The material in this section is quite complex, so to keep it reasonably short and to improve the flow of the narrative, some important issues have been put into Annexes. We would encourage readers to have a look at these, in particular the conceptual issues in Annex 4 (What is “impact assessment” and why does terminology matter?) and Annex 7 (Independence, impartiality and reducing bias in IA). 2.1.2. SPIA Mandate Area (i): To provide CGIAR members with timely, objective and credible information on the impacts at the system level SPIA has long seen CGIAR funders17 as a principal audience for its work, and has commissioned workshops and two surveys (2003?) and 2014) to find out “what donors want” in terms of impact information (Raitzer and Kelley, 2008a; SIAC, 2016; Watson, 2003). We were asked to investigate this area further in this evaluation. The results are summarized in Annex 5. In brief, funders have three main requirements for impact information: 15 Note that here and throughout the report, answers to Evaluation sub-questions have been reordered in places to improve the flow of the narrative 16 This section covers broad-brush project design issues, while the selection of specific studies under SIAC is covered in the effectiveness section which follows 17 CGIAR membership goes beyond the major funders (footnote 9), but the main demand for rigorous impact assessments has come from a relatively small number of large-scale funders. 9 iea.cgiar.org SIAC Evaluation - Report a) Justifying funding to the CGIAR, in the face of competing calls on donor funding and (often) reducing aid budgets. For this purpose, the CGIAR needs to be seen as a reliable, impact-focused organization18 that generates “big numbers”19 of beneficiaries. Large scale adoption studies are therefore a priority, but funders also require information on the distribution of benefits, in particular to the poor and women, as well as the impacts on a range of indicators (e.g. nutrition and resilience to climate change), and in particular the CGIAR Intermediate Development Outcomes (IDOs). The scale of these demands implies a very ambitious agenda for outcome and impact assessment. The SIAC project was designed to address some of these demands (e.g. collecting large scale adoption data) but to maximize its effectiveness SIAC/SPIA needs to leverage IA carried out in other parts of the CGIAR. b) Demonstrating the impact of individual donor investments already made in the CGIAR: In the past, this has resulted in a plethora of individual project studies and surveys of different donor ‘areas of influence’ and (often) overload on national statistical systems (Barrett et al 2009; CGIAR-IEA 2016). However, international agencies and funders are aware of these problems and are slowly taking some practical steps to tackle them, with plans for national agricultural surveys harmonized across agencies, looking at technology adoption inter alia.20 (SIAC’s work with LSMS-ISA was seen as a step in the right direction.). Another challenge is timing: adoption of technologies typically takes 10-20 years, while most donors are under pressure to demonstrate the impact of current investments, typically in a time frame of 3-5 years. c) Decision making on where to invest in the CGIAR: While some support was expressed for the ‘public goods’ nature of some SIAC research, funders are generally more interested in IA that is designed to generate a body of evidence to help them prioritize and justify investments in particular areas of CGIAR research. Small, situation-specific trials, and adoption studies that fail to answer distributional and ‘why’ questions, are generally seen as less useful. There continues to be strong demand for good outcome and impact data for certain areas of research that funders consider important, but that still lack large-scale evidence on outcomes and impacts, such as Natural 18 It is important to donors that the CGIAR is able to set overall research priorities, based on a critical analysis of available information and ex-ante economic analysis where possible, rather than basing funding on historical allocations or unrealistic projections. (Unfortunately, in our opinion, donors often give mixed messages on this point, with researchers often feeling that they have to make heroic assumptions to reach “big numbers”.) This issue is discussed below under SPIA mandate (iii) 19 Donors interviewed did not define “big numbers”, but to put this into perspective, World Bank health and education projects can target millions or even tens of millions of beneficiaries. 20 FAO has been addressing such issues through the" Global Strategy to improve agricultural and rural statistics" (GSARS) programme, an umbrella effort working to enhance the capacity of developing countries to produce and use agricultural and rural statistics and to strengthen statistical governance mechanisms. USAID and FAO have just (7 Sept 2016) announced a new $15M project for agricultural integrated surveys (AGRIS) which will “capture improved annual data on agricultural production, but also broader and more detailed structural information relating to farms, including … production costs, farming practices, and environmental impacts. It will incorporate recent innovations like remote sensing, GPS, mobile technology and various uses of "big data." These tools will introduce more objective approaches to measuring agricultural performance, in some cases replacing traditional, more expensive methods…. promote the integration of disparate data sources, improve data timeliness and usability, and cut data collection costs.” http://www.fao.org/news/story/en/item/430779/icode/ 10 iea.cgiar.org SIAC Evaluation - Report Resource Management21. Given the size of the IA agenda and SIAC’s limited resources, prioritization of studies is particularly important (see paragraph 2.2.2). Presentation of impact information is also important to funders, and this has implications for SIAC/SPIA communication. Donor staff are typically under time pressure and do not want to trawl through a website or many documents (however short) in search of data. They need a quick way of checking what evidence there is for particular crops, research programs and indicators (such as nutrition, climate resilience or gender)22. Key data also needs to be presented in a format that allows funders to make clear statements (with appropriate qualifications on the strength of the evidence), rather than in what was described by one interviewee as an “overly scientific, cagey” style. SIAC has already made some investments in improved communication, but are aware that this is an area where SIAC/SPIA still needs to improve. 2.1.3. SPIA Mandate Area (ii): To provide support to and complement the Centers in their ex post impact assessment activities As mentioned above, the impact assessment agenda for the CGIAR is too big for SPIA to undertake alone. Despite the boost given to SPIA’s budget by SIAC, it is still a small proportion of all funding available for IA in the CGIAR. This is illustrated in Figure 5, which shows the SIAC budget along with the planned CRP budgets for IA in Phase II23. 21 This is a long-standing issue – see e.g. Kelley and Gregersen (2005) – and NRM is a major SIAC workstream which has already produced some useful outputs (paragraph 0). 22 One format that has been well-received is the “Evidence Gap Maps” of 3iE (see paragraph 2.2.2). In contrast, despite a promising first impression, the SIAC evidence map on the website http://impact.cgiar.org/publications/map makes it hard to find evidence for specific research topics and indicators, even for CGIAR studies. 23 It is important to note that CGIAR Centers also budget for IA, and in some cases (e.g. IFPRI) organise this separately from the IA managed by the CRPs in which they participate. However we did not get comparable budget information on Center investments, which are thus missing from Figure 5. The forthcoming ISPC/SPIA evaluation will investigate this issue in more depth. 11 iea.cgiar.org SIAC Evaluation - Report Figure 5: SIAC and CGIAR Research Program Phase 2 average annual budgets for impact assessment PIM Policies Inst Markets 5.3% A4NH Nutrition Health 4.4% Roots Tubers Bananas 2.5% SIAC GRiSP Rice 3.1% FTA Forests Agroforestry 3.2% Maize 3.0% Fish 3.9% Wheat 2.3% Legumes/Dryland 1.1% Water Land Ecosystems 1.5% Livestock 1.7% CCAFS Climate Change 0.5% 0 1 2 3 4 5 6 Average annual budget (US$) for Impact Assessment (labels = % of CRP total budget) Source: Evaluation team graph, based on information compiled by SPIA of budgets from CRP Phase II proposals. The SIAC annual budget shown is the total Phase 1 SIAC budget divided by the time for Phase I, including the no- cost extension period. The graph gives a general impression of relative levels of investment, but should not be used for making direct comparisons between CRPs, due to the differing definitions of impact assessment implicitly used (see Annex 4) and the widely differing costs of some types of studies involved (e.g. a qualitative PORIA study vs a large-scale RCT). CRPs may also use project funding for IA that may not be visible in this graph. In this short project evaluation, we were not able to investigate the organization of impact assessment at Centers and CRPs in any depth. A full evaluation of the role of SIAC/SPIA in strengthening the IA system of the CGIAR would need to start with a detailed assessment of the roles and functions of all other entities carrying out IA in the system. In our view, this should be a priority for the forthcoming evaluation of ISPC/SPIA. Annex 6 summarizes our observations on the system, based mainly on interview data24. The following issues were raised in interviews about the comparative advantage of SPIA, and the ways in which 24 Our observations are in line with the conclusions reached by the recent synthesis of 15 CRP evaluations (IEA-CGIAR, 2016) that “a recurring theme … is the need for CRPs to adopt a systematic and adequately funded approach to IA, which would replace the present ad hoc processes that provide only very partial and sporadic coverage of CRP activities.” Centers potentially also have an important role to play, inter alia because they have a longer time horizon than CRPs. 12 iea.cgiar.org SIAC Evaluation - Report SIAC/SPIA could best support and complement the work of the Centers and CRPs. These have been broadly structured following the objectives of SIAC phase 1. a) Methods for impact assessment • This area is seen as a core function of SPIA by virtually all interviewees. SPIA can provide a useful service by collecting and making widely available the latest information on impact assessment approaches and methods25. SIAC/SPIA can also usefully help develop and test particular methods that are relevant across the whole CGIAR – for example SIAC’s work on comparing DNA fingerprinting to other methods has lessons for many commodities. • Given that many actors are working on developing methods in some areas, including the Centers/CRPs, it is important to clarify SIAC’s specific niche and value added. In particular, certain CRPs are internationally-recognized centers of expertise in types of impact assessment methods and indicators, for example nutrition (A4NH) and policy (PIM). Furthermore, several CRPs have an agreed cross-cutting role in advising other CRPs, and in some cases conducting impact evaluations on their behalf. SIAC has already done some collaborative work with some ‘expert’ CRPs — for example a joint workshop on policy oriented research impact assessment (PORIA) with IFPRI/PIM and participation in a working group on indicators for NRM. However, more work is needed to clarify the division of roles and responsibilities between SIAC/SPIA and the ‘expert’ CRPs. • Policy oriented research (POR) is a major focus for many Centers and CRPs. Many interviewees felt that this area needed more attention on methods for IA, in particular to increase the rigor (and perceived rigor) of qualitative methods for measuring the contribution to policy change of CGIAR research. Some research leaders however felt strongly that SPIA as currently constituted does not have the qualitative skills/interests to lead in this area (see also paragraph c below). b) Outcomes and Impacts of CGIAR research • As mentioned, there is very little systematic collection of large-scale diffusion/adoption and ex-post impact data by Centers or CRPs. These have generally collected a mixture of one-off surveys and panel data from sentinel sites. There are some large-scale adoption surveys completed/ underway (e.g. in rice, maize and wheat), mainly supported by bilateral funding. • It was generally agreed that SPIA could play a very useful role in regularly collating and synthesizing information on the results of CGIAR outcome and impact studies. At the moment, this activity is limited to informal information sharing, in particular at the biennial meeting of IAFPs. • There were mixed views on the distribution of responsibility for collecting diffusion/adoption and impact field data. Many interviewees saw diffusion/adoption studies as part of the regular research program, at least as far as the early stages of adoption, with SPIA’s role being mostly for very large-scale ex-post and multi-commodity studies. Asked about the importance of SPIA’s institutional independence for conducting diffusion/adoption studies, surprisingly few interviewees (including only one of the donors) saw this as a vital issue – and many pointed out 25 For example by scanning the latest presentations and blogs such as http://blogs.worldbank.org/impactevaluations/ for relevant and practical advances in methods and examples of use. However it is worth noting that SIAC has set up an online Discussion Board for Q&A on methods inter alia, but with no uptake to date. 13 iea.cgiar.org SIAC Evaluation - Report that SIAC has been subcontracting Centers for many of the recent adoption studies in any case26. Interviewees emphasized instead the importance of rigorous methods and the involvement of external experts in promoting independence (see further discussion of the independence issue in Annex 7)27. • There is currently no cross-CRP planning for impact studies on joint outcomes, although many outcomes depend on a series of different outputs being in place, e.g. varieties, policies and NRM practices. Similarly, as pointed out in the minutes of the 2014 IAFP meeting, “measuring indirect effects of agricultural research – non-farm economy, price effects, labour market effects etc. are beyond the focus of an individual Center or CRP”. SPIA therefore is seen as having a comparative advantage in these areas. • To sum up: there are several issues to consider in deciding on the optimal allocation of adoption and impact research between CRPs/Centers and SPIA, including the stage of research, independence and impartiality, cost and information needs for different audiences. In this evaluation we can only raise these issues for discussion, but hope that the ISPC/SPIA evaluation will be able to investigate them more thoroughly. c) Capacity development and IA community of practice Interviewees highlighted a range of issues, including: i) The management of the community of practice • Both Centers and CRPs have Impact Assessment Focal Points (IAFPs). SPIA has not defined any ToRs for IAFPs. Linkages are informal and collegial. • IAFPs generally enjoy attending SPIA meetings and linking to other IAFPs. However, IAFPs are quite diverse in their status in the organization, their level of communication with research leaders and the degree of contact that they have with impact assessment colleagues in their Center or CRP. For this reason, it is not obvious that a meeting of IAFPs is a more effective communication mechanism than a more open, virtual IA Community of Practice. • Consultation with IAFPs is not a substitute for formal consultation with research leaders, for example on SIAC workplans. • Many IAFPs also participate in the Evaluation Community of Practice (ECOP, managed by IEA) or the Monitoring Evaluation and Learning Community of Practice (MELCOP, coordinated by the System Office). There is scope for better coordination about which CoP takes the lead on certain areas: for example qualitative PORIA research, a topic of great interest to the ECOP and potentially also to the MELCOP. ii) Capacity development 26 For example, one donor said: “If you have Centers who are expert in a commodity, it’s more efficient to ask the Center to lead the adoption study – instead of giving money to SPIA who subcontracts a university who contracts the Center.” 27 For example: “[IRRI] pointed out that they try not to have IRRI researcher as the lead researcher in the study - that this increases the credibility of adoption studies and IAs. It is not necessarily the case that IRRI-led studies overstates the outcomes/impact – for instance, the ACIAR-funded study estimated higher total benefits and higher average annual benefits for Indonesia and Philippines from varietal development and releases than an IRRI-led study (even as management showcases the ACIAR study)”. (Minutes of IAFP meeting, 25 July 2014) 14 iea.cgiar.org SIAC Evaluation - Report • CRPs and Centers are generally interested in upgrading their skills on IA. Several of them have arranged training for their own staff in specific quantitative and qualitative methods. SIAC/SPIA was seen as having a potentially important role in facilitating training on methods and approaches of cross-CGIAR interest. • However, capacity development should follow good practice in being demand driven, involving institutions as well as individuals, and be linked to follow-up. One course funded by SIAC was mentioned by some interviewees as too academic and lacking in practical follow up. • Interviewees appreciated the chance to work with Advanced Research Institutions (ARIs), seeing this as a good opportunity to improve skills in specialist areas. The matchmaking functions of SIAC were highly appreciated in this regard. However, study-specific partnerships with a range of ARIs were often preferred to the longer-term one-to-one partnerships with a single ARI, for two main reasons. First, multiple partnerships could give them access to a wider range of skills and experience than a single ARI. Second, because interviewees felt they were able to access more senior ARI staff time in the short-term partnerships than in the long-term partnerships, in which they often had more contact time with the ARI students than with professors28 (this might be resolved by incorporating better incentives for senior staff input). • SIAC/SPIA could coordinate more closely with the specialist CRPs, who also have a role in supporting other CRPs with capacity development in specialist methods and indicators (including for IA) – see ‘methods’ section above. d) Standards and quality for IA • There was a mixed reaction to the proposal by SIAC/SPIA to carry out ex-post quality assessment for IAs. IEA would find this helpful (see paragraph 2.1.4). Some Center/CRP IA staff felt this might be helpful, but others wondered why peer review for published papers was not enough29. For many researchers, the possible benefits were generally outweighed by the perception of reputational risk if colleagues became aware that their IA had not passed “quality control”. This was heightened for some by the perception that SPIA does not have (or would not contract in) appropriate skills to judge the quality of certain types of IA – in particular, qualitative work – and that their research might therefore be disadvantaged. • A number of interviewees suggested that a more positive first step for SIAC/SPIA would be to establish clear standards and guidance for IA in the CGIAR, which could be agreed through a broad consultation process30. We support this idea. 28 There were exceptions: one Center interviewee mentioned the serendipitous placement of a PhD student with particular specialist skills that had been very helpful, but this appeared to us to be more a function of the individual than the institutional set-up. 29 The evaluation team does not agree that journal peer review is sufficient for a full quality assurance. Journals may not look at aspects which are important to the CGIAR, such as scale and stage of the work, or distributional effects, and not all journals cover all aspects of science quality, such as ethics. 30 Some interviewees suggested more specifically that a “living manual of best practices and literature’ would be useful, especially for new CGIAR recruits. SPIA previously did develop some guidance for IA (Walker et al., 2008) that could probably be built on (although this was not mentioned by any of our interviewees). Standards that could be drawn upon include those of 3ie and - for more depth - EvaluATE (as mentioned in Annex 4). 15 iea.cgiar.org SIAC Evaluation - Report • Most of the IAs undertaken in Centers form part of the regular research program (see Annexes 4 and 6). This then raises questions about whether the quality assurance of such IAs should be the responsibility of SPIA, or whether it should form part of the regular science quality assurance system (for individual studies) of the CGIAR. This has been rather weak in some Centers (CGIAR- IEA 2016) and ISPC is now investigating how quality of science could be improved throughout the CGIAR31. 2.1.4. SPIA Mandate Area (iii): To provide feedback to CGIAR priority setting Our interviewees highlighted two main possible routes32 for SIAC/SPIA to provide useful feedback to CGIAR priority setting33. The first is by feeding into ex-ante analysis. The second is by feeding results into planning, monitoring and evaluation – in particular through the IEA. With regard to ex-ante analysis34, there may be opportunities for more systematic feedback loops with ex-ante modellers such as those in PIM/Foresight35 and other Centers and CRPs. The results of IA can help to validate the realism of assumptions in ex-ante models, and conversely, modellers may be able to feed back information on their major gaps and assumptions which can help to prioritize studies for IA. Finally, SPIA secretariat staff also work for ISPC, so there is an opportunity to comment on the realism of the assumptions made in upcoming research proposals, based on IA experience (this is done informally but not systematically at the moment). With regard to feeding into M&E, a major opportunity is through the IEA evaluations of CRPs. These evaluations use available IAs to make judgments about CRP results. SIAC/SPIA could collect such results more systematically and also potentially provide quality control or validation (see also paragraph d below). Having said this, it would be unrealistic to think that all research priorities could be set with reference to IA results. Research priorities are set looking forward to future scenarios and new innovations, while much of the ex-post impact assessment in the CGIAR refers to research which is 10 years old or longer (due to long adoption times for much technology) so that IA results are of variable relevance to today’s 31 End of Meeting report 14th Meeting of the Independent Science & Partnership Council, 14 - 15 September, 2016, ICRISAT, Hyderabad, India, pp 4-5 32 A third possible route is to feed back into learning and priority setting within research programs and research activities. We fully support the principle that impact assessments should be designed for learning and use (3ie requires this as a condition of its grants, e.g see http://www.3ieimpact.org/en/about/what-3ie-does/policy-influenc/policy- entrepreneurship/), but most learning for prioritization within research programs will be the result of IAs carried out by those programs 33 As per the full wording of SPIA’s mandate, which is “to provide feedback to CGIAR priority setting and develop synergies by developing links to ex ante assessment and overall planning, monitoring and evaluation functions in the CGIAR”. 34 A small minority of interviewees felt that SPIA itself should undertake ex-ante analysis, but this was not supported by the vast majority, due to the different skill set required as well as the institutional location of SPIA (most felt that ex- ante analysis was the provenance of research programs, with support from specialists in IFPRI). This is an issue which could be further explored in the forthcoming ISPC/SPIA evaluation. 35 http://globalfutures.cgiar.org/ 16 iea.cgiar.org SIAC Evaluation - Report priorities. Where possible, it would be useful to structure and synthesize impact assessments to pick up cross-cutting lessons on the kinds of factors and R&D processes that promote uptake and impact. 2.1.5. SIAC’s design and theory of change How does SIAC’s design and theory of change match up to the needs outlined in the previous three sections? The summarized logic model for the SIAC proposal is shown in Figure 6 below, with our annotations in red. In our view the model36 has two major shortcomings: • It fails to specify clearly the expected outputs and outcomes in a measurable and testable way37. • It does not make explicit the linkages and assumptions between activities, outputs and outcomes. In particular, the rationale for and assumptions behind the selection, design and management of SIAC activities are not rigorously examined, and alternative approaches are not explicitly considered. There is no risk analysis. 36 These comments refer to the project documentation as a whole, not simply to the specific boxes and lines in the logic model figure. 37 This is not to say that the theory of change should be a blueprint. However, a flexible and adaptive approach to attaining project outcomes still requires more careful definition of desired aims and milestones, as well as regular reflection on these. 17 iea.cgiar.org SIAC Evaluation - Report Two examples will help to illustrate this point. First, take the output at the top left of the figure: ‘systematic collection of adoption data institutionalized’. Institutionalizing the collection of adoption data could indeed be an important objective. Without institutionalization, the tendency in the CGIAR has been for poorly-coordinated ‘snapshot’ adoption surveys at irregular intervals. As illustrated by Figure 7 which shows the adoption and disadoption of rice varieties in Indonesia over a 34-year period, it is not always easy to predict from a snapshot survey whether a particular technology (variety in this example) will go on to dominate the market, become a niche product or be abandoned after a few years. Figure 6: SIAC project logic model from proposal document, with annotations by evaluation team Outputs after 5 years Outcomes Impacts Systematic collection Greater Greater confidence CGIAR system of adoption data transparency about of donors in CGIAR becomes more institutionalized CGIAR impacts system effective at reaching its Increased number of System-Level impact assessments More evidence- Outcomes of CGIAR research based decision- making within the Greater range of CGIAR types of research and types of impact covered by impact Does not specify activities, outputs, links, assumptions and risks assessments explicitly and in a testable way Greater rigor and scale of CGIAR adoption estimates and impact assessments 18 iea.cgiar.org SIAC Evaluation - Report Figure 7 Adoption rates are difficult to interpret from a snapshot survey: Use of leading rice varieties in Indonesia, 1985-2009 Source: (Brennan and Malabayabas, 2011)p.58 However, institutionalizing the collection of adoption data is a complex matter. It requires agreement (or at least no objection) from key partners both inside and outside the CGIAR on roles and responsibilities for defining what is to be collected and when, what methods are considered valid, who will be responsible for collecting and maintaining the data and how/by whom it will be accessed, analyzed, synthesized and published – and finally, adequate resource mobilization. Through SIAC (and previous projects, in particular DIIVA), SPIA has gained some expertise in the technical side of this work, but that is only a small part of what is required38. SPIA would need to think carefully about the impact pathway(s) to institutionalization, its role and comparative advantage, and how it needs to work with partners to attain the broader objective. Another example is the box entitled ‘more evidence-based decision-making within the CGIAR’. As mentioned in paragraph 0, there are some opportunities for more systematic feedback loops. However, this will not happen except sporadically and informally, without a concerted effort, resources, and clarity about roles and responsibilities. 38 Extract from the 2014 IAFP meeting: “[The BMGF representative] noted that DIIVA was a response to the lack of systematic collection of adoption data in the system: that BMGF had hoped that DIIVA would illustrate the feasibility and approach in regular collection of such data, but that such institutionalization does not seem to have occurred.” 19 iea.cgiar.org SIAC Evaluation - Report Similar analysis is needed right across the theory of change, and difficult decisions of prioritization will be required given SIAC/SPIA’s limited resources. A specific area to consider under the Theory of Change is the range and scope of SIAC work. Traditionally, SPIA concentrated on large scale ex-post Impact Assessments, but SIAC has broadened this remit to include micro-studies of technologies and interventions prior to their wide dissemination and diffusion39. (The use of the term “Impact Assessment” for such varied types of studies can confuse the discussion, as discussed in Annex 4.) A number of SIAC studies would normally be considered to be within the provenance of “regular” CGIAR research, for example proof of concept studies (see examples in Table 3) or operational research (for example, a study on the most effective extension method for an emerging technology). These types of studies are normally conducted by CRPs/Centers because they can conduct a critical mass of studies in a specialist area, and are in a better position to synthesise the results. We are not arguing that SIAC/SPIA should never venture into this area – there may be good reasons to do so in some cases, for example methodological innovation or the demand for an independent check on efficacy claims. However in our view, such reasons need to be transparent and used as explicit criteria in the selection of SIAC studies. Moreover, Center/CRP leaders should have a meaningful input into discussions of roles and responsibilities for the different types of IA40. 39 “... the SPIA strategy (SIAC program) now encompasses a broader swathe of activities aimed at enhancing our understanding of the R-to-D impact pathway, studies that examine carefully the causal chain of events along this pathway, e.g., especially micro-level studies using experimental or quasi-experimental methods that provide evidence on the impact of CGIAR derived technologies to adoption households or other relevant populations....” SIAC (nd) unpublished proposal for a SPIA external review and quality rating system for ex-post impact assessments (ePIAs), p. 2 40 SIAC has made an effort to avoid duplication with specific CRP studies, for example A4NH leaders were consulted on the proposed studies in the nutrition area. However we think that consultation should go beyond this. 20 iea.cgiar.org SIAC Evaluation - Report 2.1.7. Summary of findings on Relevance It is difficult to summarize our complex findings on SIAC relevance in a short paragraph, so we have opted to present them as a traffic light table, ordered according to SPIA’s mandate (Table 1). Table 1: Summary findings on relevance of SIAC design to SPIA mandate SPIA Mandate SIAC Relevance 1. Timely and Broadly relevant, but could be better focused. credible information • Need to leverage Center/CRP IA to respond to the great on the impacts at the demands for impact evidence, including the demand for system level... more systematic evidence on non-conventional indicators (e.g. nutrition) and SLOs/IDOs 2. Complement the SIAC has moved beyond epIA and there is a potential overlap with Centers in their epIA IA undertaken as part of Center/CRP research activities – needs to be clearer on its role • Consultation on priorities with Centers/CRPs is currently weak • IPSC/SPIA evaluation should address wider issue of comparative advantage of SPIA and who does what 3. Support to the Broadly relevant, but could be better focused Centers in epIA - needs further analysis of comparative advantage (as above) • High demand from Centers/CRPs for SIAC/SPIA work on IA methods (working with expert CRPs where appropriate), collection and centralisation of IA data, standards and guidance • Role in IA quality assurance needs further study • Concerns about current SPIA disciplinary expertise - too narrow 4. Feedback to Potentially well placed in ISPC CGIAR priority • Feedback loops to/from ex-ante models (e.g. Foresight) are setting, links to ex weak ante assessment and • Demand from IEA for collating and synthesising IA M&E information across the CGIAR Source: evaluation team judgements based on discussion in text. Key to colours: Light green: broadly relevant, but could be better focused. Amber: area to improve in a future project phase 21 iea.cgiar.org SIAC Evaluation - Report 2.2 Effectiveness Subquestions addressed: To what extent have outputs been produced as planned under each Objective? To what extent has there been progress towards meeting the objectives of SIAC and which activities have contributed most? What are the main enabling as well as constraining factors which explain the project’s achievements (or lack of)? Has SIAC made appropriate adjustments (in terms of activities and management) in response to changed circumstances? Main evidence sources: SIAC team self-evaluation and validation through documentary evidence and interviews. For further information see: Annex 8. 2.2.1. Progress against outputs Table 2 presents a short visual ‘traffic light’ summary of progress against the main SIAC output areas. Full supporting information can be found in Annex 8. We consider that the SIAC/SPIA team has worked extremely hard to deliver a large number of complex activities in a short period. Many of these are on course to deliver outputs within the next year. The SIAC team has also reflected frequently on its progress and made some changes of direction, for example by initiating new work in the area of nutrition. Many interviewees have commented positively on the commitment and enthusiasm of the SIAC/SPIA team. Factors which have negatively affected the production of some outputs include: • The administrative systems of FAO are cumbersome (see paragraph 2.4.1) and have led to delays on initiating many studies. The small grants scheme was also discontinued after a few months, due to the high transaction costs. • The use of competitive grants as the main mechanism for launching studies has meant that in some areas, studies have not yet been commissioned to fill key information gaps because no proposals were received or proposals were of low quality. • In a few areas (e.g. the quality assurance system, the SPIA website), outputs have met the ‘letter but not the spirit’ of the original plan. This is mainly the case for Objective 4 (capacity building and development of a community of practice), which is more institutional in nature and requires significant skills and time investment for in-depth communication and consultation. While outputs have been developed, uptake has been poor, implying that they require further consultation and redesign to meet user needs. 22 iea.cgiar.org SIAC Evaluation - Report Table 2: Summary ‘traffic light’ table of progress against SIAC outputs 1 : Develop, pilot and verify innovative methods for diffusion data 1.1 Methods for crop varieties 1.2 Protocols for diffusion of NRM technologies 1.3 New institutional approaches for diffusion data 1.4 Disseminate best practices learned 2. Institutionalize the collection of the diffusion data needed for IA 2.1 Institutionalise collection of adoption data 2.2 Collect and validate NRM claims 2.3 Organise and institutionalise POR results 2.4 Institutionalise collection of adoption data 3. Assess the full range of impacts from CGIAR research 3.0 IA on nutrition and health 3.1 Long term/large scale epIAs 3.2 Experimental/quasi experimental studies 3.3. Under-evaluated areas 3.4 Meta analyses at system level 4. Support Community of Practice for ex post IA with CGIAR and partners 4.1 Small grants 4.2 Capacity development activities for IA in the CGIAR 4.3 Biennial conference on epIA (planned 2017) 4.4 Quality star rating for CGIAR IA studies ex-4.5 Support RROs and NARs 4.5 SPIA website a one-stop shop for CGIAR IA (website upgraded, not yet a one-stop shop) 4.7 Support and capdev to Consortium At least three studies by post docs Key: (for details see Annex 1) Outputs completed On track to complete in Phase 1 Progress made, but outputs will not be complete in Phase 1 Some setbacks Too early to tell Discontinued activity New or modified activity 2.2.2. Progress against outcomes The project has also made some significant progress towards reaching its outcomes (given the scale of ambition and the short time frame). The lack of a detailed theory of change for the project, as discussed under Relevance above, means that the links between outputs and outcomes are not always fully spelled out. However, the project team is working on an implicit theory of change and is putting a number of key building blocks into place. As an illustration, here are some of the activities undertaken to date under each objective: 23 iea.cgiar.org SIAC Evaluation - Report Objective 1 (Methods): Steps towards improved and more rigorous methods for collection and assembly of diffusion data 41 • testing methods of tracking adoption of crop varieties (and fish) against the ‘gold standard’ of DNA fingerprinting; • piloting methods for tracking NRM adoption. Objective 2 (Outcomes): Steps towards institutionalizing the collection of varietal adoption data • piloting the integration of varietal adoption data collection into national and World Bank LSMS- ISA surveys; • training CGIAR staff and NARS partners. Objective 3 (Impacts): Steps towards assessing the full range of impacts from CGIAR research • compiling CGIAR databases on adoption and impact claims for NRM practices and policy outcomes; • commissioning validation studies for some of the claims made of large-scale impacts ; • reviews of evidence on livestock, irrigation and water management which have pointed out gaps in the evidence and areas for further study42; • commissioning a review of investments in the CGIAR which highlights the commodities and areas which have received most investment43. Objective 4 (Community of Practice): Steps towards strengthening impact assessment across the CGIAR IA community and with partners • regular meetings with impact assessment focal points, training workshops and partnerships between US universities and CGIAR Centers; • workshops with CGIAR and external partners to discuss the latest thinking on methods for impact assessment of policy research and assessing the effect on poverty. Having said this, the evaluation team finds that the organization and management of phase 1 has been focused very strongly on achieving outputs, reflecting the original project proposal, with little time to think through the full gamut and prioritization of activities needed to reach outcomes44. We believe that for the next phase of the work, more thought should be given to the theory of change, the options 41 SIAC work on methods is not starting from scratch, but adding to previous SPIA methodological work, such as that on assessing environmental impacts (Renkow, 2011; Stevenson et al., 2011) 42 According to the Centers and CRPs working in these areas (contacted by email), these reviews have encouraged increased investment in impact assessment. However, the researchers also highlighted the risk that some donors have interpreted the reviews as meaning that these areas of research are less worth funding, either because they are lacking in impact or are too complex to demonstrate impact. See also footnote 54. 43 The changes in definitions for monitoring CGIAR investment over the years have been challenging for this study, and this highlights the need for the CGIAR to agree on and stick with a set of definitions. 44 This is not to denigrate the SPIA team, who have not only worked extremely hard to deliver the project, but are also a very reflective group as evidenced inter alia by the minuted discussions in the PSC and at the internal Mid Term Review meeting (2015). 24 iea.cgiar.org SIAC Evaluation - Report for management of the research and the institutional learning and communications investments needed to reach the wider outcomes desired. One key area for improvement is communications and learning. In Phase I, SIAC has not been able to invest enough resources in this area. For example, the IA reviews carried out by SIAC (livestock, water/irrigation and PORIA) were not known to all our (relevant) interlocutors, and there was reportedly no presentation and discussion of the findings. The second key area for improvement is the planning and management of studies. The decision to maximize the quantity and academic quality of project outputs in a wide range of areas with very limited human and financial resources has been reflected in the way that most SIAC research has been managed. The first issue identified is that management of large areas of the work has been outsourced to trusted academics (Figure 8). The most important example is that about 40% of the SIAC budget has been subcontracted to Michigan State University, managed by an ex-SPIA member45. Trusted academic colleagues have also managed other important parts of the work such as the nutrition and experimental calls. This increases SIAC productivity, but risks outsourcing some of the more detailed thinking and learning, instead of it being internalized in the CGIAR46. 45 This also gave rise to delays in implementing SIAC, as MSU was not able to allocate sufficient human resources and several areas of work (NRM methods and PORIA) were handed back to SPIA to manage. 46 Another issue is that the research topic leaders themselves were not chosen competitively in several cases. On one hand, this is explained by tight time frames and the need to deliver high quality, and it is possible that a competitive process would have been won by the same people. On the other, the lack of transparency (in particular in awarding the large MSU contract, which was written into the project proposal) is not best practice and has been criticised by several interviewees. An early PSC meeting raised these issues, and later SIAC contracts (including longer contracts like the capacity building partnerships) have been awarded competitively. Most MSU subcontracts have also been awarded competitively. 25 iea.cgiar.org SIAC Evaluation - Report Figure 8: SIAC outsources much of its research management to trusted partners: Distribution of budget across main partner types Source: graph by evaluation team, based on budget information provided by SIAC The second issue is that most of the competitive calls for research proposals (under SIAC Objective 3) have been broad in topic matter, for example ‘experimental studies’; ‘NRM impacts’ or ‘nutritional impacts’, and the predominating criteria for selection have been high academic quality together with the opportunities to leverage other initiatives, with preference often given to projects that have co- funding47. The result has indeed been to maximize the number of high-quality studies, all of which are individually interesting. However, the portfolio as a whole appears quite fragmented, with a variety of methods, research questions and parts of diverse research impact pathways being studied (see examples in Table 3 and Annex 8)48. Moreover, the SIAC portfolio contains many studies which are not “ex-post” and indeed could be considered part of the regular CGIAR research process (see Annex 4 for 47 We scored proposals which gained SIAC funding through competitive calls, and judged that 84% had moderate-high leverage (n=32). On an individual project level, SIAC can add value through these ‘add-ons’: it can potentially add technical rigor as well as funds for additional data collection. However, SIAC does not have the resources to provide this type of support across the CGIAR. Moreover, there are CRPs which specifically have this function – for example A4NH has specialists in impact assessment and indicators for nutrition, and funds to support other CRPs in this area. So SIAC needs to prioritise its ‘add-on’ support carefully. 48 Of course, studies could potentially be designed specifically to learn lessons about IA methods (as in SIAC objective 1 for example). But this does not seem to be the case in most SIAC calls. One exception is the first ‘experimental’ call, where three studies focused on whether on-farm/on-station trials can estimate impacts at scale (Minutes of PSC5). The synthesis of SIAC work underway (Stevenson et al. forthcoming) will bring out some of the methodological and other shared lessons from the studies. 26 iea.cgiar.org SIAC Evaluation - Report fuller discussion of this issue). Greater prioritization would be more effective in building a portfolio which fills key data gaps and enables synthesis of overall lessons about CGIAR impacts49. Table 3: Example of fragmentation: the full list of topics for funded proposals in the SIAC nutrition call50 Commodity system Country Main research question (type of study) Method High iron beans (HIB) Rwanda Are HIB a cost-effective health intervention in Adoption surveys with practice? (adoption/epIA) data on bean consumption and iron intake NERICA Rice Sierra Can NERICA rice potentially reduce the hungry RCT Leone season? (efficacy/proof of concept) Crop diversification Ethiopia Does combining crop diversification with the Adoption surveys with in maize systems and adoption of modern maize varieties improve data on dietary diversity, Malawi nutrition compared to each practice alone? micronutrients and (epIA) anthropometrics Dairy hubs Tanzania Can pilot dairy hubs potentially increase the Household surveys with nutrition of dairy farming households? data on dietary diversity (efficacy/proof of concept) and food expenditure Irrigated horticulture Senegal Does the complex set of horticultural RCT interventions being promoted improve diets? Does it work alone, or only when combined with behavior change communication (BCC)51? (efficacy/proof of concept) Source: compiled by evaluation team from information in SIAC (2016). 49 A respected CGIAR research leader made this comment: “... As the green revolution showed, we know there can be very powerful synergies between research on technology, natural resource management, and policy in improving productivity impacts in farmers’ fields, yet most IA studies conducted within the CGIAR, including those sponsored by SPIA, focus on only one or at best two of these three components.... Such studies miss out on the interactions between these components, the ones that make the total impact much larger than the sum of the parts. To capture this we need more coordinated IA studies undertaken at sites where several centers or CRPs are undertaking complementary work. This would not only tell us more about the real impact of the CGIAR, but also demonstrate that there is real value in having a CGIAR system and not just a bunch of semi-independent centers and CRPs. But organizing such collaborative IA studies is a classic ‘management of the commons’ problem, and SPIA is in a unique position to play a leadership role by organizing and sponsoring such studies.” 50 This example is not intended to single out or criticise the nutrition call, just to illustrate the general point that the prioritization of other criteria (such as proposal quality) may result in disparate study topics. 51 It is worth mentioning that the added value of BCC in area-based horticulture/other rural development programs is one of the major research questions that is addressed by the A4NH CRP/IFPRI, using rigorous impact assessment methodologies (Ruel and Alderman, 2013). 27 iea.cgiar.org SIAC Evaluation - Report In our view, there is scope for more systematic use of reviews by SIAC to highlight gaps and key research questions52. Originally there were plans for six reviews of under evaluated areas of the CGIAR work53. Only two of these have been finalised, and they do not follow a common methodology. One of the reviews was criticised by a number of interviewees as being too narrow in scope, ignoring evidence that did not conform to its definition of quality54. This highlights the importance of agreement on the types of evidence that can be included in such reviews and the way that incomplete or “imperfect” evidence can be handled55. One approach to consider is the “Evidence Gap Maps” of 3iE56. These are essentially online interactive tables which present the evidence linking particular interventions to specific indicators of outcomes and impacts. Each cell in the table contains hyperlinks to individual studies and reviews that meet certain evidential standards. This is not only a rigorous way to search systematically for evidence of particular outcomes (such as nutrition or resilience) but also facilitates finding the information quickly. 3iE is increasingly using gap maps for prioritizing questions in its thematic calls for impact evaluations. Finally, there is scope for more systematic consultation with research leaders across the CGIAR about priorities for SIAC work. This is particularly important when SIAC ventures into areas which are considered part of the regular research process. Without such consultation, the risk, as outlined by one (highly-respected) CGIAR research leader, is that “the result will be just another [impact] study of which we already have many, but not advancing available methods and not on a topic of priority”. SIAC has attempted to consult on its work plans by presenting them to Impact Assessment Focal Point meetings, but the meeting minutes, and our observation of one meeting, suggest that this is not an effective method for consultation — the majority of IAFPs have neither the position nor expertise to advise on the entire SIAC workplan, and a short presentation in a long meeting is inadequate for them to be able to grapple with the complexities involved. One possible forum for more systematic consultation would be the CGIAR research leaders’ annual meeting. The System Council and its new committee covering IA (see paragraph 2.4.3) could also play a useful role. 2.2.3. Summary of findings on effectiveness Through SIAC, SPIA has undertaken an ambitious and wide-ranging programme of work. Generally good progress has been made against planned outputs. This has included inter alia some innovative and influential work on methods for varietal adoption studies. 52 Another issue is the allocation of resources to different sectors. It appears (our analysis is incomplete as we are missing some of the MSU-managed studies) that the vast majority of SIAC resources have gone to varietal adoption work and NRM, with much smaller proportions (a few percent) to PORIA and other areas which are important for the CGIAR. However, it is not possible to make a judgement about whether this is a reasonable sectoral allocation without performing a broader analysis of the division of roles and responsibilities for adoption and impact studies, as discussed elsewhere in this report. 53 Source: Minutes of PSC meetings e.g. PSC 5. 54 The SPIA foreword to this review also (gently) supports this criticism. We support the general point that a wider range of evidence may be useful. Pritchett and Sandefur (2013) estimate from modeling that the potential bias from generalising a very small number of RCT results to other contexts (as happens in some systematic reviews) is “far, far greater” than the error from using a wider range of less rigorous studies. 55 (DFID, 2014) provides a set of criteria for assessing the strength of evidence. 56 http://www.3ieimpact.org/en/evidence/gap-maps/ Snilstveit et al (forthcoming), Evidence gap maps: a starting point for strategic evidence production and use, New Delhi: International Initiative for Impact Evaluation. 28 iea.cgiar.org SIAC Evaluation - Report However, the focus on quickly delivering a large number of high quality studies (under rather broad topic areas) has had some downsides, including some fragmentation of the portfolio and loss of opportunities for longer term institutional learning. We believe that for the next phase of the work, more thought should be given to the theory of change, options for management of the research and the institutional learning and communications investments needed to reach the wider outcomes desired. 2.3 Quality Of Science Subquestions addressed: Do the IA methods developed under SIAC reflect state of the art quality of science? Do the processes of designing, selecting and managing the impact assessments and technical studies being carried out under SIAC promote high quality? Main evidence sources: SIAC team analysis of SIAC calls for proposals and review documents, accepted proposals and project progress reports, and available outputs (very few). Interviews with SIAC-funded researchers both by VOIP and at/around the Boston SIAC meetings. For further information see: Annex 9. The Quality of Science Assessment followed the CGIAR-IEA (2015) framework (inputs, processes and outputs). However, it was limited in scope because it is still early in the project and there are few outputs57. We examined all 37 research project agreements58 - amounting to $9,260,780 of funding. 2.3.1. Processes for Assuring Science Quality The vast majority of work under SIAC is managed through two-stage competitive calls, with a call for Expressions of Interest (EOI) followed by shortlisting, a call for full proposals from those selected, and then selection of full research proposals. In some calls, for example for NRM, this has been complemented by design workshops with shortlisted researchers to discuss proposals, “matchmake” between partners and ultimately improve the quality of design. The calls have been managed by various individuals including SPIA staff and partners such as MSU and Paris School of Economics (see Figure 8), and have not been consistent in format during Phase I of SIAC, although SIAC/SPIA has been working towards a more standard format59. Some of the issues included: • Not all calls explicitly required researchers to set out a testable hypothesis and explain how it would be tested. 57 To be specific: there has been one peer reviewed publication in a genetics journal, published less than a year before this evaluation, and two published reviews (of impact assessments in livestock and water). There are however a large number of outputs on the way, including three that have been published in the month since the first draft of this evaluation report, and they will doubtless be considered in the forthcoming evaluation of ISPC/SPIA. 58 For example, Michigan State subcontracted with a number of CGIAR centers to conduct DNA fingerprinting but we could not recover the size of the subcontracts nor processes used by Michigan State to select their sub-awardees. 59 Some helpful examples from the USA could be those of the National Institute of Health, Department of Agriculture and the National Science Foundation. 29 iea.cgiar.org SIAC Evaluation - Report • Different types of research may require different selection criteria (see discussion in Annex 4 and in particular the example checklists by EvaluATE given in footnote 3 of that annex) but this was not evident in the calls. It is (obviously) equally important that proposals are evaluated by specialists qualified to assess the specific methods proposed (qualitative and quantitative) as well as the subject matter. • In only a few of the calls was there a request for an explicit statement on data availability and research subject safety (ethics) made by the researcher. These issues are indeed often part of contract documents written between FAO and the implementing institution. However, this part of the contract is often reviewed and signed by institutional representatives and hence at arm’s length from the implementing researchers, who may not have thought through all the implications. Best practice60 (and a legal requirement in some places, e.g. the USA) is that open data and research safety should be a part of the research proposal rather than relegated to contract administration. Inputs into the Research Process Thirty-two of the research proposals accepted for funding by SIAC61 were reviewed for science quality. The criteria used were scored on a yes/no (1/0) basis, as follows: • objectives of the proposal clearly stated; • clearly stated testable hypothesis or hypotheses62; • clearly presented data collection strategy; • data collection strategy that included collecting disaggregated data, for example between men and women, wealth groups, ethnicity or caste; • clearly-stated strategy in the proposal itself (not the contract documents) to ensure that the data complied with best-practice human subjects protocols and CGIAR open data policy; • clearly presented analytical strategy. The results are illustrated in Figure 9 and presented in more detail in Annex 9. In sum: • all but one proposal clearly stated the research objective and provided a compelling case for the research activity; • 69% of the proposals specified a clear, testable hypothesis. Most of those lacking a clear hypothesis were in the NRM call. We could often infer a hypothesis, but this is not the same as what a researcher might have in mind; • 84% of the proposals described their data collection strategy clearly. Not all proposals explained power calculations and contingencies; • 90% of proposals set out clear analytical approaches; 60 At a minimum, the researcher (not the administrator signing the contractual documents) could be required to attest that they are aware of and will abide by the CGIAR Open Data policy. A more proactive policy would require that the researcher indicate where the data will be archived, the metadata provided and the embargo period. 61 This excludes some of the MSU managed activities, for which we did not have documentation, as well as proposals received for capacity development activities. 62 We searched for clearly stated hypotheses that could be evaluated but also accepted statements that strongly alluded to a hypothesis 30 iea.cgiar.org SIAC Evaluation - Report • only about one-third of the proposals provided sufficient information on whether distributional impacts would be covered, for example between men and women, or by income, ethnic groups, or caste. Less than one third of the proposals gave an explicit plan for making the data open to external analysis (see footnote 60) or provided clarity that they understood the responsibility of the researcher to follow best ethical practices (human subjects). Currently, SIAC/SPIA requires that the research pass an Institutional Review Board (IRB), but devolves this responsibility to researchers to organize. This would be stronger if SPIA undertook explicit review of human subject protocols, informed consent statements and affiliated practices. It is important to remember that this analysis refers to written research proposals accepted by SIAC. It is possible that some of the issues highlighted as weak in the proposals (e.g. disaggregation of data) are actually being implemented in the research, but were simply not documented in the proposal. However, we would argue that proposals should not be accepted without such documentation. Figure 9: Percentage of research proposals examined with given characteristics Clearly stated objective Analytical approach described Data collection strategy defined Testable hypotheses User-group disaggregated data Human subjects statement/open data 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Source: Evaluation team. N=32 2.3.2. Summary of findings on Quality of Science The quality of science conducted under SIAC has been very good overall. We did find some areas for improvement, particularly in documentation and in the specification of research calls. Based on an analysis of accepted research proposals, the weakest areas are the lack of information on distributional impacts (disaggregated data), research ethics and open data. The project team has gradually been gaining experience in how to manage research calls and improve quality, including calls managed by partners. We encourage SIAC/SPIA to build on this experience and develop consistent documentation and procedures, which tackle the weaknesses mentioned. 2.4 Management and Governance Subquestions addressed: Are the human and financial resources of the project adequate and used efficiently? To what extent have the partnership arrangements with regard to subcontracting project components been efficient and effective? To what extent has the Project Steering Committee been effective as a mechanism for guidance and oversight? 31 iea.cgiar.org SIAC Evaluation - Report Main evidence sources: Document review, including documentation on calls, proposals and SPIA mailing lists; minutes of PSC meetings; interviews with PSC members For further information: Analysis of PSC attendance and decision-making (available on request). 2.4.1. Management The core SIAC/SPIA team is composed of three technical experts (total 2.25 FTE) and one part-time administrator (FTE), all shared with ISPC. In addition, the SPIA chair spends an estimated 0.1 FTE on SIAC business. This is a small team for an ambitious project, particularly given that some team members are personally involved in carrying out research. Over the long term, the core SPIA team, along with other central bodies of the CGIAR, is effectively limited in size by available funding, which is a portion of the ‘overhead costs’ of the CGIAR system. The overhead costs are funded by a 2% levy on all research programs of the CGIAR. There is considerable downward pressure from Centers and CRPs on these overheads, and (implicit) competition among central institutions for funds. The dilemma posed for any central institution by additional project funding (such as SIAC) is therefore how to manage the additional funds without taking on additional core staff. SPIA has handled this mainly by outsourcing large portions of the work, as well as contracting consultants for specific tasks. However, outsourcing research management and contracting temporary staff both have a potential downside in loss of institutional learning, as outlined in the Effectiveness section. There is no easy answer to this problem, short of a decision by the CGIAR System Council to invest more long-term funds in centrally-managed impact assessment63. SIAC faces challenges which are common to many projects: split funding and several reporting lines (illustrated in Figure 1). The initial funding was provided by BMGF and comes through the Consortium Office (now the System Office). BMGF (like most bilateral donors) requires regular reporting against a series of specific milestone targets, as well as financial reporting. SPIA also reports its work to the Fund Council (now the System Council) through the ISPC. The decision by the Fund Council in 201364 to channel the additional W1 funding provided by DFID through FAO further complicated matters. The rationale for routing the SIAC funding through FAO was to increase the independence of the project, but (based on our limited reading and interviews) we cannot see any sign that the BMGF funding has resulted in lower independence of studies due to being routed through the Consortium (see also Annex 7 on independence). FAO systems are not only cumbersome, but their specific rules around subcontracting and intellectual property are unacceptable to many SIAC partners (e.g. many universities), and the resulting negotiations have delayed the agreement of contracts for studies under SIAC for months, and in one case up to a year. We would suggest that these issues be investigated more closely before making a decision to route further SIAC funding through FAO65. 63 This would require further analysis of SPIA’s role and comparative advantage, which we expect will be undertaken in the ISPC/SPIA evaluation. 64 The discussion and decision are summarised in the PSC 3 minutes. 65 This refers to the operations budget, not staffing. Some interviewees have gone farther and suggested that SPIA itself be moved into the new Systems Office, but this is a question for the wider evaluation of ISPC/SPIA. 32 iea.cgiar.org SIAC Evaluation - Report The evaluation team considers that the management of the project has been efficient, within the institutional constraints it has faced, and that there is evidence of improvement in efficiency over the lifetime of the project. As an illustration, Figure 10 shows the times taken for the key stages for all the large competitive calls carried out by SIAC, ordered chronologically from the top to the bottom of the chart. Several points can be observed from this figure: • the first general point is that it takes a minimum of several months to get studies under way — and this does not include the time needed for the design of the call. So a three-year project is very short for the outputs expected in SIAC; • the second point is that there has been considerable improvement in efficiency of the process over the three years of the project. The first call took nearly a year from the time of the first call for expressions of interest (EOI) to the date of signature of the first study contract66. The final call shown took under six months. While this could still be improved, our limited benchmarking with other organisations67 showed that these times are not out of line with others; • the third point is that there is still room for improvement in two areas68: the consistency of the time allowed for proposals and EOIs, and the efficiency of the approval process by the PSC. The PSC will be discussed below under ‘governance’. Regarding the time allowed for proposals and EOIs, we would suggest that SIAC/SPIA sets a consistent timetable that allows enough time for teams to be put together. 66 The median time to contract signature was considerably longer in some cases (e.g 18 weeks for the ‘underevaluated areas’ call). 67 Based on team experience and comparison with calls by 3ie and the Feed the Future Innovation Lab for Collaborative Research on Sorghum and Millet 68 This assumes that there is no further possibility for improvement in the contracting process, which has been the subject of long negotiations with FAO and to a much lesser extent with the Consortium Office. 33 iea.cgiar.org SIAC Evaluation - Report Figure 10: Timing for key stages of major SIAC research calls Time (weeks) for each phase 0 10 20 30 40 50 2013 Nutrition call (EOI=56) 7 10 14 19 Experimental evaluations (EOI=21) 8 8 9 9 2014 Long term/ large scale studies (EOI=22) 5 7 13 9 Under-evaluated CGIAR areas (EOI=26) 8 7 7 4 Adoption of NRM practices (EOI=61) 4 3 7 11 2015 EOI call period Proposal call period Time to PSC decision Time to first contract signature Source: evaluation team based on data provided by SIAC. EOI – Expressions of interest Management of contracted partnerships SIAC/SPIA has contracted a large number of institutions at two levels: a) as activity team leaders of particular areas of work (e.g. MSU) b) as lead partners for particular research studies, which in turn have subcontractors. Figure 11 shows the geographic distribution of study lead partners (green for CGIAR Centers, and red for other institutions) and sub partners (yellow). It can be seen that SIAC has subcontracted many of the studies to CGIAR Centers to manage. This relates back to discussions of independence and SPIA value addition (paragraph 2.1.3-b and Annex 7). It can also be seen from Figure 11 that (leaving aside the CGIAR Centers) the majority of contracts and subcontracts are to United States (and to a lesser extent European) universities and research institutions69. While these Advanced Research Institutions generally promote high quality research, there is also a potential concern about the degree of institutional and national learning from the studies if the core of the design and analysis is carried out far away from the location of the study. This issue was raised by some interviewees. We did a quick investigation into the reasons behind the geographical spread of contracts: Did SIAC invite people from around the world to apply for its competitive calls? Were proposals more likely to be accepted from certain geographical areas? Figure 12 shows the result. European and North American institutions dominate the SPIA mailing list (more than three quarters of the database) and North America is the most successful region (with 26% of applications being funded). Neither Latin America 69 Figure 8 also shows that over half of the SIAC studies budget is managed by US universities. 34 iea.cgiar.org SIAC Evaluation - Report nor Africa had any successfully-funded proposals, which in the case of Latin America can be explained by the low number of invitees and very low amount of proposal submissions. Over the past 10 years, there has been a large international investment in training people from around the world in impact assessment, by institutions such as CLEAR, 3ie and many universities70. Some organizations such as 3ie have tried very hard to take advantage of this wider pool of qualified people in impact assessment, and have tried various approaches to do so, for example 3Ie calls often specify that Principal Investigators must be nationals of the country concerned. 3iE has also promoted ‘matchmaking’ of research partners from different countries, both through workshops and online. It might be useful for SPIA to investigate the experience with these approaches, in order to broaden out its partnerships and (potentially) improve country-level learning from its impact assessments. Figure 11: Geographic distribution of lead partners in SIAC study contracts MSU SPIA Source: constructed by evaluation team from SPIA contract (LOA) database. Green= CGIAR Centers, Red = other principal partners, Yellow - subcontractors 70 Both CLEAR and 3ie have useful databases of qualified impact assessment specialists around the world, including in agriculture. 35 iea.cgiar.org SIAC Evaluation - Report Figure 12: Average number of invitations, applications and funded proposals for 3 SIAC calls 160 142 140 120 108 100 80 60 40 33 39 20 17 18 11 4.3 1.3 7 0.7 0.0 5 1 0.0 0 NORTH EUROPE ASIA AFRICA LAC AMERICA invited applied funded Source: Constructed by evaluation team with data from SIAC mailing list and calls databases. The three calls were the only ones with complete data available: Nutrition call (3.0), Under-evaluated areas of CGIAR research (3.3.) and Adoption of NRM practices (2.2.). This analysis does not include CGIAR Centers. LAC – Latin America and the Caribbean In terms of contracting efficiency, a number of interviewees raised the issue of the relatively small sums on offer from SIAC and the need to find additional funding for the studies. Some felt that the transaction costs of going through SIAC calls to get relatively small amounts of funding were too high71. Transaction costs on the SIAC management side are also relatively high for small projects. However, the bigger issue here is whether it is a good investment for SIAC to leverage other funding, or seen from another perspective, to add value to existing projects with small sums which can be used to collect additional information of interest to impact assessment. This is something which needs to be considered in the analysis of SIAC/SPIA’s niche and comparative advantage. 2.4.2. The role and functioning of the PSC As explained in Section 1, SIAC sits within a rather complex structure. While SPIA reports to ISPC, and through ISPC to the Fund Council (now the System Council), the SIAC project (which funds virtually all of SPIA’s work) has its own Project Steering Committee (PSC), which includes the main two SIAC donors72. In 2013, when the proposal was presented for W1 funding of SIAC, the PSC structure caused 71 For example, one researcher told us that it took him as much time to go through the call process and negotiate the contract to get $100,000 from SIAC as for several million dollars from a bilateral donor. 72 Both SIAC donors are FC (now SC) members, and one of them (DFID) represented the FC on the PSC as from the second meeting, as well as routing its funding through the CGIAR Fund Window 1. However, for structural reasons, the FC has not been able to exercise effective governance over SIAC, so in the absence of this, a number of key stakeholders 36 iea.cgiar.org SIAC Evaluation - Report some friction with the Fund Council (FC), which noted inter alia: “The present proposal is of concern because it transfers that independence [of SPIA] into a new program [sic] with a committee structure and separate advisory council comprising various elements of the system over a large part of the core funding and activity of SPIA, given that SIAC will be the main SPIA activity” 73. However, it was accepted by the FC as a fait accompli. Box 1: The composition and ToR of the PSC 1. Project Steering Committee Composition: (changes from original project proposal in italics) • SPIA Chair - Chair • SPIA Secretary (secretary) - Member • Consortium Financial Manager - Member • A Fund Council member representative – Member (DFID) • BMGF representative - Member • Independent outside expert (nominated by Chair) – Member (Distinguished IA expert from UC Davis) • CGIAR Consortium CEO, or his representative – Observer (only attended first meeting) • Head of IEA – Observer • Each meeting was also observed by one or more staff of the Secretariat and when relevant, activity leaders, in particular MSU. Primary Function: i) Provide strategic guidance (ensure right direction/on-course); ii) Provide project-level oversight (quality control function). Operational Tasks: • Appointing the Objective/Activity Team Leaders (OTL)/(ATL) & approving the sub-grant to MSU; • Approving OTL/ATL’s and MSU’s operational plans (consistent with proposal objectives); • Setting priorities for competitive grants and approving criteria for evaluating proposals; • Making final selection of competitive grants; • Reviewing scheduled (early, mid-term and final) reports of large competitive grant projects; • Commissioning and receiving reports of internally commissioned external reviews of the full project at mid-point (December 2015); • Reviewing expenditures against budgets; • Considering/approving increased budget requests; Meeting frequency: • Face-to-face meetings with all OTL/ATLs to review progress annually; • Quarterly meetings with individual OTLs/ATL by skype; • Periodic attendance (as available) at planning meetings The composition and planned ToR of the PSC are in Box 1 above. Each meeting has generally been attended by 6 to 12 people. Over half of the participants – and in many cases over two thirds – are from SPIA or project implementers74. SPIA also provides the chair and secretariat. So despite its title as a feel that a central CGIAR body has been taken over inappropriately by a bilateral project. This issue is discussed below under ‘governance’. 73 Extract from discussion of SIAC proposal for funding through W1 in Fund Council minutes, 9th Fund Council Meeting, Apr 25-26, 2013, New Delhi, India, pp 18-20 http://library.cgiar.org/bitstream/handle/10947/2967/FC9%20SUMMARY%2c%20Final.pdf?sequence=1 74 Many are ‘observers’, but it was noted from the minutes that discussion was not confined to official members, and in fact many of the comments and suggestions which led to action points were made by ‘observers’. 37 iea.cgiar.org SIAC Evaluation - Report ‘steering committee’, the PSC minutes read more like those of a management team meeting with some interested observers. The PSC has met assiduously; as often as once a month in the first few months of the project. The external members have been very supportive to SIAC, and have taken time from their other jobs, often at short notice, to respond to a wide variety of requests, from commenting on technical study proposals to giving ‘no objections’ to staff recruitment and agreeing to details such as the removal of one country from a particular study. The minutes are clearly written and appear to reflect the meetings fairly (not only because they are approved by members, but because they appear to be open and honest about any problems and disagreements). Despite the above positive points, the evaluation team judges that the PSC is not appropriately configured at present. It performs an unusual mixture of management, technical advisory and (more rarely) governance tasks, but is not set up well for any of these in terms of composition and structure. Furthermore, the meetings are overloaded with information so that there is little time for members to step back and take strategic decisions. A structured analysis of PSC minutes, covering the first two years of the project75, showed that the PSC took relatively few strategic decisions on the direction of the work programme (the governance function). Specifically: • Out of 80 agenda items examined, only 18 (23%) involved an actual PSC decision. The majority of agenda items were information and updates, with no comments recorded in the minutes. Many of these involved administrative and financial issues. • Out of the 18 decisions taken, 12 (two thirds) related to approval of calls for proposals, sets of concept notes or proposals which had already been peer reviewed, or (in four cases, or nearly a quarter of all decisions taken) approving a single research proposal. One PSC meeting was called specifically to give the go-ahead for a specific research proposal. In our view, this is not a good use of PSC time given that the proposals have already been through a separate specialist peer review process. At best there is a risk of micromanagement by committee and duplicating the work of the peer review. At worst, there is a danger that this becomes a rubberstamping exercise, since the majority of external committee members do not have the specialist knowledge and time to realistically “approve” detailed research proposals, and in any case there is rarely the time for detailed discussion76. • The PSC were also asked to approve (on a no objection basis) various administrative decisions such as the recruitment of staff in SPIA and MSU and to give the go-ahead for contracting without competition in two cases where specialist skills were needed. 75 Available on request from the evaluation team. We did not have a complete set of PSC minutes, so the analysis covers only numbers 1-8 covering Feb 2013 to October 2014 and also an additional meeting in Dec 2014. However this represents the period when most of the key decisions on direction and activities were taken in the project. We later got hold of minutes for three further PSC meetings in 2015-16, which all dealt with approval of research proposals. 76 Only in two cases were there any questions or objections recorded to particular research proposals, and in both cases the proposals were passed without further modification. Part of the problem seems to be that by the time decisions come to the committee, much hard work has already been put into developing partnerships and thinking through particular research proposals, so it is psychologically difficult for the PSC to start raising fundamental questions. 38 iea.cgiar.org SIAC Evaluation - Report • The PSC approved the MSU workplan and budget in their second meeting, although questions were raised in discussion about the basis for prioritization and about the approach to institutionalization. Following that, there is no record in the minutes of any major decision on overall priorities77. (Having said this, most of the PSC participated in the internal review Mid- Term meeting in March 2016 where broader prioritization issues were discussed, although no decisions were taken.) • The skills of the independent IA specialist were rarely called on in the PSC meetings, according to the minutes. He was asked to help advise SIAC (offline) on plans for an IA quality assurance scheme, and he made some pertinent comments on issues such as questionnaire design, but it is not clear that the PSC was an effective means to make use of his (considerable) expertise. The PSC, while adding some value, also has substantial direct and indirect costs. One issue is that the PSC effectively constitutes an additional layer of management, although not all the members are equally qualified for such a role. The requirement for the PSC to discuss and approve individual studies (normally a management function) can delay the start-up of studies by as long as several months (see Figure 10). More importantly, however the PSC, while undertaking some governance functions, is not appropriately composed or structured to act as a governance body, and its decisions cannot substitute for effective consultation with key entities of the CGIAR when it comes to (often implicitly) approving new directions for SIAC research. For the above reasons, we suggest that for any future phase, SIAC should replace the PSC (as currently constituted) with a more conventional and smaller ‘management group’ structure for taking management decisions – this could be for example the SPIA chair, selected SPIA members and secretariat staff, and an ISPC representative. This management group should be given the power to approve research studies within the portfolio, as long as it can show that a proper review process is being followed. IA experts could be brought in as needed to discuss specific technical issues. If donors need to approve certain line items of expenditure, that could be done in a more conventional and efficient fashion - e.g. through written ‘no objections’. The CGIAR governance of SIAC/SPIA would be handled separately, and SIAC donors and the System Office would almost certainly be represented on any governance body (although this is outside the scope of this evaluation). If necessary, to complement the above, SIAC/SPIA could continue to hold key stakeholder/donor meetings at regular (for example annual) intervals to discuss progress and solicit views on the direction of travel. 2.4.3. The governance of SIAC/SPIA We have been asked to leave broader questions of SPIA governance for the forthcoming ISPC/SPIA evaluation, so only offer some initial observations here. The Fund Council has expressed78 a legitimate (in our view) concern that SIAC is a stand-alone donor project supporting a central CGIAR institution, and that bilateral donor interest should not be allowed to dictate central priorities79. On the other hand, the vast majority of those interviewed felt that IA is under-resourced in the CGIAR and that a central institution like SPIA is required to support this. SIAC is 77 In PSC 6, for example, the minutes record that (in the middle of a discussion on W1 financing) the chair informed the meeting of "major rebudgeting exercise undertaken since last PSC - reviewing and reallocating across SIAC activities". There was no (minuted) discussion on this point in this or the following meetings. 78 See Footnote 73 79 The donors concerned did try to avoid this, within their own procedural constraints – see footnote 72 39 iea.cgiar.org SIAC Evaluation - Report therefore mainly seen as a reasonable response to the mismatch between the demand for IA and the failure of the FC to allocate sufficient resources to this area. The concern shifted therefore to whether the FC had sufficient oversight of SIAC. The SIAC project has been functioning during a period of major institutional change for the CGIAR (in particular, the changeover from the Fund Council and Consortium to the System Council and System Office). One of the recurring discussion items in the PSC meetings was the failure of the Fund Council to provide an effective mechanism for governance of SPIA (and SIAC) work. The FC itself was too large, unwieldy and overburdened to serve as a direct governance body without an intermediary oversight group. A number of people, in particular the head of IEA, pressed hard for the establishment of an ‘Evaluation and Impact Assessment Committee (EIAC)’ within the FC which could take a closer look at work plans of both IEA and SPIA. However, despite some initial meetings, and the formal establishment of EIAC as a committee of the Fund Council80, EIAC never managed to get off the ground properly (no minutes are available, and the meetings petered out) and it apparently never looked at any of SPIA’s/SIAC’s work. The charter of the new CGIAR System Council includes the mandate to “approve plans and financing of ex-post impact assessment of the CGIAR Portfolio proposed by the ISPC after consultation with the System Management Board”. (CGIAR, 2016 p10). The planned establishment of new, standing ‘Strategic Impact, Monitoring and Evaluation Committee’ in the System Council could potentially act as part of the governance mechanism and provide greater scrutiny of the work of SPIA/SIAC. The other important potential route for SPIA governance is through the ISPC. SPIA has regularly presented its (SIAC) workplans and reported on progress to the ISPC, and the ISPC in turn has been the normal route through which SPIA plans and progress can be communicated to the Fund Council. However, the minutes of ISPC and FC meetings appear to show that agendas were generally packed and the time to cover SPIA/SIAC issues relatively short, and that very little if any discussion on SPIA workplans and progress took place. We also noted that the ISPC (outside SPIA members) is not officially represented on the PSC81. 2.4.4. Summary of findings on Management and Governance The management of the SIAC project has been efficient, within the institutional constraints it has faced, and there is evidence of improvement in efficiency over the lifetime of the project. However, the administrative systems of FAO have been a major constraint, leading to delays and limitations on contracting partners, and the closure of the incipient SIAC small grant scheme. One area for possible improvement is to broaden out external partnerships, in particular to get greater involvement of nationals of countries where the work is taking place. 80https://library.cgiar.org/bitstream/handle/10947/2992/Evaluation%20and%20Impact%20Assessment%20Committe e,%20TOR.pdf?sequence=1 81 The ISPC chair did attend the 7th PSC meeting as an observer, and raised several key strategic issues (similar to some issues raised in this evaluation), but no follow-up was minuted. The ISPC Chair and Executive Director also contributed to the Mid-term Review in Feb 2015, but this meeting was not limited to PSC members. 40 iea.cgiar.org SIAC Evaluation - Report Governance of the project by the CGIAR system has been weak. SIAC has operated during a period of major top-level institutional reform. Broader SPIA governance issues are outside the scope of this evaluation, but we note that the Fund Council has been unable to exercise effective governance of SPIA/SIAC to date, and that there may be an opportunity to address this with the new System Council through its ‘Strategic Impact, Monitoring and Evaluation Committee’, and/or through the ISPC82. In our view, despite the commitment and hard work of its members, the Project Steering Committee (PSC) is not appropriately composed and configured. The PSC currently performs an unusual mixture of tasks, most of which would normally be the responsibility of SIAC/SPIA management. 82 The ISPC’s remit is also under discussion and its revised ToR have not been agreed by the System Council at the time of writing. 41 iea.cgiar.org SIAC Evaluation - Report 3. Conclusions The SIAC has given a major boost to central IA resources in the CGIAR. In the first phase of SIAC, SPIA has undertaken a broad and ambitious programme of work, aided by a highly-qualified, committed, reflective and hard-working team. Regarding relevance: We found that the design of SIAC is broadly relevant to the mandate of SPIA and to the needs of CGIAR funders and Centers/CRPs. However, there are a number of areas for improvement, as detailed in the report. There are two overarching messages. First, the theory of change of SIAC needs revisiting in depth before taking major decisions on the scope and activities of any further phase of SIAC. Second, more work needs to be done to agree the comparative advantage of SPIA and its priority activities vis-a-vis IA conducted by Centers and CRPs, based on an in-depth study of IA work across the CGIAR. We believe this is a priority for the forthcoming evaluation of ISPC/SPIA. Regarding effectiveness: generally good progress has been made against planned outputs and productivity has been high. This has included inter alia some innovative and influential work, for example on methods for varietal adoption studies. However, the focus in SIAC Phase I on quickly delivering a large number of high quality studies has had some downsides, including fragmentation of the portfolio and loss of opportunities for longer term institutional learning. We believe that for the next phase of the work, more thought should be given to the theory of change, options for management of the research, and the institutional learning and communications investments needed to reach the wider outcomes desired. Regarding quality of science: it was too early to assess outputs, but we looked at research proposals and SIAC processes. Generally, quality of science was judged to be high. We did find some areas for improvement, particularly in the specification of research calls. Based on an analysis of accepted research proposals, the weakest areas are the lack of information on distributional impacts (disaggregated data), research ethics and open data. The project team has gradually been gaining experience in how to manage research calls and improve quality, including calls managed by partners. We encourage SIAC/SPIA to build on this experience and develop consistent documentation and procedures that tackle the weaknesses mentioned. Regarding management and governance: The management of the project has been efficient, within the institutional constraints it has faced (in particular, FAO systems). One area for possible improvement is to broaden out external partnerships, in particular to get greater involvement of national impact assessment specialists/institutions from countries where the work is taking place. In contrast, governance of the project by the CGIAR system has been weak. Broader SPIA governance issues are outside the scope of this evaluation, but we note that the Fund Council has been unable to exercise effective governance of SPIA/SIAC to date, and that there may be an opportunity to address this with the new System Council. Furthermore, we concluded that despite the commitment and hard work of its members, the Project Steering Committee (PSC) is not appropriately composed and configured for either a management or a governance role, and we suggest that SIAC consider replacing it with a management committee. 42 iea.cgiar.org SIAC Evaluation - Report 4. Recommendations We gathered quite a rich set of ideas and observations in the report, which we would like SIAC/SPIA to consider, but only a few have been formulated as evaluation recommendations, i.e. having an official status and requiring a management response. We have supplemented each recommendation below with some suggestions in bullets. The suggestions do not require an official response, but we hope they might be helpful. Recommendations 1-6 are intended for the SIAC/SPIA management, and the suggested timing in most cases is before submitting a Phase II proposal. Recommendation 7 is for IEA and relates to the planned evaluation of ISPC/SPIA in 2017. R1 Revisit the theory of change of the SIAC project before/while designing a new project phase Suggestions to consider: • use a longer time frame, with SIAC/SPIA aims more clearly linked to the CGIAR Strategic Results Framework; • where appropriate, consider how best to design SIAC activities and outputs to institutionalize and strengthen IA across the CGIAR; • involve more stakeholders in developing/validating the theory of change, in particular research leaders in the CGIAR; • get external assistance in facilitating the process. R2 Put in place a more systematic process for selection of IA topics and specific studies undertaken by SIAC Suggestions to consider: • make more systematic use of standardized reviews to identify knowledge gaps and research questions; • review the state of evidence on key indicators (e.g. wealth, nutrition, resilience with particular attention to indicators used for the Sustainable Development Goals and the CGIAR Intermediate Development Outcomes) as well as on economic benefits, using ‘evidence gap maps’ or similar; • group studies by topic, and aim to gather a critical mass of information on a particular theme. R3 Carry out more systematic consultation of CGIAR research leaders on needs for IA and the proposed SIAC work programme, and the division of responsibilities for IA with other parts of the CGIAR • Suggestion: make use of the annual CGIAR Science Leader Meeting. 43 iea.cgiar.org SIAC Evaluation - Report R4 Take steps to improve the utilisation of IA results in the prioritisation of CGIAR research Suggestions to consider: • make linkages with CGIAR ex-ante modellers for two-way feedback loops (as suggested in the main text); • collate, validate and synthesize IA studies across the CGIAR for use (inter alia) in IEA evaluations; • develop a communications strategy which inter alia focuses on practical use of IA results. R5 Invest more strategically in helping to institutionalise IA across the CGIAR Suggestions to consider: • develop - and reach broad agreement on - standards and guidance for IA in the CGIAR, distinguishing the requirements of different types of impact studies; • maintain an updated database of CGIAR IA studies (including ongoing studies where possible); • publish a regular (annual or biennial) review of IA funding and progress across the CGIAR; • help ensure that relevant CGIAR policies on science quality, ethics, inclusion/distribution of benefits and open data are reflected in all IA calls and contracts. Where policies are weak, help strengthen them; • agree a Terms of Reference for IAFPs, also taking into consideration their roles in other Communities of Practice (CoPs); • open up CoP meetings (not just to IAFPs) and make more use of online engagement, e.g. webinars; • invest more in communications and liaison - dedicate staff time for this – and develop a (two- way) communications strategy. R6 Revisit the management and governance of SIAC/SPIA Suggestions to consider: • seek clarification from the System Council on an appropriate and workable governance structure for SPIA, to cover SIAC work (as part of a wider review of system functions); • replace the PSC with a management group that will handle the majority of its current tasks. Continue periodic consultations with donors and other key stakeholders, but do not involve them in normal management decisions; • with the System Council, consider different funding mechanisms and channels for SIAC/SPIA. 44 iea.cgiar.org SIAC Evaluation - Report R7 (Recommendation for IEA): In the planned evaluation of SPIA, include an analysis of IA roles and responsibilities across the CGIAR • We recommend that the planned IEA evaluation of ISPC/SPIA in 2017 should not look at SPIA in isolation, but analyze IA and related activities right across the CGIAR and consider roles and responsibilities. It should also consider appropriate academic disciplines needed (qualitative as well as quantitative). 45 iea.cgiar.org SIAC Evaluation - Report 5. References cited in the text (Note that this is not a complete list of documents consulted, which would be unfeasibly long - draft research proposals, meeting minutes etc.). A4NH (2015) Using Theories of Change to Manage and Monitor Progress towards Outcomes: Updated August 2015. Unpublished report - available on website. A4NH. Available from: http://www.a4nh.cgiar.org/files/2014/03/Using-Theories-of-Change-to-Manage-and- Monitor-Progress-towards-Outcomes.pdf Alston JM, Chan-Kang C, Marra MC, et al. (2000) A meta-analysis of rates of return to agricultural R&D: Ex pede Herculem? Intl Food Policy Res Inst. Aw-Hassan A (2016) ICARDA RBM-IA strategy. Available from: https://sites.google.com/site/spiaiaworkshops/presentations/SIAC%20Program%20Report_2 2%20July%202016_draft.docx?attredirects=0&d=1 Brennan JP and Malabayabas A (2011) International Rice Research Institute’s contribution to rice varietal yield improvement in South-East Asia. ACIAR Impact Assessment Series, ACIAR. Available from: http://aciar.gov.au/files/node/13941/international_rice_research_institute_s_contribu_3906 9.pdf CGIAR (2016) CGIAR System Framework (June 2016). Available from: http://cgiarweb.s3.amazonaws.com/wp-content/uploads/2016/06/CGIAR-Framework- Document-June-17.pdf Compton J, McLean D, Emmens B, and Balagamwala, M. (2015) Independent CRP-Commissioned External Evaluation of the CGIAR Research Program on Agriculture for Nutrition and Health (A4NH). Available from: http://iea.cgiar.org/evaluation/crp-commissioned-external- evaluation-ccee-agriculture-nutrition-and-health-a4nh DFID (2014) How to Note: Assessing the Strength of Evidence. Available from: How to Note: Assessing the Strength of Evidence. Gakusi A-E and Sindzingre AN (nd) Credibility and Independence in Development Evaluation. Available from: http://repository.uneca.org/bitstream/handle/10855/3649/Bib.%2029046.pdf?sequence=1 ISPC-SPIA (2014) Adoption of Modern Varieties of Food Crops in SSA. Independent Science and Partnership Council - Standing Panel on Impact Assessment (CGIAR). Available from: http://ispc.cgiar.org/publication/adoption-modern-varieties-food-crops-ssa-brief-number-42 Johnson N, Guedenet H and Saltzman A (2015b) What Will It Take for Biofortification to Have Impact on the Ground? Theories of Change for Three Crop-country Combinations. CGIAR Research Program on Agriculture for Nutrition and Health. Available from: http://ebrary.ifpri.org/cdm/ref/collection/p15738coll2/id/129089 46 iea.cgiar.org SIAC Evaluation - Report Kelley TG and Gregersen HM (2005) NRM impact assessment in the CGIAR: meeting the challenges and implications for CGIAR centres. In: Natural Resources Management in Agriculture: Methods for Assessing Economic and Environmental Impacts, pp. 341–359. Mayne J and Johnson N (2015) Using Theories of Change in the CGIAR Research Program on Agriculture for Nutrition and Health (Draft). A4NH. Available from: http://www.a4nh.cgiar.org/files/2014/03/Mayne-and-Johnson-2015-for-Website.pdf Mohanty S, Arouna A and Labarta R (2016) RICE: Second phase CGIAR Research Program on Rice: Impact Assessment Framework. Available from: https://sites.google.com/site/spiaiaworkshops/presentations Perrin, B. (2002). How to—and how not to—evaluate innovation. Evaluation, 8(1), 13-28. Picciotto R (2013) Evaluation independence in organizations. Journal of MultiDisciplinary Evaluation 9(20): 18–32. Pritchett L (2002) It pays to be ignorant: A simple political economy of rigorous program evaluation. The Journal of Policy Reform 5(4): 251–269. Pritchett L and Sandefur J (2013) Context matters for size: why external validity claims and development practice do not mix. Journal of Globalization and Development 4(2): 161–197. Raitzer DA and Kelley TG (2008a) Assessing the contribution of impact assessment to donor decisions for international agricultural research. Research Evaluation 17(3): 187–199. Raitzer, DA., and Kelley TG (2008b) "Benefit–cost meta-analysis of investment in the International Agricultural Research Centers of the CGIAR." Agricultural Systems 96.1 (2008): 108-123. RTB (2016) FP5: Impact at Scale 2017-2022. Available from: https://sites.google.com/site/spiaiaworkshops/presentations/Session%201_CRP%20RTB_Kiku lwe.pptx?attredirects=0&d=1 Ruel MT and Alderman H (2013) Nutrition-sensitive interventions and programmes: how can they help to accelerate progress in improving maternal and child nutrition? The Lancet 382(9891): 536– 551. SIAC (2015) Strengthening Impact Assessment in the CGIAR (SIAC) Mid-term Review – Key Outcomes. 27th and 28th February 2015, Rome, Italy. SIAC (2016) Donor Demands Survey, 2014 round, brief results. Unpublished briefing paper. Available from: http://impact.cgiar.org/files/docs/Discussion%201_M%26E_Donor%20demands_Kelley- Krishnan.pdf USAID (2016) Feed the Future Indicator Handbook Definition Sheets. Available from: https://feedthefuture.gov/sites/default/files/resource/files/Feed_the_Future_Indicator_Han dbook_25_July_2016.pdf 47 iea.cgiar.org SIAC Evaluation - Report Walker T, Maredia M, La Rovere R, et al. (2008) Strategic guidance for ex post impact assessment of agricultural research. Prepared for the Standing Panel on Impact Assessment CGIAR Science Council. Available from: ftp://ftp.fao.org/docrep/fao/011/i0276e/i0276e.pdf Watson DJ (ed.) (2003) International conference on impacts of agricultural research and development: Why has impact assessment research not made more of a difference? CIMMYT. Available from: http://impact.cgiar.org/sites/default/files/images/CIMMYT2002.pdf White H (2012) Can we do small n impact evaluations? | Impact Evaluations. Development Impact. Available from: http://blogs.worldbank.org/impactevaluations/guest-post-by-howard-white- can-we-do-small-n-impact-evaluations (accessed 12 September 2016). White H (2014) Is independence always a good thing? Evidence Matters : Improving development policy and practice. Available from: http://blogs.3ieimpact.org/is-independence-always-a-good- thing/ (accessed 12 September 2016). White H and Phillips D (2012) Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework. 3iE. Available from: http://www.3ieimpact.org/media/filer/2012/06/29/working_paper_15.pdf 48 iea.cgiar.org