T erms of Reference: G eneban k Platform Evaluation June 2023 Photo: Crop Trust/CGIAR Genebanks Initiative Terms of Reference: Genebank Platform Evaluation June 2023 Contents 1 Background .................................................................................................................................................................. 1 1.1 Introduction ................................................................................................................................................................................................... 1 1.2 Platform Leadership, Management, and Governance ........................................................................................................ 3 2 Evaluation Purpose, Scope, and Users .................................................................................................................. 4 2.1 Evaluation Objectives and Criteria with Key Evaluation Questions ............................................................................. 5 2.2 Approach and Methodology .............................................................................................................................................................. 6 2.3 Consideration and Expected Limitations to the Evaluation ............................................................................................. 7 3 Evaluation Purpose, Scope, and Users ................................................................................................................... 7 3.1 Evaluation Timeline and Management ........................................................................................................................................ 7 3.1.1 Preparatory Scoping Phase ................................................................................................................................................... 8 3.1.2 Inception Phase ............................................................................................................................................................................ 8 3.1.3 Inquiry, Data Collection and Analysis Phase ............................................................................................................. 10 3.1.4 Reporting Phase .......................................................................................................................................................................... 12 3.1.5 Dissemination, Use and Knowledge Management ................................................................................................. 12 4 Evaluation Management, Roles and Responsibilities ....................................................................................... 12 4.1 CGIAR Management Engagement and Response ...............................................................................................................13 4.2 IAES Management and Responsibilites .......................................................................................................................................13 4.3 Evaluation Team ...................................................................................................................................................................................... 14 5 Evaluation Deliverables, Knowledge Management and Dissemination ...................................................... 15 6 Contract and Payment ............................................................................................................................................ 16 Annex 1: Evaluation Questions and Sub-questions by Evaluation Criteria ........................................................17 Annex 2: CGIAR Genebank Platform-Overview ...................................................................................................... 20 A2.2 Purpose and Objectives of CGIAR Genebank Platform..................................................................................................... 21 A2.3 The Genebank Platform’s Impact Pathway and Theory of Change ........................................................................22 A2.4 Platform Modules, Objectives, Targets and Activities ..................................................................................................... 23 A2.5 Platform Partnership Ecosystem and Users ........................................................................................................................... 27 A2.6 Genebank Platform Funding and Budget .............................................................................................................................. 28 Annex 3: List of Stakeholders Consulted Toward TOR Development ................................................................... 31 Annex 4: Preliminary List of Readings ........................................................................................................................ 32 Annex 5: Requested points of Engagement and Tasks for the Evaluand’s MEL Focal Point .......................... 33 List of Tables Table 1. User groups by evaluation objectives ......................................................................................................................................... 4 Table 2. Potential evaluation questions or sub-questions, Genebank Platform .................................................................. 5 Table 3. Accessions of CGIAR genebanks in 2021 ................................................................................................................................ 20 Table 4. Mapping of stated Platform outcomes per modules (source IAES).......................................................................22 Table 5. Activities, outputs and targets of the Conservation module ..................................................................................... 23 Table 6. Activities, outputs and targets of the Use module........................................................................................................... 25 Table 7. Activities, outputs and targets of the Policy module ...................................................................................................... 26 Table 8. List of Platform partnerships .......................................................................................................................................................... 27 Table 9. 2017 Genebank Platform financial summary ...................................................................................................................... 28 Table 10. 2018 Genebank Platform financial summary .................................................................................................................... 29 Table 11. 2019 Genebank Platform financial summary...................................................................................................................... 29 Table 12. 2020 Genebank Platform financial summary ................................................................................................................... 29 Table 13. 2021 Genebank Platform financial summary ..................................................................................................................... 30 List of Figures Figure 1. CGIAR Genebanks under the Genebank Platform ............................................................................................................... 1 Figure 2. Genebank Platform management and governance ....................................................................................................... 3 Figure 3. Indicative Genebank Platform evaluation timeline, 2023 .............................................................................................. 8 Figure 4. Internal and external CGIAR genebank distributions, 2017-21 .................................................................................... 21 Figure 5. External genebank distributions by user group, 2017-21 ............................................................................................... 21 Figure 6. Impact pathway and theory of change ............................................................................................................................... 23 Terms of Reference: Genebank Platform Evaluation 1 Background 1.1 Introduction Through the Genebank Platform, CGIAR genebanks managed collections of more than 20 staple crops in 12 locations on five continents.1 The collections remain freely available upon request to thousands of users worldwide under the International Treaty on Plant Genetic Resources for Food and Agriculture (ITPGRFA), accounting for a large amount of the germplasm exchanged every year under the multilateral system of access and benefit sharing.2 CGIAR genebanks safeguard some of the largest and most widely used collections of crop diversity in the world, critical to attaining global development goals to end hunger and improve food and nutrition security. The genebanks- as a key driver of the international exchange of Plant Genetic Resources for Food and Agriculture (PGRFA)-are fundamental to delivering the CGIAR 2030 Research and Innovation Strategy.3 Figure 1. CGIAR Genebanks under the Genebank Platform4 In 1989, the Food and Agriculture Organization of the United Nations (FAO) commission called for the creation of an International Network of Ex Situ Collections. In 1994, 11 of the International Agricultural Research Centers (IARCs) of CGIAR5 signed formal agreements with FAO, placing collections into the International Network, under the auspices of FAO.6 In 2004, the ITPGRFA entered into force. Article 15 of the 1 At the time of the Platform there were 11 Genebanks in 12 locations (ICARDA has two genebanks, one in Morocco and one in Lebanon). Only centers that to date have signed the CGIAR Integration Framework Agreement receive funding under the Genebank initiative (2022-2024), so the number of locations has decreased from 12 to 10. The Map on Genebanks Initiative does not depict Bioversity Genebank in Belgium, hence additional information will be assessed during inception to confirm the number of current locations. 2 See https://www.cgiar.org/initiative/genebanks/. 3 Replacing the retired CGIAR Strategy and Results Framework (SRF) with three System Level Outcomes (SLO): reduced poverty, improved food and nutrition security, and improved natural resources systems and ecosystem services. 4 CGIAR genebanks are located in the Ivory Coast (AfricaRice), Colombia and Belgium (Alliance of Bioversity and the former International Center of Tropical Agriculture, or CIAT), Lebanon and Morocco (International Center for Agricultural Research in the Dry Areas, or ICARDA); originally collections were in Syria, but they were moved in 2013), Kenya (International Centre for Research in Agroforestry, or CIFOR-ICRAF), Nigeria (International Institute of Tropical Agriculture, or IITA), Ethiopia (International Livestock Research Institute, or ILRI), India (International Crops Research Institute for the Semi-Arid Tropics, or ICRISAT), the Philippines (International Rice Research Institute, or IRRI), Mexico (International Maize and Wheat Improvement Center, or CIMMYT), and Peru (International Potato Center, or CIP). See Genebanks | CGIAR Genebank Platform 5 CIAT, CIMMYT, CIP, ICARDA, ICRAF, ICRISAT, IITA, the former International Livestock Center for Africa (now ILRI), IRRI, CIFOR- ICRAF and the former West Africa Rice Development Association (now AfricaRice). 6 To be elaborated on the Inception Report, based on Annex 1 of the agreements. 1 Terms of Reference: Genebank Platform Evaluation Treaty recognizes the importance of the ex situ collections of CGIAR while, in turn, the IARCs recognize the authority of the Treaty Governing Body when it comes to providing policy guidance on collections. Through agreements signed by each IARC and the Treaty Governing Body in 2006, IARCs are obliged to make the collections and associated data they manage available through the Multilateral System of Access and Benefit Sharing of ITPGRFA. Under the same agreements, the genebanks are bound to “manage and administer these ex situ collections in accordance with internationally accepted standards, as endorsed by the FAO Commission on Genetic Resources for Food and Agriculture” (FAO, 2014). Between 2012 and 2021, CGIAR’s 11 genebanks worked together under the independent oversight of the Global Crop Diversity Trust (Crop Trust) in the Genebank CGIAR Research Program (CRP) (2012‒16) and, subsequently in the Genebank Platform (2017‒21). The Crop Trust is an international non-profit organization working to conserve crop diversity and protect global food and nutrition security. At the core of the Crop Trust is an endowment fund dedicated to providing guaranteed, long-term financial support to key genebanks worldwide. The Crop Trust was established in October 2004 by FAO and Bioversity International on behalf of CGIAR for the purpose of sustainably supporting a global system for the conservation and use of crop diversity through its Crop Diversity Endowment Fund.7 The Crop Trust supports the Svalbard Global Seed Vault and coordinates large-scale projects around the world to secure crop diversity and make it available for use globally, forever, and for the benefit of everyone. The Crop Trust is recognized as an essential element of the funding strategy of ITPGRFA. The Genebank CRP was managed by the Crop Trust and independently evaluated in 2017. At that time, the CGIAR genebanks first reported their status and progress at a systemic level using common metrics. The development and adoption of improved and common data management tools accelerated. The Genebank Platform-a comprehensive five-year program for the management of secure and sustainable funding of the collections of PGRFA held by the CGIAR genebanks-provided a more robust governance structure and continued to improve integration and cohesion between genebanks and crops. The Platform was a partnership between the members of CGIAR and the Crop Trust. It supported the core activities of CGIAR genebanks, namely, conserving and making available crop and tree diversity to help them meet international standards, improve efficiency, and ensure more effective use of collections within a supportive policy environment. The Platform was important, as the CGIAR genebanks had different levels of experience and resources but were recognized under one brand—CGIAR—which is widely taken to symbolize the highest operating standards. The Platform’s (2017-21) three core modules were: 1. Conservation, which helped genebanks work strategically to exploit new opportunities, conserve more diversity, and respond to more demands while controlling costs. The main objective of the Conservation module was to support and improve essential genebank operations, ensure that germplasm was secure and available, and improve genebank operations and management. 2. Use, which helped genebanks align their operations toward more targeted use and exploitation of the collections. The main objective was to empower the effective use of plant genetic resources by offering more effective access to crop diversity and information. The Use module went beyond increasing the number of accessions distributed. It tackled some fundamental constraints that genebank clients, such as breeders, faced when selecting crop diversity, helping them to make more informed and precise decisions, uncovering new traits or characteristics in well-known crops, or just making it easier to rapidly get hold of clean materials. The path to success involved more—and more useful— information about each accession. The Genesys information system-originally co-developed in 2013 by Bioversity International (on behalf of CGIAR’s System-wide Genetic Resources Program), the Crop 7 Retrieved April 2023, from https://www.croptrust.org/about/. 2 Terms of Reference: Genebank Platform Evaluation Trust, and the Secretariat of ITPGRFA-has been managed and maintained by the Crop Trust. The Use module has helped enhance and facilitate its use within CGIAR genebanks. The increased use of Genesys aims to provide researchers with an effective way of searching for information on genebank accessions and ordering samples directly from the genebanks. 3. Policy, which supported the active engagement of CGIAR in shaping international genetic resource agreements. This module ensured a supportive policy environment for CGIAR genebanks, breeding programs and partners. The policy module worked closely with other CGIAR constituencies, including the Excellence in Breeding and Big Data Platform, the Centers’ intellectual property focal points and the System Office. In 2022, under its new portfolio, CGIAR launched the Genebanks Initiative, under the Genetic Innovation science group. CGIAR manages the Initiative, representing a transition from the Crop Trust’s previous oversight and management of the Platform and CRP of the same name. The Initiative aims to support the global system for the conservation and use of PGRFA.8 The Initiative built on the Platform’s modules, expanding the effort on use in breeding programs and work with national partners.9 1.2 Platform Leadership, Management, and Governance The Executive Board of the Crop Trust took ultimate responsibility for the Platform’s governance. The CGIAR representative on the Executive Board was responsible for ensuring that the opinions of the CGIAR System Board and Office were represented. The Crop Trust coordinated overall the Platform. The Platform coordinator was responsible for overseeing the technical and financial management of the Platform as a whole and coordinating the activities of the Conservation and Use Modules with the guidance of a management team. The management team comprised the platform coordinator10, the policy module coordinator, the Genebank Health Units (GHU) coordinator, the three elected members of the Executive Committee of the Article 15 group (A15)11, and the Crop Trust science team leader. Figure 2. Genebank Platform management and governance12 8 See https://www.cgiar.org/initiative/genebanks/. 9 See https://www.cgiar.org/initiative/genebanks/. 10 The Genebank Platform Coordinator was Charlotte Lusty. In September 2022, she was hired as CGIAR Genebank Initiative co-lead. 11 The Article 15 group superseded the former Inter-Center Working Group on Genetic Resources. It comprised the genebank managers of the CGIAR Centers, which had designated their collections under Article 15 of the ITPGRFA. The A15 Executive Committee was composed of three elected genebank managers (CIMMYT, CIP and IRRI). 12 Source: Genebank Platform Full Proposal Genebank Platform Full Proposal 2017-2022 (cgiar.org). 3 Terms of Reference: Genebank Platform Evaluation The platform coordinator oversaw the technical and financial management of the Platform as a whole and coordinated the activities of the conservation and use of modules with the guidance of the management team. The policy module was coordinated by Bioversity International with International Rice Research Institute (IRRI). While CGIAR Centers and genebank management had responsibility for the day-to-day operations of the genebanks, the Platform activities were implemented under the overall guidance of the management team, which had the role of: • Directing funding allocations to collective activities • Planning and monitoring collective activities • Developing indicators and targets and reviewing progress toward them • Providing overall guidance on the management of the Platform and the execution of reviews • Convening meetings and other events. 2 Evaluation Purpose, Scope, and Users The CGIAR Independent Advisory and Evaluation Services (IAES) 2022‒24 multi-year workplan (2021; Decision Reference SC/M14/DP4, confirmed) provides for the 2023 independent evaluation of the Genebank Platform.13 The CGIAR IAES Evaluation Function will execute the evaluation consistent with its mandate set in the IAES terms of reference (2018). An independent external evaluation in 2023 will contribute to Crop Trust and CGIAR institutional learning and provide evidence of the efficiency and effectiveness of the Genebank Platform (2017‒21). The exercise will also identify good practice and lessons on which the Genebanks Initiative and, potentially, CGIAR impact area platforms can build. Thus, there are five targeted users of the evaluation results. Based on the scoping phase, other stakeholder and user needs will be elaborated and articulated in the IR: Table 1. User groups by evaluation objectives Targeted user Rationale for prioritizing user-groups Envisioned objective A. CGIAR System Council - The System Council commissions Accountability (commissioner, funder) all independent and external Learning evaluations that IAES executes. B. The Crop Trust, as the lead - The evaluation will support Learning of the Genebank Platform evidence-based decision-making Performance (evaluand) at the Crop Trust, related to its engagement with CGIAR, its genebanks and other partners. C. CGIAR users, including the - For CGIAR Centers that hold collections in Learning Genetic Innovation Science genebanks, to have evidence to enhance Group and the leadership strategically exploiting new opportunities, of the Genebanks Initiative; conserve more diversity, and respond to CGIAR Centers that hold more demands while controlling costs collections in genebanks; - Management of other platforms in CGIAR those in CGIAR managing the previous and new CGIAR Platforms. 13 The Genebank Platform is the last of the four platforms in the CGIAR portfolio to have transitioned in 2022. The other three were subject to evaluations: Big Data in Agriculture (2021), Excellence in Breeding (2022) and the GENDER Platform (ongoing). 4 Terms of Reference: Genebank Platform Evaluation D. External partners such as - Involved in exploitation of the CGIAR Accountability FAO, ITPGRFA, collections Learning policymakers, national - Have a record through an independent lens governments and national about how Genebank Platform performed agricultural research and and delivered on various commitments to extension systems these groups and their needs. (NARES), researchers, the - Provide learning around engagement of breeding community. CGIAR in shaping international genetic resource agreements. E. Requestors of CGIAR - Learning around conservation and use of Accountability Genebank Accessions - PGRFA under the CGIAR Platform Learning NARES, industry, private - Learning about engagement of CGIAR with sector, other CGIAR partners stakeholders, others - Learning about Platform’s network. F. Entities outside CGIAR - Foster exchanges of knowledge and Learning System that hold plant information genetic resources (i.e. - Identify the genetic resources collection national genetic resources gaps genebanks, universities) - Reduce duplication of efforts and resource use in germplasm collection, maintenance and distribution - Formulate mutual understanding and co- development of policies to bring more germplasm of different commodities under the umbrella for exchange, evaluation and use To the extent feasible-given the resource and time allocated to the evaluation-key stakeholders will be widely consulted and engaged throughout the evaluation process through relevant channels and using the appropriate engagement tools. 2.1 Evaluation Objectives and Criteria with Key Evaluation Questions Consistent with the evaluation objectives to support learning and accountability among the key user and stakeholder groups, the evaluation will collect, analyze, and present the information to meet their diverse needs framed by the following evaluation criteria: relevance, effectiveness, efficiency, coherence, sustainability and learning for impact14 (Evaluation Policy, 2022). Table 2. Potential evaluation questions or sub-questions, Genebank Platform CGIAR evaluation criteria Key evaluation questions A. Relevance 1) How relevant was the mandate of the Genebank The extent to which the Platform and design respond Platform and ways to achieve it? to the needs, policies, and priorities of users/clients and global, regional, and country partners/institutions, as well as continue to do so if circumstances change. Consistent with the Quality of Research for 14 TBC for IR 5 Terms of Reference: Genebank Platform Evaluation Development (QoR4D) framework, attention is given to the importance, significance, and usefulness of the work implemented in the problem context, associated with CGIAR’s capacity to address the problems. B. Effectiveness 2) To what extent did the Genebank Platform achieve The extent to which the intervention achieved, and/or progress towards intended outcomes? is expected to achieve, its objectives, and its results, including any differential results across subgroups of users/clients. Consistent with the QoR4D framework and in the CGIAR context, this criterion considers the extent to which research is positioned for use and has generated knowledge, products, and services with high potential to address a problem and contribute to innovations, outcomes, and impacts. C. Efficiency 3) How did allocation of resources (funds, people, time, The extent to which the intervention delivers, or is likely expertise, etc.) support the achievement of the to deliver, results in an economical and timely way– Genebank Platform’s outputs and outcomes? that is, the overall use of resources. “Economical” refers 4) What strategies, internal or external mechanisms and to the conversion of inputs (funds, expertise, natural factors contributed to, or inhibited, timely and cost- resources, time, etc.) into outputs, outcomes, and effective achievement of outputs and outcomes, impacts in the most cost-effective way possible intended and unintended? compared with feasible alternatives in the context. D. Coherence and added-value 5) How did the research, evidence, capacity agenda of The compatibility of the intervention with other the Platform complement and/or strengthen related interventions in a country or a sector or within CGIAR; genebank-focused work in CGIAR, towards the its overall fit. Internal coherence addresses the Genebank Initiative? synergies and interlinkages between the intervention 6) How were Genebank platform operations harmonized, and other interventions carried out within CGIAR. aligned, and coordinated with non-CGIAR genebanks? E. Sustainability and learning for impact 7) What learning mechanisms have been built into the The extent to which the net benefits of the intervention Genebank Platform design and implementation to continue or are likely to continue. This criterion focuses facilitate the potential sustainability of Platform on the continuation of benefits, not on external results? funding, and highlights the multidimensional nature of 8) In what ways did the platform contribute to achieving sustainability. global development objectives, notably the SDGs, along its impact pathway? Evaluation questions, especially sub-questions, will be refined and expanded in the Inception Report, based on preliminary interviews and engagements, including with Crop Trust, CGIAR genebank initiative and external partners, such as FAO and national genebanks. During the inception phase of the evaluation, team leaders will establish rubrics to define qualitative assessments (adjectives such as “well-managed”, “appropriately governed”, and “appropriate”). 2.2 Approach and Methodology The CGIAR Evaluation Framework and Policy (2022) will guide evaluation design and implementation. The evaluation design and inquiry will build on the 2017 evaluation of the Genebank CGIAR Research Program (CRP) and will draw on insights from the other CGIAR Platform evaluations: GENDER, Excellence in Breeding, Big Data in Agriculture, as well as the 2021 Synthesis. 6 Terms of Reference: Genebank Platform Evaluation The suggested approach for evaluation is utilization-focused and participatory, using mixed methods of data collection and analysis (see section 3.1.3 for detail), including: • Documents and data reviews: a desk review of documents and related resources, prioritizing use of secondary evidence • Primary data collection: key informant interviews, focus group discussions, online surveys o attendance at the Crop Trust management event for inception discussion in Bonn o up to two field visits to CGIAR genebanks, identified based on to be defined criteria • Analysis: ToC analysis, social network analysis, and selected quantitative methods for addressing “efficiency” evaluation criteria, based on the final evaluation sub-questions and data availability. The evaluation team will be welcome to use innovative approaches in data collection, analysis, and dissemination (as applicable) throughout the evaluation. 2.3 Consideration and Expected Limitations to the Evaluation Understanding of the necessary consideration and expected limitations is grounded in the scoping exercise towards developing this TOR. The following should be considered during the evaluation design, implementation, and analysis of results: Factors that affect the implementation of the Genebank Platform • Germplasm health and use issues, as these impact the availability of crop collections • Availability of crop collections, and their relationship to the Genebank Platform, where these might be addressed in other CGIAR or external modalities, including acquisition and pre-breeding • Part of the activities of the Platform were conducted during a global pandemic (COVID-19). Factors that affect the reception and targeting of the evaluation • The evaluation happens at a time in which the Genebank Platform has already ended its mandate and transited to Genebanks Initiative in 2022, under direct CGIAR management • Real-time learning for the new Genebanks Initiative and future actions • Considerations around best modalities for supporting genebanks in the forthcoming 2025-2027 CGIAR portfolio, and supporting the long term maintenance and improvement of physical infrastructures These and other considerations and limitations should be elaborated upon in the IR, with suggested mitigation strategies. 3 Evaluation Purpose, Scope, and Users 3.1 Evaluation Timeline and Management Evaluations are process-driven and typically divided into four phases: • Preparatory/scoping, including stakeholder consultation and the selection of evaluation team; this phase for this evaluation commenced in January 2023 • Inception, including field trip, briefings, and the development of the IR 7 Terms of Reference: Genebank Platform Evaluation • Inquiry, including desk review and data collection • Reporting, including the analysis of findings and the development of a final report. Not including the preparatory stage, the evaluation will take place between June and November 2023 for transmission to the System Council, System Board, and Executive Management by end of 2023, after endorsement by the Strategic Impact, Monitoring, and Evaluation Committee. Figure 3. Indicative Genebank Platform evaluation timeline, 2023 March-May June 2023 June-Oct. Nov. 2023- Feb. 2024 2023 Inception 2023 Jan. 2024 onwards Scoping Inquiry Reporting Dissemination and use •Stakeholder •Evaluation •Desk review •Preliminary •Targeted consultation team •Analysis of findings webinars •Development induction progress on •Report •Development of Concept •Core the ToC development of blog posts Note document •Interviews •Validation •Development •Selection of review •Survey workshop of other the •Field trip to •Focus group •Draft report knowledge evaluation Bonn- discussions •Report on products, i.e. team evaluand •2-3 case quality briefs •Terms of induction studies assurance Reference •Inception •Social •Report Report (draft Network finalization and final) Analysis process •Pre-design of (SNA) survey •Field trips to •Preparation selected work for genebanks inquiry •Module phase component studies 3.1.1 Preparatory Scoping Phase The scoping for evaluation commenced in January 2023, with initial consultations with Charlotte Lusty, former Genebank Platform coordinator and current CGIAR Genebank Initiative co-lead, along with Enrico Bonaiuti, research team leader for Monitoring, Evaluation and Learning (MEL) at International Center for Agricultural Research in the Dry Areas (ICARDA), and MEL focal point for Genetic Innovation Science Group. Following this, IAES carried out the following tasks: • Reviewed key documents to define the scope and issues surrounding the evaluation • Selected two evaluation co-leaders based on their previous experience in process and performance evaluation, team leadership and knowledge of genebanks’ work • Developed a Concept Note towards the TOR, to bring to the attention of CGIAR Evaluation Reference Group (ERG) • Conducted interviews with Subject Matter Experts (SMEs) which will be involved in the evaluation of specific Platform’s modules. 3.1.2 Inception Phase The inception phase entails an initial review of existent program documentation, related evaluative evidence and relevant external literature. Supported by the IAES, the evaluation team co-leaders will coordinate the development of an IR. Its purpose is to set the basis for the final evaluation report and help ensure a shared understanding of the evaluation conduct between the commissioner (with its needs represented by SIMEC), executing office (IAES) and the external evaluation team, as well as among key stakeholders. The IR serves as a roadmap and as a guiding document in the evaluation process for all the 8 Terms of Reference: Genebank Platform Evaluation stakeholders involved, including the evaluation team members themselves and the evaluand, steering proper implementation, monitoring, and mutual quality assurance in all its stages.15 The IR will focus on the following elements: • Refinement of the evaluation questions and sub-questions, elaboration of evaluation methodology towards a comprehensive evaluation design matrix • A reconstruction of the intervention logic or theory of change of the Genebank Platform • A stakeholder analysis identifying key Platform stakeholders through desk analysis of main documents and discussion with the Platform team, to build into an online survey, SNA and other data collection • Prioritization of strategic issues of importance for emphasis during the inquiry phase, should they have changed • An evaluation report outline, including the division of roles and responsibilities between the evaluation team leader and other members of the evaluation team, identification of people to be interviewed and possible surveys to be conducted, and a debriefing and reporting timetable. These elements will be drawn together and agreed upon in an IR between the evaluation team and IAES. The cornerstone of the IR-the evaluation design matrix-will be circulated to the evaluand for feedback, with a crucial role to be played by the MEL focal point. The IR is also circulated to the evaluand for comments and factual corrections. To quality assure the key evaluation outputs, IAES’s layered quality assurance system will draw from two of its external independent evaluation stakeholder groups: (i) external peer reviewers with relevant expertise, from the IAES/ISDC roster of subject-matter experts; and (ii) some members of IAES’s Evaluation Reference Group. Both groups will be called upon to interrogate the evaluation approach and methodology, enhance the evaluation matrix and review the draft IR. For validation, IAES will circulate the draft IR to the Strategic Impact, Monitoring and Evaluation Committee (SIMEC) via its secretariat, for comments or broad guidance on the evaluation design matrix–particularly to flag if the questions posed will meet the needs of System Council, being the commissioner of the evaluation. IAES will ensure quality assurance and that the evaluation team incorporates the relevant feedback. The final IR subsequently represents the contractual basis for the evaluation team’s work and deliverables of the evaluation, and it will be published on IAES’s website. By the end of inception stage, evaluation team members are expected to have acquired strong preliminary knowledge of the Genebank Platform and the following documents and websites: - Genebank Platform website - Genebank Platform Full Proposal (2016) - Genebank Platform Annual Reports (2017, 2018, 2019, 2020, 2021) - Evaluation of CGIAR Research Support Program for Managing and Sustaining Crop Collections: Genebank CRP (2017) - Crop Trust System Level Review of Genebank Costs and Operations, 2020 - Governance and Management of the Genebank Platform, 2017 - Genebank Platform Annual Plan of Work and Budget (POWB), 2017 2018, 2019, 2020, 2021 - Other reviews and assessments, inter alia, pertinent ISPC and ISDC reviews (if any)16 15 See also blog on IAES’s approach to evaluation inception reports. 16 To be further clarified during the inception phase 9 Terms of Reference: Genebank Platform Evaluation 3.1.3 Inquiry, Data Collection and Analysis Phase The inquiry, data collection, and analysis phase is grounded in the IR. The evaluation team will collect and analyze data and evidence according to the evaluation design matrix detailed in the IR, complete its analysis, and prepare a preliminary list of findings and conclusions. Data collection will follow mixed methods, leveraging both qualitative and quantitative data from primary and secondary sources to understand operating environments and track contextual and programmatic assumptions. At a minimum, the following methods will comprise: a) Document and data review: The evaluation team will analyze the relevant documentation; a component portfolio analysis will be performed by relevant subject matter experts from the evaluation team, based on each Platform’s module. - an analysis of the comparative advantage of genebanks (2022) considering non-CGIAR genebanks - a systematic review of genebank reviews carried out by Genebank Platform management - analysis of monitoring and performance data from a Genebank Online Reporting Tool in the CGIAR Results Dashboard and any other monitoring data - analysis of capacity building and education efforts implemented through the Platform, as reported in Genebank Platform Annual Reports and associated documentation, and assessed by recipients through online survey and targeted interviews - where available, analysis of users’ surveys conducted at CGIAR center level, will review to gather knowledge about users’ experience in relation to requested accessions b) Literature review of scientific publications and impact assessments: the evaluation team will analyze relevant literature and impacts studies, focusing on studies on how to measure the value of Genebanks and on impact studies17. Smale (2020) and Gollin (2020) represent an important starting point for this piece of analysis. c) Online Survey: survey of the targeted stakeholders, including questions per module and questions to inquire about knowledge of the transition from Platform to Initiative. An online survey will also allow comparability between Platform evaluations by IAES. d) Semi-structured key informant interviews and focus group discussions with different segments of the stakeholders identified in the stakeholder analysis and mapping. 18 e) Field visits: between the inception and the inquiry phase, three field visits are forecasted to introduce evaluation and help team leaders familiarize with CGIAR genebanks and provide them with the opportunity to collect data face-to-face during the following trips: - Visit to the Crop Trust headquarters in Bonn on 21 June 2023 - Visit two genebanks: IRRI genebank in the Philippines, and CIP genebank in Peru-TBC Table with criteria will be developed for the IR. Preliminary choice of IRRI and CIP genebanks was defined during the scoping phase. They have very different characteristics in crops and conservation, providing both different learning opportunities and a wider understanding of the challenges implied for a Platform 17 As referred by SPIA, impact studies are limited and specific Genebank Platform impact studies are not available in SPIA portfolio. The most recent literature to review is the special issue on “Genebanks and Food Security in a Changing Agriculture” publish on Food Security (2020), as part of the Genebank Platform Impact Fellowship. 18 Consistent with the approach and in the two CGIAR platform evaluations, number of key informant interviews is not expected to exceed 70, reflecting results of the stakeholder mapping. 10 Terms of Reference: Genebank Platform Evaluation that brings together genebanks with different needs, expertise and crops. Targeted analysis methods would potentially include: f) Stakeholder analysis (SA): including funders, CGIAR genebanks, Platform’s partners, National Agricultural Research systems (NARS) in targeted geographies19, Platform users (e.g., universities, breeders, farmers). It is expected that SA will help better understand the evaluand’s ecosystem and the uses of the evaluation relative to each stakeholder group, as well as to prepare for SNA. g) SNA to identify relations among Platform partners and/or users.20 Through this analysis the team may also assess Platform outreach efforts, online presence and engagement outside the scientific community. h) ToC Analysis: An analysis of the Genebank Platform’s ToC and reconstruction of its intervention logic, will play a central role in the design of the evaluation, in the analysis of the data collected, in the reporting of findings, and the development of conclusions and relevant and actionable recommendations. i) Selected analyses methods for addressing “efficiency” evaluation criteria, will be elaborated to align with the final evaluation sub-questions, and based on data availability, and may include21 o Multi-criteria analysis:22 a form of appraisal which measures variables such as material costs, time savings and project sustainability as well as the social and environmental impacts in addition to monetary impacts. MCA is usually an ex-ante evaluation tool and is particularly used for the examination of the intervention's strategic choices. Multi-criteria analysis is undertaken to identify and compare different policy options by assessing their effects, performance, impacts, and trade-offs. o Cost-utility analysis: develops an overall measure of utility or value based on the preferences of individuals and stakeholders, using participatory methods. Cost-utility analysis is useful for evaluating and comparing programs that aim to reach the same goal in non-monetary terms.23 j) Case studies or deep dives embedded or as part of module studies would potentially cover: o Germplasm health units (GHU) conducted by a designated SME. Included in this would be Comparative Advantage analysis to help define CGIAR position in germplasm conservation and distribution compared to other similar organizations. o Long-term financing and covering assets, conducted by a designated SME. o Safety duplication and back up of seeds and cryopreserved materials: ICARDA and Syrian collection, other center TBC (in support of effectiveness of activity 1.1.1) Evaluability Assessment of the Genebank Initiative will be conducted in the scoping phase and the reporting phase. It will draw on Conducting and Using Evaluability Assessments in CGIAR: CGIAR Evaluation Guidelines to operationalize the “evaluability” principle,24 and will be embedded in the evaluation. The Evaluability Assessment will (i) contribute to finalizing evaluation questions to validate forward-looking 19 Geographies will be defined in the IR 20 Further details about the scope of the Social Network Analysis will be provided in the IR, after accurate study by an SNA expert. 21 Taken out from the draft TOR: Cost benefit analysis (CBA) is one of the methods commonly used to monetize selected outcomes and evaluate comparative advantages in monetary terms. Scoping exercise indicates that this evaluation will likely have to set the path for a simultaneous or future deeper CBA study. 22 Multi-criteria analysis | BetterEvaluation. 23 https://www.betterevaluation.org/methods-approaches/methods/cost-utility-analysis. 24 See CGIAR Evaluation Framework (2022). 11 Terms of Reference: Genebank Platform Evaluation learning scope and focus of the evaluation, by assessing sustainability-related elements; (ii) contribute to the evaluation matrix, clarifying the analytical frameworks to be used by the evaluation; and (iii) ground truth recommendations towards evaluability of the Initiative, to facilitate the integration of lessons learned from the evaluation of the Genebank Platform, contributing to the evaluability of the initiative by the independent external evaluation by IAES grouped within GI Science Group in 2024. Evidence from multiple data sources will be triangulated to ensure transparency and independence of judgment and to minimize bias. To increase credibility, particular value will be placed on the triangulation of the data and solid argumentation of the conclusions and recommendations. 3.1.4 Reporting Phase In the reporting phase, the evaluation team will develop an evaluation report under the overall responsibility of the team leaders. The overall evaluation report will be preceded by and based on the development of three module component reports, reflecting the Platform set-up. The module component reports will be shared with the Platform former director and module leads for optional review. Thereafter, the evaluation team will present preliminary cumulative findings by evaluation criteria, to debrief the IAES and the Platform management and seek validation (via a validation workshop), factual corrections, and feedback.25 The report will follow CGIAR evaluation reporting guidelines and quality assurance processes. The first reviewer of the draft report will be IAES in line with CGIAR’s guidelines on the final evaluation report.26 The evaluation team will be obliged to provide a revised version of the draft report if the quality is not acceptable. If the quality of the draft report is satisfactory (in form and substance), the evaluation manager will circulate it to (i) the Genebank Platform team for comments and factual corrections; and (ii) external peer reviewers and the evaluation reference group members for review and comments. With the feedback from the relevant stakeholders, the evaluation team will finalize the draft report considering comments according to the team’s judgment. A discussion version of the report will be circulated by IAES to SIMEC for acceptance via its secretariat. 3.1.5 Dissemination, Use and Knowledge Management The core dissemination and knowledge management around the evaluation would be expected from November 2023 onwards. The evaluation report, the executive summary, the evaluation brief, and other knowledge products along with the management response, will be published on the IAES website. In line with the dissemination and knowledge management strategy to be developed at the inception phase, tailored presentations will be made to targeted stakeholders, and learning events organized with internal and external stakeholders. 4 Evaluation Management, Roles and Responsibilities The CGIAR System Council as evaluation commissioner via IAES takes accountability for the evaluation. In line with the CGIAR Evaluation Framework and Policy (2022), CGIAR management share leadership and mutual responsibility for the conduct and use of evaluation. 25 See also CGIAR Guidelines on Evaluation Final Report. 26 Old guidelines on the final evaluation report are currently under revision. IAES to provide final version to the evaluation team when published. 12 Terms of Reference: Genebank Platform Evaluation 4.1 CGIAR Management Engagement and Response Consistent with the principles and standards in the Evaluation Framework, utility and use guide the engagement with the evaluand, CGIAR management and other key intended users throughout the evaluation design and implementation. To stimulate the uptake of the evaluation results and learning, early management engagement began in the scoping and design phase, which facilitated the co- development of the priority evaluation questions (Annex 1). In line with the CGIAR Evaluation policy, management responses (MR) are mandatory for all System Council-commissioned evaluations in CGIAR.27 In line with the guidelines under development, for MR, IAES will liaise with the ex-Genebank Platform and current Genebank Initiative management as well as the Portfolio Performance Unit (PPU) to coordinate the MR preparation within a stipulated timeframe.28 The draft evaluation report will be circulated to SIMEC. Once formally presented to the CGIAR System Council, the management response and evaluation report are considered final and will be published on the IAES website. PPU oversees the procedure for tracking, monitoring and outyear reporting against the implementation of evaluation recommendations when the evaluand is CGIAR. Towards mutual responsibility, throughout the evaluation, the Genebank Platform’s management, MEL and other core focal points (actors) would respond to the evaluation team’s requests: documentation and data, access to partners and staff for engagement with the evaluators, and information on partners and stakeholders. These actors will also be responsible for giving factual feedback on the draft evaluation report, module component reports, and case studies, as required. 4.2 IAES Management and Responsibilites IAES is responsible for planning, conducting the initial design, managing evaluator selection and contracts, and initiating and managing the workflows of the evaluation. The latter is done in a way that ensures the quality and independence of the evaluation process and evaluation reports, as well as the timely delivery of high-quality key outputs. IAES is also responsible for ensuring the compliance of processes and products with the CGIAR Evaluation Framework and Policy (2022). Further information on IAES’s mandate is available in its TOR. The IAES/Evaluation Function will assign an evaluation manager from among its ranks who will be specifically responsible for (i) selecting, contracting, and convening the evaluation team; (ii) contractual arrangements; (iii) monitoring and supervision of the evaluation team against agreed terms of reference and contracts; (iv) facilitating access to the evaluand so the team may proceed to data collection; (v) coordinating quality assurance and validation; (vii) guidance and support on documentation (e.g., required templates, editorial services and graphics); and (viii) developing a knowledge management approach and products. The evaluation team and evaluation manager will ensure they undertake adequate consultation with evaluation stakeholders throughout the process. IAES will facilitate a validation workshop on preliminary findings with core stakeholders. IAES will follow its layered quality assurance system, which involves: (i) an internal peer review within the evaluation team; (ii) a second-level review by IAES; and (iii) an external peer review mechanism supported by peer reviewer(s) and the evaluation reference group. 27 The co-development of the Management Engagement and Response guidelines is ongoing at the time of this evaluation. 28 See also CGIAR Guidelines on Evaluation Engagement and Management Response. 13 Terms of Reference: Genebank Platform Evaluation 4.3 Evaluation Team Under the oversight, management, and guidance of IAES, the evaluation will be conducted by an evaluation team of evaluation and subject matter experts, including in plant genetic resources ex situ conservation, use and policy. The five team members will be drawn from the IAES’s standing SME and evaluator roster, each carefully vetted for any conflicts of interest: • Two evaluation team co-leaders: one with a strong background in evaluation and one professional evaluator with formal training in breeding and extensive knowledge of the CGIAR breeding portfolio, including the Excellence in Breeding Platform. The responsibilities of the two co-leads will be strategically and operationally split by module and evaluation phase to best leverage their expertise and availability. • Up to three SMEs with strong expertise in: (a) conservation and use of genetic resources; and (b) knowledge of policies and international treaties on genetic resources. • One (1) mid-level evaluation analyst (consultant to IAES) for data collection, analysis, and knowledge management, who supported the Big Data in Agriculture Platform evaluation. • One (1) evaluation analyst expert in SNA. The evaluation team co-leads have final responsibility for the evaluation report deliverable to IAES and all findings and recommendations, subject to adherence to CGIAR Evaluation Framework and Policy. The primary responsibilities of the team leaders will be: • Technical input into evaluation TOR • Elaborating and setting out the methodology and approach in the IR • Guiding and managing the evaluation team during the evaluation phases • Overseeing the preparation of, and quality-assuring, data collection outputs by subject-matter expert members of the team • Consolidating SME team members’ inputs to the evaluation products (IR, case studies, module component studies as needed, and the evaluation report) • Convening the team towards a jointly authored and agreed set of findings, conclusions and recommendations • Where necessary, representing the evaluation team in meetings with stakeholders • Delivering the IR, draft, and final evaluation reports. Ultimately, the team leaders are responsible for ensuring the quality, consistency, and soundness of all evaluation deliverables to IAES. • Should incidents arise at any stage of the evaluation, the team leaders must immediately report issues to the evaluation manager. If this is not done, the existence of such problems may in no case be used to justify the failure to obtain the results stipulated in the TOR. The team co-leads will have a minimum of 20 years of experience with background knowledge of evaluation and genebanks’ work, breeding, and relevant policy. The team leaders must have experience in leading teams, excellent analytical, synthesis and communication skills (written and verbal), and demonstrated skills in mixed qualitative and quantitative data collection and analysis techniques. - The first co-lead will be responsible for supervising and coordinating SMEs working on Conservation and Use modules, contributing their expertise. - The second co-lead will be responsible for the design of the initial evaluation matrix as per IR, the supervision and coordination of the SME working on Policy module, for the comparative advantage analysis and the analysis of efficiency of management, governance and coordination including Crop Trust/CGIAR relationships. Lead by evaluation team co-leads, all the evaluation team members, including SMEs, are responsible for contributing to the deliverables in the evaluation timeline, to be organized by the team co-leaders. These 14 Terms of Reference: Genebank Platform Evaluation include but are not limited to: • An IR in a template provided by IAES • Quality and timely inputs into module reports and two case studies; to be elaborated upon in the IR • A brief presentation of preliminary findings for validation by the Platform management and IAES in a template provided by IAES • Draft evaluation report, N.B IAES will provide a template for the draft and final reports • A final evaluation report following the report template with a maximum of 25 pages (excluding executive summary), and written in plain English in line with IAES’s style guide • A 2-3 page executive summary, and a set of annexes with additional information to justify and supplement the main body of the report • Blog and other knowledge management/dissemination material • PowerPoint presentations covering the main points of the evaluation, including purpose, methods, findings, conclusions, recommendations, and additional notes relevant to the evaluation. IAES will provide the relevant templates. Another interim product: brief updates to the evaluation manager every two weeks. 5 Evaluation Deliverables, Knowledge Management and Dissemination The IR: will build on and expand these evaluation TOR, to outline the evaluation team’s proposed approach and detail methods to the main phase of the evaluation. The draft evaluation report: the main output of this evaluation will be in line with CGIAR’s guidelines on the evaluation final report and aligned with IAES’s style guide. It will describe findings and conclusions, based on evidence collected in the evaluation framework defined in the IR, and make recommendations logically following the conclusions. The recommendations will be evidence-based, relevant, focused, clearly formulated, and actionable. They will be prioritized and addressed to the different stakeholders responsible for their implementation. The main findings and recommendations will be summarized in an executive summary. The main report should be concise (no longer than 25 pages excluding the Executive Summary and Annexes) and written in plain English. The two deep dives and three module component study reports will be annexed to the report. The evaluation team will be expected to produce a three-page brief of key findings and lessons, following a template provided by IAES.29 Review and finalization of the evaluation report: the evaluation team will submit a draft report by email to the evaluation manager in electronic editable form (MS Word) aligned with IAES’s style guide. The evaluation report will follow a standardized structure and template to be provided by IAES. Upon the acceptance of a draft of adequate quality, IAES will share this first draft report with a team of (i) external peer reviewers with relevant expertise called up from IAES’s vetted roster; and (ii) some members of IAES’s evaluation reference group. The first draft report will also be shared with the Platform team for their review and comments, including to check for any factual errors and to highlight the significance of any such errors in the conclusions. The evaluation team will integrate the collective feedback received into a discussion version of the report which will be professionally copy-edited. Subsequently, the discussion version will be presented to SIMEC for acceptance. With the feedback of SIMEC integrated, the discussion 29 See CGIAR Guidelines on Final Evaluation Report 15 Terms of Reference: Genebank Platform Evaluation version of the report will be presented to System Council for their endorsement. The final evaluation report will be published on the IAES’s website. Presentations: The team leader and evaluation team, where necessary, will present and share the evaluation-related results to targeted audiences via various communication channels upon request by the IAES, e.g., conferences and social media. Except presentations, all deliverables that will be published on the IAES website are subject to proofreading revision by an editor engaged by IAES. 6 Contract and Payment CGIAR’s IAES is hosted by CGIAR System Organization through an arrangement with the Alliance of Bioversity International and CIAT at its offices in Rome, Italy. Contracting will be carried out by its hosting entities and under their name on behalf of the IAES/Evaluation Function. The members of the evaluation team are expected to abide by the Conflict-of-Interest policy of the IAES and must maintain independence in fact and appearance from the Genebank Platform throughout the assignment. Each evaluation team member must complete and return declarations interest and their understanding and compliance with the policies of the IAES and its host institutions. All contracting fees and conditions will be administered in line with the approved policy for consultants. Confidentiality and non-disclosure provisions are covered in these contracts. All collected data must be anonymized and kept within the IAES SharePoint repository. Informants should be duly notified to adhere to ethical evaluation principles. 16 Terms of Reference: Genebank Platform Evaluation Annex 1: Evaluation Questions and Sub- questions by Evaluation Criteria To be elaborated in the evaluation design matrix and, subsequently, the IR. Quality of Science (QoS) related questions integrated and labeled accordingly, consistent with the CGIAR Guidelines issued in 202230. CGIAR Key evaluation evaluation Sub-questions 31 questions criteria 1) How relevant was the a) How applicable and comprehensive was the mandate of the mandate of the Genebank Platform, vis-à-vis Genebanks CRP32? Genebank Platform b) How appropriate and relevant was the Platform to national and ways to achieve genebanks and genetic diversity conservation objectives, i.e., in both it? policy and intervention priorities? c) How realistic were the performance targets of the Genebank Platform? 2) To what extent did a) To which extent did the Genebank Platform achieve the modules’ the Genebank objectives? Platform achieve I. To what extent has the Platform strengthened the linkages progress to intended between conservation and use in each of the CGIAR Centers? outcomes? II. To what extent have the genebanks’ operations improved since the launch of the Genebank Platform? III. How effective were activities of the Platform in enhancing the technical performance of CGIAR genebanks? IV. To what extent did the Platform contribute to harmonization of quality standards of genebanks? b) How effectively and appropriately was the Genebank Platform managed and governed? c) How well did the Genebank Platform facilitate streamlining the quality of performance reporting against its objectives among the involved parties? d) To which extent did the Platform achieve progress on capacity building outcomes? I. To what extent did the Platform strengthen capacity in germplasm health, management and conservation? II. To what extent did the Platform strengthen capacity of CGIAR and national partners to implement and influence international policies and laws? e) QoS: Did capacity strengthening of the research team and partners address needs vis-a-vis the planned work, including non-scientific aspects? f) Gender: To what extent men and women equally benefited from sharing of accessions? 30 Applying the CGIAR Quality of Research for Development Framework to Process and Performance Evaluations | IAES | CGIAR Independent Advisory and Evaluation Services 31 The sub-questions remain broad. The inception phase will allow to narrow down and subsequent Inception Report will provide more specificity on the questions. For example, for EQ 1.1 and 1.2, it would dig deeper, which objectives? In what way were needs met? Whose needs were met, and whose were not? What needs to change, and how? 32 Assessment of degree of implementation of MR from the Genebanks CRP evaluation. 17 Effectiveness Releva nce Terms of Reference: Genebank Platform Evaluation CGIAR Key evaluation evaluation Sub-questions 31 questions criteria 3) How did allocation of a) How efficient and transparent were the Genebank Platform resources (funds, leadership and staffing arrangements? people, time, b) How does coverage of the essential capital requirements for the expertise, etc.) genebanks compare under the Genebank Platform arrangement, support the compared to CRP or Initiative? Or pre-CRP? achievement of the c) What are cost effectiveness and efficiency considerations between Genebank Platform’s externally hosted model of the Genebanks CRP & Platform, vis-à-vis outputs and CGIAR-hosted initiative? outcomes? 4) What strategies, a) What strategies and internal mechanisms and factors contributed internal or external to, or inhibited, timely and cost-effective achievement of outputs mechanisms and and outcomes? factors contributed b) What strategies and external mechanisms, and factors contributed to, or inhibited, timely to, or inhibited, timely and cost-effective achievement of outputs and cost-effective and outcomes, intended and unintended? achievement of c) QoS: Were risk assessment and mitigation strategies put in place outputs and and exercised in light of transition? outcomes, intended and unintended? 5) How did the research, a) What has been the added value from the institutional arrangements evidence, capacity of the Genebank Platform to CGIAR and to Crop Trust respectively, in agenda of the financial and non-tangible terms? Platform b) What was the difference in how the Genebank Platform enhances complement and/or genebanks’ operating standards across CGIAR, compared to CRP strengthen related arrangement? i.e., using Genebank CRP evaluation as a baseline. genebank-focused c) To what extent did the Platform streamline intersections with gender work in CGIAR, issues and whether they enhance any of the Genebank Platform towards the outcomes? Genebank Initiative? 6) How well were a) How effectively was comparative advantage of CGIAR genebanks Genebank Platform exercised and delivered on? operations b) How has the Genebank Platform engaged in relevant policy harmonized, aligned, discourses among key external organizations? and coordinated with c) To which extent did the Platform contribute to a harmonization of non-CGIAR reporting needs and formatting to serve CGIAR and the wider genetic genebanks? resources community better? 7) What learning a) How has the Genebank Platform improved the security of crop mechanisms have collections held in trust by CGIAR genebanks, with specific analysis to been built into the physical security in fragile and conflict states? Genebank Platform b) Is the financial sustainability of CGIAR genebanks better assured design and today than before the Genebank Platform was launched? implementation to c) How did the situation change when comparing the situation during facilitate the operations before the Platform and towards the end of the Platform? potential 18 Sustainability and Cohere nce Efficie ncy Learning for impact Terms of Reference: Genebank Platform Evaluation CGIAR Key evaluation evaluation Sub-questions 31 questions criteria sustainability of d) To what extent has the Genebank Platform enhanced the Platform results? sustainability of the genebanks in terms of conservation security and non-financial risks? e) What are the short- and long-term lessons learned around optimal mechanisms to fund the genebanks? Did raising Genebanks’ standards facilitate their eligibility for long-term fundings? f) To what extent were succession plans for internationally and nationally recruited staff in place? g) To what extent did the Platform address issues related to education and outreach? h) QoS: What procedures and mechanisms were in place for internal and external coherence peer-reviews, to enhance learning? 8) In what ways did the a) QoS: What is the contribution of the Platform outputs to breeding Platform contribute science-based innovations, targeted capacity development, and to achieving global advice on policy? i.e. What were the main contributions of the Platform development to ITPGRFA and international exchange of PGRFA? objectives, notably b) Did all the intended target groups-including the CGIAR and non-CGIAR the SDGs, along its genebanks and partners-benefit equally from the intervention? impact pathway? c) How transformative were interventions under the Genebank platform? Does it create enduring changes in norms and systems (e.g., ITPGRFA, PGRFA), whether intended or not? d) To what extent did the Platform enhanced changes in non-CGIAR genebanks? 19 Terms of Reference: Genebank Platform Evaluation Annex 2: CGIAR Genebank Platform-Overview A2.1 CGIAR Genebanks CGIAR genebanks safeguard some of the largest and most widely used collections of crop diversity in the world, critical to attaining global development goals to end hunger and improve food and nutrition security. Genebanks, as a key driver of international exchange of PGRFA, are fundamental to delivering the CGIAR Strategy and Results Framework and three System Level Outcomes of reduced poverty, improved food and nutrition security, and improved natural resources systems and ecosystem services. Through their work, CGIAR contributes to the achievement of Target 2.5 and 2a of the UN Sustainable Development Goals. The 11 CGIAR Genebanks conserve 736,210 accessions of cereals, gran legumes, forages, tree species, root and tuber crops, and bananas, distributed as per Table 3 below. Table 3. Accessions of CGIAR genebanks in 202133 Genebank Crops Accessions (2021) AfricaRice Rice 19,699 Bioversity International Banana, Plantain 1,682 CIAT Beans, Cassava, Tropical Forages 66,599 CIMMYT Maize 28,694 Wheat 113,418 CIP Potato, Sweat Potato, Andean Roots and Tubers 17,314 ICARDA Barley, Wheat, Legumes, Temperate Forages 151,788 ICRAF Fruit and Multipurpose trees 14,990 ICRISAT Sorghum, Millet, Chickpea, Pigeon Pea and Groundnut 128,645 IITA Maize, Banana, Cassava, Yam, Legumes 34,864 ILRI Tropical Forages 18,662 IRRI Rice 132,313 From 2017-21, CGIAR genebanks distributed a total of 473,621 germplasm samples (3.8 million accessions). Consistently over the five-year period, germplasm distribution to requesters outside CGIAR exceeded distribution within CGIAR. Of all requested samples, 212,930 (39%) were provided to CRPs, and 260,691 (61%) were distributed to non-CGIAR recipients in 139 countries (Figures 4 and 5).34 33 Information released from the Genebank Platform website Genebanks | CGIAR Genebank Platform. 34 Source CGIAR Genebank Platform Summary Report Genebank-Platform-Summary.pdf (cgiar.org). 20 Terms of Reference: Genebank Platform Evaluation Figure 4. Internal and external CGIAR genebank distributions, 2017-21 Figure 5. External genebank distributions by user group, 2017-21 A2.2 Purpose and Objectives of CGIAR Genebank Platform The term “platform” symbolizes the solid, systemwide strength of genebanks in providing the cohesion and grounding, together with the other Platforms, by which the Agri-Food System CRPs (AFS-CRPs) modernized their breeding programs and delivered genetic gains and increased productivity. The Genebank Platform aimed to strengthen governance and structure and improve integration and cohesion across Centers and crops. It followed the footsteps of previous collaborations among CGIAR genebanks through the Systemwide Program on Genetic Resources and the Genebank CRP, that had already reinforced a “global system” for ex situ conservation and sustainable use of PGRFA at a global level. Given that CGIAR genebanks have different levels of experience and resources, the introduction of shared approaches could bring greater efficiency and more powerful tools to access and collections. The main objectives of the Platform were to increase both conservation and use of PGRFA, addressing three major challenges: 1. CGIAR must do more with less: new technologies, knowledge and processes offer the possibility of further raising the standards to become more efficient and more effective 2. CGIAR must, in aiming for increased genetic gain, respond better to breeders’ needs for genetic diversity and specific traits 3. CGIAR must comply with its legal obligations and engage in shaping international genetic resources agreements. 21 Terms of Reference: Genebank Platform Evaluation A2.3 The Genebank Platform’s Impact Pathway and Theory of Change The pathway towards impact of the genebanks and GHUs may be traced by following the route of the germplasm and associated data. The genebanks contribute in multiple ways to IDOs35 for: (i) increased resilience of the poor to climate change and other shocks; (ii) improved diets for poor and vulnerable people; (iii) natural capital enhanced and protected, especially from climate change; and (iv) enhanced benefits from ecosystem goods and services. The activities of the Platform, however, are targeted specifically to bring about increased conservation and use of genetic resources and increased productivity. The major role of the Platform is to maintain the collections and facilitate GHUs to sustain these distributions. The main focus is on increasing conservation, through the improvement of efficiency and the increased level of diversity within a collection. The Platform aims also to boost the use of genetic material, through the improvement of search tools and available data. The Table below reports the specific outcomes of the Genebank Platform and through which Platform module36 they are addressed: Table 4. Mapping of stated Platform outcomes per modules (source IAES) Module Outcomes The Conservation Module helps genebanks work strategically to exploit new 1, 4 opportunities, conserve more diversity and respond to more demands while controlling costs. The Use module helps genebank align their operations towards more targeted use and 1, 2 exploitation of the collection. The main objective is to empower effective use of plant genetic resources, by offering more effective access to crop diversity and information. The Policy module supports the active engagement of CGIAR in shaping international 3 genetic resources agreements. All outcomes are essential to the successful use of germplasm by the Agri-Food System (AFS) CRPs and contribute to building a stronger global system for conservation and use. The Genebank Platform facilitated and augmented the interactions across multiple centers and crops. It provided germplasm, data, knowledge, policy advice and phytosanitary services to the system, and fed into the generalized theory of change of the AFS-CRP and Platforms.37 35 Under ‘retired’ CGIAR SRF (2016-2023). 36 Platform modules will be further elaborated in the next section. 37 Genebank Platform Full Proposal Genebank Platform Full Proposal 2017-2022 (cgiar.org). 22 Terms of Reference: Genebank Platform Evaluation Figure 6. Impact pathway and theory of change A2.4 Platform Modules, Objectives, Targets and Activities The Genebank Platform had three modules, each responding to one or more Platform outcomes. The Conservation module supported core genebank operations. This module helped genebanks work strategically to exploit new opportunities, conserve more diversity, and respond to more demands while also controlling costs. The main objective of the conservation module was to support and improve essential genebank operations, ensure that germplasm was secure and available, and improve genebank operations and management. The Conservation module addresses outcomes one and four, through the activities depicted in Table 5. Table 5. Activities, outputs and targets of the Conservation module Strategic Indicators and Activities Outputs Assumptions relevance targets Objective 1.1: To sustain core genebank operations and ensure germplasm is secure and available 1.1.1 Managing the CGIAR obligations Representative, 90% of collections Increasing institutional and genebank: seed to the ITPGRFA to documented, available conserved, other costs can be managed processing, testing, conserve and and secure documented and so that genebanks receive storage, data make available germplasm in 35 crop available. sufficient operating funds germplasm from collections. from fixed budgets. 23 Terms of Reference: Genebank Platform Evaluation Strategic Indicators and Activities Outputs Assumptions relevance targets management, in trust collections Disease-free 80% of relevant The GHU costs are fully regeneration, are fulfilled. germplasm distributed requests serviced covered in service charges. safety duplication upon request. by the global system. Investment is made in developing protocols to test Customer for new diseases. satisfaction of 80% or higher. Objective 1.2: To improve Genebank operations and management 1.2.1 Quality High standard of SOPs validated and 50 SOPs and other Appropriate level of management and operation and improved for activities minimum QMS investment in CGIAR upgrading quality including germplasm elements (e.g., risk infrastructure. management health in 11 genebanks. management, staff clearly illustrated. succession, Quality management results barcoding) in place in improved management. and validated. Equipment is up- to- date and calibrated. 1.2.2 Researching Sustainability of Step change increase Storage periods Existing technologies only conservation operations in length of time redefined in at require optimization to obtain methods improved as germplasm may be least ten crops. improvements. materials are able stored. to be kept for longer in storage. 1.2.3 Analyzing Long-term Transparent and Annual increase in Institutional support is costs and building sustainability of comparable genebank cost of core provided to build fairer and efficiencies genebanks is operations and costs operation less than more consistent application facilitated. by crop group. inflation. of CGIAR Financial Guidelines across the System. 1.2.4 Improving Increased Representation of Representation of Users provide information on representation: conservation genetic, taxonomic, crop gene pools in needs, and AFS partners analyzing diversity, globally. geographical and ex situ provide information on traits. identifying gaps, environmental conservation eliminating diversity and traits quantified. Collecting of threatened and duplicates and improved. unique germplasm is collecting Gaps in at least five politically and physically crop gene pools possible. addressed. 1.2.5 Strengthening Global system for Improved capacity in Increase in NARS have the capacity to capacity in conservation and NARS and closer exchange of comply with international germplasm health use of PGRFA is partnership with germplasm policies on germplasm management and strengthened. genebanks. between NARS and exchange. conservation CGIAR. The Use module addressed Platform outcomes one and two; to help genebanks align their operations toward more targeted use, and exploitation of the collections. The main objective was to empower the effective use of plant genetic resources by offering more effective access to crop diversity and information. 24 Terms of Reference: Genebank Platform Evaluation Table 6. Activities, outputs and targets of the Use module Strategic Activities Outputs Indicators and targets Assumptions relevance Objective 2.1: To facilitate more effective access and use through targeted delivery of germplasm that better meets the needs of users 2.1.1 Promote Users will have Genesys (and where Genesys is linked to, Genetic Gains and Big information access to a new Applicable, in-house and exchanges Data Platforms jointly integration class of information databases) contains information freely succeed in developing between with high value for or has access to all with, API and Genomics the infrastructure, tools genebanks and targeting the selection available relevant and Open-source and mechanisms for breeding of genebank information from Breeding Informatics interoperability. programs accessions to more varied user Initiative as well as precisely meet communities on each breeders databases in Collaboration with their needs. accession. at least 50% of the Genetic Gains and Big Centers. Data is effective. Unique genetic AFS-CRPs use tools identifiers available from Genetic Gains to for 50% of accessions make their data in the CGIAR available. collections. 2.1.2 Empower Users are able to Genesys (and where User-friendly Activity 2.1.1 is genebank target and select applicable, in-house custom searches successful. clients with germplasm much genebank databases) and tools. intuitive, use- more effectively, available through a driven query reducing the need user-friendly online tools for large-scale interface which screening. facilitates custom designed searches of germplasm based on integration of environmental, phenotypic and genotypic information. 2.1.3 Enrich Users encouraged Availability and use of Individual genebanks Sufficient information data on to explore larger parts data-rich subsets of have a minimum of on accessions is collections of the collections. accessions. three subsets of available through through use of germplasm available linkages to AFS-user focal subsets and distributed communities to annually. designate meaningful subsets of the collection. The Policy module addressed Platform outcome three. It supported the active engagement of CGIAR in shaping international genetic resource agreements. This module ensures a supportive policy environment for CGIAR genebanks, breeding programs and partners. 25 Terms of Reference: Genebank Platform Evaluation Table 7. Activities, outputs and targets of the Policy module Strategic Activities Outputs Indicators and targets Assumptions relevance Objective 3.1 To ensure Centers comply with international policies and laws, increase their influence in policy- making processes and strengthen capacity of national programs 3.1.1 Monitoring Failure to comply with Centers’ fully White paper for Centers are willing to and ensuring legal obligations compliant with plant internal CG audience. comply with obligations. compliance with exposes Centers and genetic resources Guidelines, tools international the CGIAR to legal policy obligations. published on Platform policies and laws liability and loss of website reputation, and generates Use of the helpdesk disaffection with the global system. 3.1.2 Engaging in CGIAR has CGIAR plays a White paper for Delegates at international the processes of opportunities to recognized role in internal CGIAR policy meetings are willing developing shape the policies influencing audience. to be engaged on CGIAR international that govern its own international policy perspectives and interests. policies and laws and partners’ genetic development. Submissions to resources work. negotiating forums: CGIAR and CGIAR Evidence-based bi- annual report to Centers contribute timely policy ITPGRFA, discussion inputs to the time-limited recommendations papers, policy briefs, opportunities to influence to international written responses to international policy bodies. Secretariats’ surveys, outcomes. forum statements, and side-events. Representation of the CGIAR in six to eight international policy meetings. 3.1.3 Strengthening Positive national National partners Online repository of Partners have the capacity of policy development have tools and decision making tools commitment from their CGIAR and that supports CGIAR mechanisms to and guides, reports for management and national partners research and support their national level governments to to implement and development implementation of implementation. implement and support influence partnerships. plant genetic appropriate national international resources policies. PGR policy sessions policy. policies and laws led by Platform scientists in CGIAR, national and regional workshops. National program representatives invited to training events for CGIAR scientists, intellectual property managers, research leaders. 26 Terms of Reference: Genebank Platform Evaluation Strategic Activities Outputs Indicators and targets Assumptions relevance 3.1.4 Convening CGIAR will be well Sound inputs guide Minutes of the Multi- All relevant stakeholder the Multi- positioned to engage CGIAR compliance stakeholder PGR Policy groups are willing to stakeholder PGR in a highly politicized and engagement in group meetings. engage. Policy group policy field. international PGR policymaking. Positive references to CGIAR PGR policy engagement in publications of other stakeholders. A2.5 Platform Partnership Ecosystem and Users Primary users of the Genebank Platform were researchers and breeders from within CGIAR, calling upon the Platform for germplasm, data, GHU services and PGRFA policy advice. Other primary users of the Platform outside CGIAR were universities, NARS, advanced research institutes, genebanks, NGOs, farmer groups and the private sector. Among users there were also national genebanks and other research partners. Organizations like EMBRAPA, USDA, CGN, AVRDC, IPK, FAO, NBPGR, RDA and MSB played a particularly significant role across the system because of the size or complementarity of their collections.38 Main partners of the Platform included the following: Table 8. List of Platform partnerships39 Partner Name Partnership description Flagship project Intergovernmental Panel on Collaboration in safe exchange of germplasm Conservation module: Support and Climate Change improve essential genebank operations Commission on Genetic Collaboration in the delivery of the Global Plan Conservation module: Support and Resources for Food and of Action improve essential genebank Agriculture operations ITPGRFA Global Information System on PGRFA Use module: Empower effective use of plant genetic resources Global Information System on plant genetic resources for food and agriculture International policy development and Policy module: Engage in genetic compliance for plant genetic resources for food resources policy development and and agriculture, capacity building for policy compliance implementation at national level International PGRFA policy development and compliance, capacity building for policy implementation at national level Convention on Biological International PGRFA policy development and Policy module: Engage in genetic Diversity compliance, capacity building for policy resources policy development and implementation at national level compliance International policy development and compliance for plant genetic resources for food 38 Information retrieved from the Genebank Platform Full Proposal. Deeper understanding of the Platform during the inception phase, will help define whether there was a congruence between intended users listed in the Proposal and actual users. 39 CGIAR Results Dashboad Results Dashboard - CGIAR. 27 Terms of Reference: Genebank Platform Evaluation Partner Name Partnership description Flagship project and agriculture, capacity building for policy implementation at national level USDA Training and support for accessions and data Conservation module: Support and management using GRIN-Global improve essential genebank operations The Crop Trust Ultimate safety duplication of CGIAR Conservation module: Support and germplasm improve essential genebank operations Svalbard Global Seed Vault Ultimate safety duplication of CGIAR Conservation module: Support and germplasm improve essential genebank operations A2.6 Genebank Platform Funding and Budget Between 2017-21, the Genebank Platform had USD 151,44 million in total funding. Over that period, Window 1&2 represented 67% of funding, Window 3 represented 1% and Bilateral 32%. The Genebank Platform top funder was the CGIAR Trust Fund for Windows 1&2, followed by the Global Crop Diversity Trust and the Bill and Melinda Gates Foundation40. Tables 7-11 report the Platform’s planned vs actual expenditure for each year (2017-21), extrapolated from Genebank Platform’s annual report. Table 9. 2017 Genebank Platform financial summary 40 CGIAR Genebank Platform Summary Report 2017-2021 Genebank-Platform-Summary.pdf (cgiar.org). 28 Terms of Reference: Genebank Platform Evaluation Table 10. 2018 Genebank Platform financial summary Table 11. 2019 Genebank Platform financial summary Table 12. 2020 Genebank Platform financial summary 29 Terms of Reference: Genebank Platform Evaluation Table 13. 2021 Genebank Platform financial summary 30 Terms of Reference: Genebank Platform Evaluation Annex 3: List of Stakeholders Consulted Toward TOR Development Name Affiliation Charlotte Lusty Former CGIAR Genebank Platform Coordinator Sarada Krishnan Director of Program, Crop Trust Nelissa Jamora Agricultural Economist, previously M&E Focal Point for the CGIAR Genebank Platform, Crop Trust Faith Wambua-Lüdeling Science Administrator, Crop Trust Enrico Bonaiuti MEL Research Team Leader, ICARDA, CGIAR Bia Carneiro Social Research and Media Specialist, CGIAR Focus Climate Security project 31 Terms of Reference: Genebank Platform Evaluation Annex 4: Preliminary List of Readings CGIAR (2021). Proposal for the Conservation and Use of Genetic Resources (Genebanks) Initiative. INIT03-Conservation- and-Use-of-Genetic-Resources-Genebanks.pdf (storage.googleapis.com) Jackson, M. T., Borja Tome, M. J., & Ford-Lloyd, B. (2017). Evaluation of CGIAR research support program for Managing and Sustaining Crop Collections: Genebanks CRP. Evaluation of CGIAR research support program for Managing and Sustaining Crop Collections: Genebanks CRP Platform, C. G. (2016). Genebanks Platform: Full Proposal 2017-2022 Cover Letter. Genebanks Platform: Full Proposal 2017- 2022 (cgiar.org) Platform, C. G. (2017). Governance and Management of the Genebank Platform Governance and Management | The Platform | CGIAR Genebank Platform (genebanks.org) Platform, C. G. (2018). Annual report 2017: CGIAR Genebank Platform. 2017-Genebank-Platform-Annual-Report.pdf (genebanks.org)https://www.genebanks.org/wp-content/uploads/2018/11/2017-Genebank-Platform- Annual-Report.pdf Platform, C. G. (2019). Annual report 2018: CGIAR Genebank Platform. 2018 CGIAR Genebank Platform Annual Report (genebanks.org) Platform, C. G. (2020). Annual report 2019: CGIAR Genebank Platform. 2019-Genebank-Platform-Annual-Report.pdf (genebanks.org) Platform, C. G. (2021). Annual report 2020: CGIAR Genebank Platform. 2020-Genebank-Platform-Annual-Report.pdf (genebanks.org) Platform, C. G. (2022). Annual report 2021: CGIAR Genebank Platform. 2021-Genebank-Platform-Annual- Report_21June2022.pdf (genebanks.org) Platform, C. G. (2022). CGIAR Genebank Platform: Summary Report 2017-2021. Genebank-Platform-Summary.pdf (cgiar.org) Platform, C. G. (2020). System level review of genebank costs and operations (GCO). GCO-Report_261020.pdf (genebanks.org) Africa-rice Center (2020). Genebank review. AfricaRice_genebank-Report_of_External_Review.pdf Bioversity (2020). Genebank review. Bioversity International | Genebanks | CGIAR Genebank Platform CIAT (2019). Genebank review. CIAT | Genebanks | CGIAR Genebank Platform CIMMYT (2019). Genebank review. CIAT | Genebanks | CGIAR Genebank Platform CIP (2020). Genebank review. CIAT | Genebanks | CGIAR Genebank Platform ICARDA (2019). Genebank review. CIAT | Genebanks | CGIAR Genebank Platform ICRAF (2020). Genebank review. CIAT | Genebanks | CGIAR Genebank Platform ICRISAT (2020). Genebank review. ICRISAT | Genebanks | CGIAR Genebank Platform IITA (2019). Genebank review. IITA | Genebanks | CGIAR Genebank Platform IRRI (2019). Genebank review. IRRI | Genebanks | CGIAR Genebank Platform Independent Science for Development Council. 2022. Identifying and Using CGIAR’s Comparative Advantage. Rome: CGIAR Independent Advisory and Evaluation Service. Identifying and Using CGIAR’s Comparative Advantage | IAES | CGIAR Independent Advisory and Evaluation Services 32 Terms of Reference: Genebank Platform Evaluation Annex 5: Requested points of Engagement and Tasks for the Evaluand’s MEL Focal Point S/N Evaluation phase MEL focal point key tasks A Scoping/Pre-Planning • Assemble relevant and reliable extant program documentation and data for the evaluation against the requested detailed list of required documentation. This will constitute the evaluation repository. • Provide access to a designated, secure Sharepoint folder for the evaluation document upload or upload to designated Sharepoint folder of IAES. • Review key evaluation questions. B Inception • Participate in the evaluability assessment; namely, provide the supporting documentation and reliable data. Complete the spreadsheet based on the condensed core parameters of the CGIAR guidelines on conducting an evaluability assessment (2022) and provide supporting documentation where necessary.41 • Review the evaluation design matrix and comment on the methods/and data sources (e.g., Annex 2 in an IR from evaluation of Big Data Platform). • Co-facilitate engagement(s)/meetings as needed, with evaluation team members. • Review the evaluation IR, developed based on the TOR, see above example for Big Data. • Review questionnaire for online survey, if applicable. • Contribute to the review of the stakeholder analysis. C Inquiry/data collection • If needed, support/facilitate access to interviewees/key informants to answer and analysis questions from the evaluation team. • Serve as a key informant about the MEL system for an interview and respond to online surveys if applicable. D Reporting/ • Participate in the validation of preliminary findings, conclusions, and dissemination and use recommendations. • Coordinate comments from the Platform team on the draft evaluation report and any sub-studies (e.g., deep dives and module component studies) and ensure they are sent to IAES within the stipulated time. • Contribute to the development of the Management Response, e.g., from Big Data Platform Evaluation. • When the evaluation is finalized and the management response is available, they support the use of findings to ensure that key actions are implemented and learning is woven into the programming. 41 In line with the CGIAR Evaluability Assessment guidelines, this follows if the evaluability assessment was conducted as an integrated part of the inception phase. 33 Independent Advisory and Evaluation Service Alliance of Bioversity International and CIAT Via di San Domenico, 1 00153 Rome, Italy IAES@cgiar.org https://iaes.cgiar.org/