Inception Report Oct 2014 Evaluation of the CGIAR Research Program on MAIZE Guido Gryseels (Team Leader) Sirkka Immonen (IEA) Sophie Zimm (IEA) EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 TABLE OF CONTENTS List of Abbreviations................................................................................. 4 1. Introduction..................................................................................... 5 2. Context ........................................................................................... 6 3. Purpose and scope of the evaluation .............................................. 16 4. Evaluation framework and criteria .................................................. 18 Relevance ......................................................................................... 18 Quality of science ............................................................................. 19 Effectiveness .................................................................................... 19 Impact and likely sustainability ......................................................... 19 Gender ............................................................................................. 20 Capacity building .............................................................................. 20 Partnerships ..................................................................................... 20 Governance and management ........................................................... 20 5. Methodology, data collection and analysis ...................................... 22 Document review .............................................................................. 22 Project activities mapped to MAIZE .................................................... 23 Analysis of sampled projects............................................................. 23 In-depth case studies ....................................................................... 24 Country visits ................................................................................... 25 Semi-structured interviews ............................................................... 26 MAIZE researcher survey ................................................................... 26 Quality of science analysis ................................................................ 26 Impact analysis ................................................................................. 28 Governance and management analysis .............................................. 28 2 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 6. Organization and Timing of the Evaluation ..................................... 29 ANNEX 1: Evaluation Matrix .................................................................... 32 ANNEX 2: Team member profiles ............................................................ 39 ANNEX 3: LIST OF PERSONS CONSULTED IN INCEPTION PERIOD ................ 41 ANNEX 4: SAMPLE PROJECT ..................................................................... 43 TABLES Table 1: Current Flagship structure and IDOs ............................................................................ 9 Table 2: Financial summary for W3&bilateral funding, 2011-2014 ........................................ 10 Table 3: Actual W1&W2 expenses for MAIZE, as reported to the Consortium and Budget approved by the MAIZE Management Committee for completing Phase I of MAIZE (Jul 2011 – Jun 2014). .............................................................................................................................. 11 Table 4: Largest MAIZE donors ................................................................................................ 12 Table 5: Major shared projects ................................................................................................ 14 Table 6: In-depth case study projects ...................................................................................... 24 Table 7: Country visit sites ....................................................................................................... 25 Table 8: Evaluation team responsibilities ................................................................................ 29 Table 9: Evaluation Reference Group ...................................................................................... 30 Table 10: Evaluation Timetable and Tentative Deliverables ................................................... 30 FIGURES Figure 1: Major Sources of Funding in the CGIAR System ......................................................... 7 Figure 2: MAIZE Expenditures (2012, 2013) and Budget predictions (2014-2016) in USD thousands ................................................................................................................................. 11 Figure 3: MAIZE expenditures per Strategic Initiative (2012 and 2013) ............................................... 12 3 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 LIST OF ABBREVIATIONS AARI Anatolia Agricultural Research Institute (Turkey) CRP CGIAR Research Program CIMMYT International Maize and Wheat Improvement Center (Mexico/Global) EPMR External Program and Management Review (CGIAR) IEA Independent Evaluation Arrangement (CGIAR, Rome) IITA International Institute on Tropical Agriculture MAIZE CGIAR Research Program on Maize MC Management Committee (MAIZE) NGO Non-governmental organization (general) PMU Program Management Unit (MAIZE at CIMMYT) SRF Strategy and Results Framework (CGIAR) SLO System-Level Outcome (CGIAR) StAC Stakeholder Advisory Committee (MAIZE) W1 Window 1 funding type (CGIAR) W2 Window 2 funding type (CGIAR) W3 Window 3 funding type (CGIAR) 4 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 1. INTRODUCTION 1.1. Rationale for the Evaluation Research in the CGIAR is guided by the Strategy and Results Framework (SRF), first approved in 2011, which sets forth the System’s common goals in terms of development impact (System-Level Outcomes [SLOs])1, strategic objectives in areas of comparative advantage and results. The CGIAR’s research agenda is implemented by the 15 CGIAR Centres and their partners through multi-partner CGIAR Research Programs (CRPs). It is funded through a pooled funding mechanism in the Fund2 and bilateral funding to Centers. In the CGIAR, the Independent Evaluation Arrangement (IEA) Office is responsible for System-level external evaluations. The main mandate of the IEA is to lead the implementation of the CGIAR Policy for Independent External Evaluations3, through the conduct of strategic evaluations of the CGIAR CRPs and institutional elements of the CGIAR and through the development of a coordinated, harmonized and cost-effective evaluation system in the CGIAR. The IEA’s first four-year Rolling Evaluation Work Plan 2014-17, approved in November 2013 by the Fund Council, foresees the evaluation of up to 10 CRPs over 2013-2015. The CIMMYT-led CRP MAIZE was chosen as one of the first CRPs to be evaluated. This CRP was approved in 2011 among the first CRPs. The major part of the research carries forward or builds on long-term research conducted by CIMMYT and IITA, the two Center partners. This research was last evaluated in CIMMYT’s External Program and Management Review (EPMR) in 2004 and IITA’s EPMR in 2007. This CRP evaluation therefore contains a summative part of research implemented mostly under the Center programs and a formative part of the current CRP research. 1.2. Structure of the Inception Report The evaluation is guided by the Terms of Reference.4 The inception report builds on the TOR elaborating on the approach and methods to address specific questions on the evaluation issues and criteria. The remainder of this Inception Report is structured as follows: Chapter 1 provides the context to this review; Chapter 2 its purpose and scope; Chapter 3 provides the purpose and scope of the evaluation, and chapter 4 discusses the evaluation framework and criteria. Chapter 5 presents the proposed methodology, data collection, and analysis. Chapter 6 gives an overview of organization and timing. Annexes 1-4 provide further details on the Evaluation matrix, Team member profiles, List of persons consulted and Sample projects. 1 Defined as four System-Level Outcomes: reduction of poverty, improvement pf food security, increasing nutrition and health; and more sustainable management of natural resources. 2 The CGIAR Fund is a multi-donor, multi-year funding mechanism that provides funding to (i) CRPs through two “Windows”; Window 1 across CRPs as per Consortium decision and Window 2 to donor-specified CRP; and to (ii) donor-specified Centers through Window 3 3 http://www.cgiarfund.org/sites/cgiarfund.org/files/Documents/PDF/CGIAR_evaluation_policy_jan2012.pdf 4 http://www.iea.cgiar.org/sites/default/files/MAIZE%20ToR%20FINAL%20Jan%202014.pdf 5 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 2. CONTEXT 2.1. External program context Maize is one of the most important commodities for the poor in developing countries. It is produced on nearly 100 million hectares in 125 developing countries, of which two thirds are low or middle income. Together with wheat and rice, maize provides more than 30% of the food calories of more than 4.5 billion people in 94 developing countries. More than 900 million poor consumers and nearly one third of all malnourished children have maize as their preferred staple food. More than 30 million farm families have maize as their dominant food crop. The demand for maize is increasing rapidly and is expected to double by 2050. By 2025 maize is projected to become the crop with the greatest production globally in the developing world. At current levels of productivity growth, demand will fall short of demand and food prices will rise. Major efforts have to be taken to accelerate yield growth of maize, increase incomes from maize based farming systems and to stabilise food prices." Research, development and delivery of maize varieties adapted to the agro-environmental and improved production systems in developing countries are the backbone and foundation for sustainable and affordable access to staple food. The MAIZE CRP goals are targeting this deployment of improved maize varieties. This effort, however, cannot be achieved alone by the MAIZE CRP program. There are key stakeholders and partners that should support and enable MAIZE CRP mandate. Furthermore, the scenario around it is evolving rapidly with the seed industry developing and delivering maize varieties in target areas and non-private institutions (e.g., NGOs, National R&D Programs) playing increasing role in regulatory and upstream research. The MAIZE CRP program can (must) play a significant role in articulating common purpose efforts across institutions to overcome increasing challenges in maize production, especially abiotic (e.g., drought and heat, soil fertility) and biotic stress (e.g., new diseases such as Maize Lethal Necrosis MLN, crop protection). The implication of both local emerging and multinational seed industry in joint efforts with the MAIZE CRP would guarantee mid and long term delivery of maize varieties and bring up economy of scale to developing countries. In addition, the structuring of a solid portfolio of maize research together with international research organizations (e.g., Universities, Foundations) remains fundamental to innovate and provide modern breeding technologies to the developing world. 2.2. CGIAR reform context The CGIAR reform was set in motion in 2008. The CGIAR donors, in a Joint Declaration agreed on the following main principles for the reform: • To harmonize our approach to funding and implementing international agricultural research for development through the CGIAR Fund (the Fund) The Strategy and Results Framework and the consortium established by the Centers (the Consortium), respectively; • To manage for results in accordance with the agreed Strategy and Results Framework (SRF) and the Mega Programs that derive from the SRF; • To ensure effective governance and efficient operations in the provision and use of our resources; and 6 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 • To collaborate and partner with and among funders, implementers, and users of SRF research, as well as other external partners supporting the SRF. The SRF was approved in 2011 at a time when the Center-led CRPs had already been developed and two of them (on rice and climate change) approved. Thus the current CRPs did not derive from the SRF although the SRF is intended to provide the broad rationale and context for the development, implementation and evaluation of the CRPs. An updated SRF is due by the end of 2014. In the approval process the CRPs were both developed and appraised following a set of common criteria that addressed the (i) strategic program coherence; (ii) focus on delivering outcomes and impacts towards the SLOs; (iii) quality of science; (iv), management of partnerships, including both research and development partners; (i) efficiency of program management; and (vi) accountability, sound financial planning and efficiency of governance. Under Consortium Office coordination, Intermediate Development Outcomes have been developed since 2012 both at the CGIAR System level and at the CRP level for linking the CGIAR research to the SLOs and for facilitating priority setting, again both at the System and at the CRP levels. Simultaneously, CRPs have been instructed to restructure the program by Flagship Projects, and cluster of activities within the FPs, each FP designed to contribute to one or more CRP IDOs through an impact pathway and to the SLOs through a Theory of Change. The CRPs were instructed to define the IDOs also in terms of clear target domains (agroecologies and end user groups) and measurable results at the outcomes level.5 The internal CGIAR context is also important for understanding the MAIZE funding. The funding sources available to CRPs are explained in Box 1. Figure 1: Major Sources of Funding in the CGIAR System To maximize coordination and harmonization of funding, donors to CGIAR are strongly encouraged to channel their resources through the CGIAR Fund. Donors to the Fund may designate their contributions to one or more of three funding “windows”: • Contributions to Window 1 (W1) are the least restricted, leaving to the Fund Council how these funds are allocated to CGIAR Research Programs, used to pay system costs or otherwise applied to achieving the CGIAR mission. • Contributions to Window 2 (W2) are designated by Fund donors to specific CGIAR Research Programs. • Contributions to Window 3 (W3) are allocated by Fund donors to specific CGIAR Centers. Participating Centers also mobilize financial resources for specific activities directly from donors as bilateral funding and negotiate agreements with their respective donors for the use of these resources. Source: CGIAR website: http://www.cgiar.org/who-we-are/cgiar-fund/ The CGIAR reform also involves shifting to central annual reporting to the Consortium, which applies to all sources of funding. This reporting has not yet replaced the requirement of reporting to 5 https://library.cgiar.org/bitstream/handle/10947/2895/2014%20- %20SRF%20Management%20Update_Final_2013_12_20.pdf?sequence=1 7 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 bilateral donors, often following donor-specific requirement. Given that bilateral funding has remained a major source of funding, the reporting burden remains currently considerable. The CRPs were initially approved for a three-year funding period in parallel with agreement on the first SRF. As the evaluation of MAIZE is beginning, an updated SRF is being developed by the Consortium Office to include System level priorities for research funding. Inputs into the reform are expected from a CGIAR Mid-Term Review due by the end of 2014. In parallel, MAIZE has completed its 3rd year of operation and a six month extension was approved based on the submission of the Program of Work and Budget 2014.6 MAIZE among all CRPs has also submitted a proposal for extension funding for 2015-16. Finally, a process for the 2nd call of CRPs is being prepared to be launched in 2015 to follow a two-stage approach, with pre-proposals to be submitted in early 2015 and full-proposals in 2016. The CGIAR is moving rapidly in its reform and consequently the CRPs are engaged in making considerable changes in the programs. This has implications for the evaluation, which is taking place in a changing context and yet is expected to inform the direction of MAIZE. 2.3. MAIZE program MAIZE objectives, structure and activities The MAIZE Strategy which was conceived as a collaborative effort for international maize research was outlined in the Proposal Document (June 2011) and aims to: “double productivity and significantly increase the incomes and livelihood opportunities from more productive, resilient and sustainable maize-based farming systems on essentially the same land area—while contending with climate change and increasing costs of fertilizer, water, and labor.” Two target groups were defined: - Stress-prone smallholders with poor market access; and Market- oriented, technology- constrained smallholders in more benign environments. The following impact targets were set for the new strategy of international maize research, assuming engagement of farming communities, international and national researchers, policy makers, the private sector, and many other development partners: 1. Boost maize productivity by at least an additional 20% by 2020 and 50% by 2050 (compared to current trends) in 60 major maize-producing countries of the developing world, thus helping ensure accessible and stable prices for the over 900 million poor maize consumers. 2. Sustainably intensify maize production and ensure stabilization of the total maize area at about 120 million hectares in developing countries, thus avoiding environmental damage. 3. Reduce the frequency of production shortfalls and price volatility in areas and countries where the probability of crop failure in maize-based farming systems is greater than 15%. 4. Diversify maize-based farming systems and enhance their productivity and sustainability, dealing specifically with the systems with the highest poverty concentrations, where over 660 million maize-dependent poor and about 62 million malnourished children live. 5. Ensure that higher rates of maize yield growth are sustained beyond 2020 in the face of climate change impacts, worsening water scarcity, and rising fertilizer prices. 6 See CGIAR 11th Fund Council Meeting Summary, May 7-8, 2014 8 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 6. Increase opportunities for diverse market participation, including locally emerging companies, women and young adults, and give developing countries access to know-how and technologies comparable to those available in high-income countries. The Strategy was initially built around a set of nine Strategic Initiatives7 and a cross-cutting theme on capacity building. For 2014 MAIZE has been re-organizing around Flagship Projects (FP) and clusters of activities. It is in the process of defining impact pathways and measurable targets for the FPs and IDOs. The transition from SIs to FPs and the IDOs addressed by the FPs are shown in Table 1. Table 1: Current Flagship structure and IDOs FLAGSHIP IDO Former SI Flagship 1-Sustainable intensification and IDO1: Productivity SI 2 and SI 3 income opportunities for the poor IDO2: Food security IDO4: Income IDO5: Gender IDO6&7: Capacity to innovate and adapt IDO9: Environment IDO11: Climate Flagship 2- Novel tools, technologies and IDO1: Productivity SI 4, SI 8, SI 9 traits for improving genetic gains and IDO2: Food security breeding efficiency IDO3: Nutrition Flagship 3 - Stress Resilient and nutritious IDO4: Income SI 4, SI 7 maize IDO5: Gender Flagship 4- Aligning with strengthening IDO9: Environment SI 5 maize seed systems for effective product IDO11: Climate delivery. Flagship 5-Inclusive and profitable maize IDO2: Food security SI 6 and SI 1 futures IDO4: Income Source: Evaluation team, based on POWB 2014. The project alignment by FP has been nearly completed. Among the SIs, the largest by far is SI 4 on Stress tolerant maize, which will be go mostly to FP 3 with some activities on novel tools to FP 2. It includes mostly bilateral projects, among them very large projects like Drought Tolerant Maize for Africa – Phase III (budget of USD 24 million), Water Efficient Maize for Africa (WEMA) Phase I and II (budget of USD 6.5 million and USD 10.5 million), Improved Maize for African Soils (IMAS) (budget of USD 14.5 million), and Achieving sustainable Striga control for poor farmers in Africa (Budget of USD 3.4 million). The second largest SI has been SI 2, which will become part of FP 1. It includes SIMLESA (Sustainable intensification of maize-legume cropping systems for food security in eastern and southern Africa) with a budget of more than USD 16 million, MasAgro conservation agriculture component (USD 7.7 million), Farm Mechanization & Conservation Agriculture for Sustainable Intensification (FACASI) with almost USD 5 million and several components of the large scale CSISA (Cereal Systems Initiative for South Asia) project which is shared with WHEAT (MAIZE has a minor share). Also SI 3 on precision agriculture will be integrated in this FP. SI 5 that will become part of FP 7 SI 1 Socioeconomics and policies for maize futures, SI 2 Sustainable intensification and income opportunities for the poor, SI 3 Smallholder precision agriculture, SI 4 Stress tolerant maize for the poorest, SI 5 Towards doubling maize productivity, SI 6 Integrated postharvest management, SI 7 Nutritious maize, SI 8 Seeds of discovery, SI 9 New tools and methods for NARS and SMEs 9 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 4 includes the MasAgro component, International Strategy for Increased Maize Yields, which makes up 90% of that Flagship Project. A MAIZE gender strategy, adopted in 2013, outlines the process and approach to integrate gender aspects in the program and conduct strategic gender research. Gender activities accelerated in 2013 (with expenditure of USD 7 million) when a UNDP aligned DAX indicator was introduced to assess gender related expenditures. MAIZE funding and expenditures Next to a fully funded MAIZE scenario estimated at USD 237.8 million over three years, the 2011 MAIZE proposal included also two more conservative funding scenarios for likely available funding which were about 70% of what was considered full funding for the program activities proposed (USD 238 million). The budget which was approved per the Programme Implementation Agreement (PIA) between the Consortium and CIMMYT was USD 170.2 million for the first three years of operations, with around 26% supposed to be funded through W1/2. The difference to the full funding was defined as “expanded” component. In the 2013 Annual Reports, MAIZE begun to report part of the CRP expenditures as “supplementary”, and thus outside of the CRP. This was based on the question by the Consortium why MAIZE was spending more W3 and bilateral funding than approved by the FUND Council. MAIZE analyzed its budget situation and explained to the Consortium on 8 March 2014 that funding increased for development type funding that typically supports scale-out and accelerates the impact pathway of a CRP. This type of funding was not envisioned in the original proposal. They were included in the CRP reporting because donors themselves aligned such funding with the CRP. The budget used for strategic CRP funding on the other hand was indeed very much aligned with the original MAIZE proposal (see Table x). MAIZE proposed to label such scale-out funding affiliated by donors with the CRP as “CRP supplementary.” Table 2: Financial summary for W3&bilateral funding, 2011-2014 MAIZE 2011 2012 2013 2014 2014 Total Total Jul - Dec Jan - Dec Jan - Dec Jan - Jun Jul - Dec Phase I Phase I + 6 months A. Budget as per PIA (Scenario 2) W3 & Bilateral 19,892 40,778 42,816 21,930 125,416 Total 19,892 40,778 42,816 21,930 125,416 B. 2011-2013 Actuals & 2014 Budget Actual Actual Actual Budget Budget W3 & Bilateral: CRP Strategic 22,601 41,864 45,432 23,967 23,967 133,864 157,831 New Research: Maize Lethal Necrotic Virus 656 398 398 1,054 1,452 Total 22,601 41,864 46,088 24,365 24,365 134,918 159,283 C. W3 & Bilateral: CRP supplementary 22,201 1 25,357 13,349 13,349 38,706 74,255 1 Retroapplied based on mapping of 2013 accounts Source: CRP management. 10 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 For Phase I (July 2011 – June 2014), around 14% of the W1&W2 funding were used for CRP Management, 56% for research at CIMMYT, 12% for research at IITA and 18% for research by partners, very much aligned with the CRP proposal and the geographic focus of CIMMYT and IITA operations (CIMMYT in Latin America, Asia and eastern and southern Africa; IITA in West and Central Africa). . Table 3: Actual W1&W2 expenses for MAIZE, as reported to the Consortium and Budget approved by the MAIZE Management Committee for completing Phase I of MAIZE (Jul 2011 – Jun 2014). Actuals Expenses Expenses Expenses Budget Total Percentage 2011 2012 2013 2014 Phase I 2012 - 2014 CRP Management 493 1,713 1,872 1,124 5,202 14% CIMMYT 2,225 8,546 6,655 3,832 21,259 56% IITA 682 1,578 1,520 860 4,640 12% Partners 24 1,664 3,091 1,894 6,674 18% Total 3,424 13,502 13,138 7,710 37,774 86% Source: CRP management. Figure 1 shows expenditures for 2012 to 2013, budget estimates (in orange) for 2014 (from POWB) and 2015-2016 (from Extension Proposal) per funding source. In the 2015-16 extension proposal, the total MAIZE budget is projected to reach USD 100 million by 2016, including supplementary funding which MAIZE continues to separate from the so-called “CRP strategic funding”. Figure 2: MAIZE Expenditures (2012, 2013) and Budget predictions (2014-2016) in USD thousands 120,000 100,000 29,366 Supplementary 80,000 28,000 25,357 23, 3 3,250 2,161 4 0 Additional W1/2 60,000 requirements 64,065 50,273 52,726 40,000 W 3/Bilateral 45,681 41,479 20,000 W1/2 22,601 13,502 13,138 13,281 16,970 18,667 0 3,424 2011 2012 2013 2014 2015 2016 (est) (est) (est) Source: Financial Reports 2012 and 2013 (L101), POWB 2014, Extension Proposal 2015-2016. 11 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Figure 3 shows the expenditures for 2012 and 2013 per Strategic Initiative, for both CIMMYT and IITA. 8 Figure 3: MAIZE expenditures per Strategic Initiative (2012 and 2013) Source: MAIZE Annual Reports 2012 and 2013. The level of W1/2 funding was established on basis of unrestricted core funding of the research in 2010 with an estimated annual increase of 5%. W1 complements W2 within these limits, which has restricted W1/2 growth beyond the initial trajectory. In its first 2 ½ years of operation (mid-2011- 2013) MAIZE received the lowest level of W1/2 funding (as proportion of total expenditures), 19% on average. The criteria by which W1 funds have been allocated have been questioned by MAIZE management. The two funding windows are coupled, which means that successful resource mobilization to W2 leads to lower level of W1. Thus, there seems to be little incentive for the CRP to mobilize W2. The largest donors (based on MAIZE Financial Reports) for the MAIZE program are shown in Table 2 below. BMGF through various projects and SAGARPA through MasAgro are by far the most important bilateral donors for MAIZE. If the CGIAR Fund (and its contribution by Window 1 and 2) would be seen as an individual donor, it would still be smaller than those two donors. Table 4: Largest MAIZE donors DONOR 2012 2013 Comments (main activities) AATF Pass through money from Water Efficient Maize for Africa (WEMA) leader: The African Agricultural Technology Foundation, actually 1,969 - funded by BMG, in 2013, CIMMYT received money directly from BMGF 8 In 2011 only overall expenditures were reported, but they were only at around a third of the expenditures in 2012 and 2013 and proportional spending along SI is likely not have been effected. 12 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 for WEMA ACIAR 2,416 2,444 SIMLESA and FACASI Support to Agricultural Research for Development on Strategic AFDB 1,087 1,840 Commodities in Africa (SARD-SC) project which is led by IITA Drought Tolerant Maize for Africa – Phase III, Improved Maize for African Soils (IMAS), Water Efficient Maize for Africa- Phase II, Achieving sustainable Striga control for poor farmers in Africa,DH Maize: A Doubled Haploid facility for strengthening maize breeding BMGF 17,543 14,873 programs in Africa CIDA 1,645 - Nutritious maize for Ethiopia Harvest Plus 1,381 - Only in 2012, since 2013 it is part of CRP on A4NH IFAD 2,791 1,548 EC/IFAD CGIAR Programme SAGARPA 22,678 12,589 MasAGro Effective Grain Storage for better livelihoods of African Farmers- Phase II, New Seed Initiative for Maize in Southern Africa (NSIMA) - Phase III, SDC 3,876 2,549 The Hill Maize Research Project (HMRP) - Phase IV Developing maize resistant to stem borer and storage insect pests for Eastern and Southern Africa- IRMA III Conventional, Affordable, SFSA 1,487 1,757 Accessible, Asian (Triple A) drought tolerant maize CSISA components which are in MAIZE, co-funder of several projects USAID 4,224 2,955 that BMGF fun, also funds IITA significantly Source: MAIZE Annual Reports 2012 and 2013. MAIZE project portfolio In 2013, CIMMYT reported 107 bilateral/W3 funded projects and IITA reported 9 bilateral/W3 funded projects within MAIZE. The current database of projects includes a total of 220 different bilateral and W3 funded projects and W1/2 activities (related W1/2 activities spread over several years have been consolidated for the purpose of establishing the project portfolio).9 It includes 22 IITA projects currently mapped to the CRP. Since the database only refers to total (often multiyear) budgets and since the grant duration vary, any real comparison would be misleading. Twelve W1/2 activities were included from the Work Plan 2014 without defined budget and therefore the total portfolio budget is slightly higher than USD 216.5 million. MAIZE Budget (in USD millions) 2013 (1) Accumulated (2011-2013) Total CIMMYT IITA Total CIMMYT IITA W1/ W2 activities 14 12.51 1.50 34 30.11 3.88 W3 Projects 20 18.47 1.93 34 28.64 5.23 Bilateral Projects 32 26.88 4.96 117 105.89 11.05 Total 66.26 57.87 8.39 184.80 164.64 20.16 (1) It does not includes Supplementary Projects Source: CRP management. 9 Ten of the IITA activities which were initially included have been re-mapped 13 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Based on the project database, when looking at the different Flagships, the by far largest one is Flagship 3 on stress resilient and nutritious maize with more than 40% of total budget of which almost all of it comes from bilateral funding. The second largest Flagship is FP 1 on sustainable intensification of maize-based farming systems which is mostly the same as the previous SI 2 and includes MasAgro components on sustainable intensification, SIMLESA and a share of CSISA. Most of MAIZE activities target sub-Saharan Africa with 72% of the total project budget allocated to African countries or regional African projects and activities. IITA covers West and Central Africa and CIMMYT mostly Eastern and Southern Africa (with regional activities like SIMLESA and Improved maize for African soils, IMAS). Of the 8% project funding marked for Latin America, the SAGARPA funded MasAgro project (shared with WHEAT) accounts for 90%. Ten percent of project funding is allocated to Asia including: Heat tolerant maize for Asia project, ACIAR-Climate Resilient systems in Nepal and the CSISA program in Bangladesh, India and Nepal as well as a Hill maize research project in Nepal. The remaining 10% is funding of global projects. In 2013, the countries which accounted for most project activity in terms of expenditure, although several of it with regional focus, were: • Kenya (IFAD contribution, DTMA, WEMA II, IRMA III, ACIAR-Adoption Pathways in eastern and southern Africa, SDC Effective Grain Storage) • India (CSISA, HTMA, GTZ-Abiotic stress tolerant maize for increasing income and food security among the poor in Asia, Syngenta-Affordable, Accessible, Asian (‘Triple A’) Drought Tolerant Maize) • Ethiopia (FACASI, Nutritious Maize for Ethiopia, IFAD CASFESA) • Zimbabwe (ILRI/ACIAR Crop and Livestock, NSIMA III, Zambia maize-legume systems) Some projects also contribute to other CRPs and are not accounted for 100% in the project portfolio. For example, IITA reports 50% of the large project “Achieving sustainable Striga control for poor farmers in Africa” and 25% of the Development of Strategic Crops Africa” (SARD-DC) project outside of MAIZE. Shares of the CIMMYT projects CSISA and MasAgro Take it to the farmer are also reported under WHEAT; to mention two examples. Table 5 below shows some of the large projects which contribute to other CRPs than MAIZE. Table 5: Major shared projects Project Title BUDGET % MAIZE Comment (USD million) CSISA II and related projects ~ 21 Ave 31% CSISA shared with WHEAT, GRISP and CSISA IRRI/USAID-CSISA expansion in Bangladesh ~2.5 Ave 31% Shared with WHEAT, GRISP and CSISA MASAGRO, several components ~20 Ave 62% Shared with WHEAT; Take it to the Farmer has larger MAIZE share, Seeds of discovery is equally spilt Achieving sustainable Striga control for ~7 50% Shared with CRP on Humid tropics poor farmers in Africa Development of Strategic Crops Africa ~4.8 75% Shared with CRP on Humid tropics Source: MAIZE Project database, as of 1 August 2014 14 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 W1/2 are assigned to specific CRP research activities, partners and CRP managementreported as ‘planned’ in the Program of Work and Budget template at the beginning of the financial year. CRPs are obliged to report on major deviations to plan. Thus, W1/2 funds become restricted through eth annual workplanning process, but can be managed as CRP activities within and across FPs (compared to W3/bilateral funding earmarked to a specific project).W1/2 funding is aimed to make the program more coherent, used as “glue” to complement bilateral projects, and to enable MAIZE to pursue an overall strategy. W1/2 is targeted specifically at some FPs. FPs 1 and 3 rely to a large extent on very large bilateral projects while FPs 2, 4 and 5 have more significant contributions from W1/2 (36%, 30%, 35%, respectively). By October 2013, more than 40 institutions had been awarded a MAIZE competitive grant addressing themes such as mechanization, gender empowerment, sustainable smallholder practices, climate change vulnerability and the management of MLN and Tar Spot Complex diseases, in addition to testing new maize varieties with resistance to diseases, Striga and environmental stresses. MAIZE Governance and Management MAIZE is governed by a set of formal Agreements: The CGIAR Fund Council and the Consortium signed a Joint Agreement in April-May 2011 that sets out the umbrella terms which govern the submission and approval of CRP proposals and the transfer and use of W1-2 funds to CRPs, and a Consortium Performance Agreement in relation to MAIZE, in which the Consortium assumes overall financial and programmatic responsibility for the implementation of MAIZE. The Program Implementation Agreement (PIA) is between the Consortium Board (CB) and CIMMYT. CIMMYT is accountable to the Consortium for the use of the W1/2 funds that are transferred to CIMMYT, and for the satisfactory performance of MAIZE. The Program Participant Agreement (PPA) is signed by CIMMYT and IITA outlining the individual Center use of W1/2 funds. CIMMYT, as the Lead Center, has the fiduciary and legal responsibility for the use of funds it receives from the Consortium, Fund Donors and Bilateral Funders; and its Board of Trustees, advised by its Director General, has the ultimate responsibility for approving workplans and budgets related to the CRP.10 MAIZE Management Committee (MC) is responsible for program implementation. It includes CIMMYT global program directors, the DDGs of CIMMYT (chair) and IITA, the R4D Director of IITA and three non-CGIAR Primary Research Partners (Syngenta Foundation, KARLo-Kenya, INIFAP- Mexico). All decisions on fund allocations, choice and allocation of competitive partner grants are done by the MAIZE Management Committee within the FUND Council approved budget. Decisions cannot be made by one institution alone and have to be supported by the non-CGIAR partners. The Management Committee also reviews annual workplans and reports. 10 In practice, the MAIZE Management Committee approves the MAIZE work plans. 15 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 The MAIZE Stakeholder Advisory Committee (StAC) provides independent oversight and overall governance. It consists of the DGs of CIMMYT and IITA and five non-CGIAR partners. The MAIZE StAC is chaired by a independent/non CGIAR member (Mary Ann Sayoc) Starting September 2013, the CIMMYT Program Committee Chair attended the MAIZE Stakeholder Committee meeting as a non-officio observer and reports his findings to the subsequent CIMMYT Board of Trustees meeting. The CIMMYT Board and its Program and Audit Committees, the MAIZE StAC and MC, and the Partner Center’s Board and Management are thus expected to oversee and manage CRP-related activities in accordance with the PIA, PPAs and these bodies’ respective Terms of Reference. A small management team handles day-to-day management, It is based at CIMMYT and staffed by a Program Manager (David Watson), three full time and two part time staff (shared with WHEAT), and overseen on a day-to-day basis by the Chair of the MAIZE Management Committee (Marianne Banziger). It is responsible for CRP administration and communication; coordination among the CIMMYT Global Program directors, IITA, FP Leaders, and Donors and Collaborators; and – as a large part of work – coordination with and reporting to the CGIAR Consortium. A recent, IEA commissioned Review on CRP Governance and Management concluded that, in general, the CRP governance and management arrangements did not give CRP leaders sufficient powers to manage for results. The Review emphasized the need for CRP governing bodies to have greater independence than generally was the case, and to exercise programmatic oversight of the CRPs while reporting to the lead-Center Board. The review concluded that in CIMMYT’s case, MAIZE StAC and MC are consolidated within CIMMYT management, thus leaving the CIMMYT board with no independent source of oversight for the program. In its recent meeting, CIMMYT Board discussed the CRP leadership and reporting arrangements to better align the CIMMYT practice with the recommendations of the Review. The CRP Management expenditures (USD 1.8 million in 2012 and 2 million in 2013) include the salaries of the CRP management unit, governance and management meeting costs, travel costs and contributions to ICT systems. Particular to MAIZE and WHEAT, FUND Council minutes show that the FUND Council explicitly requested the two CRPs to be managed together. A much smaller management budget was approved for WHEAT than MAIZE implying that MAIZE management carries part of the costs of WHEAT management. 3. PURPOSE AND SCOPE OF THE EVALUATION The primary purpose of this evaluation is to enhance the contribution that MAIZE is likely to make towards reaching the CGIAR goals and towards reducing poverty and improving food security for people whose livelihoods depend on maize and maize-based systems. As for all CRP evaluations, the purpose of the evaluation of MAIZE is to provide essential evaluative information for decision-making by Program management and its funders on issues such as extension, expansion and structuring of the program and adjustments in some aspects of the program. 16 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 A second call for CRPs will be launched in 2015 following the approval of a revised SRF by the end of 2014. In that context, the evaluation of MAIZE will provide information for decisions on the program formulation and selection of 2nd cycle CRPs. The broad scope of the evaluation is set in the TOR. The strategic issues and evaluation questions that address the main evaluation criteria are structured around two dimensions: Research/ programmatic performance and organizational performance. The criteria and aspects related to these two dimensions are outlined in Section 4 while the more specific evaluation questions are presented in detail in the Evaluation Matrix (Annex 1). The evaluation will give emphasis to gender and capacity building as cross-cutting topics particularly as they pertain to programmatic performance. It will assess partnerships as strategically important for mobilizing research competence outside the CGIAR and essential for enhancing the relevance, delivery and uptake of the CRP’s results as outlines in the impact pathways. Specific evaluation questions will address these cross-cutting topics. The evaluation will cover program planning, all research and research related activities of MAIZE, thus including activities funded from all funding sources. Thus, it will also cover the funding marked as “supplementary” in the budgets for 2013-14 and projected for 2015-16 in the extension proposal. It will cover all W1/2 funded activities and their strategic use across the MAIZE research portfolio. The evaluation will assess the purpose of competitive funding within W1/2 and its contribution to results and engagement of partners. W3 and bilaterally funded projects will be cover through sampling and in-depth case studies. Given that MAIZE carries on the long-term CGIAR research on maize and maize-based systems, the evaluation will put emphasis on the summative part, looking at the extent to which past research has led to outcomes and impacts. The evaluation will assess the results – outputs, outcomes and impacts – generated from research prior to establishment of MAIZE and filling the results pipeline also into the future for some time. The evaluation will consider impact from past activities as documented since the most recent CIMMYT and IITA EPMRs (2004 and 2007, respectively). It will look at output results since 2010 with cut-off date 1st of September 2014. The major emphasis will be on assessing the design and implementation of the MAIZE CRP as a multi-partner CRP with newly defined program structure, targets and impact pathways, its likely effectiveness and the extent to which it adds value over and above doing the research separately in two centers each with their partners. The evaluation will cover program evolution since the inception in early 2011 considering all changes and information available by the end of the evaluation inquiry phase (end of 2014). It is acknowledged that the CGIAR reform is setting the context for these changes and this context is evolving. However, the program evolution is assessed against the CRP’s overall objectives to determine the extent to which it is likely to succeed. The selection of field sites and projects for sampling and in-depth analysis reflect the MAIZE regional focus, with majority of activities in SSA, some in Asia, and limited attention in Latin America. The global research will be addressed in the evaluation. 17 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 In the context of the CGIAR reform, and based on initial document review and engagement with the CRP and its stakeholders, the evaluation, while addressing specific issues within the broad evaluation criteria presented in Annex 1, puts focus on the following issues: • Is the CRP evolving in such a way as to demonstrate added value to research on maize and maize-based systems, in comparison with the research done through two Center mandates as previously. • Is MAIZE priority setting effective in terms of program coherence and focus of research on its intended objectives, given the relatively small proportion of unrestricted (W1/2) funding and the historic mandates of the two participating Centers? • Is MAIZE designing and shaping future partnerships to articulate a sustainable research project portfolio? • Is the CRP managing well the very high and increasing level of restricted funding in terms of program quality and effectiveness (including high-quality staff), sustainability and administrative load? • Are the impact pathways in the new CRP structure sufficiently specified regarding target beneficiary groups and alternative research and industry providers, and are they clearly formulated and used in program monitoring and management? 4. EVALUATION FRAMEWORK AND CRITERIA This chapter provides an overview of the evaluation issues and framework for addressing the criteria under these two dimensions and the three cross-cutting topics; gender, capacity building and partnerships. Data collection and methods are elaborated in the subsequent sections. An evaluation matrix in Annex 1 provides an extensive list of evaluation questions. 4.1. Programmatic performance As part of programmatic performance, the evaluation will look at the following evaluation criteria: relevance, quality of science, likely effectiveness of the CRP as currently designed and implemented, impact of past research and effort made in documenting it, and sustainability of benefits. Within programmatic performance three cross-cutting topics are specifically addressed: gender, capacity building and partnerships. Relevance The assessment of relevance relates to the strategic coherence of the program regarding the CGIAR’s strategy and the System Level outcomes and the program objectives; its relevance in the global context of agricultural development and research priorities and priorities within its target agro-ecologies and beneficiary groups; and the comparative advantage of the program partners to conduct the activities and generate international public goods. The evaluation will assess the formulation of the IDOs and their consistency with the needs and priorities of the intended users and beneficiaries of MAIZE research, the integration of research within and among the FPs, and the prioritization of activities for addressing the IDOs. It will assess relevance also against emerging research opportunities. Priority setting processes will be looked at, 18 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 as also the way in which relevance is enhanced in the use of W1/2 funding, resource mobilization and strategic foresight. The evaluation will also assess the synergies among MAIZE partners, IITA and CIMMYT in particular, and integration of previously separate Center components into a coherent program. Quality of science The evaluation of science quality will look at several dimensions of quality from research management and incentives to assure quality to team and leadership competences to quality of research design and the knowledge and outputs generated. Does the CRP have in places sufficient processes and incentives for ensuring high research quality across program components and partners? Is the quality and track record of team leaders sufficient and are the competences among research staff and in teams appropriate? Given the high recruitment rate particularly at CIMMYT, are research staff adequately mentored, oriented and motivated to enable high quality inputs? Does program design at project level demonstrate state-of- the art knowledge of the research areas and sufficient extent of novelty? The quality of research outputs will be evaluated. It is acknowledged that germplasm constitutes a major output of MAIZE, in addition to scientific publications and other outputs. Effectiveness Effectiveness is assessed from the point of view of current and likely effectiveness of the program, rather than past impact. The evaluation looks at the program design, particularly the impact pathways and theories of change and their realism and plausibility of impact; the extent to which assumptions and risks have been taken into account and the program is considering constraints to uptake and adoption of results, and factors that influence scaling up, outcomes and impacts. It will look at the extent to which gender analysis is informing the impact pathways. It will look at the extent to which the program is addressing constraints, for instance by incorporating capacity building to research activities, engaging appropriate partners and addressing gender specific constraints. This includes also assessment of the linkages that MAIZE has with other CRPs for enhancing the likely effectiveness of its research. The evaluation will also assess progress towards milestones and outputs across the research portfolio. It will assess the M&E system and the extent to which it is used by management to adjust research plans and impact pathway designs; this includes learning from adoption and impact studies. Impact and likely sustainability An important part of the summative component of the evaluation will be to assess the extent to which past research has led to outcomes and impact. There are, however, limitations to this component of the evaluation as it is restricted by the availability of evidence of impact. The assessment of impact will look at the claims made by MAIZE of adoption, outcomes and impacts and the evidence to support such claims. The credibility of the evidence and quality of studies will be assessed. This assessment will take into account the challenges related to attribution of long-term impacts to research. The evaluation will also consider the adequacy of impact assessment by the 19 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 CRP and participating Centers. On basis of available evidence, the evaluation will consider the overall magnitude of impacts from past research. In addition to assessing past impact, the evaluation will also assess the processes in place for facilitating impact assessment in the future; such as collecting baseline data and budgeting for adoption and impact assessments. To the extent possible, the evaluation will look at the sustainability of benefits from MAIZE research; particularly how sustainability has been addressed in the theories of change and what measures are taken in program implementation, capacity building and managing partnerships for enhancing sustainability. 4.2. Cross-cutting topics Gender MAIZE has a gender strategy that has been developed with the guidance from the Consortium Office – as with all CRPs. MAIZE claims that it has spent more than USD 7 million on activities related to gender in 2013. The evaluation will assess the implementation of the gender strategy in terms of the quality and sufficiency of gender analysis across the research portfolio and the adequacy gender- specific research. It will look at the extent to which gender is taken into consideration in targeting research, in research implementation and data collection and in documenting lessons and impact. Capacity building Capacity building strategy and budgeting are a major component of each project. Strengthening national programme capacities to reinforce sustainability of research is a major objective of CGIAR research activities. The evaluation will look at how capacity building is prioritized for addressing partners’ needs within the boundaries of available resources; the incorporation of capacity building into research activities for mentoring and enhancing the relevance and likely uptake of research results; the consideration of capacity issues among assumptions and risks related to the theories of change; and gender equity in targeting capacity building activities, for instance training and skills development. Furthermore, the evaluation will assess the balance between capacity building and other type of activities. Partnerships The evaluation will consider the partnerships among the implementing centers (CIMMYT and IITA), linkages with other CRPs, and partnerships with both research and development partners as well as boundary partners upon whom the development outcomes depend. It will assess issues such as: strategic prioritization of partnerships, incentives for partners to contribute to MAIZE, involvement of partners in research decision-making, funding, coordination and transaction costs, and joint ownership of results. 4.3. Organizational performance Governance and management The evaluation of organizational performance pertains to governance and all aspects of management that affect the CRP’s performance and ability to produce results in timely and effective 20 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 manner. The evaluation will look at the proficiency of the program’s governance and management structures, functions, and processes in facilitating the achievement of the program’s objectives in an efficient and effective manner. With reference to the principles of good practice in the governance and management of large partnership programs11 and following the methodology of the recently completed CRP Governance and Management Review, the evaluation will assess MAIZE governance and management arrangement and functions for efficiency, accountability, transparency and fairness and - governance in particular - for independence and legitimacy. The evaluation will draw from the above mentioned Review, which also provides cross-CRP comparison and reference. As part of the organizational performance, the evaluation will assess the efficiency (cost effectiveness) and, when applicable, effectiveness (in terms of enhancing the programs ability to perform towards its objectives) of the MAIZE governance and management functions. Several aspects of management will be covered by the evaluation. These include: • program management and leadership; • the CIMMYT research management system in serving the CRP needs; • accountability and reporting; • monitoring and evaluation system in informing management decisions (learning) and for reporting; • financial management and resource mobilization; • management of intellectual property; • partnership management • risk management Human resource management, including staff performance assessment, is the responsibility of each participating Center, and therefore human resource management aspects will be evaluated in the context of the CRP and its ability to perform well. The evaluation will assess the efficiency (cost effectiveness) and, when applicable, effectiveness (in terms of enhancing the programs ability to perform towards its objectives) of the MAIZE governance and management functions. The evaluation will also investigate issues that have emerged from the initial interviews with CIMMYT and MAIZE management, namely, the management of conflicts of interests and the host relationship between CIMMYT and MAIZE. It will also look at CIMMYT in its leadership position regarding the MAIZE CRP. The CRP Governance and Management Review highlighted two issues that CIMMYT BoT and management are considering in terms of any changes that may be required in the oversight and management arrangements regarding the two CRPs CIMMYT is leading, MAIZE and WHEAT. One is the recommendation to assure the independence (and other attributes of good governance and management listed in the Review) of the CRPs’ governing bodies by establishing 11 See the IEG/DAC 2007 Sourcebook for Evaluating Global and Regional Partnership Programs (GRPPs) 21 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 direct reporting from CRP leader to the governing body and subsequently to the lead-Center BoT; rather than have the reporting line through the DG or DDG. The other recommendation debated by CIMMYT was to strengthen the CRP leaders’ powers to manage for results. The BoT is likely to reach a decision on these issues in its September meeting to be held in conjunction with the MAIZE Stakeholder Committee meeting. This evaluation will consider the MAIZE/CIMMYT decisions and possible changes to assess the governance and management arrangements and functions. 5. METHODOLOGY, DATA COLLECTION AND ANALYSIS This section presents the selected methods that are used in the evaluation. Data requirements and tools used for specific criteria and questions are also given in the Evaluation matrix (Annex 1). Data and information will be collected at multiple levels, depending on the criterion and evaluation questions. For example, some aspect of science quality will be assessed at CRP level while other will be assessed at the FP or discipline level. Relevance will be assessed for example at the level of sampled projects for drawing conclusions at the FP and CRP level. The MAIZE project portfolio and list of activities as mapped to the Flagship Projects as of 1st of August 2014, is the primary basis for assessing several criteria and issues. Some analysis can be done across the entire portfolio but mostly the evaluation will base its conclusions on sampling some 25% of the portfolio. The set of sample projects is given in Annex 4. Projects were selected to be representative of size, maturity, geographic focus and research strategy. These sampled projects will be assessed for a number of characteristics, particularly their alignment with the CRP objectives, Flagships and IDOs and other aspects of relevance; and their impact pathway design, partnerships, progress to-date, delivery strategy and other aspects influencing likely effectiveness. In-depth analysis will be conducted for a small subset of projects. Project sampling has also influenced the choice of field sites where information and perceptions will be collected specifically on the projects in addition to other aspects of the CRP. 5.1. Data collection and analysis Document review The evaluation uses several sources and several kinds of documents for basic information about the program, its approval process, design and evolution; governance, management and financial arrangements and decision-making; the CGIAR reform context and its evolution; guidance regarding expectations from the CRPs; and the evaluation conduct in the reformed CGIAR (see document specification in the Evaluation Matrix, Annex 1). The team is reviewing documents on the following areas: • development and approval of MAIZE (original MAIZE proposal; ISPC comments and CO recommendations); • governance and management processes (organisational handbooks; ToR and minutes of meetings of Center Boards, MC and STAC, Programme Teams, Country Teams and Committees) • progress of MAIZE (annual reports, etc.); 22 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 • extension of MAIZE and structural changes; • issues within the CGIAR and other CRPs relevant to MAIZE. Project activities mapped to MAIZE The portfolio analysis covers all activities mapped to MAIZE. The following minimum data have been collected for all activities: • Lead center/main implementing institution • Project/activity title • SI and FP to which project/activity is mapped • Source of funds: W1-2, W3, and bilateral • Donor(s) • Start date • End date • Maturity of the project in years • Total budget • Proportion of total budget for MAIZE • Countries and region of operation The project portfolio information allows some descriptive analysis over the total CRP portfolio and sampling of projects for further analysis and field visit planning. Analysis across the portfolio will look at the following aspects: • Size distribution • Timing and duration • Geographic distribution • Use of W1/2 vs W3/Bilateral funds Analysis of sampled projects The evaluation will undertake a review and analysis of a sample of 40 W3/bilateral projects and W1/2 activities (for list see ANNEX 4) which were selected as follows: • 10 largest projects in the Portfolio • 30 randomly selected projects by Flagship (within cut-off size) This should ensure that the matching analysis adequately covers a representative sample of the MAIZE project portfolio. As the main input, the review will use Project Documents/ Proposals and Progress Reports. It will focus on: (i) relevance and coherence of individual activities, by assessing how well activity objectives match with the overall program objectives and Flagship/cluster objectives (or with the equivalent strategic initiatives for the previous years); (ii) quality of science by looking at project design; (iii) likely effectiveness by looking at the realism of impact pathways and progress towards results; and (iv) cross-cutting topics related to relevance and effectiveness. Some of the information collected will be factual, and some will represent evaluative judgments. An IEA Evaluation Analyst will extract factual information about each activity from the activity proposals 23 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 and progress reports, and the core team members responsible for each Flagship will undertake the evaluative assessments related to the evaluation questions. In-depth case studies The evaluation will conduct five in-depth case studies based on bilateral projects in order to assess Program Management functions of MAIZE. The case studies have been purposefully chosen from the largest projects, aiming to cover different research strategy streams (germplasm/varieties and sustainable intensification) as well as Flagships, with emphasis on FP3 which has by far the largest share of MAIZE. Table 6: In-depth case study projects Project Title FP Research strategy Associated country Team members visit Drought Tolerant maize FP 3 germplasm ETHIOPIA, KENYA Scott Chapman for Africa phase III Javier Betran Improved maize for FP 3 germplasm KENYA12 Javier Betran African Soils Sieg Snapp CSISA13 FP 1 Sustainable NEPAL Sieg Snapp intensification Guido Greyseels Nutritional Maize for FP 3 germplasm ETHIOPIA Seyfu Katema Ethiopia Scott Chapman Identifying Socio FP 5 Socio-economics ETHIOPIA Guido Greyseels Economic Constraints for Seyfu Katema faster technology Adoption The case study analysis will be based on field visits observations, semi-structured interviews (with associated program directors, projects leaders and key staff, partners/collaborators and beneficiaries) and documentation analysis (proposals and progress reports for projects, external reviews, and competitive contract grants for inclusion in case studies). The analysis focuses on: • Relevance or key drivers (theory of change, targeting and impact assessments or narratives) justifying the project investment(s) and anticipated outputs (relative to desired program-level outcomes); • Coherence, quality and efficiency of project design relative to anticipated project outputs; • Appropriateness of project prioritization, project sequencing, and project funding relative to program-level IDOs; • Rationale for choice of partners (up- and down-stream); • Interrelationship between bilateral projects and W1/2 funded activities; 12 Improved maize for African Soils is also active in Ethiopia, which however is not a key country for the project. 13 Initially it was proposed to do an in-depth case study of Sustainable Intensification Smallholder Maize Livestock Systems South Asia. However the evaluation team was advised that this project has already closed and may not be representative of the MAIZE Asian activities. 24 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 • Integration of gender; • Definition and quantitative verification of project outputs delivered (what, to whom, where, when, and why); • Associated capacity building or maintenance including training and technical or financial support of essential partners/collaborators; • Adjustments in impact pathway management and impact narratives based on project progress (learning). Country visits The evaluation team’s field visits include Mexico, Ghana, Kenya, Ethiopia, Nepal and China. The headquarters of CIMMYT, the lead partner in the MAIZE CRP, is located in Mexico so it was logical that the inception phase that the whole team began there (May 2014). Ghana is a principal site of IITA MAIZE CRP in West Africa and during the first week of September an IITA MAIZE team meeting took place in Tamale, in the presence also of the IITA DG and DDG-Research. A representative of the evaluation team attended the MAIZE stakeholder meeting in Beijing on Sept 17 and the CIMMYT Board of Trustees meeting 18-20 September. Kenya (double haploid facility) and Ethiopia (nutritious maize) are among important research sites for CIMMYT and its partners in East Africa. Nepal is an important and representative research site for MAIZE program in Asia. The country visits will allow observations and analysis of stakeholder perceptions through interviews. They will also be used for in-depth analysis of selected projects to gain better understanding of processes and factors that are affecting program implementation and results. The sample of countries has been selected on basis of major activities as evident in the MAIZE project database. The criteria included: • Coverage of in-depth case study projects (please see above) • Level of MAIZE activity in the country and history of earlier Center activities • Potential to observe W1/2 and bilateral activities and their interlinkages • Potential to observe important CRP characteristic such as gender activities, competitive partner grant implementation, new research Table 7: Country visit sites Country Primary purposes of site In-depth case study projects to be covered visit Mexico Inception meeting and meetings with lead center senior management Ghana In conjunction with IITA No in-depth case study projects, but the following IITA projects: Extended Management • The Multinational - CGIAR Project: Support to Agricultural Research Meeting meet IITA for Development on Strategic Commodities in Africa (SARD-SC) managers and staff working • Drought Tolerant Maize for Africa (also in Kenya and Ethiopia) with CRP MAIZE, MAIZE • West Africa Seed Program (CORAF/WECARD partners and stakeholders Kenya CIMMYT regional office; • Drought Tolerant Maize for Africa – Phase III IITA research station • Improved Maize for African Soils Major and long-term • Identifying Socio Economic Constraints for faster technology projects on maize; large Adoption number of current CRP projects Ethiopia Attendance at the • Drought Tolerant maize for Africa phase III stakeholder meeting to • Nutritional Maize for Ethiopia interviewing major MAIZE • Identifying Socio Economic Constraints for faster technology 25 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 partners and stakeholders; Adoption specific MAIZE project activities in Ethiopia Nepal In Asia major long-term • CSISA activities related to maize. China Attendance of MAIZE • MAIZE Stakeholder Advisory Meeting. 18 Sept 2014 governance body meetings • CIMMYT BoT Program Committee Meeting, 21 or 22 Sept 2014 Semi-structured interviews The interviews are already going on and will be completed by mid-November. Team members will interview a representative selection of partners, external stakeholders and peers, donor and other individuals knowledgeable of the CGIAR, MAIZE and global maize research in agricultural development context. Most of these interviews will be conducted during the field visits. However to get a broader perspective of stakeholders, each team member will also interview around 15 additional stakeholders per telephone/skype. The list of interviewees will be completed by the team and is based on a list of partners and researchers of MAIZE, relevant stakeholders of MAIZE and the CGIAR and the peers known to the evaluation team. Suggestions will be drawn from the Reference Group, IEA and other suitable sources. The final choice should be representative in terms of institutional background, geography, gender and discipline. These interviews will cover the entire range of evaluation issues and questions. Interview templates will be developed for each category of stakeholder (partners, researchers, donors, international peer), specifying the context and the purpose for the interview (e.g. programmatic in general, quality of science, gender, management, governance). Semi-structured interviews will be used for the evaluation in general or as part of the in-depth case studies. MAIZE researcher survey MAIZE evaluation will undertake a survey of researchers (CIMMYT and IITA) working for the CRP, which will cover the research/programmatic performance of MAIZE addressing, in particular, aspects of value added of the CRP, relevance, quality of science and likely effectiveness, but also other aspects such as efficiency of management and cross-cutting issues (gender, partnerships and capacity strengthening). The survey will be confidential, conducted through Survey Monkey, and the results will be analyzed and presented in a manner that preserves confidentiality. The survey will be launched in the middle of the inquiry phase to allow for cross-validating primarily qualitative findings in document review, interviews and team observation at field sites. The survey will be administered during October. Quality of science analysis The evaluation will assess the quality of science at different levels: the program as a whole and Flagship level. Quality within disciplines will also be looked at. The framework for evaluating science 26 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 quality has four dimensions: (i) processes for assuring quality; (ii) input quality; (iii) output quality; and (iv) perceptions of quality. In the assessment comparisons will be primarily internal, looking at variability among MAIZE components; and judged against peer expectation of quality of international research of excellence. The assessment of different dimensions and the CRP as a whole aims at identifying variability within the CRP, highlighting areas of excellence and identifying areas where improvements could be made. Processes in place This assessment will be done at program level. The assessment aims at determining whether MAIZE management explicitly addresses quality through processes and whether this could be improved. The evaluation will look at all internal processes that are explicitly aimed at assuring quality. These include: • Internal peer processes and how they function; • Use of external evaluations/reviews as management tool; • Staff performance assessment (CIMMYT and IITA) and to what extent it is used for enhancing quality and as a talent management process; • Incentives for assuring and stimulating high quality; • Competitive grants process; the extent to which it is used for enhancing quality. Inputs to science quality This assessment will be done at FP level and it will include research staff that have team leader responsibilities; research support, resources and data; and research design. • Team leaders include all Principal Investigators, Flagship and Cluster Leaders, and Focal Points. For these lead scientists, information about their scientific track record will be assessed; • The adequacy of research support and resources; • Quality of data management; • Research design for sampled projects. ISPC comments on science quality will be taken into account (original proposal and extension proposal). Output quality Evaluation will look at both the quantity and the quality of science outputs, including publications and breeding material. The publications analysis will draw from a recent study conducted by Elsevier on Center publications output, and for the period when the CRP has been operating, list of publications mapped to the CRP. This analysis will include: • Qualitative assessment of a random sample of publications • Quantitative assessment (bibliometric analysis) of publications • Germplasm assessment: assessing breeding approaches and rate of gain advancement toward program objectives 27 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Perceptions of quality The evaluation will draw on perceptions of quality for assessing processes, inputs and outputs and as an important means for assessing science quality as it relates to breeding (research and breeder partners and peers). Other aspects will include overall science quality reputation: excellence and ambition, critical mass and comparison with other organizations. Impact analysis Impact here is defined to include results along the impact pathway beyond delivery of outputs; e.g. adoption, influence, outcomes and longer-term impacts towards the CGIAR goals. The assessment is based on a narrative provided by the CRP on such results from past research that has relevance in the current CRP. The narrative should contain claims made regarding the volume, scale and level (along the impact pathway) of the results supported by evidence that needs to be listed. The cut-off date for impact documentation is since the last EPMR for Centers involved. These narratives will be assessed against the evidence. The quantity and quality of evidence will also be assessed considering the extent to which the Center/CRP can be expected to document the outcomes and impacts of its research. Assessment will also be made on the overall scale of impacts. Governance and management analysis The evaluation will draw heavily from existing documents, for example: the CRP proposal, commentaries from the ISPC and FC, contractual agreements and guidance document from the Consortium and Fund Council. The evaluation will build upon the recently completed CRP Governance and Management review and the data and information collected for the Review. Other means for collecting data and information include structured interviews among selected stakeholders; researcher survey; analysis of the terms of reference of the MAIZE governance and management bodies; review and analysis of the minutes of participating Center Boards, and the CRP governance and management committees. (See Evaluation matrix, Annex 1). 5.2. Main Limitations or Constraints of the Evaluation The evaluation occurs at a time when the MAIZE CRP has been operational for slightly over three years. The summative part of the evaluation will therefore focus on research done at CIMMYT and IITA and differentiating results emerging from CRP operations as opposed to Center operations is not yet easy. The mere size of the CRP and its expanse over large geographical areas and the very large number of partners and even larger number of boundary partners and stakeholders pose limitations to the evaluation that has to approach the task through cross-scale sampling and selection of representative areas of operation. The breadth of analysis, field travel and sampling is necessarily limited by the resources and time available for the evaluation. No members of the team suffer from actual conflict of interest. Regarding any perceived conflict of interest, the team members’ familiarity with the CRP come from their expertise in the topics, maize and maize-based systems, addressed by the CRP and this familiarity is not judged to influence the objectivity of any member’s assessment of the CRP. 28 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 6. ORGANIZATION AND TIMING OF THE EVALUATION 6.1. Team Composition: Roles and Responsibilities The evaluation is conducted by a team of independent external experts (for team member biodata, see Annex 2). The team is composed by six members that have expertise in particular aspects of MAIZE program and management areas and in addition have extensive experience in international agricultural development, research for development and evaluating research programs and organizations. The team has the competences necessary to assess also the following aspects of MAIZE: • program governance, organization and management, including financial management • sociological and gender issues • capacity building issues • institutional and policy analysis in the context of development • research planning, methods and management • intellectual property issues • communication and partnership The evaluation team composition and main areas of responsibility are show in Table 8 below. Table 8: Evaluation team responsibilities Name Role Main responsibilities Economics’ and social sciences components of research, impact, Guido Gryseels Team leader research strategy, gender; Lead in FP 5 (SI 1, SI 2) with Sieg Snapp Plant breeding and breeding related research, private sector; Javier Betran Team member Lead in FP 3 (SI3, SI5, SI7) with Scott Chapman Crop physiology, genomics, ARI relations; Lead in FP 2, (SI8, SI9) Scott Chapman Team member with Javier Betran Plant breeding system, capacity building, NARS relations; Lead in Seyfu Ketema Team member FP 4 (SI2, SI3, SI6) with Guido Gryseels Pammi Sachdeva Team member Governance, management, finances Natural resources management research, conservation Sieglind Snapp Team member agriculture; Lead in FP 1 (SI2, SI3) with Seyfu Ketema 6.2. Evaluation governance: roles and responsibilities The evaluation team leader has final responsibility for the evaluation report and all findings and recommendations, subject to adherence to the CGIAR Evaluation Standards. The evaluation team is responsible for submitting the deliverables as outlined in more detail below. The IEA is responsible for planning, initial designing, initiating, and managing the evaluation. The IEA is also responsible for the quality control of the evaluation process and outputs, and dissemination of the results. The IEA has taken an active role in the preparatory phase of the evaluation by collecting background data and information and by carrying out preliminary analysis on the CRP on MAIZE. An evaluation manager supported by an evaluation analyst provides support to the team throughout the evaluation. 29 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 A Reference Group has been set-up to work with the IEA evaluation manager to ensure good communication with, learning by, and appropriate accountability to primary evaluation clients and key stakeholders, while preserving the independence of evaluators. The Reference Group composition is shown in Table 9. Table 9: Evaluation Reference Group Name Title Organization Marianne Banziger MAIZE Leader and DDG, CIMMYT CIMMYT David Watson MAIZE Manager CIMMYT David Chikoye Director for R4D, IITA IITA Ylva Hillbur DDG, IITA IITA John Snape CIMMYT Board Chair John Innes Institute Mary Ann P. Sayoc MAIZE StAC member East-West Seed MAIZE management (including the MAIZE Leader) has a key role in helping provide for the evaluation team’s information needs. It provides documentation and data, information on all MAIZE activities, access to staff for engagement with the evaluators, and information on partners and stakeholders. It facilitates arrangement of site visits and appointments within the lead Center and other stakeholders. MAIZE management is also responsible for giving factual feedback on the Draft Report and for preparing the Management Response to the Final Report. It assists in dissemination of the report and its finding and lessons and it acts on the accepted recommendations. While the evaluation is coordinated with MAIZE management, CIMMYT as the lead Center is a key stakeholder in the evaluation. It conducts most part of the research done within MAIZE. Its leadership and Board are expected to make themselves available for consultations during the evaluation process. 6.3. Quality Assurance In order to ensure technical rigor to the evaluation, the IEA will work closely with the Evaluation Team throughout the evaluation, and will ensure that the tools and methodologies, as well as the process followed, are in line with the CGIAR IEA Evaluation Policy and Standards. 6.4. Timeline and Deliverables The evaluation timeline is shown in Table 10. Table 10: Evaluation Timetable and Tentative Deliverables Phase Period Main outputs Responsibility Preparatory Phase Jan 2014 – May 2014 Final ToR IEA Evaluation team recruited Inception Phase May 2014 – Sep 2014 Inception Report Evaluation team Inquiry phase Sep 2014 – Nov 2014 Various analysis products as Evaluation team defined in inception report; Nov 2014 Preliminary findings Evaluation team presented to MAIZE stakeholders Reporting phase Drafting of Report Nov 2014 – Jan 2015 Draft Evaluation Report Evaluation team March Feed back on Draft Report IEA, MAIZE management 30 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Final Evaluation Report March 2015 Final Evaluation Report Evaluation team Management Response April 2015 Management Response CRP Management Dissemination phase June 2015 Communications products IEA CRP Management The Evaluation Report will be the main deliverable of the evaluation. Its recommended length is maximum 100 pages, excluding Annexes. It will describe the findings and conclusions that are based on the evidence collected within the framework defined for the evaluation criteria and issues and for addressing the specific evaluation questions (Annex 1). It will present a set of recommendations that are prioritized, focused and actionable, indicating the stakeholders that are responsible for their implementation. The main findings, conclusions and recommendations will be summarized in an executive summary. 6.5. Feedback and Responses to the Evaluation Adequate consultations with MAIZE stakeholders will be ensured throughout the process. In particular, debriefings on key findings will be held at various stages of the evaluation. Consultations, feedback and finalization of the Evaluation Report will take place as per IEA guidance on “CRP Evaluation Process for Finalization, Feedback and Response”. MAIZE management will prepare a response to the evaluation for the consideration of the Consortium Board. The Management Response will contain both an overall response to the evaluation, as well as response to each recommendation. The Consortium (Consortium Office, with approval of the Consortium Board) will review the Evaluation Report and MAIZE Management Response and provide their response on the Evaluation Report recommendations, Management Response and Action Plan. The Final Evaluation Report, MAIZE Management Response and the Consortium Board Response will be considered by the Fund Council’s Evaluation and Impact Assessment Committee (EIAC) that will lead the Fund Council discussion on the Evaluation Report and the MAIZE Management Response and Consortium Board Response, and propose the decisions to be taken. The Fund Council will endorse the evaluation recommendations, responses, action plans and proposed follow-up. 6.6. Dissemination plans The Team leader will prepare presentations for disseminating the Evaluation Report to targeted audiences. Several events will be organized to disseminate the evaluation results, including but not limited to: • Virtual presentation to MAIZE management on the preliminary findings (end of November – early December 2014); • Presentations of the Draft Report to MAIZE Reference Group, MAIZE Stakeholder Advisory Committee, CIMMYT and IITA Management and Board; Consortium (February 2015); • Presentation of the Final Report to the Evaluation and Impact Assessment Committee (EIAC) and the Fund Council (May 2015). 31 Independent iea.cgiar.org Evaluation Arrangement ANNEX 1: EVALUATION MATRIX Evaluation Issues and Questions Data Collection and Analysis Overarching questions To be addressed through the more detailed questions and data analysis shown below. • Is the CRP evolving in such a way as to demonstrate added value to research of maize and maze-based systems, in comparison with the research done through two Center mandates as previously? • Is MAIZE priority setting effective in terms of program coherence and focus of research on its intended objectives, given the relatively small proportion of unrestricted (W1/2) funding and the historic mandates of the two participating Centers? • Is the CRP managing well the very high and increasing level of restricted funding in terms of program quality and effectiveness (including high quality staff), sustainability and administrative load? • Are the impact pathways in the new CRP structure sufficiently specified regarding target beneficiary groups, clearly formulated and used in program monitoring and management? Research/Programmatic Performance Relevance Coherence Desk review of the CGIAR’s Strategy and Results Framework (SRF); the approved MAIZE proposal; MAIZE • Is the MAIZE CRP strategically coherent and consistent with the main objectives of the CRP 2015-16 Extension proposal; ISPC commentaries on and the goals and System Level Outcomes presented in the CGIAR’s Strategy and Results original and extension proposal; CO commentary on Framework? extension proposal. • Is there clear rationale for the five Flagship Projects and are they internally coherent? Matching analysis • Is the core funding (Windows 1 and 2) used strategically in key areas of the program, and for In-depth case studies. leveraging bilateral funding, to align bilateral projects within program strategy? Stakeholder interviews. • Is the MAIZE CRP defining, developing and prioritizing bilateral projects targeting SRF objectives? Comparative advantage Desk review as above. EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 In-depth case studies. • What is the comparative advantage of MAIZE (across its Flagships and activities) - in terms of Stakeholder interviews. the CGIAR’s mandate of delivering international public goods – relative to other international initiatives and research efforts, including the private sector, partner country research institutions and development agencies? • In the different areas of research (Flagship Projects, Clusters of Activity) does MAIZE play an appropriate role as global leader, facilitator or user of research compared to partners and other research suppliers? • Does MAIZE engage with appropriate partners, given their roles in implementation and achieving the objectives of the program • What is the MAIZE CRP comparative advantage and its expected future evolution across the research and development stages (i.e. from basic research to product delivery). Program design Desk review of MAIZE IDOs and impact pathways. Interviews of MAIZE management and principle • Does the program target an appropriate set of Intermediate Development Outcomes (IDOs) investigators. and are the activities relevant, of highest priority for targeting the IDOs? Researcher survey • Do the impact pathways logically link the principal clusters of activities to the IDOs and are the IDOs linked to the SLOs through plausible theories that take into account trade-offs between multiple objectives? Have constraints to outcomes and impacts been considered in the program design, for example through assessment of the assumptions and risks in reliance on policies, actions of national institutions, capacity and partnerships? • Have the CRP research activities been adequately prioritized, in line with resource availability? • Is MAIZE CRP implementing program management? How? Quality of Science ISPC commentaries • Do the research design, problem-setting, and choice of approaches reflect high quality in Publications analysis scientific thinking, state-of the-art knowledge and novelty in all areas of research? In-depth project analysis Interviews of peers In-depth project analysis • Is it evident that the program builds on the latest scientific thinking and research results? Interviews about internal processes • Are the internal processes and conditions, including research staff and leadership quality, H-index analysis 33 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 adequate for assuring science quality Researcher survey Analysis of publications and other outputs • Are the research outputs, such as publications, of high quality and what role do CRP scientists have in the publication? • Is the MAIZE CRP participating in state of the art breakthrough research initiatives with leading institutions? Likely effectiveness Review of MAIZE Annual reports and performance • Has the CRP stayed on track in terms of progress and milestones toward outputs, and along reports. the impact pathway toward outcomes? Interviews with CRP management and FP leaders • Is the monitoring system used effectively for adjusting the program on basis of lessons Assessment of M&E systems and its use in program learned? adjustment Review of impact pathways and theories of change and • Have adequate constraint analyses and lessons from ex post studies informed program design their use in program design and adjustment for enhancing the likelihood of impact? Review of impact pathways and theories of change and • Is the CRP adequately addressing enabling factors for scaling up outcomes? their use in program design and adjustment • Are processes clearly defined and quality reviews conducted to improve effectiveness? Interviews with partners during site visits. Impacts and Likely Sustainability Review of MAIZE impact narrative and evidence provided • What has been the record of the participating centers engaged in research on maize and in support to the claims. maize systems, measured as both outcomes and impacts from past research? What is the impact of MAIZE research and how is it been estimated? Same as above. • Have there been sufficient efforts to document outcomes and impact from past research, Interviews with stakeholders with reasonable coverage over all research areas? Same as above • What can be concluded from the findings of ex post studies, regarding the magnitude of impact in different geographical regions—and the equity of benefits? Interviews with stakeholders • To what extent have benefits from past research been—or to what extent are they likely to be—sustained? Gender 34 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Assessment of gender strategy, gender-related IDOs and • Has gender been adequately considered in research design in terms of relevance to and impact pathways in terms of gender considerations. effect on women? Analysis of gender research Analysis of incorporation of gender issues in sampled projects Interviews Researcher survey Same as above. • Has gender been adequately considered in the impact pathway analysis, in terms of the differential roles of women and men along the impact pathway, generating equitable benefits for both women and men and enhancing the overall likelihood enhancing the livelihoods of women Capacity Strengthening Assessment of capacity building strategy and • To what extent do capacity building efforts address partners’ needs? To what extent is MAIZE consideration of capacity in the FP impact pathways training new people who continue to contribute to the CRP? Analysis of incorporation of capacity building in sampled projects. Interviews. Researcher survey. Same as above. • Does capacity building target women as well as men adequately and their differential needs taken into account Same as above. • To what extent are capacity issues taken into account in the impact pathway analysis? Same as above. • Are capacity building efforts integrated with the research mandate and delivery of the CRP? Same as above. • Are the capacity building efforts and incentives among partners adequate for enhancing the long-term sustainability of program effects? Partnerships Assessment of FP impact pathways. • To what extent are the partnerships relevant and cover the relevant partner groups to Interviews at country sites. achieve program objectives? Same as above. • Are the partnerships chosen and managed so as to maximize efficiency and effectiveness and mutual benefits? 35 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Organizational performance Governance and Management Legitimacy. Desk reviews of the minutes of Fund Council, • To what extent do the governance and management arrangements permit and facilitate the Consortium Board, CIMMYT Board, StAC, and MAIZE effective participation and voice of the different categories of stakeholders in the governance MC. and management decisions, taking into account their roles and responsibilities? Interviews with selected staff from the Fund Office, Consortium Office, CIMMYT and IITA; StAC members, MC members. Review of CRP Governance and Management Review report and relevant recommendations of PWC review. Interviews with partners. Accountability. Same as above. • To what extent are the lines of accountability within the program well-defined, accepted, and Direct observation of the work of CIMMYT Board and being followed? Are there any significant gaps in either programmatic or fiduciary StAC meetings in Beijing. accountability? Review of CIMMYT Board decisions following the Beijing meeting. Transparency. Same as above. • To what extent are the program’s decision-making, reporting, and evaluation processes open Desk review of MAIZE planning documents, strategies, and available to the general public, subject to confidentiality requirements in scientific presentations, and report. research and in human resource management? Review of the MAIZE website. Conflicts of Interest. Same as above. Desk review of CGIAR, CIMMYT, and MAIZE policies on • To what extent are conflicts of interests being identified and managed transparently? conflicts of interest. Efficiency Interviews and survey with selected CIMMYT Board, StAC and MC members. • Are the MAIZE institutional arrangements, management and governance mechanisms Direct observation of the work of CIMMYT Board and efficient? StAC meetings in Beijing. 36 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Management effectiveness Draws from CIMMYT “ organizational culture” review planned for mid-2014. • Does MAIZE research management provide effective leadership, culture and ethos for Interviews with CRP staff during site visits. advancing the program’s objectives? Researcher survey • Is the significant growth, including recruitment of new staff, managed efficiently and effectively? • How is quality management conducted? What are the policies and processes? Financial management Desk review of CGIAR and CIMMYT financial guidelines and audit reports. • To what extent does the program have good financial management, budgeting, and Interviews with CIMMYT and IITA financial staff and reporting? StAC members. Resource mobilization and allocation Desk review of resource allocation criteria, procedures, and results. • How effective and efficient have been the criteria and the procedures for allocating the Desk review of minutes of MC meetings. program’s resources? How have the resource allocation processes and timing affected the Interviews with relevant managers and research implementation of the program’s research activities? leaders. • How effective has been the mobilization of financial resources for the program? Effects of CGIAR reform Interviews with CRP leadership, StAC and MC members Draws on findings of the previous CGIAR and CRP level • To what extent have the reformed CGIAR organizational structures and processes increased reviews. (or decreased) efficiency for successful program implementation • What lessons can be learned to date regarding the effectiveness of the new governance and management arrangements of the CGIAR influencing MAIZE and of MAIZE specifically? Collaboration Desk review of MAIZE proposal and Annual reports. Interviews with stakeholders. • Is the level of collaboration and coordination with other CRPs appropriate and efficient for reaching maximum synergies and enhancing partner capacity M&E and reporting Review of RMS. Interviews with CIMMYT and IITA managers and lead • Is MAIZE management using a monitoring and evaluation system efficiently for recording and CRP scientists. enhancing CRP processes, progress, and achievements? Risk management Review of CIMMYT risk analysis related to CRP, and 37 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 decisions on actions (September update). • Are CRP implementation and sustainability related risks adequately identified and managed? Interviews with concerned CIMMYT and CRP managers and StAC/CMMYT Board members. IP management Review of CIMMYT IP policies. Interview of IP staff and relevant research managers. • Is the management of Intellectual property used or generated by the CRP appropriately managed? 38 Independent iea.cgiar.org Evaluation Arrangement ANNEX 2 – TEAM MEMBER PROFILES Guido Gryseels (team leader) Guido is an economist and doctor of agricultural sciences who is currently Director General of the Royal Museum for Central Africa, Belgium, which is a major research institute on Africa in both the human and natural sciences. . He is also a member of the Board of Directors of Federal Science Policy and of the Fund for Scientific Research in Belgium, and Chair of the Programme Committee on Food and Business Research at the Netherlands NWO/WOTRO. Earlier, he served as Deputy Executive Secretary of the Technical Advisory Committee of the CGIAR, hosted by the FAO. He also held other important positions in the CGIAR, including that of Executive Secretary of the CGIAR's Impact Assessment and Evaluation Group and chairman of the Board of Trustees of ICARDA based in Syria. From 1979 to 1987, he held various senior positions at the International Livestock Center for Africa, in Ethiopia. Javier Betran (team member) Javier has a PHD in plant breeding. Javier is currently the Head of the Maize Breeding Europe, Africa and Middle East for Syngenta. Before that he was professor in the US. Javier was a postdoctoral research and breeder at CIMMYT in 1990s. He is an expert in Plant breeding, quantitative genetics, agronomy, statistics, biotechnology, environment, abiotic and biotic stresses, and people development. Javier has extensive international experience in maize breeding. He has a large publication record on maize breeding. He has collaborated with international organizations like the Rockefeller Foundation, CGIAR centers. Scott Chapman (team member) Scott has a PHD in agricultural science and currently does research on genetic and environment effects on growth of field crops, particularly in drought dominated regions, applying quantitative approaches (crop simulation and statistical methods) and phenotyping (aerial imaging, canopy monitoring). Scott was an Associate Scientist at CIMMYT’s Maize Program in the early nineties. Over the last 20 years Scott conducted research at the Commonwealth Scientific and Industrial Research Organisation and is currently an adjunct professor at The University of Queensland in Australia. Seyfu Ketema (team member) Before becoming executive secretary of the Association for Strengthening Agricultural Research in Eastern and Central Africa in 2002, Seyfu served as director general of the Ethiopian Agricultural Research Organization, Ethiopia's minister of agriculture, general manager of the Institute for Biodiversity Conservation and Research and worked for the Institute of Agricultural Research. Seyfu was the Regional Representative for Eastern Africa on the CGIAR and served as Board Member for ICARDA and ICRAF. Seyfu obtained his M.Sc. and Ph.D. in plant breeding from the University of London. Paramjit (Pammi) Sachdeva (team member) Pammi is specialized in program and institutional assessment and HR management with expertise also in capacity development, systems analysis and organizational design. Since 2001 he has worked as an independent consultant and been involved in a number of external reviews of CGIAR Centers and programs, and in international development project and human resource management EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 consultancies. Previously he worked at the World Bank as senior management specialist and advisor and earlier in his career at ISNAR as senior research officer. He has a PhD in social systems sciences. Sieglinde (Sieg) Snapp Sieg is an agronomist and plant physiologist who is currently Professor of Soils and Cropping Systems Ecology at Kellogg Biological Station and Department of Plant, Soil and Microbial Sciences, Michigan State University. Sieg is interested in participatory systems research and extension approaches to natural resource management and sustainable intensification. She has extensive experience with multidisciplinary teams including scientists, farmers, students, advisors and extension to support research for development. Current research projects include the Long term Ecological Research in Row Crops at MSU, sustainable intensification research through on-farm experimentation and modeling supported by IITA/USAID in Malawi, and systems analysis for perennial grains on smallholder farms in Africa, supported by BMGF. 40 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 ANNEX 3: LIST OF PERSONS CONSULTED IN INCEPTION PERIOD Name Organization Position Marianne Bänziger CIMMYT Deputy Director General Research & Partnership Ranajit Bandyopadhyay IITA Pathologist Michael Baum ICARDA Director – BIGMP Hans Braun CIMMYT Program Director Global Wheat Program Ernesto Briones CIMMYT Senior Systems Developer David Chikoye IITA R4D Director, IITA-Southern Africa Marisa De la O Elizagaray CIMMYT Manager, Risk Management & International Policy Olaf Erenstein CIMMYT Program Director, Socioeconomics Program Richard Fulss CIMMYT Head, Knowledge Management Bruno Gérard CIMMYT Program Director, Global Conservation Agriculture Bram Govaerts CIMMYT Associate Director, Global Conservation Agriculture Sara Hearne CIMMYT Senior Scientist, Maize Molecular Geneticist/Pre- breeder Anna Herremans CIMMYT (former) Director, International Finance Ylva Hillbur IITA DDG for Research Huntington Hobbs CIMMYT Leader, Strategic Planning and Research Coordination, MasAgro Nina Jakobi CRP WHEAT Program Assistant Victor Kommerell CRP WHEAT Program Manager Michael G. Listman CIMMYT Senior Science Writer, Corporate Communications Diana Lopez CIMMYT Project Management Unit Thomas Lumpkin CIMMYT DG Sally Mallari CRP MAIZE Program Assistant Richard Medina CIMMYT Director, Internal Audit Fernando Fernando P Mendoza CIMMYT Senior Internal Auditor Abebe Menkir IITA Team leader for maize improvement research at IITA, Focal Point for CRP at IITA Ivan Ortiz Monasterio CIMMYT Agronomist, Wheat Harvest Coordinator Patricia V Mir CIMMYT Risk Management Analyst Thomas S. Payne CIMMYT CIMMYT Board Secretary Kevin Pixley CIMMYT Program Director, Genetic Resources Program B M Prasanna CIMMYT Program Director Global Maize Program Jens Riis-Jacobsen CIMMYT Director of Int. Systems and Information Technology Nellooli P. Rajasekharan CIMMYT Director, International Human Resources Jose Ramiro T. Mondragon CIMMYT Manager, Financial Planning Geneviève Renard CRP MAIZE and WHEAT Communication Specialis Jean-Marcel Ribaut GCP Director, Generation Challenge Program Horacio Rodriguez CIMMYT MasAgro Extension Coordinator Victor Lopez Saavedra CIMMYT Manager of Institutional Relations TTF-MasAgro Felix SanVicente CIMMYT Breeding lead for the tropics in Mexico Urs Schulthess CIMMYT Crop modeler, Global Conservation Agricultural program Thomas W. Short CIMMYT DDG Support Services 41 Independent iea.cgiar.org Evaluation Arrangement EVALUATION OF CGIAR RESEARCH PROGRAM ON MAIZE, INCEPTION REPORT, OCT 2014 Graham Sim CIMMYT Director, International Finance Matthew Thornton CIMMYT Hub Coordinator Sam Trachsel CIMMYT Scientist, Global Maize Program David Watson CRP MAIZE Program Manager Martha Willcox CIMMYT Senior Scientist working with the Seeds Discovery (SeeD) project 42 Independent iea.cgiar.org Evaluation Arrangement ANNEX 4: SAMPLE PROJECT Code Lead center Title Funding Donor FP Budget 10 largest projects M0193 CIMMYT Drought Tolerant Maize for Africa – Phase III W3 BMGF 3 24,242,177 T0073 CIMMYT Sustainable intensification of maize-legume cropping systems for food Bil ACIAR 1 16,147,130 security in eastern and southern Africa (SIMLESA) (CSE2009/024) M0167 CIMMYT Improved Maize for African Soils (IMAS) Bil BMGF 3 14,541,716 PJ- IITA The Multinational - CGIAR Project: Support to Agricultural Research for Bil AfDB, SARD-SC 1 11,858,250 001371 Development on Strategic Commodities in Africa (SARD-SC) {MAIZE} M0196 CIMMYT Nutritious maize for Ethiopia Bil CIDA 3 11,666,500 M0215 CIMMYT Water Efficient Maize for Africa- Phase II Bil BMGF 3 10,541,990 and M0146 M0212 CIMMYT MASAGRO-Estrategia internacional para aumentar el rendimiento del Bil SAGARPA 4 7,879,393 Maíz C0040 CIMMYT MASAGRO-Desarrollo sustentable con el agricultor Bil SAGARPA 1 7,728,887 PJ- IITA Drought Tolerant Maize for Africa – Phase III Bil CIMMYT 3 6,829,358 001511 M0205 CIMMYT Effective Grain Storage for better livelihoods of African Farmers- Phase II Bil SDC 5 6,823,671 30 randomly selected projects A1111 CIMMYT China Contribution W3 Chinese 0 122,499 Academy of Agricultural Sciences C0037 CIMMYT Cereal Systems Initiative for South Asia (CSISA)-India W3 USAID 1 280,923 C0029 CIMMYT Take it to the Farmer (TTF)-Private sector contributions Bilateral Various 1 90,150 private sector companies C0002.01 CIMMYT 2010 EC/IFAD CGIAR Programme: Sustainable intensification of W3 IFAD 1 1,364,700 smallholder maize-livestock farming systems in hill areas of South Asia (C- ECG-43) IITA Integrated Striga management for Afrcia W1/2 CGIAR 1 854,330 C0045 CIMMYT Colaboración sobre Seguridad Alimentaria Integrado a las Plataformas de Bilateral Syngenta 1 101,250 Investigación e Innovación del CIMMYT Agro M0191 CIMMYT Increased productivity of maize-based systems in Zambia's Eastern Bilateral USAID 1 881,042 Province CIMMYT Enhancing Maize Production and Productivity Through Best Nutrient W1/2 CGIAR 1 200,000 Management Practices in Nepal T0094 CIMMYT Scale out the research results of the SIMLESA to neighbouring countries Bilateral ACIAR 1 102,600 Botswana and Uganda to make an impact in improving food security (C2011/180) CIMMYT Using decision support tools to develop innovative maize-based W1/2 CGIAR 1 200,000 technologies for enhancing crop output in northern Ghana M0185 CIMMYT Genomic Selection: The next frontier for rapid gains in maize and wheat Bilateral Cornell 2 489,653 improvement University R0135 CIMMYT Integrated breeding platform Bilateral Generation 2 178,517 Challenge Program CIMMYT Exploring transgenic approaches for ensuring low-income countries and W1/2 CGIAR 2 750,000 resource poor farmers' access to transgenic options M0224 CIMMYT Managing maize lethal necrosis (MLN) in eastern Africa through Bilateral Syngenta 3 921,782 accelerated development and delivery Foundation for Sustainable Agriculture M0153 CIMMYT Maize breeding for drought tolerance as an option to maintain maize Bilateral World Bank 3 362,385 production and decrease mycotoxin damage in the actual changing climate M0173 CIMMYT Support for Striga work - seed breeding and production and on-farm Bilateral BASF, The 3 82,509 testing Chemical Company CIMMYT Identification and dissemination of farmers' preferred nutritious maize W1/2 CGIAR 3 130,000 varieties suitable for food, feed, silage and fodder in Nepal 44 iea.cgiar.org IITA Various DTMA related activities W1/2 CGIAR 3 892,158 M0207 CIMMYT Improved Maize for African Soils (IMAS) W3 USAID 3 115,320 PJ- IITA Maize lethal necrosis disease: investigating risks and pre-emptive CGIAR CIMMYT 3 116,193 001911 management in West Africa CIMMYT Meeting the rapidly growing poultry sector requirements through value- W1/2 CGIAR 3 375,000 added maize germplasm CIMMYT Increasing yield potential and stress tolerance of QPM varieties for W1/2 CGIAR 3 375,000 southern Africa CIMMYT Increasing yield potential and stress tolerance of QPM varieties for W1/2 CGIAR 3 n/a southern Africa M0194 CIMMYT Improving disease resistance of tropical maize germplasm Bilateral VILMORIN & 3 197,558 CIE M0208 CIMMYT Heat stress resilient maize for South Asia through a public-private W3 USAID 3 806,343 partnership IITA Doubling Maize in Nigeria II Bilateral Nigerian 4 $ Gov 172,000.00 T0097 CIMMYT Identifying socioeconomic constraints to and incentives for faster W3 ACIAR 5 2,064,583 technology adoption: Pathways to sustainable intensification in eastern and southern Africa (FSC/2012/024) PJ- IITA Research Project on Aflatoxin Control in Maize Through Aflatoxin Bilateral NESTLE- 5 50,000 001322 Resistant Maize Variety Breeding And Other Aflatoxin Management SWITZERLA Methods ND M0151 CIMMYT Developing maize resistant to stem borer and storage insect pests for Bilateral Syngenta 5 2,824,950 Eastern and Southern Africa- IRMA III Conventional Foundation T0092 CIMMYT 2011 EC/IFAD CGIAR Programme: Conservation agriculture and W3 IFAD 5 491,716 smallholder farmers in Eastern and Southern Africa-Leveraging institutional innovations and policies for sustainable intensification and food security (CASFESA, 2011/260-204) 45 iea.cgiar.org