SDR/ISC: IAR/02/23 
iSC Working Document 
(Not for public citation)
CONSULTATIVE GROUP ON INTERNATIONAL AGRICULTURAL RESEARCH
INTERIM SCIENCE COUNCIL
Eighty-Third Meeting iSC/TAC, FAO and IPGRI, Rome, Italy, 26-30 August 2002
Report from
iSC’s Standing Panel on Impact Assessment (SPIA)
(Agenda Item 6)
For Discussion: The attached report from SPIA will 
be introduced by its Chair, Dr. Hans Gregersen. iSC 
Members are invited to comment on the report, and 
to provide suggestions on the future directions of 
SPIA’s work.
iSC SECRETARIAT
FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS
August 2002
TABLE OF CONTENTS
Page
1. MANDATE AND COMPOSITION OF SPIA 1
2. CURRENT STATUS OF SPIA ACTIVITIES 1
2.1 Environmental Impact Study 2
2.2 Germplasm Improvement Impact Study 2
2.3 Training Evaluation and Impact Study 2
2.4 Impact of the CGIAR on Poverty Alleviation 9
2.5 Conference on Impacts of Agricultural Research and Development: 11
Why Has Impact Assessment Research Not Made More of a Difference
2.6 Strategic Guidelines for Ex-Post IA in the System 11
2.7 CGIAR Benefit - Cost Meta-Analysis 12
2.8 African Evaluation Association Conference in Nairobi, Kenya, 10-14 June 13 
2002
2.9 CGIAR Impact in Africa Follow-up Study 14
2.10 CGIAR Impacts Website and Database Development 15
3. FUTURE OF SPIA & IMPACT ASSESSMENT ACTIVITY IN THE 15 
CGIAR
3.1 Context 15
3.2 Specific IA Needs of the System 16
3.3 The Future Impact Assessment Needs in the CGIAR 17
3.4 Concluding Comment 17
ANNEX 1 - Table of contents for the Proceedings of the Int’I Conf. on 
Impacts of Agricultural Research and Development
ANNEX 2 - Strategic Guidelines for Conducting Ex-Post IA in the 
CGIAR
ANNEX 3 - Impact Studies for Review in the CGIAR B-C Meta 
Analysis
ANNEX 4 - Sample Database Record for Review of B-C Studies 
ANNEX 5 - Agenda for the Agricultural Research Strand of AfEA
S' "
iSC STANDING PANEL ON IMPACT ASSESSMENT (SPIA)
Report to iSC 83
1. Mandate and Composition of SPIA
The current mandate of the Standing Panel on Impact Assessment (SPIA) is threefold, 
namely to:
• provide Members with timely, objective and credible information on the impacts at the 
System level of past CGIAR outputs in terms of the CGIAR goals;
• provide support to and complement the centres in their ex post impact assessment 
activities; (this includes facilitating inter-centre impact assessment efforts and providing a 
forum for exchange of experience from impact studies) and,
• provide feedback to CGIAR priority setting, and create synergies by developing links to 
ex ante assessment and overall planning and evaluation functions in the CGIAR.
The role and structure of SPIA, and indeed of the Science Council (SC) itself, may change as 
roles and responsibilities are re-defined or clarified subject to the report of the SC Working 
Group recommendations to the Group. (The Standing Panel’s own view on the future IA 
needs of the System is described in Section 3 of this report). Drs. Ruben Echeverria 
(Uruguay) and Hermann Waibel (Germany) remain the ’acting’ Members of the Standing 
Panel (not yet confirmed by the Co-sponsors). Alain de Janvry (France) and Elias Fereres 
(Spain) are ex-officio members in their capacity as Chairs of SCOPAS/iSC and SCOER/iSC 
respectively. The Chair is Hans Gregersen (USA) who also serves as an ex officio member 
of the iSC. Tim Kelley is the person assigned to SPIA from the iSC Secretariat. Members 
are chosen for their independence and impact assessment expertise and familiarity with 
international agricultural research.
2. Current Status of SPIA Activities
SPIA as a unit of the interim Science Council (iSC) will cease to exist at the end of 2002. 
Given the importance that CGIAR members give to assessment of impacts of their 
investments in the context of the mutual goals of sustainable poverty alleviation and food 
security, it is assumed that a new Systemwide impact assessment unit will be established in 
association with the establishment of the new Science Council, What follows are
descriptions of on-going and agreed upon activities being undertaken by SPIA.
All of the below mentioned activities were discussed in SPIA’s report to AGM01. Some 
have been widely discussed by the members. In several cases, SPIA is in the process of 
completing the activities and publishing final reports (e.g., germplasm enhancement impacts 
and environmental impacts assessment). The IFPRI led poverty impacts study also has been 
ongoing for some time, and significant progress has been reported at several meetings of the 
Group. It still has significant additional activity planned. SPIA recommends that the new 
Science Council give highest priority to bringing this activity to a successful completion.
2
In most of the cases reported below, activities were started during late 2001 or the first part of 
2002 (e.g., the training impact assessment, the development of strategic guidelines for 
CGIAR impact assessment studies, the meta cost benefit analysis of the aggregate investment 
in the CGIAR, the follow up study for Africa, and the CGIAR impact assessment website). 
SPIA suggests that all these activities are important ones in terms of moving CGIAR 
understanding of Systemwide impacts ahead and in terms of providing the Science Council, 
Executive Council and the System Office insights to use in developing plans for the future of 
the CGIAR. Thus, SPIA recommends that all of these activities be given high priority in 
the development of a plan of action for SPIA’s successor entity.
2.1. Environmental Impact Study
A report by prepared by Mywish Maredia and Prabhu Pingali on the negative impacts of 
productivity enhancing research entitled "Environmental Impacts of Productivity-Enhancing 
Crop Research: A Critical Review" was published and distributed earlier in the year. The 
main conclusions of these reports were highlighted in the SPIA Report to the Group at AGM 
01. The authors concluded that it is not possible to develop aggregate quantitative estimates 
of negative environmental impacts—the compounding factors accounting for the linkages 
between research and environmental impacts are too complex to sort out given present 
analytical methods and available data. Evidence of negative environmental impacts has only 
been presented in the literature for a few GR crops, e.g., for rice (from pesticide use) and for 
wheat (from fertilizer/irrigation problems), and these are often associated with other causes, 
such as institutional or policy failure. Furthermore, there is, to date, little evidence of 
environmental damage resulting from other CGIAR mandate crops.
The other main report from this study, "Environmental Impacts of the CGIAR: An 
Assessment" by Michael Nelson and Mywish Maredia has been extensively revised to take 
into account comments on data and methodology used in the earlier version. While some of 
the quantitative results have changed, i.e. are more conservative, the main findings have not 
changed: Contributions by the CGIAR in the areas of germplasm enhancement and 
agronomy have permitted significant yield increases in farmers’ fields, thus leading to less 
land being required to produce a given quantity of food crops. The Panel report is expected 
to go to green cover after a final review by SPIA members.
2.2 Germplasm Improvement Impact Study
Although a final report on this project was presented at MTM 01, subsequently a number of 
major revisions were undertaken by the authors in response to a critical external peer review 
by three referees seeking further clarification on the methodology and interpretation of 
results. The revisions are complete and CABI is now in the process of finalizing proofs. The 
23-chapter book containing the main elements of this SPIA activity which documents the 
impact of CGIAR and NARS crop germplasm improvement will be published in late 2002.
2.3. Training Evaluation and Impact Assessment
Background
In TAC79, TAC commissioned an evaluation and assessment of capacity strengthening 
activities in the CGIAR. One part of this review is a study focusing specifically on CGIAR 
training. The CGIAR members and Centre representatives reacted positively to this idea, 
considering it both important and timely. It would provide a means to develop concepts, 
methods and data to use in the assessment of the other components of the overall set of
■v " 3
capacity strengthening activities in which the CGIAR is engaged. Since TAC79, TAC/iSC 
has continued to endorse the need for this activity, which will include a rapid appraisal type 
of stripe study on the current state-of-the art of training activities in the Centres and a 
research based assessment of actual and potential impacts. The implementation of this study 
has been the joint responsibility of SCOER (Standing Panel on External Reviews) and SPIA.
The overall purpose of the training study is to evaluate the effectiveness of the processes, 
outputs and impact of the CGIAR training activities for the NARS and identify how the 
activities could be improved. The study’s broad objectives are to: (1) evaluate the quality 
and relevance of the training activities within the CGIAR, (2) evaluate the comparative 
effectiveness and efficiency of CGIAR training activities, (3) assess the intermediate outputs 
and impacts of training, and, to the extent possible, (4) assess the impacts of CGIAR training 
in the context of the CGIAR ultimate goals of poverty reduction and food security through 
sustainable production.
Given uncertainty about the available budget, TAC decided at TAC 81 to concentrate first on 
a desk study to gather, collate and analyze available information and data on Centre training. 
At TAC/ÌSC82, the iSC reinforced the decision to implement the main study with primary 
focus on selected NARS. However, the transition of TAC into a Science Council has delayed 
a commitment of resources to undertake the main study. Implementation of the main phase, 
which will involve field work and original data generation, hinges critically on the 
availability of adequate funds within the framework of the future Science Council. 
Ultimately, this evaluation will provide recommendations on ways to make the impacts from 
System training activities more relevant and sustainable in terms of CGIAR goals,
Rationale
The current context of alternative training providers, new modes of channelling capacity 
strengthening activities, and declining funding is forcing the CGIAR Centres to prioritise and 
re-design their capacity strengthening strategies. The training study is expected to provide 
information useful to the CGIAR and the Centres in setting relative priorities regarding 
training focus, identifying effective strategies for CGIAR training activities at the System 
level, and enhancing coordination of training as part of other capacity strengthening 
activities. In particular, the study aims to help: 1) the Centres improve the integration and 
effectiveness of their training activities; 2) the Donors develop justifications for support to 
the training programmes; 3) the NARS assume increasing responsibility in capacity 
strengthening and training appropriate for their own needs; and 4) the System, e.g., through 
the Science Council and other bodies, in monitoring and evaluating (self and external) of the 
training activities aimed at strengthening NARS capacities. The study also plans to inform 
the development of training in the context of the new operational model that the CGIAR is in 
the process of implementing, including particularly the cross-centre challenge programmes.
Desk Study
SCOER and SPIA have worked in close collaboration with the centre focal persons 
nominated by the DGs for the desk study. The study has four components as follows:
1. An analysis of background information from external reviews and other documents on 
important issues related to training and capacity strengthening. This analysis will produce a 
description of what is already known about the training programmes and strategies of CGIAR 
training activities.
4
2. A compilation of information and data on Centre training activities. For this 
component, each Centre was asked for a description of its training programme and annual 
data for the period 1990-2000 on all training activities (type, theme, length etc., including, to 
the extent possible, funding) and on the participants of each training event (country, sex, age, 
institution, status, etc.). The data are being analysed for each individual Centre in order to 
describe trends in strategies, thematic and operational focus, funding and output in the long­
term and for formulating hypotheses regarding alternative strategies and modalities. Then, 
because the data provided varies in form and content, a qualitative analysis will be done 
across the Centres to describe any overarching trends, themes, and issues. These two 
components of the desk study are in process and nearing completion.
3. Formulation of a conceptual model or framework showing the anticipated links 
between training processes and training impacts. Drawing on summaries prepared for the 
first two components, this component has progressed as far as an initial draft of the 
conceptual model. (See Table 1)
4. A Delphi survey among selected stakeholder in the developing countries to identify 
the crucial issues to be covered in the design of the main study and to facilitate NARS 
contributions and involvement in the main study’s field work. This component is still in the 
planning phases.
During 2002, Dr. Leslie Cooksy of the University of Delaware has assisted SCOER and SPIA 
in conducting the desk study and preparing for the main study. Her responsibilities have been 
to assist the iSC Secretariat in organising and analysing the data and information. In addition 
she is providing advice for deciding on the focus, scope and methodology for the study main 
phase and assisting in defining the ToR for the study Panel.
*
5
Table 1: Generic Model of CGIAR Training1
Inputs Activities Outputs Short-term outcomes Intermediate outcomes Long-term outcomes2
■ Results of Identify priority topics List of training priorities CGIAR training is relevant. ■ Policy makers, 
training 
Select appropriate training delivery tools and Plans for addressing priority topics managers and project needs leaders have a 
assessment strategies for topic and audience (appropriate 
based on training needs, cost, integration of comprehensive vision 
■ Skilled local knowledge, etc.) of the role of science 
trainers and technology in 
■ Funds Develop quality training materials # and type of training materials Well-designed training materials are available NARS have an increased capacity to train their agricultural developed to CGIAR partners and clients. own staff. development
Deliver training or support its delivery by ■ # of training events by type of ■ Trainees increase their knowledge and “ New knowledge and skills of trainees are " The sustainability of the 
another institution event (group, individual, etc.) develop new skills. transmitted to trainee colleagues and clients development of the 
■ # of participants/training event ■ The numbers of national scientists with (multiplier effect). agricultural sector is 
■ # of men and # of postgraduate research qualifications are ■ NARS develop and implement 
increased.
relevant and 
women/training event increased. up-to-date research programs. ■ A network of current 
■ The numbers of specialists in the use of   and future partners ■ # of participants/nationality/ ■ NARS and other development partners 
training event scientific methods and techniques are increase their capacity to acquire, apply, 
throughout the 
developing world is 
increased. access and further develop knowledge, skills, 
Participate in networks, consortia, and ■ # of networks, consortia, and established.
    ■ The exchange of information, experiences, technologies and policiesregional programmes (facilitates multiplier regional programmes 
effect) participated in by Centre staff and strategies among course participants, a R&D partners have increased capacity for and 
including course leaders (Centre staff), is interest in partnership.
■ # of Centre staff participating increased.
in networks, consortia, and/or “ R&D partnerships/collaborations increase.
regional programmes ■ Links between and among NARS scientists     ■ R&D partnerships/collaborations produce and Centre researchers are established. improved technologies more quickly.
■ Collaborative networks among countries - 
      ■ Interdisciplinary work among NARS both formal and informal - are facilitated. researchers and between NARS researchers 
■ Research-extension-user linkages are and their research partners increases.
developed or strengthened. • Community-level adaptation of improved 
technologies increases.
Support educational institutions, including # and type of educational Educational institutions adopt changes in Educational policies support the incorporation of 
primary and secondary schools, in institutions supported curriculum. appropriate technologies and natural resource 
incorporating relevant information in the management in educational activities.
curriculum
Support training courses organized by Amount and type of support to Network of training institutions to build The effectiveness of training (as measured by the 
collaborating institutions collaborating institutions synergistic linkages, increase awareness about outcomes identified above) increases.
the supply and demand for training, and share 
training and research materials
Evaluate the implementation and impact of Evaluation results ■ Improved teaching skills among Centre 
training materials, event and/or strategy trainers
■ Improved quality and relevance of training
Change training event or strategy based on Documented change in training or 
evaluation information, if change is overall training strategy
warranted
1 Source: Centre descriptions of training activities and programs.
2 The long-term outcomes support the overall goals of CGIAR to alleviate hunger and poverty and improve natural resource management.
6
Main Study
The information gathered in the desk study will provide the platform for the design of the 
main study. Specifically, the desk study will inform the selection of the strongest and most 
feasible design from among the alternatives. In addition, it will help identify promising 
regions for local interviewing, provide local contacts among trainees and their institutions, 
and prepare panel and advisory group rosters and processes.
A panel will conduct the main study. The current thinking is that there would be a small, 
three-member panel of distinguished individuals, with expertise on training, evaluation and 
training/education impact assessment. Eight to ten resource persons in the Regions, selected 
for their regional expertise and standing would form an advisory group to the panel. These 
resources persons could also oversee the regional interviewing and data gathering activities in 
each of 8-10 regions, although in the interest of uniformity and comparability among regions, 
it will be highly desirable to have at least one member of a central team involved in the actual 
interviewing in all regions. Combining the professional excellence and coverage of the 
crucial areas of expertise with regional expertise and familiarity will guarantee a credible 
final report. This suggested operational model would also keep the panel activities more
manageable and flexible, and increase the cost-efficiency of the study.
The iSC Secretariat is proposing a research design for the main study that would use two 
complementary approaches. Table 2 provides an overview of the kinds of questions that each 
approach addresses and likely methodologies and Table 3 presents an overview of the 
strengths and weaknesses of each approach. One of the approaches would use NARS or 
NARS organizations as the unit of analysis. This piece of the study would provide 
information on 1) how CGIAR training activities compare to other sources of training used 
by the NARS, 2) the causal pathways between CGIAR training and NARS effectiveness and 
impact on the poor and their environments, and 3) the cross-cutting constraints and other 
issues related to the effects of Centre training in NARS. The second approach would use
specific training activities or events as the unit of analysis. The desk study will provide 
information about the types of training activities that are most prevalent and important. The 
panel would use that information to first decide what types of training should be studied and 
then select specific training activities or events of each type to be included in the study. In 
combination, the two approaches will yield information that can be used to establish funding 
priorities for training, improve training activities, develop training partnerships with other 
organizations that provide training in NARS, and leverage comparative advantages of the 
Centres across regions and types of training activities.
7
Table 2. Main Study Questions and Methods
Starting Point Questions Answered Likely Methodology
NARS What sources of training are used by NARS? Relative to the other sources, - Survey of employees in selected NARS about 
what is the nature and extent of NARS training provided by Centres? the kinds and sources of training they have had 
What are the perceptions of NARS staff about the effectiveness of CGI AR in the past ## of years [number to be determined 
training relative to other sources of training? later]
If a large proportion of the training received by NARS employees comes from - Individual or small group interviews about the 
CGIAR and there is a perception that CGIAR training is effective, what linkage between CGIAR training and NARS 
connections can NARS employees make between their CGIAR training and the outcomes
effects of their work?
Training Activities What are the strengths and weaknesses of specific training activities? Survey of trainers and trainees involved in a 
selected set of training activities
What do the evaluation data on each activity suggest about the training process Analysis of secondary data
and its likely effects?
What are trainee perceptions of the effects of the training on (a) their career, - Survey of trainees. This could be expanded to 
(b) their NARS, (c) alleviation of poverty, and (d) natural resource include a survey of trainee supervisors as well.
management?
8
Table 3. Strengths and Weaknesses of Main Study Approaches
Starting Point Strengths Weaknesses
NARS Utility: Information about the quality of CGIAR training relative to Resources: Because this focus would involve individual or small 
other sources of training could be used to decide how to allocate group interviews, it would need more time for data collection and 
training funds. analysis.
Utility: Information about the other sources of training could be used Generalizabilitv/Utlitv: The findings will be limited to the training 
to develop training partnerships or to emphasize the kind of training provided to the NARS selected for the study, unless the NARS are 
in which the Centres working with the selected NARS seem to have a selected for diversity on various characteristics and we found a 
competitive edge. similar pattern of results across different NARS.
Attribution: Inferences about the effects of CGIAR training for the 
selected NARS will probably have a stronger foundation than 
inferences based on activity-focused survey data. First, by looking at 
CGIAR training in the context of all the related training received by a 
NARS, we can make a more sophisticated assessment for the portion 
of credit due to the Centres for any observed effects. Second, the 
individual interviews should provide more detailed information that 
would enable tracking the causal linkages from training through 
NARS activities to post-NARS outcomes.
Training Activities Feasibility: The design is straightforward and fairly easy to Generalizabilitv/Utilitv: The sample would be limited to a small set 
implement. of types of activities and to a small set of specific training events 
within each activity type. As a result, recommendations for 
Utility: The results for each of the selected training events are likely improvements may not apply to similar training activities in other 
to be useful to the Centre that provided the training. Centres or in the same Centre at different times.
Utility: Perhaps interesting comparisons could be established. For Attribution/Utilitv: With survey data as the primary source of 
example, if one set of trainees participated in more than one kind of information, the results are likely to be limited to initial outcomes 
training, we could compare the trainees’ perceptions of effectiveness only.
across the different kinds of training. If we focused on no more than 
2 or 3 types of training, we could compare different instances of the 
same kind of training in different regions. Such comparisons could 
inform decisions about how to match trainees with appropriate 
training activities or about the relative challenges to a specific kind of 
training across regions.
9
2.4. Impact of the CGIAR on Poverty Alleviation
To determine how the poor have benefited from agricultural research, CGIAR centres need a 
stronger capacity to undertake poverty impact assessments on a continuing basis, not only to 
identify the conditions under which agricultural research is a sound investment for reducing 
poverty, but also to improve the targeting of research priorities to the changing needs of the 
poor. In 1998, the Impact Assessment and Evaluation Group (now SPIA) requested EFPRI to 
develop and coordinate a system-wide project to strengthen capacity for such poverty 
assessments.
Background
The first phase of this two-phase project, completed in 1999, involved a review and synthesis 
of the literature on the links between agricultural research and poverty, and a workshop to 
develop methodologies for further CGIAR impact studies. The second phase, which began in 
September 2000, planned 14 case studies to cover a wide range of countries and types of 
CGIAR research. The first wave of seven studies, launched in 2000, provides partial 
coverage of CGIAR centres and types of research (e.g., in terms of commodity and regional 
coverage and scale of impact). The remaining seven studies—intended to target other 
CGIAR centres and types of research—were put on hold due to lack of funding.
These studies have two main objectives: (1) to test empirically methods for evaluating the 
impact of agricultural research on poverty in the context of different agricultural technologies 
and within different country, social, and institutional settings; and (2) to develop a conceptual 
framework that CGIAR centres can draw upon for impact assessment work, and that will also 
serve to guide priority-setting and technology design to increase the impacts on poverty. To 
accomplish these objectives, five of the first seven case studies used the sustainable 
livelihoods conceptual framework. Key aspects of this framework include
• expanded understanding of the dimensions of poverty and how to measure it;
• emphasis on vulnerability to natural phenomenon, market shocks and trends, and social 
conflict, and how this shapes livelihood strategies and choices in technology;
• examination of physical, natural, financial, human, and social capital assets (and the 
constraints on access to assets) and how people combine these in their livelihood 
strategies;
• study of the varied range of livelihood activities and strategies that people pursue, 
recognizing that many different activities are pursued simultaneously, including on- and 
off- farm work;
• understanding how the institutional environment at the micro and macro levels influences 
livelihood strategies and outcomes and impacts of interventions;
• looking beyond aggregated household or head counts to consider the significance of 
social differentiation by class, ethnic group, gender, and other factors.
Each case study focuses on a set of research questions driven by the nature of the technology 
under study and its context. All questions, however, are informed by the sustainable 
livelihoods conceptual framework and fall within a set of themes that cuts across the studies. 
The sustainable livelihoods framework was used to structure the analysis, synthesize the 
information gathered, and make comparisons across case studies (see Table).
10
Phase 1, Wave 1 case studies of impact of agricultural research under the IFPRI/SPIA project
Country______Technology________________Case study leader Lead CGIAR centre_________________
Bangladesh Modern rice varieties Mahabub Hussein IRRI 
Bangladesh Polyculture fishponds Kelly Hallman 1FPRI
Improved vegetables 
Modern rice varieties
Kenya Soil Fertility Replenishment Frank Place ICRAF
Zimbabwe Modern maize varieties John Hoddinott IFPRI
Mexico Creolized maize varieties Mauricio Bellon CIMMYT
China Agr. research investments* Shenggen Fan IFPRI
India Agr. research investments* Shenggen Fan IFPRI
* Uses econometric analysis of secondary data rather than sustainable livelihoods approach with integrated social and 
economic impact assessment 
The project is managed by IFPRI. An international Social Analysis Team includes
researchers from EFPRI, the London School of Economics, and Wageningen University. 
Each case study is led by a senior researcher (usually an economist) at the respective CGIAR 
centre, who works with senior socials scientists (economists and sociologists) from national 
research institutes or universities and a team of less experienced social scientists for the 
purpose of capacity development. An External Advisory Committee meets once a year.
Progress to-date
The seven case studies of Wave 1 are nearing completion. A workshop was held at IFPRI 
headquarters in Washington DC May 28-30, 2002 to bring together all the case study leaders 
and social analysis team members, and the Advisory Committee. Draft papers from all case 
studies using the Sustainable Livelihoods framework were presented and discussed in detail. 
The discussion generated preliminary findings for the synthesis report and implications for 
the CGIAR.
The initial plan called for presentation of the findings of Wave 1 at Centres’ Week. 
However, with the shortening of the Annual General Meeting, it has become more difficult to 
obtain time at the AGM and to present substantive findings. Instead, a major outreach 
workshop is being planned for the end of Wave I in Spring 2003, to present both the 
individual case studies and synthesis to key leadership of the CGIAR centres, donor 
organizations, NARs, and a wider public.
Many of the case studies were presented at the SPIA/CIMMYT workshop on impact 
assessment in Costa Rica in February, 2002. Based on the discussions at that workshop, this 
project has begun to consider how to incorporate Institutional Learning and Change (ILAC) 
in the ongoing studies, as well as how to expand this ILAC in future impact studies. This 
implies a shift from externally oriented impact assessment to evaluation that involves CG 
centres and NARs, including the biophysical scientists, much more throughout the process, so 
that the learning is incorporated in the priorities and conduct of further agricultural research 
by those institutes.
Future Work
Initially, it was planned to add a second set of case studies—to diversify the regions and 
technologies examined and to expand the number of CGIAR centres involved in this project. 
Reviewing experience with Wave 1, EFPRI has decided that it will not coordinate a second 
wave of case studies, but will work with interested centres to develop their own case studies, 
and convene a workshop to develop the concepts and methods for addressing ILAC.
11
Discussions are underway with IPGRI, CIAT, ICRISAT, CIMMYT, and ICARDA for such 
follow-up work.
2.5. Conference on Impacts of Agricultural Research and Development: Why 
Has Impact Assessment Research Not Made More of a Difference?
The main outcomes of this SPIA/iSC and CIMMYT sponsored conference, held in San Jose 
Costa Rica in February 2002, are reported in a Proceedings document which is currently 
being finalized for publication. Participants included IA practitioners from 15 Future Harvest 
Centres, as well as representatives from NARS, public and private universities, multilateral 
lending organizations, development assistance agencies, NGOs, philanthropic foundations, 
private corporations, and the media. With over 145 people attending, this conference was the 
largest gathering in CGIAR history of the international agricultural research IA community. 
The 4-day conference provided an opportunity for IA professionals to update their knowledge 
and skills in relation to both conceptual and empirical approaches to impact assessment while 
engaging in extensive discussion and networking. The program included invited and 
contributed papers sessions, panel presentations, free discussion sessions, and poster sessions 
(see table of contents in Annex 1).
Participants highlighted experiences and case studies of impact measurement in the following 
areas: agricultural productivity; equity, poverty, social health, and nutrition; the environment; 
and, institutions and human capital. Participants also described novel approaches to hard-to- 
measure impacts in such areas as: training and capacity-building; institutional strengthening; 
networking; participatory research; and policy research.
In addition to the Proceedings volume, the Quarterly Journal of International Agriculture will 
be devoting one entire issue to a set of papers presented at the Conference. These are being 
edited by a steering committee comprised of CIMMYT and SPIA members. In addition, the 
SPIA Chair and Secretary are contributing to an introductory article for that issue. Other 
journals have been contacted to publish selected papers as appropriate.
2.6 Strategic Guidelines for I A in the System
The need for establishing strategic guidelines for IA studies in the CGIAR has been re­
inforced at the last two major CGIAR sponsored IA conferences. The strategic guidelines 
will not be a detailed step-wise Tow to’ manual for carrying out IAs, but rather a set of basic 
principles and discussion of strategic issues (including user needs) for IAs in the System. 
The document would cover issues that help link what users of IAs need (donors, planners, 
administrators) with what doers of IAs can do, given resource, and time and data constraints. 
It would explore basic issues such as the criterion of plausibility in IAs, attribution, 
development of counterfactuals, logframe and impact pathways analysis generally, and issues 
related to credibility, feasibility, transparency, and communication. Donors are keenly 
supportive of developing this set of guidelines, since they that such a document would be 
helpful to them in establishing internal guidelines for judging IAs and explaining them to 
funding and political bodies.
Since the last iSC meeting, SPIA has drafted a preliminary annotated outline for these 
Guidelines. This draft was subsequently revised following a number of helpful comments 
and suggestions from various individuals, including iSC members, and some interested 
donors (see Annex 2). In particular, EIARD members and USAID are quite interested and 
supportive of this work and will be close partners in developing these guidelines. The plan is
12
to bring on a consultant for 4-6 weeks to draft and help finalize guidelines, in collaboration 
with SPIA members and a range of other stakeholders. Centres will be involved centrally in 
this activity throughout the process. The major output from this activity will be a set of 
principles and Test practices’ strategies to guide ex-post impact assessment work done by the 
CGIAR and its centres.
2.7. CGIAR Benefit - Cost (B-C) Meta-analysis
Since its establishment in 1972, the CGIAR community has invested some US $ 6.6 billion 
(2001 inclusive) in current value terms in its various research and related activities. In an era 
characterised by “donor fatigue” and scarce development resources, it is very relevant to ask: 
Do the past and expected future benefits from CGIAR research justify the total investment in 
the CGIAR so far? Although the CGIAR system has been a world leader in documenting 
research impacts, no previous study has attempted to comprehensively answer this question.
Background
The present analysis is intended to resolve on a preliminary basis whether the entire 
investment in the CGIAR over time can be justified on the basis of the benefits derived from 
its proven (and agreed-upon) major successes. If such is indeed the case, and with the 
assumption that the other research produced by the System has not caused major negative 
impacts, , then the overall investment in the System can be justified in an aggregate sense. It 
is recognized that the result will underestimate the other non-measured positive benefits, and 
as such represent a conservative estimate. This approach avoids the criticism that impact 
assessments on the whole generally tend to look only at projects already known to be 
successful, often using the results to argue the case for a much larger program of investment. 
This type of aggregate benefit and cost analysis has been used successfully in the past for 
justifying overall research budgets for agencies and and other entities.
To derive cumulative benefit values, the analysis will aggregate the most plausible benefit 
values from available, credible economic impact assessments for known successes. A list of 
studies to be included is provided in Annex 3 of this report. Each study will be rigorously 
reviewed, with key assumptions recorded in a detailed database. All major unsubstantiated 
claims and values utilised in the included analyses will be scrutinised, and compared against 
other results published in the peer-reviewed literature, if possible, so as to ensure the 
credibility of cumulative values. When necessary, included benefit values will be adjusted, 
so as to compensate for disputable assumptions. Partitioning of benefit attribution will be 
attempted, to the most plausible degree possible, on the basis of several different criteria, 
depending upon the type of benefit investigated. Derived benefit values and aggregate 
CGIAR investment costs will be discounted using a range of plausible discount rates to bring 
all values to a common point in time. Since reliable economic impact data are only presently 
available for a very small proportion of the CGIAR’s major impacts, the results of this 
analysis must be considered as preliminary. A second more detailed and time-intensive study 
is tentatively planned for early next year to follow up on these preliminary results.
Progress to Date
A graduate student from the Royal Veterinary and Agricultural University in Copenhagen is 
working with the iSC Secretariat on a short term basis to move forward with the initial 
analysis, under the guidance of SPIA. Thus far he has: surveyed the literature for studies to 
be included in the aggregate analysis through discussions with CGIAR scientists, SPIA, 
extensive searches in relevant academic databases and reviews of reference lists from
14
A variety of different evaluation approaches were employed in the studies including:
• Impact assessment
• Effectiveness evaluation
• Planning processes
• Monitoring
• Cost efficiency/Economic analysis
All papers from the strand are available on the conference web site. The three papers shown 
below will be summarized and included in the conference proceedings.
• Principles of evaluation: Methodological lessons derived from East and Central Africa, in 
the context of ASARECA by Adiel Mbabu and G. Ebong.
• Evaluating the impact of agricultural research and development: Future Harvest Research 
Centre Approaches by Patti Kristjanson et al.
• Assessment of the impact of agro-forestry based soil fertility replenishment interventions 
on the poor in Western Kenya by Mary Omosa.
2.9 CGIAR Impact in Africa Study follow-up
The major output from this activity will be an updating and extension of the work presented 
at MTM rii seeking to document the improved technology and policy impacts of CGIAR and 
partner agricultural research in sub-Saharan Africa. The analysis will rely on two sets of
information and data: field-level impact evidence from case studies and more general 
CGIAR centre and System level assessments. Another key output, derived from the field 
study component, will be a longitudinal dataset over an extended number of years. This will 
be fundamental in capturing a better understanding of the linkages between agricultural 
research and poverty alleviation.
Hence, this initiative builds on the initial assessment presented at MTM rii to develop a more 
systematic and comprehensive assessment of the impacts of the CGIAR and its partners in 
achieving the goals of reducing poverty, hunger and malnutrition in Africa. A consultant will 
be hired by SPIA to work closely with the Centres in compiling and synthesizing the available 
evidence of CGIAR research impacts in Africa.
In addition to completing the more comprehensive desk study, SPIA has been invited to 
become involved in a new initiative to assess impacts of five or six CGIAR centres in about 
eight specific locations/projects in Africa. The major focus of this initiative is on community 
level processes of who adopts, where and why, and improved understanding at the household 
level of what impacts are being generated by new technologies. Major funding for 
completing the first round of detailed baseline surveys and preliminary and final workshops 
comes from the Japanese Foundation for Advanced Studies (FASED), but additional support
for operational expenses and the second round of surveys is required. SPIA has been asked 
by the lead centre to contribute to the latter effort in the planning and design phases and, to 
the extent possible, in providing some financial support. Initial meetings have already been 
held between ILRI, CIP, ICRAF, CIMMYT and ICRISAT and over the course of the next six 
months specific project objectives, methods and sites will be identified. SPIA has already 
provided some input into the planning and overall conceptual framework.
■s'* 15
2.10 CGIAR Impact Website and Database Development
This activity, which had been in the advanced planning stages (see previous SPIA Report to 
TAC/iSC 82), has been put on hold pending clarification of the activities, budget and modus 
operandi of SPIA after 2002.
3. Future of SPIA and Impact Assessment Activity in the CGIAR
3.1 Context
The CGIAR members and the Cosponsors decided at MTM 99 in Beijing that the 
systemwide IA function (previously the IAEG) should be integrated with the work of TAC in 
order to gain efficiency and take advantage of the synergies with the System’s forward 
planning and its monitoring and evaluation functions, both of which were housed in TAC. 
This view was confirmed by the 2001 SC working group in its recommendations on the SC.
The iSC agreed that the three functions of (i) forward planning, (ii) monitoring and 
evaluation (M&E), and (iii) ex-post IA should be closely associated so that each can build on 
synergies and complementarities with the others. At the same time, the iSC was sensitive to 
the wish of the Group that the IA function should retain its independence and transparency 
and, hence, credibility.
Moving ahead to optimize the contributions of science to achieving the goals of the System 
requires an iterative process of successive approximations as new results emerge, as new 
science evolves, and as the issues are understood better. In this process, planning requires 
learning from the present progress (through M&E) and from the impacts of past activity. At 
the same time, evaluation of the relative effectiveness of on-going activity, and assessment of 
the impacts of past application of science requires knowledge of what has happened, what is 
happening now, and what likely will and should happen in the future, i.e., the context. Thus, 
SPIA and the iSC believe that close and regular linkages between the three functions are 
essential.
The System, in its systemwide IA activity through SPIA as part of the iSC, has focused so far 
mostly on major Systemwide impact assessments. Thus, there have been assessments of the 
System’s germplasm improvement impacts (Evenson et al. report), environmental impacts 
(Nelson and Maredia and Maredia and Pingali reports), and IPM activities (Waibel report). 
On-going SPIA/iSC assessments are focusing on the CGIAR’s impacts on poverty alleviation, 
the impacts of the System’s capacity strengthening activities, and the overall relationship 
between the entire System’s costs and its impacts on or benefits to society. In addition, there 
have been various information and support functions carried out over the past years, including 
several workshops for centres to consider where the system and its centres should be going in 
the field of IA, and an international conference, bringing together CGIAR investors and IA 
users with IA specialists from the centres and from outside the System.
16
3.2 Specific IA Needs of the System
Within this broader context of CGIAR forward planning, monitoring and evaluation of on­
going programs, and accountability to investors for past use of their resources, the iSC has on 
several occasions confirmed its belief that the System still needs:
(a) evidence of the impacts of its various completed and on-going Systemwide research 
and related activities,
(b) impact information useful in understanding appropriate and desirable changes in 
direction of the System’s programs, and
(c) mechanisms for strengthening the capacity to do impact assessment in the System’s 
centres.
(d) to advance a culture of impact assessment in the Centres and their related NARS
Given this confirmation of the role of impact assessment at the System level, SPIA concludes 
that there is a continuing need for four main ex post impact assessment functions at the 
System level within the CGIAR:
(1) Conducting high quality, independent impact assessments to provide results useful to 
(a) investors, in justifying their investments; and (b) System management and centres in 
planning their programs and investments and developing and allocating budgets. 
(Independence here refers to being done by individuals not associated with the research 
being assessed and having no conflicts of interest that could affect the assessment);
(2) Tracking information related to the impacts associated with centre and cross centre 
activities. This could involve routine data collection; and developing, maintaining and 
managing, in collaboration with the planning and monitoring and evaluation units, an 
appropriate data base/MIS for the System that would provide annual updates on 
accomplishments (training, research, etc.) in addition to data on other indicators of 
relevance in understanding the impacts of the System’s outputs and processes.
(3) Devloping methodologies, providing training in their use, and providing advice and 
facilitation for centres as needed, e.g., in terms of setting up programs and projects in 
such a way as to make tracking and analyzing impacts more feasible, transparent and and 
of high quality.3 This would include the “certification” of quality of internal IAs and 
organizing and “certifying” quality of external IAs. Ideally, this function would involve 
the establishment and maintenance of a CGIAR wide IA web site that also would be open 
to all outside entities with an interest and involvement in IA related to agricultural 
research and training.
(4) Delivering and facilitating the most effective use of the outputs of the IA entity, e.g., 
facilitating centre interaction and learning, and developing an effective impact culture in 
the centres. It also would involve providing insights to investors on what is and is not 
feasible in terms of carrying out IAs for such activities as natural resources management, 
social science research and capacity strengthening.
3 It should be stressed that the implication of this statement is not that the centres are lacking in high quality 
impact assessment capacity. Rather, the thinking here is that a central entity can facilitate interaction among 
centres, gain access with System level resources to expertise needed by all centres, and provide a clearing 
house for information and documentation of use to all centres. In a sense, this central entity will provide 
“System level public goods.”
17
3.3 The Future Impact Assessment Needs in the CGIAR
Based on a review of past Systemwide IA activity and experience, it seems to SPIA that its 
successor, will need to target five key areas:
(1) program evaluation that involves inter-centre cooperation and collaboration;
(2) work more closely with the System’s science monitoring and evaluation activities to 
ensure that the complementarities between IA and M&E are fully realized;
(3) provide ex post IA outputs that can serve the purposes of planning the new 
Challenge Programmes and reorienting overall System plans, i.e., work more closely 
with the forward looking, system level planning activities;
(4) open up more broadly to the IA world outside the CGIAR, through networking, a 
web site, outsourcing and putting some future assessments out for bid on a broader 
“request for proposal” basis, and,
(5) set standards and help develop a more systematic process to assure high quality, 
independent scientific peer review of the analytical ex post IA studies produced.
Within this broader context of reorientation of IA activities within the SC, the following 
activities (not in any order of priority) might productively be initiated by SPIA’s successor 
over the next few years:
• Working more closely with centres to develop a CGIAR interactive IA website and 
data base and a network of expertise in various aspects of IA;
• Developing and applying IA methods for participatory research/breeding (specific 
assessments of activities should be done through the partners involved in the 
activities);
• Continuing with the assessment of the impacts of the capacity strengthening activities 
of the System, extending out from the on-going desk study assessment of training to 
other types of capacity strengthening activities in the System and to field work 
involving systematic collection of lessons learnt from those who have been trained and 
the NARS groups in which they work;
• Moving into assessment of policy research impacts across the System, working closely 
with EFPRI and other centres and consortia dealing with this topic;
• Looking at the impacts of the System’s biodiversity activities;
• Following up on the Evenson/Gollin work on CGI, perhaps (a) doing a single crop 
more in depth; (b) looking at other crops; or (c) assessing impacts in one 
region/country in more detail, e.g., Latin America or South Asia;
• Expanding on the Africa impacts study (presented by iSC/SPIA at Africa Day in 
Durban during MTM01);
• Assessing impacts in other regions, following in the spirit of the new, stronger CGIAR 
focus on regional planning and priority setting; and,
• Assessing more in detail the impacts of the System’s natural resources management 
activities, including agroforestry, forestry, fisheries and livestock research, focusing 
initially on the development of appropriate methods and processes.
3.4 Concluding Comment
The SPIA, and more broadly the iSC, continue to see IA as a central function of the new SC, 
in agreement with the recommendation of the 2001 SC working group. The synergies
18
between the overall mandate to guard the quality as well as the relevance of science are 
strong. Understanding impacts of past activity provides central input for planning how to 
improve the efficiency, effectiveness, quality and relevance of on going and future science in 
the CGIAR System.
During the recent, somewhat lengthy process of discussions and studies leading to 
institutional change within the System, SPIA has not been in a position to start major new 
initiatives. When this process of change has been decided on at AGM02, there is an urgent 
need to again become active in major systemwide assessment activities that can provide the 
CGIAR and its stakeholders with new perspectives on the major impacts derived from 
investments in the System. As indicated in section 2 of this report, a number of such 
potential activities already have been thought through by SPIA and are available for 
consideration by SPIA’s successor. There is a need to further develop a culture of impact 
assessment at all levels in order to maintain credibility of the CGIAR’s work with its 
stakeholders and the general public. To achieve this goal requires a unit within the system 
that constantly cares about and monitors this process as it progresses.
Where ex post LA is housed in the CGIAR is not as important as (1) how it is linked to the 
System’s forward planning and monitoring and evaluation functions, and (2) how its 
independence, transparency and credibility are maintained. Strong linkages with forward 
planning and M&E, and independence and credibility to the outside world are essential 
ingredients for a strong future impact assessment capability within the CGIAR.
ANNEX I
Table of Contents for the International Conference on Impacts of Agricultural 
Research and Development:“Why Has Impacts Assessment Research Not Made More 
Of A Difference?” Proceedings of a conference organized by SPIA/iSC and CIMMYT, 
4-7 February 2002, San José, Costa Rica
Table of Contents
Page
Introduction 1
Aims and objectives 2
Summary of achievements 4
Post conference activities 5
Program 13
Donors Perspectives (summary)
1.1 Rodney Cooke IFAD 30
1.2 Dana Dalrymple USAID 32
1.3 Stephan Krall GTZ 37
Media Perspectives (summary)
2.1 Barbara Rose Future Harvest 39
2.2 G. Venkataramani The Hindu 44
2.3 Gideon Lichfield The Economist 48
Closing Session (summary)
3.1 Tim Kelley/Hans Gregersen SPIA 54
Appendix 1: Abstracts
Plenary Papers
PI 1.1 Milestones in CGIAR impacts assessment research___________ 58
PI 1.2 Assessing the impacts of international crop genetic improvement 
research: Some lessons learned 60
PI 2.1 Under investment in agricultural R&D revisited________________ 62
PI 2.2 The impact of agricultural research in Bangladesh: Productivity, 
economic returns, and varietal replacement issues_______________ 64
PI 2.3 Herding cats: Is impacts assessment the ultimate exercise in futility? 66
PI 3.1 Resolving conflicting evidence about the impact of the Green 
Revolution 68
PI 3.2 Professional error, critical awareness and good science 69
PI 3.3 Why the Green Revolution failed in Africa and how this impacted the
poor 70
PI 4.1 Measuring the benefits of international agricultural economics 
research 72
PI 4.2 Evaluating the impact of economic policy research: Concepts and 
practices_________________________________________________ 74
PI 4.3 Why has impacts assessment research not made more of a difference? 78
Annex I - Page 2
Panel Papers
PA 1.1 Estimating the benefits of plant breeding research: Methodological 
issues and practical challenges______________________________ 79
PA 1.2 Returns to investment in maintenance research: The case of leaf rust
resistance breeding in CIMMYT- related spring bread wheat________ 82
PA 1.3 Economic impacts of post-harvest research for potato and sweetpotato 
in developing countries______________________________________ 84
PA 2.1 Impact assessment in natural resource management research 85
PA 2.2 Environmental impacts of productivity-enhancing crop research: 
A critical review 89
PA 2.3 Measuring the impact of user participation in natural resource 
management research 91
PA 3.1 Golden rice: What role could it play in alleviation of Vitamin A 
deficiency?____________________________________________ 93
PA 3.2 Assessing the indirect impact of mungbeans on nutrition and 
productivity; new insights from case studies in Pakistan and India 94
PA 3.3 Agriculture and nutrition: Adversaries, bedfellows or allies? 96
PA 4.1 What GIS can (and can’t) bring to impact assessment: Novel data, 
analysis, and insights____________________________________ 97
PA 4.2 GIS Tools: They’re not just for experts anymore______________ 98
PA 4.3 Lost in space: Fulfilling the promise of spatial analysis in impact 
assessment 100
PA 5.1 Assessing the impacts of agricultural research on poverty using the 
sustainable livelihoods framework: Concepts and methods_________ 102
PA 5.2 Impact of rice research on poverty reduction: The case of Bangladesh 103
PA 5.3 Improved vegetable and fishpond technology on poverty in 
Bangladesh 105
PA 5.4 The impact of improved maize germplasm on poverty alleviation: The 
case of Tuxpeno-derived material in Mexico 107
PA 6.1 Economic costs and benefits of a participatory project to conserve 
maize landraces on farms 109
PA 6.2 The distribution of benefits from public international germplasm 
banks: The case of beans in Latin America 110
PA 6.3 Endowing future harvests: The long-term costs of conserving genetic 
resources at the CGIAR centres 111
Contributed Papers
Cl.l The simple econometrics of impact assessment: Theory with an 
application to milk-market participation in 112 the Ethiopian highlands
C1.2 Measurement and source of efficiency in Argentina’s agricultural 
science research system: A stochastic frontier analysis______________ 114
C1.3 The Tradeoff Analysis Approach: Lessons from the tradeoffs project in 
Ecuador and Peru 116  
C2.1 Factors affecting the adoption of maize technologies in the hills of 
Nepal 119
C2.2 Variety characteristics, transaction costs and maize adoption in 
Honduras 121
Annex I - Page 3
C2.3 Factors affecting the adoption of selected wheat production 
technologies by farmers in Njoro and Rongai divisions of Nakuru 123
District, Kenya________________________________________
C3.1 Expanding the use of impact assessment and other evaluation 
researchevidence 125
C3.2 Why does impact assessment continue to neglect institutional 
sustainability?______________________________________________ 127
C3.3 Demand driven technological change and the traditional cereals in Sub- 
Saharan Africa: The Malian case 129
C3.4 An assessment of IPGRTs impact on international policy- making: A 
case study of the international undertaking on plant genetic resource 130
(1996-2001)
C4.1 International R&D spillovers and productivity growth in the 
agricultural sector: A panel co integration approach_______________ 132
C4.2 Impact of modem technology adoption on output growth and 
sustainability of major cereals production in Bangladesh____________ 134
C4.3 The contribution of different components of total factor productivity in 
high potential rice-wheat systems in Indian Punjab________________ 136
C5.1 Effects of innovative wheat breeding in marginal environments 138
C5.2 The impact of bean research in Honduras 141
C5.3 A study of Philippine peanut fanning communities: Impacts of new 
peanut CRSP technology and influences on sustainability_________ 143
C5.4 Impact of public sector versus private sector in R&D and technology 
generation: The case of maize in Asia 144
C6.1 Winding up the impact pathway: An approach to strengthening the 
impact orientation of national agricultural research_____________ 146
C6.2 The importance of impact assessment studies for the Brazilian 
agricultural research system_______________________________ 149
C6.3 Potato production and pesticide use in Ecuador: Linking impact 
assessment research and rural development intervention for greater 152
ecosystem health
C7.1 The welfare effects of maize technologies in marginal and high 
potential regions in Kenya_____________________________ 154
C7.2 How agricultural research affects urban poverty in developing 
countries: The case of China 156
C7.3 Impact of the adoption of MVs of rice on productivity gains and 
income distribution for the irrigated and rain fed ecosystem 157
C7.4 Household resource endowments and impacts of soil fertility 
management__________________________________________ 159
C8.1 Socio-economic, ecological, and policy impact assessment in the 
introduction of a transgenic staple crop variety to the developing 161
world: Insect resistant maize for Africa
C8.2 A farm level evaluation of the impact of IPM on pesticide use: A 
comparative analysis of IPM trained and ordinary farmers in 163
Zimbabwe’s smallholder sector
C8.3 Impact assessment of biological control in Africa: Twenty years 
experience of the International Institute of Tropical Agriculture 165
Annex I - Page 4
C8.4 A socio-economic analysis of farmers’ field schools implemented by 
the national program in 167 IPM of Thailand
C9.1 Network approach in soil management research: IWMI’s experience 
inSoutheast Asia 169 
C9.2 The impact of Rockefeller funded Forum for Natural Resources 
Management Program in eastern and southern Africa______________ 171
C9.3 Impact of the Coordinated Rice Improvement Program on movement 
of improved 172 germplasm and productivity gains across the Indian states
C9.4 Economic benefits of research cooperation: The case of the Regional 
Maize 174 Program for Central America and the Caribbean
C10.1 Can impact analysis be used for research evaluation? 176
CIO.2 An evaluation approach for achieving and attributing impact for INRM 
and EPM 178 
CIO.3 From measuring impact to learning institutional lessons: An innovation 
systems perspective on improving the management of international 180
agricultural research_________________________________________
C10.4 Disciplines, institutions and organizations: Impact assessment in 
context 182
Cll.l Tradeoff analysis as a tool for assessment of economic and 
environmental impacts of agricultural 184 research__________
Cl 1.2 Agricultural development and impacts on the environment: 
Experiences from India 186  
Cl 1.3 Impact of salinity management research in Northwest India 188
Cl 1.4 Adoption and impact of soil conservation technologies in Central 
America 190
C12.1 Evaluating capacity development of the Plant Genetic Resources 
Centre in Bunso, Ghana 192 
C12.2 Evaluating capacity development in research and development 
organizations_______________________________________ 194
C12.3 The costs of transforming public extension services towards 
participatory approaches 196
List of participants 198
ANNEX II
Strategic Guidelines for Conducting ex-post IA in the CGIAR
Draft Annotated Outline (23 July)
I. Introduction
1.1 Justification:
The need for establishing a set of strategic guidelines for ex-post impact assessment (IA)
studies in the CGIAR is long overdue and has been re-inforced at the last two major CGIAR
IA conferences4.
• Not a detailed step-wise Trow to’ manual for carrying out ex-post IAs, but rather lays out 
the basic principles and strategic issues
• Donors keenly supportive — helpful in setting up internal guidelines forjudging IAs and 
explaining them to funding and political bodies.
• A common framework would facilitate more effective system-level integration, synthesis, 
and comparison of centre-level assessments.
1.2 Challenge /Difficulty
Despite the multitude of IA studies done in the CGIAR to-date, documenting in a convincing
way the effects of agricultural research is neither simple nor straightforward.
• Outcomes of interest, such as farm income, food and nutritional security, and 
environmental sustainability, are determined by many variables other than agricultural 
research and research related activities.
• Absence of high quality data from primary or secondary sources makes the task 
especially difficult.
1.3 Objective
Formulate a set of principles and strategic guidelines for ex-post IA in the CGIAR:
• addressing key issues such as defining user needs, plausibility criteria, attribution, 
development of counterfactuals, logframe and impact pathway analysis, credibility 
issues, transparency, and communication.
• define ’principles’ of good practice under each when appropriate.
• highlight good (credible) studies as models to follow, working towards "best 
practices".
II. User Needs
Ex-post IA research has multiple uses including improving accountability, raising awareness, 
generating support, and improving research management. Given the diversity of uses among 
potential stakeholders, it is essential that IA be demand-driven and realistic objectives 
defined up front. This can only be achieved through dialogue between those requiring IAs 
and those carrying them out. Who are the former and what are their primary requirements?
4 The SPIA-organized workshop on The Future of IA in the CGIAR: Needs, Constraints and Options, 3-5 May 
2000, FAO, Rome; and The CIMMYT/SPIA inti conference on the Impacts of Agricultural Research and 
Development, 4-7 February 2002, San Jose, Costa Rica.
Annex II - Page 2
3.1 Donors
» Primary: accountability and/or justification for future investments
> “we need all kinds of impact information in various forms” (Dana Dalrymple) 
• Secondary: useful feedback
3.2 Govts /policy makers
• Evidence of sound investment
• Planning and resource allocation
3.3 Research managers and scientists
• Re-assessing on-going programmes
• Feeding it to ex-ante priority setting
Decision makers typically require three types of impact information in order to make 
informed decisions. These relate to impact information for planning and priority setting (ex- 
ante IA), impact information from on-going activities (monitoring and evaluation) and impact 
information from past activities (ex-post IA). The focus here is on the latter.
III. What is impact assessment
2.1 Definition of terms
• ex-post vs. ex-ante IA
• adoption studies (partial IA) vs. comprehensive IA
• IA vs Evaluation
> Different types of assessment and evaluation have different functions and should 
be executed by different actors in the System. IA should not be confused with 
programme evaluation.
> According to a well known textbook on evaluation (Rossi et al. 1999), "IA are 
undertaken to find out whether interventions actually produced the intended 
effects".
■ ex-ante IA (for programme planning) -> done internally within projects
■ ex-post IA (for accountability purposes) -> done externally (independence 
essential)
2.2 Defining the principal agents, the intermediaries and the target beneficiaries, i.e., 
whose impact to be assessed.
IV. Defining Basic Principles of Good Practice (or, Enhancing the credibility of IA by 
establishing linkages between ag research and observed effects—that are in turn related 
to CGIAR goals)
Note: While following such practices may not guarantee the plausibility of one’s claims, it 
provides the foundation for building a credible argument.
4.1 Programme Theory: Defining the Impact Pathway
4.1.1 Logical framework (theory of action) — logical linkages between activities and 
outcomes (adapted from Figure 2.1 in Cooksey, 1997). Similar in concept to the "programme
Annex II - Page 3
theory" (= sets of assumptions underlying policies and indicating why these policies are 
believed to have an impact):
• Step 1. Describing project activities [i.e., the source of impact]
• Step 2. Project outputs generated
• Step 3. Project outputs utilized (e.g., new seeds, new technology,
information)
> uptake by institutional clients (NARS, etc.)
> adoption by beneficiaries/target groups (farmers)
• Step 4. Direct and indirect outcomes from adoption (e.g., yields, better
policies)
• Step 5. CGIAR-goal related outcomes / long-range benefits realised (e.g.,
increased incomes for poor households, improved nutrition, 
healthier environment)
• (also see p. 8, Krall et al., 2002)
These benefits, both direct/intermediate and ultimate must be related to the Mission and 
overall purpose of the Centre/CGIAR.
4.1.2 Impact criteria and indicators defined clearly 
• Types of impact (positive AND unintended negative ones5)
> Economic (food supply, economic returns)
> Social (poverty, nutrition, education)
> Environmental (resource base, water, air)
> Institutional (NARS capacity, policies)
> Integrated / Multi-disciplinary assessment 
• Indicators/proxies
> Economic (yield/production, producer/consumer surplus, RoR, B-C ratios)*
> Social (income, poverty #s, calorie consumption, literacy)
> Environmental (soil status, water pollutants, etc.)
> Institutional (trained staff, new policies, etc.)
> Unintended (biodiversity loss / groundwater contamination)
*Note, many of the cost - benefit or IRR studies have measured benefits in terms of 
overall estimated economic surplus. As most CGIAR centres’ mission statements 
explicitly target alleviation of poverty and enhancing food security, B-C studies only 
indirectly address the impact indicator of interest. Logical frameworks can argue that 
reasonable linkages exist between these intermediate effects and the ultimate, higher 
level aggregate outcomes.
4.1.3 Distinguishing between the direct/intermediate vs. long-range outcomes/impacts, e.g., 
how far down the chain is the analysis going?
• Krall et al. (2002) believes the main attribution (cause-and-effect) gap is here, 
between intermediate impacts (increased yields) and higher level aggregate benefits 
(greater regional food security).
• This is supported by the study of Cooksey in 1997 who concluded that most CGIAR 
centres impact studies had no information and made no claims about long-range 
outcomes, such as the alleviation of poverty and conservation of the natural resource 
base. The few centres that did make statements about long-range outcomes had
5 This may include the usual negative environmental externalities, e.g., groundwater contamination, indirect 
negative effects on non-adopters (falling output prices) and losses due to inappropriate utilization of the 
technology in certain situations (the ’poison well’phenomenon).
Annex II - Page 4
relatively weak data to support such claims. Only one impact study (from IPGRI) 
provided supporting evidence for making the claim about long-range outcomes.
4.2 Empirical measurement of changes in the impact indicator of interest (yield, income, 
food security, etc.)
4.2.1 Methods {specify by economic vs. social vs. institutional vs. environmental??} 
• Quantitative
> Economic surplus
> Econometric analysis
> Survey
> Analysis of secondary data
• Qualitative
> Case study
> Key informant/target group surveys
> Expert opinion
> Antecdotal
• Mixed
> e.g., EFPRI poverty case studies
4.2.2 Methodological rigour
• Appropriate tools used
• Assumed values are adequately justified
4.2.3 Discounting benefits and costs
• Defining and standardising opportunity costs for capital
4.2.4 Sensitivity analysis to test critical assumptions
• Selecting inputs for analysis
• Accounting for cummulative effects of covariance
4.2.5 Limitations
4.3 Impact Monitoring
Relates to situations where impact of interest cannot be adequately measured or empirically 
related to the research outcome, but, where those research products are known to positively 
affect or contribute to it are measurable.
• Establishment of the logical framework linking expected outputs of the Centre to its 
overall mission and purpose is essential.
• Monitoring of intermediate products and outputs from research (e.g., publications) as 
indicators of steps made toward achieving the longer range outcomes of poverty 
alleviation and food security (the ultimate CGIAR goals)
4.4 Impact Pathway Analysis
4.4.1 The problem: The impacts/outcomes of interest, such as farm income, food security, 
etc., are determined by many variables other than the useful products of ag research. How to 
’attribute’ / measure ag research’s role in the effect?
Annex II - Page 5
4.4.2 Impact pathway defined (logical framework, impact indicators) [see above]
4.4.3 Conceptual boundaries of analysis defined
4.4.4 Spatial and temporal dimensions of the IA analysis defined
• Time period depends on main objective of IA
> long term (15-20 years) for comprehensive aggregate level effects
> short term (5 years) for use by research managers in decision making
• Spatial dimension
> geographical mandate/target of the research programme
• Spillover effects
4.4.5 Long-range/comprehensive or intermediate impact indicators measured or monitored
4.4.6 Develop/test plausible cause-and-effect relationships between linkages (as specified in 
the logical framework)
• Development of the counterfactual
> with and without (use of models)
> before and after (baseline surveys)
• Multi-source verification and synthesis of evidence
> to limit bias from any single source or method
> assess: points of corroboration and points of inconsistency
• Data gaps and cautious reporting of conclusions
> accuracy-related
> geographic coverage
4.4.7 Stakeholder / Intended beneficiaries’ perspectives
• Other informed opinion that support or contest the findings
• Degree of consistency
4.5 Scaling up
• Basis for extrapolation
• Sampling issues
• Cumulative and sequential impacts
4.6 Measuring full programme costs
• Research and extension costs relevant to the development and dissemination of the 
technologies being assessed.
• IARC and NARS costs in the development and disseemination of the technologies
• Indirect costs (administration, depreciation, complementary services)
4.7 Ensuring transparency [DELETE??]
4.8 Dealing with attribution issues
4.8.1 When attribution is important and
• Relatively easy to do: When few actors involved and the research activity to 
research output to intended outcome/impact chain is reasonably straightforward 
and linear. When important but difficult to do
Annex II - Page 6
• Relatively difficult to do: When many partners involved, many playing an 
important role in a complex process involving others outside agriculture.
4.8.2 When attribution is not so important:
• Principle: The more effective a centre-NARS partnership is, the less desirable and 
feasible it is to attempt to attribute impacts separately to each partner. Indeed, in 
some cases, attempting attribution could be counterproductive and put at risk good 
working relationships.
4.8.3 Assessment of other mitigating factors (infrastructure, markets, policies, weather)
4.9 Cost effectiveness
• quick and clean
• trade-offs
4.10 Independence/Credibility
• Who conducts the ex-post LA?
> IA for mainly accountability purposes requires external assessment, i.e. external to 
the programme being assessed.
■ by a unit outside the organization, to achieve maximum credibility
■ or from within the organization but outside the research programme; though 
the former is perceived to be more credible. (Consider third party audit)
■ important aspect is that it is perceived to be ’independent’, without bias and 
credible.
• Role of NARS
• Role of intended beneficiaries
4.11 Drawing lessons
• shortcomings/honest attribution
• elucidating key constraints and effects thereof
• lessons learnt
4.12 Communication
• clear strategy for communication linked to specific user needs (those requiring IA), 
and others
> for political decision makers, results of IA in short, transparent and readable form
> effective dissemination of information / publicizing results
• “We need all kinds of impact information in various forms. We need summary 
information for administrators and Congress, and we need more detailed information 
to use with colleagues and for ourselves in presenting the work of the IARCs and in 
making our own internal budget decisions” (Dana Dalrymple, USAID 2002)
Annex II - Page 7
V. Models to Follow
5.1 IFPRI policy impact studies
5.2 I1TA Impact senes
5.3 Others
VI. Qualifiers/Limitations
6.1 Data constraints
6.2 Hard-to-measure impacts
• some impacts can’t be measured cost effectively (doesn’t mean impact isn’t there)
• valuation of non or partially priced goods and services
6.3 Others
VII. Future Emphases
7.1 Multidisciplinary las
7.2 Cost effectiveness (low cost, data collection)
Annex III
Benefit Studies for Review in the CGIAR Benefit/Cost Meta-Analysis
6 Evenson, R.E. and D. Gollin (eds). 2002. Crop Genetic Improvement arici Agricultural Development. Oxon, U.K.: CABI.
7 Pingali, P.L. and Hossain, M (eds). 1998. Impact of Rice Research. Los Banos, Philippines: IRRI.
Annex III
Virus-free Shandong
Fughe et al. 1999 CIP SweetpotatoPublication Planting CIP Province,
Material China
Water
De Groote et al. 2002 Unpublished Hyacinth IITA Southern
Biocontrol Benin
Byerlee and 
Traxler 1995 AJAE Wheat CGI CIMMYT Global
Heisey et al. 2002 Evenson & Gollin Book Wheat CGI CIMMYT Global
ANNEX IV
Sample Database Record for Review of Benefit Studies
9udv ~Rle
Economic Impact of Virus-Free Sweetpotato Planting Material in Shandong Province, China
[Author
jFuglie, K.O., L Zhang, LF. Salazar, and IS Walker
|Author Insti tu tion(s)
jCIP, Shandong Academy of Agricultural Sciences (SAAS)
|Year of Study I C ateqory of Innovation
1999 Crop Managem ent
Citation:
Fuglie, K.O., L Zhang, LF. Salazar, and T.S Walker. 1999. Economic Impact of Virus-Free Sweetpotato Flanting 
Material in Shandong Ftovince, China. Lima,Peru: International Potato Center.
|C enters Covered jC IP
Geographic Coverage
Shandong Province,China
Description of Innovation
With CIP assista nee, training courseswere organised to demonstrate to SAAS scientists the meansby which virus-free 
(or disease reduced) sweetpotato seed could be produced. Such a technique invovesproducing seed from virus 
tested mother plantsregenerated in a culture medium from the meristemsof selected older plants: After training, 
S^AS scientists conducted research to determine the best meansto apply these techniquesto field conditions, and ; 
found that these seedsoffered 40% better yields than did farmers' seeds In 1993-1994 large scale extension and 
demonstration triaiswere conducted with additional financial support from provincial and local governments The 
first virus-free seed wasoffered to farmers in 1934, and by 1997 40% of sweetpotato area wasplanted with these 
seeds.
Overall Ad vantage oflnnovation
Virus-free seedsoffer im proved yields, as disease incidentsare reduced. However,thiseffect is reduced during 
.multiplication, a s the seeds become reinfected.  . ........................................
Indicator of Productivity Changes r*.
yield as indicated ingroup interviews
Measurement Method for Productivity Changes
Group interviewswith farmers in 30 villages validated against demonstration plots
Disaggregation of Utilisation/Adoption
none
Percentage Ad option! |Area of Adoption (ha) |% of Productivity ChangesDue to Adoption |
78 416000 100
% Yield Increase: Ftice of Production:
22 r 0.4Yua n/kg a
Changes in Inputs. Crop Management Considered
planting density, N, P. & K fertilisers, pesticides, and labour
Mitigating Factors Considered
none
External Effects Included
none
Partitioned Credit 0 % of Credit for Innovation Claimed by IARC■ 100
Annex IV - Page 2
Basis of Credit Partitioning
The main technique is entirely attributed to CIP, but was applied locally by Chinese institutions However, the study 
notesthat thistechnique would have probably reached the province eventually,even without CIPsparticipation. 
Thus, when an IARC NFV ispresented below,it isbased on the technique being applied 5yearslater in CIPs 
absence.
OtherActorsAttributed
SAAS, local g overnm ents
CGIARRole in Development of Innovation
CIP developed the basic replication technique, which was tea died for local application by the Shandong 
Academy of Agricultural Sciences. Further demonstration, extension, and seed production were undertaken by 
local officia Is
[Explicity Described Counterfactual £7]
Basis of Counterf acuta I
Same benefits, delayed by 5years
[Research Partners Included in Counterfactual
[SAAS, local governments
Disaggregation of Counterfactual
none
Basis of Input Rices ] [Basisof Commodity Prices
interviews jin ter views
Elasticity of Supply | Elasticity of Demand | Assummed Lag Perio| 5artYear of Benefit] End Year of Benefits]
0 1000 4 1394 2020
Base Year of Study | |Mean Discount Rate] High_D_iscount Rate [ Low Discount Rate Sensitivity Analysis
1998 15 20 10 0
Factors Tested in Sensitivity Analysis
adoption (90% rather than 78%). elasticity of supply (2 rather than 0), costsof research, extension. & seed 
multiplication (doubled).yield improvement (halved), research, extension & seed multiplication doubled with half 
the observed yield improvement
Taget G roup Identified
It isassum ed that the producer isthe benefactor of the generated surplus In addition, incom e effectsfor producer 
households are presented for several groups of villages, but no pop ula tions for these villages accomp any the 
estimates, so these effectsmay not be aggregated.
Total NFV Under Main Assumptions | [Low Range of Total NFV [ jHigh Range of" Total NFV
550000000 38000000 620COCCOO
Main IARC NFV: Low IARC NFV: High IARC NFV:
212000000 I
NFV for Target Group | Most Credible NFV Estim ate
0 212000000
Assumptions for Most Credible NPV Estimate
Although the yield improvement estimatesare likely to be overstated (see below),the assumption that thisexact 
technology would be developed by the Chinese in CIPs absence in 5years probably gives CIP too little credit. If 
one assumesthat these factors "cancel ouf'then the $212 million figure may be accepted.
[Comm ents on Analysis
The participatory basisof the study isquite good, and many factorsappear to be considered. However, little 
documentation regarding the group interviewsha sbeen presented, and it is unclear how they were conducted. 
The interview and experimental plot basisof the yield improvement estim a tesisalso questionable,asfarmersmay 
not be able to readily quantify yields.and experimental plots generally have better observed yieldsthan do 
farmers It isalso unclear how villageswere selected, and how inform ants were selected within villages The 
participation of headmen and government scientistsm ay have also created incentive for overestim ation of yield 
benefitsduring interviews Additionally disconcerting isthe apparently low consistency of the interviews, asof 30 
villages interviewed, only 20 observations could be utilised for regression analysis It is repeatedly noted that yield 
advantagesdecrease from one generation to the next with virus-free seeds, but the basisof assurtiptionsregarding 
the procurement of fresh seed is unclear. Mitigating factors, such asimproved extension or infrastructure 
accompanying the provision of im proved seed are not explicitly considered. Finally it isunclear whether a 22% or 
24% provincial yield increase is claim ed, as a t one point the former value ism entioned.and at a later point the later 
value is referenced.
,:‘ì«
•: ..<1 . ; - - » T: <• ., ••  ..............■ --------------- - . -Jv; . - V . - ..
ANNEX V
Agenda for the Agricultural Research and Development Strand of the 
African Evaluation Association Conference,
June 10-14, 2002
Monday June 10th Presenter Title
Chair: Kwesi Atta-
1300-1315 Atta-Krah, Kwesi Welcome to Agriculture and Development Strand, 
introductions and opening remarks
1315-1345 Kristjanson, Patti et al Evaluating the impact of agricultural research and 
development: Future Harvest Research Centre Approaches
1345-1415 Waithaka, Michael, F. Evaluation of the impact of KARI's research programmes
Murithi, T. Cusack, E.A. 
Mukisira, P.W. Mwangi, N.M. 
Ng’and’a and J.W. 
Wamuongo
1415-1430 Mbabu, Adiel and G. Ebong Principles of evaluation: Methodological lessons derived 
from East and Central Africa, in the context of ASARECA
1430-1500 Coffee break
1500-1530 Mengistie, Getachew Process and institutional synergies for development of 
national plant genetic resources policies in Africa: Ethiopia 
case study________________________________ __
1530-1600 Ibro, Germaine, B. Moussa, Analyse Coùt -bénéfice des Technologies du Niébé : 
A. Kamaye and T. Une Application de la Matrice d’Analyse des Politiques
Nouhoheflin ( MAP) _____
'
Tuesday June 11
Chair: Patti 
Kristjanson
1300-1330 Diagne, Aliou, K. Kouadio The impact of modern varieties on rice biodiversity
Arsene and R. Gue
1330-1400 Omosa, Mary Assessment of the impact of agro-forestry based soil fertility 
replenishment interventions on the poor in Western Kenya
1400-1430 Koné, Miaman Analyse de l'impact de la vulgarisation agricole sur 
l'efficacité économique des paysans dans le Nord-Ouest et 
l'Ouest de la Còte d'Ivoire: cas des Departements de Touba 
et de Biankouma
1430-1500 Coffee Break
1500-1530 Fall, Amadou Abdoulaye Impact économique de la recherché sur le riz dans deux 
pays de l’afrique de l’ouest : Senegal & Mauritanie
1530-1600 Discussion
11Pi
Wednesday June 12
Chair: Boru 
Douthwaite
1300-1330 Igue, Alice Gender-based management and conservation of local plant 
genetic resources and their contribution to food security and 
poverty reduction in Benin_________________________ _
1330-1400 Runyoro, Gerald and A. Impact of superior banana varieties on food security and 
Gallez income for small scale farmers in Kagera Region, Tanzania
1400-1430 Nyende, Paul and R. Delve Farmer participatory evaluation of legume cover crop and 
biomass transfer technologies
1430-1500 Coffee Break
1500-1530 Ndiema, Alice, and M. Socio-economic factors affecting adoption of some selected 
Kinyua wheat production technologies by farmers in Njoro and 
Rongia divisions of Nakuru district of Kenya
1530-1600 Discussion
Annex V - Page 2
H II ........
Thursday June 13
Chair: Aliou Diagne
1300-1330 Bennett-Lartey, Samuel, R. Evaluation of capacity development of the Plant Genetic 
Vodouhe, and J. Watts Resources Centre in Ghana
1330-1400 Remington, Tom, L. Enhancing the capacity for USAID/OFDA to evaluate 
Sperling, P. Bramel, R. seed-based agriculture recovery proposals
Best, S. David and R. Kirby
1400-1430 Douthwaite, Boru, S. Schulz Bridging the attribution gap: An evaluation approach for 
and A. Olanrewaju achieving and attributing impact in African agriculture
1430-1500 Coffee Break
1500-1530 Discussion
1530-1545 Watts, Jamie Evaluation of the strand and brief discussion of its future
1545-1600 Atta-Krah, Kwesi Closure of strand