Research Report
An Assessment of the Impact
of ISNAR: 1997-2001 k
Jock R. Anderson "*| mv
Ponniah Anandajayasekeram
Eric Craswell
Mandivamba Rukuni
International Service for National Agricultural Research (ISNAR)
Until March 2004, ISNAR was one of the 16 Future Harvest Centers supported by the Consultative
Group on International Agricultural Research (CGIAR). At its 2003 Annual General Meeting, the
CGIAR approved the recommendations of a team that was charged with restructuring ISNAR. One
of the recommendations called for ISNAR to cease operating as an independent center of the CGIAR.
The Annual General Meeting requested that the Boards of ISNAR and the International Food Policy
Research Institute (IFPRI) carry out a transfer of governance and relocation of ISNAR's programs to
IFPRI. On April 1, 2004, the new ISNAR Program was established as a division within IFPRI and
began work from a base on the ILRI campus in Addis Ababa, Ethiopia.
Relocating the new ISNAR Program in sub-Saharan Africa brings it into closer contact with its main
target region. The new Program aims to work in partnership with national and regional organiza-
tions and the CGIAR Centers. The primary theme of the Program will be to produce new knowledge
on institutional change that enhances the impact of agricultural research. The secondary theme
will be to enhance the performance of agricultural research institutions by strengthening their
organization and management. A Program Advisory Committee, established by the IFPRI Board and
consisting of NARS leaders and specialists on the main themes, will help guide these activities.
Research Report 25
An Assessment of the Impact
of ISNAR: 1997-2001
Jock R. Anderson, Ponniah Anandajayasekeram, Eric Craswell,
and Mandivamba Rukuni
August 2004
Copyright © 2004 by the International Service for National Agricultural Research (ISNAR).
ISNAR encourages the fair use of this material. Proper citation is requested.
About the authors
Jock R. Anderson is an Emeritus Professor of Agricultural Economics at the University of New
England, Armidale, Australia. He spent several years working in the World Bank variously as Ad-
viser on Agricultural Technology Policy, Lead Evaluation Officer, and Adviser on Rural Strategy and
Policy, and retired to consultancy work in 2003. He has regularly assisted the CGIAR Centers in their
various external evaluations. Dr Ponniah Anandajayasekeram is a Senior Research Fellow with the
International Food Policy Research Institute (IFPRI) based in Addis-Ababa, Ethiopia. He is also a
Professor Extra-Ordinary, Post Graduate School of Agriculture and Rural Development, University of
Pretoria. He has worked in Africa for over 20 years with various international organizations such as
CIMMYT, SACCAR-SADC, FAO, and ISNAR. He has participated in a number of impact assessment
studies and evaluation missions. He has also been President of the International Farming Systems
Associations (IFSA) and the Southern African Association for Farming Systems Research and Exten-
sion (SAAFSRE). Dr Eric Craswell is the Executive Officer of the Global Water System Project (GWSP),
based at the Center for Development Research (ZEF) in Bonn, Germany. He has extensive experience
in research on tropical soil management and nutrient cycling, working at IFDC, IRRI, FAO, IAEA, and
ACIAR. During 1996-2001 he was Director General of the International Board for Soil Research and
Management in Bangkok. Dr Mandivamba Rukuni is the Program Director of the W.K. Kellogg Foun-
dation. He is also Professor Extra-Ordinary of Agricultural Economics, University of Pretoria, and a
member of the Board of Trustees of the International Food Policy Research Institute. He has a long-
standing association with a number of CGIAR centers and has led a number of evaluation missions.
He has been retainer consultant for the Rockefeller Foundation, World Bank, CIDA, SIDA, and CGIAR.
Citation
J.R. Anderson, P. Anandajayasekeram, E. Craswell, M. Rukuni. 2004. An Assessment of the Impact
of ISNAR: 1997-2001. ISNAR Research Report 25. The Hague: ISNAR.
ISSN 1021-4429
Contents
v Acknowledgements
vii Executive Summary
IX Acronyms
1 Introduction
1 Background
2 The role of ISNAR in the CGIAR
4 Recommendations of the 1997 EPMR
4 The changing environment of the NARSs
5 ISNAR's changing environment
6 Structure of the report
7 Chapter 1. Conceptual Framework
7 1.1 The role of ISNAR as a CGIAR Center
7 1.2 Impact assessment: Perspectives from the literature
13 Chapter 2. Methods
15 Chapter 3. ISNAR's Program Areas, Outputs and Assessments
15 3.1 Training
24 3.2 Management tools and procedures/service and advice
27 3.3 Research results
29 3.4 Assessment of direct collaborative activities
33 3.5 Country case studies: Summary of key findings
37 Chapter 4. ISNAR's Niche and Priorities
37 4.1 Niche
39 4.2 Priorities
41 Chapter 5. ISNAR's Impact: Constraints and Enhancement
45 Chapter 6. Institutionalizing Impact Assessment within ISNAR
47 Chapter 7. The Impact of ISNAR
47 7.1 Raising awareness and contributing to knowledge
48 7.2 Influencing policy
49 7.3 Institutional development
51 7.4 Developmental impact
52 7.5 Conclusions
53 References
The annexes to this report are on the accompanying CD on the inside back cover
ISNAR Research Report 25
Tables
16 Table 1: Number of ISNAR training workshops and trainees (1997-2001)
18 Table 2: Results of PAPA analysis for the Agricultural Research Management
Training workshops
19 Table 3: Summary of impacts at different levels - Results of PAPA analysis
for the course on New Technologies for Agricultural Research:
Managing Biotechnology in a Time of Transition
22 Table 4: Internet module downloads: 1998-2001
25 Table 5: Assessment of generic tools and procedures: Summary of survey
responses
26 Table 6: Overall assessment of generic tools and procedures
27 Table 7: ISNAR's total written output (1997-2001)
28 Table 8: ISNAR research publications (1997-2001)
28 Table 9: Respondent perceptions of the value of ISNAR publications
30 Table 10: Assessment of ISNAR's collaborative work by direct collaborators
and donors
37 Table 11: Overview of survey responses regarding the unique advantage
of ISNAR as a collaborator
38 Table 12: Overview of survey responses regarding disadvantages of ISNAR
as a collaborator
39 Table 13: Assistance that could be provided by ISNAR to improve
institutional performance: Summary of survey responses
40 Table 14: Summary of survey responses regarding ISNAR's priorities
for promoting innovation in agricultural research institutions
41 Table 15: Summary of survey responses regarding constraints on ISNAR's
impact
42 Table 16: Summary of survey responses regarding ways of enhancing the
impact of ISNAR
45 Table 17: Constraints on ISNAR's impact identified by Horton and Mackay
(1998)
Figures
3 Figure 1: ISNAR as a CGIAR center in the world of development
5 Figure 2: The evolution of ISNAR's medium term projects (MTPs)
(1997-2002)
10 Figure 3: ISNAR's impact chain
11 Figure 4: Factors affecting the performance of NARSs
Acknowledgments
The impact assessment team was greatly assisted in its work by the management and staff of
ISNAR. Particular mention must be made of several people who were exceptionally helpful, includ-
ing Leandra Julien and Mai-Britt Nielsen for general logistical support (especially in regard to the
country case studies), Jacobine Verhage for making many useful things happen, Mary Gavin for
kindly extracting information from the central database to facilitate the surveys and Monica Allmand
for assistance with library-based information. Extraction of data and text from the survey materials
was conducted mainly by Garcia Ochoa Garcia, while Jacintha Vigelandzoon was responsible for
finalizing the address list for the main survey of publication users. Final preparation of the report
was greatly helped by the generous intervention of Cristina Sette and by Frank M. Anderson, coin-
cidentally visiting his brother Jock during the finalization of this report. To all the above, the team
extends its warmest gratitude, without, of course, implicating any of those mentioned in any
limitations of the report.
Readers in 2004 and beyond will find the tense a little strange and should remind themselves that
this was the reality the team encountered in mid-2002. So much change has taken place as the new
ISNAR was launched in 2004, much of it echoing sentiments of clients and stakeholders expressed
to and recorded herein by the team. Much of what is set out in this report is thus something "for the
record" on the "old" ISNAR, yet it has much of relevance for the "new".

Executive Summary
In preparation for its fourth External Program and Management Review (EPMR), the International
Service for National Agricultural Research (ISNAR) commissioned a team to report on ISNAR's achieve-
ments, impacts and constraints in the period from 1997 to 2001. The report was to include an
assessment of ISNAR outputs (including publications, training manuals, reports, planning docu-
ments, etc.), an evaluation of changes in the capacity and performance of agricultural research
organizations that could be linked to ISNAR's activities, an assessment of the degree to which such
changes could be related to the ultimate goals of the Consultative Group on International Agricul-
tural Research (CGIAR),1 together with recommendations for increasing the impact of ISNAR's work.
The report, submitted in July 2002, was originally targeted at those conducting the EPMR, together
with ISNAR management and staff. It was subsequently felt that the report should be made avail-
able to a wider readership, including those in the CGIAR system in general (particularly donors and
investors), managers of national agricultural research institutes (NARIs) and others active in agri-
cultural research at national and regional levels.
Most of the work involved in preparing the report was conducted between April and the end of
June 2002. Given the limited time available, a framework was needed to guide the team's efforts.
Several methods were used, including surveys of various partners (e.g. a global attitude survey of
employees of national agricultural research systems (NARSs) (N=315), NARI managers (N=246),
direct collaborators (N=135), donors (associated with specific projects) (N=22), advanced research
institutes (ARIs) and international agricultural research centers (IARCs) (N=43), subregional organi-
zations (SROs) (N=8), trainees and their supervisors (N=310), those who downloaded computer-
based ISNAR training modules (N=30), and ISNAR publication users (N=500). These surveys were
complemented by 11 case studies of different countries in which ISNAR had conducted collabora-
tive work (Argentina, Cuba, Dominican Republic, Indonesia, Mexico, Pakistan, South Africa, Sri Lanka,
Uganda, Vietnam, and Zambia).
In its review of these and of ISNAR's own materials, the assessment team paid particular attention
to evidence of the following: institutional development, developmental impact in general, raising
awareness among clients, direct contributions to knowledge and indirect contributions to knowl-
edge through influencing policy. In addition, various focal projects were examined separately,
including the ISNAR Biotechnology Service (IBS) and the project on Agricultural Science and Tech-
nology Indicators (ASTI). The latter has informed debate both at national and subregional levels
(e.g. on the declining support for agricultural research, setting research priorities, etc.), while the
former has assisted the development of biotechnology and biosafety policies in a number of countries,
including Argentina, Cameroon, Chile, China, Egypt, Indonesia, Kenya, Malaysia, Mexico, Namibia,
Nigeria, Philippines, Turkey, Uganda, and Zimbabwe.
The team also addressed the effects of ISNAR's broad range of core activities, including its work on
developing policies on research funding, developing institutional policy and discourse, increasing
private sector participation in agricultural research, encouraging stakeholder participation in research
policy development (e.g. via collaborative work with the Global Forum on Agricultural Research
1 To reduce poverty, hunger and malnutrition by sustainably increasing the productivity of resources in agriculture,
forestry and fisheries.
ISNAR Research Report 25
(GFAR)), and its more general attempts to support and develop research institutions in developing
countries. Credibility in such efforts obviously depends on the consistency between in-house prac-
tice and that which is being advocated. However, in the judgment of the team, ISNAR has not used
a clear and comprehensive conceptual framework to support its capacity-building initiatives, par-
ticularly in regard to assessing the impact of research; there is a clear need for ISNAR to develop
an in-house 'impact culture'.
Training and capacity building has been one of ISNAR's principal concerns over the review period
and many useful materials and tools have been developed with this in mind. The training modules
most valued by managerial users include those on program management and evaluation, strategic
planning, and priority-setting. Training on biotechnology/biosafety has also been well received by
beneficiaries. Furthermore, the multiplier effects of'training the trainers' have been considerable,
with each ISNAR trainee subsequently training, on average, a further 20 individuals. Some of the
tools developed by ISNAR - such as the organizational performance assessment tool - have also
been highly valued.
Countries that benefited from various advisory services provided by ISNAR during the review period
included Benin, Croatia, Cyprus, Eritrea, India, Indonesia, Iran, Mozambique, Pakistan, Palestine,
South Africa, Sri Lanka, Uganda, Vietnam, and Zambia. However, surveys of the beneficiaries of
these services suffered from low response rates, so that some of the team's conclusions regarding
this aspect of ISNAR's work are rather tentative. In general, however, significant positive behavioral
changes were noted in participating staff, who, in addition to training others, were also said to
have gained confidence and increased their knowledge. Improvements were also reported in specific
areas such as writing skills, involvement in planning, proposal writing, monitoring and evaluation,
and performance in workshop facilitation.
Given the nature of the activities reviewed by the team, it was natural to try to determine the
extent to which ISNAR has generated global public goods. ISNAR's products are very different in
character from those traditionally produced by the CGIAR system, and the team found it difficult to
discern quantifiable cost effectiveness (on either a regional or global basis), especially for interme-
diate products whose full impact may not be apparent for some time. However, the team formed
the impression that ISNAR has been quite effective in sponsoring and supporting institutional
innovations that are widely used in different regions and cultures, and in stimulating systemic
policy reforms. There have also been noticeable positive changes in attitudes to, and perceptions of,
agricultural research and development that many associate with ISNAR. Although many of these
are relatively localized (or intermediate) products and activities, the impression of the team is that
they have achieved growing regional and global impact. However, because of the difficulty in-
volved in quantifying cost effectiveness, considerable scope remains for the further investigation
of ISNAR's diverse activities.
Over the period covered by this study, the overall impact of ISNAR's work has been relatively mod-
est. Attempting to fulfil its global mandate by covering all major developing regions has had the
effect of spreading ISNAR's limited resources too thinly, and impact has therefore been both limited
and fragmented. However, the team believes that the type of public goods produced by ISNAR are
intrinsically valuable and are still needed in many parts of the developing world. The challenge for
ISNAR and its parent CGIAR is to focus both strategy and resources on meeting these needs.
Acronyms
AARD Agency for Agricultural Research and Development
AIAT Assessment Institute for Agricultural Technology
AIDS Acquired Immune Deficiency Syndrome
ANR agriculture and natural resources
ARC Agricultural Research Council
ARI advanced research institute
ASARECA Association for Strengthening Agricultural Research in Eastern and Central Africa
ASTI Agricultural Science and Technology Indicators
CARP Council for Agricultural Research Policy (Sri Lanka)
CGIAR Consultative Group on International Agricultural Research
CIAT Centra Internacional de Agricultura Tropical
CIMMYT Centra Internacional de Mejoramiento de Maiz y Trigo
CTA Centre Technique de Cooperation Agricole et Rurale
EPMR External Program and Management Review
FAO Food and Agriculture Organization (of the United Nations)
FARA Forum for Agricultural Research in Africa
GDP gross domestic product
GFAR Global Forum for Agricultural Research
GMO genetically modified organism
GTZ Deutsche Gesellschaft fur Technische Zusammenarbeit
IAD institutional analysis and development
IARC international agricultural research center
IBS ISNAR Biotechnology Service
ICRISAT International Crops Research Institute for the Semi-Arid Tropics
IFA International Forum on Food and Agriculture
IFAD International Fund for Agricultural Development
IFPRI International Food Policy Research Institute
IGA ISNAR Global Associate
ILRI International Livestock Research Institute
INCO-DEV International Cooperation with Developing Countries (EU)
INFORM-R Information for Agricultural Research Managers, Relational
INIBAP International Network for the Improvement of Banana and Plantain
INIFAP Instituto Nacional de Investigaciones Forestales y Agropecuarias
INTA Instituto Nacional de Tecnologia Agropecuaria
INTG IARC/NARS Training Group
IPGRI International Plant Genetic Resources Institute
ISNflR Research Report 25
IPR intellectual property rights
ISAAA International Service for the Acquisition of Agri-Biotechnology Applications
ISNAR International Service for National Agricultural Research
IWMI International Water Management Institute
MFI multilateral finance institution
MIS management information system
MTP medium term project
NARC National Agricultural Research Center (Pakistan)
NARI national agricultural research institute
NARO National Agricultural Research Organization (Uganda)
NARS national agricultural research system
NDA National Department of Agriculture (South Africa)
NGO nongovernmental organization
OPAS organizational performance assessment system
PAPA Participant Action Plan Approach
PROCISUR Programa Cooperativo para el Desarrollo Tecnologico Agroal imentar io y
Agroindustrial del Cono Sur
R&D research and development
RIABGR Research Institute for Agricultural Biotechnology and Genetic Resources
SCRB Soils and Crops Research Branch
SRO subregional organization
TAC Technical Advisory Committee (of the CGIAR)
UNCST Uganda National Council for Science and Technology
UNDP United Nations Development Programme
WB World Bank
Introduction
In preparation for its fourth External Program and Management Review (EPMR), the International
Service for National Agricultural Research (ISNAR) commissioned a report with the primary aim of
providing the review panel with an 'objective and insightful assessment' of ISNAR's achievements,
impacts and constraints in the period since the previous EPMR (1996-97). It was hoped that the
report would also provide information that would help the ISNAR board, management and staff to
redirect their activities, if so required.
The terms of reference for the report stated that it should contain the following information:
• A description of the study methods and information sources used.
• A listing and assessment of ISNAR outputs, including publications, training manuals, reports,
planning documents, workshops, conferences, web-based publications, etc.
• An assessment of changes in the capacity and performance of agricultural research organiza-
tions that could be linked to ISNAR activities.
• Wherever possible, an assessment of how these changes have contributed to the ultimate
goals of the CGIAR (i.e., to reduce poverty, hunger and malnutrition by sustainably increasing
the productivity of resources in agriculture, forestry and fisheries).
• Recommendations for increasing the impact of ISNAR's work and on institutionalizing impact
assessment of its work.
The study addressed not only the immediate outputs generated by ISNAR's work, but also the out-
comes and eventual development impacts of various collaborative efforts. Furthermore, focusing
exclusively on activities conducted only during the review period would have been somewhat re-
strictive, in that their impact might not be apparent until some time in the future, while some of
the benefits of ISNAR's earlier work were finally realized during the review period. As a result,
unambiguous attribution was a problem in some cases, since a number of factors other than ISNAR's
activities can influence the achievement of development goals. An attempt was therefore made to
trace relevant sequences of events and their effects, identifying as clearly as possible the contribu-
tions of ISNAR, while recognizing the efforts of other partners.
Background
In 1995, a total of US$12.1 billion was invested in agricultural research in developing countries by
governments and the private sector (Pardey and Beintema 2001). Expressed in 1993 international
dollars, public sector investment grew from US$4.7 billion in 1976 to 11.5 billion in 1995, or from
0.42% to 0.62% of agricultural gross domestic product (GDP). Although this is low relative to the
average 2.64% of agricultural GDP spent on research in more developed countries in 1995, it never-
theless represents a significant level of investment for developing countries. Such investments
were made in the expectation of obtaining high rates of return (similar to those resulting from the
development of high-yielding cereal varieties in the 1960s).
The key role that national agricultural research systems (NARSs) could play in accelerating techno-
logical innovation in the post 'green revolution' agriculture of developing countries was recognized
some time ago by members of the Consultative Group on International Agricultural Research (CGIAR)
and in 1979 led to the establishment of ISNAR. ISNAR's mission was to assist developing countries
in improving the performance of their NARSs and related organizations by promoting appro-
priate agricultural research policies, sustainable research organizations, and improved research
management.
ISNAR Research Report 25
In order to maximize the impact of its work in developing countries, ISNAR has focused on three
major objectives:
1. Enhancing the capacity of agricultural research organizations to respond to their clients'
needs and to emerging challenges.
2. Contributing to global knowledge of agricultural research policy, organization and manage-
ment.
3. Improving the ability of developing countries to access this knowledge.
It is axiomatic, therefore, that ISNAR concerns itself with the impact of research. For its clients in
developing country institutions, as well as the other Future Harvest Centers supported by the CGIAR,
impact assessment provides a means of increasing accountability, improving program design and
implementation, and planning and prioritizing research.
The remainder of the Introduction consists of a brief review of the role of ISNAR in the CGIAR, the
findings of the previous EPMR, and the changing context in which both NARSs and ISNAR operate
in relation to impact assessment.
The role of ISNAR in the CGIAR
It is now over 30 years since the establishment of the CGIAR, and more than 20 years since ISNAR
was created. The CGIAR has continued to evolve throughout this time, with perhaps the most sig-
nificant change being the adoption (in the late 1980s) of sustainability as a major research goal. As
a consequence of this decision, the institutional composition of the CGIAR expanded to include
natural resource centers, and the group's Technical Advisory Committee (TAC) elaborated the con-
cept of ecoregional and global programs. The group's overall goal is now
To reduce poverty, hunger and malnutrition by sustainably increasing the productivity of resources
in agriculture, forestry and fisheries.
Each CGIAR center contributes in its own way to the attainment of this goal through the following
three 'purposes':
Purpose 1 The development by NARSs of improved production systems that will increase
productivity while conserving biodiversity, land and water.
Purpose 2 Improving the performance of NARSs and regional programs.
Purpose 3 Improving and implementing policies relating to CGIAR outputs.
ISNAR contributes to the overall goal of the CGIAR by addressing the second and third of these
purposes.
The relationship between NARSs and the CGIAR centers (including ISNAR) has been extensively
reviewed and analyzed by the CGIAR over the years. NARSs play a central role within the interna-
tional agricultural research system and in many cases share goals similar to those of the CGIAR.
Figure 1 shows the current CGIAR goals and illustrates the context in which ISNAR and its partners
operate in support of NARSs.
The CGIAR addresses its goals through intermediate impacts (Horton 1990) on production technolo-
gies and on research and development (i.e., on institutions), with the ultimate aim of producing
field-level effects (impact) on poverty eradication, environmental protection and food security. These
efforts are underpinned by donors, the development banks (i.e., the World Bank (WB) and regional
multilateral finance institutions (MFIs)), development organizations such as the International Fund
for Agricultural Development (IFAD), and development agencies such as the Food and Agriculture
Organization (FAO). ISNAR and the other CGIAR centers operate in collaboration with the NARSs,
their regional organizations and other advanced research organizations.
Following the expansion of the CGIAR at the beginning of the 1990s and after a series of funding
shortfalls, various mergers and attempts at consolidation were considered, including a suggestion
Introduction
CGIAR
Poverty
Environment
Food security
'NARS1
/ Regional ', •
organiza- AROs
tions .-•' x '•.
Key: strong link
very weak link
Figure I: ISNAR as a CCIAR center in the world of development
by the TAC that ISNAR be merged with the International Food Policy Research Institute (IFPRI) and
the International Water Management Institute (IWMI), since all three are mainly concerned with
research on public policies and institutions. However, prior to the 1997 external review of ISNAR, a
system-wide internal study was conducted on the future role of the CGIAR in the development of
NARSs (TAC/CGIAR 1996). The report concluded that there was very little overlap between the ac-
tivities of ISNAR and IFPRI, but recommended that all of the CGIAR centers should work more closely
together. It also identified a need for increased research on institutional development (particularly
with respect to providing generic tools for NARSs and in encouraging regional organizations), and
recommended that ISNAR take the lead in these efforts.
Since the 1996 report, a number of more recent CGIAR studies have addressed NARS-related issues
of direct relevance to ISNAR. In 1998, for example, the third system review of the CGIAR stated that:
The panel recommends that the CGIAR launch a special collaborative program to strengthen the
capacity of NARSs for policy research and formulation in countries where inappropriate public
policy is the major cause of a wide gap between potential and actual agricultural productivity.
Capacity building in policy research should cover economic policy, as well as environmental, science
and technology research policy.
A subsequent study of CGIAR priorities and strategies for resource allocation during the period
from 1998 to 2000 resulted in a number of suggestions regarding research aimed at institution-
strengthening and also explicitly addressed the importance of NARSs and the role of ISNAR (TAC
2000). The following statements are particularly relevant:
• Additional resources should be made available to ISNAR to undertake a more comprehensive
analysis of NARSs strengths and weaknesses to form the basis of a proposal for inter-institu-
tional collaboration. Studies on successful and unsuccessful examples of institution-building
activities should be carried out by ISNAR in collaboration with major donor agencies and/or
with selected NARSs.
• There should be closer collaboration between CGIAR centers in all facets of institution-strength-
ening support for NARSs, particularly between the other centers and ISNAR. Joint projects on
ISNAR Research Report 25
areas such as setting research priorities, program planning and evaluation, and project prepa-
ration and budgeting could be developed under the umbrella of a system-wide initiative on
institution strengthening.
• Generic, methodological tools that can be used by other organizations and consultancy firms
should be developed and disseminated, freeing resources for REJD activities. ISNAR should help
strengthen emerging regional groupings to facilitate the centers' efforts at strengthening insti-
tutions and encourage the participation ofNARSs in CGIAR priority setting.
• The information services provided to NARS partners should be distinguished from those given to
the centers' own scientists and those provided for donors and the public. A clear differentiation
is also needed between advice on the management of research in general and that on research
related to mandated commodities.
Adapted from TAC 2000
Such comments legitimized the continuing role of ISNAR as a service organization operating within,
and working closely with, the other CGIAR commodity and natural resource centers. However, like
the other centers, ISNAR must continue to adapt to CGIAR changes and actively seek donor support
in an increasingly competitive environment.
Recommendations of the 1997 EPMR
The EPMR of 1996-97 recommended that ISNAR should
• define its niche more clearly
• shift towards the research-based end of the service spectrum
• increase its capacity-building efforts
• base its operations on a broad concept ofNARSs
• focus on 'weaker' NARSs and collaborate with 'stronger' ones
• develop a clearly defined strategy with respect to NARSs.
The EPMR report was subsequently reviewed by CGIAR/TAC, which elaborated on the above recom-
mendations as follows:
• ISNAR should define its niche more clearly. This will involve strengthening the research base of
its services, enhancing knowledge relating to policy, and making better use of strategic alli-
ances.
• ISNAR should focus more on capacity building and increase its efforts in this area, e.g. by link-
ing research and research-based services, strengthening capacity building efforts in and around
NARSs (particularly in the management field) and enhancing its staff and skills mix to better
meet future needs.
• ISNAR should operate on the basis of a broad concept ofNARSs which will require, amongst
other things, supportive partnerships and facilitating the functioning of regional organizations.
• ISNAR should focus on weaker NARSs and collaborate with stronger ones as needed and within
the context of a clearly defined strategy.
The changing environment of the NARSs
The effectiveness and impact of ISNAR depends not only on its capacity to respond to changes in its
own operating environment, but also to changes in the environment of its client NARSs. The latter
are currently facing major challenges following various funding crises in the 1990s. Byerlee (1998)
reviewed the evolution of investment in NARSs and highlighted the need for a new paradigm to
respond to these funding difficulties. The main elements of Byerlee's review, the suggested new
paradigm, and other relevant developments can be summarized as follows:
• Current decline in public funding for agricultural research
• A need for pluralistic institutional structures
Introduction
Increased dependency on donors (especially in Africa)
Increasing decentralization of domestic administrative, financial and political power
Growing involvement of the private sector in agricultural research
A need for new research funding mechanisms
Various (as yet unmet) challenges facing public research organizations
Increasing importance of global scientific linkages
Increasing growth of subregional organizations (SROs) and networking
Increasing separation of funding and service provision
ISNAR's changing environment
As a CGIAR or Future Harvest Center, ISNAR either derives benefit from (or bears the cost of) mem-
bership of the system, depending on circumstances. In the period from 1997 to 2001, the CGIAR
experienced some major changes in financing, governance and organizational structure. With regard
to funding, the CGIAR centers faced both increased competition from other organizations and a
general reduction in donor support. Against this background, the importance of effective NARSs
has been repeatedly stressed, particularly as developing countries have become more important to
the CGIAR. NARSs now have a stronger, more equal voice in the main forum of the group, and many
have emphasized their commitment by becoming donors to the CGIAR centers. A further important
development was the formation of the Global Forum on Agricultural Research (GFAR) and the emer-
gence and strengthening of the role of SROs. As discussed below, ISNAR has played a key role in the
establishment of some of these organizations.
In response to these various changes, ISNAR has developed a new strategy, known as the Road Map
(ISNAR 2001). This addresses the realities of the rapid changes in the operating environment of
NARSs by broadening the target and scope of ISNAR's programs to focus on institutional innova-
tions. It also acknowledges the need for a better understanding of the ways in which various sec-
tions of civil society interact to help the rural poor. Thus the scope of ISNAR's work has expanded to
include the interaction between agriculture and related disciplines, the influence of the poor on
private and public research agendas, and the best ways of disseminating research findings to those
in greatest need. The evolution of ISNAR's medium term projects is outlined in Figure 2.
1997
Program I: Policy and
Systems Development
Themes:
• Policy development
• Research system 1998-2000
development 2002-2006
• New challenges 18 Medium Term
Projects (MTPs) Consolidation 6 MTPs and
Program 2: Management to 12 MTPs Road Map
(Annex 4)
Themes.
• Managing
research programs
• Managing
resources for research
• Managing
research organizations
J
Figure 2: The evolution of ISNAR's medium term projects (MTPs) (1997-2002)
ISNAR Research Report 25
Structure of the report
Chapter 1 is comprised of a review of the pertinent literature and a description of the conceptual
framework on which this impact assessment is based. Chapter 2 outlines the methods used, and
Chapter 3 presents the survey results arranged according to output category. Subsequent chapters
discuss the implications of these findings, in some cases adding supplementary data from the sur-
veys. Chapter 4 addresses ISNAR's niche and priorities, Chapter 5 the constraints affecting ISNAR's
impact, and Chapter 6 the institutionalization of impact assessment at ISNAR. Chapter 7 synthe-
sizes the team's overall view of the impact of ISNAR's work. References and detailed Annexes (on
accompanying CD) complete the report.
Chapter 1. Conceptual Framework
I. I The role of ISNAR as a CGIAR center
Most of the CGIAR centers carry out development-related technical research on crops, livestock
(including fish), and natural resources (including genetic resources). The only two exceptions are
ISNAR and IFPRI. ISNAR was created by the CGIAR in 1979 with the explicit aim of strengthening
national agricultural research institutions and systems (NARIs and NARSs), which are the principal
clients for, and users of, the new technologies and information produced by the other CGIAR centers
and similar advanced research institutions (ARIs). ISNAR is a relatively small institute, with a total
of 80 staff (about 40 of whom have been recruited internationally), working in some 40 countries
with an annual budget of approximately US$ 9 million. It works in partnership with others to pro-
mote sustainable institutions and to reinforce national capacities for agricultural research.
In this role, ISNAR has been (and still is) involved in developing new tools and methods for improv-
ing the management of research related to agriculture and natural resources (ANR). ISNAR has
collaborated with various NARSs in research work, the provision of advisory services and training,
and in documenting and disseminating practical experience and good practice. ISNAR is thus a
service-based research organization whose work is to a large extent demand-driven. Its primary
guiding principle is that institutional innovations are needed in NARIs and NARSs in order to inter-
nalize improved practices and to enhance research success. In other words, research organizations
should be encouraged to redesign their research processes and institutional procedures so as to
maximize the production of relevant technologies and scientific knowledge that will eventually
improve human well-being.
More generally, ISNAR is involved in improving the ways in which research resources are mobilized
and allocated by addressing the processes of technology development and dissemination. Many
NARIs need assistance in addressing the various challenges resulting from regional and global changes
and in developing administrative mechanisms for reallocating financial and human resources to
those areas expected to produce the greatest social benefit.
1.2 Impact assessment: Perspectives from the literature
The aim of this report was to assess the impact of ISNAR's work over the 5-year period from 1997 to
2002. In the literature, the term 'impact' is used in many different ways. For example, it is some-
times taken to mean any effects (both intended and unintended) that can be attributed to a specific
action. In other cases, the concept is used in a more restrictive manner, such as 'the long-term
outcomes of development programs on the people, economy, society or environment' (Kumar 1995),
or 'the ultimate effects on the country or organization' (DANIDA1994). Impact can also be identified
by behavioral changes in the target population.
The process of assessing or evaluating impact is subject to a similar variety of definitions and
interpretations. Rossi and Freeman (1993, p.185) defined the purpose of impact assessment as 'es-
tablishing whether or not an intervention is producing its intended effects'. In recent years, the
concept has been extended to encompass the impact of research on goals such as food security, pro-
tection of the environment and poverty alleviation (Cracknell 1996; Pingali 2001).
ISNAR Research Report 25
Horton (1990) classified technology into two broad categories - production technology and re-
search and development (R&D) technology. The latter refers to organizational strategies and the
methods used by R&D institutions to conduct their work. R&D technologies thus lead to institu-
tional impacts - i.e., effects on the capacity of R&D institutions to generate and disseminate new
production technologies. R&D technologies are a necessary precondition for enhancing the devel-
opment of production technologies: the two are complementary. Impact analysts are thus increas-
ingly using more comprehensive frameworks, encompassing intermediate products, as well as the
direct products of R&D activities and their effects on the ultimate beneficiaries (including attain-
ment of broader development goals) (Anandajayasekeram and Martella 1999; Maredia et al. 2000).
If the output of an R&D activity is a tangible technology, such as a new crop cultivar, the researchers
responsible for it are identified as real innovators, and the impacts of the technology can easily be
traced as the variety is adopted by the target population. In many cases, however, researchers
produce less obvious products, such as information, procedures or advice, which others use as
inputs in a much broader innovation process. Thus, although the researchers contribute to the
process, others may contribute even more. The innovation process - which is basically a social one
- involves many different interacting factors in addition to agricultural research. Politics, cultural
traits, social conditions, economic interests and the requirements of the technology are all amalga-
mated in a process of developmental change. In such cases, attributing broad development impact
to individual factors poses a significant challenge, particularly in the case of the research contribu-
tion, which typically occurs in the early stages of the innovation process, at some distance from the
development interface.
Assessing impact is particularly difficult in the case of institutions such as ISNAR, whose direct
outputs are intermediate products (i.e., R&D technologies), that contribute to the relevance, effec-
tiveness, and efficiency of the production technologies that are produced elsewhere. ISNAR's col-
laborative activities and the resulting R&D technologies are expected to influence the performance
of NARIs, but ISNAR itself has no direct impact on the target organizations' clients (i.e., the farm-
ers), or on broader economic, social or environmental variables. The chain of causal events is too
long and complex, and the variables affecting ultimate outcomes are too numerous, to permit the
clear identification and measurement of the impact of specific interventions on the latter variables
(Biggs 1990; Rossi and Freeman 1993). In order to understand how agricultural research (especially
that aimed at developing R&D technologies) contributes to economic and social development, a
thorough understanding of the process of innovation is needed. Institution building and institu-
tional innovation are interactive, nonlinear processes in which one might expect more 'gaps' in
attribution than causative straight lines (Salomon and Engel 1997). They also involve a diversity of
participants and a more or less continuous two-way flow of information.
Institutional innovation and capacity building can have significant positive impacts. Dollar and
Pritchett (1998), for example, pointed out that developing countries with sound policies and high-
quality public institutions have grown much faster than those without: 2.7% per capita per year
compared with -0.5%, respectively. In order to properly assess the contribution and impact of ISNAR
within this general picture, it was necessary to determine the extent to which ISNAR has influ-
enced institutional reform processes and institutional capacity - and hence the development of
tangible technologies.
Against this background, all available methods and procedures were reviewed in order to develop a
conceptual framework for analysis. The commonly used conventional methods for estimating rates
of return (i.e., econometric methods and economic surplus approaches) (Norton and Davis 1981;
Maredia et al. 2000) were not considered appropriate because nonresearch activities (such as train-
ing, networking, and advisory services aimed at improving institutional performance) are difficult
to accommodate within economic surplus models. To date, little methodological and practical work
has been conducted on assessing the economic impact of such nonresearch products (although
some exceptions are provided by Avila et al. 1983 and Cascio 1989). Maredia et al. (2000, p.l 7) reported
Chapter I Conceptual Framework
that no one has so far been successful in ascribing an economic value to the outcomes of capacity
building activities. They suggested that multidisciplinary teams involving specialists in economic
impact assessment, institutional analysis, and organizational science will be needed to explore
new and more useful ways of examining and understanding capacity development and the role of
organizational innovation in agricultural research and development.
A useful starting point for deriving an analytical framework for empirical studies is the organiza-
tional impact assessment framework originally proposed by Anderson et al. (1995) and subsequently
modified by Mackay et al. (1998) and Horton and Mackay (2001). This framework views organiza-
tional performance as a function of the various elements that contribute to the organization's
motivation, capacity, and external environment. Another approach relevant to this study is the
institutional analysis and development (IAD) framework proposed by Elinor Ostrom and her
colleagues at Indiana University, USA. This framework views collective outcomes such as efficiency,
equity and sustainability as arising from the interactions of three major factors: the physical at-
tributes of the system (such as goods and services), the motivation and capacities of those involved
in the system, and the institutional (or other) rules governing the patterns of interaction between
the different elements of the system. Gerrard (2000) argued that understanding the institutional
'rules of the game' can help explain why past efforts that focused primarily on improving the
performance of public-sector organizations generally failed to produce sustainable agricultural and
rural development. He also argued that although it is possible to improve the performance of a
system by changing any of the above three factors (or a combination of them), changing the rules of
the game is often the easiest approach.
There are other approaches to impact assessment. The comprehensive framework used in a number
of empirical studies in Eastern and Southern Africa used multiple criteria to analyze several aspects
simultaneously (including intermediate products, direct products, and field-level impact)
(Anandajayasekeram et al. 2002). Another analytical framework described in the literature is based
on 'program theory' or the 'theory of action'. Since the mid-1980s, many have argued that evalua-
tion should be driven not by the theories and methods of particular scientific disciplines, but by the
theories and assumptions underlying the particular program being evaluated. Program theory re-
fers to the set of beliefs on which an action is based and has been described as a 'plausible and
sensible model of how a program is supposed to work' (Bickman 1987, p.5). It identifies 'program
resources, program activities, and intended program outcomes, and specifies a chain of causal as-
sumptions, linking program resources, activities, intermediate outcomes and ultimate goals' (Bickman
1987, p.78).
More recently, Ryan (2002) made a distinction between 'process benefit' and 'socioeconomic im-
pact' in assessing the impacts of socioeconomic research. The former refers to institutional reform
or organizational capacity development, while the latter refers to the impact of the institutional
capacity on broader developmental goals. Babu and Mthindi (1995) also separated the benefits of
policy research into pre- and post- decision-making benefits. The first involves improved processes
related to capacity building and institutional strengthening, whereas the latter are primary and
secondary impacts of the emergent policies. It could be argued that while process benefits are
necessary (but not sufficient) outcomes indicating that ISNAR has been effective, socioeconomic
impacts are both necessary and sufficient.
The methods and procedures adopted in this study were intended to combine the best practices
from the various frameworks and procedures discussed above. Most of the activities of ISNAR are
collaborative in nature, and the outputs are intermediate products. Target individuals and organi-
zations have to apply and use these outputs to generate tangible technologies (production tech-
nologies) that will have field-level impacts on the ultimate beneficiaries of the research. ISNAR's
contributions and collaborative activities may result in positive impacts on the operational envi-
ronment, organizational motivation, and capacity of collaborating institutions (the direct clients),
which in turn may influence the performance of those organizations. This improved performance
ISNAR Research Report 25
should have a direct influence on the effectiveness, efficiency, and relevancy of the technologies
that are developed, and the adoption of those technologies by the ultimate beneficiaries will result
in the desired developmental impacts (Figure 3). Thus ISNAR's impact is ultimately a reflection of
the impact of NARSs/IARCs on target beneficiaries. However, the attribution and measurement prob-
lems implicit in this impact chain pose major challenges for any impact assessment system.
Input from Inputs from
collaborating Inputs from other
institutions ISNAR collaborators
Research, advisory services, workshops,
Collaborative documentation, strategy development, etc.
activities
Tools and methods, recommendations
(reports), trained professionals, training
Outputs modules, publications, master plans, etc.
Changed behaviour of beneficiaries and
Immediate organizations
outcomes
Organizational performance: quality,
Intermediate relevance, effectiveness, efficiency,
outcomes sustainability
/ ' Attribution
»j gap and
\weak casualitiy Field-level impact or developmental
Ultimate impact (micro and macro) effects on
outcome income, poverty, food security, nutrition,
(impact) gender, environment, biodiversity, etc.
Figure 3: ISNAR's impact chain
The typical impact chain (Figure 3) starts from the set of inputs and activities of a project/program
and proceeds to the most highly aggregated development results, such as poverty reduction, food
security, environmental protection, etc. The chain also specifies all the main intermediate steps,
namely the activities of a project, its outputs, the uses that others make of these outputs, direct as
well as possible indirect effects, and the eventual impact of the outputs on the ultimate beneficiar-
ies. Output, outcome, and impact generally occur sequentially and become more difficult to iden-
tify, measure and attribute as one moves from outputs to impact. The term 'output' refers to the
results of program activities (i.e., goods and services produced by the set of collaborative activities).
The outputs of ISNAR's activities are trained individuals, procedures, methods, and tools developed
for planning, priority setting, and performance assessment, the effective use of information tech-
nology and biotechnology, established networks and partnerships, new training materials, publica-
tions, reports, papers, etc. The term 'immediate outcome' refers to the first-level effect of these
outputs, i.e., the observed or documented behavioral changes in those directly affected by the
project or program. The term 'intermediate outcome' refers to the benefits and changes resulting
from the application of the outputs. Outcomes are thus measures of the use made of project out-
puts by clients and partners. They reflect the value placed on the intermediate products which are
used to inform clients' managerial decision-making processes.
In order to produce an outcome, the project or program has to cause a change in behavior. The
impact of a particular program could therefore be assessed by trying to identify and document
10
Chapter I. Conceptual Framework
expected changes in the attitudes, knowledge, perceptions, and decisions of those targeted by the
program. Immediate and intermediate outcomes can often be measured and documented directly.
Such an approach obviously involves identifying the various clients of the program and the ways in
which their behavior is expected to change. Observation and documentation of such short-term
changes then provides evidence that these impacts can be attributed to the program.
The 'ultimate outcome' (or impact) refers to the measurable effects of all outputs and outcomes on
the well-being of the ultimate beneficiaries of the R&D effort, namely the poor, those with insecure
food and nutrition, and/or the environment. Often the ultimate outcomes are closely linked to
sectoral, regional, and/or national developmental goals. Since there is often a considerable time lag
before the realization of outcome and impact, proxies or partial indicators may provide a practical
means of assessing field-level impact.
The four products of ISNAR's collaborative activities - outputs, outcomes, changes in organiza-
tional performance, and eventual impacts on welfare - are produced sequentially and become pro-
gressively more difficult to identify, document, measure, and attribute. In any empirical impact
analysis the two basic issues that must be addressed are those of causality and attribution. The
latter remains one of the major challenges facing impact studies, and this is especially true for
cases such as ISNAR, where partnerships and collaborations are central to most activities. Many
ISNAR outputs are joint products produced with significant inputs from partner organizations (i.e.,
NARIs and NARSs); as the latter mature, they become the major partners in the process, so that
attempts at apportioning project benefits may risk damaging good relationships. It is better, there-
fore, for investors and assessors to seek evidence of joint impacts, while attempting to ensure that
the inputs and contributions of all partners are appropriately acknowledged. The nature of each
contribution (e.g. discovery, development, or facilitation) should be clearly specified.
A number of factors influence the performance of an R&D organization (Figure 4). Those that could
be influenced by ISNAR through its collaborative efforts include leadership, organizational structure,
Organizational
structure
HR motivation
incentives
Production technology
Resource management*
(human and financial) R&D technology
Available resources
Modalities of operation* human
planning, priority setting financial
monitoring and evaluation
Policies and \
strategies*
* Indicates points at which ISNAR could, in principle, have an effect.
Figure 4: Factors affecting the performance of NARSs
ISNRR Research Report 25
linkages and networking, resource management, planning and priority setting, monitoring and
evaluation, policies and strategies, resource availability (human and financial), motivation and in-
centive systems. In order to take these various elements into account, and to reduce the level of
uncertainty surrounding attribution, multiple lines of evidence were sought in trying to establish
ISNAR's outcomes and impact. Such an approach assumes that, while individual pieces of evidence
may not themselves be very compelling, a large set of different and complementary evidence is
much more convincing.
12
Chapter 2. Methods
The study was conducted in accordance with the conceptual framework outlined in the previous
chapter, using both primary and secondary data. The latter were obtained from various reports and
program files, including internal program reviews, final reports from the 18 medium term projects,
(MTPs) external evaluation reports, and syntheses from the Participant Action Plan Approach (PAPA)
used by the training program and the biotechnology group. Additional information was collected
from interviews, structured (targeted) surveys, country-level case studies, and various secondary
analyses (including meta-analyses).
For primary data, an attempt was made to obtain the views of a broad spectrum of stakeholders,
including government agencies, the private sector and various representatives of civil society, in-
cluding nongovernmental organizations (NGOs). In order to assess the general perception of ISNAR
by its global beneficiaries, developing countries were grouped into three broad categories (1) cur-
rent collaborators (i.e., those with whom ISNAR was actively involved during the review period), (2)
historical collaborators (those with whom ISNAR had worked in the past, but not during the review
period), and (3) 'candidate' countries (those with whom ISNAR has never worked but which would
seemingly benefit from such a collaboration). There were 42 countries in the first category, 63 in the
second, and 14 in the third. From these, a sample of 315 potential stakeholders (including public
sector NARIs and private entities, learning institutions, NGOs, and producer organizations) was
selected for a global attitude survey designed to collect perceptions and views of ISNAR. The first
level country selections were made using systematic sampling procedures and the individuals/
groups within those countries were selected using purposive sampling.
Lists of donors and direct collaborators in NARIs were identified from the final reports of the MTPs,
ISNAR internal documents, and by consultation with project leaders. The sample used to assess
training materials developed by ISNAR was drawn from a list of those downloading computer-
based training modules, while the alumni survey was based on a list of all trainees who had at-
tended five selected training workshops. Unfortunately, a few individuals received up to three
questionnaires, having been selected in their roles as NARI managers, direct collaborators, and
former trainees; some were also selected in the survey of publication users and hence received yet
another questionnaire. Such overlap may have contributed to the relatively low response to the
various surveys.
In addition to the above, a number of e-mail based surveys were conducted to obtain further per-
ceptions of ISNAR's achievements, impacts, and constraints from an additional sample of its clients.
Furthermore, since the impact of various carefully selected outputs (such as research tools, reports,
books, etc.) was of particular interest, an inventory of ISNAR's output for the review period was
compiled (based on the final reports of the 18 MTPs) and various stakeholders were surveyed for
their responses to those outputs. The stakeholder categories included actual and potential clients
(for an attitude survey), NARI managers (for their perceptions of general output and outcomes),
direct collaborators and donors (for their perceptions of specific projects), trainees, supervisors of
trainees, managers of SROs, and information users (including those downloading training mod-
ules). In these cases, purposive sampling procedures were used to select the survey participants.
One of the major difficulties encountered in the e-mail based surveys was a lack of accurate current
addresses: the ISNAR central database had incorrect addresses for 21.6% of the global perception
sample, despite efforts to keep the information up to date. The overall percentage of incorrect
addresses for the team's survey efforts was 16.4 (27.6% for the sample of NARI managers).
ISNAR Research Report 25
The various instruments used to collect information for this study (summarized in Annex I)1 are
omitted from this report for reasons of brevity but are available on request from the ISNAR central
database. The types of information collected from the various groups of respondents are summa-
rized in Annex 2.
The information outlined above was complemented by a separate survey of 500 ISNAR publication
users (of which 82 responded: see Section 3.3) and a number of carefully selected country-level case
studies conducted by independent consultants. The use of independent assessors increases objec-
tivity and lends credibility to impact assessments, although in this case only a small number of
projects and programs could be assessed in this way because of the costs involved. Discussions
with the leaders of the MTPs helped to identify countries in receipt of significant inputs from ISNAR,
countries where most of the objectives were accomplished, and countries where achievements
were less than fully satisfactory. Several MTPs involved activities in the same country, and for the
sake of balance, relevance and effectiveness, projects deemed 'failures' were examined as well as
those considered 'successful'. The criteria used in selecting the countries for cases studies included
concentration of activities and geographical balance. Originally, 13 countries were identified for
independent case studies on the basis of concentration of activities. However, due to resource
constraints, only eleven were eventually selected, namely Argentina, Cuba, Dominican Republic,
Indonesia, Mexico, Pakistan, South Africa, Sri Lanka, Uganda, Vietnam, and Zambia. In order to
ensure uniformity in coverage and reporting across these countries, the consultants were provided
with notes on ISNAR activities, suggestions for consultation and review (based on information
provided by project leaders), and a suggested outline for their final report.
A total of 165 e-mail questionnaires were completed and returned, i.e., an overall response rate of
17%. The returns for individual stakeholder categories were: 27% for direct collaborators, 32% for
donors, 22% for the advanced research institutes (ARIs) (including IARCs), 63% for the SROs, 14%
for NARI managers, 13% for the global NARSs attitude survey, 9% for training module downloaders,
and 16% for the trainees and supervisors of trainees. The relatively low level of response could be
attributed, among other things, to the limited access to e-mail in many developing countries, espe-
cially in sub-Saharan Africa. Attempts were made to follow up with telephone reminders and postal
enquiries, but the limited time available to complete the surveys (approximately 6 weeks in total)
placed further constraints on these endeavors. Although the information obtained from these surveys
was obviously less than complete, when all of the data collected by both ISNAR and the assessment
team are considered they appear to be sufficiently robust to allow reasonably sound conclusions to
be drawn.
All of the Annexes to this report can be found on the accompanying CD on the inside back cover.
14
Chapter 3. ISNAR's Program Areas, Outputs
and Assessments
As outlined in the Introduction, ISNAR has undergone a number of transformations in terms of
structure and program content since the 1997 External Program Review (Figure 2), at which time
there were only two major programs - the Policy and Systems Development Program and the Man-
agement Program. Immediately after the review, ISNAR embarked on a new strategy and reorgan-
ized its activities under 18 MTPs.1 Most of ISNAR's work during much of the subsequent review
period was motivated by four main considerations, namely the diagnosis, planning, and implemen-
tation paradigm, the separation of research, services and training, 'critical factors', and NARSs as
coordinated and manageable entities (Janssen 2002). However, in 2000, in response to economic
constraints and various other changes, ISNAR refocused and consolidated its activities into 12 MTPs,
combining some previously separate projects and canceling or concluding others.
At the end of 2001, ISNAR once again reviewed its strategy and formulated the so-called Road Map
(ISNAR 2001), which adopted an innovation systems perspective and grouped ISNAR's activities
under six thematic areas (1) policies for institutional innovation in agricultural research, (2) linking
research organizations and stakeholders in a changing context, (3) learning for institutional inno-
vation, (4) management of new technologies for agricultural research, (5) building capacity to respond
to cross-sector demands, and (6) entrepreneurial partnerships to support agricultural research. The
criteria used in the rationalization process included ISNAR's comparative advantage (in terms of
alternative service providers and available resources), critical resource requirements, and relevancy
and strategic importance in the context of the changing operating environment.
ISNAR's activities are conducted through partnerships with a number of stakeholders. Such collabo-
ration has taken various forms, including research, the provision of advisory services, involvement
in training, and documenting and disseminating practical experiences and good practice. Many of
these activities are synergistic and iterative. For example, a research project may produce a policy
tool or method that forms the basis of a training workshop, that in turn leads to publications and a
training module. The major outputs of ISNAR's collaborative work are therefore research results
(contributions to knowledge), advice (given to specific clients such as NARIs and SROs), manage-
ment tools and procedures, trained individuals, training materials and various publications. The
achievements and impacts of ISNAR in relation to these major output categories are discussed in
the following sections, focusing mainly on the activities, outputs, and outcomes of the original 18
MTPs (since most of ISNAR's activities during the review period were completed under that frame-
work). Most of the 18 MTPs also included many subprojects, so that a very large number of projects
and programs were executed by ISNAR between 1997 and 2001. Trying to trace the outcome and
impacts of such a large number of activities would have been time-consuming and prohibitively
expensive. Hence major output categories for each of the 18 MTPs were used to assess performance
and impact.
3.1 Training
Within the broader framework of innovation systems as detailed in the Road Map (ISNAR 2001),
ISNAR has consistently emphasized a comprehensive approach to training for capacity building in
The title, objectives and geographical coverage of each of these projects are summarized in Annex 4 and their outputs
in Annex 5.
15
1SNM Research Report 25
NARSs. A conceptual framework was developed to reinforce ISNAR's strategic objective of improv-
ing access to knowledge which took into account (1) an initial assessment of partners' needs,
(2) the identification and/or development of appropriate knowledge to respond to those needs,
(3) continuous involvement of partners to ensure their ownership of- and readiness to use - knowl-
edge, and (4) identification of appropriate methods of sharing knowledge with many partners.
ISNAR has also encouraged the use of a training cycle, which requires regular assessments of train-
ing needs. ISNAR offers training through both formal channels (workshops) and informal approaches
(various 'hands-on' activities, combined with ongoing action research). Some training is offered
through the training unit (currently MTP3) and some through other MTPs. The major activities of
the former include training trainers (producing a multiplicative effect), training research managers
and researchers in the use of tools and methods related to agriculture and natural resource man-
agement, teaching training officers (e.g. to develop skills related to assessing training needs), de-
signing, implementing, and evaluating training programs, and developing training modules and
materials.
From 1997 to 2001, a total of 75 training events were conducted and over 1300 participants were
trained in various aspects of research management. The training unit alone conducted a total of 46
workshops involving 834 participants. The number of workshops increased from 5 in 1997 to 19 in
2001, with the number of trainees increasing from 67 in 1997 to 361 in 2001 (Table 1). Workshop
participants were drawn from Africa (46%), Asia (29%), Europe/USA (14%), and Latin America and
the Caribbean (11%). The fact that nearly half of the trainees came from Africa serves to emphasize
ISNAR's commitment to capacity building in the African NARIs.
Table 1: Number of ISNAR Training Workshops and Trainees* (1997-2001)
Year
Product 1997 1998 1999 2000 2001 Total number
Training workshops 5 6 5 II 19 46
Trainees 67 100 90 216 361 834
Source: ISNAR meta-evaluation
* Number trained through formal workshops by the training unit only
A number of training activities are offered regularly each year, while others are offered on demand.
The regular annual training programs include New technologies for agricultural research, Agricultural
research management training for program leaders, How to write convincing proposals and Facilitating
agricultural innovations.
Since 1998, the agricultural research management course has been delivered in Africa in partner-
ship with the IARC-NARS Training Group (INTG). This course includes training in the leadership and
management of research, formulating research programs, and planning, monitoring and evaluat-
ing research projects. So far, 78 research program leaders have participated in these workshops,
and of these, six individuals have been selected to act as trainers in future INTG events.
By the end of 2001, a total of 133 participants (75% from Africa) had attended the program for NARI
managers on How to write convincing proposals. Training in this area is particularly important in
view of the increasing competition for scarce national and international research funding and the
competitive grant schemes used by many donors and institutions for allocating research funds. The
course has also been adapted to meet the needs of other CGIAR centers as well as various NARS
competitive grant programs.
16
Chapter 3. ISNAR's Program Areas, Outputs and Assessments
The annual workshop on Facilitating the agricultural innovation process is also offered in partnership
with INTG and is targeted at NARS program leaders in sub-Saharan Africa. The objective of this
workshop is to increase the capacity and abilities of agricultural research program leaders in three
key areas of the innovation process, namely strengthening creative skills to promote innovation,
facilitating linkages between stakeholders, and stimulating participatory research so as to engage
both researchers and farmers in the innovation process.
The workshop on Managing biotechnology in a time oftransition, for participants from selected Asian
countries, was designed by ISNAR with financial support from the Government of Japan. The objec-
tive of this workshop was to provide key agricultural personnel with the opportunity of enhancing
their management and leadership skills in relation to biotechnology. The workshop was developed
primarily for senior research managers, research project or program leaders and those representa-
tives of NGOs with responsibility for biotechnology. A total of 74 individuals undertook this train-
ing during the 4-year review period.
A meta-evaluation conducted by ISNAR (Garcia 2002) summarized the final evaluations of 18 differ-
ent training workshops, in which participants were asked to rate their training on a scale of 1 to 5
(1 = poor, 2 = fair, 3 = average, 4 = good, 5 = excellent). The overall average rating was 4, with a
range of 3.8 to 4.8 (see Annex 6). In general, the training workshops were highly rated in terms of
relevance of content and presentation; only two workshops were rated below 4.
The same meta-evaluation also identified three major strengths of ISNAR's training activities, namely
the content and relevance of the workshops, the use of participatory methods (which recognize
and build on trainees' knowledge and experience), and the quality of the training materials. In
general, ISNAR's training workshops are designed and planned in collaboration with stakeholders,
which helps to ensure that they explicitly address the problems and needs of the clients. ISNAR's
partners and stakeholders also provide logistical, financial, and intellectual support for these train-
ing efforts, demonstrating considerable mobilization of outside resources by ISNAR. In general, the
training materials used are simple, easy to understand, and cover all of the workshop objectives.
Participants suggested revisions to only one training module, although additional materials were
recommended for two other workshops.
The most frequently mentioned weaknesses of the training workshops were problems associated
with time management, lack of (or inadequate) 'real life' examples, case studies and/or exercises,
limited use of field visits, and few training personnel with practical experience of the subject mat-
ter. Some participants also reported difficulty in grasping certain topics (e.g. logical frameworks in
research planning).
The ISNAR meta-evaluation also raised a number of issues with respect to workshop evaluation
that should be addressed. First, a standard evaluation instrument (questionnaire) should be used
for all workshops. Second, the questionnaire should include a section for evaluating the trainer's
performance. This will prove particularly useful if ISNAR increases the level of involvement of its
global associates in training activities. Third, in many instances, the 'objectives' listed in the evalu-
ation forms appear to be little more than the topics included in the workshops, rather than specific
objectives per se. Since most workshops are intended to impart skills, some method of appraising
specific skills (before and after training) should be included in the evaluation forms. In addition, the
present evaluation summary does not include information on either the gender or number of par-
ticipants in each workshop. In many of the cases examined in the study, the summary evaluation
reports for individual workshops were either missing or incomplete (i.e., not all of the information
collected was included in the summary). Garcia (2002) therefore recommended that an evaluation
summary should be completed for each training workshop, and that all trainers should assume
responsibility for the proper completion of the evaluation forms. Ideally, the entire set of evaluation
forms (including the evaluation summary sheets) should be revised and improved in order to facili-
tate subsequent comparisons and meta-evaluation processes.
17
ISNAR Research Report 25
For this report, five workshops were selected for further evaluation in terms of outcomes and impact.
These workshops were New technologies for agricultural research, Agricultural research management
training for program leaders. How to write convincing proposals, Facilitating agricultural innovations,
and Research network management. The objectives and skills imparted through these workshops are
summarized in Annex 7. Information of various types was sought from past trainees as well as
their immediate supervisors (see Annex 2). Of the 34 ex-trainees who responded, 18% were female
and 82% were male; 25 of the 34 had participated in the workshop on How to write convincing
proposals. The majority of the respondents were from Africa (26) followed by Asia (6), and finally
Latin America and the Caribbean (2). Of the supervisors who responded, 83% were from Africa.
On average, the responding supervisors had known the trainees for about eight years (range:
3-15 years). According to these supervisors, the main tasks performed by the trainees were re-
search, research management and extension, network coordination, and teaching and consultancy
services. The information collected from trainees and their supervisors is summarized below.
3.1.1 Impact on trainees
Ideally, the impact of training should be assessed in terms of improved technical skills and in-
creased capacity for research planning, management, and evaluation. Estimating the likely final
impact of skills-based training is a challenging task. Even at the conceptual level it is difficult to
define the limits of such impacts, since they are likely to be dispersed over many different activities
over time. The skills and work habits learned by a trainee may affect their own productivity - and
that of others - in a sequence of jobs over many years. Given the number of people trained and the
number and variety of skills imparted, measuring impacts of this nature was impractical. In this
study, therefore, qualitative indicators obtained from the trainees and their immediate supervisors
were used as proxy measures of the ultimate impact of the training.
ISNAR's training unit uses a tool called the Participant Action Plan Approach (PAPA) to check train-
ees' progress at least 6 months after the end of the training event. This approach is a participatory
strategy based on experiential learning theory (Kolb and Fry 1975; McCaffery 1986). In other words,
it is a learner-centered strategy involving active experience followed by a process of reviewing,
reflecting, and applying what has been learned. ISNAR's training unit used this approach to evalu-
ate the workshop on Agricultural research management training; 528 questionnaires were analyzed
(Odame 2001) and the results revealed that 361 action items were accomplished (Table 2); 167 action
items were abandoned or considered no longer relevant.
Table 2: Results of PAPA Analysis for the Agrieu.ltu.ral Research Management Training
Workshops
Level of impact
(number of action items achieved)
Area of training Individual Team Organization Total
Leadership in the management of research
(3 workshops) 44 46 4 94
Research program formulation
(4 workshops) II 59 24 94
Planning, monitoring, and evaluation
(4 workshops) 16 67 20 103
Financial management (4 workshops) 28 37 5 70
Total actions 99 209 53 361
Source: Odame (2001)
18
Chapter 3 ISNAR's Program Areas, Outputs and Assessments
The greatest impact was achieved within the research teams managed by program leaders. These
successes were generally related to improved communications, proposal writing, better organiza-
tion, improved management of program finances, and the application of tools concerned with pro-
gram formulation, planning, monitoring, and evaluation. Some changes were also noted at the
organization level in relation to adoption of new management tools and the preparation of success-
ful funding applications for new research programs. At the individual level, most successes oc-
curred in the area of leadership in management.
A PAPA analysis was also conducted for the four management courses on New technologies for
agricultural research: Managing biotechnology in a time of transition. Of the 96 participants who sub-
mitted an action plan, 38 responded to the questionnaire (response rate: 40%). The results are
summarized in Table 3.
Table 3: Summary of Impacts at Different Levels - Results of PAPA Analysis for the Course
on New Technologies for Agricultural Research: Managing Biotechnology in a Time
of Transition.
Impact level Achieved Not achieved Key examples of achievements
Individual participant 20 Improved management skills as scientist,
team leader or director
Studying information gained during course
Team 25 I I Sharing new information and skills with
team
Organizing internal workshops or seminars
using course materials
Organization 35 20 Applying priority-setting tools
Review research programs with regard to
biosafety/IPR
Initiate institutional biosafety/IPR policies
Initiate international collaboration
National level 12 Contribute to development of biosafety,
IPR guidelines and policies
Total 92 (68%) 43 (32%) Total number of action items = 135
Most impacts occurred at the organization level. The major constraints to achieving intended im-
pacts were lack of time, lack of certain skills (e.g. those related to priority setting), and the fact that
an intended action fell outside an individual's area of authority (particularly when institutional or
national-level changes were intended). At the institutional level, a lack of support from higher
management was the most frequently cited constraint. Overall, although 68% of the action items
were achieved, the analysis indicated a need for more active support on completion of the training.
When participants were asked to assess their skills on a scale of 1 to 5 (1 = very high, 5 = very
low), the average rating before the workshop was 3.24 (standard deviation = 1.07) while the rating
for the same skills at the time of the survey was 1.94 (standard deviation = 0.91). According to
supervisors, the average ratings for the same skills prior to and after the training workshops were
2.87 and 1.44, respectively.
19
ISNAR Research Report 25
Fifty-three percent of responding trainees had changed their careers since receiving their training
and believed that the training was at least partially responsible. Of the 18 respondents, 14 of them
had assumed increased responsibilities and one had completely different responsibilities. Increased
remuneration was reported by one respondent. Approximately 80% of respondents attributed vari-
ous professional accomplishments to ISNAR's training, including an increased number of publications
(N=2), the opportunity to attend conferences (N=4), additional external supervisory responsibilities
(N=8), the opportunity to undertake consultancy services (N=7), and development of successful
projects (N=14).
All responding supervisors stated that the job performance of the trainees had improved as a result
of ISNAR's training. Some of the benefits mentioned included an increase in the confidence and
helpfulness of the trainees, significant improvements in their knowledge of the subject area, im-
proved writing skills, and greater involvement in strategic planning, proposal writing, monitoring,
evaluation, and workshop facilitation. All supervisors confirmed that the trainees regularly used
the knowledge gained during training; 92% indicated that the training received was relevant to the
needs of their units, and 83% believed that the training received was relevant to their institution as
a whole. They also stated that project management had improved, more proposals had been ac-
cepted and funded as a result of the training, and that trainees had subsequently acted as facilitators,
supporting other staff in areas related to the training (e.g. research priority setting, monitoring,
and evaluation).
3.1.2 Impact on collaborating organizations
Sixty percent of the trainees who responded to the survey indicated that they were subsequently
involved in training their colleagues and others in the skills acquired in ISNAR's training work-
shops. Between them, 20 respondents trained a total of 1045 individuals, with the number trained
by each individual ranging from 5 to 450 (average = 52). ISNAR's training therefore resulted in
significant multiplier effects.
Seventy-four percent of respondents indicated that they had helped to transfer their newly ac-
quired knowledge of research management procedures to management levels beyond their own:
31% of these respondents claimed to have made such a contribution at the departmental level, 48%
at the institutional level, 17% at the national level, and 3% at the regional level.
Sixty-two percent of respondents felt that ISNAR's training had contributed to institution-level
changes such as the increased use of priority setting methods (as more and more people were
successfully trained in this area), strengthening of institutional management, better resource use,
more collaborative research, increased numbers of project proposals submitted and funded, and
increased use of logical frameworks for monitoring and evaluation. Evidence in support of these
claims was provided in the form of journal publications (1), conference papers (2), externally com-
missioned reports (1), institutional documents (8), training materials (8), and recommendations to
farmers (3).
All supervisors who responded to the training questionnaire indicated that trainees' subsequent
contributions in some way enhanced the performance of their units. The most frequently noted
improvements were in the operation and management of the units, fewer (but more focused) re-
search projects which attracted more funding, an increase in the quality of research projects and
proposals, and more effective team work. Such remarks reflect an improvement in both attitude
and performance on the part of the trainees. Fifty percent of respondents also indicated that there
were other improvements in the performance of trainees that could be attributed to ISNAR's train-
ing, including improved leadership and communication skills, and more systematic analysis of
problems.
20
Chapter 3 ISNAR's Program Areas. Outputs and Assessments
Participants in the proposal writing workshop were asked if they had written more proposals since
undergoing their training, and if so, how much of this increased output could be attributed to
ISNAR's training (using a scale of 1 to 6, where 1 = very high, 5 = very low and 6 = unknown).
Fifty-six percent of respondents stated that they had written more proposals; on average, they were
writing approximately three proposals per year, and felt that a significant part of their increased
output could be attributed to ISNAR's training (average rating = 2.2, range: 1 to 3). Participants
were also asked whether the success rate of their proposals (in terms of attracting funding) in-
creased after training; 44% of respondents replied in the affirmative, with 9 of them writing, on
average, 2-3 successful proposals per year. An average score of 2.4 (range: 1 to 3) was awarded for
the perceived contribution of ISNAR's training to this success. Further evidence of the effectiveness
of ISNAR's training in proposal writing is provided by the fact that ISNAR's self-training materials
and support led to the formation of 28 consortia who competed for the European Union's INCO-DEV
grants in 2001 (see Section 3.1.3); six of these consortia were successful in their bids (Annex 8).
When asked if they would recommend ISNAR's training to other members of their institution, all
but one supervisor responded positively (financial problems were cited as the reason for not recom-
mending the training). The most important reasons given for recommending the training were that
all staff needed skills in topics such as writing project proposals, research management, etc., and
that ISNAR training updates were essential for everyone.
3.1.3 Development of training materials
During the review period ISNAR developed and published a total of 56 training modules (41 in
English, 7 in French, and 8 in Spanish) for workshops lasting from 2 days to 2 weeks. Materials on
the same topic but in different languages were considered separate modules. These training modules
covered the following topics: priority setting in agricultural research programs, planning, monitor-
ing, and evaluation of research projects, information management for research, scientific writing
and presentation, research program formulation, financial management for research in agriculture
and natural resources, strategic planning, gender analysis for management of research in agriculture
and natural resources, training needs assessment and organizational constraints, participatory re-
search, leadership in the management of research, managing linkages between research and tech-
nology users, priority setting in factor-based agricultural research programs, facilitating linkages
among stakeholders in agricultural innovation systems, strengthening creative skills to promote
innovation, research network management, and writing convincing research proposals. A number
of 'mini-modules' were also included, for example, organizational learning and learning organiza-
tions, the organization as a learning laboratory, leaders in learning organizations, evaluation of the
main approaches to agricultural innovation, project cycle overview and creativity. A number of
these modules were made available on the Internet and on CD-ROM, and various distance-learning
materials were being developed. The training unit also developed a self-training package on
successful proposal writing for the European Union's program on International Cooperation with
Developing Countries (INCO-DEV). This module was originally developed with funding from the
African Development Bank in response to the needs of research managers in the NARIs of
10 francophone countries, and ISNAR now offers this module in three languages, namely English,
French, and Spanish.
3.1.4 Assessment of ISNAR's training modules
Survey participants were asked to rate training modules for content, usefulness, and presentation,
using a scale of 1 to 5 (1 = very good, 5 = very poor). The overall ratings for these three factors
were, respectively, 1.67 (range: 1 to 4, with three respondents giving a rating of 4), 1.57 (range: 1 to
5, with one respondent giving a rating of 5 and another a score of 4) and 1.78 (range: 1 to 4, with
only one respondent giving a score of 4). In general, most responses were positive. The team there-
21
ISNAR Research Report 25
fore felt that the materials developed for the modules were adequate for meeting the training
objectives.
This conclusion was supported by the results of routine assessments carried out by ISNAR during
the workshops, in which participants were asked to evaluate the training materials in relation to
the workshop objectives. Four modules were assessed in this way: The research project management
cycle: planning, monitoring and evaluation, Research program formulation. Financial management in
research on agriculture, and Leadership in management research. The ratings (using the reverse of the
scale generally used in this report) were 4.43, 4.35, 4.43, and 4.3, indicating that all four modules
were judged to be somewhere between 'good' and 'excellent'. In the project cycle workshop, par-
ticipants recommended that one module should be reviewed and improved to place more emphasis
on practical procedures, problems and solutions.
Since January 1998, various ISNAR training modules have been available via the Internet. When-
ever one of the modules (or sections of them), are downloaded, information about the user is auto-
matically registered on an electronic database. ISNAR's training unit conducted an analysis of these
data collected in two time periods: from January 1998 to May 1998, and from January 2000 to
September 2001. The analysis focused on the modules downloaded, the types of organization and
individuals using the modules, and the geographical distribution of the users (Odame and Nielson
2002) (Table 4).
Table 4: Internet Module Downloads: 1998-2001
Outcome End of 1998 By 2001
Downloads of ISNAR
internet modules
(number of users) 158 783
No. modules/materials
downloaded 532 2738
Geographical distribution Africa: 15% Africa: 24%
Asia: 14% Asia: 16%
Latin America/Caribbean: 15% Latin America/Caribbean: 2 1 %
West Asia/North Africa: 3% West Asia/North Africa: 8%
Central & Eastern Europe: 5% Central & Eastern Europe: 2%
European Union: 18% European Union: 2 1 %
USA/Canada: 25% USA/Canada: 8%
Australia/New Zealand: 5%
Type of organization Public (25%) Public (41%)
downloading Private (23%) Private ( I %)
Source: Odame and Nielson (2002)
The major findings of this analysis included the following:
• The number of users increased from 158 in 1998 to 783 by 2001, and the number of modules
downloaded increased from 532 to 2738 over the same period. During this time, the number
of users in Africa, Latin America/Caribbean, and West Asia/North Africa increased by 5% or
more.
22
Chapter 3. ISNAR's Program Areas, Outputs and Assessments
• During 2001, the three most popular modules were Planning, monitoring and evaluation, Strategic
planning, and Priority setting.
• Between the two study periods, there was a significant increase in public sector users (from
25% to 41%) and 'others' (5% to 16%), and a significant decline in the number of private
sector users (from 23% to 1%). The diversity of the user groups also increased.
For this report, a questionnaire was sent by e-mail to thirty individuals who had downloaded at
least one training module. Only 20% returned the completed questionnaire. The modules downloaded
by these respondents were Priority setting for agricultural research programs (5), Planning, monitoring
and evaluation of research projects (4), Scientific writing and presentations (3), Research program formu-
lation (4), Strategic planning (2), Gender analysis (1), Training needs and constraints assessment (1), The
research project management cycle: planning, monitoring and evaluation (1), and Financial manage-
ment in agricultural research (4). Respondents were asked to rate the various modules for content,
presentation, usefulness and their contribution to knowledge, again using a scale of 1 to 5 (with
1 = very high, 5 = very low). The average ratings for these four criteria were 2.16, 2.22,1.80, and
2.70, respectively. Most modules were given a score of 1 or 2 (with the exception of module 1, which
was given a very low rating (4) by one respondent. General comments by respondents included the
observation that the materials were useful, comprehensive, and easy to understand. Thus, in spite
of the limited response to the survey, there is evidence to suggest that the modules are of good
quality and form a useful contribution to knowledge.
The respondents used the modules for various purposes, including training, increasing their knowl-
edge, facilitating consultancies, and preparing research proposals. Two individuals indicated that
they had downloaded modules for later use. Suggestions for enhancing the usefulness of ISNAR's
training materials included translating them into different languages and using a wider variety of
contexts (not just African). Two respondents indicated that the module content could usefully be
updated.
Some of ISNAR's training materials have been adopted by other agencies - another example of the
potential multiplier effects of these products. For example, some of the material on planning,
appraisal, and monitoring is being used by a consortium of universities led by the University of
Pretoria. ISNAR has also granted permission for CIMMYT to use some of its training materials.
3.1.5 Enhancing the impact of ISNAR's training
When trainees were asked to identify ways in which ISNAR could enhance the impact of its training
activities, their suggestions included supporting national and regional workshops, providing train-
ing in other topics, sending updated training materials, providing information on calls for propos-
als issued by donors, and developing a tracer system to follow-up trainees. Supervisors of trainees
suggested offering more refresher courses, conducting training in the context of specific projects or
local conditions, providing support for regional and national training, implementing a good follow-
up mechanism, conducting surveys of training needs, and creating a forum for fostering periodic
interaction between trainees.
The meta-evaluation report (Garcia 2002) noted that, of the 27 workshops evaluated, 20 (74%) of
them had been held only once, while the remaining 26% had been held only twice (4 workshops) or
three times (5 workshops). Only four workshops run every year. Since it takes a significant level of
resources to design, plan, organize, and implement training courses and to develop and test
training modules, the assessment team believes that it would to be advantageous to more ag-
gressively promote those training workshops and modules that ISNAR has already developed.
This would make better use of the resources invested, broaden impact, and generate additional
resources to support new initiatives.
23
ISNAR Research Report 25
However, even if training is rated as 'good' during and after the workshop, there is no guarantee
that the attitudes of participants will be permanently changed, that their new skills will be applied,
or that they will improve the performance of either their parent organizations or the R&D system
as a whole. The PAPA analysis is a good starting point for addressing this issue and for collecting
evidence of the outcomes of ISNAR's training. However, analyses of this type are not conducted for
all of ISNAR's training activities, and in any case the process ends 6 months after training. In order
to better document and assess the impact of ISNAR's training activities, PAPA analyses should
therefore be complemented by some sort of tracking system (tracer studies) that will allow
assessments to be conducted at a deeper level and to continue for longer.
Finally, there is a perception among some members of ISNAR's staff that only those working in the
training unit are involved in the development of training modules. Furthermore, not all staff in-
volved in formal and informal training actually use the modules that have already been developed.
It is important that all of ISNAR's staff should 'own' the materials produced and that these materi-
als should reflect the expertise and experiences of all of ISNAR's technical staff.
3.2 Management tools and procedures/service and advice
In its role as a service agency, ISNAR produces two levels of outputs. First, it develops generic
research management tools and procedures with wide applicability. These tools are described in
Section 3.2.1. Second, ISNAR provides service, advice, and other support to NARSs in order to pro-
mote institutional development and the formulation of improved policies.
These twin components - tools and advice - operate synergistically, in that the management tools,
together with the training modules (see above), form the basis of much of ISNAR's service and
advice to NARSs. The tools are generally derived either from collaborative research projects or from
service/advisory activities. Furthermore, ISNAR has also been active in the establishment of the
Forum for Agricultural Research in Africa (FARA) and has seconded a senior officer to support the
NARS global secretariat hosted by FAO. This secretariat has assisted regional and subregional or-
ganizations in all developing regions, and has helped to strengthen the voice of the GFAR.
Tool production and service provision account for a large proportion of ISNAR staff and budgetary
resources. Of the 18 MTPs, one (MTP13) was specifically designed to strengthen NARSs via improve-
ments in diagnostic services, planning, and the facilitation of institutional development. The out-
puts from this project were generated via service missions and collaborative work organized under
17 subprojects in 15 countries in Africa, Asia, and the Middle East (Annex 4). Activities in Latin
America and the Caribbean were included in the New Paradigm project (MTP 5), while Central Asia
and the Caucasus were covered by MTP 14. An assessment of these services in 10 collaborating
countries forms part of the case studies conducted for this report (Annex 12).
Service-based outputs and tools were also derived from global projects addressing particular themes:
globalization (MTP 1), agro-industrial and environmental issues (MTP 3), governance/accountability
(MTP 4), impact assessment (MTP 6), biotechnology (MTP 7), information/communications (MTP 8),
environment (MTP 9), management tools (MTP 10), management information systems (MTP 11),
gender (MTP 12), and global information (MTP 17). These themes are reflected in the outputs listed
in Annex 5 and in the generic tools and procedures listed in Table 5.
The survey of NARI managers conducted by the assessment team provided primary data on ISNAR's
services and advice and on the 26 procedures and tools produced by ISNAR. Twenty-three individu-
als responded to the survey, 15 from Africa, 6 from Asia, 1 from Latin America and the Caribbean,
and 1 from Oceania. Of these, 13 indicated that their organizations had not conducted specific
collaborative work with ISNAR during the review period (i.e., 1997-2001).
24
Chapter 3. ISNAR's Program Areas, Outputs and Assessments
3.2.1 Generic tools and procedures
The number of survey responses obtained with respect to the 26 individual tools or procedures
ranged from 1 to 8, and averaged 3.3 (Table 5). Since 10 of the 26 tools received only one or two
responses, little, if anything, can be concluded from the average scores for individual tools. The
highest response rates were obtained in relation to the tools for management information systems
(MTP 11) and project planning, monitoring and evaluation (MTP 18).
Table 5: Assessment of Generic Tools and Procedures: Summary of Survey Responses
Generic tools/procedures generated by ISNAR Average score No. of responses
1. A methodology for mapping poverty at the village level 2.0 I
2. A methodology for incorporating socioeconomic data for
targeting antimalana measures in vulnerable communities 2.0 1
3. Methodology for assessing institutional impact 1.6 3
4. Methodology for assessing impact of information 1.5 3
5. Methodology for assessing the impacts of training 2.6 2
6. Guidelines for evaluating ecoregional programs 1.9 3
7. Methodologies for priority setting in natural
resources management 1.6 4
8. Methodologies for priority setting in livestock research 1.5 3
9. Priority setting using the analysis hierarchy approach 1.6 5
10. Management guidelines for improving university
contributions to national agricultural research 1.8 5
11. INFORM-R: a management information system 2.0 8
12. Strategic planning procedures 1.6 5
13. Project management tools and resource kits 1.6 4
14. Guidelines for building partnerships with agroindustries 2.6 2
15. Training needs assessment 2.0 5
16. Proposal writing 1.5 5
17. Scientific writing and presentation 1.6 4
18. Planning, monitoring and evaluation of research projects 1.5 6
19. Gender analysis 3.0 1
20. Research program formulation 1.7 4
21. Methods for planning effective linkages 2.4 2
22. An approach to measuring instability due to governance,
institutional and financial instability 2.8 1
23. Organizational performance assessment system (OPAS) 1.5 2
24. Agricultural science and technology indicators (ASTI) 1.6 2
25. Agriculture, health and environment 1.9 2
26. Institutional and national guidelines for bio-safety
and intellectual property rights (IPR) 1.3
25
ISNAR Research Report 25
The overall mean scores for the usefulness, quality, and contribution of the tools to institutional
performance and development goals are listed in Table 6. Not all respondents made comments
regarding specific tools, although some made more general comments (for example, that the tools
and procedures were useful and had improved efficiency). Other respondents commented specifi-
cally on their successful application of tools for priority setting (Tools 7, 8, and 9), guidelines for
universities (Tool 10), management information systems (Tool 11), and organizational performance
assessment (Tool 23). Other positive comments included statements that course materials were
relevant and useful (Tool 13), were good, with many case studies and illustrations (Tool 16), pro-
vided an opportunity for full NARS participation (Tool 26), had been used to develop training plans
(Tool 15), or to develop new projects (Tool 23). Negative comments or problems reported by re-
spondents included the fact that there had been no resources to implement the tool (No. 10), that
Tool 18 was difficult to monitor due to administrative bottlenecks, and that actual achievements
with respect to development goals were fewer than anticipated (Tool 20).
Table 6: Overall Assessment of Generic Tools and Procedures
Attribute assessed Average score Range No. of responses
scoring 3 or above
Familiarity 1.8 1-4 17 (20%)
Usefulness 1.6 1-5 10(12%)
Quality 1.7 1-4 11(13%)
Contribution to
institutional effectiveness 1.7 1-5 12(14%)
Actual/potential contribution to
development goals 1.9 1-4 15(21%)
Key: 1 = very high, 5 = very low
The aggregate analysis shown in Table 6 is based on mean scores for different attributes. Scores for
the usefulness and quality of the tools averaged 1.6 and 1.7, respectively, with only 12-13% of
respondents giving scores of three or above. They gave marginally poorer scores for contributions
to institutional effectiveness and development goals. These scores are similar to those given by
collaborators for similar attributes (Table 10).
3.2.2 ISNAR's global associates
In 1998, ISNAR created a new unit, ISNAR Global Associates (IGAs), to secure and expand their pool
of skilled experts in developing countries and to supplement ISNAR's own permanent staff. Today,
there are 17 IGAs, 10 males and 7 females. Of these, four are from academic institutions, two are
from organizations providing consultancy services, seven are from agronomy-based NARIs and four
are from other national organizations. In terms of geographical distribution, nine are from Africa
(including five from Kenya and one each from Burkina Faso, South Africa, Zambia and Zimbabwe),
three are from Asia (one each from India, Pakistan and the Philippines) and three from Latin America
(one from Chile, and two from Peru).
On average, ISNAR currently uses over 200 days of IGA services each year, mainly for training-
related activities. Since the creation of the IGA unit, its personnel have been used in the following
countries: Brazil, Burkina Faso, China, Costa Rica, Dominican Republic, Ghana, India, Kenya, Mauri-
tius, The Netherlands (at ISNAR headquarters), Nicaragua, Niger, South Africa, Swaziland, Tobago
and Trinidad, USA, and Zambia.
26
Chapter 3. ISNAR's Program Areas, Outputs and Assessments
The fact that most IGAs are former ISNAR trainees limits the input of new ideas and experiences
from other sources. Furthermore, the associates interact mainly with the training unit and have
little or no involvement with ISNAR's other technical programs. Nevertheless, the IGA initiative
provides a unique opportunity for expanding the scope of ISNAR's efforts and could result in signifi-
cant multiplier effects. In addition, it has the benefit of being relatively cost-effective and of allow-
ing ISNAR staff to focus on other emerging challenges related to their core tasks.
In the opinion of the assessment team, there are various ways in which ISNAR could make this
unit more effective, efficient, and credible, including:
• Aiming for broader participation in terms of both stakeholders and geographical coverage
• Making the selection process more competitive
• Selecting individuals with substantial agricultural R&D management experience, who are
both well respected and influential
• Introducing an explicit and systematic strategy for reviewing and evaluating performance
and for continuously upgrading the skills of associates
• Actively involving associates in action research and service-related activities other than
training.
3.3 Research results
Assessing the impact of ISNAR's research activities was challenging for several reasons, not the
least of which was identifying and defining such activities. The first step in measuring research
output was to identify published accounts of completed work. This was relatively straightforward
and the publications produced over the review period are listed in full in Annex 9 and summarized
in Table 7.
Table 7: ISNAR's Total Written Output (1997-2001)
Type Number of publications
1997 1998 1999 2000 2001 Total
Books 2 3 1 1 1 8
Briefing papers 7 6 3 2 2 20
Research reports 3 1 2 3 3 12
Research management guidelines 1 1 - - - 2
Reports of meetings
(including country reports) - 5 - - 9 14
Articles 26 29 30 21 2 108
Discussion papers 8 19 15 8 9 59
Miscellaneous 7 7 3 - - 17
Total 54 71 54 35 26 240
The next step was somewhat subjective and involved categorizing these publications into those
that were products of ISNAR's research efforts, and those that were not (Table 8).
Averaged over 5 years, this amounts to 26 research products per year. Since there are approximately
30 active ISNAR authors, this is equivalent to about one product per active author per year.
27
ISNAR Research Report 25
Table 8: 1SNAR Research Publications (1997-2001)
Type of publication Total number No. of research outputs
Research Reports 10 10
Research Management Guidelines 2 2
Discussion Papers 59 59
Books 8 6
Articles 108 53
Total 187 130
An additional aspect of ISNAR's publishing activities are those materials available from its website,
which has been online for about 5 years. Summary statistics were prepared on the publications
downloaded from the website from January 2001 to early April 2002. During this period, 4300
people visited the site, representing 107 countries and all regions. The number of visitors to the
website is steadily increasing, and currently stands at about 800 per month. Of the total visits
recorded during the survey period, 39% were from the USA, 8% from the Netherlands and 5% from
Germany. The World Bank is also a major user of ISNAR's website, accounting for approximately 5%
of all visits. During the study period, 7277 publications were downloaded, of which 3655 were
research reports, 1480 were briefing papers, 870 were country reports and 359 were corporate
publications. The five most frequently downloaded publications are listed in Annex 10 (Table A8). Of
these, three are research reports, one is a book, and one is a research management guide. Downloading
for educational purposes seemed to account for most visitor sessions from Europe and North America.
A planned analysis of citations of ISNAR publications was limited by technical difficulties and pro-
duced few fresh insights. An alternative way of attempting to assess the impact of research prod-
ucts was to solicit user opinions on the usefulness of the products. However, of the various surveys
undertaken, only that of NARI managers included detailed questions on publications, and of this
group, only 3 of the 23 respondents were direct recipients of any of the products. This particular
group rated the main research products (such as briefing papers and research reports) as being
slightly better than average in terms of accessibility, usefulness, and quality, while books and ASTI
products were viewed slightly less favorably.
A parallel survey conducted by ISNAR of 500 recipients of its publications produced 82 responses,
the results of which are summarized in Table 9 and given in more detail in Annex 10.
Table 9: Respondent Perceptions of the Value of ISNAR Publications
Increase in Adoption Application of No benefit
Region knowledge of tools recommendations claimed Other*
Latin America/Caribbean 100 30 40 0 0
Sub-Saharan Africa 94 41 35 0 18
Asia & Pacific 81 41 19 19 15
Organization for Economic
Cooperation & Development 74 13 22 26 4
West Asia & North Africa 100 100 50 0 0
Average % 85 33 27 14 10
* The 'other' category included use of publications for education, extension, or for sharing with colleagues.
Source: ISNAR publications unit survey (2002). Unpublished.
28
Chapter 3 ISNAR's Program Areas. Outputs and Assessments
According to this survey, ISNAR publications were of most value in increasing reader knowledge -
particularly in the case of respondents from developing countries, an impressive percentage of
whom reported benefiting in this way. There was also considerable agreement that ISNAR materials
were valuable as working tools and sources of advice on better ways of organizing research. On the
other hand, approximately one-quarter of European and North American respondents indicated
that the publications were of no direct assistance in terms of career building. However, the gener-
ally positive findings of this survey indicated that ISNAR publications reach a reasonably wide
audience, are generally regarded as useful, and are appreciated by readers. They thus appear to be
worthy products with high 'public good' characteristics.
Research outputs could be examined in various other ways, including, for example, analyzing the
proportion of discussion papers that evolve into refereed journal articles, and/or the number of
refereed articles produced each year by each staff member (and whether or not this is recorded and/
or rewarded in annual performance assessments). The issue of an institutional incentive structure
for encouraging journal publication or for publishing in high-quality journals is also important.
The assessment team suggests that the ISNAR management gives explicit consideration to these
matters, and that co-authorship with collaborators from the developing world be encouraged as
a form of capacity building.
The ultimate assessment of ISNAR's research work would involve examining its effect on agricultural
research management, conduct, policy, investment, evaluation, and effectiveness. Although such a
wide-ranging task was beyond the scope of this report, the assessment team supported the view
that many ISNAR research products have been influential in helping development agencies to assist
NARSs, either directly (via country-specific investment activities), or indirectly through CGIAR-linked
policies and decisions. The products of the ASTI project, for example, are unique sources of informa-
tion that have informed debate on national agricultural research investment (despite the fact that
some of the information was outdated by the time that it was published). The book Science Under
Scarcity has also probably helped many analysts around the world - both in developing and other
countries - to undertake more cogent analyses of research resource allocation. However, no data
could be obtained to support this view - not even the attempted citation analysis could assist the
team in quantifying this particular impact.
3.4 Assessment of direct collaborative activities
ISNAR's collaborative work with NARIs includes research, training, consultancy services, and
conducting workshops of various types. In order to assess the performance and impact of these
collaborative activities, questionnaires were sent to those individuals within both NARIs and the
donor community who have been directly involved in their implementation. The perceptions of
those responding are summarized in Table 10 and discussed briefly below.
3.4.1 Survey of collaborators
A total of 32 direct collaborators (26% of those sampled) returned the completed questionnaire:
26 males and 6 females. In terms of geographical distribution, 9 respondents were from Africa,
10 from Asia, 6 from Europe or the USA, and 7 from Latin America or the Caribbean. Seventeen
respondents were involved in biosafety/biotechnology-related projects and eight in projects related
to management of information systems; others were involved in projects related to priority setting
in agricultural research (5), the New Paradigm project (4), planning, monitoring, and evaluation (4),
integrating the demands of the environment, agroindustry, and competitiveness in agricultural
research (2), new technologies for agricultural research (2), AIDS, food security, and rural liveli-
hoods (2), and agricultural participatory research (2). It is worth noting that some of these projects
are heavily oriented towards training, with some action research also being involved.
29
ISNAR Research Report 25
Table 10: Assessment of ISNAR's Collaborative Work by Direct Collaborators and Donors
Direct collaborators (N=32) Donors (N=IO)
Attributes assessed Av. Score Av. Score
Effectiveness in achieving objectives 1.70 2.16
Relevance 1.30 2.22
Quality 1.70 1.80
Contribution to capacity building 1.70
Contribution to institutional performance 2.20 2.70
Actual/potential contribution to developmental goals 2.00
Contribution to:
• Economic growth - 3.25
• Poverty alleviation - 3.67
• Food security - 3.25
• Environmental protection - 3.50
• Nutrition and health - 4.00
• Gender equity - 4.00
Key: 1 = very high, 5 = very low (see Annex 11 for further details)
The survey asked participants to use a scale of 1 to 6 (1 = very high, 5 = very low and 6 = don't
know) to assess each collaborative activity for relevance, quality, contribution to capacity building,
contribution to institutional performance, actual and potential contributions to development goals,
and effectiveness in achieving the stated objectives. In total, 54 activities were assessed by ISNAR's
direct collaborators. The full list of projects and scores are presented in Annex 11, and a summary of
the results is given below. In many cases, the same project or activity was assessed by more than
one respondent, since many of ISNAR's activities involve multiple countries and/or several different
institutions. A number of collaborators associated with the New Paradigm project also shared their
perceptions with the team via written comments.
Effectiveness in achieving the stated objectives. The average score given by respondents was 1.7.
Of the total of 54 collaborative activities, only two were given a score of 4, and another two a score
of 3. Since all other collaborative activities (i.e., 93% of the total) were given a score of 1 or 2, it can
be concluded that these various activities were effective in achieving their stated objectives. Those
who awarded poor scores made various additional observations, including the comments that there
was insufficient support from the institutions involved, no follow-up after the exercise, or that the
objectives were only partially met because the workshop in question introduced relatively few new
ideas.
Relevance. The average rating for relevance was 1.3: ninety-seven percent of activities received a
score of 1 or 2, and only three activities were given a score of 3. All projects were considered rel-
evant - which is to be expected if activities are jointly planned and implemented with collaborat-
ing partners, focusing on agreed priorities.
Quality. With respect to quality, the average rating was 1.7, with 81% of joint activities being awarded
a score of 1 or 2. One activity was given a score of 4 by one respondent and five other activities were
given a score of 3. Considering the wide range of topics in which ISNAR is involved, the quality of
work in general is considered to be of a relatively high standard.
Contribution to capacity building. The average rating given for this attribute was 1.7, with 85% of
activities receiving a score of 1 or 2. Respondents claimed that collaborative activities enhanced
institutional capacity in a number of ways, including via improved research capacity, human capacity,
new methodologies and ideas, staff analytical capacity, managerial abilities, priority setting, and
30
Chapter 3. ISNAR's Program Areas. Outputs and Assessments
increased knowledge. One respondent gave a score of 5 and three others a score of 4, stating that
they needed more time to fully assess the contribution of the activity under consideration. They
also suggested that a close follow-up of activities would help to assess their value with respect to
capacity building.
Contribution to institutional performance. The average rating in this case was 2.2, with 30% of
ISNAR's activities being given a score of 3 or more; two gave a score of 5. Those who were positive
about this aspect of ISNAR's work offered a number of comments, including the observation that
their institution had greatly benefited from the activity, that the project had promoted effective use
of resources, or had been 'a tremendous positive influence on the institution', etc. On the other
hand, there were also comments such as 'it will [make a] great contribution if all recommendations
are implemented' and 'we gave more than we received'. Although 70% of respondents stated that
the activities had a moderate to significant effect on institutional performance, there is clearly a
need for more detailed follow-up to identify and document such changes.
Actual or potential contribution to development goals. Given the short duration of some of the
activities (which in some cases were still in progress at the time of the survey), respondents found
it difficult to evaluate this aspect. In many cases, therefore, individual assessments were based on
the perceived potential contribution of a particular activity. The average rating was 2.0, with 24%
of activities being given a score of 3 or more.
Some respondents indicated that the potential contribution of a particular activity would be high if
all recommendations were implemented, which to a large extent is dependent on the managers of
the various institutes and on adequate financial support. Nevertheless, the level of implementation
could be improved through better priority setting, more efficient use of resources, better use of
information technology (or biotechnology), and increased cooperation between and within institu-
tions. Again, however, there is a need to follow up completed projects in order to identify and
evaluate their contribution to developmental goals.
3.4.2 Donors' perceptions
Donors who were either directly involved in, or otherwise familiar with ISNAR's collaborative ac-
tivities, were asked to assess the performance and impacts of those activities with respect to their
effectiveness in achieving stated objectives, their relevancy and quality, and their contribution to
institutional performance, economic growth, poverty alleviation, food security, environmental pro-
tection, health and nutrition, and gender equity. Responses were received from seven donor repre-
sentatives (32% of the sample), who between them assessed ten activities. The average ratings
given to these activities are listed in Table 10.
Most donors were satisfied with the effectiveness of ISNAR's collaborative activities in achieving
their stated objectives (average rating = 2.16). However, two respondents felt that the overall
objectives of one particular project had not been met and those which had been achieved were
unlikely to be sustainable.
All donors felt that the activities and outputs were of high quality (average rating = 1.80), and that
they were relevant - perhaps not surprisingly, since most of them had been identified as priority
areas with the active involvement of potential beneficiaries. However, one respondent commented
that ISNAR needed to develop ways of strengthening ownership in research agenda-setting through
greater stakeholder participation.
With respect to institutional performance, in general the donors felt that participating institutions
had benefited from the collaborative activities and were in a position to progress further. However,
two respondents were more skeptical. One stated that 'the work did not help the organization with
31
Research Report 25
its efforts to develop strategy, progress or capacity', and that 'ISNAR staff seemed to be preoccupied
with many tasks, [limiting their] commitment to this project'. The other felt that ISNAR's team was
not fully engaged in the ongoing restructuring of agricultural research systems.
Few respondents offered specific comments regarding the effects of ISNAR's activities on broader
developmental goals. Again, they found it difficult to assess this aspect of ISNAR's activities, both
because of the timing of the assessment and the fact that the realization of developmental goals is
largely determined by external factors. The average ratings given for various aspects of develop-
ment ranged from 3.25 to 4.00 (Table 10). General comments included the fact that impacts on the
environment and on gender equity were indirect and delayed rather than direct and immediate.
One respondent expressed the opinion that ISNAR's activities would eventually assist in the
development of high quality, relevant research outputs, which in turn would contribute to eco-
nomic growth, food security, and poverty reduction. However, ISNAR's limited involvement in the
broader field of R&D was identified as a major concern with respect to achieving developmental
goals.
Overall, although some of the donors' comments did address some key issues, the team's efforts to
ascertain stakeholders' views of ISNAR in the wider context of development were in general not
very successful.
3.4.3 Perceptions of NARI managers
The views of NARI managers were also obtained via questionnaires. Of those who responded, ten
were involved in training projects with ISNAR, three in biotechnology-related collaborations and
two in NARI strengthening and consultancy.
When asked if the collaboration improved the personal performance of the respondents, 12 answered
in the affirmative, one in the negative and 10 provided no answer at all. Of those answering posi-
tively, two had been promoted, but two others commented that their tasks relating to the
collaboration had been interrupted in some way.
Regarding improvements in the performance of their staff as a result of the collaborative activities,
one negative and 11 positive answers were received. All of the latter claimed that their staff had
benefited directly or indirectly in some way - for example, by applying new methods, tools, con-
cepts, or ideas.
Fifty-two percent of respondents indicated that their institution as a whole had benefited from
collaborative activities with ISNAR, mainly by using the information, tools and/or skills acquired
through training activities. Some commented that as institutions, they were now more efficient
and made better use of their rather scarce resources.
Regarding ISNAR's perceived contribution to national development goals, six respondents felt that
the collaboration have improved national capacity, two stated that the collaboration had influ-
enced national policymakers, and two others felt that the rural population had benefited from the
collaboration.
3.4-4 Perceptions of regional research organizations
In the past, ISNAR has made considerable efforts to assist SROs, but of the eight surveyed, only
five returned their questionnaires (all from either Africa or the Latin American/Caribbean region).
Given the potential importance of SROs to NARIs and similar bodies in their respective regions
- and the extent of ISNAR's involvement with them - a higher response rate would have been
32
Chapter 3 ISNAR's Program Areas, Outputs and Assessments
invaluable. However, those that replied gave a reasonable score for the uniqueness of ISNAR's ef-
forts (average = 2.25) and some commented that ISNAR is particularly well placed to address the
concerns of NARSs and emerging SROs.
The contribution of ISNAR's work to improved staff performance within the various regional re-
search organizations was assessed at various levels:
Self. In this case, all responses were positive, although the reported self-improvement had been
achieved in different ways, for example by having contact with ISNAR staff or by taking up a fellow-
ship at ISNAR headquarters.
Colleagues. Three of the five responded quite positively to this question, mentioning, for example,
that ISNAR had provided post-doctoral fellowships and training for staff and had helped to improve
working methods in various ways.
Institutional. All answered this question in the affirmative, with one mentioning that ISNAR's work
made the wider institutional context easier to understand.
When asked for their opinion on the future role of ISNAR, one respondent felt that it should focus
on national technological innovation systems, rather than NARSs perse. Another argued that ISNAR
should provide theoretical and methodological guidance in institutional capacity building, and should
manage a database on worldwide agricultural research and innovation. Others suggested that ISNAR
should provide training to promote institutional capacity building, seek out key partners to spear-
head institutional innovation, and that it should help in priority setting and developing institu-
tional policies.
When asked to identify the key factors limiting ISNAR's impact, respondents mentioned the lack of
(1) a physical 'presence' in the regions, (2) a clear mandate, and (3) sufficient funds to collaborate
effectively with NARSs. There was also a perception that ISNAR staff lacked a clear understanding of
the politico-economic and socio-cultural realities of the institutions with which they work. Other
comments were more positive:
Attracting donor support. 'ISNAR helped PROCISUR2 to raise new funds'
Influencing NARS agendas. There is evidence of it...among NARSs ofASARECA3 member countries'.
•ISNAR helped to build a new strategy for PROCISUR, influencing NARIs' agendas'.
Linkages and networking. 'ISNAR brought new ideas on the subject to PROCISUR; it has [conducted]
many useful studies, but did not follow up to ensure institutionalization'.
ISNAR could increase the impact of its collaborative work by working more closely with national
and regional organizations and multilateral banks. More collaborative ventures should be under-
taken, rather than isolated advisory work. There is also a need to focus more on innovation systems
and networks rather then the NARSs alone and to be more proactive in identifying client's needs
and potential sources of funds.
3.5 Country case studies: Summary of key findings
Summaries of the 11 case studies are given in Annex 12. The studies illustrate a diverse set of
partnership arrangements, needs, and opportunities. For example, Argentina's NARS was one of
the first to be established in Latin America, and the Instituto Nacional de Tecnologia Agropecuaria
(INTA) has subsequently proved to be an outstanding performer in this region. However, recent
economic crises have had various adverse effects, and partnership with ISNAR has therefore be-
Programa Cooperativo para el Desarrollo Tecnologico Agroalimentario y Agroindustrial del Cono Sur (The Cooperative
Programme for Technological Development in Agrifood and Agroindustry in the Southern Cone)
Association for Strengthening Agricultural Research in Eastern and Central Africa
33
ISNAR Research Report 25
come a more important way of modernizing some parts of the NARS. This is particularly true in
relation to various biotechnology issues, which are especially important in Argentina because the
government is committed to improving agricultural productivity through the use of genetically
modified organisms (GMOs).
Uganda was an obvious choice for detailed study, since ISNAR has played a central role in improv-
ing its agricultural research capacity, particularly in the planning and development of the National
Agricultural Research Organization (NARO). Indeed, ISNAR's first collaborative activity in Uganda
was a 1988 study that helped underpin the establishment of the NARO. Since then, ISNAR has been
heavily involved in capacity building within various Ugandan institutions (albeit to a somewhat
lesser extent in recent years). ISNAR's work in Uganda must be acknowledged as one of its major
successes, although credit should be shared with a number of stakeholders - not least the Ugandan
Government for its enlightened commitment to re-establishing a national research capacity.
Zambia is another country with a long history of collaboration with ISNAR starting with a field
study of the managerial development needs of African research services conducted by ISNAR between
1982 and 1986. Collaborative work between ISNAR and the Zambian Soils and Crops Research Branch
(SCRB) started in 1984, with the participation of the latter in the ISNAR Survey of NARSs. More
recently (1998), ISNAR conducted two workshops on the implementation and institutionalization
of agricultural research management after difficulties were experienced with a newly installed
management information system (MIS). ISNAR has also trained at least 150 people in Zambia (in-
cluding staff from the SCRB, extension services, the private sector, and NGOs) and provided critical
inputs into the proposal to transform the SCRB into a Soils and Crops Research Institute. In addi-
tion, various exercises on priority setting were carried out in September 2000 and July 2001.
Most countries selected for in-depth study had a long history of interacting with ISNAR, so that the
review period constituted only a recent and relatively small part of the overall historical associa-
tion. In Indonesia, for example, ISNAR has a long history of involvement with its main partner, the
Agency for Agricultural Research and Development (AARD). More than a decade ago ISNAR helped
develop a MIS to assist AARD administrators in monitoring staff, budgets, and projects. This system
is still in use. AARD now has financial support from the World Bank and the Asian Development
Bank for the establishment of Assessment Institutes for Agricultural Technology (AIATs) in 26 prov-
inces, and it has recently created the Research Institute for Agricultural Biotechnology and Genetic
Resources (RIABGR). One challenge currently facing AARD is the decentralization policy of the Gov-
ernment of Indonesia, under which all functions of the government (except monetary policy, defense,
religion, foreign affairs, and justice) are to be decentralized. Nevertheless, the government has now
established regulations governing the licensing of GMOs and a Biosafety Regulation Group has
been formed from AARD staff, representatives of scientific societies, and various others. A scheme
for screening GMOs in glasshouses and multilocation field trials has been developed and Bt4 and
herbicide-resistant varieties of cotton, maize, and soybean have now been evaluated. Regulations
for the parallel evaluation of food safety issues have also been established and are delaying the
release of the maize and soybean varieties. However, Bt cotton has been released and seed made
available through the National Seed Board.
One country in which ISNAR's history of collaboration is relatively short is Cuba, where cooperative
work began only in the mid-1990s. Nevertheless, ISNAR has already been quite active: more than
160 workshops and meetings have been held and significant numbers of managers and researchers
have been trained. Furthermore, procedures, methods, and tools have been developed for planning,
priority setting, and performance assessment, and various networks and partnerships have been
established. All research work is managed on a project basis and yearly working plans - structured
by specific objectives - are also produced. Other significant changes that can be linked to ISNAR
Bacillus thuringiensis ('Bt') is a soil bacterium pathogenic to various insect species. Genes from different strains of the
bacterium have been incorporated into crop cultivars to confer resistance to specific insect pests.
34
Chapter 3 ISNAR's Program Areas, Outputs and Assessments
include an increased emphasis on self-evaluation and a more systematic approach to project
appraisal. Thus, despite continuing institutional rigidities in the Cuban NARS, ISNAR has managed
to make a useful and well-regarded impact.
Another relatively recent partner is South Africa, where ISNAR began to collaborate with the Agri-
cultural Research Council (ARC) in 1997 at a time of great institutional change. ISNAR assisted ARC
with various aspects of its transformation process, including the development of multidisciplinary
programs for the Council's institutes and the development of cross-program logical frameworks
('log frames'), which re-oriented research towards historically neglected resource-poor black agri-
culture. ARC staff indicated that the former contribution was the most successful, although consid-
erable impact was also achieved through various training activities, including training the staff in
the ARC'S Development Impact Assessment unit and Transformation Secretariat. Other assistance
consisted of advice, support, the provision of relevant ISNAR materials and publications, and help
in improving communications between ARC and the country's nine Provincial Departments of Ag-
riculture. For the ARC, one advantage of collaborating with ISNAR in the restructuring process
was that ISNAR could act as the principal agent of change, since there was some concern amongst
the institutes that the cross-program approach would result in some loss of autonomy.
Despite these positive features, however, ISNAR could have developed a more strategic approach to
the South African NARS: despite the fact that the ARC commands 66% of the national public
agricultural research budget, it may have been better for ISNAR to have targeted its collaborative
efforts at the National Department of Agriculture (NDA), which is possibly the only institution that
could really drive the agricultural research system. ISNAR's lack of focus on the NDA, coupled with
poor communication of the changes taking place in the ARC (with subsequent erosion of ARC fund-
ing and professional capacity) are compelling reasons for ISNAR to conduct its collaborative work
on a program basis (i.e., via strategic partnerships) rather than on a project basis. ISNAR needs to
create 'smart partnerships' with the key stakeholders in the South African NARS if it is to effectively
address the scope and complexity of problems presented by the South African situation.
The case study of Viet Nam provided evidence that ISNAR has been actively involved with
nontraditional partners, including many of the specialized agricultural research institutions that
have proliferated in Viet Nam. The analysts in this case concluded that the reports, publications,
and training that were the initial outputs of the collaboration were important both for improving
the knowledge, skill, and capacity of the staff of these institutions and for the country's agricultural
research and development in general. It is clear that in this case the needs are great and that
ISNAR's involvement should continue.
In general, ISNAR's impact on the case study countries has been in the form of better planning
systems, strengthening of planning capacity through the training of technical personnel, and the
provision of material to inform the policy debate on biotechnology and biosafety. In Sri Lanka, for
example, managers and scientists in the NARI are confident that they are better informed with
regard to biotechnology issues, and now have practical MISs. As yet, it is premature to judge the
usefulness or impact of the work aimed at decision-making in relation to research resource alloca-
tion. The Sri Lankan authorities are confident that the external review of the Council for Agricultural
Research Policy (CARP) will be strategically useful, although the extent of this usefulness will de-
pend on the future direction of government policy.
Although ISNAR originally focused on NARSs, it has subsequently distanced itself somewhat from
the concept on the grounds of having too limited a resource base for such a broad remit. However,
the question of the possible role of some of the potential partners in such systems (e.g. universities)
remains to be addressed. In general, incorporation of universities into national research systems
has been remarkably slow and often unsuccessful, despite ISNAR-supported efforts. In Uganda, a
new paradigm has been developed that is expected to lead to the establishment of a new NARS, in
which universities, the private sector, and community-based organizations will all be partners in
35
ISNAR Research Report 25
delivering demand-led research. Given the broad consultative approach being undertaken, ISNAR's
role in developing this new NARS is unclear. However, its past capacity-building efforts in the coun-
try will undoubtedly help to inform the national debate currently underway regarding the conduct
of agricultural research.
3.5.1 Constraints
Difficult decisions may be necessary in order to overcome some of the constraints that were identi-
fied in the case studies. One issue that recurred several times was that of the duration and continu-
ity of ISNAR's presence in a particular country. It was felt, for example, that ISNAR could have had a
much greater impact in the Dominican Republic if it's staff had been able to stay for longer during
the implementation phase of many of its recommendations. The observation that ISNAR's lack of
regional offices limits its outreach was expressed in many of the case studies; operating exclusively
from headquarters in the Netherlands incurs heavy overhead costs.
In the Dominican Republic, the policy environment for research has changed substantially for the
better, which, regrettably, has not generally been the case in the Latin American/Caribbean Region.
For example, in promoting institutional change in Mexico's Instituto Nacional de Investigaciones
Forestales y Agropecuarias (INIFAP), ISNAR must help overcome more than two decades of inad-
equate research funding, poor research infrastructure, and a remarkably low level of interaction
with the private sector. Most of the researchers in INIFAP are over 40 years of age and there has
been virtually no recruitment of new researchers in the past 20 years. Furthermore, most of the
research staff are suffering the effects of at least 15 years of low salaries and lack of opportunities.
On the positive side, the dialogue facilitated by ISNAR has succeeded in promoting the formation of
public-private partnerships, stronger engagement with agroindustries, and greater concern for
environmental issues in the public research agenda. The rapidly expanding role of the private sec-
tor in agricultural research will undoubtedly lead to a rapidly changing NARS that may have little
need for the types of assistance traditionally provided by ISNAR. Thus, although there is much yet
to be accomplished, what remains is primarily a responsibility for the state rather than a challenge
for ISNAR.
In some cases, ISNAR's achievements have been severely constrained by an extremely unfavorable
political environment for much of the review period. Such is the case in Pakistan, where ISNAR has
been working with the National Agricultural Research Center (NARC). In this case, although it is
rather too early to evaluate the impact of ISNAR's institution-building activities (e.g. work on
organizational performance assessments), there would appear to be few chances of attaining early
positive outcomes given the current turmoil in the country. However, the relatively modest recent
program can nevertheless be regarded as a useful step towards a longer term goal.
It is important that, prior to taking on new commitments, the best possible analysis should be
conducted of the current situation in a particular country in order to deploy scarce resources to best
effect. The South African study, for example, highlighted the need to identify the most appropriate
strategic partner if maximum impact is to be achieved, while in Sri Lanka, ISNAR should have
assessed the country's specific requirements prior to planning its training programs. A broad-based
training program invariably misses issues critical to particular countries, and in the Sri Lankan
case, the theoretical content of the training was excessive and too intense, while the practical
component was limited. One participant even claimed that no computers were provided for a train-
ing program entitled Internet and Intranet. Absence of a contact point for ISNAR in the country also
made it difficult to arrange any follow-up activities; eventually, the services of a local computer
programmer had to be used. The author of the Dominican Republic report also considered it highly
desirable for ISNAR to commit to pursuing the necessary follow-up activities in serving national
research systems. However, if such activities are to be of any value, then ISNAR must choose effec-
tive partners for the purpose.
36
Chapter 4. ISNAR's Niche and Priorities
4.1 Niche
An attempt was made to identify ISNAR's 'niche' by means of various survey questions concerning
ISNAR's services, alternative service providers, the advantages and disadvantages of collaborating
with ISNAR, and so on. The responses to these questions are discussed below.
4.1.1 Advantages
Of those who responded to the survey, 81% of collaborators, 90% of advanced research institutions,
61% of NARI managers and 86% of donors indicated that ISNAR offers unique advantages compared
to other institutions providing similar services.
Respondents from all four categories (Table 11) agreed that, in addition to its unique mandate,
ISNAR also offered experience with NARSs, high quality staff, global vision, and a network of col-
laborators. Other advantages (expressed by two or three categories of respondents), included ISNAR's
credibility, respectability and seriousness, and its good relationships and communications with
NARSs. These comments contrast somewhat with some of those recorded in Chapter 5 regarding
constraints on ISNAR's impact.
Table 11: Overview of Survey Responses Regarding the Unique Advantages of ISNAR as
a Collaborator
Unique advantages of ISNAR
as a collaborator Direct collaborators ARIs NARI managers Donors
Unique mandate and many years
of experience with NARSs, high
quality staff with knowledge
and experience, global vision,
and a network of collaborators
Credibility, respectability, and
seriousness in implementing
projects/activities, global network
Good relationship with its partners,
provides advice and guidance to
NARSs, multifocal cooperation,
good online communication
Note: + indicates agreement with the given statement by members of the respective group of respondents.
4.1.2 Disadvantages
Respondents' comments on the unique disadvantages of ISNAR are summarized in Table 12. Repre-
sentatives of all four categories commented that ISNAR lacked adequate size and representation in
different regions. Respondents in three of the four categories also made two closely related comments
37
ISNAR Research Report 25
regarding ISNAR's resource limitations and its European location. The high operational cost of ISNAR
is listed as a disadvantage by members of the ARI and donor categories, although in contrast, one
donor commented that the transaction costs of working with ISNAR were low (given as an advan-
tage but not listed in Table 11). Two comments which each arose from only one category of re-
spondent were that ISNAR is not well respected, and that other agencies are more visible and
proactive.
Table 12: Overview of Survey Responses Regarding Disadvantages of ISNAR as a Collaborator
Comments on disadvantages Direct collaborators ARIs NARI managers Donors
Locational disadvantage - based in
Europe but trying to service NARSs
elsewhere + + +
Lacking adequate size and presence
in different regions. Not very familiar
with needs or developments in
particular regions. Inadequate
regional framework for action + + + +
Not well respected +
Limited resources + + +
High operational cost, especially
overhead charges + +
Other agencies are more visible and proactive +
Note: + indicates agreement with the given statement by members of the respective group of respondents.
4.1.3 Alternative service providers
Fifty percent of those responding to the global attitude survey and 43% of responding NARI manag-
ers indicated that there are other institutions that provide the same services as ISNAR. The most
commonly mentioned services in this respect were information and publications (also provided by
ICRISAT, IFA, GTZ, CTA, FAO, Michigan State University, Rockefeller Foundation, IFPRI, CIAT, ILRI,
Virginia Polytechnic Institute and State University, ISAAA, IPGRI, INIBAP, etc.), training (also pro-
vided by GTZ, CIAT, ISAAA, CTA, IPGRI, and INIBAP), and policy analysis (also provided by IFPRI).1
Independent consultants were also mentioned as alternative service providers. One current
collaborator commented that 'many public and private institutions and NGOs can provide the same
services [at] a fraction of the cost with the same or better quality and credibility'; one donor stated
that there are many other institutions with the ability to work with institutional agenda.
4.1.4 Reasons for not collaborating with ISNAR
Those NARI managers and respondents in the global attitude survey who indicated that they were
not currently collaborating with ISNAR (57% and 31%, respectively) were asked if they would share
the underlying reason with the impact assessment team. In general, there was 'no particular reason',
with many eager to establish collaborative linkages. Other explanations included the perception
that ISNAR was 'more interested in NARIs', that the respondents lacked sufficient information
1 See list of acronyms.
38
Chapter 4- ISNAR's Niche and Priorities
regarding ISNAR's activities and areas of interest, administrative reasons (e.g. that ISNAR failed to
take a proactive role in capacity building), and the feeling that ISNAR was not involved in the kind
of activities in which collaboration was being sought. This latter problem may be addressed to
some extent by ISNAR's new focus on innovation systems.
Trainees' supervisors and NARI managers made additional suggestions regarding the types of as-
sistance that ISNAR could usefully provide to improve the performance of their respective institu-
tions (Table 13).
Table 13: Assistance That Could be Provided by ISNAR to Improve Institutional
Performance: Summary of Survey Responses
Type of assistance: Supervisors of trainees NARI managers
Assist in publishing results +
Develop research management skills including
project evaluation and impact assessment + +
Help strengthen collaboration or increase
presence in the region + +
Establish collaborative programs with
universities and other institutions/encourage
public-private partnerships + +
Assist in the commercialization of
research output +
Assist in evaluating NARIs +
Help identify research funding + +
Sponsor national and/or regional
training programs +
Note: + indicates agreement with the given statement by members of the respective group of respondents.
Overall, the various surveys suggested that effective and credible alternative providers are not
available for every service provided by ISNAR. ISNAR should therefore make every effort to main-
tain and expand its current unique niche within the R&D management arena, as well as increasing
awareness of its services amongst relevant stakeholders.
4.2 Priorities
Five categories of survey participants helped identify possible priorities for encouraging innova-
tion in agricultural research institutions (Table 14). Respondents in all categories indicated that
ISNAR should assign high priority to its service role in building the capacity of NARSs, particularly
in regard to managing changes in their operating environment. Respondents in all categories also
viewed forming networks and facilitating information exchange as a priority. The next most
frequently mentioned priorities were newer areas of science, such as biotechnology, the environ-
ment, and health (HIV/AIDS). Most other suggestions were raised by only two categories of
respondents and included such diverse topics as facilitating fundraising, broadening ISNAR's mandate
(to include areas other than agriculture), promoting self reliance rather than self sufficiency in food
production, providing training to offset staff shortages in managerial ranks, providing help with
information management, and increasing awareness of issues associated with the globalization
of trade.
39
ISNAR Research Report 25
The nature of the surveys did not allow firm conclusions to be drawn regarding ISNAR's future
priorities. Nevertheless, the widespread support for the first two suggestions given in Table 14
indicates that ISNAR's services and facilitating role are widely recognized as valuable and neces-
sary. The need for ISNAR to adapt its approach to these activities in response to regional groupings
and global changes is also clear.
Table 14: Summary of Survey Responses Regarding ISNAR's Priorities for Promoting
Innovation in Agricultural Research Institutions
Direct Global NARI
Priorities of ISNAR collaborators survey ARIs managers Donors
Continue to provide services to NARSs
and help them to build local capacity
in order to drive institutional innovation
and technological changes in response
to stakeholder needs and global changes
Facilitate partnerships and networking
between R&D institutions (civil, public,
and private); focus on exchange of
information and sharing of experiences
including technologies; focus on SROs
Focus on biotechnology/biosafety,
intellectual property rights and issues
such as the environment and HIV/AIDS
Assist with and facilitate the mobilization
of resources (i.e., help to develop new
and sustainable funding mechanisms)
Place more emphasis on participatory
research and technology transfer
Focus on institutional innovation. Facilitate
change in focus from supply-oriented to
demand-driven institutions
Broaden mandate to include rural research
and development (not just agricultural
research)
Increase research on agricultural
transformation, focusing on income
security (rather than just food security)
for rural populations
Continue to train researchers and research
managers to alleviate the effects of staff
movement and of HIV/AIDS
Continue to provide research management
tools, generate databases, and disseminate
information at regional and global levels
Increase the awareness of issues associated
with globalization amongst NARIs
Note: + indicates agreement with the given statement by members of the respective group of respondents.
40
Chapter 5. ISNAR's Impact: Constraints
and Enhancement
All surveys solicited opinions both on the factors constraining ISNAR's impact, and on ways of
enhancing it. Open questions addressing these issues were placed near the end of the question-
naires and generally elicited comments from respondents (Table 15). The financial constraints on
ISNAR's support of NARSs was the only observation made by all categories of respondents, includ-
ing some donors who presumably could contribute to the alleviation of the problem. Donors showed
less concern over the constraints resulting from ISNAR's location than did other groups of respond-
ents. Representatives of collaborators, ARIs and NARI managers all stated that since most of ISNAR's
staff were located at headquarters - far from developing countries - their knowledge of local con-
ditions (and hence the service itself) was somewhat limited.
Table 15: Summary of Survey Responses Regarding Constraints on ISNAR's Impact
Constraints on Direct Advanced NARI
ISNAR's impact collaborators research institutes managers Donors
Insufficient resources (financial and
human) to meet the needs of NARSs
and to implement collaborative projects + + + +
Most staff based at HQ, with inade-
quate presence in the various regions
and consequently poor knowledge of
local developments and limited scope
of activities + + +
Lack of (or poor) follow-up and
support of projects/activities. Poor
communication + +
Relatively high cost of services +
Lack of research capacity for collabo-
rating with research institutions in
the Northern hemisphere +
A recent shift away from service pro-
vision towards resource mobilization +
Frequent changes in NARI leadership +
Narrow and highly conservative views
of its leaders and a supply-oriented
development approach +
Restricted to work with NARIs -
a need to broaden scope for collabo-
ration beyond agriculture + +
Poor reputation for giving sound and
targeted advice to NARSs +
Note: + indicates agreement with the given statement by members of the respective group of respondents.
41
ISNAR Research Report 25
Other comments included the view that ISNAR lacks the research capacity necessary for collaborat-
ing with research institutions in the Northern hemisphere, that lately ISNAR's focus has shifted
away from a service orientation and more towards resource mobilization, and that the cost of
ISNAR's services is relatively high. As detailed in Table 15, other concerns include a 'supply-oriented
approach" to service delivery and the comment from one of the donors that ISNAR does not have a
strong reputation as a provider of sound and targeted advice to NARSs.
Many respondents provided constructive suggestions for ways in which ISNAR could enhance its
impact (Table 16). Eight out of the nine suggestions were made by respondents in more than one
group of stakeholders. However, only one was common to all groups of stakeholders, namely that
Table 16: Summary of Survey Responses Regarding Ways of Enhancing the Impact of ISNAR
Stakeholder group
Suggestions Direct collaborators ARIs NARI managers Donors
Increase resource availability so as to
improve coverage, be more effective in
collaborative work and provide more
services for NARSs
Alter ISNAR's mandate to encourage
greater innovation and broader collabora-
tion, and to allow a greater focus on the
needs of NARS
Involve NARSs in ISNAR's priority setting,
build a strong and effective communications
strategy that will enhance information ex-
change among institutions and increase
skills at all levels
Decentralize to increase ISNAR's presence
at the regional level, increase collaboration
with national and regional experts, plan and
implement programs in consultation with
NARSs
Develop a better strategy and mechanisms
for following up projects and activities,
integrate services with projects, and follow-
up the outcomes of interventions. Seek
funding for full program implementation
Concentrate work in fewer but better
focused projects and scale-up those activities
that have been most successful. Maintain
effective and regular contact with NARSs
Raise ISNAR's profile among political
leaders and donors
Move away from the original service mission
towards a greater emphasis on service-related
research
Increase collaborative work and partnerships
with other organizations (e.g. CTA, FAO, etc.)
Note: + indicates agreement with the given statement by members of the respective group of respondents.
42
Chapter 5 ISNAR's Impact Constraints and Enhancement
NARSs should be involved in ISNAR's priority setting and that communications in general should be
improved. Related suggestions included changing ISNAR's mandate to allow greater innovation
and broader collaboration in addressing the needs of NARSs, to follow projects from planning to full
program implementation, and to increase resource availability so as to accomplish more. Two groups
felt that ISNAR could enhance its impact by decentralizing and maintaining staff in the various
regions. Respondents from three groups recommended that ISNAR attempt to raise its profile with
political leaders and donors. Further suggestions regarding program strategy included adopting a
more focused approach, moving away from a service orientation towards a greater emphasis on
research and increased collaboration with organizations such as CTA and FAO.
In addition to those listed in Table 16, several other suggestions were made by donors, including
• Improve the implementation of ISNAR's work and the management of its projects
• Support regionalization (e.g. via a subregional action plan)
• Move to an IARC located in the regions
• Look for new ways of mobilizing resources
• Obtain (and learn from) feedback from those institutions adopting ISNAR's recommendations
Various comments (hopefully constructive) have been made regarding the apparent need for a more
strategic approach to ISNAR's work, including the choice of potential partners. Traditionally, ISNAR
has focused mainly on NARIs. However, new collaborators (such as private entities, NGOs, etc.) may
be needed, as well as links with representatives of ministries other than agriculture (e.g. planning,
finance and regional development). Interacting with a more diverse group of potential stakeholders
will require persuasive intellectual skills, new competencies, excellent preparatory work, and vari-
ous other transformations that will make new demands on staff and perhaps result in increased
stress levels. Such concerns are beyond the remit of the assessment team and are not pursued here.
However, they must be addressed by the ISNAR management if they are to successfully address the
wider issue of ISNAR's role and its ultimate impact.
43

Chapter 6. Institutionalizing Impact Assessment
within ISNAR
Measuring institutional impact involves identifying a broad spectrum of impacts, including training,
building institutional capacity for the production and delivery of research products and services,
increasing acceptance and utilization of these products by their intended beneficiaries, and any
other impacts on higher order institutional objectives. Leeuw (2000) argued that it is possible to
assess, both quantitatively and qualitatively, the outcomes of programs aimed at strengthening
institutions by systematically measuring improvements in their capacity to carry out their functions
and achieve desired results over time. Such an approach requires understanding and documenting
actual improvements in individual and/or organizational behavior as indicators of program
performance.
A previous study of ISNAR's impact (Horton and Mackay 1998) identified a number of constraints
which are summarized in Table 16.
Table 17: Constraints on ISNAR's Impact Identified by Horton and Mackay (1998)
1. Constraints within ISNAR's external environment:
• The loose configuration of most NARSs
• The inherent difficulty of working in the area of organizational capacity building
• The relatively low priority given to agricultural research in general and to strengthening the
organization of NARSs in particular.
2. Constraints within ISNAR (organizational motivation):
• Lack of clarity regarding the identity of ISNAR's primary clients
• Inadequate internal procedures for staff development and reward and for promoting
organizational learning
• Lack of operational procedures for determining the optimal balance between advisory services,
research, and learning.
3. Constraints within ISNAR (organizational capacity):
• A restricted core budget, which limits staffing and operating funds
• Lack of a clearly defined 'plan of action' to guide ISNAR's work
• Lack of clear priorities for responding to requests from NARSs
• Limited collaboration with other CGIAR centers and partners
• Inadequate internal procedures for monitoring and evaluation.
Source: Mackay et al. (1998)
At the time of the above study, ISNAR lacked a clear set of performance goals against which to
measure its achievements, and the 1997 EPMR recommended collecting baseline data for all new
projects in order to establish a better information base for assessing future impacts.
As ISNAR consolidates its activities under its new Road Map, an 'impact assessment culture' must
be adopted so that research and service delivery are routinely conducted in relation to specific
goals. Establishing the initial (baseline) condition in each project is a pre-requisite for establishing
45
ISNAR Research Report 25
credible evidence of impact. As ISNAR's own publications and training materials point out, monitor-
ing, evaluation and impact assessment should be an integral part of program or project planning
and implementation, and resources should be explicitly allocated for this purpose. Each research
program or project should include a logical framework that should specify data requirements and
indicators of success (including potential indicators of increased institutional capacity and improved
performance), as well as establishing responsibilities for data collection and reporting. This should
be linked to the overall institutional framework and ultimately to the CGIAR log-frame. Until such
measures are indeed an integral part of ISNAR program planning and implementation, impact
assessment will continue to be ad hoc, expensive, and of limited value.
Two complementary evaluation approaches could be adopted: project monitoring and assessment
by staff, and independent assessment by outside experts focusing on intermediate and long-term
impacts. Selected in-depth case studies of research and capacity building initiatives should be
conducted on a regular basis. Such case studies are essential for further refining methods and
approaches, for helping to define 'best practices', and for identifying ways of enhancing impact in
different settings. Conducting impact studies only in preparation for an EPMR prevents ISNAR from
deriving the maximum benefit from such activities. There are long delays between identifying (and
accepting) the need for institutional innovation, adoption of innovation, and production of measur-
able impacts. Impact studies conducted at too early a stage have little merit. Periodically, ISNAR
should routinely select a sample of completed projects and carefully study and document their
impacts. Examining 'failures' as well as 'successes' would be most instructive.
Such an approach would help establish plausible links between program or project activities and
development results, bridging the 'attribution gap' as far as is realistically possible for develop-
ment settings in which there are many service providers and contributors. Suitable databases and
indicators of outputs, outcomes, and impacts must be developed and maintained to ensure that
they become a persistent part of the 'corporate memory' and are not lost if individual staff mem-
bers leave.
The primary benefit of adopting better impact assessment processes and procedures will be in-
creased efficiency in the design and delivery of future projects, and the provision of credible evidence
of impact for various stakeholders. If ISNAR succeeds in adopting and internalizing its own tools
and procedures, then it would have the added benefit of being able to give a practical demonstra-
tion of the value of its ideas to its various partners, including both NARSs and other CGIAR centers.
46
Chapter 7. The Impact of ISNAR
The principal results of the assessment team's review of ISNAR's activities, outputs, and immediate
outcomes are reported in Chapter 3. Sufficient evidence was obtained to conclude that, over the
review period, ISNAR made a significant contribution to building capacity within various NARSs,
particularly in regard to improving leadership, organizational structure, linkages and networks,
resource management, planning, priority setting, monitoring, evaluation, and resource mobilization.
In this chapter, an assessment is made of the 'downstream' impacts of these various contributions
- i.e., of the capacity of R&D systems to identify and meet the needs of poor and often hungry
people.
The ultimate aim of ISNAR's work is to help improve agricultural R&D capacity at national and
regional levels in order to address and respond to the most pressing needs of resource-poor small-
holder farmers in less developed countries. ISNAR's goals are thus consistent with those of the
CGIAR system as a whole, in that the ultimate target is to reduce the negative impacts of poverty by
increasing incomes, enhancing food security, and using natural resources in a more sustainable
manner. Evidence that ISNAR's various activities have led - and continue to lead - to a substantive
increase in capacity within national agricultural research systems therefore addresses the core of
ISNAR's impact. With this in mind, the assessment team attempted to determine ISNAR's impact in
the following four areas:
• Raising awareness and contributing to knowledge
• Influencing policy
• Institutional development
• Development in general
One of the difficulties encountered in trying to assess ISNAR's impact was a general lack of baseline
data. During the review period, the operating environment of many NARSs deteriorated. Several
experienced a decline in the resources available for research, which had concomitant implications
for ISNAR, since one of the latter's principal roles is the provision of training and tools for the
improved planning and management of research systems. As described in Chapter 3, ISNAR's work
has suffered to some extent from the fact that no clear connection has been established between
improved management practices on the one hand, and the generation of improved technologies for
farmers on the other. However, by examining each of the above categories in turn, it is possible to
demonstrate some of the impacts that have been achieved.
7.1 Raising awareness and contributing to knowledge
Investment in agricultural research has been central to improving human nutrition in much of the
developing world, and any knowledge or information that supports and maintains this effort ought
to produce long-term positive impacts. ISNAR's most significant and most clearly demonstrable
impacts in this area were achieved through the IBS project and the ASTI work conducted in collabo-
ration with IFPRI. These two projects stand out among the diverse impacts achieved by ISNAR
through research and knowledge sharing (although there appears to be a divergence of opinion
within ISNAR as to whether or not the ASTI effort is properly regarded as a 'research' activity). The
latter has nevertheless been particularly useful in informing the development of current and future
policy on agricultural research at national and international levels and such information will con-
tinue to be of value.
47
ISNAR Research Report 25
The emergence of biotechnology industries in both rich and poor countries has also generated a
need for revised research policies, and ISNAR's work on this issue has been valuable in various parts
of the world. A number of workshops have also been effective in informing interested parties, since
they have usually attracted high-level political and stakeholder participation.
Some of the knowledge generated by ISNAR is now finding a wider application through its incorpo-
ration into university curricula. The University of Zambia and Ruhuna University in Sri Lanka, for
example, have incorporated materials on research management information systems and research
policy analysis into their respective degree programs. ISNAR should increase the use of its knowl-
edge-based products in this way by collaborating with a wide range of tertiary-level institutions
and academic programs. The New Paradigm project is already making valuable progress in this
respect within participating universities in Latin America. Overall, the production and application
of knowledge-based products probably represents ISNAR's best prospect for providing international
public goods.
According to the team's surveys, the area in which ISNAR has made its greatest impact is in the
dissemination of knowledge via publications. Various publications (e.g. those on information sys-
tems, planning and financing research) provide evidence that ISNAR has successfully facilitated the
sharing of experiences and best practice in many of the countries in which it works. Some of the
country reports, particularly those on biosafety and strategic research planning, have also been
widely distributed. However, research publications are not necessarily the best way of reaching
different audiences: currently, there appears to be no clear strategy for the 'packaging' of material
in the most appropriate way for ISNAR's various partners. There may be some merit in giving a
higher priority to 'translating' good research publications into formats better suited to different
audiences.
7.2 Influencing policy
7.2.1 Government policies and strategies
ISNAR's greatest contribution to policy development has been in the area of biotechnology and
biosafety. ISNAR has been a leader in this field, providing much useful information at a time when
the attempts of most developing countries to advance in this area have been limited by inadequate
access to relevant knowledge and technology. The countries that have benefited most from ISNAR's
support in developing policy in this area include Argentina, Cameroon, Chile, China, Egypt, Indonesia,
Kenya, Malaysia, Mexico, Namibia, Nigeria, the Philippines, Turkey, Uganda, and Zimbabwe. ISNAR's
work with the Uganda National Council for Science and Technology (UNCST) provides a good exam-
ple of a multiplier effect, since the latter has now been selected to coordinate a regional program
(BIO-EARN) within the East African Regional Program and Research Network for Biotechnology,
Biosafety and Biotechnology Policy Development.
ISNAR has probably had a lesser impact on the development of domestic policies for agricultural
research in the developing world. The ASTI project helped to document current trends in invest-
ment in agricultural research, but public sector funding for this field has continued to decline and
the much-needed political support for agricultural research is still not forthcoming. Good research
is not a sufficient condition for policy change, and ISNAR's work on policy reform must be informed
by a greater degree of understanding of the policy environments of developing countries.
Nevertheless, ISNAR has made significant contributions to issues such as raising awareness of HIV/
AIDS as an area requiring policy development in relation to agriculture in general and agricultural
research in particular.
48
Chapter 7. The Impact of ISNAR
7.2.2 Institutional policy and discourse
At the institutional level, ISNAR's work has promoted policy discussions on research Financing and
private sector participation in R&D. With regard to the latter, ISNAR's work in Mexico is perhaps the
best demonstration of how its efforts have helped increase private participation in agricultural
research.
Broader stakeholder participation in research policy development has been promoted through ISNAR's
involvement with GFAR and various SROs. ISNAR's support of the latter has been particularly effec-
tive in promoting local stakeholder participation in research activities and in setting research agendas.
The ASTI work has provided useful information that has informed debate at national and subregional
levels, particularly on the issues of declining support for agricultural research, the setting of re-
search priorities, and collaborative relations between NARSs and IARCs.
7.3 Institutional development
ISNAR's principal impacts in this area have been achieved at national and subregional levels. Its
core activities remain focused on increasing the capacity of NARSs, mainly though training activi-
ties and advisory services, although a number of its planning, management, and diagnostic tools
are also slowly being adopted by various national research systems. ISNAR is also evolving better
methods of collaborating with partners so as to increase its effectiveness in developing institu-
tional capacity.
However, in the view of the assessment team, ISNAR does not appear to have a clear and compre-
hensive conceptual framework to support its capacity-building initiatives. The approach appears
fragmented, with various component activities (e.g. training, management tools, and advisory serv-
ices) to a large extent being conducted independently. A clear capacity-building framework would
help to identify ways of collectively strengthening an institution's capacity so that results are achieved
more quickly and/or with fewer resources and less effort. It is worth pointing out that credibility in
this area is a function of the extent to which in-house practice reflects that being advocated.
On a separate but related point, ISNAR does not appear to have developed any formal networks for
advancing its agenda with its partners. This is a significant omission, particularly in relation to
policy development.
There are, however, some examples in which various conferences, training workshops, seminars,
and planning workshops have made a significant contribution to transforming an existing R&D
system into one more appropriate to a competitive agricultural sector. ISNAR's experience in Mexico,
for example, is now being used to guide similar efforts in other Latin American/Caribbean countries
(e.g. the Dominican Republic). ISNAR's initial work in Pakistan has also led to a more comprehensive
collaborative institution-building program that should prove useful when the country's situation
improves.
7.3.1 Training and capacity building
There are a number of noteworthy points in relation to ISNAR's training activities:
• ISNAR's training has improved skills in various areas, including those of leadership, team
building, research program formulation, budgeting, monitoring and evaluation, gender analy-
sis, communication, presentation and report writing, problem solving, priority setting, and
proposal writing.
• The PAPA has proved effective in improving both knowledge and skills - as evidenced by
reports of the increased confidence of trainees in areas such as leadership, management, and
49
ISNAR Research Report 25
teamwork. The experiential method and feedback approach appears to result in greater im-
pact than do other methods of training.
ISNAR's training in the field of biotechnology/biosafety has been particularly well received by
its beneficiaries.
1 The training on effective proposal writing is also highly regarded and there is growing evi-
dence that it has resulted both in more proposals being prepared and a higher success rate in
terms of the proportion funded.
1 In the area of management, the three training modules most valued by users were Strategic
planning, Priority setting and Program management and evaluation.
• Almost 50% percent of ISNAR's training has benefited individuals from Africa; this is consist-
ent with broader CGIAR priorities.
1 The multiplier effects of ISNAR's training are considerable, with clients reporting an average
of about 20 additional staff subsequently trained by each individual trained by ISNAR.
In general, there has not been adequate long-term tracking of the impact of ISNAR's training.
7.3.2 Tools
ISNAR has introduced a wide range of planning, management, and diagnostic tools for use by NARSs.
At present, none of these tools has been widely adopted, although the use of some of them is
increasing. The new INFORM-R application, for example, is regarded as a much-improved version of
the original and is now expected to be more widely adopted. Demand is also growing for the OPAS
tool, although the assessment team was unable to obtain convincing evidence of positive outcomes
from its use. ISNAR has also developed other tools that should improve the impact of its work,
including those concerned with poverty mapping and regional priority setting.
7.3.3 Collaboration and advisory services
ISNAR has continually improved its approach to collaborating with clients and target institutions
and steady improvement in this area has to some extent ameliorated the negative effect of ISNAR's
lack of physical presence among its target population. Evidence suggests that in areas where ISNAR
has been involved in a successful collaborative activity, there are significant positive effects on
capacity building (e.g., in terms of research capacity, human resources, and skills development, as
well as general management). Impacts of this type have been achieved in various biotechnology/
biosafety projects and in collaborative projects involving INFORM. The New Paradigm project, al-
though only initiated recently, promises to strengthen ISNAR's collaborative efforts in various Latin
American/Caribbean countries, including Bolivia, Brazil, Colombia, Costa Rica, Cuba, Ecuador, Mexico,
Nicaragua, Panama, and Venezuela. The Global Associates project also provides an additional op-
portunity for ISNAR to achieve greater impact through its collaborative efforts.
As reported earlier, ISNAR's work with SROs has had a significant impact on various stakeholder
groups and processes. However, many commentators, particularly donors, have suggested that
this impact would be greater if ISNAR developed a specific strategy for each subregion and made a
greater attempt to decentralize its efforts. ISNAR's work on policy support and institutional innova-
tion may be better served by establishing a critical mass of experienced staff living and working in
particular countries for extended periods of time, helping to build mutual understanding, trust,
and confidence. Such an approach would allow ISNAR to develop an action-research agenda and to
identify windows of opportunity as they arise. Moreover, it would facilitate the development of
holistic and/or integrated approaches to building capacity at institutional, national, and regional
levels, partly because of the greater scope for forming partnerships and collaborations. On the
other hand, establishing and maintaining a regional presence would incur additional costs and
difficulties. A set of guiding principles and procedures is needed that will allow ISNAR to decentral-
50
Chapter 7 The Impact of ISNAR
ize its operations in the most effective and efficient manner. Such guidelines would have to take
into account factors such as the greater need for communication across long distances, as well as
political issues such as institutional jealousy and so on.
In terms of advisory services (some of which have been provided on a cost-recovery basis), the
greatest impact has been achieved in the areas of strategy development and planning, diagnostic
reviews, and facilitating institutional development. During the review period, various advisory
services were supplied to Benin, Croatia, Cyprus, Eritrea, India, Indonesia, Iran, Mozambique, Pales-
tine, Pakistan, South Africa, Sri Lanka, Uganda, Viet Nam, and Zambia.
7.3.4 Behavioral change
It was difficult to assess whether or not ISNAR has been effective in catalyzing behavioral change
either within research and/or related organizations, or at the individual level. At the organizational
level, ISNAR's collaborative work with the United Nations Development Programme (UNDP), World
Bank, and CGIAR on gender relations in agricultural research is expected to have a number of impacts
on the activities of these influential organizations. However, at the individual level, there is more
evidence of behavioral change within NARSs, since supervisors of trainees reported the following
improvements in their staff after training:
• Increased confidence
• Improved knowledge and ability to train others
• Better writing skills
• Greater involvement in planning, proposal writing, monitoring and evaluation
• Better workshop facilitation.
7.4 Developmental impact
7.4.1 Global public goods
In the future, ISNAR's products will hopefully develop into international public goods that may fall
within any of the following categories:
• Institutional innovations widely used in different regions and cultures
• Systemic policy reforms
• Noticeable positive change in attitudes to, and perceptions of, agricultural R&D
• Local products and/or activities with growing global and/or regional impact.
It is therefore important for ISNAR to develop a review framework that allows all projects to reflect
higher order impact objectives, both in terms of CGIAR goals and in terms of promoting the wider
adoption of ISNAR's tools and ideas.
7.4.2 Stakeholders' perceptions
Stakeholder surveys and the country case studies provided evidence of some local impacts of ISNAR's
work. However, there was no evidence that the local or country level work resulted in wider impact
or other significant spillover effects. Few of the NARS stakeholders could provide examples of broader
developmental impacts of ISNAR's work, while various donors expressed concerns in this regard,
most notably over the perception that clients tend to have a less than desirable degree of owner-
ship of ISNAR-funded projects. They also recognized other limitations in ISNAR's work, for example,
due to fragmentation and the lack of an holistic, strategic approach to both R&D and institutional
issues. As noted earlier, donors also viewed ISNAR's lack of effective presence in developing coun-
tries as a possible barrier to achieving wider impact.
51
ISNRR Research Report 25
7.4-3 Impact of ISNAR's work on alleviating poverty
There is a marked lack of data concerning the field-level impact of ISNAR's work on poverty alleviation.
However, in the absence of such information, anecdotal evidence and examples may help to dem-
onstrate some of the potential impacts of ISNAR's work. For example, ISNAR's contributions to
biotechnology/biosafety policy development could lead to the accelerated release of new crop varie-
ties and consequent positive impacts on the poor. In addition, ISNAR's attempts to increase collabora-
tion between different NARSs and also between NARSs and the CGIAR centers may well accelerate the
pace of technology development and uptake, resulting in further tangible benefits for the poor.
7.5 Conclusions
Prior to 1997, ISNAR invested heavily in the development of products and tools. The review period
covered by this report should therefore provide some evidence of returns on these investments.
However, the team found it difficult to find evidence that allowed some assessment of cost effec-
tiveness, either globally or regionally.
In relation to the magnitude of the challenges facing ISNAR, its impact to date has been modest.
There are several explanations for this, of which the first and most obvious is lack of resources,
since the investment in ISNAR by CGIAR donors has been relatively modest. Second, there is a
general deficiency in the approaches used both in ISNAR's policy development work and in its
institutional development and transformation activities. Although ISNAR is producing a consider-
able amount of information for guiding policy development, in most of ISNAR's partner countries,
policy reform is a complex business. ISNAR therefore needs to invest more resources in understand-
ing the workings of policy-related processes. This also requires the development and application of
new techniques and approaches appropriate to influencing policy. ISNAR also needs a comprehen-
sive framework to guide its various activities related to institutional development, including research,
training, advisory services, and the development of institutional reforms and innovations.
In the opinion of the assessment team, one of ISNAR's core tasks should be to foster the develop-
ment of intellectual capital in the field of institutional theory and policy development for R&D
systems. Such challenging work is risky. However, it may be beneficial for ISNAR to move away
from its current 'comfort zone' of supply-driven tools and advisory services to address such chal-
lenges, perhaps by taking more calculated risks by adopting an innovation systems perspective,
generating ground-breaking action research, and adopting pragmatic policy development agendas.
It is worth noting that during this review period ISNAR has been operating in a very difficult envi-
ronment. Most of its major collaborating institutions have been faced with severe resource con-
straints and have lost staff as a result. ISNAR itself was faced with a sudden, significant reduction in
the funding provided by one of its major donors and has undergone two major strategic planning
processes. In short, the working environment of both ISNAR and its primary collaborators has
become more complex than ever before.
Against this background, ISNAR's attempt to maintain the spirit of its global mandate by covering
all three major developing regions has had the effect of spreading its limited resources very thinly
and hence to only a fragmented and modest level of impact. Resolving this dilemma will involve
making tough choices and setting clear priorities. After all, while resources are likely to continue to
be a constraint into the foreseeable future, the need to develop NARSs remains urgent - a fact that
cannot be ignored either by ISNAR or the CGIAR.
52
References
Anandajayasekeram, P. and D. Martella. 1999. Evaluation of agricultural research in Eastern and
Southern Africa. Knowledge, Technology and Policy 11(4): 13-41.
Anandajayasekeram, P., M. Rukuni and M. Mekuria. 2002. Why has impact assessment research not
made more of a difference? Lessons from Eastern and Southern Africa. A paper presented at the
International Conference on Impacts of Agricultural Research and Development: Why has Impact
Assessment Research not made more of a Difference? San Jose, Costa Rica. February, 2002.
Anderson, G., C. Lusthaus, and P. Bracegirdle. 1995. Reflections on evaluating the performance of
development organizations: in search of a framework. Paper presented at Evaluation for a New
Century: A Global Perspective, an international conference co-sponsored by the Canadian Evalu-
ation Society & the American Evaluation Association. November 1995. Vancouver, Canada.
Avila, A. F. D., J. E. Borges-Andrade, L. J. M. Irias, and T. R. Quirino. 1983. Formacao do capital humano
e retorno dos investimentos em treinamento na EMBRAPA. Brasilia: EMBRAPA-DDM/DRH.
Babu, S.G. and G. Mthindi. 1995. Costs and benefits of informed food policy decisions: A case study
of food security and nutrition monitoring in Malawi. Quarterly Journal of International Agricul-
ture 34(3): 292-308.
Bickman, L. 1987. Using program theory in evaluation. In New directions for program evaluation,
edited by L. Bickman. No. 33. San Francisco, USA: Jossey-Bass.
Biggs, S.D. 1990. A multiple source of innovation model of agricultural research and technology
promotion. World Development 18(11): 1481-1499.
Byerlee, D. 1998. The search for a new paradigm for the development of national agricultural research
systems. World Development 26(6): 1049-1055.
Cascio, W. F. 1989. Using utility analysis to assess training outcomes. In Training and development in
organizations, edited by I. L. Goldstein, pp. 63-88. San Francisco: Jossey Bass.
Cracknell, B. 1996. Evaluating development aid: Strengths and weaknesses. Evaluation 2(1): 23-34.
DANIDA. 1994. Evaluation report: Agricultural sector evaluation. Impact studies, Volume 2: Methods and
findings. Copenhagen, Denmark: Danish International Development Agency.
Dollar, D. and L. Pritchett. 1998. Assessing aid: What works, what doesn't and why. Washington D.C.:
Oxford University Press for the World Bank.
Garcia, L.O. 2002. An assessment of ISNAR training activities: Trainees reactions to workshop. A
meta evaluation report prepared for the ISNAR Training Unit. The Hague, The Netherlands: ISNAR.
Gerrard, CD. 2000. Ten institutionalist perspectives on agriculture and rural development: a con-
ceptual framework for policymakers, managers and analysts. Paper presented at the IAAE
Conference, Berlin, Germany. 2000.
Horton, D. 1990. Assessing the impact of international research: concept and challenges. In Methods
for diagnosing research system constraints and assessing the impacts of research. Vol. 11: Assessing
the impacts of research, edited by R.G. Echeverria. The Hague, The Netherlands: ISNAR.
Horton, D. and R. Mackay. 1998. Assessing the organizational impact of development co-operation:
a case from agricultural R&D. Canadian Journal of Program Evaluation 13(2): 1-28.
ISNAR. 2001. Road Map 2002-2006. Institutions matter: Let knowledge make the difference. The Hague,
The Netherlands: ISNAR.
Janssen, W. 2002. ISNAR's approach to institutional development: Evolution over the last five years.
Presentation made to EPMR and Board Meeting, February 2002. The Hague, The Netherlands:
ISNAR.
Kolb, D.A. and R. Fry. 1975. Towards an applied theory of experiential learning. In Theories of concept
processes, edited by C. Cooper. Cirencester, UK: Wiley.
Kuby, T. 2000. Turning attention towards results. How GTZ is building up its impact evaluation capacity.
GTZ internal document. Eschborn, Germany: GTZ.
53
ISNAR Research Report 25
Kumar, K. 1995. Measuring performance of agricultural and rural development programs. In Evalu-
ating country development policies and programs: New approaches for a new agenda, edited by
R. Picciotto and R.C. Rist. San Francisco, USA: Jossey-Bass.
Leeuw, F.L. 2000. Program evaluation and social and institutional impact assessment. Paper pre-
sented to the workshop organized by the Standing Panel on Impact Assessment (SPIA) of the
Technical Advisory Committee (TAC) of the CGIAR on the future of Impact Assessment in the
CGIAR: Needs, Constraints and Options. May 3-5, 2000. Rome, Italy: FAO.
Mackay, R. and D. Horton. 2002. Expanding the use of impact assessment and evaluation in agricul-
tural research and development. Paper presented at the International Conference on Impacts of
Agricultural Research and Development. February, 2001. San Jose, Costa Rica.
Mackay, R., S. Debela, T. Smutylo, J. Borges-Andrade, and C. Lusthaus. 1998. ISNAR's achievements,
impacts and constraints: an assessment of organizational performance and institutional im-
pact. The Hague, The Netherlands: ISNAR.
Maredia, M., D. Byerlee, and J.R. Anderson. 2000. Ex-post evaluation of economic impacts of agricul-
tural research programs: a tour of good practice. Paper presented to the workshop on The
future of impact assessment in CGIAR: needs, constraints, and options. Standing Panel on Impact
Assessment (SPIA) of the Technical Advisory Committee of the CGIAR. Rome, Italy.
McCaffery, J.A. 1986. Independent effectiveness: A reconsideration of cross-cultural orientation and
training. International Journal ofIntercultural Relations 10:159-178.
Norton, G.W. and S. Davis. 1981. Evaluating returns and agricultural research: a review. American
Journal of Agricultural Economics 63(4): 685-699.
Odame, H. H. 2001. From workshop to workplace: An evaluation of agricultural research manage-
ment training using the participant action plan approach. The Hague, The Netherlands: ISNAR.
Odame, H. H. and M. Nielson. 2002. Internet access to ISNAR training modules. The Hague, The
Netherlands: ISNAR.
Pardey, P.G. and N. Beintema. 2001. Slow magic: Agricultural R&D a century after Mendel. Agricultural
Science and Technology Indicators Initiative. Washington D.C.: IFPRI.
Pingali, P.L. 2001. Milestones in impact assessment research in the CGIAR, 1970-1999. With an annotated
bibliography of impact assessment studies conducted in the CGIAR, 1970-1999, prepared by
M.P. Feldmann. Standing panel on impact assessment. Technical Advisory Committee of the
Consultative Group on International Agricultural Research.
Rossi, P.H. and H.E. Freeman. 1993. Evaluation: A systematic approach. 5th edition. Beverley Hills, USA:
Sage Publications.
Ryan, J. 2002. Evaluating the impact of economic policy research: concepts, practices and lessons.
A paper presented at the International Conference on Impacts of Agricultural Research and
Development: Why has Impact Assessment Research not made more of a Difference? San Jose,
Costa Rica. February, 2002.
Salomon, M.L. and P.G.H. Engel. 1997. The social organization of innovation: a focus on stakeholder
interaction. Amsterdam, The Netherlands: Koninklijk Instituut voor de Tropen (KIT).
TAC/CGIAR. 1996. The future role of the CGIAR in the development of national agricultural research
systems: a strategy study of institutional strengthening research and services. Rome, Italy: TAC
Secretariat/CGIAR
TAC. 2000. A food secure world for all: toward a new vision and strategy for the CGIAR. Rome, Italy:
TAC Secretariat, FAO.
54
Text edit, layout and cover photo: Green Ink Ltd, UK (www.greenink.co.uk)
Printing: Colour Works, UK
In 2002 ISNAR commissioned a report on its achievements, impacts, and constraints
since its previous external review in 1996-97. The team assembled for this purpose
used several methods, including surveys of ISNAR's institutional and individual
partners and case studies of countries in which the Service had been active over
the review period. The team paid particular attention to ISNAR's efforts to influence
policy, develop institutions, achieve a developmental impact, raise awareness and
contribute to knowledge. Several of ISNAR's focal project activities were examined,
as too were its broad range of core activities.
The team found that ISNAR has been quite effective' in sponsoring and supporting
institutional innovations that are used widely across regions and cultures, and that
systemic policy reforms have been usefully triggered by ISNAR work. There have been
noticeable positive changes in attitudes to, and perceptions of, agricultural R&D that
many associate with ISNAR. Training and capacity building has been a major thrust
over the period examined, and many useful materials have been prepared and tools
provided. Even though many of ISNAR's products and/or activities are local and
intermediate in character, they have resulted in growing global and regional impacts.
However, it was difficult to quantify the cost-effectiveness of these efforts, especially
for intermediate products that will take long periods to come to fruition. The types of
public goods that have been pursued are intrinsically valuable and will continue to be
in demand from many parts of the developing world. However, attempts to cover all
the world's major developing regions have had the effect of spreading ISNAR's limited
resources rather thinly. ISNAR's endeavor to maintain the spirit of its global mandate
has led to a fragmented and modest impact. This poses challenges for the Service
and for its parent, the CGIAR, which will need to re-align ISNAR's strategy, focus,
resources and operations to address the future need for institutional development
in national agricultural research and innovation systems.
ISNAR Program
P.O. Box 5689
Addis Ababa, Ethiopia
Tel: +251-1-463215
Fax: +251-1-461252/464645
CGIAR E-mail: isnar@cgiar.org
Website: www.isnar.cgiar.org ISSN 1021-4429