I M P A C T A S S E S S M E N T B R I E F N U M B E R 8 Supported by the CGIAR The Vavilov-Frankel Fellowship Programme awards two fellowships annually to young scientists from developing countries, to en- able them to carry out innovative research on the conservation and use of plant genetic re- sources (PGR). Since 1994, 39 scientists from 24 countries have taken part in the programme. An evaluation of the impact of the Vavilov- Frankel Fellowships (VFF) reveals that overall the VFF Programme has met its stated goals of contributing to the scientific capacity of the Fellows and their home institutes, and of fos- tering the conservation and use of PGR. The evaluation also indicated ways in which the Fellowship Programme could be improved, and these are now being implemented. Background The VFF Programme commemorates the huge contributions made to the field of PGR con- servation and use by Academician Nikolai Ivanovich Vavilov of Russia and Sir Otto Frankel of Australia. Vavilov launched the idea of storing global collections of crop di- versity in genebanks, primarily for breed- ers to use in developing improved varieties. Frankel was one of the first to loudly sound the alarm about genetic erosion and the loss of agricultural biodiversity, and to establish the International Board on Plant Genetic Re- Evaluating the impact of the Vavilov-Frankel Fellowship Programme sources, with its emphasis on collections and ex-situ conservation, which eventually grew into Bioversity International. The fellowship is open to young scientists from developing countries, and selection is based on a competitive process that considers the originality of the research, the quality of the proposal and the likelihood that it will result in positive benefits for the home country. Fellows need to establish collaboration with a scientist at a host institute and are also allocated a Bioversity scientist as co-supervisor. ProgrammE goals The goal of the VFF Programme is to encour- age the conservation and use of plant genetic resources by awarding Fellowships to out- standing young scientists from developing countries. More specifically, objectives include: • To strengthen the capacity of developing countries to research urgent problems of interest to the home country as well as to the larger scientific community. • To increase the knowledge base in state-of- the-art areas of PGR science and to advance the frontiers of the science of PGR conser- vation and use. • To stimulate research linkages and ex- change between research institutions and PGR research partners around the world. The expected long-term impact of the Pro- gramme is that plant genetic resources for food and agriculture will be more equitably, more productively and more sustainably managed. study mEthods Given the nature of the VFF Programme, the approach to evaluation was to combine Kirkpatrick’s theory for the evaluation of learning and training (Kirkpatrick & Kirkpatrick 2006)1 with the Programme’s 1 Kirkpatrick, D. L. and Kirkpatrick J.D. (2006). Evaluating Training Programs (3rd ed.). San Francisco, CA: Berrett-Koehler Publishers From the Fellows’ Gallery http://www.bioversityinternational.org/training/research_fellowships.html IMPACT ASSESSMENT BRIEF NUMBER 8 logical impact assessment pathway. Kirkpatrick’s approach looks at four different levels of analysis: • Reaction: how participants react to the training pro- gramme. At its most basic, trainees should react posi- tively, because while a positive reaction does not ensure learning, a negative reaction almost certainly reduces the possibility of effective learning. • Learning: the extent to which participants change atti- tudes, improve knowledge and increase skills. • Behaviour: how the learning actually changes what the participant does, the changes that occur as a result of taking part in the training programme. • Results: the final outcomes of the changes in behav- iour brought about by participation in the training pro- gramme. Reaction and Learning are intrinsic to the training itself, while Behaviour and Results are beyond the scope of the training programme and relate more to the working envi- ronment of the trainee. Kirkpatrick’s levels offer a useful framework within which to evaluate the impact of the VFF Programme, while the logical impact assessment pathway provides much of the information around which to con- struct the evaluation. Figure 1 below shows the logical framework for the VFF Programme. The outputs, which result from the activities of the Fellows, were evaluated at the Reaction and Learn- ing levels. Fellows were not, of course, the only actors with a hand in those outputs; supervisors at host and home in- stitutes and at Bioversity also perform an important role through mentoring, and so need also to be considered. The outcomes that flowed from the outputs were evaluated at the Behaviour and Results levels, also with due considera- tion for actors other than the Fellows themselves. Data collection required many different approaches and methods. We interviewed and gathered data directly from people involved in the project. We also used online surveys and interviewed some participants by telephone. Table 1 shows the number of people in each category who were invited to take part in the survey, and the number of responses. Surveys were complemented with follow-up “ The results directly relate to the viability and importance of in-situ landrace conservation to the maintenance of genetic resources.” Figure 1. Logical impact assessment pathway for VFF Programme. IMPACT ASSESSMENT BRIEF NUMBER 8 interviews with four fellows (two men, two women) from Africa, Latin America, Southeast Asia and Eastern Europe. Four supervisors were interviewed; two from host institutes, one from a home institute, and one Bioversity International scientific advisor. Note that while some questions were specific to individual categories, others were common to one or more categories, allowing results to be triangulated in order to assess the validity of the responses. Summarising, the key questions we asked were: • To what extent are the fellows able to apply the knowledge gained through the VFF Programme. • To what extent has the Fellowship contributed to the professional development of the Fellow? • What is the impact of the Fellowship research on PGR at national, regional and international levels? • What are the strengths and weaknesses of the VFF Pro- gramme? Table 1: Numbers and response rates Invitations Responses % Fellows 25 16 64 Host Supervisors 22 9 41 Home Supervisors 5 2 40 Bioversity Scientific Advisers 22 11 50 ProjEct ImPact ApplIcAtIon of knowledge gAIned Despite some individual differences, all the fellows reported that their knowledge and skills had increased as a result of the Fellowship Programme. This was corroborated by host institute supervisors and Bioversity scientific advisors. Thirteen of 16 Fellows said that they have or will apply the knowledge and skills gained. For example, a Fellow from Benin said that training in tree domestication and genotype characterization received at Ghent University would in- crease expertise in molecular genetic methods at the Faculty of Agronomic Sciences at the University of Abomey Calavi. A Fellow from Turkey said that all the methods and proto- cols learned during the Fellowship had been applied in the laboratory and taught to students and post-docs. The most commonly cited obstacles to applying knowledge and skills were the lack of equipment and funding. Host institute supervisors and Bioversity scientific ad- visers agreed that the Fellows’ research had increased scien- tific knowledge in their areas of expertise. On a four-point scale, from completely achieved to not achieved, 23 out of 25 mentors said that increased knowledge had been mostly or completely achieved. Host institute supervisors were also asked to rate, on a three point scale, the extent to which the Fellowship increased the capacity of the Fellow. Six of the nine rated their Fellow improved to a high degree, the other three to a medium degree. The relationship between Fellow and host institution is clearly a critical factor for success, and the survey revealed that many of the Fellows had kept in contact and were developing new collaborative research projects. The relationship went further, with seven of nine host institutes creating strong partnerships with home institutes. Where there were difficulties in sustaining the relationship, the reason given was related to working culture, for example between strongly hierarchical institutions and those with a flatter organisation. contRIbutIon to pRofessIonAl development We asked Fellows to rate nine changes in their professional lives on a six-point scale. Most importantly, 14 of 16 Fellows reported “an increase in academic outputs,” and 12 felt that they had been “exposed to useful methods and technolo- gies”. Fellows identified many other positive changes that had benefited their careers, among them increased self- esteem, improved ability to negotiate and collaborate with peers, and the opportunity to build a sustaining network of professional relationships. The international prestige associ- ated with the VFF Programme was also cited as being very helpful. ImpAct on plAnt genetIc ResouRces The VFF Programme seeks to build capacity related to PGR in national agricultural research systems of developing countries. Of 16 Fellows, 15 stated that their home institute had indeed benefited from their participation. New collaborative research projects between home and host institutes, as mentioned above, are further testimony to the impact on home-country capacity. Host institute supervisors also indicated that in some cases research by fellows had made a distinct contribution to the knowledge base on PGR. One, for example, singled out a study of how farmers manage sorghum diversity in their fields. This study produced baseline data that were followed up nine years later “to produce some of the first time-series data [of] this nature. The results directly relate to the viability and importance of in-situ landrace conservation to the maintenance of genetic resources.” One of the most important factors for national impact is that Fellows actually return to their home country. Most did, only four of the 16 staying away, usually to continue or com- plete their PhD research, which leaves open the possibility that they will eventually return to their home country. One of the four indicated an intention to return within five years, and some did say that the lack of suitable positions that would make use of their newfound knowledge and skills was a factor against returning to the home country. Fellows also reported a direct impact on PGR in their home country. Fourteen of the 16 specifically said their Fellowships had benefited their home country, both through spreading knowledge and new methods to students, and through informing policy-makers directly about the role of agricultural biodiversity and consequences of genetic “ 14 of 16 Fellows reported an increase in academic outputs.” Bioversity International Via dei Tre Denari, 472/a, 00057 Maccarese, Rome, Italy Tel.: (39) 0661181 bioversity@cgiar.org www.bioversityinternational.org This work by Bioversity International is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. erosion. One Fellow was appointed a consultant for the country’s State of the World Report on Plant Genetic Resources for Food and Agriculture, visible recognition of the Fellowship and of the importance of PGR. The Global Plan of Action for the Conservation and Sus- tainable Utilization of Plant Genetic Resources for Food and Agriculture (GPA) provided a basis to assess the relevance of research by Fellows for the global PGR research agenda. The GPA lists several thematic areas, which can be consid- ered priorities for research on PGR. We assigned each Fel- low’s research to one of the GPA thematic areas. Fellowship research is dominated by two thematic areas: Supporting on-farm management and improvement of PGRFA and Expanding the characterization, evaluation and number of core collections to facilitate use. In total, Fellows’ research covered nine different thematic areas, and considering that some of the GPA’s thematic areas concern processes rather than research in the strict sense, we are pleased to note that Fellows covered more than half of the thematic areas relevant to developing countries. Fellows themselves all considered that their research had benefitted international efforts in PGR conservation and use and was relevant to many countries. lEssons lEarnEd and rIsk Factors Based on the responses received from participants, we iden- tified three aspects of the VFF Programme that needed to be addressed. • A more targeted approach, to increase the relevance of proposals received. • Increased engagement with Fellows after their fellow- ships. • Increased monitoring of results, for future assessments. Some measures have already been implemented: the call for applications now includes a list of eight priority topics on themes relevant to Bioversity International. The topics change from year to year, and the selection panel reports that it is receiving more relevant proposals as a result. A Fellows’ Gallery has been constructed for online access, which not only raises the profile of the Fellows and their research but also offers dynamic opportunities for ongoing engagement among all parties, including Fellows from other schemes ad- ministered by Bioversity. The evaluation study was unable to assess factors such as the impact on PGR management in the home countries or the extent to which Fellows had increased their skills, largely as a result of constraints in budget and timeframe. Nevertheless, the methodology developed, linking Kirkpatrick’s theoretical framework to the Fellowship Programme’s logical impact pathway, lends itself to a more in-depth analysis that could indeed illuminate these aspects. A careful ex-ante analysis, collecting data on the Fellows, host and home institute supervisors and Bioversity advisors before the start of each fellowship, combined with ex-post monitoring over several years, could shed further light on the impact of the Vavilov-Frankel Fellowship Programme and indicate further ways in which it could be strengthened. A primary risk factor for the VFF Programme is the con- tinued availability of funding. Despite the relatively small amounts required (approximately US$42,000 per Fellow- ship per year), continuing support has been difficult to ob- tain. The evaluation study indicates that there are indeed benefits to the Fellows, their home counties, and the sci- ence of PGR conservation and use, but is not able to indi- cate returns on investment or other tangible measures of the Programme’s value. To secure funding at the same time as ensuring the high quality of selected Fellows is an ongoing challenge for Bioversity. conclusIons Within the limits discussed above, the Vavilov-Frankel Fellowship Programme meets its objectives concerning the Fellows’ acquisition of new skills and knowledge and their ability to apply these in their home institutions. Receiving a Vavilov-Frankel Fellowship contributes to the professional development of Fellows, and the research that Fellows conduct contributes to knowledge about the conservation and use of plant genetic resources. This brief is based on Gotor, E. and Goldberg, E.D., Evaluating the impact of north-south research fellowships: The case of the Vavilov-Frankel Fellowship Programme on fostering conservation and use of agro-biodiversity, forthcoming in the South Asian Journal of Evaluation in Practice. Citation: Gotor E., Cherfas J., 2012. Evaluating the impact of the Vavilov-Frankel Fellowship Programme. Bioversity International Series of Impact Assessment Briefs, no. 8. Bioversity International, 4 p. S E P T E M B E R 2 0 1 2 For further information please contact Bioversity Impact Assessment Specialist Dr Elisabetta Gotor (e.gotor@cgiar.org)