A nnex to Workshop Report Eva luating Quality of Research-for-Development (QoR4D) in Process an d Performance Evaluations, February 2023, Rome Contents Annex 1: Workshop Agenda ............................................................................................................................................ 1 Annex 2: Workshop Contributors, Presenters and Participants .............................................................................6 Annex 2.1: Bios ....................................................................................................................................................................................................... 6 Annex 2.2: Analysis of participants .........................................................................................................................................................22 Gender of participants (72 in total) ..............................................................................................................................................22 ......................................................................................................................................................22 Participants affiliation (72 in total) ...............................................................................................................................................22 Participants profession type (72 in total) ..................................................................................................................................22 Annex 2.3: Attendance to the workshop............................................................................................................................................. 23 Participation by day (72 in total) .................................................................................................................................................. 23 Annex 3: Feedback on the Event .................................................................................................................................. 23 Annex 3.1: Response rate ............................................................................................................................................................................. 23 Survey response rate – 38 responses, of which 3 incomplete that were not considered. .......................... 23 Annex 3.2: Respondents’ profile............................................................................................................................................................... 24 Main place of work (35 respondents) ........................................................................................................................................ 24 Annex 3.3: Participation in the workshop ........................................................................................................................................... 25 How did you participate in the workshop? (35 respondents)...................................................................................... 25 Which sessions of the workshop did you attend? (35 respondents, multiple options allowed) ............. 25 Annex 3.4: Feedback on the event......................................................................................................................................................... 26 How satisfied were you/were you not with the event? (35 respondents) ............................................................ 26 ....... 26 Why were you/were you not satisfied with the event? (35 respondents) ............................................................ 26 Which sessions did you enjoy the most? (35 respondents, multiple options allowed) ................................. 27 Which sessions did you enjoy the least? (35 respondents, multiple options allowed) ................................. 27 Would you want to attend this kind of event in the future?............................................................................................ 27 What were your biggest takeaways of the event? ............................................................................................................. 28 What would have made this event better? ............................................................................................................................ 29 Annex 4: Cases of study ............................................................................................................................................... 30 Annex 4.1: Case Study 1 ................................................................................................................................................................................. 30 Annex 4.2: Case Study 2.................................................................................................................................................................................31 Annex 4.3: Case Study 3 .............................................................................................................................................................................. 33 Workshop Report Evaluating Quality of Research-for-Development (QoR4D) in Process and Performance Evaluations, February 2023, Rome: https://iaes.cgiar.org/evaluation/publications/workshop-report-evaluation- guidelines-applying-qor4d-frame-reference Annex to Workshop Report Annex 1: Workshop Agenda Day 1 Monday, February 27 Mins SETTING THE STAGE: EXPECTATIONS, BACKGROUND AND CGIAR CONTEXT 14:15 REGISTRATION 15 Welcome and Background: Allison 14.30 Grove-Smith, IAES Director- 10 Introduce IAES and ERG Welcome and Introductions: facilitator and support team, groups of participants, clarify objectives and desired outcomes, format, and 14.45 expectations – Svetlana 15 Negroustoueva, Function Lead, IAES Menti poll/world cloud/map- Where are you joining us from? (Facilitated with Q&A) The Quality of Research for Development in the CGIAR Context (Qo4RD) Frame of Reference: - Genesis, use and aspirations: Holger Meinke, Chair of Independent Science for 15-15.20 Development Council (ISDC) 20 - Use of QoR4D Frame of Reference in CGIAR initiative proposal reviews: Amy Beaudreault, ISDC Secretariat Lead Q&A Facilitated by Guy Poppy, ERG (Icebreaker-Discovery) Paired conversations- Identify your dream in evaluating research for development (R4D). 15.20-15.30 10 - Video on CGIAR Eval Policy and Framework (2022) Facilitated by Natasha Valentine (Panel with Q&A): The Journey Behind the Guidelines - Evaluations of science in CGIAR - Legacy - CGIAR 15.30-16.00 30 Independent Evaluation Arrangement- Federica Coccia, former evaluation analyst, CGIAR IAE 1 Annex to Workshop Report Day 1 Monday, February 27 Mins - Approach to evaluative reviews of CRPs, 2020- Paolo Sarfatti, Independent consultant, CGIAR and Adviser, Nutrition Research Facility - The QoR4D Evaluation Guidelines: the story and the gist – Svetlana Negroustoueva, IAES Evaluation Function Lead Q&A Facilitated by Roberto LaRovere, ERG 16.-16.15 Break 15 (Panel with Q&A) Common ground- Differences and similarities with other contexts: chaired by Hanna Ewell (GIZ): - Juha Uitto, Director, GEF evaluation office - Claudio Proietti, MEL Advisor, CIRAD 16.15- 16.45 - Dr. Rao Ram, former Director, 20 ICAR, India Michael Spilsbury, Director, UNEP evaluation office (remote) (Mentimeter Activity) Word cloud Similar contexts, organizations to reinforce common ground Facilitated by Natasha Valentine (Presentations with Q&A) CGIAR story continued: Approaches and Methods 25 - Technical Note on Bibliometrics- Etienne Vignola-Gagné, Research Analyst, ELSEVIER/Science-Metrix 16.45-17.20 (remote) Q&A Facilitated by Jillian Lenné (Activity) Think, pair, share: one thing that you take away from the 10 presentations. Facilitator: Natasha V Closing remarks for the 1st day: 17:20-17:30 15 Conclusions, framing the next day 2 Annex to Workshop Report Day 1 Monday, February 27 Mins IAES- Svetlana N, Evaluation Function Lead, IAES - Allison Grove-Smith, IAES Director Day 2 Tuesday, February 28 Mins APPLYING THE GUIDELINES: LEARNING TOGETHER Welcome back: Highlights and Reflections from the 1st day; Presentation of the aims and format 10 of the 2nd day: Facilitated by Svetlana Negroustoueva, IAES Evaluation Function Lead 9.00-9.25 • Quiz on the Guidelines (PPT) • Format of the day: 15 Facilitator Natasha V (Presentation with Q&A) Evaluations of CGIAR portfolio, story continued • QoS in RTB CRP evaluations- the journey – Jillian Lenné, SME for evaluations of RTB (2015, 2020) and GLDC (2020) CRPs • Video from FAO event- Andrew Ash, ISDC member 9.25-10.00 • GiZ evaluation of QoS- Jörg 35 Helmschrot, former project lead, GOPA (remote) • GENDER Platform evaluation- Lora Forsythe, Tanya Strathers –NRI, SMEs for Gender Platform evaluation (remote) Q&A Facilitated by Sonal Zaveri, ERG (Break- out) Case studies by evaluation teams: in-person and online 10.15- 11.30 60 Facilitated by ERG members & support in each group Break- flexible 15 3 Annex to Workshop Report Day 2 Tuesday, February 28 Mins (Facilitated) Results of case studies on PPT 11.30- 12.00 30 Facilitated by Svetlana Negroustoueva 12.00-13.00 Lunch 60 NEXT STEPS: TRACKING UPTAKE AND LEARNING: HOW (Facilitated) Results of case studies on PPT- continued 13.00- 13.30 30 Facilitated by Svetlana Negroustoueva (Energizer) Game, facilitated by 13.30-13.45 15 Natasha V. (Interviews) Vision and the Learning Needs • Hasna Ziraoui & FatimaEzzahra MOUTIK, Project Management & Research Assistant, Mohammed VI University, Morocco • Paolo Sarfatti – Adviser, Nutrition Research Facility, 13.45-14.15 EU 30 • Emma Rotondo- Coordinator, ReLAC/Peru • Carlos Tarazona – Evaluation officer, FAO • Enrico Bonaiuti, MEL Research Team Leader, ICARDA, a CGIAR Center Q&A Facilitated by John Gargani, ERG (Activity) Menti-world cloud - Metrics of learning & uptake- indicators Commentary from in-person and 14.15-14.30 15 online Facilitated by Natasha Valentine (Panel) Sense-making- activity / Mechanisms for learning from the uptake of the Evaluation Guidelines 14.30-15.00 in CGIAR and beyond 30 • Renata Mirulla, Facilitator, EvalForward CoP 4 Annex to Workshop Report Day 2 Tuesday, February 28 Mins • Michael Spilsbury, Director, UNEP Evaluation Office (remote) • Svetlana Negroustoueva, Evaluation Function Lead, IAES Facilitated by Ola Ogunyinka, ERG (Break- out) – Learning by doing: what and how to track uptake and use by both for donors, scientists, partners and final beneficiaries 15.00- 15.45 45 • Groups: CGIAR and Non-CGIAR: SMEs and evaluators Presentation of Results from groups 15.45-16.00 Break 15 (Panel and Facilitated discussion, Q&A) Learning Pathway and Mechanisms • Journey of the RQ+ 16.00-16.45 tool, Rob McLean, Senior 45 Program Specialist, IDRC (online) Facilitated by Allison Grove-Smith, IAES Final remarks: Holger Meinke, Chair 5 of Independent Science for Development Council (ISDC) 16.45- 17.00 Concluding remarks and next steps Group picture+ Screen shots for online participants 10 5 Annex to Workshop Report Annex 2: Workshop Contributors, Presenters and Participants Annex 2.1: Bios Workshop Contributors and Presenters Allison Grove Smith is the director of CGIAR’s Independent Advisory and Evaluation Service (IAES): the secretariat for the Independent Science for Development Council and the Standing Panel on Impact Assessment, and it implements CGIAR’s independent evaluation plan. Allison has played a leading role in development and research settings for 25 years and has planned and implemented strategies, portfolios and change processes. She directed quality assurance and oversaw M&E in the Europe/Middle East/North Africa region for Catholic Relief Services, one of the world’s largest NGOs. Allison served as a global initiative manager for genetic resources conservation at Bioversity International in Rome, among other appointments. Working alongside scientific teams across multiple disciplines in topics ranging from the world’s banana and plantain genebank to forest genetic resources to seed system policy, Allison and the genetic resources team developed and implemented projects supporting better conservation and use of agrobiodiversity. At Bioversity, she also spearheaded partnership strategies, redesigned global business processes and improved knowledge management systems for the institute. Before joining IAES in 2019, Allison worked as a consultant and contributed to projects related to sustainability financing and sustainable business practices Svetlana Negroustoueva Leads Evaluation Function under CGIAR’s Independent Advisory and Evaluation Service (IAES). She is a PMP Certified professional with over 20 years of experience designing and conducting evaluations, assessments, monitoring and research activities, and quantitative and qualitative data collection and analyses. She has worked on M&E activities at the intersection of R4D, sustainable landscapes, energy, health, food security, gender and social inclusion, for projects across a range of funders and implementing entities. Prior to joining CGIAR in 2020, Svetlana served at African Development Bank, World Bank, the Global Environmental Facility (GEF) and Climate Investment Funds (CIF) in positions concerned with demand-driven and independent evaluations. She was an independent evaluator herself, for a variety of clients. Svetlana’s passion and expertise is in the domain of gender and evaluation; she was the co-chair of EvalGender+, the global partnership to promote the demand, supply and use of Equity Focused and Gender Responsive Evaluations. Svetlana is a Russian and US national, and holds a Master’s degree in Public Affairs from the University of Texas at Austin in the US, and Advanced BA in Public Administration and Social Research from Lomonosov Moscow State University (MSU), Russia. Holger Meinke Holger is Adjunct Research Professor for Global Food Sustainability at the University of Tasmania, Australia. As an internationally renowned leader in climate, agriculture, and food systems research, he provides intellectual leadership at local, national, and global level. He has published over 130 refereed papers in disciplinary and transdisciplinary journals. He supervises three PhD candidates. Professor Meinke is passionate about education that fosters critical thinking and high-quality research that has impact. He is a member of five editorial boards. 6 Annex to Workshop Report Workshop Contributors and Presenters He was Director of the Tasmanian Institute of Agriculture for nine years (2011-2019). Before joining the University of Tasmania, Holger held the Chair of Crop and Weed Ecology at Wageningen University, The Netherlands (2007–2011). Prior to that he was a cropping systems scientist with the Queensland Government (1988–2007), where he co-developed the agricultural systems simulation platform, APSIM. In 2014 he became a Graduate of the Australian Institute of Company Directors (GAICD). He is a member of the National Committee for Agriculture, Fisheries and Food, a subcommittee of the Australian Academy of Science that advises the Australian Government. He has a BSc from Hohenheim University (Germany, 1983), an MSC from the Technical University Berlin (Germany, 1986) and PhD from Wageningen University (The Netherlands, 1996). Amy Beaudreault is the Lead for ISDC within IAES. She uses her expertise in global nutrition and food systems to support the ISDC during the One CGIAR reform, via work plan implementation and multidisciplinary activities. With a 20-year history in newly created positions, she has cultivated novel pathways to advance academic and nonprofit institutions. Her experiences range from research, communication, policy, and program development to public-private engagement and stakeholder consensus building—all underscoring the vital role of science-based solutions. Most recently she was a research fellow in the Global Food Security Project and the Global Health Policy Center at the Center for Strategic and International Studies (CSIS) in Washington, DC. In addition, she held leadership positions at the University of California, Davis’ World Food Center and the New York Academy of Sciences. Dr. Beaudreault has published in peer-reviewed journals, organized dozens of scientific symposiums, and nurtured international partnerships. She holds a BS in journalism from Ohio University, and an MS in agricultural communication, a PhD in agricultural education and extension, and a graduate certificate in survey research from The Ohio State University. Juha Uitto is Director of the Independent Evaluation Office (IEO) of the Global Environment Facility (GEF), an operational entity of the financial mechanism for five multilateral environmental conventions on biodiversity, desertification, climate change, and environmentally harmful chemicals. He has headed the GEF IEO since 2014, prior to which he worked as Deputy Director and Evaluation Advisor in the UN Development Programme (UNDP) IEO and an M&E specialist at the GEF. He has more than two decades of experience as an evaluator during which time he has managed and conducted numerous evaluations at the global, regional and country levels focusing on the nexus between environment and development. Juha spent the 1990s as environmental research and training programme manager at the UN University. He has published widely on topics related to sustainable development, environmental hazards, and evaluation, and has been a visiting scholar at Rutgers University and Kyoto University. A native of Finland, Juha was educated at the Universities of Helsinki and Lund and holds a PhD in Social and Economic Geography. Clemencia Cosentino is the Director of Evaluation at the Food and Agriculture Organization (FAO) of the United Nations. Before this, she was the Chief Evaluation Officer and Evaluation and Assessment Capability Section Head of the US National Science Foundation (NSF), a Senior Fellow and Director of STEM Research at Mathematica and the Director of the Program for Evaluation and Equity Research of the Urban Institute. Clemencia is widely recognized as an expert on science, technology, engineering, and mathematics (STEM), including the underrepresentation of minorities and women in STEM-related degree programmes, training, and careers. Over the past decade, she has been focusing on designing and developing complex data systems that leverage existing data and serve multiple functions (monitoring, research, and evaluation). She received her master’s and doctoral degrees in sociology from Princeton University, with a focus on education and international development. 7 Annex to Workshop Report Workshop Contributors and Presenters Federica Coccia has 20 years of experience with international development organizations, in the food security and agriculture sectors, with considerable experience in evaluating agriculture for development programs, including agricultural research for development programs. She was coordinator of the Monitoring, Evaluations and Impact Assessment team of the Forests Trees and Agroforestry CGIAR Research program between 2019 and 2021, after having worked as an Evaluation Analyst for the CGIAR-Independent Evaluation Arrangement from 2012 to 2018. She has a long-standing experience in evaluation approaches and methods for evaluating agricultural scientific research for development. With the FAO Office of Evaluation, She collaborated on various evaluations including the 2019 Evaluation of FAO’s Strategic Framework and the 2006 Independent External Evaluation (IEE) of FAO. Following the IEE, she was closely involved in FAO’s reform process, which allowed me to gain a good understanding of the governance and management of large complex organizations. Paolo Sarfatti has over 35 years of experience on international cooperation for agricultural development, mostly in Africa. In the first period of his career his main field of research and action has been Sustainable management of natural resources for agriculture and rural development in developing countries. In more recent years has concentrated his interest on how to improve, through international cooperation, coherence and effectiveness of agricultural and rural transformation policies and programmes in developing countries. In this context, has a long experience of working with the EU, and international organisations such as FAO, IFAD, and the CGIAR. He has a long-standing experience of collaborating with African agricultural research organisations such as FARA and the SROs. During the last years he has focused his work in the field of Monitoring and Evaluation, with particular reference to research for development. Jillian Lenné is an independent consultant in agricultural research for development with over 40 years experience in tropical agricultural research, development, management and governance, including 22 years with CGIAR institutes (CIAT, ICRISAT and IRRI) and 8 years with UK-based institutes. My experience as a consultant in project and programme review through short-term assignments spans more than 40 countries in Latin America, Asia and sub-Saharan Africa with relevant experience in semi-arid and dryland cropping systems; horticultural systems especially vegetables; tropical rice systems; tropical fodder and crop/livestock systems; root and tuber crops; and tropical agro-biodiversity management. She has worked with the CGIAR ISPC on evaluating most of the CRP Proposals and with the IEA on the 2015 Evaluation of RTB and with CAS on the 2020 Evaluations of RTB and GLDC. More recently, she has been working with the Independent Advisory and Evaluation Services in using and developing the Quality of Research for Development Framework for evaluation of CGIAR Programs and Initiatives. She is also the Editor in Chief of Outlook on Agriculture. Rao Rama Darapuneni obtained PhD from IIT, New Delhi. Has about four decades experience in ICAR. Held senior positions as National Director, NAIP cum DDG (Engg) at ICAR Hq, New Delhi, Director, ICAR-NAARM, Hyderabad and ICAR-Emeritus Scientist at PJTSAU, Hyderabad. Contributed on many policy dialogues and strategic initiatives in R&D, organisational reforms, public-private partnerships, research management, innovation management, digital learning, educational planning, open access and capacity development. Current interests in impact evaluation and digital learning. Has over 135 publications including 40 papers in peer reviewed national and international journals. Besides professional work, takes keen interest in community social and spiritual activities. 8 Annex to Workshop Report Workshop Contributors and Presenters Dr. Jörg Helmschrot, based in Namibia, holds a diploma degree in Geography, Geology and Urban Planning from the University of Jena in 1999. Dr. Helmschrot worked as researcher at the Universities of Jena, Hamburg and Washington. He is appointed as extraordinary Professorship with the University of Stellenbosch Water Management Institute since 2016. As SASSCAL’s Director of Science and Technology and Capacity Development (2016-2021) in Windhoek, Namibia, he was in charge for implementation and further coordinating the development of the SASSCAL research and capacity development agenda as well as the establishment of the Open Access Data Centre. Dr Helmschrot is currently supporting the CO-HYDIM-SA project lead by KIT IMK-TRO in developing a 4-year project in southern Africa and GOPA in a FIA/GIZ supported research on Quality of Science of the CGIAR portfolio. In his career, he coordinated and was involved in more than 45 research projects focusing on water and global change science as well as food security. His research is reflected in more than 90 scientific publications. He also serves as scientific advisor to the German, South African and Namibia Governments, in various international committees/panels, as reviewer and advisor to funding agencies and as reviewer for more than 20 and editor for 4 high- ranked scientific journals. Tanya Strathers Dr Tanya Stathers, Professor of Sustainable Agri-food and Postharvest Systems and Practice at the Natural Resources Institute (NRI), University of Greenwich, UK. Tanya’s research has focused on supporting sustainable agri-food systems across sub-Saharan Africa, with particular emphasis on the reduction of and quantification of food losses at and after harvest. She has worked within transdisciplinary teams on deepening understanding of a broad range of agricultural development issues including: urbanizing food systems, rural-urban interdependencies, drivers of food choice, agricultural adaptation to climate change, innovative finance, seed systems, agricultural innovation systems, experiential and multi-stakeholder social-learning processes, poverty impacts of market certification standards, gender and diversity aspects of agri-food systems, as well as field and laboratory research trials into pre and postharvest constraints across a variety of tropical grain, root and tuber, and tree crops. She has lived and worked long-term in Tanzania, Kenya and Papua New Guinea. She has evaluated and monitored projects and reviewed proposals for the EU, DFID, CGIAR and philanthropic organizations, and led systematic evidence syntheses and meta-analyses. She has also co-developed several hands-on agricultural training programs, and teaches and co-supervises postgraduate students registered at African and UK universities. Lora Forsythe is Associate Professor of Gender, Inequalities and Food systems at the Natural Resources, University of Greenwich. Lora leads the Gender and Social Difference Research Group and is a specialist in social inequalities in the context of agriculture, food systems and natural resource-based livelihoods. Lora has expertise in gender- based violence and food systems; vulnerability and adaptive capacity to climate change; land and natural resources rights, management and governance; food and nutrition security; agricultural livelihoods and market participation, and food cultures. She is a mixed-method interdisciplinary researcher with experience in high-level policy-oriented research, with competencies in in-depth, survey and longitudinal interviewing; case studies; participatory action research; research co-design; capacity strengthening, and monitoring, evaluation, learning (MEL) and impact methodologies. 9 Annex to Workshop Report Workshop Contributors and Presenters Etienne Vignola Gagné holds 15 years of experience as an analyst of public policy, innovation systems and organizational change. He has led in-depth case studies of major biomedical innovation initiatives and policies, conducting hundreds of expert interviews with researchers, policymakers, industry representatives and others along the way. He has conducted mandated enquiries for scientific agencies such as the European Commission, the Wellcome, or the Canadian IRSC/NSERC/SSHRC focused on improving public policy in areas such as the evaluation of transdisciplinary research programs; developing bibliometric metrics for capturing societal outcomes of research, or international comparisons of COVID-19 test and trace administrations and systems; or tracking genomic innovation in cancer care. He has published or co- authored scientific contributions in venues such as Quantitative Science Studies, History and Philosophy of the Life Sciences or Science and Public Policy. Dr. Vignola- Gagné holds a doctoral degree in political science from the University of Vienna, with prior training in science and technology studies. Enrico Bonaiuti is the Research Team Leader of the Monitoring, Evaluation, and Learning (MEL) Team at ICARDA. He coordinates and leads large multi-country teams, ensuring timely deliverables and effective implementation across operations and service units.  He has 15+ years of experience as a coordinator and service partner for large, complex international research programs/projects with sustainable development and demand-driven mandates. Previously, Enrico served as Program Coordinator for the CGIAR Research Program on Roots, Tubers, Bananas, and Dryland Systems. He was responsible for implementing a MEL framework, defining indicators, conducting evidence-based evaluations at the systems level, and enhancing co- learning among stakeholders. He holds a Master’s Degree in Agricultural Sciences from the University of Florence, Italy, and a Master’s Degree in Marketing and Economics from the Bocconi University in Milan, Italy.   Hanna Ewel is a technical advisor for monitoring and evaluation, and gender focal person for the German Development Corporation’s (GIZ) Fund International Agricultural Research (FIA). My career interests are rooted in sustainable development and climate resilience, gender, food security and nutrition. My research experience includes qualitative studies in fields of gender and nutrition in the context of orange fleshed sweet potato adoption in Kenya and Malawi; human impacts on biogeochemical cycles in Colorado; sustainable fisheries management in the Turks and Caicos and the Galapagos Islands; and crop breeding priorities in sub-Saharan Africa. Hasna Ziraoui is currently a Chargée de Mission within the Office Of The President at University Mohammed VI Polyctechnic (UM6P). As a Chargée de Mission she works on several files related to research, partnerships, knowledge management, structuring projects as well as broader initiatives such as the Great Green Wall Initiative. In 2017 Hasna joined  Hicham El Habti’s (current UM6P President) team within the Mouvement (organizational change initiative) at OCP Group. She facilitated and co-lead initiatives within the Mouvement to reach their full potential. Her role ranged from advice process, project management, strategy and deployment as well as building bridges with other initiatives.  Hasna first joined OCP in 2012 to improve and streamline customer service processes. She then worked on implementing, monitoring and reporting processes in the shipping and logistics departments.    Prior to joining OCP Hasna worked for one of USA’s leading food and beverage innovation consultancy in San Francisco. She was the lead R&D specialist for major US food chains. Hasna is a graduate from Grenoble Ecole de Management, 2005.   10 Annex to Workshop Report Workshop Contributors and Presenters Norbert François TCHOUAFFE TCHIADJE is currently, senior lecturer at the University of Dschang and researcher at the Pan-African Institute for Development in Cameroon. Previously Norbert was a visiting scholar at the MIT-DUSP (USA) from 2014 to 2015 and was also assistant lecturer (Distance learning) at the International Institute for Water and Environmental Engineering (2iE), Ouagadougou, Burkina Faso from 2007-2012. He is an Agricultural engineer holding a Postdoctoral Certificate in Social Vulnerability and Disasters, a Postdoctoral certificate in Building Skills for a Successful Career; a PhD in Applied Development Sciences, a Certificate in advanced studies (CAS) in integrated water resources management from Bern University of Applied sciences. Emma Rotondo is a Peruvian social anthropologist with 30 years of experience in program planning, design, monitoring and evaluation in 15 countries in the Latin American region. She has conducted internal and external evaluations, capacity building process, as well as program monitoring. She facilitates and trains in M&E, in different cultural contexts and levels from national, sectoral, sub-national and community level institutions. She applies mixed approaches in evaluation and develops knowledge management to facilitate the use of evaluation for different audiences. She is the author of about 40 texts including guides, articles, manuals in her area of expertise. Currently, she is President of the Peruvian Evaluation Network (EvalPeru) and co-coordinator of the working group on Human Rights and crisis of the Latin American Evaluation Network (ReLAC).  Claudio Proietti works at the French Agricultural Research Center for International Development (Cirad) where he provides training and methodological support to project teams in his function of monitoring and evaluation advisor. He holds an MSc in Tropical Agriculture from the University of Florence (Italy) and has more than 20 years of experience in Africa and Latin America. Prior to joining Cirad, he worked with international research and development organizations (CGIAR, GIZ) and non- governmental organization in the agricultural, rural development and local governance sectors. During the last years, his work focuses on theory-based and outcome-oriented planning, monitoring and evaluation approaches particularly in the agricultural research for development sector. Fatima Ezzara Moutik is currently Research Assistant at Entity of strategy, University Mohamed VI Polytechnic (UM6P), Morocco. Recently she has been working on quality analysis of Web of Science publications using selected bibliometrics. She did her first degree in engineering in Agronomy at Institute of Agronomy and Veterinary Hassan II (IAV) in 2018. She also obtained her Master of Biotechnology & Agrobioscience at UM6P in 2021.   Carlos Tarazona is a Senior Evaluation Officer at the Food and Agriculture Organization of the United Nations (FAO) where he is responsible for coordinating selected thematic and country evaluations. He is an evaluation specialist with more than 17 years of experience in the evaluation of agricultural and rural development programmes. Before joining OED, Carlos served as an evaluation officer at the International Atomic Energy Agency and managed the Independent External Evaluation of FAO. Carlos holds a master’s degree in international agriculture from Reading University in the United Kingdom and Larenstein University in the Netherlands. 11 Annex to Workshop Report Workshop Contributors and Presenters Renata Mirulla is a facilitator of EvalForward Community of Practice by supporting knowledge sharing and information exchange among members, summarizing topics discussed and highlighting good practices in evaluation, ensuring updates on events, resources, guidelines, that are relevant to evaluation in agriculture, food security and rural development. She has more than 20 years of experience working in different development organizations on Food Security, Nutrition, Agriculture and Rural Development. Dr. Robert McLean is a Senior Program Specialist in Policy and Evaluation at Canada’s International Development Research Centre (IDRC) and concurrently a Fellow of the Integrated Knowledge Translation Research Network (IKTRN) at the Ottawa Hospital/University of Ottawa. Rob’s broad interests lie in understanding how human creativity can create a better world. Pursuing that aim, he has worked across government, private, and NGO sectors and has published scientific research and invited commentary in venues ranging from Nature to the Stanford Social Innovation Review. He is co-author of the new book, Scaling Impact: Innovation for the Public Good published by Routledge NYC. Rob earned his Ph.D. in the Department of Medicine at Stellenbosch University, South Africa. He holds an M.Sc. from the Global Development Institute of the University of Manchester, England, and two undergraduate degrees following studies at Carleton University, Canada and the University of KwaZulu-Natal, South Africa. John Gargani, PhD, MBA has 30 years of experience as an evaluation practitioner, researcher, writer, speaker, and teacher. With Gargani + Company, he directs projects around the world that range from early-stage program design to multi-site randomized control trials. As an Adjunct Professor at Claremont Graduate University, he teaches classes on program design, social impact measurement, and social entrepreneurship. He coauthored Scaling Impact: Innovation for the Public Good with Robert McLean of IDRC, which explores the meaning and practice of scaling from the perspective of Southern innovators. He also publishes per-reviewed journal articles on topics related to impact measurement, evaluation theory, and human-centered practice. (Full bios available at ERG page)   Sonal Zaveri has over 30 years’ experience in strategic planning, program design, capacity building, mentoring, and evaluation. She has a Ph.D. in Social Work and was a Fleishman Fellow at Duke University, USA. She has worked in more than 25 countries in Asia, Asia-Pacific, and Central Asia, East and West Africa, Middle East, and Eastern Europe at grassroots, sub-national, national, regional, and international levels bringing a broad contextual experience. She holds regional and global leadership positions – including being a board member of the International Evaluation Academy (IEAc).   Ola Ogunyinka, a Resource Economist by training, was a Senior Research Fellow and Monitoring, Learning and Evaluation Specialist at the Natural Resources Institute, University of Greenwich, United Kingdom. He has had a long association with the CGIAR system dating back to early 1991 to 2006 while working with IITA, Ibadan Nigeria as the Head of the then Monitoring and Evaluation Unit. Dr. Ogunyinka’s has a Ph.D in program evaluation, and his 35 year’s evaluation experience cut across sub-Saharan Africa and South-East Asia and includes a range of large-scale, multi-country program, process, performance, and impact evaluations.  (Full bios available at ERG page)   12 Annex to Workshop Report Workshop Contributors and Presenters Guy Poppy is a Professor, served as the UK’s Food Standards Agency’s (FSA’s) Chief Scientific Adviser from 2014 to 2020. He continues with his research in global food security at the University of Southampton, where he is Professor of ecology and directs interdisciplinary research as the Associate Vice President. He has significant research experience in food systems and food security and has advised governments around the world on these issues. He is currently a member of the Research Excellence Framework (REF2021) panel assessing the quality of agriculture, food, and veterinary science in the UK, having previously served on the REF2014 panel.  (Full bios available at ERG page)   Roberto La Rovere is an independent evaluator. He previously worked at three CGIAR Centers in West and East Africa, Syria, and Mexico, being focal point for the Standing Panel on Impact Assessment (SPIA) and the Science Council until 2010. He has consulted extensively for the UN Rome-Based Agencies, the Global Environment Facility (GEF) and for more than a decade he has been a senior evaluation advisor at UNDP, at the Green Climate Fund (GCF), and at the Pan American Health Organization (PAHO). He authored a chapter on Evaluation of Climate Change, Environment and NRM in the CGIAR in the book “Evaluating Environment in International Development”. He holds a Ph.D. in natural resource management (NRM) economics, and degrees in development economics and agricultural sciences and systems as well as specializations in evaluation and climate change. (Full bios available at ERG page)   Frank Place is a senior advisor with IFPRI following upon several years being the director of the Policies, Institutions, and Markets CGIAR Research Program, hosted at IFPRI.  In 2022-23, he is working in three main areas with the CGIAR: (1) supporting the Systems Transformation science group, notably in the areas of oversight of the impact area platforms and MELIA related topics, (2) leading a workstream on strengthening portfolio level coherence in outcome and impact assessment with the Portfolio Performance Unit and (3) co-leading the policies and institutions work package of the agroecology initiative.  He has significant experiences in impact assessment with PIM and previously with the World Agroforestry Center and was a longstanding member of the MEL CoP within CGIAR. Karen Jay-Yina has over a decade's experience with M&E, research, knowledge management - translating evaluation evidence into action for multi-sectoral programmes including food and agriculture, policy and advocacy, climate change, health systems among others. She has served with the United Nations Industrial Development Organisation, in the private sector-led development space, the academia and research centres, using quantitative and qualitative approaches to inform the evidence-based strategy of action for adapting development programmes. Before joining CAS secretariat, she worked as an Action Researcher with the International Fertilizer Development Centre, as a part of the Partnership Resource Centre team; and as an M&E expert with UNIDO-Nigeria. An alumnus of the United Nations Executive Leadership Program in Evaluation and Sustainable Development Goals, she has developed expertise in mainstreaming the 2030 Agenda for Sustainable Development. A Nigerian national, Karen Erdoo holds the University of Exeter’s One Planet MBA, a sustainable business education that equips MBAs to be responsible economic, social, and environmental stewards. She also holds an M.Sc. in Public Health from the University of Lagos 13 Annex to Workshop Report Workshop Contributors and Presenters Natasha Valentine is a Canadian Human Resources Generalist with over 20 years of consulting and operational experience. She has a unique combination of strengths, including facilitation, lecturer, and HR experience. An energetic collaborator, she develops interactive, engaging educational content for on campus and online students. Natasha has notable experience in talent management and career coaching. The last six years have been in the academic sector, primarily with Rome Business School as Program Director for the Master of Human Resources Management. She has designed and delivered hundreds of presentations/lectures to a variety of universities in Rome. Topics have included global talent management, diversity, biases in the workplace and cross culture communications. Having a culturally diverse mentality has helped her to understand and interact better with employees, students, and professionals. Participants In-Person and Online Michael J Spilsbury is the Director of the Evaluation Office of the UN Environment Programme. Mike has more than 27 years professional evaluation experience in international organizations. He joined UNEP in 2005 as an Evaluation Officer and has performed various roles in the Evaluation Office, serving as its Director since 2013. He is active in the United Nations Evaluation Group’s Peer Review Working Group. Prior to joining UNEP, he worked as an international research scientist for the Center for International Forestry Research (CIFOR, a CGIAR center) based in Bogor, Indonesia (1995- 2005). He was responsible for impact assessment, evaluation of research and strategic planning. A significant contribution, in 1996, was to introduce the concept of ‘impact pathways’ (theories of change) into project and programme planning in the centre. Mike has also worked in the field and in the private sector. He was a postdoctoral researcher studying lowland tropical forest hydrology in remote logging concessions and protected areas in Malaysian Borneo and a manager and designer of environmental development projects in the private sector. He holds a Doctorate in forest ecology and modelling and a Master of Science in forest management from Oxford University and Bachelor of Science in biology from York University, UK. He has authored more than 40 publications. Andrew Ash has recently retired as a Chief Research Scientist in CSIRO Agriculture and Food. He has 35 years research experience in tropical rangelands and agricultural systems in northern Australia and south-east Asia with a particular emphasis on developing management systems to improve profitability and environmental outcomes for rural communities. Andrew has a particular interest in achieving on-ground impact from research and implementing appropriate monitoring and evaluation methodologies. He also has taken national leadership roles in adapting to the impacts of climate variability and change. He has published over 200 scientific papers, book chapters, conference papers and technical reports. He is currently Chair of the Reef and Rainforest Research Centre and sits on a number of independent science panels in Australia. Alison Poulos leads the design, implementation and continuous improvement of Bioversity International’s Results Based Management (RBM) monitoring system; and is a member of the Project Coordination Unit (PCU) of CGIAR. She also coordinates the institutional Annual Plan of Work and Budget and is our focal point for requests coming from the CGIAR Research Program Lead Centers. 14 Annex to Workshop Report Workshop Contributors and Presenters Mariafernanda Cataño is a people-oriented and highly organized Project Management Specialist working for CIMMYT-HQ in Mexico. She is a certified professional in project management (PMD-Pro) with 5+ years of experience developing and implementing high-level project management methodologies, standards, and tools for research and development projects across multiple geographies, including South Asia, Eastern Africa, and LATAM. Mariafernanda also serves as the interim advisor on project coordination for the CGIAR Project Coordination Unit, bringing her experience in setting up PM standards and supporting PCU’s efforts in consolidating a project management framework for CGIAR. Adebayo Solomon is a member of the Project Coordination Unit (PCU) of CGIAR, representing IITA. Previously, he was an Education Specialist at World Bank Group. He has worked with Deloitte as Organizational Development Advisor, Strategic Planning Specialist at Options Consultancy Services Ltd., and Senior Manager for Action Aid. Oluchi Ezekannagha is a seasoned professional with extensive experience in research, project design, portfolio management, implementation and evaluation. With academic expertise in nutrition sensitivity, child malnutrition, and nutrition policy, and practice expertise in agriculture R&D, food security, global health, and maternal and child nutrition, she has contributed significantly to various organizations in Africa. Oluchi has a PhD in Human Nutrition from Stellenbosch University and has held several positions, including Technical Assistance for Nutrition (TAN) Capacity Building Sub-consultant, Graduate Research Fellow at the International Institute of Tropical Agriculture (IITA), Nutrition Program/Research Officer at Food Basket Foundation International, and Health Officer at Jos South LGA Primary Health Care. Donald Menzies is an economist specialised in climate change and development, with a background in impact management processes. Donald is seconded from the UK Government’s Foreign, Commonwealth and Development Office to support the CGIAR’s pivot towards impact in a climate crisis. Rodrigo Paz Ybarnegaray is a design, planning, monitoring, evaluation and learning researcher and practitioner with twenty years’ experience. My work focuses on evaluating and enabling innovation, and facilitating critical transitions towards transformative change. He has evaluated research and development initiatives funded by DFID, USAID, EU, CIP, CIAT, IFPRI, WorldFish, CIMMYT, OXFAM, CARE, HEIFER International, Practical Action, COWATER-SOGEMA, SNV, IDRC, The McKnight Foundation, IFAD, SIDA, NIRAS, ECORYS, IIED and FAO. He has extensive experience using an array of methods in design, planning, monitoring, evaluation and learning: program theory, theory of change, developmental evaluation, realist evaluation, contribution analysis, modus operandi, utilization-focused evaluation, outcome mapping, outcome harvesting, outcome evidencing, most significant change, participatory impact pathways analysis, and social network mapping and analysis. Also designing and implementing quantitative impact evaluations.   15 Annex to Workshop Report Workshop Contributors and Presenters Valentine Joseph Gandhi is an Implementation Scientist, Development Economist, Policy Advisor, Poverty and Livelihood Impact Evaluation Specialist, ICT4D and Knowledge Manager, currently between Brighton, Jakarta and Nairobi, with over 20 years of interdisciplinary, quantitative and qualitative research, teaching/training and project management, M&E, Strategic consulting, knowledge sharing and capacity building and development. Rachel Davel MDev is a research assistant with the Sustainability Research Effectiveness Program at Royal Roads University in Victoria, Canada. With over five years of experience in theory-based research evaluation methods and transdisciplinary research quality standards, her current work focuses on outcome and impact assessment to understand how research projects contribute to societal change as well as how research can be made more effective via intentional design, implementation, and adaptive management. Rachel Claus works as a research assistant with the Sustainability Research Effectiveness Program at Royal Roads University. Her research expertise and interests are in research design for impact and theory-based research evaluation methods. She has six years of experience developing and applying theories of change to projects and programs to support monitoring and evaluation strategies to optimize effectiveness. Rachel is currently enrolled in the Doctor of Social Sciences program at Royal Roads University, where she hopes to examine the application and interpretation of research quality criteria in grant peer review processes. Stefania Sellitti is a development economist with a strong background in agriculture and rural development. She is currently working as evaluation analyst consultant at the CGIAR Evaluation Function. She worked on several research projects with CIAT and CropTrust, focusing on the empowerment of workers in coffee estates in Latina America, on the knowledge about climate change in Nicaragua and Colombia, and the impact of CIATS's Genebank and bean collection. She previously worked as teaching assistant at the NOVA School of Business and Economics in Lisbon and as Research Assistant at Nocafrica. She has experience in Monitoring, Evaluation, and Impact Assessment, both within CGIAR, as an intern at the DG Agri of the European Commission, and as an external consultant for private companies, such as a Plan-Eval in Brazil and COATL in Portugal. Pierre Boulanger currently serves as ISDC Secretariat advisor. Prior joining CGIAR, Pierre was scientific officer at the Economics of Agriculture Unit of the European Commission’ Joint Research Centre. His areas of expertise include agricultural and trade policies, economics of food and nutrition security, investment programs and sustainable development. Pierre held research and teaching positions at Science Po Paris and the University of Frankfurt am Main, and worked as a consultant for the World Bank. Pierre completed undergraduate degrees in economics at Paris Dauphine University and Autonomous University of Madrid, and his MSc and PhD in international economics at Sciences Po Paris. Magali Garcia is a Bolivian Agronomist Engineer, with a strong interest in the field of bioclimatology, climate change, and ecosystem management. She holds a Master of Science from the University of Wageningen in The Netherlands and a Master and Ph.D. in Climate and Water Sciences from the Katholieke Universiteit Leuven of Belgium. She coordinates projects on watershed strategies to reduce land degradation in the Andes, training programs of Land and Water resources responding to climate pressures, and several climate change impacts projects in Bolivia and elsewhere in the Andes focused 16 Annex to Workshop Report Workshop Contributors and Presenters on irrigation strategy, vulnerability, and adaptation of small Andean communities and ecosystems and climate services linked to local knowledge. Keith Child (PhD) is an independent consultant and evaluation specialist working in the areas of food security, climate change and agricultural development. Keith has held several senior positions, including Senior Scientist, Impact and Learning at the International Livestock Research Institute, Senior Evaluation Specialist with the Research Program on Water, Land and Ecosystems, and Director of Evaluation, Research and Learning at the Aga Khan Foundation. As an evaluative consultant, Keith has most recently worked with the Adaptation Fund, World Bank, the UNFAO and CGIAR Evaluation Function, among others. Graham Thiele is presently an independent consultant. He was Director of the CGIAR Research Program on Roots, Tubers and Bananas (RTB) from 2012 to 2021 which was widely seen as a model for collaborative international agricultural research for development programs. As RTB director he provided intellectual leadership to develop a shared vision, build strong partnerships, and support learning around what worked through monitoring and evaluation and manage for outcomes. He played a key role in validating, testing and documenting the Scaling Readiness Approach. He holds an MSc in Agricultural Economics and a PhD in Anthropology from London and Cambridge Universities, respectively. He began his career at DFID in 1984 working on market studies, farming systems, and research-extension liaison. He joined the International Potato Center (CIP) in 1994, developing, and disseminating approaches for promoting innovation in value chains through public private partnerships and contributing to participatory varietal selection and seed system development. From 2005 through 2011 Graham was the leader of CIP’s Social Sciences Division working on targeting, priority setting, and adoption studies of new agricultural technology and participatory methods. He has mentored, backstopped and led projects in Bolivia, Peru, Ecuador, Benin, Kenya, Nigeria, Rwanda, Tanzania, the Philippines, and Indonesia. He has authored over 90 peer-reviewed articles, two books, numerous book chapters and magazine pieces. Recently he was the lead editor on a volume on innovation in root, tuber and banana food systems synthesizing RTB’s achievements.  Ibtissem Jouini is a manager of evaluations at FAO in the Regional Office for the Near East and North Africa. She has more than ten years of experience working in the International Development field which includes having developed skills in M&E. Before FAO, that she led and contributed to several external evaluations, including for the evaluation function under IAES of CGIAR. Noor Alifa Ardianingrum joined FAO as their Communication officer in 2017. She managed daily digital publishing and knowledge management activities. The team is responsible for digital-first publishing strategy, and template development for digital reports and recently started the initiatives to improve uptake and analysis of impact. Prior to UNFAO, she worked with several non-governmental organizations in natural resources management and in the disaster risk management area. She was the Social Specialist at Earthworm Foundation (previously known as The Forest Trust) and the Training Coordinator at Arbeiter-Samariter-Bund Deutschland e.V in Indonesia. She graduated from The University of Melbourne with a Master of Environment science. When she is not keeping busy with he work you can find her exploring some natural wonders of the world. 17 Annex to Workshop Report Workshop Contributors and Presenters Gaia Kouki joined the Food and Agriculture Organization of the United Nations (FAO) in 2019 after finishing her bachelor's degree in international relations and marketing from John Cabot University. Since then, she has worked in the FAO publications and library branch and have operated closely with the several teams of the unit, covering tasks raging from marketing to digital publishing and data collection activities meant at measuring the performance of FAO publications. Lately, she has been coordinating actions around the production of digital reports and has been managing the communication between our internal team and the external stakeholders. She has also joined a small team/task force that is currently dedicated on defining, measuring and improving the impact of FAO's knowledge products. Finally, she recently graduated from the SOAS University of London after completing an online Master in Global Diplomacy with a focus on the MENA (Middle East and North Africa) region. Amy Jersild has 25 years of experience in the international development sector as a senior evaluator, manager, technical advisor, and educator working in partnership with donors, governments and civil society organizations in Asia, Africa and the Middle East on development effectiveness and quality programming and policies. She has senior level of expertise in management, capacity development, and institutional strengthening with a strong thematic focus on labor and mobility. She has high level of technical knowledge and experience in evaluation and research strategies and methods. Amy holds MA degree in sustainable development, and PhD candidate in interdisciplinary evaluation studies at Western Michigan University, USA. Maggie Gill is a Professor of Integrated Land Use at Aberdeen University. She has worked with the United Nations Food and Agriculture Organisation as chair of the Independent Science and Partnership Council of the CGIAR as well as Chair of an EU Think-tank for Food 2030 and Chair of a Science Advisory Panel for the “Our Land and Water” National Science Challenge in New Zealand. Maggie has had a varied career in agricultural sciences starting with research in ruminant nutrition before broadening her interests to the impact of agriculture on the environment and subsequently managing and advising on interdisciplinary research. Brian Belcher is the Ashoka Chair in Research Effectiveness and Professor in the College of Interdisciplinary Studies at Royal Roads University. He leads a research program that is developing theory, methodology and methods for evaluating research in complex transdisciplinary contexts. This work helps to demonstrate the societal value and impact of research and learns lessons to improve future research Christo Fabricius is an independent advisor and Professor at Nelson Mandela University where he leads the Capacity for African Resource Management (CARMa- Afrika) programme. A former Lead Author (Scenarios) of the IPBES Sustainable Use assessment, he specializes in Developmental Monitoring and Evaluation, community- based resource management, participatory action research and resilience assessment of social- ecological systems. He has published more than 60 articles related to communities, landscapes and livelihoods and was an author of CGIAR Research Program 2020 Reviews: Water, Land and Ecosystem (WLE). In 2021 he led a CGIAR synthesis of resilience principles for agro-ecological systems management. 18 Annex to Workshop Report Workshop Contributors and Presenters Bia Carneiro co-leads the Digital Innovation thematic area of the CGIAR FOCUS Climate Security, where she develops digital methods research to map narratives and actor dynamics in online spaces. She is a social research specialist with over 20 years’ experience in communications and international development, focusing  on managing information systems, critically assessing data, and developing innovative research strategies. She holds a Joint Major in Communication and Latin American Studies from Simon Fraser University, Canada, with a specialisation in Social Policy Management from Pontifícia Universidade Católica do Paraná, Brazil and a MSc Social Policy and Development from London School of Economics, UK. She is currently a PhD Sociology candidate at the University of Coimbra, Portugal. Marina Apgar is Research Fellow in the Participation, Inclusion and Social Change cluster at the Institute of Development Studies, ranked first in the world for Development Studies for a sixth year running. She has over 15 years’ experience working in research and evaluation with marginalised communities in international development supporting learning and change in complex systems. She specialises in combining contribution analysis and participatory evaluation and accompanies development and humanitarian organisations as they build, test and improve their evaluation systems to maximise learning for adaptive programming. Prior to her role at IDS she was Knowledge Sharing and Learning Scientist at WorldFish, leading work in the CGIAR on innovative participatory approaches for complexity-aware and learning- focused programming and embedding appropriate evaluation methodologies. As a core team member of the Centre for Development Impact she has delivered evaluation design and methodology training to over 150 evaluation professionals over the last five years. She is currently the evaluation lead for the Mali transforming peacebuilding programme funded by Humanity United using contribution analysis. Past learning and evaluation contracts include working with/for USAID, Swiss Development Cooperation, Caritas Austria, Oxfam GB, the Irish Agricultural Development Fund and the UK Foreign and Commonwealth Development Office. She recently edited the evaluation section of the SAGE Handbook of Participatory Research and Inquiry and is the editor of a special issue in the European Journal of Development Research on Evaluating Research for Development. Raul Guerrero is an Evaluator with over 20 years experiences at international level. He has designed, led, and managed more than 50 complex evaluations across different sectors and fields around the world (in-field experience in 50+ countries. He has previously worked with European Investment Bank, European Commission and European External Action Service. Ben Bennet is a marketing and value chain economist with 35 years experience of managing and delivering research. He is currently Interim Director of the Natural Resources Institute (NRI), UK. His work has been largely in the field of trade and market economics for tropical commodities in developing countries with a particular focus on aspects of agricultural value chains for novel products. He has undertaken consultancy and research in over 40 countries working on cereals, legumes, horticulture, fisheries, fibres, non-timber forestry products, livestock, oilseeds, and industrial crops. 19 Annex to Workshop Report Workshop Contributors and Presenters Kouété Paul Jimmy graduated in Agricultural Sciences (Agr.Ing Dipl) with major in Agricultural Economics and Rural Sociology and holds an MSc in Sociology of Natural Sciences from the Univeristy of Parakou, Benin. He worked as a research assistant with the Faculty of Agronomy at this university, and was involved in many projects, including analysis of participatory video use in agricultural extension; and women’s knowledge systems on local dairy processing in agropastoral settings. Based on these experiences, he co-authored and published five peer-reviewed papers. Anthony Osei Asare is a Researcher, Monitoring and Evaluation expert and a Development Planner and holds a Master’s degree in Development Studies and a BSc in Development Planning. He has been in the field of research and Monitoring and Evaluation for over 11 years and has vast experience in research and consultancy, conducting large scale households and institutional surveys in the areas of food security and agriculture, women empowerment and livelihood, education, water and sanitation, housing, and local governance. Anthony has served as a research manager or a monitoring and evaluation expert for number of projects funded by both international and local agencies like the International Food Policy Research Institute (IFPRI); Canadian Feed the Children (CFTC); USAID, World Bank, World Vision, among others. Michael Apamaku is a consultant in climate smart livestock as a basis to promote low carbon and climate resilient livestock in Africa. Michael has over 20 years of experience in livestock development with the Government of Uganda, Veterinarians Without Borders and the UN FAO. Michael is the Founder and Co-Director of Livestock Futures which offers consulting in the design, implementation and evaluation of climate smart agriculture projects in East Africa. Michael serves as the Business Development Director with AgCoTech Global (in the mission to reduce poverty and fight climate change in livestock keepers in Africa) and Director for Strategic Partnerships with The Conservatory Group (in programs for regenerative agriculture). As a Senior Expert with GOPA Worldwide Consultants GmBH, Michael evaluated the BMZ-funded Program for Climate Smart Livestock, implemented by World Bank, ILRI and GIZ. With GOPA, Michael is currently supporting research on Quality of Science of the CGIAR portfolio. Michael is a veterinarian with a Master in epidemiology (University of California), a PhD in virology (University of California) and MS-training in agricultural economics (Kansas State University). Michael is based in Nairobi, Kenya. Elli Travis is a PhD Candidate at Virginia Tech in the Department of Agricultural, Leadership, and Community Education. Her dissertation research is focused on conceptions of causal validity and epistemic justice in international development impact evaluation. She has 15 years of experience in community and economic development and evaluation and currently serves as a Senior Economic Development Specialist at the Virginia Tech Center for Economic and Community Engagement. Elli holds an MS from Virginia Tech in Agricultural and Applied Economics and a BA in Economics and Russian from Indiana University. She serves her community as president of Downtown Christiansburg, Inc., an aspiring Main Street community development organization. Dr. Ingrid Fromm is a research associate and lecturer in international agriculture at the Bern University of Applied Sciences, School of Agricultural, Forest and Food Sciences, in Zollikofen Switzerland where she teaches in the Bachelor and Master’s programs. She holds a PhD in Development Economics from Leipzig University in Germany and a BSc. in Agriculture from Zamorano University in Honduras. Her work focuses on food systems and global value chains and their impact on economic, environmental, and social development. For the past 18 years, Dr. Fromm has focused on value chain research in multi-country trade commodities such as cocoa, coffee, banana, and palm oil as well 20 Annex to Workshop Report Workshop Contributors and Presenters as fresh vegetables and pulses. She has conducted research and consultancies in numerous developing countries in Latin America, West Africa, and Asia. Samriti Maharjan Samriti has 4 years of experience working on Gender Equality, Research and Evaluation. She worked as Junior Consultant to support the “Real-Time Evaluation of Gender Integration in the UNICEF COVID-19 Response in South Asia”. She has supported conducting quantitative and qualitative research and drafting evaluation reports. Further, she has gained knowledge on Gender Equality, Women Empowerment and Social Inclusion while doing Internships in UNICEF South Asia and UN Women Nepal. Further, she has Master’s degree in International Cooperation and Development from Nepal. She has also won Cooperating and Development Network Grant 2021/2022 to conduct her Master’s thesis in “Socio-Economic Impacts of COVID- 19 on Healthcare Workers in Kathmandu Based Hospital”. Lenyora Sihwa has 8 years of experience in Monitoring, Evaluation, Accountability and Learning Specialist and Statistical data analyst within humanitarian and development work in Africa and Europe. Her demonstrated history of working is in the education, development economics, and governance sectors. Her skills are in gender and disability inclusion, human rights-based approach to development and MEAL capacity assessments, capacity building and organisational learning. Strong quantitative and qualitative research skills. She has designed complexity-aware MEAL Systems and impact evaluations, designing MEAL frameworks for various sector and experienced in Monitoring and Evaluating projects funded by DfID/FCDO, BMZ, and UN. Huber Paulmer is a management consultant focused on international trade and development with more than 25 years of experience in planning and managing complex evaluations and multi-sectoral/multi-country projects in the private, public, and non-profit sectors in more than 60 countries (in-country) across Africa, Asia- Pacific, Europe, and the Americas. He has completed diverse assignments with UN Agencies (FAO, UNDP, UNICEF, UNFPA, UN Women, UNECA, ITC, and ICAO) and international financial institutions (AfDB, ADB, IFAD, CDB, and the World Bank), as well as several bilateral agencies, intergovernmental organizations, global foundations, and international NGOs. In addition to leading evaluations (corporate, thematic, multi- country and programme), Mr. Paulmer has extensive experience in organizational assessments, strategic planning, risk assessment and management, market/sector studies and value-for-money assessments. Mr. Paulmer is recognized for his expertise and glocal experience in climate-smart agriculture and value chains, food systems, MSME and social enterprise development/management, trade development, green growth strategies and women’s economic empowerment. He has been a Credentialed Evaluator (CE) of the Canadian Evaluation Society (CES) since 2012 and a CES Credentialing Board Member since 2019. He holds a Master’s degree in Agriculture, an MBA and a Post-graduate Diploma in International Planning and Development. Richard Caldwell is Senior Program Officer MLE at Bill and Melinda Gates Foundation. He has over twenty-six years of professional international research and development experience with expertise in project monitoring and evaluation, quantitative data analysis, climate change, ecology, and agronomy. 21 Annex to Workshop Report Workshop Contributors and Presenters Melanie Mueller is an experienced creative facilitator who is highly skilled in teaching, developing education curriculum for university classrooms to corporate training centres. She has worked in management and the education industry for over 25 years and much of her experience has been gained in an international environment. A lateral thinker that makes international projects happen within a projected timeframe both onsite and remotely, Melanie has won awards for her dedication to inspiring others. Her professional experience encompasses working at WFP in Rome writing HR training programs a well as an International Academic Supervisor of Rome Business School and Griffith University in Australia. Currently, as a consultant, she is writing training materials for a global audience that will train 30,000 workers. Melanie loves helping people grow, especially in an international environment. Annex 2.2: Analysis of participants Gender of participants (72 in total) F 47% M 53% Participants affiliation (72 in total) CGIAR Non-CGIAR 26% 74% Participants profession type (72 in total) 22 Annex to Workshop Report SME MEL Partner/MEL SME/MEL Researcher Other 3% 4% 8% 22% 9% 54% Annex 2.3: Attendance to the workshop Participation by day (72 in total) 70 62 63 60 60 50 50 40 40 30 30 53 20 20 10 10 9 10 0 0 Total Day 1 Total Day 2 Only Day 1 Only Day 2 Both Annex 3: Feedback on the Event Annex 3.1: Response rate Survey response rate – 38 responses, of which 3 incomplete that were not considered. 23 Annex to Workshop Report No 49% Yes 51% Annex 3.2: Respondents’ profile Main place of work (35 respondents) CGIAR 6% Multilateral Organization (CGIAR, UN 17% Women, FAO, AfDB, etc) 14% University/Academia 3% 11% Non-for-profit sector NGOs, CSOs, Foundations Govermental and public institutions Privator sector (i.e. farm, small-medium 49% enterprises, consulting) Other 24 Annex to Workshop Report Annex 3.3: Participation in the workshop How did you participate in the workshop? (35 respondents) Online In-person 46% 54% Which sessions of the workshop did you attend? (35 respondents, multiple options allowed) 25 Annex to Workshop Report Annex 3.4: Feedback on the event How satisfied were you/were you not with the event? (35 respondents) All respondents (35) Very satisfied Satisfied Not satisfied Online (16) In-person (19) 6% 13% 48% 37% 42% 46% 58% 50% Why were you/were you not satisfied with the event? (35 respondents) Why were you/were you not satisfied with the event? • Rich and interesting discussions • Presentation of practical applications • Valuable learning opportunity • Gained a better understanding of the guidelines Very satisfied • Shared understanding of the way forward (17 respondents) • Possibility to listen to a wide range of perspectives • Good spread and diversity of participants • Good networking opportunity • Excellent facilitation • Excellent social events • Good quality of training • Good quality of presentations and discussions • Comprehensive and laid the foundation for future development Satisfied • Too short time, required at least 2.5 days (16 respondents) • Facilitation could have been clearer, it was a bit unclear for online participants • Required better facilitation for the online breakout groups • Difficulty in hearing Not satisfied • The workshop lost focus on the guidelines and their application. (2 respondents) • Difference between QoS and QoR4D was not clarified • Breakout rooms were not well designed/facilitated and framing of questions was poor • Some of the speakers were not well prepared • Poor time management 26 Annex to Workshop Report Which sessions did you enjoy the most? (35 respondents, multiple options allowed) Which sessions did you enjoy the least? (35 respondents, multiple options allowed) Would you want to attend this kind of event in the future? 27 Annex to Workshop Report Yes No 3% 97% What were your biggest takeaways of the event? What were your biggest takeaways of the event? About the guidelines • Guidelines use, evaluations and time implications • Need for a harmonized approach and clear guidelines to allow for comparisons across different evaluations • Assessing the quality of R4D requires establishing criteria and tools and above all a consensus among the scientific and development public on the challenges and best approaches. The CGIAR R4D framework is a good basis for continuing that discussion • Need to contextualize the guidelines • Flexibility of the guidelines in the choice of criteria and dimension to use in each specific case Very satisfied • Evaluation of quality of science needs to go beyond bibliometrics (17 respondents) • Research for development needs to be relevant and useful for the users • The convincing that science is every time more aware of the need to be applicable, and that metrics need to change • Need for a timely delivery of the evaluation to ensure that findings are used Cooperation among stakeholders • Need for cooperation and enriching dialogues between scientists and evaluators Need for dynamic evaluation involving multi-stakeholders recognising the pluralism of knowledge About the guidelines • A good update on how guidelines were creatively updated • Importance of the evaluation of research and science • Relevance of having an applicable QoS framework for R4D • Need for flexible and well thought application of the guidelines in Satisfied combination with other resources (16 respondents) • Better understanding of the process of evaluation • How to translate knowledge into action Cooperation among stakeholders • Learning and co-design are key to uptake and impact 28 Annex to Workshop Report What were your biggest takeaways of the event? • Need for interdisciplinary research for better cooperation between scientists and evaluators • Need for increased resourcing for evaluation • The integration of evaluation and science is quite complex, and different institutions have different ways of viewing the connection, so there’s need of cooperation Not satisfied Presence of some great speakers (2 respondents) What would have made this event better? What would have made this event better? Content • Going through the guidelines together section by section • More stories outlining co-creation success • Less presentations and more time for exchange of ideas • Deeper sessions for more critical discussions Participants • Invite more diverse participants from different sectors to bring more ideas and different perspectives Very satisfied • Invite more CG practitioners, including SPIA (17 respondents) Logistics • More consideration for the time zones • Consider more time for the event, for group works and cases • Earlier understanding of the agenda Online participants • Having all participants online, rather than hybrid Content • Going through the guidelines together section by section • More stories outlining co-creation success • Less presentations and more time for exchange of ideas • Deeper sessions for more critical discussions Logistics Satisfied • Timeframe extension (16 respondents) • More programmed discussion and time • More facilitated discussions in breakout sessions Online participants • Better sound • A dedicated online facilitator • Organization of hybrid groups between virtual and in-person participants Not satisfied Content 29 Annex to Workshop Report What would have made this event better? (2 respondents) • Better design of the questions Logistics Better facilitation Annex 4: Cases of study Annex 4.1: Case Study 1 Type of evaluation CGIAR Background Genebanks are a type of biorepository that preserves genetic material. For plants, this is done by in vitro storage, freezing cuttings from the plants or stocking the seeds. Most genebanks contain living things, or parts of them, which themselves contain the collection of the different genes that make each plant variety or animal breed unique. Genebanks ensure that these genetic materials are safely conserved and available for people to use. Taken together, CGIAR genebanks represent the largest and most widely used collections of crop diversity in the world. The CGIAR Genebank Platform is a comprehensive 5-year program for the management, as well as the secure and sustainable funding, of the collections of plant genetic resources held by the 11 CGIAR genebanks. It is a partnership between the members of the CGIAR and the Global Crop Diversity Trust. The Platform is a service that supports the core activities of the CGIAR genebanks: conserving and making available crop and tree diversity. It helps them to meet international standards, improve efficiency and ensure more effective use of collections within a supportive policy environment. The three core modules of the Genebank platform are: 1. Conservation, that helps genebanks work strategically to exploit new opportunities, conserve more diversity and respond to more demands while controlling costs; 2. Use, that helps genebank align their operations towards more targeted use and exploitation of the collections; 3. Policy, that supports the active engagement of CGIAR in shaping international genetic resources agreement. Evaluation purpose: IAES’ Evaluation Function is commissioning a final performance evaluation in 2023 to assess the effectiveness of the Platform. 30 Annex to Workshop Report Type of evaluation CGIAR The Task You are an evaluation team and are asked to evaluate the Genebank Platform. With the objective of harmonizing the overall approach for evaluating research and science quality in CGIAR and similar context, IAES developed Evaluation Guidelines on “Applying the QoR4D Framework to Process and Performance Evaluation”. The task of your team is to design an evaluation, that complies with an Evaluation Framework and Policy. With this objective in mind, you start going through the Guidelines in detail, to define which criteria apply and if/how to use the Quality of Science criterion. To start, you will need to answer a set of questions, making the best use of Guidelines. Be as specific and concrete as possible, but be also critical and creative! You do not have any limitations on how to solve this case, there might be one or more possible solutions to make a good use of the Guidelines. Questions for consideration During the exercise (45 minutes) 1. How would you apply the Guidelines? 2. What approach would you take? 3. Which evaluation criteria would you use? a. The QoS criterion? Please, explain why or why not. b. IF not, what aspects of QoS criterion would be integrate into other evaluation criteria? 4. Which QoS evaluation dimensions would you use? 5. Which evaluation questions did you include? Which are specific to QoS? 6. Which methods would you use in this case (textbox 3, figure 8 and tables 5, 6 and 8) or else? Why? a. Primary/secondary? b. Are there methods that are different from the ones you would have usually used (before reading the Guidelines)? c. What about data analysis? 7. What limitations to the approach and methods do you foresee, and how you would mitigate? After the exercise (15 minutes) 8. Was there a benefit of using the Guidelines? Could you have done without? 9. What aspects did you find helpful? What difficulties have you found in applying them? Annex 4.2: Case Study 2 Type of evaluation CGIAR Background Genebanks are a type of biorepository that preserves genetic material. For plants, this is done by in vitro storage, freezing cuttings from the plants or stocking the seeds. Most genebanks contain living things, or parts of them, which themselves contain the collection of the different genes that make each plant variety or animal breed unique. 31 Annex to Workshop Report Type of evaluation CGIAR Genebanks ensure that these genetic materials are safely conserved and available for people to use. Taken together, CGIAR genebanks represent the largest and most widely used collections of crop diversity in the world. The CGIAR Genebank Platform is a comprehensive 5-year program for the management, as well as the secure and sustainable funding, of the collections of plant genetic resources held by the 11 CGIAR genebanks. It is a partnership between the members of the CGIAR and the Global Crop Diversity Trust. The Platform is a service that supports the core activities of the CGIAR genebanks: conserving and making available crop and tree diversity. It helps them to meet international standards, improve efficiency and ensure more effective use of collections within a supportive policy environment. The three core modules of the Genebank platform are: 4. Conservation, that helps genebanks work strategically to exploit new opportunities, conserve more diversity and respond to more demands while controlling costs; 5. Use, that helps genebank align their operations towards more targeted use and exploitation of the collections; 6. Policy, that supports the active engagement of CGIAR in shaping international genetic resources agreement. Evaluation purpose: IAES’ Evaluation Function is commissioning a final performance evaluation in 2023 to assess the effectiveness of the Platform. The Task You are an evaluation team and are asked to evaluate the Genebank Platform. With the objective of harmonizing the overall approach for evaluating research and science quality in CGIAR and similar context, IAES developed Evaluation Guidelines on “Applying the QoR4D Framework to Process and Performance Evaluation”. The task of your team is to design an evaluation, that complies with an Evaluation Framework and Policy. With this objective in mind, you start going through the Guidelines in detail, to define which criteria apply and if/how to use the Quality of Science criterion. To start, you will need to answer a set of questions, making the best use of Guidelines. Be as specific and concrete as possible, but be also critical and creative! You do not have any limitations on how to solve this case, there might be one or more possible solutions to make a good use of the Guidelines. Questions for consideration During the exercise (45 minutes) 10. How would you apply the Guidelines? 11. What approach would you take? 12. Which evaluation criteria would you use? a. The QoS criterion? Please, explain why or why not. b. IF not, what aspects of QoS criterion would be integrate into other evaluation criteria? 13. Which QoS evaluation dimensions would you use? 14. Which evaluation questions did you include? Which are specific to QoS? 15. Which methods would you use in this case (textbox 3, figure 8 and tables 5, 6 and 8) or else? Why? a. Primary/secondary? 32 Annex to Workshop Report Type of evaluation CGIAR b. Are there methods that are different from the ones you would have usually used (before reading the Guidelines)? c. What about data analysis? 16. What limitations to the approach and methods do you foresee, and how you would mitigate? After the exercise (15 minutes) 17. Was there a benefit of using the Guidelines? Could you have done without? 18. What aspects did you find helpful? What difficulties have you found in applying them? Annex 4.3: Case Study 3 Type of evaluation Background The Avaclim project is a 1.1 million project started in 2019 that aims to create the necessary conditions for the deployment of agroecology in arid areas. To achieve this, CARI, the NGO that is carrying out the project, and its partners gave themselves three years, from 2020 to 2022. CARI is the main executing partner for the project based (i) on a financing agreement signed between FFEM and CARI and (ii) on Letters of Agreement signed between FAO and CARI. Practitioners, farmers and scientists are studying agroecological initiatives in seven countries: Burkina Faso, Senegal, Morocco, Ethiopia, South Africa, Brazil and India to promote agro-ecology to the political authorities of these countries and to intergovernmental bodies. Agroecology enables the improvement of agricultural production through the enhancement of local natural resources and traditional know-how. It contributes to maintaining biodiversity and restoring land in drylands in these generally poor regions, where physical constraints are significant. They are particularly threatened by global warming and food insecurity. Avaclim's objective is to generate the knowledge needed to engage policy makers and stakeholders in the seven project countries to prioritize and implement agroecological systems in drylands in order to support the productivity of agroecosystems for food security, agricultural livelihoods and reduce environmental degradation and GHG (greenhouse gas) emissions. The project is building a scientifically-based argument in collaboration with agroecological practitioners. This argument is built in two key steps: the sharing of acquired knowledge between the practitioners themselves and the multidimensional evaluation of the impact of these initiatives using a scientific method. Its results are then taken to the political level, at the country level but also to donors and international institutions. Under the AVACLIM project, specific interventions are implemented that address the identified barriers. These interventions are divided into four components: 1. Building partnerships for experience sharing and capacity building of agroecology practitioners at the landscape and local levels with international connections among the project participants 2. Assessment of existing initiatives for evidence-based decision-making at the national, local and landscape levels 3. Advocacy for informed decision-making 4. Communication, learning, knowledge management and adaptive management The implementation of the project interventions under these four components aim to generate multiple global environment benefits including: i) supporting the transition towards improved agricultural practices enabling the sustainable management of land, soil, water and forest resources, and promoting the restoration of degraded 33 Annex to Workshop Report Type of evaluation land; ii) increasing carbon storage; and iii) supporting biodiversity. The AVACLIM project interventions provide reliable information on and advocate for the integration of the agroecology approach into development planning across drylands. This will enable the mainstreaming of agroecological innovations across drylands to increase food security, diversify agricultural livelihoods, reduce environmental degradation and increase soil carbon sequestration. This project will contribute to overcome the main barriers to the wide and systematic implementation of agroecology, particularly across the drylands, namely i) limited interactions between agroecology practitioners which hinders knowledge sharing; ii) limited availability of validated knowledge on the impacts and success factors of agroecology; iii) limited prioritisation of agroecology by policy-makers; and iv) insufficient awareness of the general public on agroecological innovations and their benefits. AVACLIM was primarily implemented by 10 partners that work in different contexts. The scientific community both from the southern and the northern hemisphere had the opportunity to test their hypothesis on the ground to address knowledge gaps on agroecological innovations. They also had the opportunity to compare scientific results with experience-based results. This will help bridge the gap between scientific findings and methods versus practical experimentations and observed results towards the scientific validation of on-the-ground observations. A mid-term review of AVACLIM was carried out between October and November 2021. The review assessed the status of the project in terms of its achievements and challenges, while developing recommendations for corrective actions to ensure that the project will be on track in achieving its desired results. Evaluation Purpose: FAO/GEF commissioned a final evaluation to identify important lessons to guide future actions and to serve as an input to improve formulation and implementation of projects that may follow similar approaches. It will assess the project’s results, their value relevant to target beneficiaries, national needs and priorities. This evaluation will also have the purpose of pointing at future actions to sustain project results and disseminate products and best- practices. This terminal evaluation is a requirement of the Global Environment Facility (GEF), the Food and Agriculture Organization of the United Nations (FAO) and the Fonds Français pour l’Environnement Mondial (FFEM) for project monitoring and reporting purposes. It is being conducted for both accountability and learning purposes. The evaluation also aspires to provide accountability with respect to communities –and in particular vulnerable groups in Drylands– that the project has sought to assist indirectly. The scope of this evaluation will cover results achieved by the project throughout its implementation period from January 2020 to November 2022. Although focused on results, the evaluation will also seek to assess a set of performance-related issues deriving from the donor’s evaluation criteria. The Task You are an evaluation team and are asked to evaluate the AVACLIM project. With the objective of harmonizing the overall approach for evaluating research and science quality in CGIAR and similar context, IAES developed Evaluation Guidelines on “Applying the QoR4D Framework to Process and Performance Evaluation”. The task of your team is to design an evaluation, that complies with an Evaluation Framework and Policy. With this objective in mind, you start going through the Guidelines in detail, to define which criteria apply and if/how to use the Quality of Science criterion. To start, you will need to answer a set of questions, making the best use of Guidelines. Be as specific and concrete as possible, but be also critical and creative! You do not have any limitations on how to solve this case, there might be one or more possible solutions to make a good use of the Guidelines. Questions for consideration 34 Annex to Workshop Report Type of evaluation During the exercise (45 minutes) 1. How would you apply the Guidelines? 2. What approach would you take? 3. Which evaluation criteria would you use? a. The QoS criterion? Please, explain why or why not. b. IF not, what aspects of QoS criterion would be integrate into other evaluation criteria? 4. Which QoS evaluation dimensions would you use? 5. Which evaluation questions did you include? Which are specific to QoS? 6. Which methods would you use in this case (textbox 3, figure 8 and tables 5, 6 and 8) or else? Why? a. Primary/secondary? b. Are there methods that are different from the ones you would have usually used (before reading the Guidelines)? 35 Independent Advisory and Evaluation Service Alliance of Bioversity International and CIAT Via di San Domenico, 1 00153 Rome, Italy IAES@cgiar.org https://iaes.cgiar.org/