^^CGIAR Stripe Review of Social Sciences in the CGIAR October 2009 P.nnciiltatix/p Hrmin nn International Aoricultural Research SCIENCE COUNCIL ^CGIAR Stripe Review of Social Sciences in the CGIAR Review Panel: Christopher B. Barrett (Chair) Arun Agrawal Oliver T. Coomes Jean-Philippe Platteau OCTOBER 2009 i : The Science Council of the CGIAR encourages fair use of this material provided proper citation is made. Correct citation: CGIAR Science Council (2009) Stripe Review of Social Sciences in the CGIAR. Rome, Italy: Science Council Secretariat ii Science Council Commentary on the Stripe Review of Social Sciences in the CGIAR October 2009 The impetus for the Stripe Review on Social Sciences in the CGIAR arose from repeated concerns expressed in many EPMRs that social science capacity in Centers was deteriorating. Accordingly, the SC commissioned an initial scoping paper in 2008 that presented the justification for conducting an in-depth review. The review Panel's mandate was to provide a System-level assessment of social sciences in the CGIAR including its quality, coherence, relevance, productivity and capacity, and to make recommendations for improvement. The main phase of the Review was implemented from January to August, 2009. The key elements were: collecting comprehensive data on social science activities and results; formulating a normative framework of "optimal" social sciences roles in the CGIAR against which the current state could be assessed; e-consultation with Center social scientists to prioritize issues and define hypotheses; analysis of publications and their citations, with a review of selected publications and projects; interviews with staff and partners at nine Center headquarters or regional offices; and soliciting feedback from Centers and external experts on the draft report. The approach taken provides "good practice" for future studies of a similar nature. With one exception, all Centers and CPs were responsive to nearly all the Panel's requests.1 The Science Council discussed the Panel's report at its 12th meeting held at CIFOR in Bogor, Indonesia. The Chairman of the Panel, Professor Chris Barrett, presented the report. The Science Council conveys it's thanks to the Panel Chair and the three members, Drs Arun Agrawal, Oliver Coomes and Jean-Philippe Platteau, for a frank and insightful report that presents a compelling analysis of the current state of social science in the System. The SC is impressed by the volume of information the Panel assembled to support its analysis including the feedback it solicited from Center staff and social scientists outside the System. The SC emphasizes that this is an important and timely report that deserves full attention from Center management and CGIAR membership to ensure serious consideration of its findings and recommend ations. Key Diagnosis The Panel forms its main diagnosis around what it has termed the "new business model". This model is characterized by: a) a shift from largely unrestricted core funding to relying excessively on restricted funding, most of which comes through very small, short-duration development- oriented grants; b) a lack of vision on what CGIAR social science should be doing that has led to an almost unlimited expansion of social scientists' agenda; and c) the Centers' desire to maintain a large cadre of international staff in relation to the work carried out. This has led to a loss of focus and fragmentation in social science work, an erosion of the quality and effectiveness of social science research, a fall of staff morale, and difficulties recruiting and retaining high quality IITA opted not to participate in this review. iii staff. The strategy of maintaining a large cadre of internationally recruited social science staff represents a high-cost approach for the work being done, much of which could be carried out by well-trained MS degree holders. The Science Council agrees with this core analysis which reveals shortcomings in Centers' ability to plan coherent research agendas based on clear strategies and prioritization and defend these agendas in their negotiations with the donors thereby resisting funding-driven fragmentation. The SC observes that this diagnosis signals broader problems within the System that triggered the CGIAR change management process. The effects of the trends described in the Panel's report, therefore, are likely to go well beyond social sciences. Nonetheless, the problem may be more serious for social science, than for other disciplines, if the grant money is more fragmented and developmental in orientation and if labor markets for social scientist recruitment are more competitive. These problems may also reflect an inability of the discipline to explain and demonstrate its value in conducting more long-term and strategic research —both potential and actual—to Center management and donors, and the demand for social science research, therefore, has not been clearly articulated in the Center's strategic planning. The CGIAR change management process is intended to reverse these trends and improve the coherence and relevance of the CGIAR's research and the sustainability of long-term funding. It is imperative that the CGIAR System, including its Centers and donors, take the Stripe Review Panel's analysis seriously. Special attention to improving social science capacity is needed as part of the change management process. Quality of research The Panel assessed quality primarily by looking at the quality of staff and the quality and impacts of publications. It also reviewed on-going projects that shed light on the scope and role of social sciences, project funding and the planned partnerships. Close analyses of a sample of projects allowed assessment of methodological quality and the match between objectives and approaches. The CGIAR System should be extremely concerned by the findings that social science research is judged to be methodologically weak and declining in quality. In addition, there appears to be a lack of critical mass of high quality social scientists with sufficiently diverse disciplinary skills. The Panel also found that partnerships are increasingly transactional (even competitive in the case of NARS) and the quality of the partnerships with ARIs is weak. Another aspect of declining quality is that data collection, management and exploitation were found to be deficient. The SC strongly endorses the need to systematically improve the management of data and its use internally and externally, including attention to ethical issues in data collection. Nonetheless, the SC emphasizes that the review has also revealed pockets of excellence in the CGIAR. The Panel notes that excellence seems to be associated with: sufficient seniority among the researchers, clearly defined long-term projects with assured funding, long-term partnerships with ARIs, and inclusion of an inter-disciplinary component. Center management can influence many of these factors despite funding pressures. iv In its assessment of quality, the Panel applied criteria used in academia, such as number of publications and citation. The SC accepts that these criteria are also applicable to a highly mission-driven organization. It emphasizes, however, that publications and citations as used in the profession at large is too narrow a perspective. In the CGIAR social science research ambition needs to also include a strong commitment to solving important problems that impact poor people. Comparative advantage and relevance The report lists the attributes that give social scientists in the Centers a comparative advantage in generating international public goods. The most important attribute is the ability to organize research around problems rather than disciplines — the ease of conducting inter-disciplinary research—and work at an international scale and sustained over time. Other elements of comparative advantage include the Centers' physical infrastructure and long-term presence in developing countries, and a cadre of well-trained, internationally recruited staff. Given comparative advantage, the Panel identified three areas where it sees the greatest opportunities for CGIAR social science research to be effective in serving its mission: (i) technological change that sustainably increases agricultural productivity by and for the poor; (ii) natural resource conservation that benefits the rural poor directly or indirectly via environmental services, especially those that help agricultural productivity; and (iii) institutional innovations and policies that enhance the quality of life for poor and marginalized agrarian populations. Of these, the Panel considers the first as the single most important area of CGIAR social science comparative advantage because it most fully capitalizes on the opportunity for close interaction between social and biophysical scientists that is distinctive to the CGIAR. In the Panel's view, a plurality if not a majority of staff should be in the first area. The Panel hypothesizes that, as Centers' missions have expanded, the most competent social scientists have been drawn into the marketing and policy area, with fewer engaged in research on productivity enhancement. The SC agrees with the Panel's judgment that Center management needs to take action to protect the focus and relevance of the Center's research agenda. Drifting with funding trends cannot be fully avoided but it can be sharply reduced. However, in order to improve the overall relevance, measures at the System level among the donor community are also clearly needed. For example, critical long-term system studies using sentinel sites would require commitment from donors to fund such work and commitment from scientists and management to engage in long-term work to accumulate data, comparisons and experiences. Donors and Center management need to understand better what good social science research can do for the CGIAR and for development more broadly. Impact What ultimately determines the relevance of CGIAR's social science is its ability to generate impact. The SC finds in the Panel's discussion of impact three distinct dimensions that are not clearly separated in the report: i) impact on long-term development goals from CGIAR's research; ii) impact of social sciences on influencing research priorities; and iii) impact evaluation of research as a component of the social scientists' research agenda. The SC would have liked to see much clearer analysis of the demand for and value of social science research in the context of enhancing impact of the CGIAR's research. Examples of strategie outputs from social science in different areas that very likely have enhanced the likelihood of development impact would have been helpful. However, the pathway from social science research quality to impacts on CGIAR goals is very indirect and highly uncertain. The Panel observes that social science research can most directly influence the likelihood of impact through improved research priority setting and by influencing policies. The SC also notes that in the past attempts to use a systematic ex ante analysis to determine priorities have often not been very successful. It would be important to trace the influence of ex ante priority setting research on what have eventually turned out to be successful choices in terms of real impact generated. Regarding the third aspect of impacts, the Panel identified an "impact measurement obsession". According to the Panel, the number of ex post impact assessments (EPIAs) has proliferated, and these are methodologically weak and being conducted on small projects (rather than programs) too soon after termination. The Panel suspects that this trend has been fuelled by the donors' desire to report back to their constituencies on effective use of funds, and making all research projects accountable for showing impact, despite the well-known uncertainties of research payoffs. At the same time, the Centers are responding to a requirement of the CGIAR's performance measurement system (PMS) to conduct a certain number of EPIAs annually. The Panel acknowledges and the SC emphasizes that SC/SPIA has strongly pushed for the PMS to require a limited number of high quality EPIAs to be conducted on large research efforts and sufficiently long after research has been completed for increasing the ability of the studies to detect and credibly document sustainable impacts. Confirming the Panel's observations, SC/SPIA has observed that a very large proportion of studies submitted each year to the PMS do not qualify as EPIAs at all but are end-of-project evaluations. Recommendations The Panel provides four broadly-based recommendations, which cover management reforms, re­ organization and re-focus in social sciences, updating social science staff management practices, and fostering a culture of rigorous social science research. The SC has the following comments on the recommendations and suggestions on how they might be implemented at the System and Center level. Recommendation 1: Undertake essential management reforms la) Resolve the mismatch between the business model and staffing patterns lb) Realign management incentives lc) Improve leadership selection ld) Tighten the focus on comparative advantage le) Focus on impact but end the impact measurement obsession lf) Mainstream gender equity as a basic axiom of CGIAR research lg) Require full indirect cost recovery The SC agrees in principle with these recommendations. The CGIAR reform process promises to address many of the problems underlying this recommendation (especially components la and Id) by improving the coherence of the overall CGIAR program and the commitment of core funding. Implementation of this recommendation should, however, not wait for nor solely rely on the change process. There are important steps Centers can take in the meantime such as vi improving their own strategic and medium-term planning to guide the inclusion or exclusion of activities and choice of research programs and the roles of social science within them (as strong Centers are already doing). Components lb, lc and lg are clearly the immediate responsibility of individual Center management with Board oversight but should also be pursued by the Consortium for affecting change more uniformly across Centers and programs. Component le on impacts can be best addressed through appropriate design of Mega-programs and agreement on the outcome targets and steps for generating impacts that will be monitored in a systematic fashion. Component If on mainstreaming gender addresses a concern that has been identified in several reviews and needs to be appropriately addressed at the System level. Recommendation 2: Re-organize and re-focus CGIAR Social Science 2a) Restore longer-term partnerships, especially upstream 2b) Focus training and capacity building on research mentoring 2c) Organize a Regional Systems Analysis Mega-Program 2d) Organize a Mega-Program on Stimulating and Evaluating Innovations 2e) Shrink unproductive social science units The Science Council sees merit in what is being recommended, but suggests that the implementation requires further discussion. Addressing recommendations 2a and 2b will be an important challenge in moving to a programmatic mode in research implementation that will increasingly involve partners outside the System. In its response to the recommendations to establish Mega-programs on particular subjects, the SC interprets this to mean that, rather than Mega-programs per se, a systematic way of organizing research units or platforms for addressing research topics across Centers should be formed. The elaboration in the report related to these components (2c and 2d) should be incorporated into the design of future Mega-programs, providing important roles for social science in these programs. Recommendation 3: Update social science personnel management practices 3a) Introduce a CGIAR Young Scientists Program 3b) Increase entry-level compensation packages 3c) Establish a clear research career track for social scientists 3d) Restore competitive travel and sabbatical programs 3e) Employ modern human resources management practices This cluster of recommendations is intended to address several shortcomings hampering recruitment and retention of good quality social scientists in the CGIAR. The SC appreciates that the Center managers are aware of many of these issues and of the importance to engage well- trained and highly motivated young scientists from sufficiently diverse disciplinary background. Recommendations 3a and 3b would help but other measures will also be needed. The CGIAR would need additional funding for implementing 3a to re-establish a post doctoral program. Even then it would be important to consider ways in which the new program can be designed to meet emerging needs. vii The SC considers it important that the Centers restore their attractiveness as a long-term employment option for high quality scientists by exploring a range of both full and part time (shared) appointments, and maintaining locational flexibility. This depends not only on human resource management but also on the excitement that the System's mission can generate. The implementation of recommendations 3b-3e must be tailored to the highly variable circumstances and conditions at each Center. A more uniform approach can only be led from a central unit that would determine human resources policies and practices across Centers. Even then, location- specific circumstances need to be taken into consideration. Recommendation 4: Foster a culture of rigorous social science research 4a) Restore social science research seminar programs 4b) Individual performance measurement 4c) Stop wasting money on in-house publications other than policy or research briefs 4d) Establish a CGIAR Institutional Review Board In this recommendation, components 4a and 4b propose readily supportable and practical actions to stimulate motivation and quality. The SC believes, however, that while dissemination of research results (4c) needs to be cost- and time-efficient, strategies for publications and other means of dissemination need to consider different target groups and multiple outlets may be necessary for disseminating the same results. The Panel's recommendation on Institutional Review Board echoes the recommendations from the SC's recent studies on ethics that suggested it is imperative for scientists to have their research plans approved by a review board. This is particularly important for research that involves human subjects, and the SC again flags this need for Center Boards and the Consortium Board. Conclusions and next steps The main messages in this report and its recommendations in the context of a much changed research and development environment pose new challenges for research, involving social sciences, as well as research management. The SC fully agrees with the Panel's optimism that there are now excellent opportunities for building strengthened and relevant social science components into the CGIAR. The causes for this optimism include the renewed recognition among donors and governments of the importance of agriculture and agricultural research to development; renewed interest in applied development research in the social sciences; and the CGIAR reform process itself. The SC strongly recommends that the System and the Centers collectively and without delay put in place a process to further discuss and implement the very timely and relevant recommendations of this report. This must involve further discussion with Center managers as well as with social scientists themselves. Some of the recommendations could be implemented at relatively low cost, such as revamping recruitment processes to attract higher quality social scientists. Others will need sustained additional funding. Fortunately, there are indications that some donors would be interested in providing such support if the System is ready to move forward. viii Christopher B. Barrett Cornell University Stephen B. and Janice G. Ashley Professor of Department of Applied Economics and Management Applied Economics and Management International Professor of Agriculture Co-Director, African Food Security & Natural Resources Management Program Cornell Institute for International Food, Agriculture and Development 315 Warren Hall http://ciifad.cornell.edu/ Ithaca, NY 14853-7801 USA Telephone: 607-255-4489 Associate Director for Economic Development Programs Fax: 607-255-9984 Cornell Center for a Sustainable Future E-mail: cbb2@cornell.edu http://sustainablefuture.cornell.edu/ http://aem.cornell.edu/faculty_sites/cbb2/ November 20, 2009 Dr. Rudy Rabbinge Chair, Science Council Consultative Group on International Agricultural Research c/o FAO Viale delle Terme di Caracalla 00153 Rome, Italy Dear Dr. Rabbinge, Attached please find the report of the Social Science Stripe Review (SSSR) panel that the Science Council appointed late last year. The other three panel members - Professori Agrawal, Coomes and Platteau - and I thank you and the rest of the Science Council for granting us this opportunity to learn about the wide-ranging work of social scientists within the Consultative Group on International Agricultural Research (CGIAR) and, we hope, to offer constructive insights on the strengths, weaknesses and potential of the social sciences to serve the broader CGIAR mission. Per the terms of reference, our report offers a comprehensive assessment of CGIAR social science research followed by a set of clear recommendations based on the analyses in the assessment. An undertaking of this scale and scope requires considerable skilled support, The panel's work was aided invaluably by a large number of people scattered throughout the world. We especially thank Dr. Sirkka Immonen of the Science Council Secretariat, who expertly coordinated the panel's activities, data collection and interaction with Centers and Challenge Programs, and provided insightful Cornell University is an equal opportunity affirmative action educator and employer. ix feedback throughout both phases of the SSSR. Ms. Natalia Morazzo did a superb job processing data and establishing and maintain the panel's online Wiki under Dr. Immonen's direction. Ms. Irmi Braun-Castaldi handled travel arrangements and other administrative matters for the panel with her usual skill. Dr. Paswel Marenya of the University of Nairobi gathered, organized and effectively summarized a tremendous amount of information in his Phase 1 background report and then patiently answered questions of clarification and investigated queries about specific data issues in the months since he thought he was finished. Erin Lentz and Kelsey Mitchell provided outstanding background research support at Cornell University. The focal persons from the fourteen participating Centers and the Challenge Programs were extremely helpful in providing necessary data and documentation and in providing feedback and insights in the January 2009 e-consultation and to various queries. Stacy Roberts at IFPRI, Frank Place at ICRAF, Ade Freeman at ILRI, Mark Giordano at IWMI and Cynthia Bantilan at ICRISAT graciously organized excellent local programs and logistics for the panel's site visits to their Centers' headquarters, including coordination with co-resident staff from other Centers and Challenge Programs. Forty-two different CGIAR staff took time from their busy schedules to participate in the January 2009 e-consultation the panel held, offering valuable contributions that materially shaped the panel's work. Nearly three hundred people took time to meet with us, speak with us by telephone or Skype, respond to our email queries or take the online survey the Secretariat coordinated. The panel conveys its sincere thanks to each of these people. This enormous effort would have been impossible without their cooperation. The report we submit has benefitted considerably from helpful comments on the first draft provided by several Centers. (Africa Rice, Bioversity, CIFOR, CIP, ICARDA, ICRAF, IFPRI, ILRI, ERRI) and Challenge Programs (Generation, Water and Food), as well as individual feedback from Jacqui Ashby, Randy Barker, Derek Byerlee, Gelia Castillo, Alain de Janvry, Ken Fischer, Bob Herdt, Doug Horton, Sirkka Immonen, Tim Kelley, Urna Lele, John Pender, Per Pinstrup-Andersen, Jim Ryan and Rob Tripp. The panel appreciates the time taken by these experienced observers of the CGIAR in sharing with us their insights. The attached report conveys the panel's unanimous assessment that CGIAR social science has enjoyed many significant accomplishments and remains central to the Cornell University is an equal opportunity affirmative action educator and employer. current and prospective successes of the System. CGIAR social scientists are making fundamental contributions to the social sciences while simultaneously effectively undertaking research, outreach and capacity building activities essential for the System to achieve broad-based impact in reducing poverty and food insecurity, protecting the natural environment and advancing broader development and equity goals worldwide. The System and its social scientists can rightly take pride in many accomplishments that add to the stock of useful, actionable knowledge that helps advance the CGIAR mission. CGIAR social science also exhibits some serious weaknesses and faces several major challenges. Some of those challenges appear common to all the CGIAR; others are specific to the social sciences. The panel believes these weaknesses are remediable and the challenges surmountable with appropriate action by Center and Challenge Program management as well as by System-level managers and stakeholders. In the penultimate section of our report, the panel offers 4 major recommendations, which nest within them 21 more specific sub-recommendations, as to how CGIAR social science might be put on more solid footing moving forward so as to increase its impact in support of the System's broader goals. We welcome dialogue with you, your Science Council colleagues and other stakeholders about these recommendations and the evidence that leads us to these conclusions and counsel. On behalf of Professors Agrawal, Coomes and Platteau, I extend our sincere thanks for the opportunity to undertake this review and our equally sincere hope that our assessment and recommendations will prove helpful to System management and to social scientists throughout the CGIAR. The world needs outstanding CGIAR social science. Sincerely yours, ■' Christopher B. Barrett Chair, Social Science Stripe Review Panel Cornell University is an equal opportunity affirmative action educator and employer. xi CONSULTATIVE GROUP ON INTERNATIONAL AGRICULTURAL RESEARCH CGIAR SCIENCE COUNCIL Stripe Review of Social Sciences in the CGIAR Review Panel: Christopher B. Barrett (Chair) Arun Agrawal Oliver T. Coomes Jean-Philippe Platteau SCIENCE COUNCIL SECRETARIAT OCTOBER 2009 CONTENTS 1. INTRODUCTION 7 2. THE EXTERNAL ENVIRONMENT 11 3. CGIAR COMPARATIVE ADVANTAGE IN THE SOCIAL SCIENCES 15 4 ASSESSMENT OF CURRENT CGIAR SOCIAL SCIENCE.... 19 4.1 A New Business Model, but with Old Staffing Patterns 20 4.2 Finance................................................................................ 23 4.3 Staffing................................................................................ 26 4.4 The Pursuit and Measurement of Impact....................... 31 4.5 Organization and Partnerships........................................ 36 4.6 Research Quality............................................................... 42 4.7 Summary Assessment...................................................... 54 5. RECOMMENDATIONS FOR CGIAR SOCIAL SCIENCE 57 6. CONCLUSIONS 73 REFERENCES 73 FIGURES 3.1 CGIAR Social Science Comparative Advantage, Research Foci and Specific Lines of Research................................................ 18 4.1 Real CGIAR Expenditures, 1985-2008 24 TABLES 4.1 Center funding, 2003-2007...................................................................................... 25 4.2 Center-reported social science staff numbers, 2007............................................. 26 4.3 Top 25 cited papers published since 2003 in 16 primary journal outlets........... 51 4.4 Bibliometrics for Social Scientists at Selected CGIAR and Other Development Research Units, 2003-8............................................................................................ 52 4.5 Citation Statistics per IRS Social Scientist (n=223)................................................ 53 ANNEXES 1 Terms of Reference............................................................................... A-l 2 Panel Member Biographical Details.................................................. A-8 3 CGIAR Social Science Stripe Review e-consultation wrap-up........ A-ll 4 Non-CGIAR Individuals and Organizations Consulted.................. A-30 5 ARI Social Scientists Survey Results.................................................. A-31 6 Summary of Relevant Comments from Recent CCERs and EPMRs A-39 7 Best publications analysis............................................................ .A-61 XV EXECUTIVE SUMMARY The CGIAR faces a world rapidly awakening from a long period of complacency about international agricultural research, a world in which there is at once increased international development assistance, for agricultural development and environmental issues in particular, and heightened high-level attention to issues on which the CGIAR should be well positioned to contribute. Yet donors are also more demanding than they once were and the landscape is considerably more competitive. A window of widespread and high-level demand is now opening wide for agricultural and environmental researchers, presenting a favorable but highly competitive environment for CGIAR social science research. In order to take full advantage of this opportunity, the CGIAR must carefully assess its own comparative advantage in the social sciences within this space, assess its performance in those areas of comparative advantage, and organize itself to enhance its performance going forward. This review, commissioned by the CGIAR Science Council, aims to inform the CGIAR community on these matters, assessing the present state of CGIAR social science research, culminating in recommendations on how best to advance the CGIAR Mission through cutting edge social science. Assessment The CGIAR's mission fundamentally concerns the well-being of humans - "to achieve sustainable food security and reduce poverty in developing countries" - and, secondarily, mitigation of adverse anthropogenic change in the natural environment. The System's core instrument for achieving its mission —research - turns entirely on human agency in the processes of discovery, adaptation, adoption and diffusion of new technologies, policies and institutions. The core mission and instrument are all social science topics. It is therefore hard to imagine how the CGIAR could possibly achieve its mission without strong social science to complement the natural science intrinsic to its thematic concerns with agriculture and the natural environment. Generalizations about CGIAR social science necessarily oversimplify. There are pockets of outstanding social science, but also much weakness and great heterogeneity among and within 15 Centers and four Challenge Programs. Although the panel's assessment is frank in identifying several aspects of CGIAR social science that need serious, immediate attention, it is equally important to emphasize that the System has many excellent scholars generating high quality work that both influences broader global dialogues on agricultural development and feeds directly into high impact development interventions and policies. CGIAR social science has comparative advantage in generating international public goods (IPGs) related to (i) sustainable agricultural productivity increases by and for the poor, (ii) conservation of the natural resources on which the rural poor directly or indirectly depend for crucial environmental services, especially concerning agricultural productivity, and (iii) institutional, policy and technological innovations that enhance the quality of life for poor and marginalized agrarian populations. CGIAR social science research should emphasize support of productivity growth via technological innovation, institutional innovation, and agricultural and rural development policy analysis. Of these, the first is without question the single most important area of CGIAR social science comparative advantage as it most fully capitalizes on the close interactions between social and biophysical scientists that are distinctive to the CGIAR. Honoring CGIAR comparative advantage in social science requires more than just identifying and emphasizing these core research themes. It equally requires identifying and enforcing strict 1 boundaries so as to avoid (i) unnecessary duplication of efforts more effectively performed by alternative providers, (ii) crowding out research and outreach capacity in developing country institutions whose emergence the System is meant to support, and (iii) the high opportunity cost associated with surrendering the space necessary for serendipitous scientific discovery. CGIAR leadership and social scientists need to guard against mission drift into other areas, perhaps especially on-the-ground development activities and more general development studies and policy analysis unrelated to agriculture and NRM, for which there is constant, even growing demand from donors. The panel senses that many donors expect the CGIAR to perform an impossible task, generating research that delivers large-scale, sustained poverty reduction in a very short time. Unrealistic expectations of short-term impact draw CGIAR social science away from its areas of comparative advantage, leading to problems of deteriorating research quality and staff morale. Given the growing importance of social science to the CGIAR as the System's mission has broadened, it is little surprise that social science has grown rapidly. But most growth has been on restricted project funding, much of it short-term. By all accounts, a significant majority of social science staff time in the 1980s and into the 1990s was covered by unrestricted or quasi- unrestricted (long-term, large-scale) funding. But by 2007 all Centers operated with more restricted funds than unrestricted. Indeed, only four Centers had unrestricted/restricted ratios greater than 0.6 in 2007, with most in the 0.4-0.5 range, consistent with the roughly 30-40% core funding of staff time that seems to have become the norm for CGIAR social scientists. Total CGIAR internationally recruited staff (IRS) grew by about 17% overall from 1995-2008, but social science IRS nearly doubled over the same period, to more than one quarter of total IRS. Nonetheless, the panel finds low morale among CGIAR social scientists, especially at the Centers with smaller social science staffs, largely as a consequence of increased fragmentation of social science research effort and significant increases in bureaucratic demands on scientists' time while expectations of scientific output and development impact have grown. The "doing more with less" management culture that now pervades the CGIAR is a path that inevitably leads to shortcuts, lower quality work and less job satisfaction, such that CGIAR social science is, in many ways ironically, actually "doing less with more," as resources and staffing have increased yet research quality seems to have declined and the panel found no evidence of increased development impact. This heavily impacts recruitment and retention, especially of skilled mid­ level and senior social scientists. The CGIAR is not well organized to generate big successes in the social sciences; its direct focus is on the agricultural sciences and natural resources, not on social processes. In order to fully seize emergent opportunities in global agricultural research, some reorganization of CGIAR social science is almost surely necessary. One problem is the lack of critical mass. Although staffing has increased, social scientists are spatially dispersed and their time is highly fragmented to the point that perhaps half the Centers lack effective critical mass in social science research, especially outside of economics. A second organizational problem concerns the general lack of boundaries such that even where critical mass seems to exist, it quickly becomes excessively diluted across too large and diverse a portfolio of work. The panel asked scores of CGIAR social scientists to identify activities currently undertaken that fall outside their comparative advantage and that they therefore should not be doing. Astonishingly, not a single respondent identified any such activity. They see no boundaries. Third, the heavy reliance on restricted funding has eroded the quality of the partnerships that are central to CGIAR science, especially with advanced research institutes (ARIs) but with NARS as well. 2 There is much strong social science done within the CGIAR, with well-deserved international research awards, international accolades and citations to prove it. But there are also many CGIAR social scientists whose toolkits appear methodologically out of date, many projects that evince little careful thought about research design and methods, many publications that are poorly conceptualized and executed, and data sets of questionable quality. A significant share of the social science research undertaken in recent years by CGIAR IRS social scientists could be done just as easily - and far more cheaply - by strong recent graduates from Masters programs at ARIs and leading national universities in the global South. The heterogeneity of research quality within the System is alarming and must be addressed swiftly and directly. Tangible development impact from high quality research is uncertain; but it is virtually certain that no major sustained improvements in poverty result from poor quality research. There are pockets of excellent social science research by internationally renowned experts, good morale and real impact within the CGIAR, to be sure, especially where there remains critical mass, a local culture that prizes rigorous research, active collaborations with leading ARI researchers and adequate long-term funding to minimize fragmentation of scientists' attention and effort. But in all Centers, capable staff are put in untenable situations and too often driven out by the new business model (explained in detail in Section IVA) and by management pursuit of bigger Centers that are not necessarily better ones. Key IPGs are not being produced; instead a large amount of development consulting-type activities are undertaken in order to meet payroll expenses, Centers' organizational drive for growth, and immediate donor demands. IFPRI can certainly survive as a standalone social science entity; perhaps 4-6 other Centers' social science units can as well. But most CGIAR social science units are either quite weak already or extremely fragile. Hiring patterns in response to development-oriented restricted funding are reinforcing this mission drift. And all CGIAR social science units are significantly underperforming their potential. In aggregate, CGIAR social science has lost much of its research coherence and quality through dilution effects resulting from excessive reliance on short-term, restricted project funding and the associated pursuit of increased immediate development impact. The irony is that there is little or no evidence that CGIAR social science has in fact increased its development impact by focusing more directly on that end goal. Indeed, the panel suspects the opposite, i.e. that CGIAR social science's impact has lessened as donors and management have pushed harder for immediately visible development impact because it has meant less attention to the fundamentals of the intermediate research steps. The increasingly transactional - and occasionally competitive - nature of relations with many NARS and among Centers is but one troubling sign of diminished sustainable development impact. Rapid revenue growth associated with loss of disciplined focus on core competencies is at the root of these problems. This can be reversed if Center, CP and System leadership is prepared to make serious, sustained efforts at rehabilitating CGIAR social science so that it can make the major contributions to CGIAR goals and to global agricultural and rural development of which it is plainly capable. CGIAR social science is essential to the System and has long made seminal contributions to global social science as well as to generating broader development impacts through innovation in agricultural technologies, NRM practices, institutions and policies. But CGIAR social science is overtaxed and underperforming. The CGIAR once set the agenda in the social science of agricultural and rural development and NRM. It no longer does. Mission drift and fragmentation of social scientist time is universal, leading to high variance in the quality of research and partnerships and in impact, such as the System measures it. But the prominence 3 and performance of CGIAR social science can be restored if System and Center leadership and donors engage in substantive reforms to realign the business model, mission and staffing. Recommendations Looking to the future, one must grasp simultaneously the tremendous opportunities facing CGIAR social science today, the crucial role the social sciences play within the CGIAR, its distinguished past, and the perilous state of CGIAR social science today. The SSSR panel offers 4 broad recommendations, encompassing 21 specific sub-recommendations. We believe these recommendations will allow the System to seize emerging opportunities and rehabilitate its social science activities. The four recommended reforms relate to management, reorganization of social science activities, human resources and research quality standards. Recommendation 1: Undertake essential management reforms The root problems of CGIAR social science are directly traceable to management problems associated with the business model pushed by donors and accepted by CGIAR leadership, the incentives facing Center senior management, the selection of senior management with too little regard to their comprehension or vision of the role of social science within the CGIAR, insufficient focus on core areas of comparative advantage, and on excessive emphasis on measuring impact in all its minutiae. Thus the panel's first and most fundamental recommendation is that the CGIAR must undertake essential management reforms in order to improve its social science research. Without management reforms, progress is exceedingly unlikely. The essential management reforms comprise seven discrete sub-recommendations: la) Resolve the mismatch between the business model and staffing patterns lb) Realign management incentives lc) Improve leadership selection ld) Tighten the focus on comparative advantage le) Focus on impact but end the impact measurement obsession lf) Mainstream gender equity as a basic axiom of CGIAR research lg) Require full indirect cost recovery Recommendation 2: Re-organize and re-focus CGIAR Social Science The present, diffuse organization of CGIAR social science limits its impact. Partnerships have become excessively transactional and ties with ARIs, in particular, must be actively restored. Training and capacity building with NARS and other local partners have suffered. The System needs renewed focus on direct research mentoring to help build and sustain partners' social science research competencies. It needs to organize so as to realize the IPGs and large-scale impact achievable through regional and global integration of research designs and findings, perhaps through a new Mega-Program. The System likewise needs to invest in a larger-scale center of excellence on impact - how to achieve it through research prioritization and the design and conduct of project implementation, and how to measure it both ex ante and ex post. This would marshal what are presently excessively dispersed and ineffective Center-specific units and reinforce the core objectives of a SPIA that is too detached from Centers and too poorly resourced. Finally, as key functions that need to be undertaken at larger scale are reorganized 4 into alternative structures, the System should shrink its unproductive Center social science units. This can be accomplished through five sub-recommendations: 2a) Restore longer-term partnerships, especially upstream 2b) Focus training and capacity building on research mentoring 2c) Organize a Regional Systems Analysis Mega-Program 2d) Organize a Mega-Program on Stimulating and Evaluating Innovations 2e) Shrink unproductive social science units Recommendation 3: Update social science personnel management practices The key input to any research enterprise is skilled human capital. Thus recruitment and retention of the world's best researchers with relevant interests is the highest order priority for CGIAR social science. As indicated earlier, field-based, empirical research is resurgent within many of the social sciences today. This opens up exciting possibilities for CGIAR recruitment of highly qualified staff with field experience and relevant, rigorous toolkits. But the CGIAR must make some substantive reforms in personnel management in order to realize those possibilities. The SSSR panel has five specific sub-recommendations that would be important steps in the right direction: 3a) Introduce a CGIAR Young Scientists Program 3b) Increase entry-level compensation packages 3c) Establish a clear research career track for social scientists 3d) Restore competitive travel and sabbatical programs 3e) Employ modern human resources management practices Recommendation 4: Foster a cidtnre of rigorous social science research Critical intellectual mass and strong ARI linkages are necessary but certainly not sufficient to upgrade CGIAR social science research quality. Centers must introduce clear incentives and inculcate a culture of rigorous research in order to sustainably upgrade CGIAR social science. The SSSR panel has four specific sub-recommendations in this regard: 4a) Restore social science research seminar programs 4b) Individual performance measurement 4c) Stop wasting money on in-house publications other than policy or research briefs 4d) Establish a CGIAR Institutional Review Board The CGIAR desperately needs high quality social science in order to achieve the laudable and much needed impacts on poverty, food insecurity and environmental degradation to which the System aspires. It is equally true that the global social science community needs the CGIAR, which has, for decades, contributed important data, findings, capacity building and applied research that have significantly shaped international research on agricultural and rural development, natural resources management, and broader human behavior and well-being. Today, however, the model for producing high quality social scientific work within the CGIAR is broken. The System's new business model is mismatched with both staffing patterns and the 5 impacts the CGIAR Mission seeks to achieve. This compromises staff and research quality, and appears unsustainable. There has been growth in the number of social scientists in the CGIAR. But it reflects not reinforcement of areas of strength so much as loss of focus on CGIAR comparative advantage in the social sciences. There is distressingly high heterogeneity in and low average quality in research, capacity building and support to non-social science research within the System. In order to enhance its social science activities and accomplishments in support of the System Mission, the CGIAR must take major steps. The panel's recommendations can help the System seize the attractive opportunities presented by resurgent global demand for high quality research on international agriculture and related questions of rural development, climate change, land and water management, genetically modified organisms, and food policy. Without fundamental reforms to sharply increase the unrestricted (or quasi-unrestricted) share of funding that supports scientists, to more tightly focus on its areas of comparative advantage, and to attract and retain the best social scientists with interests in agricultural and rural development, however, the likelihood of realizing the considerable potential of CGIAR social science will be low. This need not be. The time is ripe for change in CGIAR social science and the likely rewards to effecting such change are great. 6 Figure 4.1 Real CGIAR Expenditures, 1985-2008 350 .2 = 300 E # 250 ITI ” 200 t/1 £ 150 c 100 ai Q. aX) 50 Q) Cd 0 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 ■ Unrestricted ■ Restricted But the panel strongly cautions against treating revenue growth as even an intermediate objective, as it can foster excessive reliance on short-term, small-scale restricted funding. Indeed, Figure 1 shows that growth in CGIAR funding has come entirely from restricted funding. Unrestricted funding declined 39.0% in real terms, an annualized rate of 2.5%. Dramatic growth in restricted funding combined with steady shrinkage in core, unrestricted funding has gone hand-in-hand with a loss of research focus from areas of comparative advantage, to problems in retention and recruitment of skilled social scientists and deteriorating morale among those retained, to increasingly heterogeneous (and, on average, lower) quality research and partnerships, and to many comparatively low-quality EPIA activities. Of course, the shift in funding roughly parallels the significant expansion of the System's mission to encompass poverty reduction more broadly, not just agricultural productivity growth for developing countries, and growth in the number of Centers. So the causal relation between funding composition and mission creep is difficult or impossible to untangle. What is clear is that the CGIAR needs better Centers, not necessarily bigger or more Centers. As the CGIAR Financial Information System data in Table 4.1 show, total funding increased sharply in the CGIAR over the past five years, with double digit percentage increases in 11 of 15 Centers. In all but four Centers, overall funding has expanded faster than IRS staffing, indicating that resources available per scientist have grown over this period, substantially in some Centers. By 2003, only two Centers operated with more unrestricted funds than restricted; by 2007 none did. Indeed, only four Centers had unrestricted/restricted ratios greater than 0.6 in 2007, with most in the 0.4-0.5 range, consistent with the roughly 30-40% core funding of staff time that seems to have become the norm for CGIAR social scientists. The four Centers that have enjoyed the faster growth in funding - IFPRI, ICRISAT, CIP and CIAT - are all in this range, underscoring that budget growth is coming almost entirely from restricted project funding. 24 As they have grown increasingly oriented downstream like NGOs and development consulting firms, Centers have also gravitated increasingly toward activities that resemble development project implementation work. These activities are essential to development impact. But is this the right role for the CGIAR and for massive international public investment in the System? If so, is CGIAR social science properly staffed for that role? The System is now caught in an untenable situation. Its old staffing pattern - lots of expensive, internationally recruited Ph.D. researchers - is a mismatch for its new business model, which is based on restricted-funding, downstream-oriented projects, which predictably lead to transactional partnerships and heavy administrative and impact assessment burdens. This makes it ever harder to recruit and retain the highest quality social science Ph.D.s, without whom it is extraordinarily difficult to design and implement large-scale, long-term, IPG- generating SSR. This is an unsustainable mismatch. Without a concerted effort to shift course, CGIAR social science will naturally transform into what might be more aptly labeled a Consultative Group on International Agricultural Development (CGIAD) as its high-level research capacity atrophies to trivial scale, leaving it as little more than a loose network of high- cost, high-quality development consultancies. If the CGIAR embraces this path, its development impact is likely to decline even if this affords access to substantially higher aggregate resources: Even a billion dollars devoted to producing direct development impacts is not likely to increase the average income of more than a billion poor in the world by much more than a dollar each year. The System is inherently trading away high-return, longer-term investments for short­ term, lower-return ones. The path CGIAR social science follows is ultimately a leadership and management issue that far transcends the System's social sciences. Indeed, our sense is that many of these problems are not exclusive to the social sciences within the CGIAR, although they are perhaps most acute in CGIAR social science. 4.2 Finance Total expenditures in the CGIAR are extremely difficult to allocate between social sciences and the rest, as many projects are multidisciplinary, Centers' accounting systems vary markedly so appropriate data were simply not available for enough Centers. Thus we look at expenditures on social science staff, which provides a reliable indicator of CGIAR investments in social science since laboratory and equipment costs for social science research are typically modest. The imperfect data available suggest that CGIAR social science has grown considerably, but largely in response to short term restricted funding opportunities. This is clearly reflected in Figure 1, which depicts the real (i.e., inflation-adjusted) total expenditures of the CGIAR from 1985-2008.1 Across multiple Centers, the panel got the impression that Center management's incentives favor revenue and staffing growth above all other objectives. On those terms, CGIAR social science is succeeding. Total real expenditures by the CGIAR increased 69.4% from 1985- 2008, an impressive rate of 7% annually. 1 The data come from the CGIAR Secretariat. Nominal expenditures are adjusted for industrialized country average consumer price indices, as reported by the International Monetary Fund in the serial publication International Financial Statistics. The base year in the figure is 1985. 23 expansion of core resources. It is unclear if this shift was ever defined as a clear strategy. The panel's sense is that it has been more of an incremental and inadvertent evolution in response to a range of incentives and pressures internal and external. System expansion and restricted funding have been largely demand driven, with donors seeking international expertise to address development agendas and contribute to demonstrable short term products and outputs. Donors' interest in large-scale, strategic questions that involve agriculture waned steadily, largely as donors have become far more short-sighted and much less interested in agriculture. As indicated earlier, this has begun shifting in the past couple of years. But the momentum of the past twenty years continues to drive CGIAR social science. The business model has been essentially inverted over the past twenty years. Inflation-adjusted core funding shrank but restricted funding rose rapidly. The CGIAR is now at the point where senior social scientists typically have only 30% - and often less - of their time supported on core funding. They depend heavily on restricted funding - much of it small-scale and short-term - for the bulk of their and their team's salary support, a model similar to that of consulting companies, which universally provide core funding to key staff. Scientists also now rely almost entirely on restricted funds to cover operating costs. Many Centers have adopted intra-day time tracking programs characteristic of law firms, consulting companies and accounting firms. While accurate accounting for effort on different activities is clearly desirable, this development is a clear signal of the kind of short-sighted fragmentation and bureaucratization that chokes off creative, path-breaking research. Data reported by Centers indicates that IRS social scientists spend nearly 30% of their time, on average, in administration. Although slightly lower than for non-social scientists (34%), this surely fails to capture time spent in project meetings with stakeholders, donors, administrators, etc. nor time spent writing research funding proposals. The fragmentation and diversion of scientist time away from scientific research is draining the System of a key source of comparative advantage: its skilled, internationally recruited scientific workforce. These same pressures and incentives have drawn Center partnerships downstream, heavily focused on NARS and NGOs. Collaborations with ARIs have grown markedly weaker over the past twenty years, leading to and reflecting an increased disjuncture between academic researchers and development practitioners. This has reinforced the pre-existing tendency toward a heavy reliance on economists and has also induced non-social scientists to migrate into non-technical social science research. As a further indication of growing donor control over CGIAR research, impact assessment demands on Centers have grown exponentially. No one disputes the need to establish the impact of System activities. There must be accountability to funders to justify the massive investments made in the System. But the CGIAR may now be the most over-scrutinized research enterprise in the world, burdened by donors levying constant demands for short-term ex post impact assessment (EPIA) on even small-scale projects. The resulting EPIA work, generally based on weak design and poorly controlled data, generates assessments of low quality and credibility. Conflict of interest problems abound as Centers and scientists are typically responsible for evaluating their own project impacts. SPIA has made notable efforts to correct these flaws, but its resources and influence are limited and the panel's impression from reading many Center and CP EPIA studies and speaking with scores of stakeholders is that Systemwide progress has been scant. 22 The professional rewards to CGIAR social scientists were relatively generous in that era: they enjoyed considerable latitude and ample core funding to design rigorous, often long-term studies and carry them through to analysis, publication and outreach if they were self-motivated to do so. There was significant mobility between universities and Centers during that era. Centers invested in sabbaticals and travel to international conferences for senior scientists to help them maintain and update their technical skills and to support retention of the System's most skilled social scientists. Center social scientists, however, did not have complete autonomy. CGIAR social scientists had extensive responsibilities in supporting Center biophysical scientists and in facilitating capacity building at national- and local-level partner institutions. They were often somewhat professionally isolated and absorbed in administrative duties once they hit mid-career. But these positions were attractive and respected, not radically different from academic positions where scientists divided time between research, outreach/extension, service and student teaching and supervision. Beyond the natural attraction of these positions, recruitment of CGIAR social scientists was fostered for more than 20 years by the exceptionally successful Rocky Doc program, which provided Centers with a subsidized stream of high-quality, pre-screened, internationally- recruited junior research staff from many different social sciences. The Rocky Doc program was the System's main channel for recruitment of talented non-economists. Many of the Rocky Docs went on to become international leaders in the field, some of them based at CGIAR Centers, many at ARIs or donor organizations where they became strong external champions of the CGIAR. The Rocky Doc program was an effective counterweight to the structural tendencies of the CGIAR to separate its social scientists from active ARI researchers and to deemphasize peer- reviewed publication. The New Business Model Beginning in the early 1990s, the System has added Centers, expanded the System and Centers' mission, especially to encompass policy-oriented and natural resources management research and more direct attention to poverty and gender issues. As discussed in Section 3, these are areas in which the CGIAR holds comparative advantage in particular lines of research, so expansion into these areas was natural and appropriate. The panel would certainly not recommend a retreat to the research domain of the 1970s and 1980s CGIAR. But expansion and growth seems to have become the central driving ambition of Center and System management. While over the long-term growth may well be an indicator of quality if investors reinforce successes, over the short-to-medium-term the relationship between growth and performance is tenuous at best. A certain amount of CGIAR growth has been instead through conglomeration and expansion beyond original core competencies into new lines of business, some of which prove a good fit ex post, others of which do not. A central problem is that core funding available per Center has not kept pace with System expansion, which has been based entirely on growth in restricted funding, as we show in Section 4.2. At the same time, policy-oriented social science research proved able to attract considerable short-term restricted funding. Core funding in the social sciences shrunk especially fast, it seems, although the System lacks the consistent time series data to test that hypothesis rigorously. Overall, however, there was indisputably a substantial broadening of the CGIAR social science research agenda as Centers 'followed the money' without commensurate 21 4.1 A New Business Model, but with Old Staffing Patterns The Old Business Model This section will necessarily oversimplify; but the panel sees a stark contrast that has emerged within CGIAR social science over the past generation. Until about 1990, research in the CGIAR was supported primarily by unrestricted core funding and focused relatively tightly on improving agricultural productivity (World Bank 2003). Its social science reflected this broader pattern, emphasizing long-term studies in support of farming systems characterization, agricultural technology adoption, adaptation and diffusion, and associated policy questions. The vast majority of CGIAR social science was concentrated on the first area of research focus identified above: sustainable agricultural productivity increases by and for the poor. Poverty reduction had not yet become an explicit focus of the CGIAR; that began in the 1990s. Thus there were major, important gaps in CGIAR social science of this era, especially around poverty, gender, natural resources and institutions. The broader social science of agricultural development outside of the CGIAR was also relatively immature at the time, so there was much "low-hanging fruit" available to those who undertook intensive, field-based data collection. Outside of IFPRI, the high-performing social science units were never large, rarely more than 5-6 internationally recruited PhD scientists, and commonly just 2 or 3. The strong interdisciplinary orientation of the Centers enabled much fruitful research but reduced both interactions with cutting edge social science in the ARIs and incentives to publish. This longstanding structural challenge is intrinsic to CGIAR social science. Many Center social scientists nonetheless skillfully leveraged ARI partners on the cutting-edge in terms of theory and methods and took full advantage of their ARI collaborators' greater incentives to publish. Graduate students' thesis research and ARI researchers with shared interests in agricultural and rural development multiplied the human capital available to Centers, typically at very low cost. NARS partners, including local universities, were actively and creatively engaged in intensive field-level work and as sources of promising students, many of whom worked their way up through graduate training and the CGIAR to international visibility. Mentoring of the next generation of social scientists trained to undertake field-based, rigorous applied research was a high priority, especially for nationals from low-income countries. And research results were frequently and directly shared with senior policymakers and local academics. These links with young talent and senior leaders built a few deep partnerships with NARS. A strong research program leveraged through long-term ARI partnerships was the foundation on which a suite of activities - training, outreach, NARS partnerships, policy advisory services - rested. This freed Center social scientists to engage in critical synthesis - "big think" - about central issues in agricultural and rural development: about the role of risk, which is disproportionate in agriculture relative to other sectors of the economy, about the processes and behavioral and welfare effects of technological change, about the economy-wide multiplier effects of agricultural productivity growth, about the political economy of agricultural policy, etc. CGIAR social scientists such as Barker, Binswanger, Byerlee, Hayami, Herdt, Mellor, Pingali, Pinstrup- Andersen, Ruttan, et al. earned global reputations as intellectual leaders on agricultural and rural development and had a significant impact in their academic disciplines, as did many of their academic and NARS collaborators. 20 4 ASSESSMENT OF CURRENT CGIAR SOCIAL SCIENCE Generalizations about CGIAR social science necessarily oversimplify. The heterogeneity among and within 15 Centers and four CPs is considerable, as is to be expected of any global enterprise with more than 300 internationally recruited staff. Some CGIAR social scientists are internationally recognized intellectual leaders whose work commands respect and contributes to real development impact. It shapes development agendas and others' research activities. But most CGIAR social scientists lack strategic direction, current technical toolkits, and the intellectual space needed for thoughtful research. Most struggle to balance competing demands in a System that increasingly exhibits little sense of boundaries on its work: any potentially fundable task related to agriculture, food and natural resources in the service of development seems fair game in most Centers today. Both scholarly and development successes and systemic weaknesses are readily apparent. Ultimately, most of those weaknesses appear traceable to (i) excessive reliance on restricted funding and the incentives and constraints inherent to the business model that has emerged within the CGIAR over the past couple of decades (see Section 4.1 below), and (ii) lack of management vision for and commitment to social sciences. There is a lack of vision and leadership at System level and at many Centers and CPs, apparent in, among other manifestations, a lack of senior-level consensus on what CGIAR social science should be doing. A surprising share of past and current senior Center leadership evinces little substantive understanding or appreciation of social science research and its essential role in advancing Center, CP and CGIAR missions. Funding pressures and lack of vision and commitment from management have led to substantial mission drift and increased variability in the quality of staff, their outputs and impact. Because the core problems revolve around financing and management, in shorthand, the “business model" for CGIAR social science, the panel begins by describing the fundamental problem of CGIAR social science (and perhaps of the CGIAR more broadly): it has, perhaps inadvertently, adopted a new business model while retaining old staffing patterns. The mismatch between the two is unsustainable and must be resolved. As the subsequent subsections enumerate, that mismatch has led to widespread problems in the financing of CGIAR social science, in staffing, in the organization of social science, including the System's partnerships, in research quality, and in the pursuit and measurement of impact. There remains considerable talent in the System, with many impressive, skilled, hard-working social scientists deeply committed to the CGIAR Mission. The System also has considerable resources with which to achieve important discoveries that can, directly or indirectly, have substantive development impacts. But CGIAR social science is overtaxed and underperforming. The CGIAR once set the agenda in the social science of agricultural and rural development and NRM. It no longer does. But its prominence can be restored if CGIAR leadership and donors engaSe 'n substantive reforms to realign the business model, mission and staffing. 19 Figure 3.1 CGIAR Social Science Comparative Advantage, Research Foci and Specific Lines of Research CGIAR Comparative Advantage CGIAR Social Science Research Focus Specific Lines of Research Multidisciplinary research on Productivity growth and poverty • Systems and farmer characterization work agricultural productivity growth reduction via technological innovation. • Participatory plant breeding by and for the poor. • Ex ante impact assessment for priority setting • Technology adoption studies to establish cross-sectional and Close interaction with stakeholders inter-temporal patterns of uptake and adaptation of CGIAR at local, national and global levels. innovations • Ex post impact assessment for evaluation Producer of significant new • Natural resources management (NRM) for enhancing and intellectual property in agricultural sustaining productivity growth sciences. • Emergent research on innovation systems and impact pathways • Intellectual property rights management Close interaction with stakeholders Productivity growth and poverty • Investigation of market and non-market resource allocation 00 at local, national and global levels. reduction through institutional mechanisms innovation. • Sociocultural constraints on and incentives for productivity­ Multidisciplinary organization enhancing innovations with opportunities for data • Collective action and property rights integration and interoperability • Agricultural input and output distribution systems, including farmer-based organizations • The design and management of agricultural and NRM research institutions, whether around intellectual property rights, impact assessment methods, or related topics Unique combination of Productivity growth and poverty • Policy analysis related to international agricultural trade, international-caliber technical reduction and by directly informing domestic and regional agricultural input and output markets expertise and its multinational agricultural and rural development • Intellectual property rights and agricultural research policy, nature. policy. and the policies for the conservation of animal and plant genetic resources Widespread “honest broker" perception for policy guidance. management of agricultural and NRM research institutions, whether around intellectual property rights, impact assessment methods, or related topics. The third and final area is social science research to directly inform agricultural and rural development policy. This encompasses policy analysis related to international agricultural trade, domestic and regional agricultural input and output markets, intellectual property rights and agricultural research policy, systems of payments for environmental services and the policies for the conservation of animal and plant genetic resources. The CGIAR occupies a rare and important - perhaps even unique - space in the universe of policy analysis suppliers due to its combination of international-caliber technical expertise and its multinational nature. These characteristics induce a widespread perception of a more objective, "honest broker" source of policy guidance than might come from (especially government-funded) ARIs or NARS, or more politically-driven international organizations. Honoring CGIAR comparative advantage in social science requires more than just identifying and emphasizing these core research themes. It equally requires identifying and enforcing strict boundaries so as to avoid (i) unnecessary duplication of efforts more effectively performed by alternative providers, (ii) crowding out emergent research and outreach capacity in developing country institutions that the System is meant to support, and (iii) the high opportunity cost associated with surrendering the space necessary for serendipitous scientific discovery. CGIAR leadership and social scientists need to guard against mission drift into other areas, perhaps especially on-the-ground development activities and more general development studies and policy analysis unrelated to agriculture and NRM, for which there is constant, even growing demand from donors. Whereas some CGIAR social scientists may have the skills to undertake such work, the opportunity costs are high, as other organizations typically hold comparative advantage in these broader topics, and in many cases these topics are only weakly related to the core mission and instrument of CG institutions. The need to respect the boundaries implied by comparative advantage applies equally to donors and other stakeholders outside the Centers. As is true of any research organization (including those in the private sector), only a small proportion of CGIAR innovations are ultimately adopted by and diffused among farmers in developing countries. This is a source of widespread frustration, indeed perhaps excessively so given that estimated average and marginal rates of return on CGIAR research remain extremely high (Alston et al. 2000, Gardner 2003, Raitzer and Kelley 2008). Furthermore, innovation and impact pathways are multiple, complex, indirect and necessarily involve many organizations. And given the modest scale of CGIAR social science, in order to achieve impact it must leverage partnerships upstream, downstream and laterally. When donors implicitly hold the CGIAR accountable for functions that are partners' comparative advantage, they compel self-defeating mission drift. Expectations of CGIAR social science must be commensurate with its scale and comparative advantage. We sense that many donors expect the CGIAR to perform an impossible task, generating research that delivers large-scale, sustained poverty reduction in a very short time. Unrealistic expectations of short-term impact draw CGIAR social science - and the System more broadly - away from its areas of comparative advantage, leading to problems of deteriorating research quality and staff morale. 17 of highly skilled social scientists able and willing to collaborate actively with other, non-social scientists. Somewhat more precisely, CGIAR social science has comparative advantage in generating international public goods (IPGs)1 related to (i) sustainable agricultural productivity increases of, by, and for the poor, (ii) conservation of the natural resources on which the rural poor directly or indirectly depend for crucial environmental services, especially concerning agricultural productivity, and (iii) institutional, policy and technological innovations that enhance the quality of life for poor and marginalized agrarian populations. CGIAR social science research should emphasize support of productivity growth via technological innovation, via institutional innovation, and via agricultural and rural development policy analysis, The relation between the CGIAR's sources of comparative advantage, areas of appropriate social science research emphasis, and sample of specific lines of research activities are reflected in Figure 1, adapted slightly from the Phase 1 normative framework. The first and single most important area of CGIAR social science comparative advantage revolves around technological innovation. This involves traditional work on systems and farmer characterization work, participatory plant breeding, ex ante impact assessment for priority setting, technology adoption studies to establish cross-sectional and intertemporal patterns of uptake and adaptation of CGIAR innovations, and ex post impact assessment for evaluation and learning. But it also encompasses newer lines of research. Research on and capacity building with respect to intellectual property rights management and policy also fall into this domain. Thoughtful research to identify impact pathways and generalizable lessons about the institutional arrangements that underpin effective innovation systems likewise merit inclusion. Natural resources management (NRM) research essential to increasing and sustaining productivity is of growing importance in the face of climate change, growing water scarcity and degrading soil fertility. In the panel's view, research on productivity growth by and for the poor should represent most CGIAR social science resources, outputs and impact. This is the bread- and-butter of CGIAR social science research and appears to offer the highest return of all lines of CGIAR social science research. The second area of CGIAR comparative advantage in social science research focuses on productivity growth through institutional innovation. This encompasses investigation of market and non-market resource allocation mechanisms, including sociocultural constraints on and incentives for productivity-enhancing innovations and collective action and property rights, as well as agricultural input and output distribution systems, including farmer-based organizations. It also relates to research, outreach and capacity building on the design and 1 It is apparent to the panel that there is much misunderstanding of the IPG concept within the CGIAR, with too many people failing to recognize that work can be at once locally relevant and contextualized and yet internationally generalizable and thus an IPG. As a reminder, a public good is nonrival (i.e., one person's enjoyment in no way degrades the availability or quality for others) and nonexcludable (i.e., anyone in the relevant population can enjoy it). Some public goods must be produced or provided (e.g., knowledge, a new technology) while others must be conserved or protected (e.g., forests or the existence of species); some may be both produced and conserved (e.g., genetic material). The "international" modifier restricts the relevant set of public goods to those relevant to multi-national populations. The primary justification for this more restrictive spatial definition arises from the fundamental design principle of subsidiarity, which stipulates that any sort of spillover - of which public goods are one particular form - should be handled by the agency possessing the necessary technical capacity whose functional and geographic mandate most closely match the functional and geographic reach of the spillover. International organizations such as CGIAR Centers are thus tasked to provide international public goods only, leaving more localized public goods to more localized entities. But note that the public good is the knowledge generated, not the location in which that knowledge is generated. Well designed studies fielded in only a few specific locations, even within a single country, can generate knowledge of quite general, international value. This has long been true of much of the CGIAR's best social science. 16 3. CGIAR Comparative Advantage in the Social Sciences In defining the comparative advantage of CGIAR social science, we must begin by emphasizing the importance of the social sciences to the CGIAR. The CGIAR's mission fundamentally concerns the well-being of humans - "to achieve sustainable food security and reduce poverty in developing countries" - and, secondarily, mitigation of adverse anthropogenic change in the natural environment. The System's core instrument for achieving its mission—research - turns entirely on human agency in the processes of discovery, adaptation, adoption and diffusion of new technologies, policies and institutions. The core mission and instrument are all social science topics. It is therefore hard to imagine how the CGIAR could possibly achieve its mission without strong social science to complement the natural science intrinsic to its thematic concerns with agriculture and the natural environment. Just as agricultural technologies and NRM practices are deeply embedded in the farming systems in which they are found, so are those farming systems equally embedded in agroecological processes and economic, political and social structures that fundamentally shape their evolution and performance (Conway et al. 2006). In many of the settings of greatest interest to the CGIAR areas with the highest rates of poverty, malnutrition and resource degradation - these broader structures limit the speed and extent of adoption of improved technologies, practices and policies. Overcoming the obstacles posed by such structures in order to achieve development impact requires careful attention to institutional frameworks, to externalities and feedback effects among sub-systems, and to behavioral patterns that are the core concerns of social scientists (Lynam 2007). The SSR Phase 1 normative framework paper articulated a vision of CGIAR comparative advantage in the social sciences that was repeatedly and widely reaffirmed in Phase 2. CGIAR staff and stakeholders routinely emphasized one or more of three key sources of comparative advantage the System possesses relative to alternative suppliers of mission-relevant research. First, (most of) the Centers' location in developing countries facilitates highly contextualized, participatory, problem-driven, longitudinal research that is both more difficult and more costly to undertake from ARIs and consulting firms based in the more developed countries. This locational advantage is more about personal and institutional connections and tangible long­ term presence than about air miles efficiencies. Second, a relatively large cadre of highly-trained, internationally-recruited staff enables Centers and CPs to undertake more advanced and complex lines of social science research than many - but certainly not all - developing country NARS, NGOs or private firms can tackle. As the preceding section emphasized, the CGIAR represents but a small part of aggregate global social science research capacity on agricultural and rural development. Nevertheless, as a System it is a relatively large individual player that can undertake larger-scale IPG-generating activities than can many - perhaps any - other entity in this space. Third, the multidisciplinary nature of most Centers and CPs and the relatively low disciplinary boundaries within them make it relatively easier to organize diverse teams well- eqnipped to address inherently interdisciplinary challenges that research institutes or faculties organized along more narrow disciplinary lines have difficulty undertaking effectively. The CGIAR's capacity and tradition to organize around problems rather than disciplines and to work at a scale, over a time frame and with physical infrastructure difficult to replicate elsewhere is probably its single greatest source of comparative advantage in social science research. The intersection of these three attributes implies CGIAR comparative advantage in applied agricultural and rural research that demands both a presence in developing countries and teams 15 agriculture and natural resources management. The playing field for CGIAR social science has grown appreciably more crowded over the past generation. In summary, the CGIAR faces a world rapidly awakening from a long period of complacency about international agricultural research, a world in which there is at once increased international development assistance, for agricultural development and environmental issues in particular, and heightened high-level attention to issues on which the CGIAR should be well positioned to contribute. Yet donors are also more demanding than they once were and the landscape is considerably more competitive. The CGIAR can do nothing about these external environmental conditions. A window of widespread and high-level demand is now opening wide for agricultural and environmental researchers, presenting a favorable but highly competitive environment for CGIAR social science research. In order to take full advantage of this opportunity, the CGIAR must carefully assess its own comparative advantage in the social sciences within this space, assess its performance in those areas of comparative advantage, and organize itself to enhance its performance going forward. The ensuing three sections address those three topics: comparative advantage, assessment of recent performance, and recommendations for the future. 14 sensing, with profound effects on social science research on natural resources management. The emergence of new methodological tools related to network theory and agent-based modeling promise to add greater value to the analysis of primary and secondary data collected. Interest in topics traditionally at the heart of CGIAR research is strong and growing in the global social science community, which should make recruitment of strong partners and staff feasible and enhance opportunities for publication, citation, and impact of carefully executed research. Over the past couple of decades, there have been significant changes in social science training. Economics, political science and geography (at least the people-environment field) have grown significantly more quantitative, making it harder for graduates from weaker university systems in low-income countries to earn admission to and a Ph.D. from leading programs in the global North. The collapse of many low-income countries' university systems has also shrunk the supply of well-trained local scientific talent available to Centers, a few notable outliers in Asia and Latin America notwithstanding. The more qualitative social sciences - anthropology and large fields within political science, human geography and sociology - have increasingly disengaged from agricultural development as a central line of research, moving more into environmental and regional studies. In some sense, this is only a slight change from earlier eras, as anthropologists and sociologists have long been generally hostile to the Green Revolution and the CGIAR, making recruitment of the best young PhDs in those disciplines more difficult. The Rocky Doc program (on which more below) helped overcome this while it operated. Although the CGIAR can be attractive to all social science disciplines when actively recruited into appropriate positions, on balance, the supply of interested, well-trained social scientists in more qualitative fields has diminished over the past decade or so while supply in more quantitative areas has grown. Meanwhile, donors of all sorts bilateral, multilaterals, foundations and individual philanthropists - have become far more demanding of clear metrics of impact, often within unreasonably short timeframes that reflect more the career interests of program officers than realistic expectations about the pace at which applied scientific research can deliver large-scale, sustained changes on the ground. Aid fatigue has led to increased cynicism about development assistance, leading to far more stringent oversight and reporting requirements from funding recipients. And an ever-growing share of research funding, not just for CGIAR social science, but for all research by all providers, comes in the form of short-term restricted grants that limit scientists' degrees of freedom in research design and resource allocation. Donors' increasing impatience, combined with the increasing void left by weak NARS and local universities, has induced donors to push CGIAR Centers and CPs further downstream to fill the void left by national- and local-level actors. Other providers have emerged in that space as well, most notably international and local not-for-profit non-governmental organizations (NGOs) and for-profit consulting firms. The development practitioner community has exploded over the past twenty or so years, feeding (and feeding off of) the increasingly short-term orientation of donors and governments. At the same time, the stronger NARS have grown in size and skills and emerged as alternative regional suppliers of research and capacity building services. Many ARIs have also developed active, grant-based field research programs that compete directly with CGIAR Centers. Northern universities such as MIT, Michigan State and Yale - as well as several in middle income countries - have established long-term field research platforms in rural Africa and Asia where they conduct longitudinal surveys and randomized controlled trials, providing direct competition with the CGIAR. Other multilateral agencies - the World Bank, FAO, IFAD, etc. - have likewise expanded their field-based research operations in developing country 13 reflected by both the quarter century gap between issues of the World Bank's flagship World Development Report dedicated to agricultural development, the most recent of which was released in late 2007, and the familiarity of much of the theory, evidence and arguments contained in that highly influential report. The apparent stagnation in CGIAR “big think" remarked upon by many observers we spoke with during this review seems to reflect a broader global pattern over that period. But demand for agricultural research is now resurgent. The global food price crisis of 2007-8 highlighted the urgent need for research to stimulate sustainable agricultural productivity growth, especially to close the substantial yield gaps that persist in developing country agriculture. The food price crisis likewise sparked policymakers' interest in policy questions as to how best to dampen price volatility and to facilitate effective and equitable market access. Ideology has weighed heavily in such debates in the absence of much hard empirical evidence, opening up new opportunities for applied agricultural policy research. Meanwhile, there is now near-universal, high-level acceptance of the urgent need for research on how to mitigate and adapt to climate change and of the central role of agriculture in attending to that major challenge. And there are growing concerns about impending water scarcity and the possibility of violent conflict over water for agricultural uses, as well as about the environmental effects of both genetically modified (GM) crops and (especially liquid) biofuels. At the same time, global leaders have committed to sharp growth in development assistance funding, which increased again in 2008 in spite of the global economic downturn. After years of steady, sharp decline in investment in international agricultural research (World Bank 2007) that hit the CGIAR hard, international donor investments in agriculture has begun a rapid reversal, reflected most recently in the Obama Administration's April 2009 commitment to double United States assistance to developing country agriculture, the dramatic entry of the Bill and Melinda Gates Foundation into agricultural development grantsmaking and the July 2009 G-8 announcement of concerted efforts by OECD countries to redouble efforts on global food security. In developing countries, government funding for agriculture is likewise increasing again after a long period of decline. Further, within at least some of the social sciences - economics and geography in particular - there is renewed interest within the global research community in empirical research and in the sorts of technology, policy and institutional questions that lie at the heart of CGIAR social science research. The 1980s-90s era of “Washington Consensus"3 prescriptions based on overly simplistic neoclassical macroeconomic models has given way to newfound appreciation of careful empirical research on problems of considerable societal importance and the value of evidence-based praxis. For example, within mainstream economics, topics of agricultural technology adoption, the functioning of markets and rural institutions, and poverty dynamics are major lines of high-profile research. Land use and land cover change and sustainability science research have become vibrant areas of interdisciplinary research within the social sciences more broadly. Rapid improvements in information and communications technologies have fuelled renewed interest in field data collection in many social science disciplines and have enabled the rapid rise of geographic information systems (GIS), spatial analysis and remote 3 The term "Washington consensus" was coined in the 1980s by economist John Williamson to describe a set of specific economic policy prescriptions that had been widely accepted as the foundations of market-oriented reforms recommended to developing country governments by the International Monetary Fund, the World Bank and other international financial institutions, The term thereafter became synonymous with market fundamentalism. 12 2. THE EXTERNAL ENVIRONMENT In assessing the state of CGIAR social science and its prospects into the future, it is essential to first consider relevant patterns of external demand for and alternative suppliers of agricultural social science research. There have been significant changes in these factors over the past twenty years, indeed even within the past few years. Those changes have materially affected CGIAR social science, generally for the worse, although new opportunities are emerging from more recent changes. If decision makers in the System can seize those opportunities and undertake important reforms, the prospects for rehabilitation of CGIAR social science appear reasonably good. The CGIAR mission is "to achieve sustainable food security and reduce poverty in developing countries through scientific research and research-related activities in the fields of agriculture, forestry, fisheries, policy, and environment, " i This mission statement makes the System's functional emphases on research and impact on poverty and food security, as well as its thematic focus on agriculture and natural resources, abundantly clear. These emphases are distinctive. Yet CGIAR social science does not exist in a vacuum. Indeed, it is a relatively small player, even within the limited domain of agricultural research, much less if one adds in the vast array of practitioner groups working in international agricultural development. The approximately 310 internationally recruited social scientists within the CGIAR are dwarfed by the global community. An estimated 750 social science faculty work on international agriculture at land grant colleges and universities in the United States alone,2 and a similarly large community of researchers is employed in other ARIs worldwide. Then there is the still-larger community of development practitioners in government agencies, multinational organizations, and local and international NGOs. Although CGIAR research is highly valued, generates significant impacts and can have a significant multiplier effect through its influence on other actors, because of its relatively modest size, the CGIAR is necessarily not the source of most of the useful - much less high-impact - scientific discoveries within its mission areas (Pardey et al. 2006). Private firms, ARIs, NARS, think tanks, etc. are all major alternative suppliers of related research, perhaps especially on social science topics related to policies, institutions, markets, and human behavior and well-being. We found that CGIAR management and scientists are frequently under­ informed about the broader social science research community within which they operate, with implications for key questions of focus, framework, and partnerships. The broader agricultural research community has experienced a long period of widespread global complacency and overconfidence about the resiliency and performance of international agricultural systems. This was fostered, in part, by the CGIAR's early successes in helping stimulate agricultural productivity growth that outstripped population and income growth, leading to declining real food prices and a sense that the prospect of famine had been forever banished. Donors and governments steadily reduced investment in agriculture during the 1980s and 1990s. And there has been a marked decline in innovative, strategic thinking about agricultural and rural development, not just in the CGIAR but worldwide. This was vividly As stated on http://www.cgiar.org accessed July 2, 2009. 2 These rough estimates are based on informal conversations with staff at the Association for International Agriculture and Rural Development (AIARD) and the National Association of State Universities and Land-Grant Colleges (NASULGC). 11 (EPMR) of each Center. Annex 6 summarizes findings from those prior reviews of Centers and their social science-related programs. The SC Secretariat undertook a brief survey of Centers and CPs with respect to their procedures for the protection of human research subjects and use of Institutional Review Boards (IRBs), if any, as well as their policies and practices with respect to data management. Finally, the panel also read many related papers and collections of papers concerning CGIAR social science, including a substantial literature on innovation systems/institutional learning and change, since the SSSR TOR explicitly requested the panel to address that emergent area of work. It also obtained and reviewed additional data (beyond those assembled and reported in Phase 1) on paper and report downloads and on paper and scientist citations. The exact boundaries of CGIAR social science are difficult to define and vary by Center and CP. The panel has tried to err on the side of being inclusive by exploring SSR activities undertaken by any Center and CP scientist, regardless of formal disciplinary training. But there are surely relatively minor errors of inclusion and exclusion within both the enormous mass of documents and the data the panel reviewed. We should also note that different data sources yield slightly different quantitative results, although the qualitative conclusions appear robust to the specific data sources we could tap. The data reported here are the most current available to the panel, sometimes reflecting corrections and updates to the Phase 1 data. This report offers the panel's summary assessment and recommendations, per the SSSR TOR and is based on the considerable mass of qualitative and quantitative evidence assembled and the extensive consultations in which the panel engaged over the past seven months. In the next section we contextualize CGIAR social science by describing the external environment within which CGIAR social science activities take place. In section 3 we identify the comparative advantage of CGIAR social science, which frames our assessment of the current state and recommended future directions for the System. Section 4 offers the panel's assessment of CGIAR social science today. Section V concludes with a series of 4 general recommendations, nesting 21 specific sub-recommendations, with an eye towards the new integrated strategy and results framework under discussion within the CGIAR's Change Management process. Background details are contained in multiple appendices and in the Phase 1 background documents. In closing this introduction, we note that many of the conclusions we draw and the recommendations we offer, although focused on the social sciences, may well apply to the CGIAR more broadly. Our charge and our professional expertise are confined to the social sciences, so we focused there and do not claim broader generalizability. However, our sense from many conversations with natural scientists in the CGIAR and with longtime observers of the System - and from many comments we received on the first draft of this report - suggest that at least some of the panel's findings and recommendations may indeed apply more broadly. 10 CPs,6 as well as many Systemwide and Ecoregional Programs, the Gender and Diversity Program, the Standing Panel on Impact Assessment (SPIA) and the CGIAR Secretariat, plus 46 personnel from 15 different non-CGIAR stakeholder organizations (donors, NARS, and other research partners). A large number of those meetings were with non-social scientists, in both research and senior management positions. Panel members supplemented these meetings with interviews with another 50 international experts with some past or present association with the CGIAR. Special attention was paid to alumni of the Rockefeller Foundation's 1974-96 social science post-doc ("Rocky Doc") program. Annex 4 lists the organizations and individual experts with whom the panel consulted. This is certainly not an exhaustive list of all interested and knowledgeable CGIAR stakeholders. But this large group of informants provided a wide array of perspectives and insights reflecting virtually every conceivable widely-held position on the state of and preferred directions for CGIAR social science. The SSSR panel also reviewed voluminous project materials. In early 2009 the SC Secretariat requested basic information from Centers and CPs on all projects with "a substantial social science contribution,"— as reflected in at least three months' aggregate time commitment annually from IRS social scientists — with (restricted or unrestricted) funding of at least 12 months duration and a budget of at least US$100,000/year, completed after 31 December 2007, ongoing or fully approved to being in 2009. This focused the panel's attention on current or very recent projects with substantive social science inputs while intentionally ruling out short duration and small-scale projects. The panel then selected a stratified sample of 16 projects from 12 different Centers and 3 CPs, split evenly by project size (above and below the median of roughly US$300,000/year) and between projects completed or near completed (end date of 30 April 2009 or earlier) and those with significant time remaining (end date after 2009). Centers and CPs were then asked to provide the panel with all relevant documents (proposals, reports, evaluations) on those sample projects. The panel read the documents, in a few cases interacted with the project lead social scientists as needed to clarify particular points, and made qualitative, general assessments of the project design and performance. The SC Secretariat then fielded two online surveys (using the Survey Monkey tool, http://www.surveymonkey.com/) aimed at eliciting external stakeholders' and partners' impressions of CGIAR social science. One survey covered NARS senior social scientists; the other, ARI social scientists. In each case, the panel drew heavily on names provided by Centers of recent collaborators, supplemented by names suggested by panel members. Response to the NARS survey was relatively weak, with only 29 complete responses (of 110 requested, a 26% response rate), 48% of those from Asia, 31% from Africa. The ARI survey had a somewhat higher response rate of 34% (78 responses of 230 requested). Annex 5 summarizes the ARI survey responses. Interestingly, ARI and NARS survey respondents offered quite similar perspectives on the range of qualitative assessments we asked them to make with respect to CGIAR social science. The survey responses are consistent with the input received in individual and group interviews by telephone and in person. Thus when we invoke impressions of ARI or NARS social scientists, we draw on both interview and survey results. The panel also reviewed all of the Center-Commissioned External Reviews (CCERs) since 2003 relevant to social science as well as the most recent External Program and Management Review We met with representatives of Bioversity, CIAT, CIFOR, CIMMYT, CIP, ICRAF, ICRISAT, IFPRI, ILRI, IRRI, IWMI, as well as the Harvest Plus and Water and Food Challenge Programmes. The only Centers and CPs with which we did not have direct meetings were Africa Rice, Generation CP, ICARDA, IITA, Sub-Saharan Africa CP and WorldFish. Other than IITA, which opted out of the SSSR entirely, they all participated in the e-consultation. 9 The SSSR panel was tasked to provide a comprehensive system level assessment of the CGIAR's SSR agenda by examining: i) the current quality, coverage, relevance and productivity of social science activities to CGIAR priority research, with a focus on three key areas of SSR: • SSR support for productivity growth and poverty reduction via technological innovation. This focus area should include assessment of the social science contribution to rural livelihoods enhancement through natural resources management practices; • SSR support for productivity growth and poverty reduction via institutional innovation (including NRM institutions) and policy analysis; and • SSR for informing agricultural and rural development policy ii) the capacity and incentives in the CGIAR for conducting high quality research that can contribute to the CGIAR goals, and iii) the opportunities for improving the organization and partnership models for enhancing the quality and impact of SSR. The panel was explicitly asked to address aspects of productivity and effectiveness by assessing the quantity and quality of publications, the quality of linkages among scientists within the Centers and collaboration with advanced research institutions (ARIs), and the outcomes and impacts of the work of social scientists. It was tasked to comment on emergent - and sometimes controversial - research in innovation systems, and on the apparent level of awareness and use of appropriate and state-of-the-art methods by System social scientists. The review was to explicitly include: (i) comparison of social sciences research prioritization and management within the CGIAR against current global best practices; (ii) analysis of the organization, financing and staffing of the social sciences in support of CGIAR priority research; and, (iii) review of collaborations between IFPRI and the other Centers. In addressing its charge, the panel first reviewed the Phase 1 materials, including the extensive data and documentation provided by the SC Secretariat and Center and CP focal persons. Much of that data is summarized in the phase 1 background report, so we do not replicate that report's scores of tables and figures here; readers seeking the detailed, Center-by-Center bean counting are directed there. It read and undertook a quality assessment of each of the 216 "best publications" self-identified by the Centers, as well as all of the 70-plus impact and partnership statements the Centers provided in Phase 1. Phase 2 then publicly began with an e-consultation for 15 days in January 2009 (a summary is included as Annex 3). The e-consultation involved 42 subscribed participants, with at least one from each of the participating Centers and CPs, 23 of whom - representing 12 Centers, 1 CP, SC Secretariat and the SSSR panel - offered one or more written statements over the course of the e-consultation. The rich insights offered by the e- consultation participants and their reactions to and refinements of the Phase 1 documents were essential inputs to the panel as it framed its investigations over the ensuing seven months. Per the TOR, from March-July 2009 the panel undertook site visits in Colombo, Hyderabad, Nairobi and Washington, aiming to cover commodity-, policy- and NRM-oriented Centers and the System's two largest clusters of social scientists (in Washington and Nairobi). During those missions, the panel visited with 169 different CGIAR staff representing 13 different Centers and 8 1. INTRODUCTION 1 "Social sciences" comprise those disciplines that apply scientific methods to the study of individual and collective human behavior and outcomes. This includes anthropology, communications, economics, geography, law, management, political science, psychology, and sociology, as well as derivatives of those fields (e.g., agricultural economics, development anthropology, rural sociology), interdisciplinary fields (e.g., development studies, environmental studies, human ecology, public policy ) and social science-oriented branches of the biological sciences (e.g., community nutrition). 2 Scott Rozelle and Rob Tripp, "Stripe Review of Social Science in the CGIAR System: Scoping Paper," December 2007 (http://www.sciencecouncil.cgiar.org/fileadmin/user_upload/sciencecouncil/Monitoring_and_Evaluation/Stripe_ Review_scoping_paper_Dec_2007.pdf). 3 The International Institute for Tropical Agriculture (IITA) declined to participate in the SSSR. IITA data are included only where they were contained in pre-existing databases available to the Science Council Secretariat. All other Centers and CPs provided and verified additional data. 4 Paswel P. Marenya, "A Background Study for the CGIAR Social Sciences Stripe Review Commissioned by the CGIAR Science Council," September 2008 (http://www.sciencecouncil.cgiar.org/fileadmin/user_upload/sciencecouncil/Monitoring_and_Evaluation/SSR_Ph ase_l_Backgroud_Report__21_Sep_08_CBB_Version.pdf). 5 Christopher B. Barrett, "A Normative Framework for Social Science Activities in the CGIAR," October 2008 (http://www.sciencecouncil.cgiar.org/fileadmin/user_upload/sciencecouncil/Monitoring_and_Evaluation/Normat ive_Framework_October_27_2008_final.pdf). 7 Table 4.1 Center funding, 2003-2007 Center Increase in funding Unrestricted/Restricted Change in U/R Change in IRS 2003-7 (%) (U/R ratio) 2003 ratio 2003-7 2003-7 (%) IFPRI 75.1 0.7 -0.3 59.7 ICRISAT 61.4 0.7 -0.2 20.4 CIP 44.2 0.9 -0.4 13.5 CIAT 40.9 0.5 -0.1 -17.9 Bioversity 39.9 0.8 0.1 57.8 CIFOR 33.3 1.1 -0.2 -27.5 IRRI 31.1 0.8 0.0 2.8 IITA 23.2 0.5 -0.1 13.3 CIMMYT 19.4 0.7 -0.2 -28.0 ILRI 19.3 0.9 -0.3 31.4 ICRAF 15.7 0.5 0.0 -5.8 ICARDA 9.1 0.7 -0.1 -18.1 World Fish 4.4 0.7 -0.1 22.9 IWMI 2.2 0.5 -0.1 2.2 Africa Rice -4.7 2.3 -1.4 32.4 The panel could not obtain long-term data for social science staff funding. However, by all accounts, a significant majority of social science staff time in the 1980s and into the 1990s was covered by unrestricted or quasi-unrestricted (long-term, large-scale) funding. In the period 2003-2007 the unrestricted to restricted funding ratio fluctuated a great deal and was on average 0.8 (data for nine Centers); in 2007 only 38% of staff expenditures were covered by core funds. The panel emphasizes that restricted funding is a desirable part of a research funding portfolio. Cutting-edge social science researchers everywhere raise operational grant money for their research programs. A certain amount of restricted funding is healthy; it keeps cutting-edge researchers connected to the evolving priorities of stakeholders (donors, governments, development agencies, NARS, NGOs, communities), who also tend to be more deeply engaged in uptake of results in which they directly invest, and it enables some portfolio reallocation as needs shifts, performance varies, etc. Too much restricted funding, particularly when it has a short time horizon, however, shaves scientists' efforts too thinly, depriving them of the time needed to engage in more thoughtful, creative and rigorous work, to write up results at the level of detail and rigor required to achieve real impact from research, to think outside the boxes of their immediate projects and programs, and to synthesize their and others' findings into a larger vision. Excessive restricted funding also creates considerable job insecurity that drives out the most talented scientists and prevents recruitment of the most promising. Those who remain get no sabbaticals. They do not have opportunities to regularly attend major professional association conferences to keep abreast of the most recent scientific advances. There are no institutional incentives to invest in maintaining, much less building, skilled human capital in the form of technical research skills. IRS social scientists' time is heavily fragmented across multiple projects, with restricted projects necessarily requiring far greater administration, meetings, reporting, and accounting for time and resources. The pursuit of restricted funding also induces excessive inter-Center competition and directly unproductive rent-seeking behavior, and leads to short-term "partnerships of convenience", a term the panel heard repeatedly. 25 An ARI university professor in the United States typically has nine months' time covered by core funding that places multiple demands on her time - research, teaching, outreach, service - but with minimal ancillary administrative demands. In Canada and Europe, core funding of professors' salary generally extends for 12 months. CGIAR senior social scientists are now down to just 3-4 months covered by core funds (and effectively much less once administrative responsibilities and annual leave are factored in); that's far too little. The panel considers 50% coverage of scientist time on unrestricted funds an absolute minimum, with 60-80% staff coverage on unrestricted resources the optimal range for CGIAR IRS. The hazards of heavy and growing dependence on restricted funding, much of it short-term and small-scale, are reflected in myriad ways we describe in the remainder of this section. 4.3 Staffing The main asset of CGIAR social science is its skilled human resources. Social science research is far more labor intensive than biophysical science, making human resources management higher stakes in the social sciences than elsewhere in the CGIAR. The System must recruit, motivate, maintain the skills of, and retain outstanding social scientists able to engage both the development practitioner and global research communities, and in sufficient numbers to cover the broadening research mission of the System. While expenditures on social scientists have grown rapidly, as just demonstrated, there are troubling signs elsewhere with respect to social science staffing in the CGIAR. There seems to have been quite rapid growth in social science staff. Social scientists represented 26.7% (310/1163) of all CGIAR internationally recruited staff (IRS) in mid-2008. By way of comparison, TAC (1996) indicates there were 163 social scientists out of 937 IRS in 1995, equal to 17% of the total. While IRS grew overall by about 17% overall over those dozen years, social science IRS nearly doubled.2 The highly imperfect comparability of different data series makes strict comparison problematic. Nonetheless, the CGIAR Gender and Diversity (G&D) survey staffing data3 suggest that 18.5% of all IRS were social scientists in 2003, implying that most of the growth in social science staffing has come in the past five years. Over the 2003-2008 period, the G&D survey data estimate a 33.0% increase in total social science staffing, ten times the 3.3% estimated increase in non-social scientists. By all accounts and available data, social science appears to be growing within the CGIAR. Table 4.2 Center-reported social science staff numbers, 2007 Staff Category Ph.D. Masters Other Total Internationally Recruited 275 33 2 310 Nationally Recruited 15 96 39 150 All Staff 290 129 41 460 2 It is not clear how strictly comparable these numbers are, for example, whether the 1995 figure uses full-time equivalents rather than a pure headcount and how redefinition of nationally recruited staff (NRS) affects the figures. 3 These and the gender data reported later in this subsection come from the CGIAR Gender and Diversity (G&D) program staff survey. Hence the subtle discrepancy in some figures relative to those reported elsewhere using other data. 26 Of the 310 IRS engaged in social science research in the System, 89% hold a Ph.D.,4 as do 10% of nationally recruited staff (NRS) social scientists (Table 4.2). This is comparable to the 91% of non-social science IRS with Ph.D.s.,5 although the social scientists are significantly more junior than their non-social science colleagues, with more than one-third within 8 years of earning their last degree, versus just 21% for non-social scientists. Social scientists' longevity at Centers (as measured by years since joining their current Center) is, however, indistinguishable from that of non-social scientists; the two groups move at nearly identical rates. In 2008, 33% of CGIAR social scientists were women, versus only 22% in the non-social sciences. Both figures were up significantly from 2003, when only 24% (16%) of CGIAR social scientists (non-social scientists) were women. Although fewer female social scientists have a Ph.D. than do males, 74% versus 87%, the gap closed appreciably between 2003 and 2008, from 22% (86% for men less 64% for women) to 13%. Over the same period, the share of social scientists in senior positions - Senior Scientist, Principal Scientist, Directors or beyond - who were women also increased from 19% to 29%. According to female scientists in or with significant experience in the System with whom we spoke, working conditions for women have improved, especially in terms of accommodating parental leave, dual career management, telecommuting to accommodate family circumstances, etc. But there remain some signs of subtle sexism. Women scientists often report feeling less likely to be in the "in crowd" that gets early access to funding or co-authorship opportunities. And, as is true in many organizations, women appear more likely to get dragged into administrative service as management seeks "a gender perspective" on various matters. Economists constitute the largest share of social science staff by discipline (60%), followed by geography (7%) and anthropology (6%). Social science in the CGIAR is clearly overwhelmingly dominated by economists, as has always been the case. Non-economists commonly report finding Centers difficult places to work, in part because of the dearth of colleagues with whom they can consult on disciplinary questions. Lack of critical mass is a serious issue within the non­ economics social sciences. Several non-economists commented to us that economist colleagues try to engage them, that the greater problem was that biophysical scientists often have a hard time understanding how to engage social scientists who employ primarily qualitative methods and simply ask for an economist to help, whether or not an economist's toolkit is the most appropriate one.6 Indeed, the panel was struck by the frequency with which economists are leading CGIAR social research for which other disciplines' toolkits appear better suited - e.g., in networks analysis, participatory research prioritization, spatial analysis. The need for greater non-economics social science is palpable at the Centers. How best to recruit, retain and deploy non-economics social scientists remains much less clear and remains an important challenge. Another issue that arises from the disciplinary profile of CGIAR social scientists is that Centers listed a considerable number of IRS who do not have advanced degrees in a social science but have, instead, migrated into social science-related research post-doctorally. As one Center report put it, they have 'people who are trained as biophysical scientists but who have become adept at 4 This includes Juris Doctor as a terminal degree equivalent to the Ph.D. 5 This figure and subsequent comparisons with non-social scientists come from data compiled from research staff curriculum vitae submitted to the SC Secretariat for Centers' most recent External Program and Management Reviews. 6 While this appears the most common alignment, in a few Centers, conflicts between economists and non­ economics social scientists have been greatest, with each group working with the natural scientists reasonably effectively, albeit in parallel. The key point is that interdisciplinary engagement is valuable and attractive to most CGIAR social scientists but that it is often difficult, especially for the non-economists who lack a significant community of their own to fall back on when necessary. 27 a number of social science functions'. This is a surprisingly large cohort (8%) in the total pool of CGIAR IRS working on SSR; if they comprised a discipline, they would be second only to economists in number. While these personnel are sometimes among the most vocal champions of social science's role in CGIAR research - hence their migration - a large share have little substantive understanding of nor the methods and toolkits of social science, leading to research of at best uneven quality. Social scientists are not distributed evenly among Centers. IFPRI naturally employs the most, 86 IRS and another 46 NRS, accounting for 28% (29%) of CGIAR IRS (total) social scientists. Among the other Centers, only ILRI, CIAT, IWMI and IITA employ at least 20 IRS social scientists. Six Centers (Africa Rice, Bioversity, CIMMYT, CIP, ICARDA and ICRISAT) employ 16 or fewer total social scientists. Less than half (46%) of social science IRS are posted at Center headquarters, as compared to 57% of non-social science IRS. Given that Center social scientists are commonly dispersed across multiple regional offices and the small size of most social science units, concerns about lack of critical mass in social science are widespread at Centers other than IFPRI, in spite of the rapid growth in social science staffing in the System as a whole. By "critical mass" or "effective critical mass" as we sometimes term it below, the panel does not mean a simple head count. Rather, the concept relates to the time effectively available for scientists' substantive interaction around shared research interests. Critical mass is absent even where headcounts are satisfactory if (i) staff are physically separated by space, (ii) functionally separated with little time or incentive to interact across distinct projects, or (iii) excessively fragmented in individual and collective effort so that the small fractions of even several social scientists' time nevertheless adds up to very little. The concept is difficult to quantify in the data available. But at a qualitative level, the panel is convinced that in most Centers the social science staffing numbers are deceptive in suggesting effective critical mass exists. For the three reasons just enumerated, perhaps half the Centers almost surely lack effective critical mass in social science research, especially outside of economics. Based on discussions with current and former staff at multiple Centers, the panel found morale among CGIAR social scientists to be low, especially at the Centers with smaller social science staffs. This the natural consequence of fragmentation of social science research effort, and significant increases in bureaucratic demands on scientists' time while at the same time expecting greater scientific output and development impact. The "doing more with less" management culture that now pervades the CGIAR is a path that inevitably leads to shortcuts, lower quality work and less job satisfaction, such that CGIAR social science is, in many ways ironically, actually "doing less with more," as resources and staffing have increased yet research quality seems to have declined and the panel found no evidence of increased development impact. Several long-time observers of the System point to degraded staff morale as the biggest difference between CGIAR social science today and that of 20 years ago, remarking that there are adequate resources, but the terms under which those resources are available, the micro- management by upper management and donors, and the lack of strong, visionary, consistent leadership have sapped the creativity and energy of the System's social scientists. Low morale is both cause and consequence of increasing difficulty in recruiting top social scientists into the System. Several Centers have struggled for two years or more to find qualified candidates for mid-and higher-level social science positions. Even post-doc positions attract limited numbers of qualified applicants, at least outside of Washington, DC-based positions with IFPRI. Location, spousal employment, children's education, health care for special needs, and other considerations make it difficult to attract mid-career and senior staff, as is true of all 28 research organizations, although the problems are certainly worse in some venues plagued by violence or very strict labor laws. While dual career concerns have certainly become a greater concern over the past generation, this is true for all social science employers, including many non-CGIAR institutions not located in major metropolitan areas in OECD countries. Most of the other challenges are not new; locational challenges have bedeviled CGIAR recruitment over the years. Yet recruitment has clearly become appreciably more difficult, seemingly more because the quality of the work and the work environment have deteriorated over time than due to factors exogenous to the System. Among NARS and ARI survey respondents, 64% and 72%, respectively, rated the CGIAR at the midpoint or worse of the Likert scale (i.e., good, fair or poor) on "attractiveness as a social science employer". Meanwhile, Centers are losing many of their most productive mid-career social scientists. They leave not only for ARIs, but to multilateral organizations, bilateral donors, and even national research institutes and NGOs. Many of the problems plaguing recruitment apply to retention issues as well. These are compounded by the widespread absence of retraining opportunities for updating skills, which are magnified by decreasing substantive interactions with ARI colleagues who might help mid-career scientists learn new theories and methods. But CGIAR social scientists face a few distinctive pressures that hurt retention. Unlike for biophysical scientists, there is no clear research career path for social scientists in the CGIAR today. IRS staff must assume significant project or program management and fundraising responsibilities within 6-8 years after completing their Ph.D. and by about 15 years post-Ph.D., they typically need to move into management or leave the System. There are very few opportunities for anything like the career research positions available in ARIs or national research institutions. And with virtually no sabbatical opportunities and scant core funds for travel to professional conferences and other venues for short-term retraining and skills acquisition, established social scientists suffer skills degradation at a faster rate than they should. The confluence of recruitment and retention problems and heavy dependence on restricted funding for development-oriented projects is prompting a discernible shift in social science staffing. The CGIAR has always attracted social scientists who want to "make a difference" through their research, but more recent hires do not seem to fully grasp the "development through research" essence of the CGIAR mission. Newer hires strike the panel as more likely than long-time System hands to speak of a "development mission," routinely asserting that "our mandate is to lift people out of poverty," while noticeably omitting the crucial "through research" element of the CGIAR mission statement. They are also far more likely to dismiss the IPG focus of the CGIAR as irrelevant, undesirable, or impossible - often even questioning the meaning of IPGs. These patterns strike the panel as indicative of a sharp drift downstream into development practice and away from one of the three central characteristics that is the source of CGIAR comparative advantage over other development actors (especially NARS, NGOs and private firms): its skilled scientific staff and their capacity to do quality research when funding, expectations, incentives and organization are aligned toward achieving impact through research. As the CGIAR's social science staffing mix shifts, it reinforces System mission drift. As a result of these factors, the actual conduct of CGIAR social science - detailed research design, data collection and analysis, modeling and theory development, write-up and publication of results, etc. - falls disproportionately on post-docs and NRS with Master's degrees. The post docs are widely discouraged by the lack of research mentoring in the System, by the lack of time available to employ the more advanced skills they acquired in their doctoral training or even to 29 write. One post-doc we interviewed was using vacation time to write up research results; another was taking leave without pay to do the same!7 The contrast with the Rocky Doc experience is stark. From 1974-96, the Rockefeller Foundation (RF) placed 114 carefully screened and matched post-docs from Africa and North America in CGIAR and associate Centers for two-year assignments.8 This "Rocky Doc" program greatly enhanced not only the career development of these exceptional young scholars, it also helped Centers incorporate the best new theory and methods into their research programs, to enhance outreach to NARS and to farmers, and to deepen connections to leading ARI social science units. The roughly two dozen Rocky Docs the panel interviewed uniformly found their experience highly beneficial to their development as scholars; we do not sense the current crop of CGIAR post docs will offer similar assessments a decade or so from now. When the Rocky Doc program ended, steady recruitment of high quality young social scientists into the System largely ended as well, with the partial exception of IFPRI. Meanwhile, at some Centers nationally recruited social science staff operate on markedly lower pay scales than internationally recruited Ph.D.s, which both sows resentment within teams and creates retention problems as the best performing Master's holders leave Centers to pursue a Ph.D. in the hope of returning to the System later on better terms. Centers lose the experiential expertise such staff have accumulated, often permanently as many of these staff go on to other organizations once out of the CGIAR fold. Overall, human resources management policies in many Centers strike the panel as somewhat out of date. They are commonly very slow to post and refill key positions, they fail to keep good tabs on compensation packages in competitor institutions, and there remain too-frequent allegations of unprofessional behavior by managers, etc. We were especially surprised at how poorly the CGIAR understands the international labor market for Ph.D. social scientists. Variation in compensation packages - both in terms of total value and the portfolio of salary and benefits on offer - is high across Centers. Even net of (relatively low) taxes and inclusive of (relatively generous) benefits, entry level compensations packages within the CGIAR appear uniformly low (Hewitt Associates 2005, Marenya 2008). Combined with the lack of cachet once conferred by the Rocky Doc affiliation, this may help explain recruitment problems at the post­ doc and junior scientist levels. But as social scientists progress through the ranks, compensation packages commonly catch up with and surpass those offered by competitor employers, such that senior level managers are typically well compensated. Although the ranges across and within Centers appear considerable, uncompetitive compensation does not appear to explain recruitment and retention difficulties at mid-career and beyond. It is an issue for more junior recruitment, however, and since this is the "feeder pool" for future mid-career and senior social scientists in the System, compensation at lower ranks merits close attention. The panel's sense is that high staff quality and leadership capacity are more significant issues than critical mass at System level. CGIAR social science has expanded rapidly, but mainly by inadvertently trading off quality for quantity as its new business model has made retention of 7 More senior IRS frequently do this as well. 8 Conway et al. (2006) describe the Rocky Doc program in detail. Insiders tell us that the RF terminated the program for a variety of reasons, including the Foundation's perception that Centers were increasingly cutting corners in providing basic logistical and research support to Rocky Docs. In spite of universal declarations of appreciation for the Rocky Doc program, Centers did not always back up their enthusiastic rhetorical support with real resources. 30 skills and the best performers more difficult and recruitment of talented new blood highly uneven across Centers. While female scientists have made real progress, non-economists continue to struggle to establish an effective collective presence with CGIAR social science. Meanwhile, social science is heavily concentrated at IFPRI, with insufficient and often strained relations with other Centers' social scientists (on which, more in section 4.5 below), while human resource policies and junior staff compensation have grown outdated. 4.4 The Pursuit and Measurement of Impact Given the CGIAR mission, Centers and CPs indisputably need to focus on research that contributes to real development impact through significant new scientific discoveries. Historically, CGIAR research has generated substantial impact in terms of higher crop yields, greater efficiency in input (e.g., seed, water, labor, land) use, increased farmer and farm worker incomes, lower real food prices and improved nutritional content for consumers, etc. To cite but one important impact metric, Alston et al. (2000), Gardner (2003) and Raitzer and Kelley (2008) report on the mass of findings of consistently very high rates of return on CGIAR crop genetic improvement research, typically in excess of a 50% internal rate of return annually. Social science research's contribution to the high historical impact of CGIAR research has been universally accepted, at least in its traditional research focus on agricultural productivity growth through germplasm improvement, even though it is difficult to establish precisely due to methodological problems.9 One such problem is how to separate social science research from other research given that most agricultural productivity and NRM research involves close multidisciplinary collaboration. Just as disciplinary attribution is difficult, so is organizational attribution to the CGIAR, given that System social scientists can effect negligible change on their own. Centers typically work with partners carefully chosen to increase the likelihood of impact. More fundamentally, change comes through the actions of farmers, traders, workers, community organizers, resource managers, policymakers and others, whose behaviors might be affected by CGIAR research among many other influences; but it is intrinsically difficult-to-impossible to establish by how much. Furthermore, research projects and programs are (appropriately) not designed and applied in explicitly random fashion, i.e., they target areas where impact is perceived ex ante as attainable. As a result, it is exceedingly difficult to establish appropriate counter-factuals and proper control for confounding factors is difficult in the best of cases. And as CGIAR social science research has branched out beyond the support of agricultural productivity increases into policy-oriented, NRM and markets and institutions research, these methodological challenges have increased exponentially, especially since many intended benefits from such research - improved environmental indicators, enhanced security and status of women, more equitable distribution of the gains from policies and technologies, etc. - are difficult to value quantitatively (Adato and Meinzen-Dick 2007, Waibel and Zilberman 2007, CGIAR Science Council 2008b). Hard as it may be to measure, generating sustainable impact is central to CGIAR goals. Therefore, impact must be anticipated and planned for by establishing clear impact pathways and by engaging in ex ante impact assessment to inform research prioritization of the investments most likely to yield the greatest desired impacts per dollar invested. The panel perceives that much of the impact of CGIAR social science has historically been achieved 9 Renkow and Byerlee (2009) offer one recent, thoughtful review of impact evaluation research in the CGIAR. 31 indirectly, through its effect on broader research prioritization and midstream adjustment of the conduct of research - especially within Centers and the CGIAR System, but even on the broader global agricultural research agenda - and on global social science that cumulatively shifts thinking and policy through a mass of evidence contributed by researchers in many different institutions. Direct, ex post development impact can and must equally be measured carefully using rigorous methods. That way the System can establish what investments yield dividends and which ones do not. In principle, those findings should feed back into research prioritization at all levels, from the System down through individual research projects and programs. The comparison of measured ex post impact against ex ante predictions is needed for scientific learning to steadily improve research prioritization. Furthermore, good impact assessment equips donor agencies in the political competition for resources on which the CGIAR depends. The CGIAR therefore takes impact assessment seriously, and appropriately so.10 SPIA has played a central role in trying to promote more rigorous EPIA to establish "whether and to what extent agricultural research is an effective instrument for achieving poverty alleviation and other development goals" (Kelley et al. 2008, p. 202). But the nature of impact from research nonetheless remains too often misunderstood by System stakeholders and reveals a mismatch akin to that between the staffing pattern and the new business model. There is inherent stochasticity in scientific discovery; major breakthroughs occur with only limited predictability and over long periods. It takes deliberate, sustained and focused effort, and breakthroughs often arise serendipitously and on a schedule beyond the researchers' control. However, current performance evaluation in the System, including much EPIA, assesses CGIAR research as if it were producing a manufactured commodity that should have highly predictable patterns of output and impact that are discernible in just a very few years. But research is more like exploration. Many years sometimes go by with no major new discovery other than knowing where not to look; "failures" are an important and inevitable part of the process that must be acknowledged. The incentives in the CGIAR under the new business model, however, fall squarely against acknowledging any failures. Too many donors insist on short-term, concrete and measurable development outcomes and impacts to take back to their bureaucratic and legislative masters. This leads to a wasteful search for "magic bullet" solutions that rarely exist, to overinvestment in low-return activities that guarantee some near-term successes but at high opportunity cost (in terms of longer-term, higher-expected-impact activities foregone) and enormous pressures for Centers and CPs to move downstream to fill the void left by weak NARS. Strengthening NARS (and their regional and subregional agricultural research networks) is a worthwhile objective for international donors to pursue. But the CGIAR cannot begin to solve the problem of weak NARS, which result principally from national level political economy issues beyond the influence of any Center. And as Centers move downstream out of frustration for insufficient near-term development impact, they increasingly distance themselves from the scientific capacity that enables that impact. The distortion of incentives by the laudable pursuit of impact - and empirical validation of that impact - weighs heavily on CGIAR social science. In most Centers, one or two evaluation specialists - typically with graduate economics training - are tasked with meeting ever-growing demands for impact assessment. But these staff lack the 10 The System has a benchmark standard that 1.25% of a Center's total budget should be spent on impact assessment. This seems relatively modest to the panel. 32 time and professional networks (and often the technical skills) to perform high quality assessments using state-of-the-art methods on the large number of projects and programs they are tasked to evaluate. The intent of rigorous EPIA is to credibly identify effects causally attributable to the research being evaluated. Relatively few of the impact assessment studies read by or described to the panel employed designs with sufficient rigor to establish such causality convincingly. The result is a burgeoning mass of small-scale, one-off evaluations of remarkably heterogeneous - and low average - quality. This undermines the intent of holding the System accountable to demonstrate impact and of learning what lines of research really deliver on their promise. The panel suspects that this ultimately undersells the value of CGIAR research. A focus on impact by itself cannot lead to rigor in identifying causation; it requires enabling conditions that lead to greater rigor, not just in EPIA but in all social science research. Broader problems in the System lead to perverse incentives that favor a large mass of low quality EPIA studies. The panel found widespread agreement that focus on EPIA has led to an emphasis on quantity over quality, while rigorous ex ante impact assessment has been underemphasized by Centers and CPs. Traditionally, a central form of CGIAR social science impact came through agenda­ setting internally via research prioritization and mid-stream adjustment in response to social science monitoring of innovations. Systematic modeling of expected returns on Center investment in different lines of research and input on policy issues related to research ethics, intellectual property, etc. has typically been of very high impact, although usually internal to Centers or the System and thus often neglected and undervalued by outsiders. This capacity has fallen markedly in most Centers, to the point that much strategic research prioritization has been ceded to donors, who typically lack the necessary technical expertise to make careful assessments and thus are especially vulnerable to political and intellectual fads. Indeed, the panel found it revealing in reviewing the social science impact statements filed by each Center that impact on Center, System or global research prioritization was conspicuously scarce. The focus was almost entirely on tangible, on-the-ground, near-term development achievements, many of them quite incremental. And even then, the panel found the vast majority of the impact statements insufficient to arrive at or provide any meaningful independent assessment of impact. There were many claims on a wide variety of indicators, but scant evidence. That might be because these were short, informal statements, or because there is little hard evidence. The panel simply could not tell. In reviewing project documents, the panel found claims of development impacts that range widely in scope, importance and credibility. As with the impact statements prepared for the SSSR, the evidence of impact was typically thin and imprecise. End-of-project reports commonly attributed limited impacts to insufficient budget, although it struck the panel that the real problem was more commonly overly ambitious project scope, diffuse and unrealistic objectives, and/or an overemphasis on development outcomes over research. One Center remarked in an email, "The programme brought together a diversity of partners to address a multitude of issues touching farmer's [sic] livelihood. Resources and time are limited to fully cover all the problems." This might be said for many CGIAR social science projects aimed primarily at downstream impacts. The credibility of most CGIAR EPIA is low, partly because the methods remain underdeveloped in the broader literature. The resulting evidence is meager and often unconvincing, partly because of conflict of interest problems inherent to having any organization assess its own 33 impact. As a result, little EPIA work seems to influence Center management or donor decisions. We asked multiple persons to cite examples of impact assessments that subsequently changed strategies; most respondents dodged the question, while a few frankly acknowledged that they could not cite an example. In the absence of rigorous ex ante research prioritization research that internalizes the results of rigorous ex post evaluations, investments are driven more by the drive for Center resource growth and donors' immediate programmatic needs than by impact studies. SPIA has attempted to remedy some of the weaknesses of CGIAR EPIA analysis, by pushing for greater rigor, a focus on program rather than project evaluation, an adequate lag - typically ten or more years - between program completion and EPIA, and by involving external experts to provide quality control and technical guidance. SPIA seems to have helped foster more synthesis and methodological expansion in EPIA studies by Centers..11 The panel applauds this, not the least because of the significant problems associated with near-term impact assessment.12 The search for near-term impacts typically understates ultimate impacts, especially because indirect impacts (e.g., on consumers or workers via induced price and wage effects, respectively) typically take longer to materialize than direct effects (e.g., on technology adopters), and mismeasures the distribution of those impacts among distinct subpopulations, due to intertemporal variation in group exposure and differential dependence on indirect impacts (King and Behrman 2009). Since most CGIAR social science research aims at long-term improvements in productivity, sustainability and equity, the near-term evaluation culture that the new business model has spawned seems biased towards finding little or no effect regardless of the true, ultimate impacts. SPIA is sometimes criticized as being dominated by economists and remaining wedded to strictly economic approaches. Indeed, it has struggled to advance multi-dimensional impact measures that account for non-monetary benefits related to environmental and social phenomena. But such advances have not been forthcoming from ARIs or other research systems either; this is a difficult, but high return area in which the CGIAR has a natural comparative advantage. SPIA has sponsored recent efforts in this direction, on poverty and livelihoods (Adato and Meinzen-Dick 2007), on NRM research (Waibel and Zilberman 2007) and on policy- oriented research (CGIAR Science Council 2008b). The problem is that SPIA fundamentally depends on Centers' own efforts to undertake EPIA. It can offer only very modest funding (typically just US$15,000-25,000) and the attraction of limited interaction with a few expert facilitators/ reviewers to undertake specialized impact assessment studies. This is grossly insufficient to make necessary, major methodological advances in the area and to generate the rigorous, objective impact evaluation evidence the System needs. In the face of massive pressure from donors and Center management to demonstrate near-term development impact, the perverse incentives for low quality EPIA have largely trumped SPIA and other's efforts at trying to upgrade EPIA within the CGIAR. A community of critics of technology-centered models of innovation and of SPIA-style impact assessment has emerged within the CGIAR over the past decade or so, loosely organized around the "innovation systems" (IS) perspective and drawn together somewhat under the Institutional 11 Kelley et al. (2008) describe SPIA's recent efforts and findings. SPIA recently convened a meeting with Centers, CPs and donors to attempt to define and refine best practices in ex-post impact assessment (CGIAR Science Council 2009). 12 For example, recent research by the World Bank finds that impact evaluation often takes place too soon after an intervention to measure accurately the true net impact of the program. 34 Learning and Change (ILAC) initiative, hosted by Bioversity. The ILAC/IS community emphasizes that Centers have taken on a radically expanded research agenda, adding environmental sustainability and a far more explicit emphasis on poverty reduction to the System's traditional agricultural productivity improvement goal, thereby broadening the range of innovations and impacts required of the CGIAR. This community rejects the traditional, simplistic neoclassical economic idea that exogenous technological change drives social and economic development, instead seeing technological change and learning more generally as the endogenous outcome of institutional arrangements that are the key drivers of development. Of course, this proposition has significant support even within the neoclassical economics community today,13 and echoes insights made long ago by CGIAR social science pioneers such as Ruttan and Binswanger about induced institutional and technological innovation. So the fundamental premises of the ILAC/IS community are far less contentious than the tone of the intra-CGIAR debate on this topic might suggest. As several senior scientists observed to the panel, many productivity improvements associated with CGIAR research over the years have, without expressly using the label, followed an innovation systems approach, integrating research with mentoring, partnerships and a focus on dynamic learning through interactions among all stakeholders. There is certainly much value in updating those within the System who still cling to antiquated notions of single-minded technology-push approaches to agricultural development. But such perspectives appear uncommon in the CGIAR today. The core ideas underpinning ILAC/IS are neither new nor highly contentious. The importance of accountability of the agricultural research system to farmers and other research users dates to the very first social scientist hired into the CGIAR, Vernon Ruttan, the potential value of participatory approaches has been widely appreciated at least since the seminal writings of Robert Chambers in the 1980s, and the importance of institutional design to path dynamics has long been a central result of game theory. Indeed, the innovation systems literature originated in the study of manufacturing processes in industrialized countries in the 1980s and is being adapted now to developing country agriculture. Innovation systems perspectives can offer some attractive insights into processes of institutional learning and change and into feasible pathways for achieving the sorts of poverty impact the CGIAR seeks. But IS theory remains underdeveloped and exceedingly difficult to operationalize empirically. This is the main reason why the field has declined in its original applications in manufacturing in the industrialized world. As one of the more thoughtful CGIAR contributors to this literature writes, "there is little evidence to suggest that an innovation systems framework approach to developing-country agriculture is, in fact, providing real solutions to today's challenges. While the framework is helping to change the mind-set of researchers and policymakers by encouraging them to consider new and unconventional actors and relationships, several methodological and analytical shortcomings are limiting its relevance to policy and policymaking processes. Its effect on social welfare improvement in developing countries is therefore limited" (Spielman 2006, p.42). The panel's impression is that the concerns of the ILAC/IS community are important and valid, but the System lacks highly skilled social scientists with appropriate training - e.g., in communications, organizational behavior, social psychology and other disciplines entirely unrepresented in CGIAR IRS staffing - needed to make major methodological advances and 13 See, for example, the excellent recent text by Wydick (2008) which is built around this thesis. 35 empirical discoveries in an area that, following a flurry of initial interest, has largely fallen out of favor within the mainstream research community. The Sub-Saharan Africa CP promised to be an important crucible for testing key ILAC/IS propositions through its integrated agricultural research for development (IAR4D) approach. But the CP's quite limited successes to date with IAR4D have failed to build confidence in the CGIAR's capacity to develop useful new insights and practices based around ILAC/IS. Innovation systems shows more promise as an implementation tool than as a research instrument. Partnerships with some of the world's leading thinkers in this area have helped generate some intriguing, if still somewhat vague, propositions as to key attributes of research projects likely to lead to development impact (Kristjanson et al. 2009). And there are certainly researchable questions within the IS domain that could generate important IPGs. For example, what are the costs and benefits of different partnership models in relation to ultimate impact? Or, what are the most effective stimuli for different classes of change agents under different conditions? But while the panel finds merit to the innovation systems perspective on the process of discovery, we see only a very limited role for this line of research within CGIAR social science while the concepts and methods remain seriously underdeveloped and the CGIAR lacks appropriately trained staff to enjoy a high likelihood of generating breakthroughs in an area that has had limited uptake elsewhere and has largely stalled within its original application domain. Overall, the CGIAR's new business model has led to dramatic expansion of the quantity of EPIA undertaken. But most of it remains of low quality, unconvincing even to many of those who commission or generate the estimates. Meanwhile, rigorous ex ante impact assessment for research prioritization has declined precipitously. And substantial resources have been devoted in recent years to innovation systems approaches that have yet to deliver tangible gains and in which the CGIAR holds no comparative advantage for significant conceptual or methodological advances. Given the inherently stochastic nature of research discovery, the panel favors a significant reorientation of impact assessment effort towards restored ex ante impact assessment capacity for research prioritization and far more selective, larger-scale, long-term EPIA based on methodological innovations that the CGIAR should be able to generate if it makes a serious investment in this domain. Furthermore, CGIAR EPIA should more actively involve external researchers so as to avoid the perceived conflict of interest problems inherent to reliance on claims made by staff involved in the projects being evaluated. What an organization measures carefully is what it treats as important. The questionable quality of impact assessment in the CGIAR today raises questions about how seriously the System really values research that generates scientific discoveries that lead to major and sustainable impact. 4.5 Organization and Partnerships Many of the challenges CGIAR social scientists face arise from institutional rigidities associated with Center and System structures (e.g., SWEPs, CPs) designed primarily for biophysical scientific objectives. The overall result is a bricolage wherein new structures add onto rather than resolve organizational design problems. In the process, new structures have commonly diverted still more time of social scientists away from research and into administration. This is a major concern with respect to the current Change Management Initiative (CMI) and the Mega Programs (MPs) that appear poised to become the centerpieces of unrestricted and quasi- unrestricted (i.e., long-term, large-scale restricted) CGIAR funding moving forward. The panel 36 strongly believes systemic governance and financing change is essential if the System is to restore quality and impact. If they are to succeed in that aim, MPs must significantly change research perspectives - towards longer-term, IPG-oriented activities focused tightly in areas of comparative advantage - and incentives - away from chasing marginal grant dollars and low- impact publications. Otherwise, MPs will likely just add new layers of bureaucracy and further expansion of mission without substantially shifting incentives and drawing sharper boundaries around the scope and funding of CGIAR social science, thereby seriously compounding the growth and quality dilution problems that confront the System now. It remains unclear to the panel - and to the vast majority of the CGIAR scientists and managers with whom we spoke - as to what CMI will ultimately mean for the funding and organization of CGIAR research. This concern is especially acute for the social sciences because at a fundamental level the CGIAR is not organized to generate big successes in the social sciences - its direct focus is on the agricultural sciences and natural resources, not on social processes. CGIAR social science has nonetheless achieved big successes, largely when it has capitalized on close interactions with the biophysical sciences and succeeded in cultivating or attracting true international stars, either onto Center staff or into substantial long-term collaborations. Sizeable development impacts came from having the leading global minds working with and within the System, focused on a few key issues on which the CGIAR held global comparative (indeed, absolute) advantage. ARI stars pushed CGIAR stars and vice versa, leading to very high quality, high impact work. The highest quality work in CGIAR social science still follows that model. But it is far less frequent than it once was, in part because the System's new business model is less well suited to producing or attracting stars, in part because of diffusion of effort across Centers and topics, in part due to changed partnership models that are less effective in generating high impact social science. One problem with the present organization of CGIAR social science is widespread lack of effective critical mass, at least outside of Washington and perhaps one or two other locations. The issue is not just insufficient numbers of social scientists physically co-located. It is equally too little time for substantive, sustained interaction among social scientists whose time is excessively fragmented across multiple restricted funding projects. CGIAR social science is characterized by far too many small and disconnected groups. The result is loss of essential technical capacity and failure to employ individuals according to comparative advantage; every IRS must become a Jack-or-Jill-of-all-trades. Social science units do not need to be big. It is sufficient to have even just 4-6 people with active involvement in international research networks and the time to interact spontaneously within their group. But staying in close contact with one's disciplinary colleagues is essential to maintain morale and technical capacity. Several Centers' experience of disbanding social science units and dispersing social scientists in ones and twos across thematic units and regional offices has proved catastrophic. Nevertheless, other Centers appear insistent on replicating this error. The panel strongly urges the restoration and protection of social science units, albeit integrated with thematic or ecoregional units through matrix management designs. Matrix management is difficult, as all recognize. But rather like Winston Churchill's observation that democracy is the worst of all possible forms of government, except for all the alternatives, matrix management is the best option available for cross-cutting research enterprises like the CGIAR in spite of its many, well-known imperfections. Strictly disciplinary units lead to silos that are relatively good at producing output for disciplinary audiences. But they are excessively disconnected from other disciplines and therefore too often have limited development impact. 37 They are also quite vulnerable when funding grows tight in a Center and management needs discrete units to be cut. On the other hand, thematic units advance the problem-oriented, interdisciplinary mission, but have a strong tendency to get pulled downstream and to fail to maintain technical skills and rigor in research as the work devolves to lowest common denominators across disciplines. While there is no one-size-fits-all approach appropriate to CGIAR social science research, the panel sees matrix management, with social scientists appointed in a disciplinary unit and affiliated with (or even bought up by) thematic or regional units as the default arrangement of choice, With the advent of Mega-Programs, matrix management will almost surely become the primary arrangement for many social scientists, many of whom we expect will have a Center home and a Mega-Program appointment. The second big problem with the organization of CGIAR social science is the general lack of boundaries such that even where critical mass seems to exist, it becomes excessively diluted across too large and diverse a portfolio of work. The panel asked scores of CGIAR social scientists to identify activities currently undertaken that fall outside their comparative advantage and that they therefore should not be doing. Astonishingly, not a single respondent identified any such activity. They see no boundaries. Center staff matter-of-factly reported spending significant time on such activities as preparing Powerpoint presentations to be given by donor agency program officers or assisting sub-national, state-level officials in the design of rural development or NRM policies. We found qualitatively similar examples in virtually every Center. To the panel, this is a clear sign of a revenue-driven enterprise rather than mission- and quality-driven organization. A striking number of projects the panel reviewed or was told about respond to requests to assist sub-national entities with specific agricultural or NRM issues, with little evidence of a careful research design that would ensure the generalizability of any findings such that the research might generate an IPG. These are straight development projects in which the CGIAR holds no comparative advantage. Centers frequently chase new donor funding opportunities of this sort, hiring new staff to pursue and complete the project. Then they need to pursue more such projects to keep those staff on payroll and on projects they can handle, thereby locking in the financial opportunism of the initial undisciplined pursuit of restricted funding in the absence of serious strategic thinking about comparative advantage and disciplined respect for the boundaries implied by such strategic assessment. The overly broad agenda pursued by CGIAR social scientists dilutes quality and impact by fragmenting attention and resources and overstressing key scientists, Time is the scarcest resource for CGIAR social scientists. The fixed search and transactions costs associated with each project consume an ever-greater share of aggregate time invested as the number of projects grows and their size and longevity shrinks. Center social scientists repeatedly indicate that they are responsible for working on anywhere between 4 and 12 different research projects simultaneously. Such fragmentation clearly signals insufficient focus. The third organizational problem concerns the changed nature of Center partnerships. Effective partnerships with external organizations and social scientists has always been central to CGIAR science, both because of its modest scale in the broader agricultural development research universe and because the CGIAR is positioned midstream, between basic researchers located primarily in ARIs and development practitioners located primarily in local and national agencies. The number of partnerships in which CGIAR social scientists are engaged appears by all accounts to have increased, although no hard data exist to test this hypothesis. Increasing partnerships appears desirable in the abstract, but establishing and maintaining relationships has a cost.< Many of these partnerships impose a high opportunity cost - primarily 38 in terms of IRS scientist time - with low rewards in terms of added research outputs or impact. Too often, interactions with other organizations, especially local and national institutions, are treated as instrumentally valuable by Centers because donors naively think of them as intrinsically important, without regard to whether these partnerships multiply or degrade the overall impact of CGIAR science. And far too often all scientists are expected to join in the various stakeholder events, regardless of their aptitude for such interactions. This merely reflects the widespread absence of attention paid to issues of comparative advantage, in this case at individual level. A "work with everyone on everything" approach is far too commonplace and inimical to achieving impact. The increase in CGIAR social science partnerships has been directed downstream. NARS widely view these as transactional "partnerships of convenience," as several NARS and former IRS social scientists labeled them, and only rarely as substantive, deep partnerships. Thus even though downstream partnerships have grown in number, the depth and durability of those partnerships appears to have suffered relative to past relationships. A larger share of those partnerships are directed at local NGOs, perhaps especially in NRM research, More partnerships of less depth and longevity is the natural consequence of increased focus on generating partnerships as an objective unto itself. An important implication of increased reliance on restricted funding is - as the recent Independent External Review of the CGIAR observed - increased competition with NARS for resources. NARS also commonly report that the quality of CGIAR capacity building in social science is lower than it was a decade or two ago. This is consistent with the System's very weak data on training and capacity building activities that signal the lack of systematic attention and emphasis on this activity. The problem is not enthusiasm for capacity building. Rather, IRS social scientists lack the core-funded time needed to supervise M.S. and Ph.D. thesis fieldwork and data analysis the way they did in years past. Historically, the best capacity building took place in the context of long-term, core-funded research programs with a significant, sustained field presence. Embedded students and NARS collaborators acquired valuable skills through close collaboration on extended research projects and from focused short courses and workshops to disseminate findings. With the transition in funding models, capacity building has suffered. A few highly dedicated individual CGIAR social scientists make time for training students and NARS colleagues in spite of the institutional incentives aligned against substantive time investments in capacity building. But they are exceptions to the rule. The best students tend to use Centers merely as logistical hubs - rather than as intellectual hosts and collaborators - while weaker students struggle for want of adequate guidance and supervision within the System as Center scientists juggle the excessive demands of too many short-term projects. Meanwhile, the CGIAR has launched an open distance learning education program, the Agricultural Open Curriculum and Learning Initiative (AGROCURI), formerly known as the Global Open Food and Agriculture University program. This higher agricultural education initiative, directed from IFPRI, is a standalone training platform markedly different from the CGIAR's historically successful approach of training individuals within the context of a shared research program. Observers of this effort and the panel uniformly see it as clearly falling outside the System's domain of comparative advantage, as carrying a high opportunity cost of IRS scientist time that could be better directed toward CGIAR core competencies, and running significant risks of crowding out service provision by developing country universities. 39 Upstream partnerships have likewise suffered. Long-time CGIAR observers, current CGIAR social scientists and ARI social scientists universally bemoan the degradation over time of CGIAR-ARI partnerships. This has hurt the CGIAR more than ARIs, which have developed vibrant alternative pathways to field-based research and to influencing development policy. But ARI survey respondents overwhelmingly (81%) find CGIAR collaborations helpful or very helpful to their own research programs. The main incentives ARI social scientists report for their CGIAR collaborations are Centers' strengths in relevant natural sciences, in selected cases Centers' strengths in social sciences and the opportunity to engage Centers' development practitioner partners through the collaboration. The benefits flow both ways. ARI partners were a key source of the System's major successes in the social sciences in the 1970s and 1980s. And just as in years past, much of the current research impact of CGIAR social science comes from leveraging strong collaborators in ARIs. CGIAR scientists especially look to ARI collaborators for guidance in research design, in choosing the most appropriate methods, in more technical analysis, and in synthesis of results from disparate small-scale studies into larger IPGs. From the panel's review of Centers' best publications, it is apparent that a disproportionate share of the higher quality CGIAR social science research emerges from longer-term collaborations with leading ARI social scientists. But partnerships with ARIs, like those with downstream partners, have become largely short-term and transactional, funded by specific projects. There is shockingly little engagement with the international research community, as manifest in low rates of participation in high-level scientific research panels and networks, limited awareness of cutting-edge research trends and methods, scant participation at major professional association annual meetings, and modest levels of representation on the editorial boards of major journals. Scientific interactions outside of Centers typically take place within the narrow confines of funded projects, special-purpose thematic conferences, and Systemwide and Ecoregional Programs that facilitate inter-Center dialogue. Substantial, recurring interactions are rare outside the CGIAR network and project-based relationships. Several donors (e.g., CIDA, USAID) operate small linkage grants programs to connect Centers and ARIs. These seem underutilized and underfunded relative to their impact and potential. The CGIAR can be attractive to established academics yearning to connect better to real-world development activities and to those engaged in serious field research who need in situ help with local logistics, research clearances, introductions to local stakeholders, etc. Centers have a comparative advantage associated with in situ continuity and intensive engagement with development practitioners, whereas ARIs have comparative advantage in intensive engagement with innovators of methods and theories. Although many Centers speak of playing a "boundary spanning" role, bridging between development practitioners downstream and basic researchers upstream, in practice this seldom occurs effectively. As one highly respected, longtime CGIAR social scientist told us, "I never found a viable way to combine upstream and downstream work." Time is short and the transactions costs of multi-party engagement and coordination are high. There have been a few notable examples of success in this regard - examples include CIP's collaboration with Montana State University and Ecuador's national agricultural research institute and ILRI's Maasai work with University College London and Maasai communities in southern Kenya - but these remain exceptions, not the rule. 40 Inter-CGIAR partnerships likewise appear more limited than seems desirable. There are certainly productive collaborations, perhaps especially through the more effective Systemwide Programs (e.g., CAPRI). And the Collection Action networks established to link Centers, subregional networks and NARS in some regions are working to overcome inter-center barriers. But the institutional incentives and pressures on social scientists are almost entirely toward intra- Center partnerships. Social scientists bridge to colleagues in other Centers based on pre-existing personal relationships, a keen professional desire to connect with a colleague elsewhere with particular skills, or, most commonly, when a funding opportunity arises. A large share of inter- Center partnerships appears transactional in nature, just like the other partnerships. As documented below, co-authorship by social scientists in different Centers is far more rare than co-authorship with collaborators outside the CGIAR, a curious pattern given the wealth of Systemwide programs and similarity of Centers' missions. Of particular concern, social scientists in both IFPRI and other Centers freely and frequently report poor relations between these groups. Inter-Center competition for research funding is clearly part of the problem; heterogeneity of research cultures and quality standards seems another. In the commodity Centers, many scientists view IFPRI as irrelevant to their interdisciplinary work on productivity improvement and NRM and assert that it has done little to stimulate innovation, while many IFPRI social scientists express concern about the level of rigor of the social science practiced at the commodity Centers. The most significant consequence of the dearth of substantive, sustained inter-Center collaborations is the System's failure to achieve valuable integration and synthesis of research findings. The CGIAR should be able to constitute collaborative bodies to explore and summarize scientific issues central to issues of poverty, food security and protection of the natural environment, akin to the Intergovernmental Panel on Climate Change, but focused on agricultural and rural development issues. There has been some effort at doing this through Systemwide and Challenge Programs, and some success in cases such as Alternatives to Slash and Burn or Ffarvest Plus. But the CGIAR is generally failing to realize these opportunities due to insufficient inter-Center collaboration. There is no mechanism for defining coherent, large- scale, long-term research themes that cut across commodities and specific resources or agroecosystems and then pursuing and coming to closure on those themes, in the process building an indisputable evidence base to guide private behavior and public policy. The System is big enough to generate cumulative research that is at once scientifically rigorous and pathbreaking but equally leads to identifiable outcomes and impact on poverty, with clear delivery channels in partnership with both local and upstream organizations. There are many themes that matter to the CGIAR mission and that remain under-explored by academic researchers and the private sector, such as avoiding and escaping natural resource degradation poverty traps, developing and implementing effective biosafety systems for new agricultural technologies, or connecting small farmers to markets, especially in contexts characterized by low population densities and remote locations. There have been repeated efforts to use joint appointments of IRS social scientists to achieve inter-Center cooperation. Such experiments have almost uniformly failed. Differences in cultures, management objectives, performance metrics, etc. have tended to create conflicts that can be difficult even for experienced scientists to resolve. When a scientist is based in one Center, the host has a natural advantage over its co-sponsoring Center in commanding the attention of the scientist. Either one Center completely dominates the relationship, typically because the staff member is based at the one headquarters, or both Centers try to actively 41 manage the scientist, but have different expectations, incentives and cultures that make it impossible to serve both masters effectively. The final organizational problem the panel identified concerns Centers' failure to seize economies of scale in research support and to communicate the gains they have realized to System social scientists. Examples include the design and maintenance of data repositories and web sites, the use of electronic libraries, licensing of software and contracting for technical support in computer equipment, even shared staff in statistical support. There is progress in some of these areas, but typically between small sets of Centers rather than Systemwide. We were surprised, for example, at the frequency with which IRS social scientists were unaware of the electronic journals to which they have access through an inter-Center consortium. The organizational problems facing CGIAR social science are considerable and multi­ dimensional. Loss of focus and highly asymmetric growth of social science units leaves the System with at most eight locations with a critical mass of social scientists necessary to steadily generate high quality research. Especially with the imminent arrival of Mega-Programs, greater reliance on matrix management forms of organization will be essential to maintain social science quality. Meanwhile, the System has turned inward and downstream with little evidence of greater impact as a result. ARI partnerships have degraded, as has capacity building in NARS and with students. And as core resources have dwindled, Centers have largely failed to achieve economies of scale in research support. Most of these problems are correctable. The panel offers specific suggestions in section 5. 4.6 Research Quality CGIAR social science has the capacity to generate high quality IPGs that contribute to CGIAR goals. Many of its social scientists and research lines do this regularly. But with loss of focus and a business model that emphasizes growth through restricted funding, the incentives and constraints facing System social scientists too often lead to weak research design and outputs with negligible impact. The aggregate result is highly uneven performance of modest average quality. There is a substantial amount of strong social science done within the CGIAR, with well- deserved international research awards, international accolades and citations to prove it. IFPRI is commonly singled out among CGIAR Centers for high quality social science research by survey respondents and interviewees both upstream and downstream. CIFOR's work on forest policy also elicits frequent compliments from a range of respondents. Pockets of work across Centers (most frequently CIP, ICRAF and ILRI) and CPs (mainly Harvest Plus) on agricultural productivity growth, on consumer and producer acceptance of new varieties, and on collective action and property rights likewise drew repeated praise. But there are also many CGIAR social scientists whose toolkits appear methodologically out of date, many projects that evince little careful thought about research design and methods, many publications that are poorly conceptualized and executed, and data sets of questionable quality. A significant share of the social science research undertaken in recent years by CGIAR IRS social scientists could be done just as easily - and far more cheaply - by strong recent graduates from Masters programs at ARIs and leading national universities in the global South, The heterogeneity of research quality within the System is alarming and must be addressed swiftly and directly. Tangible development impact from high quality research is uncertain; but it is 42 virtually certain that no major sustained improvements in poverty result from poor quality research. ARI social scientist survey respondents overwhelmingly echo the general assessment of extraordinary heterogeneity and widespread mediocrity of CGIAR social science research quality. Only 27-35% of them rated CGIAR social science "excellent" or "very good" in terms of quality of publications, quality of research staff, quality of research conducted or research impact. More than half of ARI respondents believe the CGIAR demonstrates comparative advantage in multidisciplinary research concerned with agricultural technologies and productivity or NRM. In other research areas (e.g., policy research at country or international level, methods development, research on innovation systems), only 8-39% felt CGIAR social scientists evince comparative advantage. Center social scientists and management speak frequently of a difficult tradeoff between quality and relevance in research. The panel believes this claim is typically misplaced, conflating quality with particular lines of research and relevance with direct development impact. Such conflation reflects widespread lack of focus in selection of research topics and projects. It is indisputably true that high quality but esoteric academic research often relies on rigorous but narrow theory and methods and has little relevance to the CGIAR mission and the System's stakeholders. This reflects not an intrinsic quality-relevance tradeoff, but rather a failure to recognize the necessary boundaries on CGIAR social science research based on the System's core competencies and areas of comparative advantage. The impact achieved through research necessarily and indeed should turn on the quality of the work undertaken within lines of research deemed most likely to contribute to ultimate development impact. Conditional on the choice of line of research, Centers must strive to maximize research quality, using the best possible theory, designs, data, and analytical methods appropriate to the problem under investigation. It is equally true that publications, citations and other standard academic criteria are imperfect measures of research quality. Thus the panel cautions against overreliance on those metrics even though it strongly recommends them as a valuable element in a portfolio of quality indicators. But so long as Centers maintain a tight focus on areas of comparative advantage, such that issue and methods selection is appropriate to generate findings that can contribute to poverty reduction impact, then the higher the quality, the greater the contribution toward impact. The need to maintain quality is magnified by the fact that when CGIAR social scientists begin to drop out of the cutting-edge dialogue within the global social science research community out of a concern to preserve "relevance", their ability to mobilize their most able counterparts in ARIs, NARS and the private sector wanes. This matters because the System will never have the critical mass to achieve its laudable-but-audacious goals without leveraging external science. Research prioritization The problems begin with research prioritization, or lack thereof. Far too much CGIAR social science appears reactive, chasing marginal grant dollars and, as a result, lacking in focus on Center or CP core competencies. Very few Centers still engage extensively in classical social science research prioritization work, in the form of ex ante modeling of the expected impacts of alternative lines of research. This work is extremely valuable to Centers and the System. And in many Centers, Medium Term Plans are treated mainly as bureaucratic requirements rather than as strategic plans that require input and buy-in from all Center staff and Boards and that should then guide the pursuit of funding and the allocation of time and other resources. 43 Moreover, rigorous research prioritization analysis remains publishable, as the mass of well- cited peer-reviewed articles based on IFPRI's IMPACT model demonstrate.14 The IFPRI-led FFarvest Choice initiative, launched in late 2006, is another valuable and innovative undertaking aimed at gathering meta data so as to generate a comprehensive data base on crop agriculture in developing countries. This model can guide research prioritization and investment decisions by CGIAR stakeholders following a novel spatial approach to research prioritization, a clear IPG based on emergent information and communications technologies. ITarvest Choice also demonstrates the value of ARI partnerships, as the University of Minnesota's International Science and Technology Practice and Policy center plays a central role in the initiative. Research design Good quality research almost inevitably requires a rigorous design from the outset, clearly identifying the hypothesis to be tested, the parameter to be estimated, the puzzle to be explained or some other research objective, as well as the specific methods to be used in achieving the research objective. In applied research, this typically involves data collection, analysis or both. The panel reviewed 16 different substantive (at least US$100,000/year) research projects recently undertaken by CGIAR social science units. Panelists have also reviewed quite a few other research proposals from CGIAR social scientists in recent years under other auspices. And in carefully reviewing the 216 best publications Centers submitted, the panel could typically discern and comment on the research designs employed. Like other dimensions of CGIAR social science research, the quality heterogeneity in research design is tremendous. A sizable minority of projects are impressively motivated and constructed, employing very clear communication of objectives, null and alternative hypotheses, cutting-edge methods and results dissemination strategies. Any repu table social scientist would be proud to have authored many of these proposals and studies. But a distressing majority of project proposals are of substandard quality. Typically the motivating problem statement is clear and compelling. Most of these larger projects focus, at least initially, on producing IPGs, although in several projects the IPG nature of the research got lost as designs were revised mid-project. Projects uniformly aim to tackle topics worthy of good research. But the research designs break down thereafter. Hypotheses are poorly specified, if at all, and then often not tested even when specified clearly. The logic of case selection is often unclear or unstated. Necessary controls for obvious alternative explanations are routinely omitted. Data collection plans are vague and often do not conform to current global best practices. Analytical methods proposed are often inappropriate, outdated, or both. Even output dissemination strategies remain unspecified and anticipated impact pathways unclear in too many proposals. Much emphasis is placed on process and partnerships and hoped-for impacts, without clearly specifying how the processes or partnerships are likely to contribute towards the desired impacts. A sizable minority share are focused squarely on local development impacts, with no indication that any IPGs are to be generated, even internationally generalizable lessons from a 14 Once upon a time, the CGIAR held comparative advantage in the development and evaluation of methods of research prioritization (Barker 1988). While that strength has perhaps ebbed, it could readily be restored if the System were to recommit to serious research prioritization. The deeply divisive debates within the CGIAR about the System Priorities (CGIAR Science Council 2005) lay bare the challenge of making such a recommitment. 44 specific context. The scientific details of proposals are typically deemphasized such that the coherence and credibility of the basic design is often substandard. The panel concludes that research quality control by Centers and CPs is highly uneven. In reviewing projects stratified by project size and timing, the panel found no significant correlation between either project size or project age and quality of research design, although one might have expected more recently designed and larger projects to be more rigorous in their specification of hypotheses, in the use of appropriate, cutting-edge methods, etc. Projects aimed at research priority setting, with clear strategies for disseminating findings to breeders, policymakers, firms and Center and System managers, were somewhat more likely to have clear, rigorous designs than were projects aimed at more general development topics. And projects with substantive engagement from ARI researchers were likewise routinely more rigorous in their design than those with at best token links to ARIs. CGIAR social science research proposals regularly invoke cutting-edge methods, but rather uneven understanding of when those methods are and are not appropriate and of how to employ them. For example, the panel notes that the term "experimental" gets used frequently in research design narratives, but often in cavalier fashion with little evidence of any true experimental design. Careful use of experimental design for randomized controlled trials is useful in some circumstances, but there are considerable natural limits to the usefulness of such methods in social science research.15 Similarly, although the CGIAR has been a rich source of integrated qualitative and quantitative research methods,16 a surprising number of proposals advance awkward or untenable attempts at such integration. And in spite of a long tradition of longitudinal data collection by CGIAR social scientists, very few proposals attempt in-depth longitudinal tracking of households, individuals and communities in order to be able to control for confounding variables. Most are one-off, cross-sectional studies of limited usefulness. Finally, there is surprisingly little use of advanced data collection and quality control methods that take advantage of advances in information and communications technologies (e.g., use of cell phones for high frequency data collection, mobile GPS units for tracking of subjects across space and time, mobile networks for performing field experiments, etc.), although Centers' field locations and technical sophistication should give them uncommon opportunities to innovate using such techniques. For many Centers, updating social science research methods is clearly a priority and could seemingly be remedied relatively easily through closer collaboration with internationally recognized researchers at other Centers, ARIs or even leading NARS. Overall, the panel's review of research designs left the impression that in core areas of comparative advantage, especially when focused on agricultural productivity growth and when working closely with ARIs, research design is solid. Otherwise, it is commonly substandard, especially for the considerable sums invested. Much of this variation in project design appears related to already-noted heterogeneity in the performance of individual CGIAR social scientists, or at best to the variation in research cultures among Centers and 15 See Deaton (2009) for a detailed articulation of the strengths and limits of randomization and related instrumental variable methods for identifying what works and what doesn't in development, from an economics perspective. The past few volumes of the interdisciplinary journals American Journal of Evaluation, Evaluation and Evaluation Review likewise contain a range of substantive assessments of the range of conditions under which randomized controlled trials are appropriate means of establishing impact. 16 For useful collections explaining and illustrating the benefits and modalities of integrated qualitative- quantitative research methods, see Kanbur (2003) or Adato and Meinzen-Dick (2007). 45 CPs, although with few observations per Center or CP, we cannot test either hypothesis satisfactorily. The panel's sense, consistent with the prior observation about the general dearth of research prioritization work, is that System social scientists are, of necessity, commonly scrambling to pull in one-off, short-term studies without the benefit of a clear vision of a precise research agenda to shape involvement in strategically selected projects so as to build up a larger body of evidence and findings on an important question. Groups with impressive sequences of coherent and high quality outputs are dearly following this longer-term approach in spite of their heavy dependence on restricted funding, an impressive achievement. But it seems that heavy dependence on restricted funding has led most social science groups within the CGIAR into a scatter shot approach wherein the whole is decidedly less than the sum of its varied parts. Data collection and management Centers and CPs collect enormous amounts of data, often in partnership with NARS or ARI partners. A few major CGIAR social science data sets are publicly accessible, of generally high quality and widely used. But the vast majority are one-time, special purpose data sets that are very small in size, poorly documented and not publicly accessible. And too many are based on questionable statistical sampling designs, questionnaires of uneven quality, data collection, entry and cleaning that is poorly supervised, etc. This applies not only to survey-based, quantitative research, but to qualitative social science data collection as well. Skilled senior CGIAR social scientists are rarely directly involved in the design and supervision of data collection and entry today; they simply do not have time. Instead, much of the work is subcontracted out to private consultancies or NARS partners or delegated to junior staff and students, with scant quality control built into the system. As a result, data quality has suffered at each stage: collection, entry, cleaning, documentation and storage. Even some of the more celebrated CGIAR social science data sets are rife with serious problems well-known to the small community of academics who have used those data. A large share of the data collected by Centers and CPs never get used in sophisticated analysis because of the extent and gravity of these quality problems. Of course, the "garbage in, garbage out" rule implies that the resulting analyses' quality gets compromised by questionable data quality. Development impact is also inevitably sharply limited when end-users cannot have confidence in the quality of the data on which all ensuing analyses and recommendations depend. It seems there has been a steady decline over the past two decades in the production, quality, and maintenance of general purpose data sets by CGIAR social scientists. A few well-regarded, longitudinal household surveys have been exceptionally rich sources of empirical insights on rural development and continue today (e.g., ICRISAT's recently resuscitated Village Level Survey in India, IFPRI's Ethiopia Rural Household Survey, IRRI's "loop survey" in the Philippines). Those data have yielded hundreds of theses and journal articles and have exerted considerable influence on development thinking and policymaking. Such data are major international public goods, the social science analog to the System's invaluable germplasm collections. Longitudinal data permit social scientists to trace the evolution of rural societies and to go back to identify key interventions, much as germplasm collections permit breeders to identifying the lineage of crops and to identify key changes in genetic material. The difference is that while the CGIAR clearly recognizes and invests in its germplasm collections, senior System and Center management and donors do not uniformly 46 recognize the enormous value of high quality longitudinal data. As a result, they systematically underinvest in its production. These data collections have been among the first victims of declining core funding over the past two decades, and yet such longitudinal data represent one of the most important legacies of CGIAR social science research since the 1970s. While the global research community and donors increasingly value such data, and have been funding its collection in recent years, the CGIAR is increasingly viewed as lacking the capacity to undertake large-scale, nationally or agroecoregionally representative survey data collection and analysis. For example, the Gates Foundation recently awarded US$19 million to the World Bank, not to the CGIAR, for a six country initiative to improve household level data on agriculture in sub-Saharan Africa (the Living Standards Measurement Study : Integrated Surveys on Agriculture, LSMS-ISA). While the Gates Foundation intends to fund a major new CGIAR initiative to track germplasm adoption and diffusion in South Asia and Sub-Saharan Africa - following up Bob Evenson's seminal work - the World Bank's large-scale survey skills trumped the CGIAR's expertise in agriculture in the LSMS-ISA project. No other organization than the CGIAR could really have undertaken the germplasm tracking study. Where alternative providers exist, they are increasingly outcompeting CGIAR social science, which should have a comparative advantage in such IPG production, were it a System priority. Centers and CPs must redouble their commitments to longitudinal data collection, committing to necessary staffing to ensure high quality data collection, cleaning, storage, documentation, and dissemination. The management of collected data to facilitate access to the broader scientific community is a widespread problem as well. In response to the panel's survey of Centers and CPs, only 7 Centers and 2 CPs indicated that they routinely make primary and secondary social science data they collect -and the associated documentation -available to other researchers. And most of those only make data available on request; the data are not online for open access download. In a few cases, restricted data access arises because the data collection was (co-) financed by NARS collaborators who are reluctant to make data publicly available. Such reluctance should be challenged, however, as virtually any such data collection exercise relies on some measure of public funding with the reasonable associated expectation of public availability of the resulting public good. Restricted data access should be an exception; presently it is the rule. IFPRI clearly sets the standard within the CGIAR, with a relatively rich and up-to-date web site making data publicly available. But even those data are commonly insufficiently documented to be readily employed, even by researchers quite familiar with the setting but not with the particulars of the survey. And not all IFPRI data are publicly accessible. Most Centers (appropriately, in the panel's view) gives staff one or two years following project completion to exclusively exploit the data they worked hard to collect. Outside of IFPRI, data dissemination by CGIAR social scientists is poor. The summary report from the June 2008 CG Data Management Workshop hosted by Bioversity in Rome sums it up this way in its opening sentences: "Empirical data from field and lab observation are, when connected with secondary information, the raw material of all our research outputs. They must therefore be valuable, yet we look after them in a surprisingly casual way. Compared with our financial data, we have few standards or recognized good practices, few professional staff responsible, few incentives for good performance and no indicators of success." The panel completely agrees. Centers and CPs recognize they have a problem; but they are doing little or nothing about it other than crafting policies that too often remain poorly implemented. Data are still viewed as 47 the property of individual scientists, an antiquated approach that invites unnecessary waste and does not safeguard against scientific fraud by limiting opportunities for replication. Inter-Center cooperation on data management issues is surprisingly limited although there are enormous economies of scale in data management. The CGIAR ICT-KM Program (http://ictkm.cgiar.org/) is attentive to this issue; but progress to date has been minimal. Outside of the Harvest Choice project, there has been no significant investment in creating metadata that might make publicly available the data sets CGIAR scientists invest scarce time and skill in collecting. Making data available helps investigators in other research institutions make more and greater discoveries and with less duplication; it is an important IPG. Doing so requires adequate careful documentation of the survey methods and instruments, data cleaning and variable construction procedures, etc. so that the data are properly contextualized and interpreted. Although there are clearly major economies of scale and scope involved in meta­ data creation, maintenance and dissemination, we saw little evidence of substantive Systemwide effort in this direction, whether through the CGIAR ICT-KM Program or the Consortium for Spatial Information, either of which could, in principle, make important advances in this area with the right leadership and incentives. Current best research practice in the social sciences is to run all research designs through an Institutional Review Board (IRB) prior to the commencement of any field data collection in order to ensure the protection of human research subjects. For example, major scientific funding agencies - e.g., the National Institutes of Health and the National Science Foundation in the United States - will not make research awards without IRB approval. Concerns about research on human subjects are greatest in the biomedical sciences - including in many human nutrition and health studies, in which some Centers and CPs (especially IFPRI) engage - but concerns exist in and the rules apply to the traditional social sciences as well. CGIAR social scientists are often unaware of IRBs and routinely fail to adhere to current global practices regarding the ethical protection of human subjects in data collection. In response to the panel's questionnaire, only two Centers (IFPRI and World Fish) appeared to have an IRB to clear ethical issues as a routine part of project approval. One CP (Harvest Plus) employed the IFPRI IRB and ICARDA and ICRISAT had alternative procedures for ethical review, but not a special purpose IRB. Multiple Centers remarked that this is "not an issue", they "are not really 'using' human subjects in research," that "researchers are responsible enough to know the level of confidentiality of the data that they are collecting," and similar responses indicating a disappointing lack of awareness of and, in some cases, concern about fundamental research ethics issues in the social sciences. The Science Council commissioned a sequence of studies on research ethics that made clear recommendations for the System-wide use of IRBs by Centers and CPs in research that engages human subjects.17 That panel concluded that "despite some advances in particular Centers, the ethical review of CGIAR research processes is deficient in many instances. In particular, the ethical review of research involving human subjects and surveys or the use of data with personal identifiers should be implemented in all Centers conducting such research. Deficiencies in ethical review leave the CGIAR vulnerable to considerable liability and leave a gap that must be addressed as a System-wide issue" (CGIAR Science Council 2008a, p.36). While there are routine grounds for exempting studies from informed consent and other standard IRB requirements, 17 These studies are available online at http://www.sciencecouncil.cgiar.org/home/priorities-strategies/en/ and summarized in CGIAR Science Council (2008a). 48 IRBs have standard guidelines for making such exemptions and for expediting approval for low- risk studies. Investigators do not have the moral authority to make decisions as to which ethical guidelines should apply to their particular study and Center management that permits such treatment of human subjects runs serious liability risks. The CGIAR's cavalier approach to the protection of human subjects concerns the panel. Research outputs Research output quality is an elusive characteristic related to the originality of the topic, rigor of the concepts, design and methods used, scientific and practical weight of findings presented, and clarity and uptake of the results. Publication in internationally respected, peer-reviewed journals - or in some disciplines, in books from major scholarly presses - is an indicator of research output quality, but an imperfect one and merely instrumental in fulfilling the CGIAR's core impact objectives. High impact CGIAR social science need not get published in journals - much less in leading disciplinary journals - if, for example, the target is to guide research prioritization within the System. Although the panel is strongly supportive of peer-reviewed journal publication as the default form of research output for CGIAR social science, it cautions against journal fetishism of the sort that can emerge if mechanical enumeration of articles and their citation numbers become a dominant metric for judging research output quality. The panel emphasizes that peer review publications are important primarily because the peer review system provides a useful, low-cost means of quality control and because such publications are an effective means to leverage external science by inducing new research outside the CGIAR in the System's areas of interest. Peer review can be a straightjacket, to be sure, and not all innovative and relevant work fits neatly into leading journals, which are often not receptive to truly path-breaking research. But when authors are highly respected, editors are typically more open to more innovative research; innovation without reputation is difficult to publish. Given the latent nature of research output quality, the SSSR panel employed several commonly- used indicators to assess research output quality. The indicators ranged from bibliometric indicators such as publication and citation rates, to more intensive but subjective measures based on the opinions of NARS and ARI survey respondents and on the panels' reading of the 216 submitted best publications, to indications of impact from development practitioners and NARS with whom we spoke. All Centers and CPs expect publications and related research outputs (e.g., data sets, software, training manuals, etc.) from their social scientists. These expectations vary in quantity and outlet by Center, and even by program within Centers. But there is an appropriate, clear, universal focus on generating tangible research outputs. Given that social scientists account for more than one quarter of CGIAR IRS and the System is approaching a US$500 million/year aggregate budget, the panel would expect to see at least 100- 150 high quality articles each year- papers that appear in respected, peer-reviewed journals and attract downloads and citations within a few years - as well as multiple high quality data sets produced and publicly released. With more than 300 IRS social scientists, that amounts to at least one high quality article every other year, not an unreasonable minimum standard. As reported in the Phase 1 Background Study (Marenya 2008), over the period 2005-7, CGIAR social scientists published an annual average of 1.08 journal articles, 0.44 book chapters and 0.05 49 books. Roughly half of the journal articles (0.58 articles/year per social scientist) were in Thomson ISI journals, the rest in typically lower quality journals. The average publications rate of CGIAR social scientists appears adequate. This average masks considerable dispersion, however; more than 15% of CGIAR social scientists averaged at least four Thomson ISI journal articles per year, 2005-7 while fully a quarter had none. CGIAR social scientists publish primarily in well-respected applied journals, the sort both researchers, policy makers and informed practitioners read regularly. Over the period 2005-7, World Development was the most frequent outlet for CGIAR social science, followed by Agricultural Economics, Food Policy, Forest Policy and Economics and Agriculture and Human Values. No other journal published ten or more CGIAR articles over that three year period. A few papers appeared in premier journals (e.g., Lancet, Science), but these are uncommon albeit desirable because of their exceptionally high visibility. But with increased use of search engines to track down research results, researchers can more easily search a broader set of outlets, reducing the premium on publication in a small core set of elite journals. Indeed, recent analysis clearly shows that citations are increasingly dispersed across a larger number of journals and papers (Larivière et al. in press), making a wider array of journals appropriate outlets for applied research aiming at generating tangible impact. A key to generating high quality research is that quality be highly valued; and that low quality will not be tolerated. Yet there is little shared sense within the CGIAR as to what constitutes high quality social science research and thus few clear incentives for investing in research quality, as opposed to quantity, and few penalties for generating poor quality outputs. The panel noted in reviewing project proposals that while scientific research papers are virtually always an explicit intended output, these is a common tendency to attempt post-hoc production of a paper or two from the research, with little ex ante prior design. Even in stronger Centers, little attention is paid to citations, downloads, professional honors or other indicators of influence on the development or research communities. More than one observer remarked that there appears a growing disjuncture between what contemporary academic social science publishers expect and what CGIAR social scientists can produce, given their skills, incentives and competing demands on their time. One metric of quality is paper downloads, a clear indicator of current demand for the published research. Elsevier provided the panel with data on the top 25 most downloaded articles in 2007- 8 for six journals that are among the most frequent outlets for CGIAR social scientists: Ecological Economics, Food Policy, Forest Economics and Policy, Land Use Policy, Journal of Development Economics, and World Development. The results suggest limited demand for CGIAR social science articles. Only three CGIAR articles were among the top 25 downloaded articles for the above journals, two in Ecological Economics and one in Forest Economics and Policy. No paper with a CGIAR author was among the top 25 downloads in 2007-8 for four of the six journals. The absence of CGIAR articles among top downloads from Food Policy and World Development, two of the three main journal outlets for CGIAR social scientists, jointly accounting for 25% of all CGIAR social science articles published in Thomson ISI journals, 2005-7, is an especially discouraging sign. Citations rates are another imperfect indicator of research quality, as they signal medium term impact on other researchers' work. The citations data are similarly discouraging. The panel used two different citations databases as a check on its findings. We use data from Scopus on the 16 primary journals in which CGIAR social scientists published, 2005-7 (representing 82% of 50 total articles over the period), as well as ISI Web of Science citations data on 2005-7 publications and on individual Center social scientists. The Scopus data reported in Table 4.3 show that CGIAR social scientists contributed 8% (32/400) of the top 25 most cited articles published in those 16 journals since 2003, lead authored 5% of them (19/400) and received 6% of the citations (434/7593). 62% of these papers had ARI co­ authors; 32% had NARS co-authors. For these highest profile papers, citations rates were typically around the median. CGIAR papers more commonly appeared among the 25 most cited in the journals with lower impact factors in this group - 10% of the top 25 articles in the lower eight journals versus 6% in the upper eight. Table 4.3 Top 25 cited papers published since 2003 in 16 primary journal outlets Source: Scopus Impact Total # cites Median #CG papers Median Cites/ #CG lead Journal Factor in top 25 cites/article in top 25 CG article in top 25 World Development 1.57 1144 42 1 31 1 Ecological Economics 1.55 1066 33 2 31 0 Human Ecology 1.25 412 13 1 15 1 Land Use Policy 1.21 608 22 1 22 0 Food Policy 1.06 430 14 6 20 3 Am J of Agric Econ 1.03 690 23 0 0 0 J of Dev Econ 0.91 686 25 0 0 0 Forest Econ & Policy 0.90 387 15 1 14 1 Econ Dev & Cult Change 0.88 327 14 4 14 2 Land Deg & Dev 0.86 386 15 0 0 0 J of Dev Studies 0.67 371 13 4 16 2 Ag and Human Values 0.61 263 10 0 0 0 Agricultural Economics 0.60 299 10 6 11 4 Rev Agric Economics 0.47 292 9 0 0 0 J. of African Econ 0.41 134 4 4 6 3 Developing Economies 0.38 108 3 2 6 2 Total: 7593 32 19 We likewise used Scopus to compare bibliometric indicators for CGIAR Center social science against those of some benchmark development research institutes. As shown in Table 4.4, IFPRI produces far more publications than any of the benchmark non-CGIAR institutes, which in turn generate more publications than do the social scientists at any of the three other Centers we selected for this analysis. As measured by the maximum number of cites of any paper since 2003 or by Hirsch's h-index,18 the CGIAR social scientists seem to perform similarly to these development institutes. CGIAR social science research productivity seems roughly on par with is per Hirsch (2005), a scholar with an index of h has published h papers each of which has been cited by others at least h times. Thus, the h-index reflects both the number of publications and the number of citations per publication. 51 other non-academic research institutes in the group that are heavily dependent on restricted funding. Turning now to the ISI data, CGIAR social scientists' 2005-7 publications were cited an average of 0.54 times per year since publication (Marenya 2008). By comparison, the median 2007-8 impact factor of the journals in which these articles were published was 0.88. Since the impact factor measures average annual citations per article in the initial two years after publication, and we use only the impact factors for the journals in which CGIAR papers appeared, this represents a strict comparison of 2005-7 publications using the same units in the same outlets.19 CGIAR articles were cited at only 61% of the rate of the average article in the journals in which they appeared. Again, this masks considerable heterogeneity, with several well-cited - and some award-winning - papers included in the 2005-7 set of articles. These high impact papers are heavily outnumbered, however, by the majority of CGIAR social science articles that went uncited during that three year period. ISI Web of Science data on CGIAR social scientists' career citations paint a similar picture: much heterogeneity with quite weak average performance. As reflected in Table 4.5, for the 223 IRS social scientists for whom reliable data could be assembled, half of them had four or fewer publications ever cited in journals indexed by ISI, averaging just over one citation per year cumulatively across all of their publications, with an average h-index - and indicator of breadth of scholarly impact - of merely 2.0, meaning that half had no more than two publications with at least two citations each. By way of comparison, the median (mean) h-index in 2007 for 635 faculty in 20 leading agricultural and resource economics departments in the United States was 4.00 (4.44), double that in the CGIAR. Significantly higher means than medians reflects the positive skewness in the citations distribution, reinforcing the pattern of heterogeneity, with a small number of very solid performers outnumbered by a very large number of weak performers Table 4.4 Bibliometrics for Social Scientists at Selected CGIAR and Other Development Research Units, 2003-8 Source: Scopus Approximate # Average Max cites/ publications #/year publication H-index CGIAR Centers: IFPRI 1000 75-100 96 32 ICRAF 60 1-10 157 12 IWMI 70 5-8 54 11 CIMMYT 25 1-3 121 10 Benchmark Development Research Institutes: Overseas Development Institute (UK) 300 20-30 78 18 World Resources Institute (USA) 200 10-15 183 18 International Dev't Research Centre (Canada) 150 5-10 110 14 Institute of Development Studies (UK) 100 1-5 184 17 Center for Global Development (USA) 80 10-15 76 10 Note: First two columns are rounded. 19 See Amin and Mabe (2000) for further insights into impact factors and related bibliometric data. 52 Recognizing the significant limitations of bibliometric data in assessing publication quality, the panel read each of the 216 best publications Centers submitted for the SSSR. Not surprisingly, we found citations counts to be only moderately positively correlated with our subjective assessments of publication quality. As summarized in Annex 7, these papers appear in journals that are actually somewhat weaker (in terms of impact factors) than the average CGIAR social science journal article and are cited at no higher a rate. Thus the pool of best publications seems surprisingly representative of CGIAR social science publications more generally. Table 4.5 Citation Statistics per IRS Social Scientist (n=223) Statistic Median Mean Cited publications/IRS 4.00 6.95 Citations/year per IRS 1.19 3.52 Hirsch's h-index 2.00 2.59 Source: ISI Web of Science data compiled for Marenya (2008) and update subsequently. The panel noted considerably heterogeneity in the quality of these papers, both within and across Centers. A small number of scientists were consistently associated with the best papers, reinforcing the impression of a moderate sized core of strong social scientists surrounded by a sizeable majority of underperformers. The panel consistently rated the papers meant to reflect excellent multidisciplinary research and strong collaborations as superior to those meant to reflect cutting-edge disciplinary research, in each of three areas: use of best practice, weight of findings and international public goods generation. This is consistent with the observation that Centers' comparative advantage is associated less with the production of fundamental new discoveries within disciplines than in research that integrates across disciplines and organizations. The panel also found that CGIAR social science research related to agricultural productivity and technologies or on institutions and markets were significantly stronger than those on natural resources management or on general theory or methods. The very best of these best publications - as identified by the panelists in flagging the most impressive 10-20% of the publications they read - were disproportionately (roughly 40%) about markets and institutions, followed by those on agricultural productivity and technologies (nearly 30%), and then natural resources management, with essentially none making core contributions to disciplinary methods or theory. Policy briefs and other short, non-technical summaries of research play a valuable role in communicating broadly and quickly key research findings. Printed briefs commonly have far higher download rates from Centers' web sites than research papers do. Email briefs have become extremely popular and effective as well. An oft-cited example is CIFOR's POLEX (short for Policy Experts), which distributes roughly one short electronic brief per month to keep key opinion leaders, policy makers and researchers informed about recent policy-relevant research results from CIFOR and its partners. And they are a primary means by which downstream partners absorb results they might not read in longer or more technical publications. Indeed, briefs often induce stakeholders to read the complete, peer-reviewed publication from which the brief is drawn. Finally, the panel observes that Centers still rely heavily on outdated methods of disseminating research results, especially formal in-house publications such as research reports and discussion papers. Before the era of internet access of informal working papers, these sorts of institutional series undeniably played an important role in getting research results out long before they appeared in journals or books and in providing details that would otherwise never be published. 53 But today such series have been largely eclipsed in the global social science research community by online posting of informal working papers by individual scientists or institutions. These serve the same function, with virtually identical accessibility, but at far lower cost than formal, in-house publications. These in-house series are relatively expensive to produce, often relying on paid external reviewers and hard copy publication and postal distribution. And quality control in these series is uneven. The most productive CGIAR social science researchers typically dislike these series as they compete for scarce writing time with other, more visible outputs. And citations data indicate that few other researchers bother to use and reference these publications. IFPRI's research report series is almost universally regarded as the pre-eminent internal publication series in CGIAR social science. Yet of the 115 IFPRI research reports published between 1983 and 2008, Scopus reports only 22 had ever been cited, and only one had been cited five or more times. Perhaps these in-house research reports are being used by downstream practitioners - although those we interviewed told us otherwise - but they are certainly not connecting cost-effectively with the broader scientific community. The panel sees little value addition from Center internal publications other than briefs. Peer- reviewed journals and books should be CGIAR social science's targeted outlets, with unreviewed manuscripts and unpublishable, detailed supporting materials made freely and widely available on Centers' web sites with minimal added cost through informal in-house series or individual scientists' web pages. These series are emblematic of somewhat outdated SSR in the CGIAR. 4.7 Summary Assessment CGIAR social science is essential to the System and has long made seminal contributions to the social sciences as well as to broader development impacts that agricultural innovation can yield. But the panel considers CGIAR social science on the cusp of an outright crisis. Mission drift and fragmentation of social scientist time is universal, leading to high variance in the quality of research and partnerships and in impact, such as the System measures it. There are pockets of excellent social science research by internationally renowned experts, good morale and real impact within the CGIAR, to be sure, especially where there remains critical mass, a local culture that prizes rigorous research, and adequate long-term funding to minimize fragmentation of scientists' attention and effort. But in all Centers, capable staff are put in untenable situations and too often driven out by the new business model. Key IPGs are not being produced; instead a large amount of development consulting-type activities are undertaken in order to meet payroll expenses, the Centers' organizational drive for growth, and immediate donor demands. IFPRI can certainly survive as a standalone social science entity; perhaps 4-6 other Centers' social science units can as well. But most CGIAR social science units are either quite weak already or extremely fragile. Hiring patterns in response to development- oriented restricted funding are reinforcing this mission drift. And all CGIAR social science units are significantly underperforming their potential. In the aggregate, CGIAR social science has lost much of its research coherence and quality through dilution effects resulting from the pursuit of increased relevance; the irony is that there is little or no evidence that CGIAR social science has in fact increased its development impact by focusing more directly on that end goal. Indeed, the panel suspects the opposite, that CGIAR social science's impact has lessened as donors and management have pushed harder for immediately visible development impact because it has meant less attention to the fundamentals 54 of the intermediate research steps. The increasingly transactional - and occasionally competitive - nature of relations with many NARS and among Centers is but one troubling sign of diminished sustainable development impact. Rapid revenue growth associated with a loss of disciplined focus on core competencies is at the root of these problems. This can be reversed if Center, CP and System leadership is prepared to make serious, sustained efforts at rehabilitating CGIAR social science so that it can make the major contributions to CGIAR goals and to global agricultural and rural development of which it is plainly capable. 55 5. RECOMMENDATIONS FOR CGIAR SOCIAL SCIENCE Looking to the future, one must grasp simultaneously the tremendous opportunities facing CGIAR social science today, the crucial role the social sciences play within the CGIAR, its distinguished past, and the perilous state of CGIAR social science today. The SSSR panel offers four broad recommendations, encompassing 21 specific sub-recommendations. We believe these recommendations will allow the System to seize emerging opportunities and rehabilitate its social science activities. The four recommended reforms relate to management, reorganization of social science activities, human resources and research quality standards. Recommendation 1: Undertake essential management reforms The root problems of CGIAR social science are directly traceable to management problems associated with the business model pushed by donors and accepted by CGIAR leadership, the incentives facing Center senior management, the selection of senior management with too little regard to their comprehension or vision of the role of social science within the CGIAR, insufficient focus on core areas of comparative advantage, and on excessive emphasis on measuring impact at a scale and frequency that are impractical and ultimately ineffective. Thus the panel's first and most fundamental recommendation is that the CGIAR must undertake essential management reforms in order to improve its social science research. Without management reforms, progress is exceedingly unlikely. We break down the essential management reforms into seven discrete sub-recommendations. These are necessarily interrelated but perhaps best discussed individually, even though they may be tackled in concert. la) Resolve the mismatch between the business model and staffing patterns The fundamental question CGIAR leadership must address is the System's strategic direction. The CGIAR is too small and uncoordinated across Centers to both generate IPGs through research and also engage in direct development activities. System leadership must decide on its primary line of business, then focus its activities sharply and let the chosen function determine the System's form. The root problem of CGIAR social science - and we suspect of CGIAR science more broadly - is that the function has expanded radically to encompass a vast array of development topics, but the form - Centers populated with Ph.D. scientists - has remained the same and is ill-suited to this expanded business model. The panel recommends restoring the old business model, which would require a significant sharpening of focus and increase in the share of funding that is unrestricted or at least long-term and large-scale ("quasi-unrestricted"). Many of the most serious problems plaguing CGIAR social science research today can be resolved once the scientists' burden of managing a large portfolio of donor-driven, small-scale, short-term projects are sharply reduced. Restoring adequate unrestricted or quasi-unrestricted funding would do much to address CGIAR social science's loss of focus, short-termism, insufficient attention to research quality, degrading vigor or partnerships, and problems recruiting and retaining the best scientists. Changing the mix of funding might require attracting non-traditional members from science, research or agriculture ministries, foundations and national academies of science that may better appreciate the value of long-term research than do many of the short-term-impacts-oriented development agencies that presently account for most CGIAR social science funding. If additional unrestricted funds are not 57 forthcoming, changing the funding mix might require letting restricted projects wind down without replacement, shrinking the scope and staffing in order to return to sharper focus on core areas of comparative advantage, underpinned with adequate unrestricted resources. But changing the funding mix, although necessary, is not sufficient. It will require far greater discipline by management, in developing medium term plans in full collaboration with research staff who fully buy into the strategic direction, and in turning away funding that does not directly facilitate the agreed strategy. Under a revised business model, CGIAR social science should focus more tightly on IPGs associated with the three core research areas identified earlier: (i) sustainable agricultural productivity increases of, by, and for the poor, (ii) conservation of the natural resources on which the rural poor directly or indirectly depend for crucial environmental services, especially concerning agricultural productivity, and (iii) institutional, policy and technological innovations that enhance the quality of life for poor and marginalized agrarian populations. This will require a return to adequate core support for scientists' time, with at least 66-75% unrestricted and long-term, large-scale restricted funding so that at least three-quarters of scientists' time is devoted to no more than 2 or 3 major, sustained research efforts within Centers' core competencies. Unless Centers reduce the excessive fragmentation of social scientist effort they will not improve research quality and research-driven development impact. Our remaining recommendations below all follow from that preferred course of action. The panel recognizes, however, that donors might not be willing or able to provide more core and long-term, larger-scale unrestricted funding. If not, then the CGIAR will have to bring social science recruitment/ evaluation into alignment with a more action research-oriented, development organization (for profit or not-for-profit) model. Those organizations have very few Ph.D.s and a cadre of very bright, effective people who undertake applied research using mature methods (e.g., adoption analyses, value chain studies, basic impact assessment work, policy analysis with off-the-shelf trade models, etc.). They tap ARI expertise as needed for technical backstopping. But they are fundamentally oriented towards generating development outcomes. They are consumers not producers of IPG research. Publications are relatively uncommon and are typically in report form only, with little resulting impact on or leveraging of the broader research community, but often significant, if almost only local, practical impacts. Research design is opportunistic and the time-scales of most work is short. Engagement with communities, governments, and local business communities can be deep. Such a System might be more accurately labeled a Consultative Group on International Agricultural Development (CGIAD). There is not much to recommend in this model, however. It is crowded space and the CGIAR is a high-cost provider of such services. If donors are unwilling to restore adequate core and quasi- unrestricted funding, the panel believes CGIAR social science could potentially be more effective as a development agent - as distinct from a research-for-development agent - particularly after adjustments to its social science staffing model and by dropping the unrealistic demands of doing high quality, publishable, internationally visible research off of almost-exclusively short­ term project financing. The developmental impacts though of such a drastically revamped CGIAD are likely to be in line with the moderate aggregate support provided the System. The suggestion was made to the panel that an alternative design to in-house provision of social science capacity might be contractual out-sourcing of social science research to a consortium of international universities. While this might obviate current problems of career management, recruitment, quality control, etc. the panel considers this radical alternative inadvisable. It would move CGIAR social science too upstream and exacerbate the risks of decoupling the 58 natural and social sciences, which is a primary source of CGIAR comparative advantage in the social sciences. The panel favors improved linkages with ARIs (see sub-recommendation 2a below) but considers the out-sourcing model a serious overreaction to the challenges presently faced by CGIAR social science. lb) Realign management incentives Currently, Center management's incentives in practice emphasize growth in Center revenues and size. Yet impact is meant to be achieved through research, and scientists are expected to publish and generate important discoveries. Managers' incentives are inevitably passed on to the scientists, both through formal performance monitoring systems, and through myriad informal cues, creating mixed signals as to what Centers really want from scientists. Management needs to support social science research in order for social science research, in turn, to effectively support the management's pursuit of the CGIAR's core mission of impact through research. To promote this virtuous dynamic requires a significant realignment in management incentives in order to promote quality of science for development impact over revenue growth and Center expansion. lc) Improve leadership selection Center leadership is extremely uneven - over time and across Centers and CPs - in its understanding of, appreciation for, and investment in social science research. By virtue of the nature of its mission, the CGIAR cannot succeed without strong social science, so leaders who fail to recognize and reinforce the value of strong social science research effectively condemn their Centers to under-perform. Boards and senior management need to take a far closer look at how candidates for senior management positions view and have interacted with social scientists in the past and at their proposed strategy for recruitment and deployment of social science expertise. Id) Tighten the focus on comparative advantage Each Center and CP social science group needs to clearly articulate for its Board its precise areas of comparative advantage it holds relative to the Center mission and to alternative suppliers. Then management and Boards should firmly hold staff to those boundaries. The research foci and lines of research reflected in Figure 3.1 are a good place to start; expansion beyond those areas probably requires careful deliberation to ensure against mission drift. The panel strongly recommends renewed emphasis on multidisciplinary social science research on productivity growth by and for the poor, perhaps especially on ex ante research prioritization, on long-term, field-based data collection in a range of sentinel sites in order to identify and measure changes in the behavior and well-being of rural peoples, especially the poor. The panel further emphasizes the need for stricter application of the international public good criterion to judge research proposals before they are submitted, so as to enhance the likely quality and generalizability of the resulting research impact. Contemporary CGIAR social science undertakes far too much highly-local research outside of broader comparative projects that can synthesize findings usefully and convincingly across contexts. A few Centers have begun implementing default policies of rejecting small-scale, short-term restricted funding projects that do not squarely fit the Center's Medium Term Plan. The panel applauds this strategy and recommends that all Centers begin to employ similar management processes to reduce fragmentation of social science effort. This strategy is of particular 59 importance in the short term as the CG transitions to the new MP model. Projects that do not cover at least one-third of a scientist's time over two years or more should bear the burden of proof that they provide essential support of medium-and long-term research programs in areas of comparative advantage. le) Focus on impact but end the impact measurement obsession The ever-greater demand to record ever-more-minute, punctual project achievements leads to the enumeration exercise becoming an end in itself, crowding out the production of high-impact research programs and the achievement of major, scalable, sustainable impact. System and Center management need to devise means by which they can maintain necessary and appropriate monitoring yet reduce the commodification of scientist time in order to separate the normal and ordinary from the impressive and extraordinary. Structured time displaces creative time. Moreover, excessive focus on near-term impacts has fostered serious mission drift as achieving impact has become the coin of the realm such that Centers are transforming into development agencies, leading to the business model-staffing pattern mismatch. If) Mainstream gender equity as a basic axiom ofCGlAR research The CGIAR's Participatory Research and Gender Analysis Program has supported a range of studies, many of high quality. Several Centers (notably CIAT, IFPRI, ILRI and IRRI) are widely recognized as having made substantial contributions to research on women's role in agriculture, NRM and rural development and to the impact of technologies, institutions and policies on women. Nonetheless, as noted by the 2008 CGIAR Independent Review, gender is not yet well integrated into CGIAR research. There is little evidence of consistent attention to gender issues by senior- level Center or System management, in research prioritization, research design or performance monitoring. Social scientists should take particular note of this organizational shortcoming and actively champion explicit attention to gender issues where appropriate. Women both disproportionately suffer the burden of poverty and food insecurity and in most of the developing world, contribute far more labor to agricultural production, post-harvest processing and renewable natural resources management than men do. Given the CGIAR mission, women should therefore assume a certain primacy of place in research prioritization, resource allocation and outreach, if only indirectly by virtue of effective targeting of CGIAR activities to address its stated mission. Whenever relevant, gender analysis must be mainstreamed into assessments at all stages of research - ex ante, mid-term and ex post - in order to establish how CGIAR activities affect women, especially the poorest and most food insecure, and to help advance Millennium Development Goal #3. Operationally, this requires managers take explicit responsibility for gender equity in research, not just in human resources management. Many Centers have troubled histories of mistakenly assuming that female scientists should be the ones to do the work on gender issues. An MDG should be everyone's business, not just that of women with Ph.D.s. Center Boards and the mooted new CGIAR consortium management body should immediately make gender equity a basic axiom of CGIAR research and should begin incorporating gender impact equity indicators in performance appraisals.1 1 The panel emphasizes the focus on gender impacts, rather than on weak staffing indicators, such as percent women among managers, which can lead to tokenism and misallocation of scientists that actually diminishes the intended development impact. 60 Ig) Require full indirect cost recovery Scarce unrestricted resources must not be used to subsidize restricted funding projects. For donors unwilling to pay indirect costs, management must ensure that all direct costs are covered, including support services, supplies, etc. Scientists and donors have clear incentives to keep project IDC rates low, so only management can prevent a tragedy of the commons with respect to unrestricted funding by being firm about full cost recovery and requiring DG approval (and DG accountability to the Center's Board) for waivers. Recommendation 2: Re-organize and re-focus CGIAR Social Science The present, diffuse organization of CGIAR social science limits its impact. Partnerships have become excessively transactional and ties with ARIs, in particular, must be actively restored. Training and capacity building with NARS and other local partners have suffered. The System needs renewed focus on direct research mentoring to help build and sustain partners' social science research competencies. It needs to organize so as to realize the IPGs and large-scale impact achievable through regional and global integration of research designs and findings, perhaps through a new Mega-Program. The System likewise needs to invest in a larger-scale center of excellence on impact - how to achieve it through research prioritization and the design and conduct of project implementation, and how to measure it both ex ante and ex post. This would marshal what are presently excessively dispersed and ineffective Center-specific units and reinforce the core objectives of a SPIA that is too detached from Centers and too poorly resourced. Finally, as key functions that need to be undertaken at larger scale are reorganized into alternative structures, the System should shrink its unproductive Center social science units. This can be accomplished through the five sub-recommendations that follow. Before discussing these in detail, the panel notes that the new Mega-Program (MP) idea under the Change Management process is untested. Many observers justifiably wonder whether it will yield either new, unrestricted resources or any appreciable change in the management and organization of CGIAR research. There are certainly vulnerabilities associated with these new arrangements; but the need for change is abundantly evident within CGIAR social science. So the panel makes its recommendations under the premise that there will be thoughtful design and monitoring of new MPs. If MPs do not, in fact, bring a shift in the CGIAR business model, however, the primary alternative for CGIAR social science will be to shrink its size so as to concentrate far more tightly on areas of comparative advantage. Downsizing is not the first-best option, but if the sorts of restructuring recommended here proves infeasible, it may be the only path to retain CGIAR social science research rather than drift irreversibly into a CGIAD. 2a) Restore longer-term partnerships, especially upstream The CGIAR occupies an essential and pivotal middle ground between the basic research conducted in ARIs (and leading NARS) and the development programming undertaken by NGOs, community-based organizations and most NARS. Its effectiveness intermediating that intellectual, cross-cultural and international space so as to stimulate research-driven reduction in poverty and food insecurity turns fundamentally on the quality of its relationships both upstream and downstream. The System's new business model has compromised both those relationships. Partnerships with ARIs have become especially fragile and demand the most immediate attention. The System needs access to global stars with relevant (latent or current) research interests in order to keep up with rapidly-moving research frontiers. There is need for 61 expanded, longer-term ARI-CGIAR linkage programs or creative, Center- and MP-driven arrangements to leverage leading ARI social scientists via collaborative work agreements, joint appointments, exchange programs, sabbatical hosting programs, etc. Durable ARI partnerships with leading scholars provide a more rapid and cost-effective means of promoting skills transfer in the social sciences than hiring does. This is especially true in the non-economics social sciences, where it would be unproductive to hire junior people without senior mentors accessible, if only part-time and electronically. There would also be strong synergies between a Young Scientists Program (see recommendation 3a), since many of those post-docs will have trained with elite ARI researchers, whose students are more likely to join Centers with which their doctoral supervisors are affiliated, and the senior scholars are more likely to affiliate with Centers where their former students work. The CGIAR must continue to work with and through NARS; the System is too small to generate large-scale, geographically diffuse impact without extensive and well-functioning NARS partnerships. But NARS also told the SSSR panel that they want the CGIAR to intermediate for them with ARIs on fast-moving areas of science that the NARS cannot hope to keep up with on their own. There is thus a natural synergy between upstream and downstream partnerships, which could even be made explicit through multi-year ARI-CGIAR-NARS linkage grants. 2b) Focus training and capacity building on research mentoring Historically, CGIAR researchers have played a valuable role in building local and global research capacity by hosting and mentoring field research by students, especially from developing country universities. By most accounts, as CGIAR social scientists' time has increasingly been diverted to restricted projects that afford little time for intensive student mentoring, the scale, scope and quality of such informal capacity building has deteriorated. The System needs to restore this vital function. The RSA and SEI Mega Programs (see sub-sub-recommendations 2c and 2d) would provide natural platforms for training and capacity building built more around standardized and sustained research mentoring. Similarly, the CGIAR needs to guard against mission drift into degree-based training and non­ degree distance learning (e.g., AGROCURI) for which it holds no comparative advantage, lacking a pedagogical mission, history and infrastructure. Given the steady decline of tertiary education in many low-income countries, the demand for such services is great. But the opportunity costs of CGIAR social scientists time diverted from other, higher return activities is likewise considerable. Furthermore, the risk of displacement of latent local university providers of such training is considerable. The CGIAR must support - not compete for - donors' direct investment in building training capacity directly in universities and NARS in low-income countries instead. The CGIAR will grow stronger as its partners grow stronger. 2c) Organize a Regional Systems Analysis (RSA) Mega-Program CGIAR social science today is plagued by too much "small think". The CGIAR has a hard- earned reputation for micro-level studies in rural areas of developing countries and it must continue excellence in that area. But it also needs to aggregate better to larger-scale, more strategic issues concerning agricultural development at regional and global scale, both for strategic research prioritization and for policy analysis and advocacy. The present void at larger scale impedes the emergence of a culture of evidence-based agricultural and rural policymaking, in both public and private spheres, which hampers the pursuit of CGIAR goals. 62 Technology, institutional and policy development take place within ecological, economic, political and socio-cultural contexts that far transcend specific commodities or natural resources. There is a delicate balance to be struck between veering off into exploration of that broader context - in which the CGIAR holds no comparative advantage - and appropriately contextualizing CGIAR research within the ecological and human systems within which it takes place. The latter is essential if technology development, institutional innovations and policy prescriptions are to succeed in improving the human condition. But CGIAR social science is not currently organized to do this. Further, the commodity or resource focus of most Centers discourages interest from many high quality social scientists, especially in the more qualitative areas (e.g., anthropology, law, political science, sociology), but even in economics and geography, as social scientists' interests typically revolve around people and landscapes more holistically. Attracting and keeping excellent social scientists is essential but extremely difficult when research programs are designed around a single commodity - or even a small number of commodities - in complex human and agroecological systems. The result is a paucity of high quality social scientists available to undertake much needed research on commodity-related issues, A move away from a commodity focus would come at little cost in terms of lost multidisciplinary collaboration with non-social scientists, as such interdisciplinary interactions are equally essential to understand regional systems. Framing more CGIAR activities in an integrative systems perspective should attract a broader range of talented field researchers interested in the behavior and well-being of peoples who generally pursue livelihoods that integrate multiple crops, livestock and non- agricultural pursuits. The natural and most desirable way to organize most CGIAR social science research is to complement small, Center-specific SSR teams with a network 6-8 region-specific systems analysis teams, each of 7-10 co-located IRS social scientists and natural scientists (probably ecologists, primarily) providing a platform for long-term, multi-location, high-quality, mixed methods data collection to track the evolving state of rural systems. This design would reinforce the essential link between biophysical and social science research, replicating at systems scale what presently takes place primarily at micro scale within single Centers. The Canada Excellence Research Chairs program (http://www.cerc.gc.ca/cpov-pcap-eng.shtml) offers a helpful model - whereby particular Centers compete to become focal points for research on specific domains of high strategic research priority over an extend term (seven years in the case of CERC), drawing top scientists through limited-term endowed chairs. The RSA network would be based at a 'CGIAR RSA Center of Excellence', with teams embedded at other Centers or partner institutions worldwide. Regions would be chosen to encompass agroecosystems of particularly high importance for agriculture and human welfare, situated in countries where the CGIAR already has considerable depth of experience. Each multidisciplinary regional team would monitor sentinel sites carefully selected to observe key economic and social processes in order to discern changing patterns of agriculture, resource availability and use, and welfare outcomes within regional-scale agroecosystems, marketsheds and populations. Each region would need sufficient locations to control adequately for statistical clustering effects, with fewer households and individuals within each location - and more locations - than is the norm for Center surveys today. Selection of sites would need to pay explicit attention to the prospective external validity of findings from those specific locations. Each site would support both qualitative and quantitative social science research using the best current approaches to mixed methods research design. This framework also allows for generalizable analysis at multiple scales of analysis, from intensive studies specific to a single 63 location - which can often generate high-value IPGs when done within a conceptual framework and research design that clearly draws out generalizable findings2 - to national, ecoregional and international level analysis using large-scale samples. The resulting data series would feed into research prioritization based on ex ante impact assessment in response to evolving constraints and opportunities in the system, including commodity-specific research prioritization. The sentinel sites would also provide natural locations for careful ex post impact assessment based on longitudinal monitoring and, where appropriate, randomized controlled trials using repeated experimental designs to more convincingly establish the impacts of CGIAR (and other) interventions. Each team would produce a biennial State of the [Specific] Region Agricultural Systems report summarizing relevant agricultural, economic, ecological, demographic, and social trends at regional scale and highlighting specific thematic issues of local importance, which might vary across regions some years and be coordinated in others, helping to prioritize at multinational level in a way that will impact Ministries of Finance and regional development banks as well as Ministries of Agriculture. The network would be organized under a cross-cutting Mega-Program that would (i) ensure sufficient standardization of data collection and analysis methods across regions to ensure comparability of results, (ii) aggregate, maintain and release usable meta-data, (iii) convene regular inter-regional meetings to engage in explicitly comparative analysis to identify global patterns, and (iii) feed the aggregated information into the global-scale analyses called for in sub­ recommendation 2c and use them to set the global agricultural development agenda. Other actors - e.g., the World Bank - currently fill that vacuum, although they lack the CGIAR's technical expertise. The CG could be far better organized to do global synthesis, partnering with the World Bank and other research bodies to run the sentinel sites, compile and analyze the data, and disseminate findings to stakeholder communities at local, national, regional and global levels. The RSA Mega Program would equip the CGIAR to become the global repository of agricultural and rural data if it could establish an interoperable database system that includes study design documentation, raw data, studies that employ those data, and related background studies and data. This would be an immensely valuable IPG that would likely attract broader scale interest in CGIAR research topics by external scholars, thereby leveraging CGIAR human capital considerably. It would also guard against the considerable loss of information that takes place annually in the CGIAR as staff move and data or essential documentation is lost. The Human Studies Database Project (http://rctbank.ucsf.edu/home/hsdb.html), recently funded by the United States' National Institutes of Health, provides a cutting-edge example of a related informatics project for large-scale data sharing and reuse. A model for the RSA Mega-Program might be the United States National Science Foundation's Long-Term Ecological Research Network (LTERNet, http://www.lternet.edu/). LTERNet currently operates a network of 26 sites using a core of standardized research protocols. Sites are competitively chosen and are subject to review every several years by the scientific governing board representing each of the sites. The network holds an annual meeting to explicitly integrate 2 Famous examples of intensive, highly local social science studies that generated major IPGs include James Scott's Moral Economy of the Peasant (1976) and Christopher Bliss and Nicholas Stern's Palanpur: The Economy of an Indian Village (1982). The key to IPG generation is not the spatial scale of analysis by that the research project has clear external validity internationally by virtue of its design. 64 findings across the various sites, lest they merely run in parallel, as is true of CGIAR social science research today. The RSA network of sentinel sites could be developed and maintained in concert with the World Bank's LSMS-ISA project in Africa and with several other current initiatives by developing country governments and ARIs (e.g., in China, Colombia, Ghana, India, Vietnam) to build high quality longitudinal (panel) household data. CGIAR Centers would not have to directly manage all the sites. Indeed, it would be desirable to draw in ARI and NARS partners explicitly in collaborative work agreements around sentinel sites. The CGIAR is uniquely positioned to lead an effort focused on long-term monitoring and analysis of rural communities and agroecosystems in the developing world if it can establish effective funding and management mechanisms - as should be feasible under a core-funded Mega-Program - and create incentives and funding for the protocol standardization, meta data compilation and results dissemination necessary to create a true international public good from the data collection and analysis efforts. The RSA research sites would also provide an extraordinary opportunity for long-term capacity building among NARS partners on field research and data analysis methods, as well as for inter­ regional cooperation based on adequately standardized methods. The resulting meta-data would be an invaluable IPG, the social science equivalent of the CGIAR's germplasm and animal genetic material collections for the biological sciences. 2d) Organize a Mega-Program on Stimulating and Evaluating Innovations (SEI) Stimulating innovations is the essence of what the CGIAR does in advancing poverty reduction through agricultural research; evaluating innovations is essential in order to use scarce resources to best effect in that mission. CGIAR social science has always had a central mission of helping to foster and document increased uptake of innovations generated by the System. This Mega- Program would be based at a 'CGIAR SEI Center of Excellence' and integrate what are currently disjoint efforts at social science activities aimed squarely at stimulating innovations - e.g., participatory plant breeding, linking farmers to markets - and System-level ex ante and ex post impact assessment and put greater emphasis on advancing methodological frontiers in these areas in order to improve the impact of CGIAR research on desired development outcomes. IFPRI provides some of the ex ante research prioritization capacity through its IMPACT model and Harvest Choice program. And some Centers produce commodity-specific outlook reports that play a similar role. But these efforts are insufficiently integrated right now. And they draw very little on longitudinal research on demographic, economic, ecological and social trends of the sort advocated in sub-recommendation 2c (the Regional Systems Analysis Network Mega- Program), even though such dynamics fundamentally affect the demand for and supply of agricultural commodities and the inputs that produce them, as well as poverty and food insecurity patterns. The SEI Mega Program would essentially reinforce SPIA's present functions and emphasize the development and application of more rigorous, selective and multi-dimensional (i.e., economic, environmental and social) ex post impact assessment (EPIA), freed of some of the Center-level conflict of interest problems and donor pressures that bedevil current EPIA work. Its EPIA research would concentrate multidisciplinary teams using more sophisticated, larger-scale - internationally comparable, rather than project-specific - research designs to generate significant new actionable knowledge as to what works, what doesn't, where, why and with what returns. The sentinel sites from the RSAN Mega-Program would provide natural locations for carefully 65 controlled EPIA. Explicitly integrating ex ante and ex post impact assessment under a single Mega-Program would increase the demand for and uptake of high quality EPIA research and stop overburdening line researchers to generate what are, quite predictably, small-scale, one-off EPIAs that lack external validity and are typically of poor quality, thus effectively generating no IPGs. The SEI Mega-Program would also be the natural home for a small initiative exploring more systematically how institutional designs condition the emergence and uptake of innovations having real development impact. The objective would be the specification of explicit hypotheses and design of rigorous tests to establish how different partnership models and processes affect the development, dissemination and impact of agricultural & NRM technologies. That would likely encompass a scaled down version of what is now IFPRI's Knowledge Innovation and Capacity Division - formerly ISNAR - and the ILAC initiative, recast somewhat with disciplinary expertise presently absent from the CGIAR (e.g., in communications, networks analysis and organizational behavior). And it would need to build strong partnerships with ARIs with extant capacity (e.g., Harvard, UNU). The objectives of this activity would be to stimulate increased innovations in agriculture and NRM directly and to identify what institutional arrangements yield the most effective stimulus under different conditions. There would be great value to research designs that looked at synergies or tradeoffs among technological, NRM and policy or institutional interventions through designs that carefully employed different combinations of interventions believed (through ex ante assessment) reasonably likely to succeed individually or jointly. Only the CGIAR could undertake such large-scale integrative work and this sort of high-impact, complex research should inform research priority setting. But it is just not happening with highly decentralized, project-level impact evaluation work disconnected from priority setting research. An SEI Mega Program could coordinate such strategic research. This sort of upscaling will require standardized protocols like those used in other multi-national primary data collection networks (e.g., the International Forestry Resources and Institutions program). All partners agree to administer standardized instruments, but are free to add questions and other supplementary data collection exercises to meet their specific research objectives. This allows comparisons across contexts, as is needed to establish truly general patterns, while still allowing for tailored local research appropriate to the context and to advancing high quality System-wide social science on stimulating and evaluating innovations. 2e) Shrink unproductive social science units CGIAR social science suffers from dilution of quality and focus. Centers are trying to do too much and as a result, are doing too little of it very well. There are a few essential social science tasks that need to be addressed with in-house staff at any Center: technology evaluation, adoption monitoring, systems characterization, etc. But in every commodity and NRM Center the panel studied, those essential support tasks could be effectively accomplished with one or two Ph.D. social scientists directing a team of half a dozen or a dozen strong, M.S.-level nationally or internationally recruited staff and leveraging ARI and NARS partners. That is essentially the model of the successful commodity Centers in the early years of the CGIAR. Growth in social science staff has not expanded Centers' capacity to do research at scale for the simple reason that Centers' natural commodity or agroecosystem mandates are limiting. All Centers - with the possible exception of IFPRI - are struggling to do the integrative work needed 66 for the CGIAR to lead to impact at scale. Hence the need for something like the RSA Mega- Program proposed in sub-recommendation 2c, to supplement small, commodity-focused or NRM-focused teams working at relatively micro scale with small, systems-oriented teams working on integration at larger, regional and global scales. Today, at most 8 of the 15 Centers have strong, potentially sustainable social science capacity. Lack of critical effective mass in many Centers and regional offices is a large part of the problem, making it harder to recruit top flight social scientists who understandably do not want to be intellectually isolated. But Centers cannot attract large scale, cutting edge projects without the minimum size and quality of staff; and the absence of such projects makes it harder to attract the best and brightest. The self-reinforcing dynamic of the critical mass problem acts to trap many Centers and CPs in a low-quality, small-scale social science equilibrium from which the CGIAR needs to break free. Unless the System or individual Centers move to the CGIAD model, social science groups at such Centers are just too small to effectively advance the broader CGIAR Mission. Plus, the duplication of activities and sometimes-cutthroat Inter-Center competition for scarce short-term project funding generates considerable directly unproductive activity. At current staffing levels outside of IFPRI, there are enough social scientists for no more than 8- 10 high quality social science units in the System. So the CGIAR needs either a massive expansion of social science - which is unlikely and seems inadvisable since social science staff already represents 27% of IRS in the System - or consolidation to achieve the minimum necessary scale to maintain high quality research and remain adequately focused, thematically and geographically. The panel strongly recommends against complete consolidation of CGIAR SSR, presumably into a radically expanded IFPRI, because there are diseconomies of scale as well. A single mega- Center for social science would sharply reduce close interactions with biophysical scientists, which is the single greatest source of comparative advantage for CGIAR social science and absolutely essential to achieving sustainable productivity growth and poverty reduction through improvements to agricultural productivity and NRM. Co-location of social and biophysical scientists is vital to the broader research mission of the CGIAR, but a sufficient critical mass of social scientists is also needed, so that social scientists may be adequately stimulated by colleagues and can effectively maintain their skills and quality of their work. As such, neither the status quo nor the alternative of a social science mega-Center are advisable. The panel recognizes that this sub-recommendation will be difficult to implement. But the problem of efficient and effective allocation of limited resources requires hard choices and new approaches. If, as expected, the MPs increase core resources, it brings a propitious moment for the reform of social science staffing at Centres. The creation of a Mega-Program on Regional Systems Analysis (sub-recommendation 2c) could deploy multidisciplinary social science teams that simultaneously serve multiple Centers operating in the same agroecosystems, obviating those Centers' need for dedicated social scientists working on broader institutional and policy issues. Likewise, a Mega-Program on Stimulating and Evaluating Innovations (see sub­ recommendation 2d) could similarly deploy multidisciplinary teams to work with Centers and their external partners on research prioritization, identification and facilitation of impact pathways, and ex post impact assessment. That expertise does not need to reside within each Center, especially if diffusion comes at a steep cost in terms of quality. 67 Recommendation 3: Update social science personnel management practices The key input to any research enterprise is skilled human capital. Thus recruitment and retention of the world's best researchers with relevant interests is the highest order priority for CGIAR social science. As indicated earlier, field-based, empirical research is resurgent within many of the social sciences today. This opens up exciting possibilities for CGIAR recruitment of highly qualified staff with field experience and relevant, rigorous toolkits. But the CGIAR must make some substantive reforms in personnel management in order to realize those possibilities. The SSSR panel has five specific sub-recommendations that would be important steps in the right direction. 3a) Introduce a CGIAR Young Scientists Program The Rocky Doc program was invaluable for recruitment of high quality young social scientists - especially in the non-economics disciplines - into the CGIAR. The System has not recovered from that program's termination. The CGIAR needs to reintroduce a 2-3 year post-doc program like the old Rocky Doc. A CGIAR Young Scientists Program (YSP) could also include young MS recipients from low-income countries, a cohort that is seriously underutilized in the System today. At US$4-5 million/year, roughly one percent of the System's budget, the CGIAR could recruit a steady stream of 20 new YSPs annually (and save significantly on Center-level recruitment costs). YSPs would feed into Centers or Mega-Programs through a matching process similar to that the Rockefeller Foundation employed in the Rocky Doc program. Two years gives Centers ample time to assess whether individual scientists would make good permanent staff and a natural way to let those who would not exit the System. It would likewise increase the cachet of placements within the CGIAR and reduce some of the present problems Centers run into from project-driving hiring cycles. A simple System-level office, rather like that organized to run the AWARD program, would handle recruitment, screening, matching and regular YSP conferences every 12-24 months, which could be held in conjunction with regular social science association conferences so as to encourage engagement with the broader global research community and to facilitate recruitment. Much as was the case for the Rocky Doc program, the YSP would improve the CGIAR's capacity to recruit high quality skills (e.g., GIS and spatial analysis, intellectual property law, networks analysis, organizational behavior) from outside its traditional domain of agricultural and development economics. Not only would a YSP program provide a more reliable stream of high-quality, vetted social science researchers, the program's alumni would also comprise a network of long-term collaborators and external supporters within ARIs and donor agencies, just as the Rocky Doc program did so effectively. 3b) Increase entry-level compensation packages With or without a YSP (recommendation 3a), the CGIAR needs to raise salaries and benefits for entry level IRS social scientists. Centers are almost universally uncompetitive at the new-PhD level, relative to ARIs and multilateral institutions, even some NGOs. If they cannot attract talented new blood, the System will continue to deteriorate. By contrast, there is no clear evidence that at mid- or senior level compensation packages for social scientists are non­ competitive. 68 3c) Establish a clear research career track for social scientists While compensation packages are not a Systemwide issue for mid-career social scientists, access to a true research career track beyond 10 or 12 years in the System certainly is. With the exception of IFPRI, whose size allows it to have mid-career social scientists maintain active research programs, Center social scientists almost invariably either leave the CGIAR or move into program and division management in order to maintain a career in the CGIAR beyond 10- 12 years, and virtually abandon any significant personal research program. Not all good researchers want to become managers, however, and many lack the personal and professional skills to serve effectively in managerial roles. Drawn by administrative talent or by default, mid­ career social scientists occupying managerial positions represent a loss to the research base at Centers. At least five ingredients are vital for the establishment of clear research career tracks at Centers: 1) retention of senior social scientists to lead cutting edge research and to mentor a healthy inflow of bright young scientists that form a critical mass in social sciences; 2) protection of senior scientists' time from administrative exigencies unrelated to their research; 3) an opportunity for social scientists to focus on important medium-to-long-term research problems, rather than almost only on short-term, one-time development contracts; 4) opportunities for periodic scientific re-tooling and intellectual renewal; 5) a promotional structure that emphasizes rewarding high quality social science research, particularly those scientists making path­ breaking advances. If Centers and MPs do not develop clear research career tracks in the social sciences, retention and recruitment problems at all levels will continue. This issue applies to nationally recruited staff (NRS) as well as to IRS, albeit with compensation as a central part of the NRS career track issue. The best of the NRS should be able to ascend to positions with compensation and substantive responsibility similar to those of junior-to-mid- level IRS. Pay and authority scales should be overlapping rather than disjoint, as they commonly are in Centers today. 3d) Restore competitive travel and sabbatical programs It is essential for CGIAR social scientists to maintain connections with the international research community and to update technical skills through regular attendance at professional meetings, participation in global scientific networks, and sabbaticals. Such perquisites were frequent under the old business model, with adequate core funding, and need to be restored. Awarded competitively to researchers who consistently generate high quality outputs that lead to impact, small investments in these opportunities yield high dividends and are natural means to enhance partnerships with ARIs (see sub-recommendation 2a). 3e) Employ modern human resources management practices The panel applauds the marked improvements observed in working conditions for female social scientists over the past decade within the CGIAR. But there remain unnecessarily long delays in posting vacancies for IRS positions and in basic introductory functions for new hires (e.g., issuing identification cards, arranging entry visas and working papers). Many HR units appear not to stay abreast of conditions in the labor market for Ph.D. social scientists. They must also stay vigilant and highly responsive to concerns about unprofessional behavior by managers. 69 Recommendation 4: Foster a culture of rigorous social science research Critical intellectual mass and strong ARI linkages are necessary but certainly not sufficient to upgrade CGIAR social science research quality. Centers must introduce clear incentives and inculcate a culture of rigorous research in order to sustainably upgrade CGIAR social science. The SSSR panel has four specific sub-recommendations in this regard. 4a) Restore social science research seminar programs Several Centers maintain active seminar series for sharing research designs, methods and results internally. The panel applauds this. But the panel was struck by the remarkably limited awareness within several other Centers of colleagues' research. A regular, active seminar series offers a tried-and-true, low-cost approach to building intellectual communities. A fortnightly seminar organized around disciplinary lines can maintain critical mass even among units of only half a dozen IRS, giving visiting scholars and students as well as Center staff a platform to share technical work in progress. Some sessions can be used for skills upgrading sessions by visiting experts. Those Centers that do not have social science research seminars should restore or establish them. At the same time, it is essential to maintain or establish regular interdisciplinary workshops or seminars so as to maintain - or in some cases restore or enhance - strong intellectual interactions between natural and social scientists outside the context of specific projects. These interactions should enable social scientists to carefully explain to natural scientists the specific approaches, concepts, methodologies, and underlying assumptions used in social science research. It was clear to the panel that many natural scientists in the System still see social science (especially other than economics) more as a source of insights on local customs and protocol or as a perspective on complex social phenomena than as scientific disciplines with established theory and methods for specifying and testing hypotheses and the capacity to design rigorous research and generate robust findings that can contribute to tangible outcomes and impacts. A culture of active, widely attended seminars can do a great deal to advance mutual understanding and respect across disciplinary boundaries. Since this is the bedrock of CGIAR comparative advantage, management should prioritize such interactions within Centers and CPs. 4b) Individual performance measurement High standards need to be set for individual research performance, but without relying exclusively on narrow measures that distort incentives to do high quality work that contributes towards real development impact. Since impact takes quite some time, however, intermediate indicators are necessarily, even if imperfect. The panel strongly recommends an emphasis on peer-reviewed publication as a standard research output for IRS social scientists, both as a means of quality control and for leveraging external science. Progress reports, research reports, technical bulletins and other in-house publications should count, as should working papers or discussion papers that are typically intermediate steps toward peer-reviewed publications. But these latter outputs should be given significantly lower weight in assessing research productivity as reflected in publications. Of course, if one only measures and rewards publications counts, quantity will trump quality, as seems to be happening currently. So the panel recommends that Centers also measure and 70 reward citations, professional honors and other metrics of professional impact. CGIAR social science needs to influence the broader scientific research agenda in order to mobilize global science around the CGIAR mission. Citations and professional (extra-CGIAR) honors, while imperfect, are solid indicators that it does. When partners and students one trains publish or earn awards for their work with a CGIAR social scientist it reflects well on the quality and impact of capacity building and maintenance activities. These impacts should be reported, recognized and rewarded. The System must also measure and reward data downloads and other scientists' use of data in publications. The production and dissemination of high quality, geo-referenced, usable data and meta-data is a heavily undervalued IPG, thus it is seriously undersupplied. If researchers begin to earn recognition for producing data that gets used, they will have better incentives to collect, clean, document and disseminate high quality data. 4c) Stop wasting money on in-house printed publications other than policy or research briefs In a bygone era when publications were largely inaccessible to developing country researchers and policymakers, in-house publications in hard copy filled an important niche. That is no longer true. While short non-technical policy and research briefs - distributed electronically or in hard copy - are heavily used, the data show that Centers' in-house research publications series are rarely cited and a range of stakeholders told the panel that such publications are rarely consulted, with a few prominent exceptions. CGIAR scientists should be relying on refereed journal articles, books and book chapters as their published outlets - with web-accessible discussion and working papers as temporary outputs until papers get published - and briefs for non-technical audiences. Only rarely should there be need for technical reports or outreach bulletins and those can typically be posted online and printed on demand to minimize printing and distribution costs. Use the external peer review processes of journals and publishers; stop paying to replicate the same system at lower quality and higher cost. 4d) Establish a CGIAR Institutional Review Board As discussed previously, current best research practices in the social sciences - and many scientific grantsmaking bodies - require Institutional Review Board (IRB) approval prior to the commencement of any data collection from human subjects. Yet few CGIAR Centers or CPs understand or use an IRB. This must be corrected swiftly. There is, however, no reason for each Center to establish its own IRB. A System-level IRB could ensure standard, appropriate protections of human research subjects using a simple electronic platform for training researchers in research ethics, receiving approval and renewal requests, and sharing best practices on consent scripts, data handling practice, etc. A System-level IRB would also help the development of meta-data, which requires common data handling protocols to enable data integration across research projects. A simple, punctual CGIAR IRB applying international best practices would be relatively easy and inexpensive to implement and would help shore up social science research quality in those areas where it is weak. 71 6. CONCLUSIONS The CGIAR desperately needs high quality social science in order to achieve the laudable and much needed impacts on poverty, food insecurity and environmental degradation to which the CG aspires. It is equally true that globally, the social science of development needs the CGIAR, which has, for decades, contributed important data, findings, capacity building and applied research that have significantly shaped international research on agricultural and rural development, natural resources management, and broader human well-being. Today, however, the model for producing high quality social scientific work within the CGIAR is broken. The System's new business model is mismatched with both staffing patterns and the impacts the CGIAR Mission seeks to achieve. This is compromising staff and research quality, and appears unsustainable. There has been growth in the number of social scientists in the CGIAR. But it reflects increased loss of focus on CGIAR comparative advantage in the social sciences. There is high heterogeneity and low average level of quality in research, capacity building and support to non-social science research within the System. In order to enhance its social science activities and accomplishments in support of the System Mission, the CGIAR must take major steps. The four general recommendations the panel offers, if followed, can help the System to seize the attractive opportunities presented by resurgent global demand for high quality research on international agriculture and related questions of rural development, climate change, land and water management, genetically modified organisms, and food policy. Without fundamental reforms to sharply increase the unrestricted (or quasi-unrestricted) share supporting scientists, to more tightly focus on its areas of comparative advantage, and to attract and retain the best social scientists with interests in agricultural and rural development, however, the likelihood of realizing the considerable potential of CGIAR social science will be low. This need not be. The time is ripe for change in CGIAR social science and the likely rewards to effecting such change are great. 73 REFERENCES Adato, M. and R. Meinzen-Dick, eds. (2007). Agricultural Research, Livelihoods and Poverty. Baltimore: Johns Hopkins University Press. Alston, J.M., T.J. Wyatt, P.G. Pardey, M.C. Marra, and C. Chan-Kang (2000). A meta-analysis of rates of return to agricultural R & D: Ex pede Herculem? Research Report 113. Washington: International Food Policy Research Institute. Amin, M. and M. Mabe (2000). Impact Factors: Use and Abuse. Perspectives in Publishing No. 1. Oxford: Elsevier. Barker, R. (1988). Methods for Setting Agricultural Research Priorities: Report of a Bellagio Conference. Cornell University Working Papers in Agricultural Economics no. 88-3. CGIAR Independent Review Panel (2008). Bringing Together the Best of Science and the Best of Development. Independent Review of the CGIAR System. Report to the Executive Council. Washington, DC. CGIAR Science Council. 2005. System Priorities for CGIAR Research 2005-2015. Rome: CGIAR Science Council Secretariat. CGIAR Science Council. 2008a. Ethical challenges for the CGIAR: Report of three studies. Rome: CGIAR Science Council Secretariat. CGIAR Science Council. 2008b. Impact assessment of policy-oriented research in the CGIAR: Evidence and insights from case studies. Rome: CGIAR Science Council Secretariat. CGIAR Science Council. 2009. Defining and Refining Good Practice in Ex-post Impact Assessment - Synthesis Report. Rome: CGIAR Science Council Secretariat. Conway, G., A. Adesina, J. Lynam and J. Moock (2006). The Rockefeller Foundation and Social Research in the CGIAR. In M.M. Cernea and A.H. Kassam, eds. (2006). Researching the Culture in Agri-Culture: Social Research for International Development. Wallingford, UK: CABI Publishing. Deaton, A. (2009). Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development. NBER working paper #14690. Cambridge, MA: National Bureau for Economic Research. Hewitt Associates (2005). CGIAR - Consultative Group on International Agricultural Research Phase II - Review of Total Compensation: Report of Findings and Commentary. Mimeo. Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences 102 (46): 16569-16572. Kanbur, R., ed. (2003). Q-Squared: Combining Qualitative and Quantitative Methods Of Poverty Appraisal (Delhi: Permanent Black). Kassam, A.H. and M.M. Cernea (2005). Studying the Culture of Agri-Culture: The Uphill Battle for Social Research in CGIAR. Culture and Agriculture,27(2):73-87. Kelley, T., J. Ryan and H. Gregersen (2008). Enhancing ex post impact assessment of agricultural research: the CGIAR experience. Research Evaluation 17(3): 201-212. King, E.M. and J.R. Behrman (2009). Timing and Duration of Exposure in Evaluations of Social Programs. World Bank Research Observer 24(1): 55-82. Kristjanson, P., R.S. Reid, N. Dickson, W.C. Clark, D. Romney, R. Puskur, S. MacMillan and D. Grace (2009). Linking international agricultural research knowledge with action for sustainable development. Proceedings of the National Academy of Sciences 106 (13): 5047-5052. Larivière, V., Y. Gingras and É. Archambault (in press). The decline in the concentration of citations, 1900-2007. Journal of the American Society for Information Science. 74 Lynam, J. (2007). Research into Development: Assessing CGIAR Research Priorities from the Perspective of Development Challenges. Mimeo. Pardey, P.G., N. Beintema, S. Dehmer, and S. Wood (2006), Agricultural Research: A Growing Global Divide? IFPRI Food Policy Report No. 17. Raitzer, D.A. and T.G. Kelley (2008). Benefit-cost meta-analysis of investment in the International Agricultural Research Centers of the CGIAR. Agricultural Systems 96 (1- 3): 108-123. Renkow, M. and D. Byerlee (2009). The Impacts of CGIAR Research: A Review of Recent Evidence. Mimeo. Spielman, D.J. (2006). A Critique of Innovation Systems Perspectives on Agricultural Research in Developing Countries. Innovation Strategy Today 2(1): 41-54. TAC Secretariat (1996). Perspectives on Policy and Management Research in the CGIAR. Rome: CGIAR Technical Advisory Committee. Waibel, H. and D. Zilberman (2007). International Research On Natural Resource Management: Advances in Impact Assessment. Wallingford, UK: CABI. World Bank (2003), The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research. Washington: World Bank. World Bank (2007), World Development Report: Agriculture for Development. Washington: World Bank. Wydick, B. (2008). Games in Economic Development. Cambridge: Cambridge University Press. 75 Annex 1 Terms of Reference Phase 2, Stripe Review of Social Science in the CGIAR October 2008 Background At SC08, Septemeber 2007, the Science Council (SC) proposed to undertake a Stripe review of the Social Science research in the CGIAR.1 The reasons for and the main objective of the Social Science Stripe Review (SSSR) were laid out in a scoping paper.2 The Executive Council in its 13th meeting approved the review plan as part of the SC's workplan for 2008-2009. The purpose of the social science stripe review is to provide an analysis of the capacity of Centers and Challenge Programs (CPs) to conduct social science research and of the quality and relevance of the social science research regarding its functions and products, and to make recommendations for improvement to enhance the delivery of relevant research results that effectively advance the CGIAR's goals. The stripe review was planned in two parts. The first phase has been conducted as a desk study. It generated two interrelated outputs a) a background report on the current status of social sciences in the CGIAR (URL when available), and b) a normative framework describing a putative CGIAR social science research agenda against which the current capacity and relevance of CGIAR social science can be assessed (URL when available).3 Despite limitations related to data (gaps in data; limited and inconsistent data on expenditures; difficulty of comparisons over time) some broad diagnostics emerged from the desk study: • Over the past decade or two, the agenda for social science research in CGIAR has expanded considerably, beyond the adoption-impact assessment nexus to include such factors as impact of agricultural technology on poverty, livelihoods, equity, environment, gender and nutrition, in addition to the role of agricultural technology in the broad context of rural development, the role of policies, institutions and technology in natural resource management, and a growing portfolio of policy and management research. • The number of social scientists in the system seems to have increased, but trends are not clear due to changes in classification of staff, methods used to collect information on staff and blurring of disciplinary specifications. In 2007, 27% of scientists in the CGIAR were social scientists or otherwise conducting social science research. • Credible estimates of expenditure on social science research were difficult to get but at the System's level expenditure to activities classified under policy research has 1 "Social sciences" comprise those disciplines that apply scientific methods to the study of individual and collective human behavior and outcomes. This includes anthropology, economics, geography, law, political science, psychology, and sociology, as well as derivatives of those fields (e.g., agricultural economics, rural sociology) and social science-oriented branches of the biological sciences (e.g., community nutrition) 2http://www.sciencecouncil.cgiar.org/fileadmin/user_upload/sciencecouncil/Monitoring_and_Evaluation/Stripe_ Review_scoping_paper_Dec_2007.pdf 3 A Normative Framework for Social Science Activities in the CGIAR. Chris Barrett, October 2008 (URL when available). A-l increased in real terms since 1992 from less than 30 million to about 45 million in 2006 (1992 US$). Among the social scientists economists are the largest disciplinary group (65%). The second largest group is comprised of staff with no formal social science training (16%) yet working on social science research topics. The numbers in the various non­ economist social science disciplines (sociology, anthropology, geography, policy, law) are small (19% in total) and fail to reach a critical mass. • A large portion of activities is in policy research and surprisingly limited attention is given to technology development and adoption issues. • Centers claim largest impact in influencing priority setting and in multidisciplinary work; less disciplinary impact. • Compared to ARIs, the CGIAR social scientists are younger and relatively inexperienced (judging by year when highest degree was earned); there is a reasonable (by ARI standards) proportion of women. • There are indications that increasing dependency on restricted funding may drive staffing patterns (including recruitment and retention) and may influence choice and quality of research. • Partnerships cover a broad range of institutions but ARI seem less common. • The publication output is variable and in general few are in high quality and high impact outlets. Even within the same journal, the Center publications are cited at substantially lower rates than other articles. There is a large portion in grey reports which have not undergone peer review. In addition to these observations many recent Center commissioned external reviews (CCERs) and external program and management reviews (EPMRs)4 point to weaknesses in social science research capacity, productivity, research quality and the IPG nature of the products. The 2nd output of the Phase 1 study - The Normative Framework -highlighted the comparative strengths of social science research of the CGIAR as: • The Centers' location in developing countries facilitates highly contextualized, problem-driven research that is both more difficult and more costly to undertake from ARIs based in the more developed countries. • A large cadre of highly-trained, internationally-recruited staff enables Centers and CPs to undertake more advanced and complex lines of research than many - but certainly not all - developing country NARES can tackle. • The multidisciplinary nature of most Centers and CPs and the relatively low disciplinary boundaries within them make it relatively easier to organize diverse teams well-equipped to address inherently interdisciplinary challenges beyond the reach of more narrowly staffed research institutes or faculties. 4 Cited in the scoping paper and Phase 1 Background paper. A-2 The paper also provided a framework of three key priority research areas for SS research, derived from the sources of CGIAR comparative advantage in social science research. Specific lines of research activities in each priority research area are identified in Appendix 1 These tentative findings and the normative framework are being used to guide the TOR of the main phase (Phase 2) of the review. Terms of reference for the Main Phase (Phase 2) 1. The objectives of the SSSR are to provide a comprehensive assessment of the CGIAR's social science research agenda by assessing i) the relevance of the social science research regarding its functions and products with focus on three key areas elaborated below; ii) the capacity and incentives in the CGIAR for conducting high quality research that can contribute to the CGIAR goals, and iii) the opportunities for improving the organization and partnership models for enhancing the impact of social science research. The review aims at suggesting new ways of thinking about how to improve social science research in the CGIAR System, contributing to a forward-looking design for social science research under the new integrated strategy and results framework emerging from the Change Management process. 2. The review should focus on the assessment, at System level, of the current quality, coverage, relevance and productivity of social science activities to CGIAR priority research, with a focus on three key areas social science research: a. social science research support for productivity growth and poverty reduction via technological innovation. This focus area should include assessment of the social science contribution to rural livelihoods enhancement through natural resources management. b. social science research support for productivity growth and poverty reduction via institutional innovation and policy analysis. c. social science research for informing agricultural and rural development policy. The assessment will cover any social science disciplines that are involved in these key areas and, to the extent possible within the scope of the review, relevant linkages to other disciplines. It will require filling in essential information gaps from the phase 1 descriptive report through both new data collection from Centers and CPs, as well as visits to selected Centers and CPs. It should capture aspects of productivity and effectiveness through assessment of the quantity and quality of publications, quality of linkages among scientists within the Center, collaboration with advanced institutions and assessment of outcomes and impacts from the social scientists' work. It should explicitly address emergent research in innovation systems (and related topics) as well as more traditional lines of research. It should explicitly examine and assess the awareness and use of appropriate and state-of-the-art methods by System social scientists. Furthermore the review should include: o Comparison of social sciences research prioritization and management within the CGIAR against current global best practices, including: how the demand for social science research is derived and articulated at the System, Center and Program levels; research ethics for the protection of human subjects, management of meta-data, project and staff monitoring and evaluation, use of information and communications technologies for transnational project management, etc. A-3 o Analysis of the organization, financing and staffing of the social sciences in support of CGIAR priority research, with an eye to disciplinary and skill mixes necessary to address emerging new challenges related to, for example, biofuels, climate change and genetically modified organisms. The recommendations of the Change Management and Independent Review processes already underway will need to be taken into consideration in this analysis. o Explicit attention to collaboration between IFPRI and the other Centers, looking especially for ways of potentially generating greater synergies from the interface between the critical mass of social scientists at IFPRI, and the critical masses of natural scientists and small groups of social scientists at the other Centers and in light of the Change Management reforms. 3. The output of the review. The output of the review is clear, doable recommendations based on the analyses in points 2a-f above, for strengthening social sciences activities in the CGIAR, paying specific attention to emergent priorities as well as current social science activities in which the System holds, or does not hold, comparative advantage globally in light of the current and prospective array of alternative suppliers. The recommendations should clearly identify the System's core competencies and appropriate boundaries for social science activities in the CGIAR. 4. Panel Membership In consultation with the review panel Chair, two or three additional members will be selected by the SC to complete the review panel for phase 2. The new members will add disciplinary diversity to the panel and where possible a familiarity of the relevant issues at the local national level. 5. The Work plan The Panel members will be recruited in November 2008. The panel will provide an interim report to SC 11 (March, 2009) and a DRAFT of the final report by July 15, 2009. An SC ad hoc team will provide a quality audit of the DRAFT report. The DRAFT will then be shared with the CGIAR Centers and CPs for factual commentary before being presented to SC11. The normative framework paper and the revised Phase 1 background report will be annexed to the main SSSR report. Over the course of the nine months from November 2008 through July 2009, the panel will hold an initial virtual meeting, engaging through appropriate means with the Center and CP focal persons, to discuss the purpose and process of the review and to get the Center focal persons' input at an early stage, as well as commitments to cooperation in the provision of essential data not generated during the phase I study. The panel - at least two of three members in each case - will visit at least four Centers/localities for at least two or three days each: o IFPRI, as the largest social science research Center and host of one of the first CPs (FFarvest Plus). This should be the first Center visited and should be preceded by a day-long meeting of the full panel. o ILRI and ICRAF as hosts for visits to the Nairobi-based complex of Center s' research teams in Sub-Saharan Africa and, next to Washington, the largest geographic center of CGIAR SSR. A-4 o CIMMYT or IRRI as representative of a major commodity-focused Center. o IWMI, CIFOR and/or ICRISAT to address more issues more specific to natural resources management and ecoregional centers and to Asian concerns. The panel will use surveys (to be coordinated by the SC Secretariat) and interviews to collect information from a range of personnel within the CGIAR and more broadly social scientists and research managers in ARIs, NARES and elsewhere as deemed necessary. The review panel will be supported by the CGIAR Science Council Secretariat (with Sirkka Immonen as the point of contact for the review). The panel will work in close consultation with the SC Task Force that acts as a resource body to the review and provides SC oversight and with the focal persons designated by each Center and CP. A-5 Appendix 1: CGIAR Comparative Advantage CGIAR Social Science Research Focus Specific Lines of Research Activities Multidisciplinary research on Productivity growth, poverty reduction and Systems and farmer characterization work agricultural productivity growth by and sustainable NRM via technological innovation, Participatory plant breeding for the poor. in close collaboration with natural scientists. Ex ante impact assessment for priority setting at project and program levels Close interaction with stakeholders at Technology adoption studies to establish cross-sectional local, national and global levels. and intertemporal patterns of uptake and adaptation of CGIAR innovations Producer of significant new intellectual Ex post impact assessment for evaluation and property in agricultural sciences. institutional learning Natural resources management (NRM) for enhancing and sustaining productivity growth Emergent research on innovation systems and impact pathways including innovations that improve the efficiency and effectiveness of global research system Intellectual property rights management Close interaction with stakeholders at Productivity growth poverty reduction and Investigation of market and non-market resource local, national and global levels. sustainable NRM through institutional allocation mechanisms innovation. Sociocultural constraints on and incentives for productivity-enhancing innovations Collective action and property rights Agricultural input and output distribution systems, including farmer-based organizations The design and management of agricultural and NRM research institutions, whether around intellectual property rights, impact assessment methods, or related topics_____________________________________________ A_£ Unique combination of international- Productivity growth and poverty reduction and Policy analysis related to international agricultural caliber technical expertise and its by directly informing agricultural and rural trade, domestic and regional agricultural input and multinational nature. development policy. output markets Intellectual property rights and agricultural research Widespread "honest broker" perception policy, and the policies for the conservation of animal for policy guidance.___________________ and plant genetic resources A-7 Annex 2 Panel Member Biographical Details Professor Arun Agrawal (http://www-personal.umich.edu/~arunagra/') Arun Agrawal is Professor and Associate Dean of Research in the School of Natural Resources and Environment at the University of Michigan, Ann Arbor. Prior to teaching at the University of Michigan, Agrawal taught at McGill University, Yale University, and University of Florida. Trained as a political scientist at Duke University, Agrawal's research focuses broadly on the political economy of natural resource management and development; more specifically, he works on community-based natural resource governance, sustainable development, indigenous knowledge, adaptation to climate change, decentralization, and environmental governance. He has written more than 50 peer reviewed papers and books related to institutional analysis, rural development, sustainable development, decentralization of environmental policy, environmental governance, climate change, and forest management. His journal articles have appeared in such outlets as Science, World Development, Development and Change, Conservation Biology, and Current Anthropology, among other journals. He has also authored and edited 5 books published by different university presses in the United States. His consulting work has spanned organizations such as the World Bank, the USAID, the Wildlife Conservation Society, the International Development Research Center, and the United Nations. He is currently assisting in the preparation of the 2010 World Development Report on development and climate change as a member of the WDR 2010 core team. He is also the coordinator of the International Forestry Resources and Institutions network. This network is a collaborative of research institutions focusing on local governance of forests in 12 countries in South Asia, East Africa, and Latin America. His other major research projects concern Climate Adaptation, Livelihoods and Institutions (CALI - a comparative study of adaptation to climate change in six Latin American and West African countries supported by the World Bank); and the Central African Forests Initiative (Comparative study of logging and environmental governance in Cameroon and Republic of Congo supported by the National Science Foundation). Professor Christopher B. Barrett (http://aem.cornell.edu/faculty sites/cbb2/) Chris Barrett is the Stephen B. and Janice G. Ashley Professor of Applied Economics and Management and International Professor of Agriculture at Cornell University where he also serves as the Cornell Center for a Sustainable Future's Associate Director for Economic Development Programs and the Director of the Cornell Institute for International Food, Agriculture and Development's initiative on Stimulating Agricultural and Rural Transformation. He holds degrees from Princeton (A.B. 1984), Oxford (M.S. 1985) and the University of Wisconsin-Madison (dual Ph.D. 1994). At Cornell and Utah State Universities, he has taught a range of undergraduate and graduate courses. There are three basic, interrelated thrusts to Prof. Barrett's research program. The first concerns poverty, hunger, food security, economic policy and the structural transformation of low-income societies. The second considers issues of individual and market behavior under risk and uncertainty. The third revolves around the interrelationship between poverty, food security and environmental stress in developing countries. Professor Barrett has published or in press 10 books and more than 200 journal articles and book chapters. He has been principal investigator (PI) or co-PI on more than $21 million in extramural research grants from the National Science Foundation, The Pew Charitable Trusts, the Rockefeller Foundation, A-8 USAID and other sponsors. He has supervised more than 50 graduate students and post­ docs, many of whom are now on leading faculties and in research institutes worldwide. He served as editor of the American Journal of Agricultural Economics from 2003-2008, is presently as an associate editor or editorial board member of the African Journal of Agricultural and Resource Economics, the Egerton (Kenya) Journal of Humanities, Social Sciences and Education, the Journal of African Economies, the Journal of Development Studies and World Development, and was previously President of the Association of Christian Economists. He has served on a variety of boards and has won several university, national and international awards for teaching, research and public outreach. Professor Oliver T. Coomes (http://www.geog.mcgill.ca/faculty/coomes/') Oliver Coomes is an Associate Professor of Geography at McGill University in Montreal, Canada. He serves as the Editor-in-Chief of the journal World Development, and Associate Editor of Economic Botany, and has been a member of the editorial boards of the Journal of Land Use Science, Annals of the Association of American Geographers, Journal of Latin American Geography. He completed his degrees at the University of Victoria (B.Sc. Honours, 1979), University of Toronto (M.A., 1982) and University of Wisconsin-Madison (Ph.D., 1992). Prior to undertaking his Ph.D., he worked for six years as an environmental consultant with SNC-Lavalin, a large Canadian engineering company. He took up his professorship at McGill in 1992 where he teaches courses on Development and Livelihoods, Human Tropical Environments and field courses in Quebec and Panama. His research program focuses on natural resource use - present and past - rural livelihoods, conservation and development among rain forest peoples of the neotropics, particularly in the Peruvian Amazon. Professor Coomes' research has appeared in a wide range of disciplinary and interdisciplinary journals, including the American Anthropologist, Journal of Latin American Studies, Latin American Research Review, Hispanic American Historical Review, Forest and Conservation History, Ambio, Economic Botany, Agroforestry Systems, Environmental Conservation, Forest Ecology and Management, Ecological Economics, Land Economics, Environment and Development Economics and World Development. His historical work with Dr. Bradford Barham appears in their book, Prosperity's Promise: The Amazon Rubber Boom and Distorted Economic Development (Westview Press, 1996). He has served a member of the scientific steering committees of the Land Use and Land Cover Change Core project of the IGBP/IHDP, of the International Geographical Union (IGU) Commission on Land Use Change, the Global Land Project transition team, and is a member of CIFOR's Poverty and Environment Network. Professor Coomes' current research examines agrobiodiversity and seed networks, peasant vulnerability and adaptation to abrupt environmental change, the role of peasant co-operative labor in land accumulation, and landscape-household linkages in land use/cover change in neotropical forest environments. He serves on the Board of Directors of the Learning Associates of Montreal, a non-profit organization for assisting children with learning difficulties. Professor Jean-Philippe Platteau (http://perso.fundp.ac.be/~jpplatte/cvjpplatteau.htm), Jean-Philippe Platteau is a Professor in the Department of Economics, Facultés Universitaires Notre-Dame de la Paix (FUNDP) in Belgium, where he has been on faculty since 1978 and where he has served as Director of the Centre for Research in the Economics of Development since its creation in 1994. He is widely published and cited and has earned multiple international research awards. Most of Professor Platteau's work has been concerned with the understanding of the role of institutions in economic development, and the processes of A-9 institutional change, especially under the joint impact of population growth and market penetration. The influence of non-economic factors and various frontier issues at the interface between economics and sociology, have been a central focus. His attention has centered mostly on agrarian institutions in developing countries, implying that many of my contributions have a detailed empirical basis, always grounded in original primary data collected in Asia and Sub-Saharan Africa. The most important themes in his research are: the determinants of village-level collective action, and the impact of wealth inequality on its level and efficiency (with special reference to the production of local public goods and the management of local natural resources); common property resource management in village societies; informal risk-sharing mechanisms; rules, including inheritance customs, governing access to land and other natural resources, and their transformation under the influence of increasing land pressure; the development and functioning of land markets under the same influence; choice of contracts in artisanal fisheries; systems of marriage payments and their evolution in Africa; the role of social and moral norms in economic development; 'elite capture problems' in decentralized development programs. Professor Platteau is currently President of the European Development Network and a member of the Board of Global Development Network, among other major professional service roles. Professor Platteau has served as an associate editor or on the editorial board of Development and Change, the Journal of Development Studies, the Middle East Development Journal, Oxford Development Studies and World Development. A-10 Annex 3 CGIAR Social Science Stripe Review e-consultation wrap-up January 2009 What follows constitutes a very rough, semi-structured summary of contributions to the e- consultation. Thus far over the past week, the listserv set up for this consultation distributed 60 messages sent by 23 different persons representing 12 Centers and 1 Challenge Program (CP), as well as the Science Council Secretariat (SCS) and the social science stripe review panel (SSSRP). There have been 42 subscribed participants, with at least one from each of the Centers and CPs. This digest makes a first, very crude attempt at grouping related comments and gathering extended quotations from participants to help flesh out broader themes from the e- consultation. It does not reflect any conclusions taken away by the review panel and it is not the product of any substantive consultation among the panel members. It is meant purely as a quick digest of many of the excellent points made over the past week by e-consultation participants. Not all the questions or comments posed by the review panel in its daily prepared posts were addressed. That is fine; those scripts were intended mainly to seed the discussion, which was lively and rich. We thank all the participants for taking valuable time to offer their thoughts and to respond to others' posts. The e-consultation has been extremely informative for the SSSRP. At this point, we welcome your reactions to this summary. Do you think it accurately captures not just the spirit and content of the e-consultation, but - more importantly - the crucial issues the SSSRP needs to explore in order that the review helps advance high quality social science in support of the CGIAR mission? The e-consultation listserv will remain open a bit longer than originally planned in order to allow participants time to review the discussions and this digest and to respond to the full forum. The listserv will close on the afternoon (Rome time) of Friday, January 23. The Big Picture The big picture question facing the SSSRP is that posed early on by Doug Pachico (CIAT), and stated similarly by several other participants, "FFow can social science best contribute to the overall mission of the CGIAR?" This question raises subsidiary questions of funding, research prioritization, organization, staffing and incentives. The strategic focus of CGIAR social science research raises the question, what is the source of demand for CGIAR social science research? The (perhaps multiple) answer(s) to this question will indicate "Is there higher payoff in social scientists working more closely with counterpart Center scientists in the field to better assess the demand for technology (i.e., for better targeting biological scientists’ own research), or with senior management in helping focus Center priorities, or in carrying out the more traditional consumer/producer behavior and/or policy type analysis? It would be worthwhile to bring into the discussion the primary intended users of social science research, including DGs, DDG(R)s, and fellow CG scientists, in addition to peers in the field (academics) to answer Doug's question of how can A-ll social science research best contribute to the mission of the CGIAR" (Tim Kelley, SCS). Different audiences have different needs. So who are System social scientists trying to serve and what is the appropriate ordering of audiences and their demands in CGIAR social science research prioritization? John Dixon (CIMMYT) identifies the relevant audiences as "NARS (sensu lata), especially smaller and mid-size. Secondary audiences: System/Center management and non-SS scientists. For better impact on the primary audience, deepen partnership modalities for joint research (e.g., select hubs among NARS lead on selected research)." Ruth Meinzen-Dick (IFPRI and CAPRI) argues that "functionally, I think a key role of social science in the CGIAR is to identify how agricultural systems (including technology and natural resources) connect with people, and to use that understanding for more effective poverty reduction. Part of that is in the ex ante priority setting (starting from an understanding of who the poor are, and how agricultural systems can help them); part of that requires serious research to develop that understanding (which might also be ex post evaluation of outcomes of programs, along the lines Meenakshi mentions). Otherwise we're just prioritizing based on our premises and conjectures. I think one of the frustrations many social scientists have had is that getting the resources to do that research has been difficult, because it takes time and skilled people to go to the field to not only collect the data, but develop a contextual understanding to interpret it." Funding, Research Prioritization and Organization Donor-driven research appears a major concern that the review panel needs to investigate. "Usually big research projects are defined by donors" (Pie Moya, IRRI). There seems limited freedom in research design or management, unlike in most high-level research organizations. JV Meenakshi (Harvest Plus) wonders "To what extent has the economics research at the Centers/CPs been driven by donor demands versus being driven by the issues as defined by the Center social scientists themselves; and to what extent has this affected the quality of the economics research at the centers"? There is widespread sense that this concern is closely bound up with well-documented increases in restricted funding for CGIAR research. "The centers have been going to a more or lesser extent towards a model that resembles increasingly a consulting firm (in fact, it my understanding that the change process should help address this problem). Unrestricted funds are decreasing at least as the proportion of overall funds, and restricted funds have been increasing. We are increasingly bound by developing and implementing donor-funded project with very specific aims (even scientists salaries increasingly depend on this). The fact that there is less unrestricted funds in my view limits our ability to carry work that may be very interesting and valuable but that may not be of immediate interest to donors. I believe that without unrestricted funds to explore new ideas, synthesize our work, etc. the work of social scientists becomes more routine and with less capacity to innovate. It could be argued that it is on us to convince donors that this innovative work is worth funding, but sometimes is not that easy. In my own experience without a substantial amount of unrestricted funds I would not have been available to carry out much of the research I did when I was a "practicing" social scientist, and even today as a manager I believe that limited unrestricted funds limit our ability to invest and add value to our work (not just in social sciences, but overall)" (Mauricio Bellon, Bioversity). A-12 The effects of restricted vs. unrestricted funding may be somewhat exaggerated, however, suggests Yemi Katerere (CIFOR), "Restricted and unrestricted funding in the CGIAR is short-term. In fact restricted funding tends to be more certain than unrestricted in the sense that a centre can predict with certainty available funding based on signed contracts. So we need to be careful about claims that dependence on restricted funding has affected recruitment patterns and choice and quality of research. Unrestricted funded would be great if only the donors would be willing to commit to multi-year unrestricted funding." Eddie Allison (WorldFish) suggests that Center social scientists play a major role in defining research agendas, as reflected in Centers' medium term plans (MTPs): "The extent and quality of participation in research goal-setting is likely to vary between centers - but I can say that in ours, social scientists have been very influential in setting our current research agenda, and all scientists from post-doc upwards were involved in workshops to inform the formulation of our institutional research plan (MTP - in CG-speak)." Cynthia Bantilan (ICRISAT) notes that "the role of social science as a key driver of research prioritization has been recognized and appreciated well. The willingness of social science researchers to work closely with bio- physical and agro-biological scientists gave rise to this leadership. To deepen and extend this impact, it should be made mandatory to the CGIAR institutions to go through a research prioritization exercise once in every 5 to 10 years." Eddie Allison (World Fish) weighs in on the closely-related question of adequate core funding, "the single largest factor impacting quality of social science (and other) research in the CGIAR is the low level of core funding, meaning that much of our limited core time is spent writing funding proposals, rather than improving the quality of our project-based research outputs or building research capacity. Although the part 1 review document (normative document by Chris Barrett) recommended a desired level of 70% core funded time for social science, most colleagues regarded this as utopian or even undesirable due to likely separation from the need to engage in practical development processes. Figures in the range 40- 60% were deemed as being acceptable. Although the CGIAR average is 30% core funded time (around 25% for us in 2009), many of my colleagues find this time eaten up by a combination of time spent developing funding proposals, doing research administration and subsidizing under-funded projects. Dependence on outside funding also encourages the pursuit of low-risk funding strategies and avoidance of competitive grant agencies which have rigorous peer-review processes, which, alongside peer review of outputs, is a key means of maintaining and enhancing research quality. Thus, much social science research funding comes from donor country-office funds allocated non-competitively, or from consultancy-style tendering processes, often with strong pressure not to mention the word 'research', but to recast it as lesson-learning, monitoring & evaluation or dissemination. All this limits scope for research innovation and quality. This lack of core-funded time also affects staff attraction and retention - and may be one way in which we have lost competitive edge to ARIs on recruitment of economists (Pie Moya's point). Research staff in the CGIAR - and those we want to attract - are motivated both by their work having development impact, and by having time to devote to the process of research. Without the latter, incentives to stay can be eroded. To some extent, coming to the CGIAR from a university involves trading some research time for greater involvement in having development impact through more direct engagement in policy advisory work and facilitation of technology transfer, for example. But if there is no time to do high-quality research then there is little new to promote and have impact with and the CGIAR gets into a spiral of decline. I don't think we are in it yet but we are close enough to worry about it. A-13 Can the panel, in its final review, address the options for increasing the core funding contribution and analyze how the CGIAR reform process is likely to facilitate this?" To what extent will the core/restricted funding situation and the follow-on effects on research design and social scientist behavior and recruitment/retention change with the new "mega­ programs" under the new Consortium, especially if only investors/donors have decision­ making power in the new Fund Council, which ultimately makes all resource allocation decisions? Or will the influence of individual donors recede and System-level guidance, as provided by the Science Council (or its successor body) and others, become more influential? Relatedly, this raises questions about "the organizational and partnership arrangements that can enhance the impact of social science research" (Ade Freeman, ILRI). Ruth Meinzen-Dick (IFPRI and CAPRI) draws on her experience directing a social science-heavy Systemwide Program (CAPRI) to emphasize that "a key factor for CAPRi has been bringing together the social scientists (and some others) within the CGIAR with top people from outside to create a critical mass that is often missing in an individual center. So the partnerships really do help, but social scientists also need to have some colleagues within their own center who understand what they are trying to do, so they don't feel too isolated." John Dixon (CIMMYT) and others opine that CGIAR social science must "enhance/invest more in communication of knowledge developed through SS research to SS in NARS. Part of the investment is needed in weak NARS to assist them to access SS knowledge from CG and ARIs. Of course, it is also necessary to maintain the communication with non-SS users in the same NARS and NGOs." Fie notes that "the CIMMYT Knowledge sharing coordinator estimated in January 2008 that NARS scientists had direct access to no more than 15% of the 2007 articles published by CIMMYT scientists (all disciplines). Of course, one can safely assume a lower accessibility across disciplines, i.e., NARS' breeders reading CG SS articles, regardless of the journal." At the same time, he notes that "ARIs/universities in OECD countries are often competitors, driven by a similar need to tap external resources, as well as active collaborators with CG system (even in the same project). Advanced NARS are not yet significant competitors but I expect that they could become so in the coming decade." A parallel question exists about integration within the System with non-social scientists, given that "many of the institutional, social, economic and cultural issues are completely interlinked with agricultural development and the associated research" (Jamie Watts, Bioversity), what organizational arrangements promote close integration with natural science research it is meant to support? What tradeoffs, if any, are there with promoting higher quality disciplinary research within the social sciences? Where is the demand for social science work? Mauricio Bellon (Bioversity) remarks on the trick of balancing more innovative social science with research in support of the natural sciences, "Social scientists have an important service role contributing to the work of natural scientists in pursuit of our mission. At the same time as scientists we want to do innovative and interesting research. Obviously there is not an inherent contradiction between these two aims. It is more of a continuum, where in many cases our service function provides the opportunity to do innovative research, but at the same time in many cases what we are asked or expected to do are the standard participatory appraisals, the diagnostic studies, the baseline surveys, the stakeholder analysis, the cost-benefit analysis, etc. which in most cases are just routine work and if we only do that the job becomes quite uninteresting from a professional perspective. At the same time our colleagues may not appreciate the value of the more interesting and innovative work that is professional rewarding and that also may be extremely valuable for A-14 accomplishing our mission, but that it is not obvious to others, at least in the earliest stages. Clearly there is a need to balance the service function and the innovative research." Cynthia Bantilan (ICRISAT) raises a key related question: "What is the critical mass of Social Scientists and funds available to do Social Science research? The critical mass of social scientists has declined in most CG centers while the issues to be addressed by them have increased. As Crop and NRM scientists have moved from purely basic research to promoting their products downstream, applied research is now practiced while the basic research is continued to keep the pipeline of improved products going. Donors and development agencies are increasingly looking for impacts at the ground level related to productivity gains, income enhancement and poverty reduction of the most vulnerable group dependent on agriculture. This has widened the demand for social scientists and raised expectations from them. The buzz words today are ex-ante assessment, targeting, monitoring and evaluation, market linkages and impact assessment. All projects irrespective of the technology being developed have to build into the project the above aspects for better buy in. These changes bear on the lessons learnt from both successes and past failures in promoting technology/ innovations on a wider scale. Productivity increase is not the only goal of agricultural research as it was several decades back when food security was the primary concern. Today, other areas of concern (besides productivity growth) are sustainable livelihood pathways, higher incomes, food and nutrition security (health), value addition or overall quality of life. Clearly social scientists have a bigger role to play if the new technologies from the CGIAR have to impact on some or all of the above aspects." Social Science Staffing, Incentives and Constraints on Quality and Effectiveness A separate thread of the discussion encouraged the review panel to focus on assessing the quality and effectiveness of CGIAR social science. Brent Swallow (ICRAF and ASB) and Aliou Diagne (WARDA) suggest "that the review focus on questions related to ... how effective are social scientists working in the CG, what are the major constraints on that effectiveness, and what are some promising options for improving that effectiveness?" Cynthia Bantilan (ICRISAT) asks "what are the present roles and functions of social scientists in the CG system. How well are these functions being carried out? How can the social science research be further improved to meet the goals of respective institutions and the over-all CG system?" Aliou Diagne (WARDA) sums up the issue as largely about getting the right people and giving them the incentives and resources to do the work they know needs doing: "I think the system is not presently short of analyses/recommendations of what the system' and centers' SSR agenda should be (the never ending MTP rewritings and SC comments on them, the strategic plans, CCER, EPMR, Systems priorities, etc...). I think at this point the problem is not "knowing what to do and how do it." But rather, having the adequate resources (time being the most important one) and the right incentives for the CG SSR staff to perform as desired. In that regard, parts of the answers are already in the last section of the normative framework paper (the Staffing section), which I find to be the most useful part of the paper. So my recommendation to the panel would be to focus the final report on making the case for the excellent and concrete ideas for change in that section." Ruth Meinzen-Dick (IFPRI and CAPRI) identifies what she considers some key factors driving improved quality in CGIAR social science research: A-15 Resources: Having a designateci budget for the social science component gives you more say in the design of the study. Senior and junior people: Either a good senior social scientist within the system (this is where retention is important, because many of those have come up from post doc) or a strong external collaborator from an ARI, usually working with more junior people who spend more time in the field. Yes, it takes time, but when it works, it's great for both sides. Web-based publications—both electronic journals and online working papers make it much easier to get social science literature. Formerly many centers did not invest a lot in social science journal subscriptions. When CAPRi started we did a lot of sending key references out to people, but now they can get things much more easily. The CG V-library has really been wonderful for expanding this, as the CG has pooled journal access. Professional and other meetings. Centers vary in how much they encourage their people to attend these, but they do really help people keep up and get fresh ideas." Brent Swallow (ICRAF and ASB) confirmed that online journals access has improved significantly in the main CGIAR offices - albeit not in all the smaller outposts - such that electronic access to current research is no longer significantly limiting for CGIAR social scientists. The issue of recruitment, retention and mentoring elicited significant and sometimes- divergent comments. One specific thread of the discussion concerned senior scientist recruitment and having time to mentor junior social scientists so as to ensure the quality, relevance and innovativeness of the work. "The quality of supervision of junior researchers by senior scientist is a critical condition of success, precisely because there are many ways of carrying out a particular topic and because a topic is perhaps less strictly defined. In that case, the hiring of highly qualified senior social scientist is very important or critical to the quality of social science research in the CGIAR" (Pie Moya, IRRI). "Motivated and enthusiastic Senior Scientists, with the wiliness to share their knowledge and experience with Junior Scientist are fundamental for the good quality of the research. The ability to abstract lessons learnt from the field and uptake them to a more theoretical level is something that can be done under good guidance of a senior person able to merge both academic and field experience. In social science this is even more important given the many facets included in our field of concentration. Bridging theory and practice is something that is learnt by experience and having a supervisor able to balance and transmit both to us is crucial. Junior Scientist education (both at PhD or Master level) is often lacking of a practical problem solving background" (Elisabetta Gotor, Bioversity). "I am not convinced that the apparent premise of this discussion, that there is a retention problem among social scientists in the centers, is correct." (Doug Pachico, CIAT) Likewise, "we are not fully convinced that retention is the key issue, so much as recruitment is — are we comfortable that the CGIAR can continue to attract high-quality social scientists able to address current and emerging issues of relevance to the CGIAR goals? Will we sufficiently competitive vis-à-vis academic, think-tanks, and other development agencies?" (Rajul Pandya-Lorch and Stacy Roberts, IFPRI). Recruitment problems seem especially acute at senior levels. Eddie Allison (World Fish) suggests thought be given to joint hiring in specialized areas: "Although I am in general against the suggestion of consolidating social A-16 Science expertise in one CG center, because it reduces the important day-to-day interactions that lead to cross-discipline collaboration and innovation, it may be beneficial for senior social science researchers in discipline areas we have difficulty recruiting for to be appointed jointly by centers with similar mandates e.g. in natural resource management, or in crop improvement research." The joint issues of recruitment and retention prompted Mauricio Bellon (Bioversity) to raise a related question of career paths. "As was pointed out we need a balance between junior and senior scientists, both are needed. A lack of junior scientists decreases innovation, but a lack of senior scientists decreases leadership and experience, we need both. What is the career path as one becomes more senior? Some become managers, but not everybody wants or is equipped to do this, so what happens? Discussing with a colleague some years ago, he pointed out that it was increasingly difficult to move from a career in a center to an academic job, which I understand used to be a common career path in the past. I think that the lack of clear longer term career paths for social scientists as they become more senior is also having an effect on recruiting and retaining senior social scientists that the centers require. On the other hand, we have problems of funding to retain bright young scientists that usually come as Associate Experts, financed by certain governments (obviously not all should be retain, but some should)." One specific hypothesis that relates recruitment to research and outputs was posed by Brent Swallow (ICRAF and ASB), "I hypothesize that the Rocky Doc program, which ended in the mid-1990s, had a major impact on the nature of social science in the CG. It led to American­ ization of social science, with bright young American scientists eager to do publishable work and with lots of connections to US universities and a bias toward publishing in the mainstream English-language journals. We shouldn't downplay that influence on the overall system. We need to ascertain whether changes in social science output reflect real changes in 'quality' or rather greater diversity in outputs and publication expectations. It would be good for the team to interview Rocky Doc veterans and alumni. At the same time as we've lost the Rocky Doc program, we have multiplied the expectations of CG social scientists and had a fragmentation of social science research effort - caused by fragmentation and discontinuity of funding of restricted projects. Testing that hypothesis would require some kind of empirical analysis of the continuity of funding, projects and research efforts. There were divergent opinions expressed on whether it is a problem that a relatively large share of CGIAR social science researchers have no formal social science training. "A surprising proportion of those filling social scientist functions in the centers do not have professional degrees in social sciences. This too has been the specific case at CIAT. While at times they can make valuable practical contributions, rarely are they effective at conducting publishable research. To do so would require re-training or mentoring, but it has been our experience that these self taught or amateur social scientists feel threatened by mentoring from experts in social science. Thus I think that a serious problem degrading the performance of social science in the centers is the presence of a number of non-professionals filling social science functions who are not capable of publishable quality research." (Doug Pachico, CIAT). Eddie Allison (World Fish) takes exception to this, arguing that many people have learned a good deal of social science over the course of their careers following graduate training. "What you trained in more than twenty years ago becomes decreasingly relevant if you have had a progressive and varied research career, and are careful to collaborate with good A-17 disciplinary specialists - collaboration, humility and willingness to learn can substitute for mentorship and re-training. Two of the three scientists in our group who have published and are cited most in social science literature are from a natural science background, so I have no evidence to substantiate Doug's observation that such scientists struggle to publish in the peer-reviewed social-science literature. Thus I also have an alternative take on the concern (expressed in the phase 1 review documents) that a substantive proportion of CGIAR social scientists are just under-performing natural scientists. Twenty or more years ago, if you were interested in forestry, fisheries or agricultural development, you probably trained in the natural sciences, as fields such as development economics, rural sociology, human geography or political ecology were less well-known and perhaps not the obvious choice for a career in agricultural development. Once entering the professional arena, the more progressive thinkers modified their concepts, toolkits, reading lists and research networks, to enable them to answer the questions they encountered, rather than addressing the world through the blinkers supplied along with their original disciplinary degree." Ruth Meinzen-Dick (IFPRI and CAPRI), however, responds "I take Eddie's point that many people have evolved and learned a great deal of social science over the course of their careers, though they started out in the natural sciences. But I've also encountered a lot of people who think that the "hard" sciences are hard, and that social sciences are intuitive, who go on to claim to be doing social science research without ever looking at the fundamental principles. That leads not only to quality problems in their work, but "real" social scientists often then don't feel respected for their knowledge and skills. (I could once do a water balance, but don't call myself a hydrologist; conducting a "focus group" of whoever shows up under the tree does not make one a social scientist)." Another, related concern surrounds the innovativeness of CGIAR Social Science. "Is the CGIAR "pushing frontiers of Social Science research forward? I think that issues and themes we are dealing with as CGIAR should be tackled in a broader way. Innovation might have a direct impact on productivity but what are the other effects generated by the innovation in place? How the innovation adopted reflects itself into other sectors? How externalities generated are analyzed?" (Elisabetta Gotor, Bioversity). It was wisely suggested that non-social scientist perspectives need to be solicited as well, especially if non-social scientists are a key audience for CGIAR social science. "Relevance and effectiveness also need to be judged from the perspective of the biological scientists who are the majority in most centers. Do perspectives of biological and social scientists in each center coincide about social science relevance and effectiveness?" (Graham Thiele, CIP) Brent Swallow (ICRAF and ASB) also urged the SSSRP to closely examine recent EPMRs that have addressed some of the same questions (e.g., ASB and CAPRI). Ruth Meinzen-Dick (IFPRI and CAPRI) questions the widespread, if often latent suggestion that the quality of and respect for CGIAR social science have declined over time; "I have now been a sociologist working in the CGIAR for 20 years (this month), and I certainly feel that both the quality of social science research and the respect of social scientists (even non­ economists) has increased within the CG—probably an iterative relationship." Yemi Katerere (CIFOR), Jamie Watts (Bioversity) and others raise the issue of the appropriate balance between peer-reviewed journal articles and other research outputs in the social sciences. Yemi wrote "A recent donor review of CIFOR criticized the centre for placing too much emphasis on peer reviewed journals and publication in high end journals. A-18 The reviewers felt that as a policy centre we should also be publishing in other more assessable "publications". So what should the right balance be?" to which Jamie replied "The issue I think was related more to either getting research information to people who do not have access to journals (which is likely to include many people still in developing countries whether policy makers, researchers or others), translating information generated by research to extensionists or other people closer to the farmer level, or information generation and sharing in research that takes place in an innovation systems context where innovation (i.e. improved practice for development) is something that emerges from collaboration among different actors in the innovation system. In these cases, peer reviewed publications may not be the most effective means of sharing information. I would not think anyone is arguing for not publishing in peer reviewed publications, but there may be a sense with the emergence of the performance indicators related to peer reviewed articles, that we have swung too far in the direction of not giving adequate value to other types of information. I don't remember seeing any real analysis of what we should be aiming for in terms of balancing peer reviewed with other publications, but that would be helpful." Arun Agrawal (SSSRP) noted that "this is a perennial issue in most research organizations that aim to influence policy - with the big point being that policy makers and decision makers have no time to read lengthy articles and books that tend to be the type of things that academics publish. Yet, I wonder if that is entirely true because of two reasons. 1. Take some quick examples: many accomplished researchers such as Elinor Ostrom, Vernon Ruttan, Yujiro Hayami and more recently E. Duflo, and A. Banerjee, etc publish almost nothing that is in the nature of briefing notes or policy review notes and other vehicles that are supposed to be better for policy impact. Yet, the policy influence and impact of their work and ideas is likely quite high. So, is the specific form of output that important? 2. Are policy oriented publication forms (more assessable and accessible) like briefing notes, etc. really what policy makers look at and have times to read? I imagine the well known POLEX briefings and the ODI policy briefs are an example of the form being important - but is the content of these also what sets them apart?" Ruth Meinzen-Dick (IFPRI and CAPRI) replied, "You're right about this tension between briefs and "scholarly" publications. I do know from CAPRi that our policy briefs are extremely popular. We've gone through thousands of copies of one set of them in hard copy and the briefs have very high downloads. But more importantly, when I have gone into various people's offices, I have seen the briefs on their shelf, dog-eared. I would like to say that is because of the content, but the feedback has also been because they are very understandable pieces. However, most of these also have a journal article or other piece that is peer reviewed to back it up, so they should be complementary (and, like Polex, the short pieces to induce people to read the longer ones)." Ruth later commented on the phase 1 background report's finding that CGIAR social science journal articles are cited at a lower rate than other articles in the same journals. "I was disturbed to note in the background statement that CGIAR journal articles are not cited as much as average for the same journals. I was wondering if this is because they tend to be case studies rather than theoretical or synthesis/reviews? In many journals, synthesis papers are the most widely cited. Also, if, as the background question notes, "technology development and adoption issues that were once the bread-and-butter of CGIAR social A-19 science", are these as much of interest to journals, and for citation? This work is certainly very important for the application of CGIAR technologies, but how much scope is there for that to be really innovative in terms of social science research or methods, and for non-CG social scientists to cite?" Chris Barrett (SSSRP) responded with an alternative hypothesis: "another, perhaps more obvious hypothesis that a number of readers of the phase 1 report have made: the quality is just lower on average, with less truly innovative empirical results (or methods or theory) than in most other articles appearing in the journals. That's clearly not true universally, as reflected in various outstanding article awards (for example, Gilligan and Hoddinott at IFPRI won last year's outstanding article award from the AJAE ... a very high honor). But many are interpreting the evidence as suggesting that is true on average. This strikes me as a serious issue worth pondering and hypothesizing about a bit more ... why is it that CGIAR social science gets cited relatively infrequently, even when one controls for the journal (i.e., it's not a matter of publishing in more applied journals rather than leading, theory-and-methods-oriented journals)????" Arun Agrawal (SSSRP) floated yet another hypothesis about the seemingly relatively low scholarly impact of CGIAR social science, "An issue worth considering here is that numbers of citations for similar quality articles are at least partially dependent on the networks of scholarship in which authors are located. A priori, I would have expected that papers published by System social scientists would have the inbuilt advantage that other social scientists in the CG System would be more prone to citing them. Thus, for example, when I think of working papers and reports published at the World Bank, I have the distinct impression (not tested rigorously) that a far greater proportion of Bank documents is cited in Bank Reports than in peer-reviewed journal articles. And that articles in peer-reviewed journals cite far fewer of Bank reports than do Bank reports. But perhaps that is not the case where social scientists in the CG system are concerned? And that is what brings me to the following questions: Do social scientists in the system keep up with the research of other system social scientists? Is there a sense of connectedness among different social scientists in the CG system? (And I ask this as a way of going beyond working on specific collaborative research projects for which we are asking questions separately.) Or is it the case that distances between the centers, research staff turnover, and immediate needs make it difficult or impossible for such a sense of connectedness from developing?" J.V. Meenakshi (HarvestPlus) responded that "I don't believe that it is the intrinsic nature of the topics of research in themselves that causes them to be cited less often." Rather, she continues, "the "core versus restricted funding" tension has implications not just for the choice of journals in which to submit, but in the amount of time one is able to devote to a given paper. In economics at least, turn-around times for the refereeing process continue to be immense, certainly in relation to the biological sciences. The inability to allocate time to 'core' for any writing (because the cost center has long since expired) is a frustration I have heard (and shared) with several economist colleagues, both within IFPRI and across sister centers. I have also had the experience of being asked by a donor about whether the level of rigor that was in the submitted report was not excessive given the mandate of the challenge program and the particular project. "Is this rigor necessary?" is a near-verbatim quote. Rigor is, unfortunately, often (but not always) expensive, and takes time." Tim Kelley (SCS) replies '"Is this rigor necessary' is a valid question for a donor to ask, since if a lesser intensity of effort (and resources) produces 'sufficient rigor' to generate the kind of impact the donors and the CGIAR are looking for, then that should be enough, whether it's published or not. Impact is the goal. Publishing is a means to an end, a mechanism for A-20 establishing the credibility of the research result (and the social scientist) and only indirectly may, or may not, have impact ... If generating specific social science related information [is] of high relevance to biological scientists or managers at the Centre, e.g., in focusing/re-targeting their work, is the opportunity lost in deepening the rigor and writing publishable pieces, then donors and Centers are not well served by that tradeoff.... IF there is that tradeoff, and in many cases there may be. I'm not defending poor quality un­ publishable social science work. But it is relevant to question the trade-off between well done applied and highly relevant to the Center research (e.g., technology evaluation/adoption studies, socio-economic characterization of major ecosystems), which may well be less publishable by their nature vs. research and related activity that lends itself to publishing in top notch journals. In this case there could be a trade-off between relevance to impact and advancing general knowledge/testing development hypotheses. ... At the margin one can consider a tradeoff between effort expended on the former vs. the latter. But the only legitimate basis for assessing that tradeoff would be in terms of efficiency in achieving CGIAR goals. FFow much importance should be given to CGIAR social scientists publishing in the best journals in their fields? Will that necessarily further the opportunities for impact? It is no doubt part of most CG social scientists’ objective function as it establishes his/her credibility in the field, but is it the same for donors, or for the Center? Major emphasis on publishing in top journals is clearly appropriate for university researchers that are in the business of and rewarded for generating more generalizable knowledge (which also may produce impact, but again only indirectly and in ways often difficult if not impossible to track), but the CGIAR has a different objective function where publishing is important, but only indirectly." Sirkka Immonen (SCS) writes "Tim’s points are very relevant about a potential trade-off between effort spent on research that is publishable in the top journals and likely to be cited and research that is highly likely to advance impact. The former can easily be measured, the latter needs more assessment and there could be a risk of declaring "impact orientation" as reason for not seeking for peer quality clearance of research (as in the formal publishing process). This discussion has drawn some comments about the performance measurement system. Along with the Change the PMS in place will also change. In my opinion the current PMS (and indicators in general) has limitations because it is difficult (if not impossible) to express relevance, impact and even quality through simple indicators. Due to these limitations, the PMS can turn against its objectives if it is not carefully used. Rather than being a management tool, the CGIAR’s PMS has uniquely been donor demanded and driven in the sense that the World Bank (and one or more other donors) have from the start made funding decisions on basis of the indicators. It is unfortunate (although maybe not surprising) if that has drawn Centers' attention from other (not easily measurable) performance issues, such as can be gauged through external reviews better than indicators. In the new CGIAR I hope much of the important management related performance (quality, good record keeping, best practices, etc.) will become the Consortium's internal matter (with no direct funding implications), relevance can be established through program selection and review, and the Performance Contracts will identify indicators for progress and success at a different level and more related to the relevance, utility of results and outcomes. " Mark Giordano (IWMI) follows up with the observation that "Another pressure which runs counter to the "real impact" issues Tim brings up is the need to produce International Public Goods and score high on the CGIAR performance measures. Important outlets like the A-21 Indian Economic and Political Weekly don't score well, or at all, on either front. As a result, centers and researchers tend not to be formally incentivized to put efforts on such venues." Jamie Watts (Bioversity) echoes the need to get social scientists' incentives right, writing "Clearer decisions need to be made by CG management (and funders) as to our priorities and then performance assessment systems adjusted as appropriate to assess the directions given. The review panel might look at the link between what we are doing (or what we think we should be doing) and the performance assessment systems in place to see to what extent they match up or don't. In terms of information for use of CG management or researchers or other policy makers, or information to address more basic questions, we clearly will need both, not one or the other. Some of the work going on like much of the impact assessment, impact pathways analysis, social network analysis, learning and change approaches and others are done for the purpose of informing CG management and decision making (though these can be published as journal articles and often should be), some is for other policy decisions and these should be managed as such with the standards appropriate for that purpose. Other areas of research are more investigations of basic questions and need a different approach. It's not an either or situation, we need both, and then we need to recognize both realms of activity as valid, find appropriate communications mechanisms for each and judge them according to the appropriate standards. Perhaps the panel could provide some insights to help us develop an overarching framework that makes sense of the different approaches to social science needed in the CGIAR. If the analysis of publication citation compared those papers that are more basic research with those papers that are more management or policy information oriented, perhaps we would see more citation of the more basic research type of work." Brent Swallow (ICRAF and ASB) added "three dis-jointed thoughts on this topic: 1. Lower citation rates in the same journals may be due to two things other than quality or relevance of the work: a. citation practices that may lead to less self-citation among CG scientists; and b. lack of pro-active promotion of the work of CG scientists by the scientists themselves or their institutions. For example, I note that among the CG centers only IFPRI is very proactive in listing its working papers, journal articles etc in RePEc (Research Papers in Economics). I've just got ICRAF listed and am working with our library to upload the relevant papers. A very useful contribution by the review team would be to provide some pointers on how to do this more effectively. 2. I think it misleading to assume a dichotomy between quality and potential impact, even if donors, project managers and others assume that there is. I think of a few examples of how a track record and reputation for quality supports greater policy impact. A CG example is the work of IFPRI in world food market predictions - IFPRI's reputation for solid research increases the credibility of its predictions, making it a much more respected and trusted source than alternative suppliers of the same information. Promotion of that work through the media, etc makes donors feel good about their investments in IFPRI, which should feed forward into more stable and secure funding. The Woods Hole Research Centre publishes routinely in the world's top science journals (e.g. Science, Nature), increasing the credibility of its policy advise. In ASB we are now relying upon our past record of research accomplishment to support our policy work - I'm sure that a few publications in top journals will further increase this credibility. Donors are a big part of the problem that centers must manage. The World Bank has recently shown great interest in supporting an extension of our work: they say they A-22 Oliver Coomes (SSSRP) pointed out that CGIAR social scientists are highly valued as reviewers by the editors of leading journals (e.g., World Development, American Journal of Agricultural Economics). Several participants remarked that they are glad to hear this, but there is no direct recognition or reward for such service, as is likewise true in the ARI or NARS communities. The benefits are purely reputational and indirect. Mauricio Bellon (Bioversity) also emphasized the importance of leadership in establishing a productive and satisfying professional environment, "Creating an environment where innovation and service coexist and actually complement each other requires leadership, both by the managers of social scientists and the overall management of the centers. Leadership is needed to create the intellectual and financial "spaces” needed to develop innovative research and to identify and implement research where service and innovation complement each other." CGIAR Social Science Comparative Advantage A key task facing the review panel is to identify where CGIAR social science holds comparative advantage. Participants offered several suggestions. JV Meenakshi (Harvest Plus) wrote that "Social scientists based in CGIAR centers are uniquely positioned to deepen the discipline's understanding of the experimental approach to impact assessment that has come to dominate the development economics literature. There is the often-voiced but typically unpublished discontent with the purist randomized experiment approach in economics. As social scientists in CG centers, we work with natural scientists for whom experimental design issues are bread-and-butter tools. Such methodological cross­ fertilization is bound to yield rich dividends and to enrich this literature. This potential is, I believe, largely untapped at present." Cynthia Bantilan (ICRISAT) notes three areas of CGIAR social science comparative advantage: "technology assessments, impact evaluation, and databases on household economies of farmers in their respective recommendation domains." She emphasizes that "In order to improve effectiveness with limited funds, analysis of adoption and impact assessment in crop, livestock and NRM research using cutting-edge methods and tools will remain a crosscutting priority." In a separate post, she emphasizes the CGIAR's comparative advantage in longitudinal data collection in rural Africa, Asia and Latin America: "There is arguably no more major, long­ term issue for the CGIAR than research that helps to document and interpret the dynamics of poverty and vulnerability or the changes that are taking place in rural areas of sub- Saharan Africa, South Asia and Latin America over the longer term. As poverty dynamics have become central to the emerging development discourse, with their attendant emphasis on risk, productivity, and systems approaches, the CGIAR finds itself in a unique strategic position. The primary limiting factor to generalizable knowledge production on the rural transformation and socioeconomic mobility is high quality, micro-level data over extended periods - decades and generations - over a wide range of cultures, agro-ecologies and policy regimes. Effective policy making for sustainable hunger and poverty reduction depends fundamentally on scientifically-defensible evidence establishing clearly not only the constraints to smallholder innovation and technology adoption, capital investment and A-23 market entry, but perhaps more importantly the interventions and incentives that enable the poor to overcome those constraints. For at least twenty years, the CGIAR has long been the world's premier source of collections of high quality, longitudinal micro-level data from carefully stratified geographical locations. For example, ICRISAT's Village Level Studies (VLS) run in rural India from 1975-85 and restarted in 2002, have been an unparalleled source of information to inform development strategies for the region and truly to transform development economics and development studies more broadly. While no other social science data set has had the scope, continuity, depth or international impact of ICRISAT's Indian VLS studies, several other CGIAR-based efforts have been noteworthy and similarly extremely influential in producing international public goods. For example, IRRI's Loop Survey in the Philippines, IFPRI's longitudinal surveys in Ethiopia, Guatemala, Mali, the Philippines and South Africa, ICRISAT's VLS in Burkina Faso and Niger, and ICRAF's panel data studies in east Africa have generated exceptionally detailed, high quality data of great value to policy and social science researchers locally, regionally and globally. Such data collections are the fundamental building blocks necessary to the design of effective institutional innovations and policies, much as collections of animal and plant genetic material are the fundamental building blocks necessary to the design of improved agricultural technologies. Precisely because so much of the value of these long-term, micro­ level studies accrues beyond the individual Centers and emerges through comparison across sites regionally and globally - i.e., because such research represents a true international public good - there has been serious underinvestment in such research, especially during the 1990s. There is, however, renewed interest among donors in such research partnerships among CGIAR centers in collaboration with ARIs and the NARS (including universities in the south) as the necessity of longitudinal data has become clearer and as more agencies and governments have developed a culture of evidence-based decision-making. The extension of existing panel studies in space and time will enable unprecedented tracking of changes in the dynamics of poverty and wealth, identification of the drivers of socioeconomic mobility and agricultural rural transformation - particularly the role(s) played by innovations in agricultural institutions, markets and technologies - and determination of shifts in the investment and livelihood strategies of the rural poor. Further, extended and coordinated VLS-type studies can enable path-breaking contributions to the understanding of the dynamics of poverty, behavior under risk, investment decisions, technology adoption, market participation, and the impact monitoring and evaluation of technologies, social protection policy and other government and NGO programs. Creation of meta-data sets that draw together the information available from longitudinal micro-level studies in a range of countries would be an invaluable international public good. The production of the underlying data for the different farming systems of the world is an international public good akin to the production and conservation of germplasm." There was limited response to the SSSRP's query about best social science practices in the CGIAR. One response, again from Cynthia Bantilan (ICRISAT), argued that "the best examples of current social science practice within the CGIAR are baseline surveys of representative farmers in the target locations and tracking the dynamics of adoption and impact of technologies developed and released by them and their partners." Brent Swallow (ICRAF and ASB) pointed to CAPRI as an example of best social science practices in the System: "A good example of effective social science practice in the CG system is the Collective Action and Property Rights systemwide program, CAPRI. Ruth Meinzen-Dick's year-end report for 2008 shows the magnitude of the intellectual contribution of CAPRI over the past year - typical of sustained high output over the last several years (e.g., 91 working A-24 papers, almost all of which are subsequently published in the international peer reviewed literature, several books, several special issues, invitations to engage in a number of policy processes, etc.). Capri has been extremely effective in harnessing the comparative advantage of the CG system in empirical studies in contrasting sites, comparisons and synthesis of lessons across sites, resources and contexts, in harnessing inputs from a range of the social science sub-disciplines, it has supported work at almost every centre, and has effectively tapped into the intellectual capacities of universities across the world. The governance structure is effective, but not onerous. It is very well convened by IFPRI and has benefited from great continuity in the person of Ruth Meinzen-Dick, whose reputation extends well beyond the CGIAR. CAPRI deserved to win the CG partnership award." In response to the SSSRP query about major emergent lines of research in which CGIAR social science must engage more extensively in order to ensure the continued relevance and impact of the System, John Dixon (CIMMYT) replied "Risk and vulnerability (best seen as re- emergent response to growing economic and climatic volatility); institutional innovation and learning; climate change (not sure if this should still be termed "emergent")." CGIAR Social Science in Upcoming Mega-programs There seems to be a desire, suggested by Jamie Watts (Bioversity), Doug Pachico (CIAT) and John Dixon (CIMMYT), for guidance as to the role of CGIAR social science in the upcoming mega-programs and that the review panel interact directly with the transition team that is developing the list and content of mega-programs for the new Consortium. John writes that "a coherent position paper is urgently required CG Social Science (perhaps as a "by- or interim product of this Stripe Review process) to feed into the Change Management process and also the preliminary and final definitions of CG Megaprograms." "Is there a social science (poverty and livelihoods for example) mega program that is mostly social science oriented and within which we aim to build a critical mass of SS capacity, or is social science more rationally built into the other mega programs in the interest of ensuring the non-social science research adequately contributes to broader development goals. Perhaps we would propose a kind of modus operandi across the board for all mega programs that includes aspects of social science research, or at least more deliberate gender goals. Impact assessment in the CGIAR, largely if not completely a social science area of work, is another example of social science research that could be very much affected by taking a mega program rather than center oriented view. It would be a missed opportunity if the review did not really focus more on the mega program level implications as compared to a center orientation." (Jamie Watts, Bioversity) "Clearly CG social science has been organized more than 90% around centers and this mode of operation is not going to evaporate overnight, as Chris points out. Consequently the review has to be concerned about the performance of social science in the centers and what its future prospects may be in the centers. However, as we think about future prospects for CG social science Jamie is urging that we think not only about the centers but also about how social science would be organized in the future mega­ programs. This seems like a timely challenge worth some thought. Although the nature of mega-programs is vague at this point, can this social science review provide any insights as to how social science might fit into future mega-programs? I suspect that the nature of some topics (e.g. gender, policy, impact) is such that if there were mega-programs around them, they would be implemented principally by social scientists. I would be inclined to think that all mega-programs should have some social science element in them, and it would be an error to envisage mega-programs without any social science or with their social science A-25 provided by an external mega-program." (Doug Pachico, CIAT) Sirkka Immonen (SCS) suggested the review panel consider "what social science will be needed in the outcome oriented programs (most of which are unlikely to be social science programs) to increase their impact, where does the capacity come from, where will the social science dealing with, for example, ex post impact assessment and research prioritization take place (and be funded) if these areas are not or cannot be explicitly included in the funding of a mega­ program, and what are the challenges for the Centers to remain the "homes" for disciplinary excellence and mentoring." She hypothesized that by looking closely at the social science within Challenge Programs the review panel would gain useful insights as to the prospective role of and challenges to be faced by social science in the context of the upcoming mega-programs. It is worth noting, perhaps, that many of the Center-oriented forms of research organization, monitoring and assessment - SPs, MTPs, EPMRs, etc. - appear likely to vanish (or at least be changed considerably in scope, frequency and impact) with the creation of the Consortium. And it seems an open question whether mega-programs will relieve some of the burden on scientists' time associated with writing grants and reporting on many small, restricted funding projects. Especially without knowing more about what sorts of results are expected, however, it is hard to know whether the new organization will allow more scope for CGIAR social science to influence ex ante research prioritization and understanding causal relations (as emphasized by Ruth Meinzen-Dick) and how much, if any, truly innovative social science research will be authorized or encouraged. Other Topics The e-consultation also elicited some comments on specific social science research topics. For example: 1. "The issue of protection of human subjects is not given enough serious consideration in the CGIAR in terms of research practice. We don't seem to have this as a strongly held value that is reinforced in various ways from management leadership right on through to research review committees and performance standards (individual and organizational) etc. Social science researchers (and this review) should rightly drive the CGIAR towards a higher level in this area." (Jamie Watts, Bioversity) 2. "The issue of data goes beyond dissemination to data management. There are specialized data analysis issues associated with social science research that need support (i.e. qualitative data management and analysis), and we need to reach a higher standard in terms of archiving, meta analysis or other kinds of synthesis, data management/ownership/intellectual property in partnership projects etc. There was a workshop held last year involving several centers and they produced a report entitled Summary Report on CG Data Management which was presented at the ADE meeting in June 2008." (Jamie Watts, Bioversity) 3. "Gender is one key area for CGIAR social science research. The external review found that the system as a whole has not done a good enough job on this, despite the evidence of the importance of gender analysis in agriculture and in poverty reduction. But the recent AGM has made a commitment to much more serious attention to gender, which I hope will go beyond lip service. This will require substantial social science expertise, A-26 4. "One service that the review team could do for us is to provide a list of "high quality and high impact outlets" for CGIAR social scientists to publish in (besides World Development, which almost everyone thinks of). This is a request we get a lot at CAPRi. It may be partly due to lack of exposure to social science journals (more on that tomorrow from the IFPRI librarian), but also because a lot of what we do is interdisciplinary, and therefore the regular disciplinary journals are not as interested, [and just as importantly, much of the audience I want to reach is not going to be reading the sociology journals, for example.] But is this part of why we are not considered to be publishing in the "top" journals? It has been cited as a reason why some very good CGIAR people have a hard time getting academic positions. "(Ruth Meinzen-Dick, IFPRI and CAPRI) 5. The issue of innovation systems research itself attracted a little bit of discussion. John Dixon (CIMMYT) observed that the "theory and methods exist albeit not widely applied in agricultural R&D contexts. Within a decade innovation systems and learning should be integrated with mainstream SS, not managed as a separate line of research." Post-Summary Comments from e-Consultation Participants Graham Thiele (CIP): "This seems a fair summary which does reflect most of the issues affecting social science in the centers. I see a couple of gaps: 1. More careful discussion of core funding. In the case of CIP, declining core funding has affected operating budgets for social science research (actually for all research) much more than staff time because the center covers fixed costs first. The core funding for operating budget in CIP's Impact Enhancement Division where much social science work is based is probably less than 20% of the total operating budget. As a result core funding for priority setting and impact assessment which can't be easily assigned to a project is extremely limited. In my opinion centers should be at least 50% core funded to sustain more strategic and longer term research across all disciplines. 2. Social scientists in the CG have played a major role in working with natural scientists to consider or engage users in developing new technology. In this consultation there seemed to be an implication that this is a mundane service type function. Much of this work has been in the area of participatory research, for example with participatory variety selection and participatory plant breeding. Another area in CIP is seed system development where social scientists work with breeders and virologists. This type of work may be harder to publish than cutting edge social science work, but publication certainly is possible and from my personal experience this one of the areas where social science is most appreciated by natural scientists. This kind of research requires a lot of commodity specific knowledge (and collegial relationships nurtured over a considerable period of time) and would be hard to provide from a single center of social science excellence." Carmen de Vicente (Generation CP and CIMMYT), a biological scientist, replied to the e- consultation summary that he "respect[s] what social scientists do and could do in and for the System, but I admit that most of my knowledge about you is quite recent. About 5 years ago, I took the role of Subprogram Leader (Capacity Building and Enabling Delivery) with A-27 the Generation Challenge Program. Already in the early days, I realized that we had to set priorities. Expectations from the CP were very high, the resources limited and the main trait to study (drought) a very elusive one. On top of these, about 19 food security crops and all resource-poor farmers in drought-prone environments were "anticipated" to benefit from our contribution. In this situation, I convinced my colleagues in the GCP Management Team that we had to embark on what we called "socio-economic studies", i.e. commissioning research for a number of topics that I found very relevant to target our areas of intervention, the crops and even the traits. At that time, I was based at CIAT so I talked to the GIS researchers and a couple of socio-economists sketching the matters we needed to elucidate (where are the major drought areas? Where do most of the rural poor people live? Etc...). In 2008, those first results were published: "Strategic approaches to targeting technology generation: Assessing the coincidence of poverty and drought-prone crop production" in Agricultural Systems 98 (2008) 50-61. Now, we continue to commission some research in the social sciences, although at times these find hard competition with the core subjects of the GCP and making room for them in our portfolio is not a given. And importantly, I am certain my colleagues in the GCP respect your work too, at least more than a few years ago, especially when we experience the benefits resulting from our previous activities. The fact is that, having a different professional background, I thought that social scientists in the CGIAR were here to do this type of job, for example gathering information to help prioritize; but I found that most of the information and data we needed weren't there, so we decided to commission research to find out, hoping that the results would also benefit other people. Subsequently, I agree that social science in the CGIAR should be the source for other biological scientists to know the demand for and appropriateness of a particular technology; but also for senior management in helping focus Center priorities, setting research agendas, guiding a sound resource allocation strategy and advising policy. We, in the GCP, would have had a very hard time to make some decisions without the "objectivity" provided by the results of these studies. All our community recognized we had to make choices, however no one with biological experience only is able to analyze the combination of elements that play a role in the right decisions: we know the core, you know the contextual environment. If we want to make sure that what we do is going to make the desired difference, we need you. Perhaps what we did in the GCP was somehow "donor-driven research". However, being all of us in the System, I prefer to think of it as a service. I agree very much with the idea that social scientists in the CGIAR should think of their job being in a high percentage a service. But this does not apply only to social sciences; it does also to the biological or natural sciences. Most of the work in the CGIAR environment, in my opinion, should deal with applied science, and that does not necessarily mean it has to be void of originality and of course be of bad quality. This is a matter of discussion in biological scientists' fora in the System: the procedures in place to assess performance give a lot of importance to publications and publications in high-impact peer-reviewed journals. You know more about assessing impact than we do. In our R4D environment, is it better to have a paper in Science or to ensure capacity exists in a network of national programs to take up the research outputs of one of our projects? I believe that what we do and what you do should be components of all-inclusive projects. We should work together as teams and each component should have its own budget. In this effort to do applied science, I am convinced we would find gaps of knowledge with which we could exercise our desire for innovation." John Hoddinott (IFPRI) responded "I would like to offer two observations on the discussion about the quality of research in the CGIAR. These are much more about nuance than they are substantive disagreements with what was presented in the phase 1 A-28 stripe review, but these nuances carry with them important implications for recommendations that may come next. 1. I noticed that the review makes use of something called the Hirsch h-index (see page 49 of the review); which is an index combining both quantity and quality of publications. A nice feature of the H-index is that it penalizes researchers who are "one hit wonders" and those who publish many papers of little value; it rewards researchers who are both productive and innovative (and thus are cited.) As it happens, the h-index is calculated for researchers who are registered at RePEc and RePEc publishes the names of the top 5% of social scientists as ranked by the h-index. Reviewing this ranking, my quick review suggests that there are something like 50 social scientists working primarily on economic development issues in this top 5% and that 6 of these 50 social scientists are employed within the CGIAR. So by this metric, CGIAR social scientists account for 12% (or about 1 in 8) of the top ranked social scientists working on economic development issues. To me, that seems like a high percentage and read in this light, it would seem reasonable to assume that there is indeed some high quality social science research going on within the CGIAR. But it's also clear from the stripe report that the quality of social science work is very poor. Juxtaposing these observations suggests, at least to me, that the dominant characteristic of the quality of social research in the CGIAR is not so much one of "low average quality" but rather that of immense heterogeneity in that quality of that work. While the stripe report makes some acknowledgement of this heterogeneity, I think the point could be made more strongly. 2. The report is strangely silent on the roles played by the Director Generals of the CG centers in the decline of social science research at a number of centers. If DG's are ultimately responsible for the quality of research in their institutes, then they are responsible, ultimately, for this decline. I do not make this observation in order to look for scapegoats, but rather because I feel it should inform recommendations for change. If, for example, some centre DG's have no meaningful interest in social science research, then tinkering with staffing, funding, incentive structures etc, is unlikely to produce meaningful change." A-29 Annex 4 Non-CGIAR Individuals and Organizations Consulted Individual Experts Interviewed (50, not including those who represented the organizations listed below in meetings with the panel) Akin Adesina Angelique Haugerud John Pender Jock Anderson Paul Heisey Prabhu Pingali Randy Barker Bob Herdt Per Pinstrup-Andersen Brian Belcher Doug Horton Tom Reardon Julio Berdegué Sally Humphries Mitch Renkow Hans Binswanger Narpat Jodha Diane Rocheleau Tom Brian Tim Johns Jim Ryan Derek Byerlee David Kaimowitz Emmy Simmons David Campbell Tim Kelley Melinda Smale Judith Carney Urna Lele William Sunderlin Dana Dalrymple Travis Lybbert Greg Traxler Jim Delehanty John Lynam Matt Turner Cynthia Donovan Dan Maxwell Laurian Unnevehr Michael Dove Michael Morris Norman Uphoff Abe Goldman Rebecca Nelson Timothy Williams Barun Gurung Chuck Nicholson Paul Winters Rashid Hassan Alice Pell Non-CGIAR Organizations (15) Africa Economic Research Consortium Alliance for a Green Revolution in Africa Bill and Melinda Gates Foundation Ford Foundation (Indian) Federation of Farmers Associations (Indian) National Center for Agricultural Research International Development Research Centre Kenya Agricultural Research Institute Segal Foundation Tegemeo Institute of Agricultural Policy and Development United Nations Environment Programme United States Agency for International Development United States Department of Agriculture University of Nairobi World Bank A-30 Annex 5 ARI Social Scientists Survey Results The survey questionnaire was sent to about 230 people. The response rate was 34% (78 responses). The survey was conducted through Survey Monkey, an on-line tool. 1. Your area of expertise/discipline (all respondents) 80% 70% i .... '_*K:: ....?«• ....... " ..... -te' 60% 1 50% 40% " ................." ‘ “ ' . 30% 20% 10% 0% 2. The geographic region where you are employed (all respondents) 80% 70% 60% 50% 40% 30% 20% 10% SÉÉ ljfejSÌljÌ o% J North America Europe Africa, Asia Latin America Middle East Australia and Pacific and Caribbean A-31 3. The developing region(s) where you have research experience (all respondents) 80% 60% 40% 20% 0% Sub-Saharan Latin America and Caucasus Asia and Oceania West-Asia and Africa the Caribbean North Africa 4. Have you worked at a CGIAR Center previously? (all respondents) 40% (31) respondents have worked at least one CGIAR Center. 5. Please list the Centers where you have worked (31 respondents) 17 (22%) respondents have worked at only one CGIAR center, 11 (14%) at 2 centers, 3 (4%) at 3 centers. 41% of the all the positions held were as Junior scientist, 27% as Senior scientist, 20% in Program/institute director position, 12% as Board members. Average period: 4,7 years [ Cases of work 10 9 8 7 6 A-32 6. How helpful was your CGIAR experience to your subsequent (post-CGIAR) professional career? (31 respondents) 25 20 15 10 5 0 I _____ | n very unhelpful unhelpful neither helpful nor helpful very helpful unhelpful 7. In the past five years, have you collaborated with a CGIAR scientist while you were not working at a CGIAR institution? (all respondents) 73% (57) respondents have collaborated with at least one CGIAR scientist, (current and past collaborations) 8. List the CGIAR Centers with which you have collaborated in the past five years on research, whether you currently have an active research collaboration, and whether the collaboration at the Center(s) is with social scientists, natural scientists or both. (57 respondents) 18 (23%) have collaborated with only one CGIAR center; 20 (26%) have collaborated with 2 centers 10 (13%) with 3 centers; 5 (6%) with 4 centers; 4 (5%) with 5 centers (current and past collaborations) Collaboration has been with Social scientists in 63% of the cases, with natural scientists in 8% of the cases and with both Social scientists and natural scientists in 29% of the cases. (current and past collaborations) Current collaborations are 43% of the cases. A-33 Collaboration cases with CGIAR centers 40 35 30 25 20 15 ■ 10 o i—i D o □ D é 4? ^ ^ ✓ <4 G^ 4y 9. Please identify the role you and your organization played in the CGIAR research collaboration(s) you just described.1 (57 respondents) i «OTHER" responses: Funding provision; Roles varied depending on the collaborating institution above; I have collaborated with different CGIAR researchers on different projects with different roles. For most activities listed my role has ranged from minor to leading; CGIAR staff wrote a chapter for my edited book; Chair, program review committee; Invited to a conference; Interviewing scientists and evaluating their work for others; Translate scientific evidence in policy decisions; Board Member; We exchange faculty members and students with centers; Organizing meetings. A-34 10. How helpful has your CGIAR collaboration been to your own research program? (57 respondents) 25 20 15 10 5 0 - very unhelpful unhelpful neither helpful helpful very helpful nor unhelpful 11. What is your main incentive for collaboration with CGIAR Centers? 2 (54 respondents) 100% 80% 1 60% W/ 40% 20% ■ o% u L_ Center’s Center’s Availability of Access to Access to CGIAR’s Other strengths in strengths in funds due to data field sites development natural SS research collaboration oriented science res. research □ a disincentive □ not important □ somewhat important S very important □ extremely important 2 "OTHER" responses: Collaborative colleagues who know the regions and the issues; Partnership for gender mainstreaming; Collaboration with certain scientists, not the Center as a whole; Respected, objective research institution; Opportunity to collaborate on research important to poverty alleviation; Entry into a world where some changes may be useful; Quality and availability of staff; My former PhD student just happened to be employed there as a post doc; Contract requirement -1 had to interview them; Fervent belief in the mission of the CGIAR; Work on Institutional learning and change; IFPRI’s strength in Nutrition, Poverty and Hunger; Facilitation of collaboration with local partners, research impact on policies; CGIAR's dominant role in agricultural research while (often) using ineffective paradigm; We use Centers as sites for faculty and student research and collaboration; Some good individual scientists. A-35 12. What are your perceptions of CGIAR social science, relative to other social science research groups worldwide? (75 respondents) 100% 80% 60% 40% 20% I 0% Attractiveness as a Linkages with ARIs Impact of research Quality of Quality of research Quality of research SS employer publications conducted staff □ poor S fair a good □ very good ■ excellent Perceptions of CGIAR Social Science 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 mi 0.0 L AttractivenessLinkages with Impact of Quality of Quality of Quality of Relevance of as a SS ARIs research publications research research staff research employer conducted □ Worked at a CGIAR Center O Not Worked at a CGIAR Center A-36 Perceptions of CGIAR Social Science 4.0 3.5 3.0 2.5 2.0 mm ff1.5 11.00.50.0 Attractiveness Linkages with Impact of Quality of Quality of Quality of Relevance of as a SS ARIs research publications research research staff research employer conducted □ Collaborated with a CGIAR Center □ Not collaborated with a CGIAR Center 13. What are your perceptions of the CGIAR's comparative advantage in social science within the international research community and the importance of these activity areas to the CGIAR mission "to achieve sustainable food security and reduce poverty in developing countries through scientific research and research-related activities in the fields of agriculture, forestry, fisheries, policy, and environment." (75 respondents) ._1._ L fl ■ □ no comparative advantage ■ potential but not current comparative advantage q clear current comparative advantage A-37 Perceptions of CGIAR comparative advantage 3.0 2.5 2.0 1.5 1.0 0.5 0.0 J m m [M m u □ Worked at a CGIAR Center □ Not Worked at a CGIAR Center Perceptions of CGIAR comparative advantage □ Collaborated with a CGIAR Center □ Not collaborated with a CGIAR Center 14. Please add anything you want to bring to the review panel's attention regarding social sciences in the CGIAR and how its quality and relevance could be improved. (46 respondents) Open-ended responses omitted for confidentiality reasons, with the comments incorporated into the main body of the report. A-38 Annex 6 Summary of Relevant Comments from Recent CCERs and EPMRs All quotations come directly from the original source documents. EPMR reports are available at http://www.sciencecouncil.cgiar.org/documents/external-reviews/en/. CCERs were obtained from Centers directly. Source information on the year and type of evaluation is included in the heading for each evaluation. Research Priorities and Focus 1) CIMMYT 5th EPMR, 2005 a) "The Panel could not find any documented evidence of the Economics Program having conducted, as suggested by the 4th EPMR, a priority-setting exercise for its own work" (p. 40). 2) CIP, 6th EPMR, 2007 a) "What the Panel sees instead in CIP's current strategy for Social Sciences — at least from the perspective of the Impact Enhancement Division ~ is a research agenda that on the one hand is lacking of important themes such as adoption and research priorities, and on the other has been changing rather abruptly as its outputs for the periods 06-08, 07-09 and 08-10" (p. 36). 3) WorldFish, 3rd EPMR, 2006 a) "...significant prioritization work is required to determine the relative importance and time scheduling of each research activity... significant attention must be given to the definition of the modes and strategies for partnership in the research work of the Program" (p. 67). 4) IRRI, 7th EPMR, 2009 a) "The [social science] division (SSD) is currently under-resourced, and a large proportion of its activity is funded by special projects. The Panel underlines the importance of identifying clear priorities for SSD to help ensure IRRTs capacity to deliver its products" (p. 7). b) "The Panel believes that it will be necessary to map out a social science strategy that includes a number of other institutions (many better placed than IRRI for Africa- based research) while defining IRRI's specific contributions that take advantage of its particular skills and interests, such as work around specific technological thrusts like small-scale mechanization, or work in domains such as rice marketing (both improving local efficiency and understanding the nature of Africa's rice import market)" (p. 63). 5) CIFOR, 2nd EPMR, 2006 a) "The Panel suggests that the GOV (Forests and Governance) Programme revisits and prioritize its research activities based on its goals and the activities' potential impact on poverty alleviation and the Programme's comparative advantages. The Panel also cautions the GOV Programme to maintain clarity as its role as a provider of information to inform policy formulation rather than being an advocate for particular policies" (p. 39). b) "The Panel Recommends that as a first step towards a more transparent and systematic priority setting process, CIFOR needs to formally document its current practice better by developing an integrated framework that consolidates the steps followed at CIFOR for exclusion and inclusion of projects, giving a full description of criteria, quantitative or qualitative scoring and aggregation methods used. CIFOR A-39 should then strive to move forward to improve the deficiencies of its current practice based on the features of such a formal framework. It is also clear that CIFOR needs to improve on stakeholder consultation and participation in its priority-setting activities" (p. 43). 6) IITA, 6th EPMR, 2007 a) "In the area of socioeconomics, research focus (and perhaps rigor) appears to be threatened by overload and lack of critical mass in addressing priority themes (p. 41). b) "...encourages IITA to be more assertive in developing researchable hypotheses and research strategies in each of its major restricted core projects so that technology generation becomes a clearly identified component of the development effort" (41). c) "...economists need to develop a clear set of priority themes on which to build a reputation; and they need to contribute to the institute's planning and assessment of major investments in technology generation" (p. 81). d) Socio-economists should engage in planning and impact activities with the intent that "it should encourage a healthy mix of disciplines that provide a broad perspective on the progress and potential for other projects, as well as provide syntheses and cross-project analysis for IITA" (p. 61). e) "...coherent set of objectives in which planning and assessment play a clearer role in establishing research priorities..." (p. 79). 7) IWMI, IWMI-Tata Water Policy Research Program (ITP) CCER, 2005 a) "In comparison with other similar international programmes and water 'think tanks', it is hard to beat the relevance and depth of the ITP output. Many of the themes that are addressed in the ITP studies—and the questions raised are simply not addressed by international organisations and NGOs .... More significantly, the quality of the outputs appear to several orders of magnitude ahead of the research outputs coming from the current set of Indian State water institutions (p. 13). b) "Finally, the original objectives of ITP would be well served with a longer-term perspective that would stimulate a search for a wider array of mechanisms to develop partnerships and build capabilities for a "meaningful engagement with the problem (of water)", avoiding the fragmentation of research efforts. But overall, much more clarity needs to be given to the programme design and not just for the Reviewers to follow the relevant dimensions of ITP work. The publication program needs to be evaluated to reflect the relative weight of the research efforts and assess how many research areas are appropriate within an annual work program (p. 34). c) "The Review is, however, concerned about (i) the broad scope of the research themes (ii) the balance of research themes (from micro-economic research and state/national macro-policy recommendations) (iii) the development of an effective communications strategy and (iv) sustaining the ITP momentum into a second programme cycle" (p.l). d) "The question of water and livelihoods in different ecological and socio-economic settings calls for more sustained attention and diversified engagement, notwithstanding CInl [Institutional Aspects of Irrigation - Central India Initiative] and the macro-studies on irrigation-poverty linkages. In fact, the CInl kind of approach that looks at the water-livelihoods nexus across a region with well-defined characteristics might be useful for the mountains, the northeast, and the entire traditional paddy region" (p. 18). A-40 8) IWMI, 3rd EPMR, 2007 a) "Mainstreaming of both gender and poverty research issues would help deliver the required re-orientation of research across all themes. In contrast, IWMI resources have been directed to the generation of data bases and GIS capacity" (p. 2). b) "research done on water poverty mapping and the gains made in refining the methodology should be used more widely by the Center in informing its research projects" (p. 51). c) "Theme 4 (Water Management and Environment) research activities would be strengthened by being more tightly focused on a smaller number of projects, more closely directed toward the goals of IWMI, shifted from being predominantly descriptive and static to being more dynamic and analytical, strengthened with the application of economic and social science expertise..." (p. 37). d) "The Panel recommends that the cross-cutting issues of poverty and gender be clearly conceptualized, meaningfully integrated and mainstreamed into research by appointing a Task Force and specifying core resources for this purpose. IWMI should articulate the expected intermediate and long term impacts of this research and the milestones for measuring progress" (p. 53). 9) ICRISAT, Institutions, Markets, Policy and Impacts (GT-IMPI) and Knowledge Management and Sharing (KMS) CCER, 2007 a) "Research areas that might at first glance seem more weakly tied to the GT-IMPI, ICRISAT and CG goals (e.g., linking to relief programs, HIV/AIDS impacts in ESA [East and Southern Africa], social networks and social capital) seem to have been carefully considered and developed in ways in which they are complementary to the established goals" (p. 48). b) "While in a certain sense the GT-IMPI priorities have been narrowed to four research areas and they seem consistent with goals of GT-IMPI, ICRISAT and broader CGIAR, they still are fairly broad, which means that the priorities are not as clear as they might be" (p. 50). c) "... in addition to providing support to both the management and research functions of the Institute, KMS also is a leader on advanced research on ICT-mediated extension and training... At the same time, KMS performs critical support to management" (p. 29). 10) ICRISAT, 6th EPMR, 2009 a) "...but the panel believes that there could be an IPG-type (International Public Good) role for GT-IMPI [Global Theme - Institutions, Markets, Policies and Impact] in analyzing the development of the seed delivery and supply sector in Asia. This would involve such matters as: description of the evolution of the system; analysis of the effectiveness of various systems (including OP versus hybrids) on the poor, and how best to position ICRISAT in the milieu of the changing seed supply parameters (role of the private sector etc). In short, analyzing economic and policy aspects of the systems (especially from a poverty alleviation viewpoint) and helping to position ICRISAT's ongoing germplasm improvement work in Asia appropriately in relation to the emergent socioeconomic trends. If undertaken, this would be a logical counterbalance to the work in Africa and would demonstrate an appealing strategic approach to project definition according to region" ->define projects according to region and situation, (p. 42). A-41 11 ) ICARDA, NRM & Social Science CCER, 2005 a) "The potential for improving livelihoods through improvement of the livestock component in these systems is under-researched in ICARDA's current programme. The panel recommends that the specific requirements of an enhanced livestock element in the overall programme be examined" (p. 3). b) "Given the certainty of water scarcity throughout the CWANA region and the increased competition amongst alternative users for this scarce resource, ICARDA has an opportunity to take the lead in the increasingly important area of the re-use of wastewater for irrigation. Policies and technologies for the use of such waters have not been well-defined in much of the region and, in many cases; this may be the only source of water available for irrigation or supplemental irrigation. Thus the panel recommends that ICARDA continue and intensify its research to better utilize wastewater as a source of water for irrigation" (p. 14). c) "ICARDA has effectively reshaped its research agenda to be strongly relevant and in harmony with the new CGIAR research priorities. The latter include among others, 1) the sustainable and equitable management and intensification of water and land resources, 2) the provision of affordable, adequate, and nutritious food supply through genetic improvement, 3) income enhancement through agricultural diversification and value-addition linking the poor to markets, 4) policy improvements, institutional innovations and capacity strengthening to support sustainable reduction of poverty and hunger" (p. 23). 12) ICARDA, 5th EPMR, 2006 a) "In achieving success in improving agricultural systems, access to appropriate knowledge and technologies has to go hand in hand with the existence of adequate markets and policies that enable farmers to take up innovation and to advance...Therefore, social science research has to be an important and integral part of any research program aimed at meeting the challenge of increasing productivity of dryland systems as a means of improving livelihoods" (p. 12). 13) IPGRI/BIOVERSITY, 5th EPMR, 2003 a) "The Panel encourages IPGRI to more explicitly address economic and social issues in its Plant Genetic Resources (PGR) policy strategy. These issues should encompass gender considerations in access to genetic resources and benefit sharing, economic impacts of PGR guidelines and laws and participatory approaches in regulating access to on-farm PGR systems. This is consistent with the first recommendation of the CCER 1999 on IPGRI's work on human and policy aspects of plant genetic resources. In particular, the Institute should give more attention to the need "to conduct systematic research planning exercises to identify broad conceptual frameworks based on social, economic and ecological patterns, for priority setting and determination of different types of social science and policy research activities" (P-26). 14) ICRAF, 3rd EPMR, 2006 a) "...program would benefit by developing a comprehensive framework that links agroforestry (and other land uses) at the field-farm-landscape-watershed levels and links biophysical, socio-economic and environmental outcomes" (p. 89). b) "...balance between biophysical research and socio-economic research is too heavily weighted towards the former with a deficiency in research on market processes and market development" (p. 39). A-42 15) ILRI, 2nd EPMR, 2008 a) "Overall the Panel believed the Theme contains some gems that can be missed by the OPs' [operating project] diversity of projects. Those with technical elements combined with social science research seem the most likely to generate the systems information that ILRI seeks" (p. 45). 16) Cl AT, Rural Innovations Systems CCER, 2006 a) "There needs to be overall planning and vision in the research so that the pieces sum to greater than the sum of the parts...More time need to be allocated for proposal development, which should build in strategies to turn the develop experience into IPGs, [international public goods]" (p. 21). b) "Shift the focus from inputs (development action plus manuals) to outputs (research leading to new insights/public goods creation that can be used by others beyond the narrow client groups of the particular development action." (p. 23). c) Recommendation: "Reformulate the work of the Institute to be first "issue-driven" (such as responses to major impending shocks, such as climate change, trade liberalization, supermarket penetration, and HIV/AIDS, then formulate research questions that derive from those issues, and then choose the appropriate methods (a combination of various kinds of action, research, and meta-research, discussed below). The categories of general issues fall roughly into: (a) policies (impacts of, design of), (b) institutions (social, economic, etc.), (c) organization (of farmers, of markets, of households), (d) technology (design of, adoption of, impacts of). One can then cross that with various specific issues" (p. 18). d) "The Institute at present is undertaking development action that is, in its great majority, not produced as IPG, but is rather a localized, private good only ...at present, there is little connection between the development actions being undertaken, and the research and meta-research. In fact much of the work of the institute at present consists of (1) general methodological and training guides for development action (that comprises some participatory analysis activities), and (2) development actions that are informed by those guides. There is very little research based on insights from the development action work, nor meta research on how the research (incarnated in the guides) influenced the action, and an evaluation of that action, and feedback to research methods or even development methods. All this "enclave" development action work should either cease (and be transferred to NARS and NGOs and government, who are doing most of the action work at present anyway) or transformed" (p. 18). e) "When participatory research methods, such a PM&E and network analysis, are used to help individual communities, they only become IPG when the findings relating contexts, processes, and impacts are systematically analyzed and made available through the refereed publications, procedural handbooks, and international capacity building with implementing partners...Thus unless research protocols are set up and systematically implemented, in order to document the conditions under which particular strategies work under what conditions" (p. 23). 17) IFPRI, Institutional CCER, 2004 a) "Development Strategies and Governance Division [DSGD] is building on a solid background of research, produces international public goods, and has the potential to use state of the art tools. Earlier efforts by IFPRI related to strategies, has gone into building tools, and less on applications. Greater emphasis on issue-oriented research A-43 b) "Much of the work of [Environment and Production Technology Division] is based on case studies, but there is also considerable use of formal modeling and simulation tools. Innovative, integrated conceptual frameworks are needed to be able to draw conclusions. There is scope for more such approaches in the synthesis of case studies to define strategies for replication" (p. 6). c) " [Food Consumption and Nutrition Division] has incorporated significant multidisciplinary work... The new research plan shows a clear response to the new strategy of the institute. However, it remains to be seen how the division can actually perform the task required with the present team. A stronger multidisciplinary of the team is needed. Moreover, cooperation between nutritionists and economists needs to be improved. Some interesting research activities in this division have not, or will not, lead to professional publications because of lack of funding" (p. 6). d) "The division [Markets, Trade, and Institutions Division] has been restructured recently through the merger with elements of another division. It is understandable that the division's new research agenda is not yet as coherent as it needs to be. Institutional aspects urgently need to be integrated explicitly into present research activities. Close cooperation with FCND is highly recommended. The division also needs to integrate external trade issues strongly with internal market research" (p. 7). e) "IFPRI has accepted—even more strongly in the new strategy—that it has to be active not only in research but also in capacity strengthening and outreach. The new emphasis on involvement in the policy debate may require a rebalancing of expenditures. ... [W]e caution against weakening the research base" (pp. 20-21). 18) IFPRI, 4th EPMR, 2006 a) "The Panel recommends that IFPRI sharpen its system of priority setting so that it will be more transparent how it decides what projects are most appropriately included in IFPRI's research agenda, and which are best left undone or left to other research institutions" (p. 10) b) "The challenge facing IFPRI is that social science impact is notoriously difficult to measure, and there are no pathways to carrying out impact assessment that will be convincing to everyone. The Panel commends IFPRI for its serious and sustained efforts to move forward on the impact assessment agenda (p. viii).... it is important to provide evidence that spending on such research [social science] generates returns to investors that justify the costs. IFPRI's impact assessment program attempts to establish methods to conduct such assessment, and apply the methods to obtain evidence of research impacts" (p. 39). c) "Research on agricultural trade negotiations in the WTO and their implications for developing countries should rank high on MTID's [Markets, Trade, and Institutions Division] agenda. Developing country negotiations have to assess the relative merits of many competing negotiation proposals, for which they often lack the required technical support from their own countries. In recent years, except for very few studiesl2, IFPRI's WTO-related trade research appears weak in terms of research capabilities, output and presence in the field, principally due to lack of experienced staff. In the comments the Panel received from peers, disappointment that IFPRI was not more prominent in current discussions on trade negotiations was one of the criticisms that emerged most. The Panel believes that considerably more effort and focus are required if IFPRI is to re-establish its identity in this area" (p. 23). A-44 19) IFPRI, International Service for National Agricultural Research (ISNAR) Division CCER, 2007 a) "[A]s a result of difficulties encountered by ISNAR staff in raising restricted funding for their research, they have increasingly worked in support of projects led by other IFPRI divisions... Their involvement has, no doubt, enhanced overall IFPRI's accomplishment, but at a cost of internal coherence and effectiveness of the ISNAR's program" (p. 21). b) "[T]he process of developing the ISNAR Division's medium-term strategy, defining its concept and formulating its priorities has been in the Review Team's opinion transparent and systematic. However, translating the concept and priorities into a work program with individual projects and activities has not been pursued with the same consistency and rigor......The departure of IRS [internationally recruited staff] research staff may also be partly explained by these deficiencies. It should also be noted that unrestricted as well as restricted funding during 2005 and 2006 declined, putting increasing pressure on staff to pursue projects even if they were not in line with subtheme priorities. These problems affected subthemes and programs differently. Subthemes, where senior research staff provided continued and consistent guidance, the work program and performance were good; subthemes where staff, particularly senior staff, left or changed suffered to a larger extent" (p. 23). c) "There are ... several factors constraining the [ISNAR] Division's ability to execute its mandate of capacity strengthening [including: inadequate staffing, lack of explicit inclusion of capacity building in projects, lack of incentives to capacity build relative to publishing, lack of funding, lack of operational definition of capacity]" (p. 20). d) "The Review Team recommends that ISNAR continues with its efforts to shift focus from services to research. It further recommends that the IFPRI Task Force on capacity strengthening comes forward with incentive systems which will encourage researchers, both in ISNAR and the rest of IFPRI, to undertake capacity strengthening research and outreach" (p. viii). 20) WARD A, 5th EPMR, 2007 a) "Rice policy and development research needs to be well focused...results-oriented focus is needed to identify what works, what does not and what new approaches should be tried" (p.61). Staff Issues 1) CIMMYT 5th EPMR, 2005 a) "In 2000, the Economics Program peaked in terms of staff and budget after which the program size has been reduced" (p. 38). 2) CIP, 6th EPMR, 2007 a) Decline in number of social science staff since 2002.... "In addition, not all staff members with socioeconomics background are involved in socioeconomic research" (p. 37). 3) CIAT, Rural Innovation Systems CCER, 2006 a) "The staff needs training in formulating research plans and consulting the literature. That training should be done "participatively" with mentors. A one-off training will not do. There is much doubt that a number of the staff will benefit from the training as their initial training/background is inadequate to use it effectively" (p. 21). A-45 4) ICARDA, NRM & Social Science CCER, 2005 a) "...the panel recommends that a critical mass of social science researchers with required competence profiles (institutions, policy and gender, among others) must be secured to allow timely delivery, continuity, and high quality output" (p. 2). b) "The current nature of social science at ICARDA indicates optimism. Individual social scientists do not display the disfunctionality common among social scientists in natural science-dominated environments. They perceive themselves as having a key role to CCER of ICARDA's Natural Resource Management Program and Socio- Economics play in the NRM Program and in the Center as a whole. This role has been attained through a great deal of hard work, persuasion and mutual learning and is exemplified by intense involvement in development of future research strategies and commitment to the goals of individual projects" (p. 19). 5) ICARDA, 5th EPMR, 2006 a) "The Panel recommends that additional staff is hired to increase the capacity of the socioeconomics Program" (p. 67). 6) ICRISAT, 6th EPMR, 2009 a) "There has been a positive increase in core competencies and staff numbers within GT-IMPI (Global Theme - Institutions, Markets, Policy and Impacts) in recent years. Examination of staff CV's indicates that, broadly speaking, people with the requisite experience and training are being hired" (p. 38). 7) ICRISAT, Institutions, Markets, Policy and Impacts (GT-IMPI) and Knowledge Management and Sharing (KMS) CCER, 2007 a) "[T]he number of socioeconomists has increased in 2006 to 15 since the drop in the late 1990s to eight when ICRISAT suffered serious budget reductions" (p. 10). b) "KMS, particularly the ISU [Information Systems Unit] seems understaffed and critical work on IT systems and applications is undertaken by interns on short term assignments" (p. 31). c) "The Panel suggests giving lead responsibilities to the team leaders for the areas in which they have core competencies even if it means cutting across regions" (p. 12). d) Of 959 trainees based at ICRISTAT during 2002-05, "only 29 trainees working with GT-IMPI for 2002-05, of which 12 (4 males and 8 females) are Doctoral students spending about 6 months to one year at ICRISAT" (p. 40-41). e) "While the Panel respects the need to decentralize expertise in social science across the regions, with decentralization come other challenges. One is that the staff is inevitably spread thinly and gaps appear close to the surf ace....The Panel sees another important challenge in decentralization - that it can weaken the global perspective unless particular attention is taken to avoid this" (p. 58). 8) IITA, 6th EPMR, 2007 a) "...survey asked respondents to rate the Centers' abilities to hire and retain high quality staff. IITA scored 11 points below the average Center score for this attribute" (p. 43). b) "There are currently 11 socioeconomics at IITA, six of whom joined since the last EPMR. This staffs are no longer housed in a single unit or project but are dispersed across a range of projects. This has some advantages, but there is a risk of insufficient disciplinary leadership and coherence" (p. 79). A-46 9) IPGRI/BIOVERSITY, 5th EPMR, 2003 a) "IPGRI does not have a full time in-house economist, but relies on a half time staff based at IFPRI. IPGRI needs to build its capacity in this aspect. (See Recommendation 6)"(p.28). 10) IWMI, IWMI-Tata Water Policy Research Program (ITP) CCER, 2005 a) "The present staff structure provides adequate high quality support and guidance to a young team of Indian researchers who are going on to PhD programmes ... It is not felt that the Principal Scientist is overburdened in terms of supervision and management workloads.... The [team] do not feel that they lack supervision from senior staff and that the process of peer review within the office and with colleagues at the IWMI office in Hyderabad provides them with enough 'reality checks'" (p. 12). b) "The current staff list offers a mix of backgrounds with emphasis on policy and economics but some engineering skills at senior levels... Intellectual exchange appears to be vibrant" (p. 34). c) "One possible way forward is to create an Advisory Committee comprising of individuals of standing in relevant fields, such as water, livelihoods, natural resources et al to guide the governance of the program. ... ITP would also be well served if it had a constellation of three to four senior persons to act as a sounding board to the Principal Scientist" (p. 33). 11) IWMI, 3rd EPMR, 2007 a) "...IWMI should strengthen its social science capacity so that it is able to undertake these kinds of analyzes with the needed rigor" (p. 51). b) "The Panel is concerned that by dropping the Theme area of Policies, Institutions and Management, IWMI has lost its concentration of research into a key niche area and with it, experienced senior social scientists..." (p. 55). c) "Frequently in the panel's discussions with the NARs and partner agencies, economics was specifically identified as an area of their need. Yet economics was also identified as a deficiency in IWMI's skill base" (p. 56). 12) WorldFish, 3rd EPMR, 2006 a) "With respect to staff, in 2005 PESS (Policy, Economics and Social Sciences) Discipline had a total of 18 scientists, out of which a total of 13 are Ph.D. (72%). Thus, overall, there seems to be a good representation of qualified scientist in PESS" (p. 66). b) But only six are classified as "senior", (p. 66). c) Since then the numbers have reduced (p. 67). 13) WARD A, Social Science CCER, 2006 a) "The Panel feels that had the positions of production economist, policy economist and sociologist not remained vacant for long, the contribution of social science unit could have been many times more" (p. 76). b) "The panel recommends that the team of social scientists at WARDA should be expanded to include two production economists, two policy economists, two sociologists and one each of technology transfer scientist, impact assessment economist, marketing and trade economist, and resource economist" (p. 77). 14) WARDA, 5th EPMR, 2007 a) "What is also apparent is the difficulty in recruiting competent agricultural economists at WARD A... this is particularly evident in the senior economist's positions" (p. 62). b) "WARDA must mainstream social sciences research regarding adoption in the technology generation programs, ensuring better integration and feedback" (p. 100). A-47 15) IFPRI, Institutional CCER, 2004 a) "It is possible that the quality of IFPRI's research could be improved were IFPRI to increase the attractiveness of its employment packages to keep existing staff and attract potential staff. We suggest considering tenured positions for division directors after their first term and for some senior research fellows after the second term (p. 3). b) "Some economists need experience in institutional economics, governance, and political economy research, but more than in the past, IFPRI's staff need to be complemented with political science scholars or sociologists" (p. 4). c) "IFPRI's redefined agenda requires more in ter divisional cooperation. This concerns participation in cross-divisional projects in addition to participation through relocating researchers from one division to another for periods of time. This second type of cooperation will become more important in making the best use of small numbers of non-economists" (pp 4-5). d) "This limited experience of new research staff, and their short stay at IFPRI, may be of concern" (p. 13). 16) IFPRI, EPMR, 2006 a) "Indeed IFPRI is not explicitly identified as a social science institution in the mission or strategy documents. It is implicit, however, in as much that the DG (Director General), Division Directors, and the great majority of the professional staff are social scientists, largely economists-- the remainder being nutritional scientists" (p. 66). b) "The new structure of MTID [Markets, Trade, and Institutions Division] covers a diverse set of issues for a relatively small Division. Given the human resources available, the set of research questions is perhaps too large, not because the themes and projects selected are unimportant or not related to one another, but because the structure and diversity of the agenda is potentially unsuited to the capacities of the limited number of experienced researchers comprising the MTID. The limited human resources available restrict the degree of specialization of research skills of the Division and inhibit the potential complementarities that might arise from a greater number of specialists 26 engaging across the diverse set of themes" (p. 25). c) "...the high turnover of post docs and the difficultly in attracting and holding experienced researchers more generally reduces both the depth and continuity of lines of research" (p. 26). 17) IFPRI, International Service for National Agricultural Research (ISNAR) Division CCER, 2007 a) "Management of the ISNAR Division has changed twice during the merger period (see Annex 3). ... Internationally Recruited Staff (IRS) has also been unstable.... As a result of these changes the work of ISNAR has been carried with continuity by one Senior Research Fellow, four young Research Fellows and the Head of ASTI. They are to be commended for their excellent work, particularly considering the enormous pressure on them for fund raising, in which younger staff are still less experienced" (P-23). b) "[T]he Review Team sees a high priority in recruiting additional high caliber senior researchers.... The Review Team recommends that IFPRI management consider the possibility of seconding more of its senior researchers to the ISNAR Division" (pp. vii, viii). c) "There are, however, several factors constraining the [ISNAR] Division's ability to execute its mandate of capacity strengthening. Perhaps the most critical constraint is the level of staffing... The mix of professional disciplines is also inadequate for the purpose of undertaking broad-based capacity strengthening. Previous evaluations have highlighted the imperative of recruiting professionals with background in A-48 18) ILRI, 2nd EPMR, 2008 a) "...currently, economists form a large proportion (33%) of ILRI research staff, whereas animal sciences and animal health account for 27%, biology for 18 and plant sciences for 10. While the Panel supports the need for a disciplinary mix, it believes that ILRI should carefully monitor the skills it needs and assure itself that it has the diversity and depth required. The Panel suggests that ILRI carefully assess shared staff time and expertise among Themes and OPs to ensure that ILRI maintains the right mix of expertise and makes best use of it" (p. 51). 19) IRRI, SOCIAL SCIENCE CCER, 2007 a) "In 2005, with the departure of an economic development and policy specialist, the vacancy was filled with an agricultural economist in part to support the IRRI initiative in Africa but also to provide input into ecosystem management. In short, these changes have resulted in trade-offs that strengthen research capacity in some areas but weaken the potential for macro-policy research" (p. 4). b) "In this environment [expanding research areas and activities] and with the growing emphasis on the matrix management that separates program and division leadership responsibilities; there is a danger that SSD will lose its capacity for independent research. What is the solution? Some combination of the following would seem warranted: (1) an increase in IRS [internationally recruited staff] from six to seven, with particular attention to the needs for policy research and the increasing demand for social science research as IRRI moves into Africa; (2) hiring of IRS on selected projects; (3) a refocus of the support to biological sciences to extract more cross­ cutting international public goods (IPGs) through comparative and synthesis studies—a good example is the possibility for additional synthesis work on participatory methods related to scaling-up issues; and (4) 9 strengthening linkages or collaboration with IFPRI, universities, and other institutions, including those in Asia and Africa, to provide support and collaboration on policy issues" (p. 8). Publications 1) CIMMYT 5th EPMR, 2005 a) High number of publications with good collaborations, "academic level and relevance has been excellent" (p. 39). b) "No publications dealing with impacts at the local level were found" (p. 39). 2) CIP, 6th EPMR, 2007 a) "...the Panel's assessment of CIP's IRS research staff productivity in terms of their publishing record during the last five years reveals that social scientists (including economists) have been highly prolific...CIP's social scientists produced 137 publications in the last five years, including peer-reviewed journal articles, books and book chapters, and conference papers. This represents nearly a fifth of all publications produced by CIP's IRS researchers...This Panel reviewed a number of social science publications, from peer reviewed journal articles to books and manuals, and notes their high quality with respect to methodology and analytical rigor. The findings are also interesting and relevant...the Panel notes that a great majority of the key publications are produced in English only, which greatly limits their potential utilization" (p. 38). A-49 3) CIAT, Rural Innovations Systems CCER, 2006 a) "The Institute at present is mainly producing guides and training manuals, and descriptions of the application of those guides, but producing little refereed journal research neither on those applications nor on processes and behavior. There is a small amount of analysis of markets, and adoption of technologies, but very little and not mainly published in journals. The journals in which the work is published are usually not major ones" (p. 20). 4) IITA, 6th EPMR, 2007 a) "The unfortunate reality is that the requirements of journal publications are often inimical to practical adoption analysis as part of an on-going program of research" (P-80) 5) IPGRI/BIOVERSITY, 5th EPMR, 2003 a) "Its publications have been cited and used internationally" - on indigenous rights and knowledge (p. 27). b) "IPGRI was instrumental in preparing some of the background papers that guided the evolution of and negotiations on the International Treaty of Plant and Genetic Research (ITPGRFA)" (p. 28). 6) IWMI, IWMI-Tata Water Policy Research Program (ITP) CCER, 2005 a) "The stated aim of ITP's communication and dissemination effort is to effectively transmit 'convincing messages' distilled from research outputs to various levels of decision makers. The number and range of research publications produced form a tremendous resource for the preparation of communication and policy related materials. But much of the communication effort has concentrated on dissemination to researchers and NGOs. Even this outreach can be expanded by having more papers published in well-known journals" (p. 3). b) "Results published in external publications are also limited at this stage" (p. 21). c) "Overall, ITP would benefit from a communications strategy that builds on the IWMI marque, elaborating the research results into several key outputs using the in-house IWMI skills in publishing where they can boost the appearance and presentation of the material. More exchange between ITP and IWMI HQ on publication guidelines and editorial styles would help significantly" (p. 14). d) " 'While the outputs from the IWMI-Tata Water Policy Program are a useful contribution to the debate on water policy, there are ways in which program outputs could be improved substantially. These include: o More rigorous editorial control over the outputs. The standard of some briefing notes is quite poor and, even in the better notes, some of the writing is really quite sloppy. o More attention referencing publications. The innovative thinking in, say, the recent publications of the "Fluid Mosaic" group is not reflected in IWMI-Tata publications. o Also, some briefing papers selectively reference work that is itself marginal, o A greater willingness to challenge the sanctioned discourses that have developed in relation to water harvesting, groundwater management, linking of rivers etc. o A greater willingness to discuss sensitive issues that have a major bearing on water management such as corruption, cooking of water-related statistics, political interference, electioneering etc. o Better handling of issues of scale. Statements such as, "groundwater recharge needs to be augmented, for example, by means of mass rainwater harvesting and A-50 —personal communication from Dr. Charles Batchelor to J Burke April 2004" (p. 22). 7) ICRISAT, 6th EPMR, 2009 a) Regarding GT-IMPI (Global Theme - Institutions, Markets, Policy and Impacts) "Overall publication quality is viewed as adequate, after giving due consideration to factors such as relevance and the service function to the Center overall" (p. 40). 8) ICRISAT, Institutions, Markets, Policy and Impacts (GT-IMPI) and Knowledge Management and Sharing (KMS) CCER, 2007 a) "The Panel commends ICRISAT for resuming work on the VLS [Village Level Studies]... However the Panel believes that ICRISAT should also be at the forefront of the analysis of the data. While the data is a true IPG [International Public Good] and can serve the interests of many partners, ICRISAT should seize the opportunity to seek collaboration to analyze the data especially in the context of the big challenges ahead of SAT [Semi-arid Tropical] agriculture" (p. 17). b) "The quality of the research is generally good, and a number of studies have successfully satisfied criteria of peer-reviewed international journals. There are areas for potential improvement, nevertheless, such as putting the VLS [Village Level Studies] data in a broader context as noted, treating important right-side variables such as social networks as reflecting behavioural choices, and adding experimental components to the data collection (e.g., random distribution of information about new varieties) that would not only permit the evaluation of the experiment but would permit investigating using instrumental variable methods the analysis of the impact of the variables directly affected (e.g., seed adoption) on other outcomes (e.g., yields)" (p. 21). 9) IWMI, 3rd EPMR, 2007 a) "This research - but especially the ADB publications which were many and were targeted to a wider and more general audience - has generated interest globally and at different levels and was/is widely referred to and celebrated by many in the international community" (p. 50). b) "The Panel further notes the delay in producing peer-reviewed publications from the high profile and multi-country ADB funded project" (p. 51). 10) IRRI, SOCIAL SCIENCE CCER, 2007 a) "SSD has produced a large and increasing number of publications (Table 2), but relatively few have been published in top peer-reviewed journals (about 12 per year on average and 19% of the total). Although difficulty of publishing social science research outcomes in top international journals is a problem common to CG centers (perhaps except for IFPRI), IRRI must take this problem seriously and try to find ways to prevent the reputation of its social science research capability from eroding" (p. 9). 11) ILRI, 2nd EPMR, 2008 a) "Between 2003-2005, 11% of the papers in peer-reviewed journals were authored solely by ILRI scientists; 60% were co-authored with NARS scientists; and 53% were co-authored with ARI scientists. The first author on the papers was 36% from ILRI, 37% from ARI, 26% from NARS and 2% from others. The frequency with which ILRI is a co-author is an indication of the level and frequency of ILRI's collaborations with partners in the South and North" (p. 57). A-51 12) ICARDA, NRM & Social Science CCER, 2005 a) "Such publication is the quality control point at the crossroads between research and development. Making due allowance for the varying nature of the work, the panel noted that the output of such referred publications from ICARDA's Natural Resources Management and Socio Economics Programmes is below expectation" (p. 3). 13) IFPRI, Institutional CCER, 2004 a) "Researchers at IFPRI are under significant time pressure; the dependence on project funding does not always allow them to complete projects with a bona fide research output, which could be a journal article, a research report, or a book contribution" (p. 3). b) "IFPRI has a sound system to support high-quality research. Research proposals are carefully prepared and discussed in seminars, and the Communications Division has prepared guidelines for proposal writing and presentations, which are very helpful. All IFPRI research publications —research reports, books, and some discussion papers— undergo a rigorous review process" (p. 15). c) "We recommend appointing an external chair to the Publications Review Committee. Moreover, external reviewers should not be chosen from candidates proposed by authors" (p. 4). d) "IFPRI staff is quite successful in publishing peer reviewed articles" (p. 16). 14) IFPRI, 4th EPMR, 2006 a) "During the last six years, IFPRI has worked to develop improved methodologies for assessing the impacts of policy and social science research. The recently published book, What's Economics Worth? Valuing Policy Research, is based on two IFPRI- sponsored conferences focused on measuring policy-focused social science research. The most recent conference was held in 2001 in The Hague, Netherlands and focused on how to measure or value the economic impact of policy-oriented social science research and how to enhance such research in policymaking environments" (p. 95). 15) IFPRI, International Service for National Agricultural Research (ISNAR) Division CCER, 2007 a) "Clearly, the ASTI [Agriculture Science and Technology Indicator] work is well established ... Also, the research output of the other programs of this subtheme [Agriculture Science and Technology Policy] has been presented at international conferences and symposia; from 2005 up to June 2007, five peer reviewed journal articles have been published and two discussion papers. This is a respectable output of a young and dynamic research staff, produced in a difficult environment of senior researchers' departures and a change in the Division's management" (p. 9). b) "Project research outputs under this subtheme [Institutional Change in Agricultural Innovation Systems] include six peer-reviewed journal articles... four IFPRI food policy reports/briefs discussion papers and country reports, a discussion paper ..., a searchable online database..., and numerous presentations...Again, with a dynamic and active young research staff, this group has the potential to substantially contribute to ISNAR's output and impact. The program could benefit from some guidance in selecting and focusing individual activities to become more in line with subtheme objectives" (p. 13). c) "Thus, work of this subtheme [Organization and Management for Strengthening Agricultural Research] has led to more service output and less to peer-reviewed publications (except ... four publications based on previous research). The Review A-52 d) "The research output of this subtheme [Learning and Capacity Strengthening] is excellent" (p. 20). CGIAR social science collaborations with other scientists, centers, NGO's etc. 1) IRRI, 7th EPMR, 2009 a) "The work of Social Science Department is widely appreciated within IRRI and its activities are quite well integrated with the rest of the institute's work" (p 61). b) A further constraint is the shortage of strong NARES collaborators. Social scientists have always been in short supply in NARES (and even when present are often ineffectively managed and motivated)" (p. 61). 2) ICRISAT, Institutions, Markets, Policy and Impacts (GT-IMPI) and Knowledge Management and Sharing (KMS) CCER, 2007 a) "This renewal of partnerships is now being followed up by the development of a proposal for a coordinated, goal-oriented, fundamental research on Socioeconomic Mobility, Agriculture and the Rural Transformation in a partnership among multiple CGIAR Centres, selected ARIs and NARS and universities from throughout the South" (p. 18). b) "Looking forward the Panel suggests that the culture of actively engaging all partners in such collaborative ventures early on with explicit discussion of costs and responsibilities be encouraged." (p. 27). c) "The Panel also recommends, as noted above, that GT-IMPI aggressively pursue partnerships with whom to collaborate in the analysis of these rich data" (p. 20). d) "The one area that the Panel felt could be strengthened in terms of collaboration is in encouraging all scientists to understand that scientific data collected while at ICRISAT are Center public goods, and often global public goods that belong to the Institution and not to them or their research group alone. Renewed emphasis on developing a "sharing culture for scientific data" among scientists within ICRISAT would strengthen collaboration across research groups" (p. 47). e) "Within ICRISAT, concerns about the time required in discussions with social scientists are sometimes seen as a disincentive to collaborate with GT-IMPI. GT-IMPI may wish to evaluate how it could realize better time management to achieve the same goals" (p. 56). 3) ICRISAT, 6th EPMR, 2009 a) "GT-IMPI (Global Theme - Institutions, Markets, Policy and Impacts) has worked with an impressive list of research partners - NGOs, NARS, private sector, other CGIAR Centers, and other ARIs. Field visits by the panel confirmed the quality and productivity of these partnerships" (p. 38). b) "Apart from partnerships with other social science disciplines, there are also partnerships with biophysical scientists both within ICRISAT, and beyond. This adds further responsibility to GT-IMPI (Global Theme - Institutions, Markets, Policy and Impacts) for partnership building - that responsibility appears to have been well- handled by the GT-IMPI team" (p. 44). 4) IPGRI/BIOVERSITY, 5th EPMR, 2003 a) "Its collaboration with IFPRI, though at its early stages, has enabled it to work on economics of PGR (Plant Genetic Resources). The Panel commends IPGRI for initiating this collaboration and encourages it to strengthen its partnership with IFPRI" (p. 28). A-53 b) "The Panel is, however, concerned that with the growing demand for IPGRI's engagement in socioeconomic work, particularly on the economic aspects of PGR [plant genetic resources], the Institute will not be able to adequately respond to the needs of its clients'^ p. 28). 5) IWMI, IWMI-Tata Water Policy Research Program (ITP) CCER, 2005 a) "ITP has a significant number of effective partnerships among NGOs and researchers. Government agencies and mainstream scientific research community, however, remain poorly represented, as do certain regions of the country... Future development of partnerships under ITP may need to be managed more strategically to produce programme impact by way of creating a wider "water community" and significant policy changes" (p. 2). b) "... it is felt that the ITP Partner institutions are not remunerated sufficiently for the work they undertake" (p. 13). 6) IWMI, 3rd EPMR, 2007 a) "They can do this by further developing strategic partnerships with IRC (International Water and Sanitation Center) and others, and by clearly identifying the host of issues in domestic water supply (and sanitation) that directly relate to wider uses of water in this case what we can call "irrigation-plus" rather than "domestic- plus" so that the entry point for IWMI is clearly from the agricultural production end" (p. 51). 7) ICRAF, 3rd EPMR, 2006 a) Achievements in scaling-up due to good collaborations. "These methods (called "prongs") include scientist to partner trainer (staff of NGOs and CBO); scientist to government extension service training, scientist to farmer trainer; and farmer to farmer training" (p. 39). 8) WARD A, Social Science CCER, 2006 a) "WARDA - in comparison with other CG Centers - is in a unique position, as it has very strong links to the NARES of the original West African member countries. In fact, when asked about their links with WARDA, all of the representatives from these countries indicated having very close links with WARDA...Given this background, it would seem surprising to find that a survey on CGIAR Center collaboration reports WARDA to have the minimum number of only 35 collaborators while for most other centers the numbers are around 100 or up to nearly 600 for IRRI. But for WARDA its member countries and their research institutes are not just collaborators but partners and therefore do not appear in the count" (p. 49). b) "WARDA's collaboration with six other CG centers (as reported in the survey) is quite similar and in line with collaborative links of other centers of this size. However, the survey data do not cover the results of the recent development of an MTP [medium-term plan] for West and Central Africa [WCA], which WARDA has prepared. The preparation of this MTP/WCA has resulted in a framework for systematic, national and balanced cooperation in the domain of agricultural research and innovation. In the quest to find ways " for research that produces International Public Goods (IPG) efficiently and at least cost", the plan defines three types of collaborative undertaking: i) inter-center alignment of activities, ii) integrated programs for major new opportunities or challenges and iii) research platforms to realize economies of scale. The MTP for West and Central Africa suggests 13 alignment activities and 15 integrated projects for WCA. This places WARDA right into the middle of the process as "WARDA was appointed by AE (Alliance Executive) to lead the WCA/MTP..." (p. 51). A-54 c) "The Panel recommends that the social scientists at WARDA be combined into one Social Science Research Unit, even when individual scientists contribute to different projects under Program 1 and 2. The members of the social science research unit to be constituted at WARDA are expected to work across different projects together with other scientists in disciplinary or interdisciplinary projects, as the case may be. In order to facilitate such disciplinary or interdisciplinary collaboration within WARDA all research projects should be defined also by listing the discipline involved. The panel recommends that research projects should more clearly define which disciplinary and interdisciplinary collaboration between different scientists is expected" (p. 78). 9) WARDA, 5th EPMR, 2007 a) "...the integration between technology developers, particularly breeders, and social scientists, leaves much to be desired. Integration and feedback, particularly regarding adoption and yield impact findings, appear inadequate" (p. 63). 10) IFPRI, Institutional CCER, 2004 a) "IFPRI might embark on even more joint projects with the economics unit of other CGIAR centers that relate to the issues of those centers. The involvement of local partners may encourage receptivity and ownership" (p. 8). b) "IFPRI has had many collaborators and visitors. It would be interesting to assess how much capacity strengthening was really achieved" (p. 8). 11) IFPRI, 4th EPMR, 2006 a) "...an important purpose of IFPRI has been to serve as a source of social science expertise for other CGIAR Centers, as partners with social scientists at those centers and as a system wide resource for some integrative and evaluative tasks, notably the measurement and dissemination of information about the CG system's economic contributions. More recently, IFPRI has been a leader in System wide Programs and Initiatives...in these enterprises IFPRI has been asked to provide project management services as well as social science expertise"(p. 72). 12) IFPRI, International Service for National Agricultural Research (ISNAR) Division CCER, 2007 a) "The ISNAR Division maintains a large network of collaborating institutions, researchers and users of ISNAR products. The ASTI initiative is leading ISNAR Division's international networking effort with about sixty cooperating institutions, of which 28 in Sub-Saharan Africa, 16 in Asia and 8 each in Latin America and 13 in North Africa and West and Central Asia. The number of collaborating institutions and researchers varies by subtheme and can only be estimated approximately; they number in total around 65" (p. 25). b) "In summary, interdivisional cooperation [within IFPRI] with the ISNAR Division has been reported to have improved a lot during the last six months... On the other hand, ISNAR staff in Addis still see a challenge in overcoming a distinctly felt "we- they" attitude" (p. 25). Impact Assessment 1) CIAT 6th EPMR, 2007 a) "The impact analysis project no longer exists and no senior staff is assigned fulltime to impact analysis at headquarters" (p. 76). b) "Apart from these studies, the quality of impact assessment work has been unimpressive" (p. 76). 2) ICARDA, NRM & Social Science CCER, 2005 A-55 a) "An important area of new work for socio-economists would be to develop appropriate integrated Management approach. These need to be consistent across different spatial scales, and coherently assimilate different styles of evaluation, ranging from the highly quantitative to the qualitative. The panel recommends that such methods be developed and implemented" (p. 2). 3) ICARDA, NRM & Social Science CCER, 2005 a) Recommendation: "That appropriate integrated impact assessment methods be developed to complement the integrated NRM approach". 4) WorldFish, 3rd EPMR, 2006 a) "Work on Impact Assessment will go from the ex post analysis of the transfer of the GIFT technology and on the research on Integrated Agriculture-Aquaculture to further developments, including: analysis of impacts from natural resources management and knowledge-intensive technologies, the analysis and assessment of impacts on biophysical and socio-economic aspects and the need to further institutionalize impact assessment approaches and methods" (p. 65). 5) IRRI, Social Science CCER, 2007 a) "There is a need for more ex post impact assessments of IRRTs work, especially to show the impacts on poverty and environmental outcomes. Ex post impact assessments should play a key audit function for IRRI and its donors. They can also be important for promoting IRRTs work within the larger international community. IRRI has already suffered much unwarranted but largely unanswered criticism of the impact of its Green Revolution technologies on poverty, nutrition, and the environment" (p. 22). 6) ICRISAT, Institutions, Markets, Policy and Impacts (GT-IMPI) and Knowledge Management and Sharing (KMS) CCER, 2007 a) "Impact evaluation, including both benefits and costs, is a central ongoing aspect of GT-IMPI activities... The other GT groups apparently generally willingly cooperate with the GT-IMPI in such evaluations, GT- IMPI recognizes that there are some problems that should be worked out regarding the distribution of responsibilities and (particularly time) costs between collaborating partners in impact evaluation, the transaction costs of undertaking such collaborations, and the extent to which information is provided by GT-IMPI for ex ante impact evaluations that increasingly are desirable for successful external funding proposals" (p. 27). b) "The Panel also recommends that VASAT be subject to careful impact evaluation in the near future" (p. 27). c) "A mind set be adopted in which evaluation of impacts is approached by asking what experiment would be desirable to ascertain the impacts and how, given the actual information constraints, this experiment can be approximated with actual or obtainable data and appropriate estimation techniques" (p. 48). 7) ICRISAT, 6th EPMR, 2009 a) "...impact assessments be more explicitly seen as being part of a feedback/priority setting loop..." (p. 40). 8) WARD A, 5th EPMR, 2007 a) "ITowever, WARDA's adoption and impact studies have been limited by their "snapshot" approach" (p. 3). b) "Panel had difficulty understanding the results of WARDA's adoption and impact studies, perhaps because the agroecologic and socio-economic contexts of such studies are not sufficiently clear and explicit" (p. 3). A-56 9) IFPRI, Institutional CCER, 2004 a) "Impact assessment is a fairly new initiative at IFPRI. Two approaches have been applied. First, papers were commissioned to assess the impact of specific research activities. Second, researchers were asked to provide some evidence of the impact of their own research and to incorporate impact considerations into the identification and design of their research projects ... Nearly all the papers limit the analysis to investigating the impact of specific IFPRI activities ... The papers do not aim to derive benefit-cost ratios" (pp. 61-62). 10) IFPRI, 4th EPMR, 2006 a) "Impact is another matter. IFPRI itself has given sustained and thoughtful attention to "impact assessment", which the Panel commends. They have concluded, rightly in the Panel's view, that there is no way of generating a quantitatively meaningful indicator of impact of IFPRI's (or other social science) research in practice. So one necessarily falls back on citations, downloads, sales and distribution of publications, attendance at conferences, briefings of key policy officials, and so forth. Unfortunately, when all is said and done, one cannot count these measures as impact. As is the case of research quality, it is the testimony of those who have dealt with IFPRI that constitutes the most credible information. The Panel's assessment is that IFPRI is having substantial influence, if not impact, and that the influence is beneficial" (p. 72). 11) IFPRI, International Service for National Agricultural Research (ISNAR) Division CCER, 2007 a) "A system has been put in place to ensure that the impact of research activities is subjected to evaluation, both ex-ante and ex-post. This is in line with the recommendations of Impact Assessment Report of 2004. ... As ISNAR Division projects are fairly recent, none has undergone ex-post evaluations so far" (p. 21). 12) ILRI, 2nd EPMR, 2008 a) "In cases, where ex post impact assessment is not possible, due to funding limitations, for instance, efforts should be made to conduct adoption studies for completed projects. The Panel has not seen any reports on adoption studies and suggests that such studies be part of the Theme's strategy and should be carried-out wherever feasible" (p. 32). b) "The Panel considers that the shorter term monitoring of outcomes from projects complement but do not substitute to ex post impact assessment. The latter is necessary for both accountability and to document what impacts actually occur. Despite time lag between the research and when its impacts can be evidenced among the ultimate beneficiaries, ex post impact is needed also for guiding new research directions. The results of ex post impact assessment can be used to guide decision making by various stakeholders, including donors, policy makers and research managers. As a critical component of ILRI's systems approach, the Panel recommends that ILRI management charge the research themes to conduct ex post impact studies on selected programs using methodologies developed by the new Innovation and Impact Unit, and using external inputs where needed" (p. 59). Attention to Policy Issues 1) CIMMYT 5th EPMR, 2005 a) "there has been lack of attention to policy analysis" (p. 39). b) "With respect to Recommendation 14, on the need for a gradual increase in emphasis on macroeconomic policy analysis of selected countries, the Panel has noticed that A-57 c) "Should the national political winds shift direction, it would be extremely valuable to have readily available to the national governments the results and policy recommendations of those studies as soon as the demand arises" (p. 49). 2) ICARDA, 5th EPMR, 2006 a) "...it is probable that research on the policy and social issues would be of more strategic value than in developing new technologies. For this topic, ICARDA should shift from a role of conducting research, to one of networking between advanced institutions and NARS, being the broker and the source for information needed to answer NARS needs" (p. 92). 3) IPGRI/BIOVERSITY, 5th EPMR, 2003 a) "It is innovative in the sense that it focuses on research on and analysis of, the policy processes as opposed to the conventional Plant Genetic Resources (PGR) policy research approach which is dedicated to research for policy processes" (p. 30). 4) IFPRI, International Service for National Agricultural Research (ISNAR) Division CCER, 2007 a) "Its [the Agriculture Science and Technology Indicator (ASTI) program of the ISNAR subtheme Agriculture Science and Technology Policy] value and importance is highly recognized by IFPRI insiders, but apparently not yet sufficiently by outsiders, notably the CG system and donor community, as evidenced by a decline in funding. A wider use of ASTI data in research and policy advice should help to underline the priority of these activities and to convince donors of its importance" (p. 9). 5) IWMI, IWMI-Tata Water Policy Research Program (ITP) CCER, 2005 b) "... the ITP management is confident that the team will produce the outputs, but how these outputs can be presented and 'pitched' into the policy debate in India needs to be thought through. Experience to date is that while there is reasonable uptake across the science, development NGOs and national press, the engagement of government has been disappointing" (p. 13). Research Quality and Innovativeness 1) WorldFish, 3rd EPMR, 2006 a) "Discipline should play a leading role not only in its own research area, but also for the documentation of impacts from Center-wide research activities and in providing guidance to, and support of, the Center's planning, monitoring and evaluation in the short and long term" (p. 67). 2) IITA, 6th EPMR, 2007 a) "Many are confined to the immediate areas of a development project's activity and often treat adoption as a yes/no variable rather than examining the detail of farmer adaptation" (p. 80). 3) IRRI, Social Science CCER, 2007 a) "At the same time, the panel sees opportunities for more cross-cutting and synthesis work to derive larger international public goods from ongoing support to the biological science 19 programs. A good example is the possibility of additional synthesis work on social science methods for generating and disseminating knowledge-intensive technologies at larger scales based on successful project experiences" (p. 18). 4) IRRI, 7th EPMR, 2009 a) "Social Science Division (SSD) staff involvement in IRRI projects for technology testing and development includes the conduct of baseline studies and follow-up analyses (often involving training of local collaborators). The site-specific work was also an opportunity A-58 5) ICRISAT, Institutions, Markets, Policy and Impacts (GT-IMPI) and Knowledge Management and Sharing (KMS) CCER, 2007 a) “Recommendation: Scientists engaged in human subject research be required to obtain human subject certification by completing successfully an appropriate online training/testing for social science research" (p. 51). b) “Providing it maintains a strong capacity within KMS to support its ICT infrastructure and functions, ICRISAT is extremely well-placed to play a lead role within the CGIAR, in innovation for technology-mediated agricultural decision support, education and agricultural extension" (p. 30). c) “More attention be devoted to issues and strategies for dealing with possible endogeneity bias in the estimation of the impact of what would seem to be one or more choice variables (e.g., social networks) on others (e.g., seed adoption)" (p. 48). d) “Recommendation: The quality of GT-IMPI's research on scaling-up processes be strengthened and a good framework for distinguishing between scaling-up per se and research thereon be developed" (p. 59). e) "The Panel also recommends ... that more attention be given to the possibility of incorporating true experimental design in evaluations both for the purpose of evaluating the experiments themselves (e.g., the mode of providing information to various potential clients) and of using the experiment to identify the effects of farmers' decisions (e.g., regarding inputs) on outcomes of interest" (p. 27). 6) ICRISAT, 6th EPMR, 2009 a) “The Panel recommends that GT-IMPI (Global Theme - Institutions, Markets, Policy and Impacts) work on the development of hypotheses that determine the IPG (International Public Good) potential of ICRISAT's downstream work on technology development, testing and adaptation" (p. 42). 7) IWMI, IWMI-Tata Water Policy Research Program CCER, 2005 a) "The quality across such an array of some 250 individual research efforts is understandably variable" (p. 13). b) "ITP needs to devise an effective communication strategy which focuses on decision makers, within a larger framework of providing critical knowledge inputs to the variety of actors associated with workshops and (ii) regular reporting of ITP research findings in the popular print and electronic media. To do this, ITP must improve the communication and media expertise within the Program by developing partnerships with communication organizations, media-centric organizations and individuals with expertise in policy making" (p. 3). c) “With regard to the technical aspects of the programme, it is sensible to ask what are the best set of tools to approach such natural resource/socio-economic mixes. Much of the point data related to groundwater and groundwater use is imperfect data, fraught with errors and approximations. Therefore, some systematic thinking about the role of sampling multiple locations and remote sensing data needs to be given. There is evidence of a lot of local discrete analysis, but macro-analysis is sometimes lacking, although such synoptic perspectives are necessary in order to derive policy analysis and policy outputs" (p. 34). 8) ILRI, 2nd EPMR, 2008 a) "Essentially, the “innovation systems approach" studies all interactions implicated in plans to introduce change. In many ways it counters a narrowness that has crept into A-59 9) IFPRI, International Service for National Agricultural Research (ISNAR) Division CCER, 2007 a) "IFPRI, known as a center of excellence in research" (p. 11). b) "Effective mechanisms for accessing relevant indigenous knowledge systems must be built into each research project... Across the whole spectrum of IFPRI communication channels (especially newsletters, issue briefs, and the web site) there is room for more systematic solicitation of input from stakeholders and target audiences" (p. 73). Gender issues 1) WARD A, 5th EPMR, 2006 a) EPMR mentions a need to focus its research around gender issues more in general. 2) ICRISTAT, 6th EPMR, 2009 a) "...including differentiating benefits by gender, age and wealth groups now comprise research activities that contribute substantially to improving technology diffusion"... "Overall the panel is impressed with the attention paid by GT-IMPI (Global Theme - Institutions, Markets, Policy and Impacts) to gender at the project level in designing and evaluating their research activities" (p. 42-43). 3) CIFOR, 2nd EPMR, 2006 a) "The Panel recommends that CIFOR's Programmes and Projects in their diagnosis, design and implementation increase attention to gender, especially in regard to poverty alleviation. Positive initial steps would be to disaggregate relevant project data by gender, and to define impact pathways that focus specifically on women and their households" (p. 41). 4) IFPRI, Institutional CCER, 2004 a) "This project [Gender and Intrahousehold Aspects of Food Policy] falls within the spirit of IFPRI's new strategy because it is able to connect macro-oriented policies with micro­ level effects. The importance of the issues and the success of the project contributed to the decision to adopt these topics insti tute-wide, not just as special projects" (p. 50). 5) IWMI, IWMI-Tata Water Policy Research Program CCER, 2005 a) "... the Review notes that several geographic regions (such as the poverty stricken and flood-prone Ganga-Brahmaputa-Meghna basin, the mountains), issues (gender, political economy, domestic water, water quality) and phenomena (land-water-forests interactions) are excluded or poorly represented" (p. 2). b) "The issue of drinking water—more generally, domestic water—has received scant attention, and often obliquely in terms of the emerging town-hinterland divide. The Review has been given to understand that this reflects IWMI policy. ITP, however, is a "joint venture" and drinking water is a major policy issue in India, attracting huge public investments. Women and girl children bear the brunt of drinking water shortages and it affects the wellbeing of all citizens" (p. 19). A-60 Annex 7 Best publications analysis Centers were asked to provide 15 best social science publications each for the period 2005-2007 in specific areas, allocated equally among publications focused on disciplinary social science, on multidisciplinary research, and those reflecting good collaborations. Some Centers used the same publication for multiple categories. Others submitted more than the 15 requested. The panel accepted whatever self-nominated best publications the Centers submitted as the intent was to examine the nature and quality of the published social science research of which the Centers are most proud. 216 total publications were submitted. Type of publication submitted by Center The first table shows that 60% of the Centers' self-selected best publications appeared in Thomson ISI journals. Meanwhile, 19% were self-published by Centers, with ICRISAT and IFPRI especially reliant on in-house publication series to disseminate what they consider their Center's best work. This seems an undesirably high rate, reflecting a broader impression that Centers are commonly not publishing their best work in high visibility outlets. Journal Center Not Thomson Book Center Total Thomson ISI ISI chapter Book publication Africa Rice 11 9 2 Bioversity 14 8 1 4 1 CIAT 15 15 CIFOR 14 14 CIMMYT 21 12 4 5 CIP 13 7 2 2 2 ICARDA 5 3 2 ICRISAT 17 6 2 2 1 6 IFPRI 17 6 1 10 ILRI 28 15 1 4 2 6 IRRI 12 5 1 3 3 IWMI 15 8 2 1 4 World Agroforestry 15 11 1 2 1 World Fish 16 11 1 4 3 Total 216 130 9 25 11 41 Citations analysis of best publications As reported in the Phase 1 background study, among the best publications, the average annual citation per Thomson journal article per year was only 0.57, with a maximum of just 2.33. Centers' judgments as to what constitutes their best publications do not seem heavily influenced by impact on the broader scientific community, as manifest in peers' citation of the work. Overall, these best publications have been cited at only the average rate of other CGIAR social science articles and at only the average citations rate of other articles in the same journals, as reflected in those journals' average impact factor. It is also striking that the average impact factor A-61 of the journals in which Centers report publishing their best social science research is significantly lower than that of the full set of journals in which they published (0.55 versus 0.88). Given that these publications represent Centers' representation of their best work, one might reasonably expect to see stronger citations rates for these publications, at least citations rates in excess of the average publication appearing in the same set of journals in which these papers appear, and to see them appear in stronger journals. Authorship of best publications % Best papers with coauthor(s) in Center Center lead Other author (%) ARI Centers NARS Africa Rice 72.7 36.4 36.4 36.4 Bioversity International 85.7 28.6 14.3 35.7 CIAT 80.0 33.3 33.3 46.7 CIFOR 42.9 71.4 14.3 50.0 CIMMYT 71.4 28.6 14.3 57.1 CIP 46.2 58.3 0.0 33.3 ICARDA 80.0 60.0 0.0 20.0 ICRISAT 82.4 29.4 11.8 41.2 IFPRI 76.5 25.0 6.3 31.3 ILRI 57.1 75.0 25.0 25.0 IRRI 66.7 50.0 8.3 50.0 IWMI 60.0 53.3 6.7 20.0 World Agroforestry 53.3 46.2 46.2 38.5 World Fish 57.9 14.3 21.4 14.3 % of all best publications 65.7% 42.7% 17.6% 36.7% Roughly two-thirds of Centers' best publications had a lead author from that Center, a respectable rate.1 The overwhelming majority of these publications were co-authored (keep in mind that one-third were supposed to reflect good collaborations). ARI collaborators were the most frequently represented on these best publications and the panel observed that the very best of the 216 publications were almost entirely co-authored with ARI social scientists. NARS scientists are well represented as co-authors in the set of Centers' best publications. Strikingly, co-authorship among Centers is relatively uncommon in this set of best publications, with only one of every six, on average, co-authored with a scientist from another Center, less than half the rate of coauthorship with NARS. Review Panel Assessments of Publication Quality The review panel read virtually all of the 216 submitted best publications and offered ordinal assessments of the quality of each publication along multiple dimensions. All publications were judged by similar criteria, irrespective of whether they had been identified as a submission meant to represent best disciplinary research, multidisciplinary research or collaborative research. We summarize the key assessment criteria graphically. i This figure could be downwardly biased by (i) lead authors who were with the Center at the time of the research, but who departed before publication, and (ii) alphabetical listing of coauthors who share seniority. We therefore view this rate as a minimum. A-62 The first criterion concerned the use of current best practice methods and theory in the publication. As shown, slightly more than half the publications scored in the top two categories (out of five) in the use of best practices. Just under one-quarter fell in the bottom two categories. Use of best practice methods and theory 35% 30% 25% 20% 15% 10% ■ 5% 0% n 1 2 3 4 5 1= best current practice, 5 = out of date methods ortheory, n=174 The second criterion related to the weight of the findings, in terms of the originality and importance of the evidence generated in the publication. Just under half of the publications fell into the top two categories, one-quarter in the lowest two categories. Weight of findings 40% 35% 30% 25% 20% r 15% 10% 5% 0% 1 2 3 4 5 l=heavyweight/excellent, 5= lightweight/unimpressive, n=190 The third criterion related to the international public good nature of the research reported in the publication. 55% fell into the top two categories; 15% in the bottom two. A-63 International Public Good 35% 30% 25% 20% 15% 10% 5% 0% 1 2 3 4 5 Impure IPG, 5=pure local good, n=177 The panel consistently rated the papers meant to reflect excellent multidisciplinary research and strong collaborations as superior to those meant to reflect cutting-edge disciplinary research, in each of these three areas: use of best practice, weight of findings and international public goods generation. This is consistent with the observation that Centers' comparative advantage is associated less with the production of fundamental new discoveries within the disciplines than in research that integrates across disciplines and organizations. This is consistent as well with the observation that research on agricultural productivity and technologies or on institutions and markets represented roughly one-third of the best publications each, with another 20% on natural resources management and less than 15% on general theory or methods. The very best of these best publications — as identified by the panelists in flagging the most impressive 10-20% of the publications they read - were disproportionately about markets and institutions, about 40%, with nearly 30% on agricultural productivity and technologies, and roughly 20% about natural resources management. A-64