about the Evaluation GLOBE

Concept paper for the fourth volume, 2022:

Concept paper for the third volume, 2020/21:

1. Introduction

The increasing use of evaluation can be observed on a global scale. Many countries experienced a shift towards evidence-based policy making and customer-oriented new public management – focusing on outputs, outcomes and impacts instead of input – and therefore require reliable scientific data. Evaluation can make the necessary data and facts available and provide assessments on the basis of previously agreed criteria. Evaluation can be performed due to three purposes: On the program level evaluation can provide steering information for the management and thereby contribute to a more efficient and effective decision making, which is focused on sustainable impacts, and program implementation. In this context evaluation provides learning potentials and recommendations for action. Another purpose of evaluation is its contribution to good governance due to the provision of information about the impact and sustainability of policies. Thereby political steering mechanisms are improved and legitimacy and credibility of politics are improved as well. The last purpose is the possible enlightenment of society. When evaluation´s findings are fed into the public discourse they can contribute to the enlightenment of society and thereby allow a dialogue about sociopolitical developments (cf. Stockmann 2013:74).

Besides the increasing practical use of evaluation, the scientific research about evaluation increased as well. Furobo, Rist and Sandahl (2002) were the first to provide a systematic overview about different country specific evaluation cultures. In 2015 Jacob, Speer and Furubo published an article with updated findings and a first comparison with the results from 2002. Other scientific work dealt with topics such as National Evaluation Policies (NEPs), institutionalization of evaluation or professionalization of evaluation (cf. Rosenstein 2013, 2015; Meyer 2015,2016; Widmer, Beywl and Fabian 2009). Most of the research about evaluation has a clear focus on the so-called pioneer-countries in Europe or the United States of America and Canada. The recently edited book “The Future of Evaluation”, however, provides information about evaluation in countries of all continents. It becomes clear that many of the emerging or so-called developing, countries are catching up in institutionalizing and using evaluation (cf. Stockmann/ Meyer 2016). This led to the intention of the present research project, which aims to generate an “Evaluation Globe”, covering countries of all continents and discovering systematically the “institutionalization of evaluation”. The specific aim of the Asia-Pacific volume is to give voice to countries that have often been neglected in previous research, resulting in a scientific bias in many international comparative studies.

Before presenting in detail the methodological approach implemented to achieve this goal, the following chapter briefly outlines the state of research conducted so far.

2. State of knowledge

In this chapter a short introduction about relevant scientific work in the field of evaluation research will be provided, focusing on the literature of country-comparisons and institutionalization of evaluation.

As already mentioned above, Furubo, Rist and Sandahl provided the first systematic overview about evaluation systems in 21 different countries and 3 international organizations. The purpose of the research was to describe the countries’ individual systems as well as global trends and developments, and to explain these developments regarding possible driving forces and consequences. The methodological approach was to conduct country case studies oriented on 9 indicators. In 2015 an update was presented by Jacob, Speer and Furubo[1], offering a cautious longitudinal comparison. The finding in this longitudinal comparison was a general “lift” effect of institutionalization in all countries, or like Furubo et al. state it “evaluation culture has matured over the last decade” (Jacob, Speer and Furubo 2011: 23). While the former forerunner countries from 2001 (USA, Canada, Australia and Sweden) remained quite static at the high level they achieved previously, others fulfilled huge efforts to catch up and thereby managed to establish themselves at the rankings top (such as Spain, New Zealand and Japan). This ranking can be considered as very useful to get a first impression about the actual state of evaluation in a country, but two points lead to a limited comparability, namely that evaluation, evaluation culture and the institutionalization of evaluation lack a clear definition, and the fact that in the update from 2015 different experts have been questioned than in 2002, which might have induced a shift in the results. Besides that, it seems that no theoretical deduction of the indicators took place and an indicator measuring training and education is missing. The focus lays, like already mentioned above, clearly on the European and North American continent, while all Latin American countries and nearly all African and Asian countries were left out. Still the findings presented by Furobo et al. are important contributions towards how evaluation is institutionalized. It has been possible to work out three different internal driving forces for the development of evaluations. Those are the political constellation, the fiscal situation and constitutional features. External driving forces might be significant external pressures by donor countries or organizations (cf. Stockmann and Meyer 2014: 27 f.; Furubo, Rist and Sandahl 2002: 1-22; 441-455; Jacob, Speer and Furubo 2015: 7-28).

Another publication dealing with the institutionalization of evaluation was presented by Widmer, Beywl and Fabian (2009). This aims to capture the development of evaluation in a systematic way in 10 different topic areas, thereby comparing Switzerland, Germany and Austria. It covers the institutionalization of evaluation, focusing on the constitutional and other legislative foundation of evaluation as well as the anchoring of evaluation in parliament, government and administration, different topic areas and the use of evaluations as well as sectoral and national trends. The obvious difference in comparison to Furubo et al. is the focus on specific sectors. Each chapter is written by a different expert from the specific sector in the specific country. Each thematic rubric ends with a comparison of all three countries. Thereby it is possible to identify differences and similarities of the countries and sectors. To enable the already mentioned comparisons the authors set a strict framework for the chapters and defined evaluation as systematic, transparent and data based. The authors had to follow this definition. It can be criticized that the methodological approach of the single chapters is not systematic – some relied only on their own expertise in this area while others carried out written surveys or systematic analysis of databases (cf. Widmer, Beywl and Fabian 2009: 13-23; 509-527). Still the book provides a detailed description of the differences and similarities of all three countries regarding the institutionalization of evaluation.

Two studies about National Evaluation Policies (NEPs) have been conducted by Rosenstein in 2013 and 2015. In 2013 Rosenstein carried out an internet research of 115 countries and found that 20 already possess a written, legislated evaluation policy, while others are either developing a policy (23) or conducting evaluation routinely without a policy (34). 38 did not provide any information indicating that they are developing one at the moment (cf. Rosenstein 2013: 4). These findings have to be interpreted very cautiously due to the fact that the study of Rosenstein was performed solely in the internet, which can lead to a false categorization, if one is not familiar with the country’s political system, language or other issues. The cross-country comparison about different legislation of evaluation can be seen as a starting point for further research, but systematic assessment of the quality of governance as well as the development of evaluation will be necessary (cf. Stockmann/ Meyer 2016: 19).

In the “Future of Evaluation”, by Stockmann and Meyer, more than 30 different authors from 20 countries placed in all continents provide an overview about the professionalization of evaluation in their countries. The focus lays, like the title expresses, on the future of evaluation, meaning how evaluation will develop in different countries around the world. Topics covered in this book are challenges, which might be able to weaken, stop or reverse the increase of evaluation or the question, if there will be a globalization process leading towards “one” evaluation culture, or if there will be differentiation of the evaluation culture according to the political culture of various countries and their specific use of evaluation. The strength of the publication clearly lies in providing a new perspective on evaluation, also including countries that are normally not on the radar of evaluation research. But although the editors provided indicators to ensure comparability, these were not systematically explored in all contributions. Therefore, also this publication does not provide a systematic analysis of the institutionalization of evaluation in different countries (cf. Stockmann/ Meyer 2016).

Meyer also provided a journal article and an internal working paper about the “Professionalization of Evaluation”. In this he presents a theoretical deduction of the indicators he uses for measuring professionalization, based on theories of the sociology of professions. As far as it is known to the authors of this research proposal, this is the first approach towards a theoretical foundation of indicators. According to Meyer an increase of scientific journals and the foundation of professional organizations can be stated over recent years, but regarding the academic education a striking fluctuation can be observed. In conclusion of his findings, it can be said that a full professionalization so far has not taken place (cf. Meyer 2015, 2016).

During the work on the “Future of Evaluation” and the “Professionalization of Evaluation” the idea was born to expand working on the institutionalization of evaluation from a world-wide perspective, namely to carry out a systematic comparison about the institutionalization of evaluation on a global scale. This is the goal of the presented research project.

In the next chapter the methodological approach will be shortly introduced.

3. Methodological approach and research dimensions

The goal of the research project is a worldwide description of the institutionalization of evaluation in different countries, followed by a comparative analysis of the history, current and future situation. The whole compendium will consist of four volumes, starting with America and Europe, proceeding with Asia-Pacific and Africa.

Each case study is expected to provide a detailed description of the situation, supported by as many quantitative and qualitative data as possible. Literature and document analysis are expected as database as well as the conduction of about 5 interviews with different evaluation experts. For guaranteeing comparability an obligatory analytical framework is provided by the editors, covering three dimensions elaborated from systems theory, namely the social system, the political system and the system of professionalization (see Table 1). Please note, contributions should be 15 to 20 pages maximum.

As it can be seen above, three systems are considered relevant. Those are the integration in the political system, the dissemination and acceptance of evaluations in society and the professionalization of evaluations. While the first two systems are considered to be demanding evaluations the latter one represents the supply side. The authors of the case studies were asked to contribute to this framework in the first two volumes before the project began, as the editors believe that their expertise is and has been very valuable in developing a high-quality framework. For this occasion, a round-table took place at the European Evaluation Society (EES) conference in September 2017, where the analytical framework was discussed and finally agreed on. Following this conference, the European book will be the first to start. The analytical framework is developed to be a lean and feasible instrument, on the one hand to keep the efforts for the experts within an acceptable frame, on the other hand to guarantee that the case studies are focused on the most relevant aspects and still keep an acceptable length. Following this conference, the Europe volume was the first to be tackled. The analysis framework has been developed to be a lean and workable tool, on the one hand to keep the effort for the experts within acceptable limits, on the other hand to guarantee that the case studies are focused on the most relevant aspects and still keep an acceptable length. In this context, the editors also see it as a pragmatic investigation tool that is subject to a learning process across the different volumes and is continuously optimized.

Table 1 Dimensions of the Institutionalization of Evaluation

Thus, for the third volume, a further minimal adjustment of the analytical framework will be made, based on the experience of the previous research work in the GLOBE project. The goal is to improve the applicability of the framework and to provide even more adequate coverage of the relevant content without changing the essential dimensions or specifying the concepts differently, in other words without jeopardizing comparability.

The following paragraph gives a brief summary of relevant information about this research project.

4. Brief summary and milestones so far

As it has been demonstrated, the publications about the institutionalization, professionalization, future or culture of evaluation lack either the world-wide perspective, also including newly industrialized (‘emerging’) or developing countries, or a systematic, theoretical and/or methodological approach. The goal of the present research project is to close this gap and carry out a world-wide, comprehensive study of the institutionalization of evaluation. The methodological approach is based on country-case studies consisting of literature and documentary analysis as well as expert interviews. For approaching an understanding what institutionalization of evaluation means, three dimensions were further elaborated from modernization and institutional theory, namely the social system, the political system and the system of professionalization. Based on these dimensions an analytical framework has been developed, which needs to be covered during the case studies

The analytical framework was presented and opened for discussion at the 2017 EES autumn conference and subsequently optimized. All participating authors were then asked to start their work. The Europe volume was published in spring 2019 (Stockmann, Meyer and Taube 2020, Link), and the Americas volume was finished in 2020 and will be published in spring 2021 (Stockmann, Meyer and Szentmarjay 2021; a Spanish-language translation from English is in preparation). In addition, a German translation of the Europe volume was completed in 2020 and will be published in spring 2021 (Stockmann and Meyer 2021). The research results available so far have already been presented at various international conferences.

5. The road ahead: The Asia-Pacific Volume

The launch of the Asia volume is scheduled for early 2021. By the end of 2020, systematic literature searches and expert inquiries had already been carried out to obtain a list of countries to be considered in particular. The aim was to identify countries in which evaluation plays at least a partially detectable role and. In the course of this process, it became increasingly apparent that the MENA region would have to be covered in another volume, as there were already many suitable and varying candidates within the Pacific regions. In order to promote further important insights into the institutionalization of evaluation, the Asia-Pacific volume, like the previous volumes, should also consider international organizations.

In early 2021, authors and author teams will be contacted to encourage them to take part in the project. In cases where no potential authors have yet been clearly identified, the evaluation professional society is first contacted during this time, or other scholars working in related disciplines (such as Public Policy Studies, Public Administration Research) are asked for advice. Authors are asked to respond to the request within two weeks, indicating their agreement to contribute and that they agree with the methodology and analysis grid of the ‘Evaluation GLOBE’. From March 2021, the joint starting signal will then be given for the creation of the contribution. After six months, by end of September 2021, the contributors should submit their contribution, which the editors will comment on. All manuscripts should be there by then at the latest, as they will be subject to a review process from then on. The editorial revision will be done by fall 2021. For final synchronization of findings, authors are invited to read the final conclusion presented by the editors and provide feedback by winter 2021 for the final publication of the volume, which is scheduled for the summer 2022.

5. Literature

Furubo, J.-E. / Rist, R.C. / Sandahl, R. [Ed.] (2002): International Atlas of Evaluation. New Brunswick, N.J.: Transaction Publishers.

Meyer, W. (2015): Professionalisierung von Evaluation: ein globaler Blick. In: Zeitschrift für Evaluation – ZfEv 14/2. S. 215-246.

Meyer, W. (2016): CPEI – Beschreibung des CEval Professionelle Evaluation Index. [not yet published]

Meyer, Wolfgang/Stockmann, Reinhard/Taube, Lena (2020): The Institutionalisation of Evaluation: Theoretical Background, Analytical Concept and Methods. In: Stockmann, Reinhard/Meyer, Wolfgang/Taube, Lena (Hg.) (2020): The Institutionalization of Evaluation in Europe. London: Palgrave/Macmillan, S. 3-34.

Rosenstein, B. (2013): Mapping the Status of National Evaluation Policies. Online available: https://gpffe.org/wp-content/uploads/2019/06/MappingNEPReportDecember2013.pdf. Last accessed: 07.01.2020.

Rosenstein, B. (2015): Mapping the Status of National Evaluation Policies. 2^nd edition. Online available: https://globalparliamentarianforum.files.wordpress.com/2016/02/the-status-of-evaluation-policies.pdf. Last accessed: 07.01.2020.

Speer, S. / Jacob, S. / Furubo, J.-E. (2015): The institutionalization of evaluation matters: Updating the International Atlas of Evaluation 10 Years Later, in: Evaluation – the International Journal of Theory, Research and Practice.

Stockmann, R. (2013): Science-Based Evaluation. In: Stockmann, R./ Meyer, W. (2013). Functions, Methods and Concepts in Evaluation Research. Basingstoke Hampshire: Palgrave Macmillan.

Stockmann, R. / Meyer, W. (2013): Functions, Methods and Concepts in Evaluation Research. Basingstoke Hampshire: Palgrave Macmillan.

Stockmann, R. / Meyer, W. [Ed.] (2016): The Future of Evaluation. Global Trends, New Challenges, Shared Perspectives. Basingstoke Hampshire: Palgrave Macmillan.

Stockmann, R. / Meyer, W. [Ed.] (2021): Die Institutionalisierung der Evaluation in Europa. Münster: Waxmann. Manuscript submitted for publication.

Stockmann, Reinhard/Meyer, Wolfgang/ Szentmarjay, Laszlo (Ed.) (2021): The Institutionalization of Evaluation in the Americas. London: Palgrave/Macmillan. Manuscript submitted for publication.

Stockmann, Reinhard/Meyer, Wolfgang/Taube, Lena (Ed.) (2020): The Institutionalization of Evaluation in Europe. London: Palgrave/Macmillan.

Widmer, T. / Beywl, W. / Fabian, C. (2009): Evaluation. Ein systematisches Handbuch. Wiesbaden: VS Verlag für Sozialwissenschaften.

---

[1] Using the same indicators to ensure comparability.

---