Evaluation culture and evaluation policy as guides to practice: reflections on the Brazilian experience

Cultura de evaluación y política de evaluación como guías para su práctica: reflexiones sobre la experiencia brasileña

Cultura de avaliação e política de avaliação como guias para a sua prática: reflexões a respeito da experiência brasileira

COMMUNICATIONS AND REPORTS

Evaluation culture and evaluation policy as guides to practice: reflections on the Brazilian experience¹

Cultura de avaliação e política de avaliação como guias para a sua prática: reflexões a respeito da experiência brasileira

Cultura de evaluación y política de evaluación como guías para su práctica: reflexiones sobre la experiencia brasileña

Thereza Penna Firme^I; Ana Carolina Letichevsky^II; Ângela Cristina Dannemann^III; Vathsala Stone^IV

^IPh.D em Educação e Psicologia da Criança e do Adolescente, Stanford University, EUA; Consultora da Fundação Cesgranrio. E-mail: [e-mail]

^IIDra. Em Engenharia Elétrica, PUC-Rio; Estatística da Fundação Cesgranrio. E-mail: [e-mail]

^III Mestre em Administração de Empresas,BMEC-RJ; Sócia Gerente da D&S Engenheiras Associadas. E-mail: [e-mail]

^IVCo-Principal Investigator, University at Buffalo, New York. E-mail: [e-mail]

ABSTRACT

There is no doubt that having evaluation policies makes the evaluator's job easier, by providing more transparency to the evaluation process and more security to those involved. However, this can only happen when policies are presented to potential stakeholders in clear language, as they are disseminated and utilized for guiding practice. This suggests that building an evaluation culture is necessary in order to effectively implement an evaluation policy, so it can be fully utilized. This paper briefly introduces and presents what the authors understand as the concept of " evaluation culture" and " evaluation policy" . Then, it discusses the importance of evaluation policies for practice of both evaluation and meta-evaluation, and points out the possible consequences of absence of policies. Recommendations are derived from these considerations in order to inspire procedures to build and implement evaluation policies. Finally, reflections are presented based on the Brazilian experience about " culture-policy-practice" inter-relations in evaluation.

Keywords: Evaluation. Evaluation policies. Evaluation culture. Practice. Brazilian experience.

RESUMO

Não há dúvida de que as políticas de avaliação tornam o trabalho do avaliador uma tarefa mais fácil por darem mais transparência ao processo avaliativo e mais segurança às pessoas envolvidas. No entanto, isso só se dá quando essas políticas são apresentadas a stakeholders (interessados) em potencial, em uma linguagem clara, já que elas são divulgadas e utilizadas para guiar a prática. Sugere-se, assim, que a construção de uma cultura de avaliação é necessária para que uma política de avaliação seja implementada efetivamente de modo a ser plenamente utilizada. Esse artigo apresenta, de maneira concisa, o que os autores entendem por " cultura de avaliação" e " política de avaliação" ; discute a importância de políticas de avaliação na prática da avaliação e da meta-avaliação; e destaca as possíveis conseqüências da ausência dessas políticas. A partir de tais considerações, surgem recomendações que têm como intuito inspirar o desenvolvimento e a implementação dessas políticas. Finalmente, são apresentadas reflexões baseadas na experiência brasileira com relação às inter-relações " cultura-política-prática" na avaliação.

Palavras-chave: Avaliação. Políticas de avaliação. Cultura de avaliação. Prática. Experiência brasileira.

RESUMEN

No hay duda de que las políticas de evaluación tornan el trabajo del evaluador una tarea más fácil por dar más transparencia al proceso de evaluación y más seguridad a las personas participantes. Todavía, eso solo se da cuando esas políticas son presentadas a stakeholders (interesados) en potencial, utilizando lenguaje clara, ya que ellas son divulgadas y utilizadas para guiar la práctica. Se sugiere, así, que la construcción de una cultura de evaluación es necesaria para que una política de evaluación sea implementada efectivamente para ser plenamente utilizada. Ese artículo presenta, de modo conciso, lo que los autores entienden por " cultura de evaluación" y " política de evaluación" ; discute la importancia de políticas de evaluación en la práctica de la evaluación y de la meta-evaluación; y destaca las posibles consecuencias de la ausencia de esas políticas. A partir de tales consideraciones, surgen recomendaciones que tienen como intuito inspirar el desarrollo y la implementación de esas políticas. Finalmente, son presentadas reflexiones basadas en la experiencia brasileña considerando las interrelaciones " cultura-política-práctica" en la evaluación.

Palavras clave: Evaluación. Políticas de evaluación. Cultura de evaluación. Práctica. Experiencia brasileña.

Introduction

Brazil has had a significant increase in the number of evaluations, especially in the educational and the social arena since the beginning of the 90's due to demand from the government and the Third Sector. But this growth occurred without previous planning. The combination of intense demand with lack of proper planning generated several challenges: a lack of professionals in the evaluation field; systems, organizations, institutions and people not prepared to be evaluated and to use the results of evaluations; lack of a network among the evaluators and those interested in evaluation, to stimulate collective development; scarce evaluation literature in Portuguese and lack of training courses in order to adequately prepare professional evaluators. Consequently evaluations of diverse natures were produced which can be grouped into three categories, as related to evaluation policy: (1) evaluation without a guiding policy; (2) evaluation with a pre-determined policy designed by a small group, not submitted for wide and substantial discussion, nor using language accessible to specific groups, often lacking transparency and wide dissemination; (3) evaluation with a pre-determined policy designed in an open and collective manner, and widely disseminated.

Discussion needs to be promoted about the strengths and weaknesses of these three categories as well as the consequences of not having an evaluation culture within society in general. Debate is also needed over the concepts of evaluation policy and evaluation culture in order to achieve convergence toward significant progress. Careful consideration of these aspects is of utmost importance for the democratic development of public policy in Brazil. The country is currently undergoing a phase when evaluators and evaluees argue about several issues: evaluation quality and the effect of evaluation upon communities; the importance of constructing an evaluation culture; and to what extent evaluations have contributed to the growth of stakeholder's social capital.

This turmoil of many doubts and few certainties, where policies attempt to guide practice, while practice incites policies, in fact offers a precious opportunity for one to observe and study the seeds – seeds that are mindlessly and unintentionally thrown in fertile soil, that nevertheless sprout beautifully, even without proper care. One must not ignore the outcomes that likely have been produced. Lessons must be learned throughout about the way to improve the processes of " sowing of the seeds" of evaluation policy and the " consumption" of evaluation practice. Brazil characterizes the Azure Jay syndrome – a bird from southern Brazil which, while feeding on the " araucaria" cones (pine seeds), seeks to hide it for future consumption, thus unintentionally cultivating woods of " araucaria" pine trees, a rare and valuable wood nowadays, for the benefit of mankind.

Evaluation culture and evaluation policy

Concept

Culture can be understood as common beliefs and practices of a group of people, in other words, the integrated pattern of human knowledge, belief, and behavior built around learning and transmitting knowledge to succeeding generations. " Evaluation culture" in Brazil is presently understood as an overall comprehension and acceptance in the realm of institutions, organizations and society in general, of evaluation importance, need, practice and utilization. In other words, when " evaluation culture" is present, there is a set of learned beliefs, values, styles and behaviors that makes evaluation welcome. Evaluation here does not threaten or condemn.

On the other hand, understanding policy as a deliberate plan of action to guide decisions and achieve rational outcome(s) " evaluation policy" is perceived as a set of guidelines that establishes rules and procedures to properly conduct planning, implementation and utilization of evaluation, in all circumstances or levels of possible implementation.. However, such guidelines must be directive enough to ensure consensus that is essential to achieve common major purposes in a disciplined way. On the other hand, they must be flexible enough to respect diversity and pluralities of interests among the various settings of evaluation conception and practice. Evaluation policies should be well documented and be transparent for both evaluator and evaluee. An " evaluation policy" must answer questions such as: " What makes up the evaluation? Who is responsible for what? What standards do we use? Thus an " evaluation policy" should contain: a clear explanation of the concept and role of evaluation; the adopted Evaluation Standards; clear definition of roles and responsibilities; emphasis on the need to operationalize the policy through evaluation guidelines; how evaluations are planned and implemented; mechanisms for the follow-up of evaluations; and the dissemination procedures for evaluations

One single country in different evaluation stages

As said before evaluations of diverse natures were produced and these can be grouped into three categories: (1) evaluation without a guiding policy; (2) evaluation with a pre-determined policy designed by a small group, not submitted for wide and substantial discussion, nor using language accessible to specific groups, often lacking transparency and wide dissemination; (3) evaluation with a pre-determined policy designed in an open and collective manner, and widely disseminated.

It is important to point out that our experiences showed that the further one develops an evaluation culture the closer one approaches the third type of evaluation practice mentioned above. Nevertheless, in situations where evaluation culture is not yet fully developed, building evaluation policies in a collective way has been successful in promoting integration and commitment among stakeholders. This result represents, indeed, a starting point to develop an evaluation culture. It is, indeed, one way of improving the process of " sowing the seeds" .

Especially in the educational and social areas where many evaluation systems are implemented by the Federal Government, Brazil has largely expanded its educational policy development efforts.

The significance of Evaluation Policy for evaluation and meta-evaluation

Evaluation policy is meaningful to evaluation and meta-evaluation in the sense of ensuring that everybody would have the same understanding about evaluation, its purpose, and the way it must be conducted. Moreover, policy is, in general, a formal written statement that is useful to guide and determine how an evaluation should be implemented so that it is possible to establish a clear limit, avoiding ethical problems.

When policies are not clearly stated evaluation borders become nebulous and easily crossed. Consequently, evaluators are asked to answer questions that are not within the scope of the evaluation and evaluees are judged about issues that do not belong to the content of the evaluation.

Although Brazil has advanced regarding evaluation and evaluation policy, it still lags behind in including meta-evaluation in such policy. Evaluation policy is in fact useful to orient evaluation users to conduct meta-evaluation, which is the best way one knows to guarantee the quality of the evaluation process. In this sense, it is worthwhile mentioning that in spite of having a long way to go in reaching the highest expectations about meta-evaluation, there have been some remarkable efforts in the Brazilian evaluation scenario in terms of training evaluation users themselves as evaluation critics and evaluation partners.

However, training professionals for the practice of evaluation as well -expanding knowledge and experience in the field by reaching all those involved and interested in developing and improving the the quality of the evaluand is still a challenge.

Possible consequences of lack of policy

Evaluation is usually a challenge to both evaluation and the evaluee. As mentioned before, presence of an evaluation culture facilitates facing this challenge; its absence frequently generates fear of evaluation. Although it is hard to frame an evaluation policy in a context of poor evaluation culture, there should necessarily be an effort from evaluators towards constructing a policy. The evaluation policy does not specify the steps that must be followed but does guide how to conduct evaluations. Conflicts in the evaluative process emerge either when principles and considerations are not clear to all involved or when they are violated. Briefly, these conflicts appear in the form of evaluations of deteriorated quality, failure in identifying and involving stakeholders, and inadequate training of the evaluator, or results not fully utilized. Conducting an evaluation without evaluation policy, in a field without evaluation culture, provokes chaos. Evaluation then starts working without directions, and evaluees give information without any commitment. When this happens it is important to ask why such evaluation is being conducted. If evaluators are not sure about why, where, and when they want to go, and evaluees do not know why, how, and what they must answer, it is important to ask: Is it really worthwhile to conduct this evaluation? Will someone use its results? Is it possible to trust the results? Is it fair dedicating human and financial efforts to this process?

All these questions remind us of a statement by Michael Scriven (2004):

In my view, one of the most important questions professional evaluators should regularly consider is the extent to which evaluation has made a contribution to the welfare of humankind and, more generally, to the welfare of the planet we inhabit – and, while we're at it, to the welfare of the other celestial bodies we are beginning to invade.

Brazilian experiences about culture-policy-practice inter-relations in evaluation: a success case

The Program

Brazil is a country seriously affected by social inequalities. Further, the social scenario is so complex that proposed solutions could be completely inappropriate as the apparent cues of poverty could be disguising deeper aspects that are actually causing the problem deserving attention. If these elements are not properly tackled, the real problem may continue and probably become worse. In this sense, a sound evaluation is needed to uncover what is actually happening instead of what we would like to happen (PATTON, 2005). Although ambitious, there is reason to believe that evaluation can construct a new reality as long as it is present in the different sectors of society and it counts on multidisciplinary teams of competent professionals. The case we describe here refers to the joint action of a community development program and its evaluation in a slum area of Rio de Janeiro, Brazil, with a population of approximately 2.000 people. This program named " Betting on the Future" is sponsored by Cesgranrio Foundation, a non-profit organization located near the target community, with a mission to promote and innovate evaluation theory, methodology and practice.

This specific program was developed in the context of the city of Rio de Janeiro, and addressed an area that includes four extremely poor communities: Parque André Rebouças, Vila Santa Alexandrina, Paula Ramos e Escadaria. The main purpose of this initiative was to contribute to the social and economical development of those communities, aimed at improving their quality of life. Two important questions were initially raised. The first one relates to our awareness of the information (or lack of it) concerning the target population. The second relates to the identification of priorities by the target population. A situation analysis was conducted to answer these questions. This generated an overall view of each community in terms of their needs, growth opportunities, realizations and limitations. Through this it was possible to detect basic information such as the number of people in each community, their home status, years of schooling, occupations and professions, as well as specific information about the needs of different segments in each community. The results of this analysis prompted the development of a program called " Betting on the Future" . It aims at promoting sustained gains concerning the quality of life of the communities. To guarantee the success of the program both in terms of attaining objectives and sustaining results, an evaluation process was designed. Additionally, to ensure the excellence of such evaluation process, a meta-evaluation procedure was also planned.

The unique aspects of this experience were: (a) giving due recognition to the community's knowledge for detecting its own needs; (b) selecting the best partnership to develop the activities and a competent team to conduct evaluation and meta-evaluation; (c) emphasis on community capacity building in evaluation, self-determination, and self-sustainability, and continuous feedback to partners, sponsors and the community.

The birth of an evaluation policy attached to a social policy

The Brazilian sociopolitical context as mentioned earlier requires that any form of evaluation intervention be integrated with social environment , if one intends to reach success.

Simultaneous to the identification of priorities, Cesgranrio Foundation designed a social policy and an evaluation policy as it commissioned an external evaluation for the Program. A fundamental assumption of the social policy was the belief that social programs should aim at social promotion instead of a mere non empowering assistance. In this sense such programs should incorporate planning, coordination, monitoring and evaluation to avoid duplication of efforts or lost resources, in order to guarantee a meaningful change in the social context. Furthermore, a social policy would concretely be applied through properly evaluated projects and actions. Thus, with the purpose of consolidating its intervention in the social area, Cesgranrio Foundation designed and implemented a social policy to guide its action in such area. The evaluation process had to be accurate and timely, but subtle and sensitive, approaching the right people at the right time making sure no harm was done either to the evaluation team or the community. Fortunately, the commitment and enthusiasm of those involved in the evaluation process overcame these limitations. In addition, the team tried to integrate different evaluation models focused around a responsive approach (STAKE, 2004). Our purpose was to have an external view on the overall Program in order to portray the quality as well as the possible limitations and the anxieties and opinions of the beneficiaries. Thus, the evaluation reproduced the peculiarities of the program, demonstrating a case of an evaluation historian (CRONBACH, 1980) where we describe, register and interpret the singular day-to-day process of each focused scenario. During its development, the evaluation considered the values, concerns, and perceptions of the program stakeholders and strived to identify the program's merit, i.e., its internal value, and also to discover its relevance in the success demonstrated by its results, repercussion and possible impacts.

Since it was an external evaluation, the evaluation team was composed of professionals with no direct connection with the Program. The advantage gained by this procedure was that we could offer a vision complementary to the self-perception of the program stakeholders. Both internal and external perspectives were necessary to achieve the desired success of the evaluation. The approach used in the external evaluation was essentially inclusive (MERTENS, 2003) because it emphasized the insertion and participation of all communities affected by the decisions based on the evaluation process.

This approach was present during all steps of the evaluative process. In this context, the role of the evaluator was a proactive one in the sense of intervening to change the elements that favor social injustice.

The starting point of the evaluation was the following evaluative question:

–To what extent is the project being developed efficiently? (Merit)

– What are the evidences of impact on beneficiaries and communities at large in regards to the improvement of their quality of life? (Impact or Worth)

It was made sure that the identified merit and impact indicators were consistent with the adopted evaluation approach. We collected data from three different sources: (a) beneficiaries, (b) partners, and (c) community. Several data collection procedures were used: (a) checklists, (b) interviews, (c) unobtrusive measures, and (d) focus groups.

A unique characteristic of the evaluation process has been that of offering continuous feedback to all the stakeholders of the program during the whole trajectory of its implementation. This procedure has undoubtedly prompted favorable changes among beneficiaries, partners and communities overall.

Results

An essential feature of the whole evaluative task was the continuous utilization of its process and results (PATTON, 1997). The present evaluation was constantly oriented towards such an effort. . The evaluation team understood that in the social service area more than in any other, there is a need for permanent follow-up to the development of actions and opportunities during the intervention of any program from its conception to its conclusion and repercussions.

In this sense, it is hoped that the results of this program will affect not only the target communities but the overall Brazilian society, for the benefit of many, including those who are responsible for conducting programs, for developing evaluative or research studies, or holding policy discussions in the social arena.

Changes observed as evidences of merit and impact are summarized below.

In relation to merit:

• Quality of Program Actions: the results show that: (1) there is a persistent interest for the best possible implementation of the program activities in all the areas of action, and (2) an increase of competency among the professionals that work in the program.

• Environmental conditions: there are indeed poor conditions in the environment over which the inhabitants have no power to intervene. These conditions hamper the possibilities of a healthy life. This is the case of the sewer, The program, however, has been acting insistently and little by little some changes have been observed such as " not pulling leaves out of trees" , " not mistreating animals" , " not throwing rocks" etc. Children and adolescents already show these behaviors with great enthusiasm and conviction. In school the improvement is remarkable. Thus, from school to community, they are growing.

• Diversified activities: the results show a constant adjustment of program activities in the sense of responding to the interests of the beneficiaries and rapidly resolving conflicts. There are also evidences of the integration of actions between projects. This has been ensuring the quality of results and at the same becoming a model for the beneficiaries and the community.

In relation to impact:

• Personal development: a remarkable result is the evidence of respect for moral values that have been emerging little by little. The program has showed impact on solidarity which, already existent, has been further reinforced. An improved relationship among the inhabitants has been noted. Mothers are more involved in the program and fathers, little by little. Self esteem has been enhanced.

• Vulnerability: results show that children and adolescents still work but only to collaborate at home. Violence against children and youngsters has been considerably reduced.

• Health: the strongest evidence of change was the reduction of the answer " never" when the beneficiaries were inquired about (a) men forty years or older submitting themselves to prostate examination and (b) women submitting themselves to preventive cancer examination. A greater care towards the hygienic conditions of the community physical space was also observed.

• Education: the results point to visible improvement in school performance. A great increase in interest for reading and writing was evident. School dropout was drastically reduced.

However, the most important asset of this whole experience was the support of the evaluation practice by an evaluation policy, which was itself attached to a social policy. From now on, all the evaluation interventions should be in harmony with the evaluation policy. This approach has been an irreplaceable coach to develop the needed evaluation culture in the target community.

Conclusions and recommendations

In synthesis, collectively constructing an evaluation policy prevents misunderstandings, conflicts, and avoids inappropriate use of results, in other words, it avoids evaluation pain. Moreover, evaluation culture emerges from the positive impact of the evaluation process, which must respect the values of the community or society involved. In this sense, our recommendations for those who practice, study or make decisions in relation to evaluation, and even more to the evaluees, are:

• Evaluation culture, policy and practice should work together;

• Evaluation policy has to be a public document;

• Evaluation policies have to be constructed in a participative way;

• Evaluation policies are a necessary guide to true evaluations;

• Evaluation should not be developed in a community just because evaluators want to do it but only and solely when the community understands its importance and takes it on its own.

Finally, to paraphrase what as Patton (1997) wisely said once, evaluation is too important to be left to evaluators alone.

Recebido em: 16/01/2009

Aceito para publicação em: 02/03/2009

• CRONBACH, L. et al. Towards reform of program evaluation San Francisco: Jossey-Bass, 1980.
• MERTENS, D. M. The inclusive view of evaluation: visions for the new millenium. In: DONALDSON, S. I.; SCRIVEN, M. (Ed.). Evaluating social programs and problems: visions for the new millenium. New Jersey: Lawrence Eribaum, 2003.
• PATTON, M. Q. The challenge of making evaluation useful. Ensaio: avaliação e políticas públicas em educação. Rio de Janeiro, v. 13, n. 46, p. 67-68, 2005.
• PATTON, M. Q. Utilization focused evaluation: the new century text. Thousand Oaks: Sage, 1997.
• SCRIVEN, M. Reflections. In: ALKIN, M. C. Evaluation roots-tracing theorists views and influences. Thousand Oaks, CA: Sage, 2004.
• STAKE, R. E. Standards-based & responsive evaluation Thousand Oaks: Sage, 2004.

1

Paper Presented at the 2008 Conference of the American Evaluation Association. Evaluation Policy and Evaluation Pratice, Denver/ Colorado, USA

Publication Dates

• Publication in this collection
  22 June 2009
• Date of issue
  Mar 2009

History

• Accepted
  02 Mar 2009
• Received
  16 Jan 2009