Setting up the Brazilian Evaluation Network: a challenging work with no boundaries

Estabelecendo a Rede Brasileira de Avaliação: um trabalho desafiador sem limites

INFORMES E PARTICIPAÇÕES

Setting up the Brazilian Evaluation Network: a challenging work with no boundaries^*

Estabelecendo a Rede Brasileira de Avaliação: um trabalho desafiador sem limites

Angela Cristina Dannemann^{**, I}; Thereza Penna Firme^{**, II}; Ana Carolina Letichevsky^{**, III}

^IMaster in Business Administration by IBMEC-RJ Specialization in Program Evaluation by CEATS/FEA/USP Leader of the Avina Foundation Board Member of FIRJAN's Social Responsibility Council

^IIPh.D. Education and Psychology – Stanford University – California Coordinator of the Evaluation Center – Cesgranrio Foundation Senior Evaluation Consultant Teacher (retired) of the Federal University of Rio de Janeiro

^IIIDoctoral (in process) in Methods of Support to Decision/ Department of Electrical Engineering PUC—RJ Master in Production Engineering PUC-RJ Statistician at Cesgranrio Foundation Professor of Statistics at PUC-RJ

ABSTRACT

Evaluation may be a relevant tool for the practice of democracy. In order for it to occur, quality and proper utilization of its results must be assured (PATTON, 1997). Therefore, the consolidation of democratic principles in a society and the promotion of effective social interactions (FETTERMAN; WANDERSMAN, 2005) will benefit enormously from a greater and better use of evaluation in all possible formats. The Brazilian Evaluation Network was created approximately three years ago in a country with a recent history of democratic practice. Within a theoretical framework networks do not have boundaries or limits for debate. However, cultural and historical aspects create limits for the discussions and boundaries among its participants. This paper intends to shed light into some of these limits and boundaries, reflecting over how they are being overcome.

RESUMO

Uma avaliação pode ser um importante instrumento para o exercício da democracia. Para que isso ocorra é preciso assegurar a qualidade da avaliação, bem como a utilização apropriada dos seus resultados (PATTON,1997). Desta forma, a construção de uma sociedade democrática, que busca a melhoria permanente dos seus processos sociais (FETTERMAN; WANDERSMAN, 2005), depende do aprimoramento contínuo da sua prática de avaliação, em suas variadas manifestações. No Brasil, há aproximadamente 3 anos, foi criada a Rede Brasileira de Avaliação. Uma rede que, em sua concepção teórica, não possui limites ou fronteiras de discussão. Contudo, existem aspectos culturais e históricos (envolvendo valores), que criam limites para as referidas discussões e estabelecem fronteiras entre as diferentes instâncias. Este trabalho apresenta alguns destes limites e fronteiras, refletindo sobre a forma como aqueles que encomendam e que conduzem avaliações vêm contornando os limites e ultrapassando fronteiras.

Introduction

After 3 years of steady growth of the Brazilian Evaluation Network (hereafter BEN) several members are arguing its challenges. Presently, its members place emphasis on transparency of planning and implementation, respect for stakeholders values and taking into account all characteristics of the situation; application of proper methods; and assurance that all stakeholders must understand the significance of evaluation, agree with its application and, most important, be able to understand, debate and use its results. As a consequence, this paper's proposal is to further promote this debate guided by the following question:

- How to evolve to a more transparent model, with a well defined identity and well organized internal evaluation processes?

Rationale

Brazil, with its 500 years of history has only recently reached a full democratic condition, the dream of generations – free presidential elections happened again only 16 years ago, after a long military dictatorship that abolished civil rights and reduced social initiatives to a bare minimum. This new situation has met with a population not entirely prepared for its demands, especially due to a historical lack of importance to universal education with high quality. Some of the reasons for this are summarized below:

- lack of vision of the influential elite and nonexistent pressure groups;

- vested interests of powerful political groups, who benefited from keeping entire populations uneducated in order to delay the natural process of demand for social and political rights resulting from educational enlightenment;

- education not a priority because it is a task demanding high investment, constant supervision, and aimed at long term results;

- historical difficulties with mid and long term activities, due to the short term of political mandates and excessive dependence on government decisions.

Thus, the need for a liberating evaluation practice had to face the lack of education and democratic experience, added to a punitive vision of evaluation, due to it being introduced within the context of an educational tradition. Radical groups of several denominations introduced and sustained formal education in Brazil for centuries and are to be praised for this, but must also bear the responsibility for the culture of blaming individuals and not processes, methods or concepts for something gone wrong. Also, it must be observed that the practice of evaluation as a systematic process of data collection for proper value judgment was established only 30 years ago. Not the least important is the fact that the network dynamic is a very recent organizational arrangement, especially in Brazil – the first networks were started in the 90's, aimed at environmental education.

Added to that, Brazil's huge social inequality will benefit enormously from adequate measurement of the results and outcomes of its public policies and programs - 10% of Brazil's population amass 50% of the GDP, which in terms of monthly salaries means that all those earning US$ 200 or above comprise these 10%. In spite of commendable efforts from several social organizations, very low or non-existent income does not promote sustainability and certainly promotes alternative ways of survival such as begging, child labor, and other worse situations.

Belief in evaluation as a means to promote important values for the improvement of democracy (FETTERMAN, 2001), such as transparency, reflection, critical thinking, (SCRIVEN, 2003) participation and debate, has been the reason behind the creation of the Brazilian Evaluation Network whose current mission statement reads: "Develop and enhance the culture and practice of public interest evaluation within the Brazilian society."

In fact the Brazilian Evaluation Network¹ has been bringing invaluable contributions to this cause, either by supporting this practice among its members (persons from several different sectors of society interested in evaluation), or by promoting and sponsoring activities related to it. In other words, the network is promoting the development of the practice of evaluation in Brazil, pushing boundaries ever further and overcoming moral, political and academic barriers. A debate over some critical aspects of this network is extremely relevant towards achieving better interactions within a traditionally horizontal, non-hierarchical system, such as this one, for members coming from an ingrained vertical, highly hierarchical culture. The considerations developed herein may be an important contribution towards the improvement of evaluation everywhere specially in developing countries such as Brazil (GUARNIERI, 2004).

Overview

The Brazilian Evaluation Network (hereafter BEN) was originally born in August, 2002, as part of an international cooperation for the promotion and development of evaluation stimulated by UN organisms such as UNICEF (SEGONE, 2002). It was initially constituted as a network of individuals and individuals within institutions with the following mission: "develop and strengthen public interest evaluation culture and practice by means of articulating individuals and organizations, knowledge management, professional development and the advocacy of evaluation as a tool for political transformation." Its founding members believed in evaluation as a social learning process that contributes with society's development, being also an important organizational development tool that promotes transparency and disseminates best practices, reliable information and results (ENCONTRO ANUAL DE PLANEJAMENTO ESTRATÉGICO, 2003).

The first strategic planning meeting for BEN was held in São Paulo on August 2003. It was attended by 13 persons, representing the 5 established hubs at the time (see Table III below). BEN's mission statement was consolidated (already mentioned at the beginning of this chapter), its goals were reviewed, a vision for its future and an action plan for the next 18 months were designed and consensus on organizational formats was achieved.

The vision of the group about the future of evaluation and its practice:

• a learning process

• promoter of empowerment and transparency

• accessible, ethical and participative

• part of the organizational culture

• a management tool

• a reliable source of information

• a tool for political transformation

The four main goals were clearly stated within the mission statement itself: (i) articulate individuals and organizations, directly or indirectly involved with evaluation debate and practice, (ii) create, disseminate and manage knowledge on evaluation, (iii) promote the training and development of professional evaluators, and (iv) advocate the inclusion of evaluation practice in management and planning strategies of public and private institutions (ENCONTRO ANUAL DE PLANEJAMENTO ESTRATÉGICO, 2003).

In addition, a fifth goal aimed at BEN's own organizational development was agreed upon: to strengthen its structure and expand its reach. Issues about BEN's organization were extensively debated breaching the following themes: constitution (by individuals and individuals within institutions), financing (only specifically per project and under the responsibility of an executive office rotating voluntarily between hubs) and government (by a group of facilitators chosen at the rate of 2 per regional hub).

Almost one year later, on July 2004, a second strategic planning meeting was held in Brasilia with 16 persons attending, representing 6 regional hubs. Only 3 of the persons attending the first meeting were present to provide continuity – the majority was not aware of the previous planning and was updated during the first day. Having overcome this difficulty, with the help of the "veterans", activities proceeded with a report per regional hub, reflections and debate about the advances and setbacks on the application of the previous strategic planning, review of the mission statement and the setting of thematic goals and the action plan for the 2004/5 year. Initially, reports of the 2003/2004 year were presented by representatives of the hubs. See below.

Improvement was identified overall, as well as per regional hub, with the sustainability of the existing hubs, the creation of a new hub at Minas Gerais, participation in the creation of the international evaluation network (IOCE) and in the Latin American and Caribbean network (RELAC), and the celebration of the agreement with the Third Sector Information Network (RITS) to host BEN's site and e-group. Notwithstanding these improvements, and the gains pointed out at Table I above, BEN's growth met with several difficulties:

• communication failure between hubs and within hubs

• long distance between hubs

• need of an office or decision level to centralize information

• need to aggregate and advertise local and national references for BEN

• need to invest in obtaining precise knowledge about evaluation practice and development overall (government, private sector and NG)

• lack of a clear definition of identity for BEN

Discussion was then initiated in order to reach consensus about a preliminary agenda for the two and a half days of work ahead. The list of agenda issues included the following:

• revision of previous mission to emphasize evaluation as a public asset and reassessment of the designation as goal of the previously assigned fifth goal of organizational development

• open a broad debate on the two essential themes of BEN's identity (name and abbreviation publicly applied) and governance (decision making and representation)

• discussion on transparency with the management of funds and fund raising

• debate about the propriety or necessity of formalizing BEN into a national association

• need to have a better knowledge of individual members' profiles and activities in order to stimulate interchange and a sense of belonging between members, and generate further knowledge through joint paper publication

• ratify the concept of diversity – there shall not be a consensus about evaluation methodology, concepts, themes or professional approach

• approach the themes of web site development and the creation of a video conferencing channel by cooperation between institutions

After ample discussion of several themes the outcomes were very positive. The mission statement and the four original goals were kept in their essence, with slight alterations in structure for simplification's sake. See flowchart below.

Discussions were organized into five broad themes: governance, network identity, sustainability, communication and development. Five Work Groups were formed to deal with specific activities, and attendees volunteered as members and suggested other possible group members amongst BEN, the latter to be confirmed after a first contact. The groups were organized into activities that are self-explanatory: rules and procedures group, relationship with RITS group, video conferencing group, web site development group and support for hub startup group. The activities to be executed by these committees were defined as the 2004/5 action plan and a date for the 2005/6 strategic planning meeting was proposed by the representative of the Pernambuco Hub and accepted: the 2^nd half of August, 2005 in Recife (ENCONTRO ANUAL DE PLANEJAMENTO ESTRATÉGICO, 2004).

Common ground was found for several items of the pre-defined agenda:

• Governance: no formalized national association; an Executive Committee created to coordinate the action plan by articulating Work Groups, solely comprised of attendees to this meeting; the executive office to be experimented as a shared model – seat in Rio de Janeiro and shared decisions with other hubs; national rules and procedures to be defined by a Work Group and hub's sub-networks will have independent decision making.

• Network Identity: survey to be developed for achieving in-depth knowledge of membership; preferential public for new membership are evaluation professionals, management professionals, academic community, students, educators and social leaders; spokesperson or group of people must be chosen for public representation of BEN; recommendation for the rules and procedures Work Group to build a proposal on the issue of admission and dismissal of members; BEN's name discussion must be reinitiated at the next meeting.

• Sustainability: funds will continue to be raised per project; recommendation for the rules and procedures Work Group to build a proposal on the issue of fund raising.

• Communication: urgent compliance with all items of the agreement with RITS; discuss and disseminate ways to make evaluation more accessible for diverse stakeholders as well as society; immediate development of the new web site; strengthen its political action in order to influence public policy and decision making as well as to further include evaluation in planning within the governmental, non-governmental and private sectors.

• Development: employ the best efforts to acquire information on training and development in evaluation, disseminate these events and facilitate attendance for BEN members; not to develop its own training program; take advantage of the experience of senior managers to promote the development of new members; stimulate the creation of study groups.

Discussion and Conclusion

By December, 2004 BEN had 264 subscribed members and regional hubs in 6 state capitals within the Northeastern, Central and Southeastern regions of Brazil. Most of these hubs had their own independent regional sub-networks, with diverse levels of activity. The profile was as follows:

Members within the Brazilian Internet comprise 78.4% (207 email addresses) of the Network, where the overwhelming majority may be considered isolated individuals or persons within businesses – they are 45.4% of the above, almost double the amount of those connected with government 26.1% and three times or more of those within NGO's and educational institutions, 15.5% and 13% respectively.

This profile may give us a strong indication of the use of personal emails for the Network in order to avoid the clogging of professional inboxes. Another possibility is that there are many members who work as consultants in monitoring and evaluation, may not be associated to any institution, and are typically connecting themselves through personal emails within email providers (by this angle we may also add the .com addresses and raise the previous 45.4 total to 51.3%). On the other hand it shows us how much still has to be done towards achieving BEN's mission – the need for enhancement and development is clearly demonstrated by the small number of members and the concentration of hubs in parts of the country with higher income.

Proper measurement of a public policy or program will assure that its results and outcomes are clearly identified and success or failure is distinguished. If you can't clearly identify success you may be rewarding failure; if you can't see success you cannot learn from it; and if the results of a program cannot be clearly demonstrated, you cannot win public support (PATTON, 1997). Thus was created and still is maintained in Brazil this condition of lack of trust in government and low level of trust with civil society's public initiatives (a recent survey associates high credibility with firemen and the post office, but not with the three levels of government or with NGO's).

Enhancement and development of the culture and practice of public interest evaluation within civil society, governmental and educational institutions are tantamount at this time in Brazil, and the sheer existence of a strong institution advocating evaluation practice and development is decisive for the adequate transition into a fully democratic condition. Proper knowledge and application of evaluation's 4 attributes of utility, feasibility, propriety and accuracy, around which all standards are organized, are the bare minimum for evolution in this direction in Brazil, and they must be disseminated in all levels of public interest, especially in educational institutions.

Let's try this exercise of applying the 4 attributes of evaluation and take BEN's planning for instance. A close look at its goals and the associated activities defined at the 2004/2005 meeting with this approach, trying to ask, for each one of them, how useful, proper, feasible and accurate they are, may be a good starting point.

The governance theme, for instance, which raised a high level of philosophical and political arguments and promoted the creation of the Rules and Procedures Work Group, is a good example. With a transversal aspect to it, this group had the potential to influence decisions that could be positive or negative for all 4 strategic goals. But this group, in order to be "useful", must in fact produce suggestions with high "accuracy" and post them within the predefined dates, for ample discussion, after reflecting on its own "feasibility" and the "propriety" of its stance. Some risks associated to our attributes may be predicted here – distance problems due to having members from 2 or more different states, lack of time availability because of members professional commitments, difficulties with task division and assignment of responsibility with group coordination in order to concentrate and organize the work at hand and to post the group consensus. These issues should have been raised at the very beginning, at the risk of non-completion of the task at hand. In fact they weren't raised and the task was not completed – some of us were part of this group and these issues were not properly addressed while the group was created. A consequence of this failure was the number of excessive complaints of inadequate posting, such as an anonymous survey trying to confirm active membership.

All 4 other themes, network identity, communication, sustainability and development, can undergo this same exercise, which we will not endeavor to do in this paper, and they will all give us similar outcomes. In fact some of the risks identified above, must be considered as an overall risk for BEN's action plan. Two of them are: lack of time availability because of members' professional commitments and difficulties with task division and assignment of responsibility with group coordination. Several Work Groups have not completed their task due to these reasons. Voluntary dedication to any activity takes time and compromises financial survival in a country with low income levels and this may be a good reason to start a discussion on the possibility of coordinating an annual fund raising for the sustainability of the rotating executive office – this would satisfy the need for centralized information and dissemination of knowledge and practice. Either that or the creation of a national association with hired professional staff for management of national activities, hosting of the e-group and centralization of knowledge and dissemination.

The crux of the matter here is to what should we do when developing evaluations in order to reduce uncertainty, improve effectiveness and make relevant decisions. The noblest intent here is to strengthen a movement that is willing and dedicated to promote social transformation, and to follow its course in order to intervene, whenever needed. The 4 attributes call for what is needed – evaluations that are sensible to situational and conditional aspects, methodologically flexible, dynamic in the political sense and substantially creative in order to promote the development and improvement of all stakeholders (PENNA FIRME, 2004).

A good start for BEN would be to find guidance on these attributes of evaluation for any and all of its future planning activities. Things could improve even further through the revision of current activities on the national level, maybe disseminating this knowledge towards the regional independent sub-networks. "Do unto you " is good advice for any and all relationships in society and application of the standards should better begin inside our own house. Our role of measuring and placing proper value unto social programs and public policies might be made easier if we consider advocating and applying something as simple as the 4 attributes, in every single act.

BEN's expansion finds solid ground on its internal and external debates and it must still meet the challenge of breaking international boundaries to link the Brazilian evaluation community to the greater worldwide evaluation community, well represented here in this conference.

• ENCONTRO ANUAL DE PLANEJAMENTO ESTRATÉGICO, 1., 2003, São Paulo. Relatório ... São Paulo: Rede Brasileira de Avaliação, 2003. Disponível em: <http://www.avaliabrasil.org.br/relatorio_planejamento_estrategico.pdf>. Acesso em: abr. 2005.
• ENCONTRO ANUAL DE PLANEJAMENTO ESTRATÉGICO, 2., 2004, Brasília. Relatório… Brasília, DF: Rede Brasileira de Avaliação, 2004. Disponível em: <http://www.avaliabrasil.org.br/RelatorioPlanejamento_final.doc>. Acesso em: abr. 2005.
• FETTERMAN, D. Foundations of empowerment evaluation. Thousand Oaks, CA: Sage Publications, 2001.
• FETTERMAN, D.; WANDERSMAN, A. (Ed.). Empowerment evaluation principles in practice. New York, NY: The Guilford Press, 2005.
• GUARNIERI, M. C. L. Governança de (em) redes sociais: uma contradição? Rio de Janeiro, 2004. Disponível em: < http://www.rits.org.br/redes_teste/rd_tmes_ago2004.cfm >. Acesso em: ago. 2004.
• MARTINHO, C. Redes: uma introdução às dinâmicas da conectividade e da auto-organização. Brasília, DF: WWF, 2003. Disponível em: <http://www.wwf.org.br/wwf/opencms/site/list_news.jsp?channelId=117&newsChannelId =117>. Acesso em: 11 jan. 2006.
• PATTON, M. Utilization-focused evaluation: the new century text. Thousand Oaks, CA: Sage Publications, 1997.
• PENNA FIRME, T. Avaliação: tendências e tendenciosidades. Ensaio: avaliação de políticas públicas em educação: revista da Fundação Cesgranrio, Rio de Janeiro, v.1, n. 2, p. 5-12, jan./mar. 1994.
• PENNA FIRME, T. et al. Avaliação em rede. Rio de Janeiro, set. 2003. Disponível em: <http://www.rits.org.br/redes_teste/rd_tmes_set2003.cfm>. Acesso em: ago. 2004.
• SCRIVEN, M. Evaluation in the new millenium: the transdisciplinary vision. In: DONALDSON, S. I.; SCRIVEN, M. Evaluating social programs and problems: visions for the new millenium. Manwah, New Jersey: Lawrence Earlbaum Associates Publishers, 2003.
• SEGONE, M. A comunidade internacional de monitoramento e avaliação: um movimento global para fortalecer a cultura e função de M&A no mundo. Brasília, DF, 2002. Disponível em: < http://avaliabrasil.org.br>. Acesso em: abr. 2005.

*

Paper presented at the 2005 Joint Conference of the Canadian Evaluation Society and American Evaluation Association "Crossing Borders, Crossing Boundaries", Toronto, Canada, October 24-30, 2005

**

The authors want to thank all members of the Brazilian Evaluation Network for their indirect contribution in the development of this paper.

1

Disponível em: <

http://www.avaliabrasil.org.br>.

Publication Dates

• Publication in this collection
  17 May 2006
• Date of issue
  Dec 2005