Improved community conditions in a brazilian slum: a significant consequence of its evaluation

Mejoría de la situación de una comunidad brasileña de baja renta: expresiva consecuenciadesuevaluación

Melhoria da situação de uma comunidade brasileira de baixa renda: expressiva conseqüência de sua avaliação

INFORMES E PARTICIPAÇÕES

Improved community conditions in a brazilian slum: a significant consequence of its evaluation^*

Melhoria da situação de uma comunidade brasileira de baixa renda: Expressiva conseqüência de sua avaliação

Mejoría de la situación de una comunidad brasileña de baja renta. Expresiva consecuenciadesuevaluación

Thereza Penna Firme^I; Ana Carolina Letichevsky^II

^IPh.D. em Educação e Psicologia da Criança e do Adolescente, Stanford University, EUA. Consultora em avaliação da Fundação Cesgranrio. [e-mail]

^IIDoutora em Engenharia Elétrica, PUC-Rio. Estatística da Fundação Cesgranrio. [e-mail]

ABSTRACT

This paper describes the activities of a slum-improvement community Program and its concomitant evaluation, both conducted by a non-profit evaluation organization in Brazil, as well as their imminent consequences. The unique aspects of this experience were: (a) recognizing the community's knowledge for detecting its own needs; (b) selecting the best partnership to develop the activities and a competent team to conduct evaluation and metaevaluation; (c) emphasis on community capacity building in evaluation, self-determination, and self-sustainability, and (d) continuous feedback to partners, sponsors and the community. Some visible consequences of this evaluation were: (a) an observed increase in the number of children in school, their performance and respect for moral values, (b) reduced violence, sex abuse, child labor and drug consumption and (c) a notable impact on the evaluating organization itself seen in their commitment to building an innovative methodology. Hopefully the betterment of the present situation is predictive of a dignified future opening up for the community.

Keywords: Evaluation. Evaluation of social programs. Meta-evaluation. Consequences of Evaluation

RESUMO

Este trabalho descreve as atividades de um programa de desenvolvimento social local em comunidades de baixa renda, bem como, sua concomitante avaliação e suas principais conseqüências, conduzidos por uma organização sem fins lucrativos, no Brasil. Os aspectos mais originais desta experiência foram: (a) reconhecimento da capacidade da comunidade em detectar suas próprias necessidades; (b) seleção dos melhores parceiros para realizar as ações e de uma equipe competente para conduzir a avaliação e a meta-avaliação; (c) ênfase na capacitação das comunidades em avaliação, auto-determinação, e auto-sustentabilidade; e ainda (d) devolução contínua de processo e resultados a parceiros, financiadores, e comunidades. Algumas das conseqüências visíveis desta avaliação foram: (a) a percepção do crescimento do número de crianças na escola, do seu desempenho e do respeito por valores morais; (b) redução de violência, abuso sexual, trabalho infantil e consumo de droga; (c) substancial impacto na própria instituição avaliadora no que se refere ao seu compromisso de construir e inovar metodologias. Espera-se que a melhoria da situação atual seja um preditor do desenvolvimento de um futuro digno para as comunidades.

Palavras-chave: Avaliação. Avaliação de programas sociais. Meta-avaliação. Conseqüências da avaliação.

RESUMEN

Este trabajo, conducido por una organización sin fines lucrativos, describe las actividades de un programa de desenvolvimiento social en comunidades de baja rienda, bien como, su concomitante evaluación y sus principales consecuencias. Los aspectos más originales de esta experiencia fueron: (a) reconocimiento de la capacidad de la comunidad en detectar sus propias necesidades; (b) selección de los mejores aparceros para realizar las acciones y de una equipe competente para conducir la evaluación y la meta-evaluación; (c) énfasis en la capacitación de las comunidades para la evaluación, autodeterminación y autosustentabilidad; y aún, (d) retorno de los resultados a los aparceros, patrocinadores y comunidad. Algunas de las consecuencias visibles de esta evaluación fueron: (a) percepción del aumento del número de estudiantes, de la mejoría de su desempeño y desenvolvimiento de sus valores morales; (b) reducción de violencia, abuso sexual, trabajo infantil y consumo de drogas; (c) impacto substancial en la propia institución evaluadora con relación a su compromiso en construir y innovar procedimientos metodológicos. Se espera que la mejoría de la situación sea preanunciador del desenvolvimiento de un futuro digno para las comunidades.

Palabras clave: Evaluación. Evaluación de programas sociales. Meta-evaluación. Consecuencias de la evaluación.

Introduction

Brazil is a country seriously affected by social inequalities. Further, the social scenario is so complex that proposed solutions could be completely inappropriate as the apparent cues of poverty could be disguising deeper aspects that are actually causing the problem deserving attention. If these elements are not properly tackled, the real problem may continue and probably become worse. In this sense, a sound evaluation is needed to uncover what is actually happening instead of what we would like to happen (PATTON, 2005). Although ambitious, there is reason to believe that evaluation can construct a new reality as long as it is present in the different sectors of society and it counts on multidisciplinary teams of competent professionals. This paper describes the joint action of a community development Program and its evaluation in a slum area of Rio de Janeiro, Brazil, with a population of approximately 2.000 people. This Program named "Betting on the Future" is conducted by Cesgranrio Foundation, a non-profit organization located near the target community, with a mission to promote and innovate evaluation theory, methodology and practice.

This specific Program was developed in the city of Rio de Janeiro, and addressed an area that includes four extremely poor communities: Parque André Rebouças, Vila Santa Alexandrina, Paula Ramos e Escadaria. The main purpose of this initiative was to contribute to the social and economical development of those communities, aimed at improving their quality of life. Two important questions were initially raised. The first one relates to our awareness of the lack of information concerning the target population. The second relates to the identification of priorities by the target population. A situation analysis was conducted to answer these questions. This generated an overall view of each community in terms of their needs, growth opportunities, realizations and limitations. In this sense it was possible to detect basic information such as the number of people in each community, their home status, years of schooling, occupations and professions, as well as specific information about the needs of different segments in each community. The results of this analysis prompted the development of a Program called "Betting on the Future". It aims at promoting sustained gains concerning the quality of life of the communities. To guarantee the success of the Program both in terms of attaining objectives and sustaining results, an evaluation process was designed. Additionally, to assure the excellence of such evaluation process, a metaevaluation procedure was also planned.

The unique aspects of this experience were: (a) giving due recognition to the community's knowledge for detecting its own needs; (b) selecting the best partnership to develop the activities and a competent team to conduct evaluation and metaevaluation; (c) emphasis on community capacity building in evaluation, self-determination, and self-sustainability, and continuous feedback to partners, sponsors and the community.

The Program

The Program was designed taking into account the following information provided by the situation analysis.

- Heads of households were equally divided between men and women. They are mostly Brazilians born in the city of Rio de Janeiro. Some are married and some continue in marital status. They live in houses that they own, which are built with bricks, have four or more rooms, running water and sewer. Most have completed primary school. They work and make up to two minimum wages, although very few have proper jobs. They have been living for at least twenty-years in these communities, although they were not born there. Most of them do not have health insurance; most do not do voluntary work.

Some have exposure to art and culture, and have been to the theater or museum.

- Most of the target families have one to three members, and most do not have all their identity documents; they make up to three minimum wages; most are not assisted by social Programs; very few do voluntary work; the great majority declare the absence of artists in their family; and very few know their communities history.

- The communities have television and radio; they practice different sports, mainly soccer. They do not have access to technology; they are interested in learning about drug abuse, family planning, child and adolescent health, old age, sexuality, nutrition, pre-natal care, and maternity. They are interested in training courses, in computer skills, mechanics, secretarial skills, tourism, and foreign languages. They lack daycare centers, preschools and schools in general, police department, health care centers, jobs office, professional training courses, among others. They also complain about the quality of the few opportunities offered in their communities. However, they approve the quality of their church, soccer team, and neighborhood association.

The information gathered allowed us to set our priorities. From the beginning, Cesgranrio Foundation believed that it should not be involved directly with the Program's actions, and kept just to the design, coordination and evaluation of the Program.

The first priority was the civil documentation sector. The Child and Youth Justice in a cooperative effort with Cesgranrio Foundation, the organizations responsible for the issuance of civil documentation, and others experienced in mobilization actions, served the inhabitants issuing birth and marriage certificates, IDs and developing adoption and custody processes.

The second priority was attention to children aged zero to three. In order to develop this activity, the World Organization of Pre-school Education (OMEP), Rio de Janeiro Section, designed and implemented a Project. OMEP's Project involved the creation of a group of mothers that would become an awareness generating center within the community, demonstrating the relevance of due attention to children and adolescents at this age level that would replicate in the communities the notion of human rights for these children. The main achievement of this effort was a sign-in list developed by the mothers demanding the installation of a day-care center within the community, to Rio de Janeiro's Town Hall. Beyond that, several courses have been developed, always according to community mothers' requests. OMEP also built a toy center that started operating within the neighbors' association.

The third priority was professional qualification. We invited an experienced institution (XEROX Institute) which developed a digital inclusion project in answer to the communities' main demand. It was distributed on two dimensions: one towards children and adolescents, with a more educational focus, and the other towards adults to provide access to the digital world.

The fourth priority was the accomplishment of cultural sports and leisure activities for all age levels. The XEROX Institute proposed and designed a proposition for this activity later discussed with Cesgranrio Foundation and converted into a project. The Project approaches seventeen kinds of sports: gymnastics for elders, volleyball, adults gymnastics, "capoeira", court soccer (men and women), and other activities such as video shows followed by debates, storyteller, dramatizing readings, popular dance (seven to fourteen), circles (seven to fourteen), theme workshops, poetry, popular percussion, theater (ten to thirty), choir (seven to nineteen), and recreation (seven to fourteen).

The fifth priority was what we called preventive actions towards children and adolescents (seven to fourteen years old). To design this project we invited a non-governmental organization (NGO) called "AMAR" beneficent Association. We must point out that the inhabitants were very much conscious that most social and personal high-risk situations involving children and adolescents occurred during the time they were not attending school due to the absolute lack of any other type of activity – which was clearly demonstrated by the situation analysis. This problem resulted in early pregnancy, drug abuse, involvement with criminal activities, child labor, in short, in the varied risk situations that are well known to us. "AMAR", thus, developed a project for eighty children of this age level contemplating a two-hour long extra-curricular activity, a snack-hour, play activities and lectures and trips.

As said before, a unique aspect of this experience was respect for the community's knowledge in detecting its own needs and strengths as well as raising its expectations for improvement long before designing the Program's development and evaluation. Another asset was putting together best partnerships in terms of their capacity to act in the areas of need and to conduct evaluation and metaevaluation (Joint Committee for Educational Evaluation, 1994), including continuous feedback from supervisors. These procedures helped guarantee overall excellence of the work. Along with collective action, emphasis was placed also on self-determination (Fetterman, 2005), social justice, capacity building and transformation. Importantly, these are communities severely affected by violence, mainly caused by drug trafficking, which imposed great risks both on collecting data and giving information.

The evaluation process

Simultaneously to the identification of priorities, Cesgranrio Foundation designed an external evaluation for the Program. The evaluation process had to be accurate and timely, but subtle, approaching the right people at the right time making sure no harm was done either to the evaluation team or the community. However, the commitment and enthusiasm of those involved in the evaluation process overcame these limitations. In addition, the team tried to integrate different evaluation models focused around a responsive approach (STAKE, 2004). Our purpose was of having an external view on the overall Program in order to portray the quality as well as the possible limitations and the anxieties and opinions of the beneficiaries. Thus, the evaluation reproduced the peculiarities of the Program, showing that in this case, we have an evaluation historian (CRONBACH, 1980) when we describe, register and interpret the singular day to day process of each focused scenario. During its development, the evaluation considered the values, the concerns, and the perceptions of the Program stakeholders and strived to identify the Program's merit, i.e., its internal value, and to discover its relevance in the success demonstrated by its results, repercussion and possible impacts.

Since it was an external evaluation, the evaluation team was composed of professionals with no direct connection with the Program. The advantage gained by this procedure was that we could offer a complementary vision to the self-perception of the Program stakeholders. Both, internal and external perspectives were necessary to achieve the desired success of the evaluation. The approach used in the above mentioned external evaluation was essentially inclusive (MERTENS, 2003) because it emphasized the insertion and participation of all communities affected by the decisions based on the evaluation process. This approach was present during all steps of the evaluative process. In this context, the role of the evaluator was a proactive one in the sense of intervening to change the elements that favor social injustice.

The starting point of the evaluation was in constructing the following evaluative question:

- To what extent is the Project being developed efficiently? (Merit)

- What are the evidences of impact on beneficiaries and communities at large in regards to the improvement of their quality of life? (Impact or Worth)

Cesgranrio Foundation's team of professionals involved in the Project constructed social indicators. This was done in order to allow for merit and impact evaluations, in terms of the intrinsic quality of actions, and benefits from them.

It was made sure that the identified merit and impact social indicators were consistent with the adopted evaluation approach. We collected data from three different sources: (a) beneficiaries, (b) partners, and (c) community. Several data collection procedures were used: (a) checklists, (b) interviews, (c) unobtrusive measures, and (d) focus groups.

Three kinds of checklists were developed to be used with partners (institutions in charge of program development), beneficiaries (children, youngsters and adult participants of the Program), and members of the communities in general, respectively. The items of the checklists were related to the previously mentioned indicators.

As for the interviews, different sets of questions were outlined to approach beneficiaries, partners and members of the communities in general.

Unobtrusive measures were utilized to collect spontaneous evidences of merit and impact.

Focus groups were conducted with representatives of all program participants lasting one hour each. All sessions were tape recorded. The data collected were organized according to the carefully constructed indicators of merit and impact.

Results

An essential feature of the whole evaluative task was the continuous utilization of its process and results (PATTON, 1997). The present evaluation was constantly oriented towards such recommendation. The evaluation team understood that in the social service area more than in any other, there is a need for permanent follow-up to the development of actions and opportunities during the intervention of any program from its conception to its conclusion and repercussions.

In this sense, it is hoped that the results of this program will affect not only the target communities but the overall Brazilian society, for the benefit of many, including those who are responsible for conducting programs, for developing evaluative or research studies, and holding policy discussions in the social arena.

Concretely, the results of this present evaluation were examined both quantitatively and qualitatively, in relation to indicators of merit and impact, grouped into categories as listed below. They emerged from the initial contact with the communities during the situational analysis previously mentioned.

Changes observed as evidences of merit and impact are summarized below.

In relation to merit:

• Quality of Program Actions: the results show that: (1) there is a constant interest for the best implementation of the Program activities in all the areas of action, and (2) an increase of competency among the professionals that work in the Program.

• Environmental conditions: there are indeed poor conditions in the environment over which the inhabitants have no power to intervene. These conditions hamper the possibilities of a healthy life. This is the case of sewer, The Program, however, has been acting consistently and little by little some changes have been observed such as "not pulling out leaves from trees", "not mistreating animals", "not throwing rocks" etc. Children and adolescents already show these behaviors with great enthusiasm and conviction. In school the improvement is remarkable. Thus, from school to community, they are growing.

• Diversified activities: the results show a constant adjustment of Program activities in the sense of responding to the interests of the beneficiaries and rapidly resolving conflicts. There are also evidences of the integration of actions between projects. This has been guaranteeing the quality of results and at the same becoming a constant example for the beneficiaries and the community.

In relation to impact:

• Personal development: a remarkable result is the evidence of respect for moral values that have been emerging little by little. The Program has showed impact on solidarity that although already existent, has been reinforced. An improved relationship among the inhabitants has been noted. Mothers are more involved in the Program and fathers, little by little. Self-esteem has been enhanced.

• Vulnerability: results show that children and adolescents still work but only to collaborate at home. Violence against children and youngsters has been considerably reduced.

• Health: the strongest evidence of change was the reduction of the answer "never" when the beneficiaries were inquired about (a) men forty years or older submitting themselves to prostate examination and (b) women submitting themselves to preventive cancer examination. A greater care towards the hygienic conditions of the community physical space was also observed.

• Education: the results point out to visible improvement in school performance. A great increase in interest for reading and writing was evident. School dropout was drastically reduced.

• Family: There are strong evidences of the improvement of the relationships within the families especially in relation to respect for children and adolescents who have also been encouraged to participate in the Program activities and to attend school.

• Culture: the results indicate a greater cultural production in the communities especially in what concerns music and theatre. Painting, dance and literature are also more visible among children, adolescents and adults.

• Work: there are evidences of a greater effort in searching for job among adults. However, more time seems to be necessary to perceive increase in employment possibilities and income improvement.

• Sport and leisure: the results show an increase of participation of children, youngsters and adults in sport and leisure activities. Concretely, there is a greater number of answer "always" by the beneficiaries when inquired if: (a) "I go to shows", (b) "I go to parties", and (c) "I practice sports".

• Development of the local social Program: the results indicate a remarkable increase in the Program being valued. In the initial phase of the Program, there were uncertainties about who would participate. The inhabitants of the target communities used to inquire among themselves whether they would be selected or not, whether the Program would give money to the Community Association, and pose other similar questions and worries. Today they have discovered that, as one of the beneficiaries said: "The situation is different – the 'Betting on Future' Program spread out, expanded all over the communities reaching all the children and youngsters. It spread opportunities, reinforced whatever was good in the community and whatever was worthwhile to bet on and everybody bet on it. This way, the Program has been integrating the community, through its diversified activities, especially involving women, children, youngsters and elders".

A unique characteristic of the evaluation process has been that of offering continuous feedback to all the stakeholders of the Program during the whole trajectory of its development. This procedure has undoubtedly prompted favorable changes among beneficiaries, partners and communities overall.

Conclusions

Combined with a collective action approach, emphasis was placed in this effort also on self-determination (FETTERMAN; WANDERSMAN, 2005), social justice, capacity building and transformation. Importantly, these communities have been severely affected by violence, mainly caused by drug trafficking, which imposed great risks both on collecting data and giving information. The evaluation process had to be accurate and timely, but subtle, approaching the right people at the right time making sure no harm was done either to the evaluation team or the communities. However, the commitment and enthusiasm of those involved in the evaluation process overcame these limitations. In addition, the team tried to integrate different evaluation models focused around a responsive approach (STAKE, 2004). Permanent feedback to the communities and to the partnership helped to enhance their self-esteem and shed light on their decisions and actions. Evaluation was the mainstream of this whole work, with several consequences: (a) the community showed an increase in the number of children and adolescents in school, with improved performance, and respect for moral values; it also showed reduced violence, sex abuse, child labor and drug consumption among others; (b) the impact on the organization responsible for the evaluation task was highly visible in the way they built an innovative methodology where sensitivity to ethical concerns, and creativity in putting evaluative procedures into practice were constantly present, alongside the physical risks. Hopefully, the betterment seen in the current situation is predictive of a dignified future that will be achieved by this community.

Sample expressions by children and adolescents

"The Program teaches values. Value is betting on the future."

"After entering the Program, I am not ashamed to speak out publicly."

"The start of the Program calmed down the local community."

"The persons of the Program want to help us to overcome difficult situations."

"In the future, I want to work, anything that would take me out of this life here in the slum."

"The Program is offering us an education that can continue in the future so that we will be able to get a good job, a house, resources and then be someone in life."

"Before the Program, we only talked about the slum. Now, we talk about classes, what we learned, and the activities we participate on, the things that are legal and those which are not. We now have more things to talk about."

Recebido em: 12/03/2007

Aceito para publicação em: 20/05/2007

• CRONBACH, L. et al. Towards reform of program evaluation. San Francisco: Jossey-Bass, 1980.
• FETTERMAN, D.; WANDERSMAN, A. (Ed.). Empowerment evaluation : principles in practice. New York: The Guilford Press, 2005.
• THE JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION. The program evaluation standards. 2^nd ed. Thousand Oaks: Sage Publications, 1994.
• MERTENS, D. M. The inclusive view of evaluation: visions for tne New Millenium. In: DONALDSON, S. I.; SCRIVEN, M. (Ed.). Evaluating social programs and problems: visions for the new millenium. New Jersey: Lawrence Eribaum, 2003.
• PATTON, M. Q. The challenge of making evaluation useful. Ensaio: avaliação e políticas públicas em educação; revista da Fundação Cesgranrio, Rio de Janeiro, v. 13, n. 46, p. 67-68, jan./mar. 2005.
• ______. Utilization focused evaluation: the new century text . Thousand Oaks: Sage Publications, 1997.
• STAKE, R. E. Standards-based & responsive evaluation Thousand Oaks: Sage Publications, 2004.

*

Paper presented at the 2006 Annual Conference of the American Evaluation Association, "Consequences of evaluation", Portland, Oregon, USA.

Publication Dates

• Publication in this collection
  25 Sept 2007
• Date of issue
  June 2007

History

• Accepted
  20 May 2007
• Received
  12 Mar 2007