Monitoring the crime prevention policy in Minas Gerais

Abstract

Methods of public policy monitoring have gained relevance with the emergence of managerial reforms of the public administration. However, the use of information by managers persists as an imminent challenge. The objective of this study is to identify and categorize, based on the literature, the use of the information collected from the monitoring strategies adopted in the Crime Prevention Policy in the Brazilian state of Minas Gerais, based on qualitative research. The study observed that managers make direct and political use of information in an instrumental, procedural, persuasive, and imposed form. However, the policy does not have a structured monitoring system. The production of information is concentrated in the local units of implementation, the publicity of the data is insufficient, and the activities are seen as instruments of control and inspection.

Keywords:
monitoring and evaluation; uses of monitoring; social crime prevention policy; Minas Gerais

Resumo

As sistemáticas de monitoramento de políticas públicas ganharam relevância no âmbito governamental a partir das reformas gerenciais da administração pública. A despeito disso, a utilização das informações pelos gestores persiste como desafio iminente. O objetivo do artigo é identificar e categorizar, à luz da literatura de referência, o uso dado às informações geradas pelas estratégias de monitoramento da Política de Prevenção Social à Criminalidade (PPSC) do Estado de Minas Gerais. São utilizados dados qualitativos coletados em pesquisa realizada com atores que atuam em diferentes áreas de execução da política. Constatou-se que os gestores fazem uso direto e político das informações, de tipo instrumental, processual, persuasiva e imposta. Apesar disso, observou-se que a PPSC não dispõe de um sistema de monitoramento estruturado. Assim, a produção de registros e dados fica concentrada nas unidades locais de implementação dos programas que compõem a política. Notou-se ainda que a publicização destes dados é insuficiente e que as atividades de monitoramento são percebidas como instrumentos de controle e fiscalização.

Palavras-chave:
monitoramento e avaliação; usos do monitoramento; política de prevenção social à criminalidade; Estado de Minas Gerais

Resumen

Las sistemáticas de monitoreo de políticas públicas adquirieron relevancia en el ámbito gubernamental a partir de las reformas gerenciales de la administración pública. Sin embargo, la utilización de la información por los gestores persiste como un desafío inminente. El artículo identifica y categoriza, a la luz de la literatura de referencia, el uso de las informaciones generadas por las estrategias de monitoreo de la Política de Prevención Social a la Criminalidad (PPSC) del estado de Minas Gerais. Se utilizan dados cualitativos de investigación realizada con actores involucrados en la gestión de la política. Se notó que los gestores hacen uso directo y político de la información, en la forma instrumental, procesual, persuasiva e impuesta. Sin embargo, la política no tiene un sistema de monitoreo estructurado, la producción de información se concentra en las unidades locales de implementación de los programas que componen la política, la divulgación de estos datos es insuficiente y las actividades de monitoreo se ven como instrumentos de control y fiscalización.

Palabras clave:
monitoreo y evaluación; usos del monitoreo; política de prevención social a la criminalidad; Minas Gerais

1. INTRODUCTION

The Policy on Social Prevention of Crimes (PSPC) of the Brazilian State of Minas Gerais started in 2003 as a pilot project. After 15 years, the policy was institutionalized in the state agenda, gaining new design and orientation. The PSPC counts on monitoring instruments since its early stages. At the beginning of the pilot project, these instruments were improvised and not well systematized, but have improved over time. The policy’s emphasis on monitoring and evaluation (M&E) systems as management tools is crucial. It may be explained by a result-based management approach that marked the state government in Minas Gerais during the period analyzed in this study.

There are numerous challenges around the effective implementation of M&E systems, particularly the use of data and information obtained through such tools. At the international level, studies on the use of evaluation results are not recent. The concerns about the topic emerged when evaluators realized that the decision-makers commonly disregarded the results of evaluation processes, losing opportunities to improve their initiatives.

In Brazil, although public policies M&E have gained space after the 1990s administrative, the monitoring and evaluation systems were limited - when observed - evidencing the failure to consolidate an organizational culture focused on these processes. Moreover, the effectiveness of such tools has not been well studied. The available references in Portuguese that deals with the outcomes of M&E processes is insufficient.

In the case of public safety policies, structuring a monitoring system is even more complicated. Institutional, methodological, and technical factors turn public safety policies, such as the PSPC in Minas Gerais, into something complex. Among these factors are a) the lack of integration among information produced; b) the traditional policy evaluation; c) the lack of crime indicators that measure the relationship between the sense of insecurity and the actual crime; d) the low reliability of data; and e) the fragility of the institutional culture of using information to improve policies.

PSPC consists of four programs, a) Fica Vivo! (Stay alive); b) Programa Mediação de Conflitos (PMC) (Conflict Mediation Program); c) Central de Acompanhamento de Alternativas Penais (Ceapa) (Center for Supervision of Alternative Sentences); and d) Inclusão Social dos Egressos do Sistema Prisional (PrEsp) (Social Inclusion of Former Prisoners of the State Prison System). The programs’ goals converge and contribute to “the prevention and reduction of violence and crime among vulnerable territories and groups, increasing safety in the state” (Estado de Minas Gerais, 2017, p. 7). Given a complex public security policy that strives to tackle multidimensional problems, it is clear the importance of using data and information obtained through monitoring as a management tool. The theme of this paper is related to this problem, and its aim is to analyze and categorize the use of information generated while monitoring the PSPC monitoring.

The methodology included a field study with a qualitative approach and data obtained through semi-structured interviews conducted with technicians and managers involved in the policy. The analytical model guiding the data collection and analysis consists of four dimensions: a) the process of building the PSPC monitoring instrument; b) the evaluation of the participating actors about the policy; c) the third dimension addresses the use of data and information; and d) the fourth dimension detects the main challenges involved in monitoring the policy and utilizing the data produced to improve the policy. The study offers suggestions regarding the improvement in using the tools presented here.

This article has three sections: a) theoretical and conceptual discussion on monitoring and uses of public program evaluation; b) brief presentation of PSPC and its programs; and c) presentation and analysis of results.

2. PUBLIC POLICIES MONITORING: CONCEPTS, TYPES, SYSTEMS, AND CHALLENGES

The study by Ramos and Schabbach (2012) indicated the polysemy of the field, presenting several concepts of program evaluation. In general, there is some consensus that this is a retrospective, prospective, and systematic value-forming process that can refer to numerous aspects of public policy, according to the evaluation’s purpose (Boullosa & Araújo, 2009; Ramos & Schabbach, 2012; Mokate, 2006).

Based on a normative perspective, the literature points out that evaluations should contribute to the improvement of public programs, “improve decision-making, obtain appropriate resource allocation, and foster decision-makers’ accountability” (Ramos & Schabbach, 2012, p. 1273).

There are many concepts for monitoring as well. The concepts refer to monitoring as a strategic process of government (Cardoso, 2014), a managerial tool to monitor activities to achieve specific goals (Garcia, 2001), or as a continuous evaluation process (Rua, 2004). In this article, monitoring is understood as a follow-up activity of a program’s key processes, timely identifying abnormalities during its implementation (Jannuzzi, 2016). Boullosa and Araújo (2009) use a similar approach, considering it a systematic follow-up of a program’s implementation and performance, a routine process of observations, records, and analyzes.

Monitoring and evaluation are complementary activities. According to Jannuzzi (2014, p. 32), M&E corresponds to “organically coordinated analytical processes that complement each other over time.” Ramos and Schabbach (2012, p. 1280) add that “evaluation requires information obtained from monitoring activities, offering a conclusion regarding a program’s efficiency, efficacy, and effectiveness; both studies [monitoring and evaluation] are, therefore, complementary.” M&E activities are crucial for public policies, as they provide information to support decision making and enable transparent implementation.

There are several ways to perform monitoring activities. Ideally, they should be based on solid models that are suited to different realities. According to Boullosa and Araújo (2009), “it is crucial that organizations make an effort to create their own monitoring models, even if only adapting existing or required models to local and organizational needs” (p. 185). Jannuzzi (2016) classifies monitoring as a) managerial; b) analytical; or c) strategic.

Managerial monitoring refers to follow up activities focused on processes, products, and results, according to the emphasis given to a program’s set of indicators. Regarding processes, professionals and managers involved in the production area are engaged in the analysis, whereas the organization’s central coordination observes the results, and the personnel from sectoral coordination is engaged in analyzing the products (Jannuzzi, 2016).

Analytical monitoring is a systematic analysis of information “on flows of financial disbursement, middle activities, product delivery, and the inference of program effects on a target audience” (Jannuzzi, 2011b, p. 41). When the design of a program is properly established, the analytical monitoring is a “quasi-evaluation” because it may suggest cause and effect relationships between activities and products, products and results, or results and potential impacts.

Managerial and analytical monitoring have different focuses. The first observes the management and its subcategories (processes, products, and results) to understand the balance and abnormalities in the input-output chain. The second emphasizes the evaluation, ascertaining whether the results, products, and processes are reaching the desired goal, considering the available resources (Jannuzzi, 2016).

Finally, Jannuzzi (2016) refers to the strategic monitoring as being a combination of managerial and analytical monitoring. According to the author, it can be applied to government plans, priority policies, or intersectoral policies. It requires specific tools and a better information system for timely data collection and comprehensive mediation of activities, products, deliverables, and results (Jannuzzi, 2016).

M&E mechanisms in Brazil are not as well developed as the instruments adopted in other countries with a consolidated culture of M&E (Jannuzzi, Resende, Silva, & Sousa, 2009). In Brazil, the information is collected, but it is not synthesized and shared intelligently. This “paradox of scarcity amidst abundance” demands improvements regarding processes of public policy information management (Jannuzzi, 2011a). Therefore, policies demand tailor-made (regarding design and management model) structuring of M&E systems.

M&E systems should respond to the allocation of financial and human resources; to intermediary processes of contracting services and integration of agents involved; and to product delivery. These systems must count on the support of four elements: a) Processes and outcomes mapping, an instrument that outlines a program’s intervention theory, and points out a lot of information to be monitored; b) key-processes indicator panels, organized according to the level of relevance of the information; c) definition of monitoring indicators that reflect the “general situation” of the program; and d) sources of information to build indicators, such as official websites of statistical agencies and international organizations (Jannuzzi, 2016).

However, establishing a culture of M&E in the public sector involves some challenges. For Mokate (2000), evaluation is crucial to improve policies, but public officials still perceive it as a threat, particularly when it is associated with mechanisms of control and auditing. Dulci (2012) discusses the importance of “creating evaluation capacity,” counting on a motivated and competent team, aware of technical, social, and methodological aspects, which requires financial, political, and intellectual investment. The use and communication of the evaluation results is another challenge, especially in decision-making processes (which is the topic addressed in this article).

The policies on crime prevention show other challenges for building a culture of M&E. Cano and Rojido (2016) offer a critical overview of using tools to improve these policies. According to the authors, less than 20% of the homicide prevention programs in Latin America go through impact evaluation processes. The reasons for this scenario involve - in addition to the lack of planning and technical limitations - particularities related to crime prevention, such as the complexity of prevention measures; the low reliability of the data; the displacement of violence in the territories; the lack of clear objectives; evaluations counting on internal data exclusively; variations in homicide rates; the difficulties in building control groups; and changes in short, medium, and long term outcomes.The low rate of programs evaluated may also be associated with insufficient monitoring strategies. Also, the absence of systematic data precludes fully useful evaluations.

At the national level in Brazil, the production of criminal statistics is insufficient and restricted to crime distribution among cities (Silva, 2017). In addition, the lack of information to measure crime at the national level leads to the use of death registration statistics from the Ministry of Health, for example, or from other institutions.

This lack of data from national criminal institutions demonstrates that there is little technical concern in measuring crime for managerial or operational purposes. The current registrations are merely for judicial and administrative use (Silva, 2017). Moreover, the lack of crime indicators that measure the relationship between insecurity and real crime reinforces the occurrence of decisions based on media pressure (Dias, 2008).

Silva (2017) emphasizes that, although the development of a national criminal statistics system is not an easy task, it would be essential to build comprehensive and reliable information. Such a system would result in constant and rich records, with details to identify patterns and regularity (the type of crime, place, characteristics, crime committed by whom and against whom), which is fundamental to formulate crime control strategies.

2.1 Using evaluation results: concepts and typology

When assuming evaluation as a complementary activity, organically linked to monitoring, its results can be treated in the same way as the information produced in processes of public policy monitoring. Such equivalence between monitoring and evaluation processes is justified by the fact that the Brazilian and international literature do not portray the use of monitoring activities in a specific way.

According to Serpa (2010), although the topic is not so discussed in Brazilian academic references, the use of evaluation results has been studied internationally since the 1970s. At that time, practitioners started to question the effective use of evaluations in decision-making and policy improvement. Other authors such as Bechelaine (2013) and Faria (2005) also observe the flaws regarding the evaluations’ feedback effect, pointing out that evaluators naively believe that the results of their work would be automatically used in decision-making. According to Faria (2005), organizations do not necessarily offer the conditions needed to use the evaluation results, as well as this practice often competing with other proposals and sources of information during the decision-making process.

Evaluation results are any information, interpretations, data, or recommendations, communicated at the end of the evaluation or as the evaluation was proceeding (Cousins & Leithwood, 1986 apud Bechelaine, 2013). As for the concept of ‘utilization,’ it includes here “new and intangible ways to influence individuals, programs, and communities” (Serpa, 2010, p. 40). Therefore, this study considers that the evaluation use means the “purposeful application of evaluation processes, findings, or knowledge to produce an effect” (Lawrenz, Gullickson, & Toal apud Serpa, 2010, p. 41).

Alkin and Coyle (as quoted in Bechelaine, 2013) differentiate the concepts of non-use, misuse, and abuse. For the authors, ‘non-use’ would be the “stall” of the evaluation reports, and it can be justified if the evaluation was not properly conducted. ‘Misuse’ would be the poor elaboration and misinterpretation of evaluation reports. Finally, ‘abuse’ would be the manipulation of evaluation results for disclosure.

There are different dimensions to measure the use of evaluation results; some of them may lead to allowing profound changes in both policies and involved actors (Bechelaine, 2013).

The first dimension is the ‘instrumental’ use, which occurs when the evaluation results aim to subsidize decision-making or solving a problem. In such cases, the evaluator should be knowledgeable about the operation of the program and the other issues surrounding it. The second dimension is that of conceptual use, which is limited to professionals implementing the policy. Although in this dimension the evaluation results are not adequate to subsidize decision-making, they may inspire new ideas and promote changes in the program activities. The third dimension concerns the symbolic or political use. It serves to justify a process of change in the program when policymakers already know what should be changed. Evaluation is, therefore, used as an instrument of persuasion (Bechelaine, 2013).

As studies on evaluation and the uses of evaluation results have expanded, new typologies of utilization have emerged (Patton, 2008). In addition to the instrumental, conceptual, and symbolic or political dimensions, Weiss (1998) adds the use of enlightenment. For the author, evaluations may inspire other institutions or events beyond the scope of the program evaluated.

Patton (2008) includes five other types of use for evaluation results. The process use occurs when the actors involved in the evaluation, except the evaluators, present behavioral and cognitive changes. These changes occur in the process of learning values and reasoning due to coexistence with the evaluators. Thus, it overlaps the instrumental and conceptual use of evaluation results, as it involves better use of the results of the evaluations, improvement of the skills of the actors involved, and, as a consequence, better decision-making. ‘Legitimative’ use occurs when the results of evaluation support or justify a particular decision made before the evaluation takes place. ‘Persuasive’ use refers to the use of results selectively, i.e., using parts of the results to support political and budgetary decisions. As for the imposed use, it occurs when higher-level agents force those actors in lower hierarchical levels to use the evaluation results. The last type added by Patton (2008) was the mechanical use, which refers to an evaluation conducted to meet a requirement (an evaluation required to approve the budget increase, for instance). The analysis of the use of PSPC monitoring data will be performed in light of the categories proposed by Patton (2008).

3. POLICY ON SOCIAL PREVENTION OF CRIME IN THE STATE OF MINAS GERAIS

The State Secretary of Social Defense (SEDS) of the Brazilian state of Minas Gerais was created in 2003, to reverse the high crime rates and provide more effective responses in the area of public security (Estado de Minas Gerais, 2009). The new State Secretary aimed to challenge the perspective of repression in fighting crime, expanding the sense of public security toward the idea of “social defense” (Oliveira, 2017). The rationale behind this development and the emergence of the Policy on Social Prevention of Crime in the State of Minas Gerais (PSPC) is that fighting crime and violence is a task that does not depend only on government strategies, but also on the participation of civil society. Therefore, interventions on conflicts, violence, and processes of criminalization must occur with the public’s direct participation (Souza, 2016). Oliveira (2017, p. 95) emphasizes that strengthening policy institutionalization process, expanding the formal internal structure and the scope of crime prevention units (UPCs), are practices that require “budget prioritization, publicization of the responsible government agency, support to public officials in charge, and parity with other areas engaged in public security.”

This section introduces the four programs that are part of the PSPC: a) Fica Vivo! (Stay alive); b) Programa Mediação de Conflitos (PMC) (Conflict Mediation Program); c) Central de Acompanhamento de Alternativas Penais (Ceapa) (Center for Supervision of Alternative Sentences); and d) Inclusão Social dos Egressos do Sistema Prisional (PrEsp) (Social Inclusion of Former Prisoners of the State Prison System).

3.1 “Fica Vivo!”

The creation of the Fica Vivo! Program (FV) in the 2000s, was based on the significant increase in homicides in Belo Horizonte in the 1990s in an attempt to find alternative ways of fighting violence (Marques, 2015). In addition, studies showed that the growth in homicides in the capital was concentrated in certain regions, especially in the peripheral areas and favelas. The population more frequently involved in crimes was formed by Brazilian “Pardos” and Black people, male, and aged between 15 and 29 years (Estado de Minas Gerais, 2009).

In this context, the state government implemented a pilot project in a specific part of Belo Horizonte (the capital city of Minas Gerais) that ran from August to December 2002. An evaluation conducted six months after the implementation showed a 47% reduction in homicides in the locality (Decreto Estadual n. 43.299, 2003).

According to Miranda (2015), after the pilot project, the FV was institutionalized with the State Decree 43,334 of 2003. The program aimed to contribute to the prevention and reduction of intentional homicides of adolescents and young people living in the areas covered by local units of Centros de Prevenção a Criminalidade (CPCs) (Crime Prevention Centers.)

According to Souza (2016), the program observes two main variables, age group, and territory. The first refers to young people between 12 and 24 years old. As for territory, the program focuses on areas of high rates of intentional homicides with the participation of young people, both as victims and as perpetrators.

The program’s methodology observes the coordination of two lines of action, a) social protection and b) strategic intervention. Social protection covers the analysis of the social dynamics of violence and crime and offers opportunities such as sports, culture, and art workshops, personal assistance, community projects, and forums. Strategic interventions consist in inter-institutional coordination between the different Brazilian police forces (Military and Civil Police), the Public Ministry, the Judiciary, the municipal public security agencies, the State Secretariat of Public Security (SESP) and the State Secretariat of Prison Administration (SEAP) (Estado de Minas Gerais, 2017).

3.2 Mediation

In 2005, the Programa de Mediação de Conflitos (PMC) (Conflict Mediation Program) was included in the PSPC (Marques, 2015). The program uses peaceful conflict resolution techniques to minimize social risks, reduce inequalities, and address violence. The PMC implements activities involving residents, community leaders, and partner organizations. It prioritizes dialogue, understanding the reality of the territory, and the involvement of various actors, to promote solutions to community problems. The program serves territories characterized by limited access to basic rights, high violence rates, and low social capital (Estado de Minas Gerais, 2017).

The general objective of the PMC is to “promote peaceful forms of conflict management at the interpersonal, community, and institutional spheres, to minimize, prevent, and avoid these conflicts unfolding in violence and crime” (Estado de Minas Gerais, 2017, p. 31). The program’s activities take place through social and institutional participation in order to reduce homicides resulting from violence against women, conflicts between neighbors, domestic and family violence, and violation of rights (Estado de Minas Gerais, 2017).

The PMC’s methodology is organized in four areas, a) individual assistance; b) group assistance; c) thematic projects; and d) institutional projects (Estado de Minas Gerais, 2017).

3.3 Supervision of Alternative Sentences

The Central de Acompanhamento de Alternativas Penais (Ceapa) (Center for Supervision of Alternative Sentences) was established in 2002, following guidelines of the National Center for Support and Supervision of Alternative Sentences (Cenapa) of the Ministry of Justice (Estado de Minas Gerais, 2009). The State Secretary of Justice and Human Rights (SEJDH) was the agency responsible for the implementation of Ceapa. In 2003, after the creation of SEDS, the program became part of the structure of the Department of Social Prevention Policies (Supec), supported by the State Decree 43,294 of 2003 (Marques, 2015).

The overall objective of the program is “to contribute to strengthen and consolidate alternatives to imprisonment in the State of Minas Gerais, offering guidelines to activities that combine rendering convicted persons accountable at the same time as offering the opportunity for them to enjoy freedom” (Estado de Minas Gerais, 2017, p. 50). The objective of the program is timely, considering the context of the country’s increasing prison population.

Ceapa’s methodology, differently from the FV and the program for social inclusion of former prisoners (presented below), is not organized in lines of action. It is based on a set of protocols, flows, procedures, according to how the alternative sentences unfold. First, the judiciary decides an alternative sentence. After that, the partner network receives training and support. The convicted person then is conducted to serve time, always supervised and assisted by the program (Souza, 2016).

The modalities of alternative sentences offered in the program are a) community services; b) participating in thematic projects related to alternative sentences (designed based on the type of felony); c) individual assistance or specialized groups of men convicted of violence against women according to Brazilian Law (Maria da Penha Law); d) special projects; and e) recuperation projects and practices (Estado de Minas Gerais, 2017).

3.4 Social Inclusion of Former Prisoners

Promoted by the Ministry of Justice, the program Inclusão Social dos Egressos do Sistema Prisional (PrEsp) (Social Inclusion of Former Prisoners of the State Prison System), formerly known as Social Reintegration Program, started in 2002. A year later, it was redesigned and included in the PSPC with the new name (Estado de Minas Gerais, 2017).

Although international and national legal provisions consider the sentence of deprivation of liberty due to a criminal act as a temporary penalty - without prejudice to the physical, mental, and social integrity of the convicted, and with the aim of resocialization - it is known that Brazil does not fulfill the promises of reintegration and social inclusion (Souza, 2016).

Against this backdrop, PrEsp aims to intervene in this transition between the execution of the sentence of imprisonment, and the return of the felon to society. The program’s objective is to “promote the access to rights and conditions for social inclusion of former prisoners from the State Prison System, minimizing the vulnerabilities related to criminalization and aggravated by imprisonment” (Estado de Minas Gerais, 2017, p. 68). According to Souza (2016), such action makes the return to social life, not a purely individual effort, but a balance between constitutional provisions and public policies.

The program’s target audience are felons (and their family members) serving in open-prison, house arrest, felony probation, and former prisoners. The methodology designed involves assistance, referrals, connections with the municipalities assistance network, discussion of cases, home visits, assistance groups with other former prisoners, and prisoners about to leave the state prison system (Estado de Minas Gerais, 2017).

4. PPSC MONITORING INSTRUMENT IN MINAS GERAIS AND THE USE OF THE EVALUATION RESULTS

The results of this study are presented according to the four dimensions of the analytical model: a) the process of building the PSPC monitoring instrument; b) the evaluation of the participating actors about the policy; c) the use of data and information; and d) the main challenges involved in monitoring the policy and utilizing the data produced to improve the policy.

The fact that the policy counts on some sort of monitoring is explained by the use of Regarding the PSPC monitoring instrument, it is possible to say that the policy has always been monitored, albeit poorly, considering the tools available throughout time. This happened because the implementation contracts (Partnership Agreements) are legal instruments that require goals and instruments to monitor the implementation partner’s performance, under the supervision of the SESP and the state’s central monitoring agency, the Secretariat of Planning and Management (Seplag).

The actors involved in the process of setting goals and indicators are perceived as strategic for the policy (deputy secretary, superintendents, directors, managers of the department of planning of SESP, and managers of Seplag). The interviewees understand that the process of collaboration between these actors was based on productive conversations and very positive.

It was also noted that there was little correspondence of the PSPC monitoring instrument with the model proposed by Jannuzzi (2016). Although the PSPC model did not have a panel of indicators, i.e., the display of indicators according to their relevance and the user’ needs, as proposed by Jannuzi (2016), the policy adopted a model that allowed to monitor indicators, exposing the overall situation of the programs. Furthermore, the elaboration of the programs’ logical framework, similar to the processes and outcomes mapping, was crucial for delimiting the scope of monitoring and defining the goals and indicators. It was possible to identify that the PSPC adopted a managerial type of monitoring. For Jannuzzi (2016), this type of management aims to monitor the goals, deadlines, and indicators of processes and products.

As for the second dimension, almost all respondents agreed that the monitoring activities support the public policy’s improvement. The exception, probably based on specificities of the program, was pointed out at the Ceapa, where the monitoring activity was understood as inspection instead of an instrument to improve the program. Because Ceapa is a program of secondary prevention, i.e., the individuals benefited were already convicted by the judiciary and must comply with the sentence attributed, the activity of monitoring may be more easily related to the typical supervision or inspection of fulfilling duties. In addition, the spreadsheet of the program used to record the progress of the convicted person’s compliance with the sentence is called “monitoring spreadsheet,” a particularity that may influence the perception observed in the interviews.

In the Fica Vivo! program, some resistance to monitoring was observed. According to reports, for a long time, the program advertised that “the priority was the quality of the work,” which led the person to consider the activity of completing the spreadsheets related to M&E as less important. However, it is noteworthy that the existence of monitoring does not presuppose the exclusion of the quality of activities and vice versa. According to Mokate (2000), it is necessary to work on the understanding that the “quantitative” and the “qualitative” content are complementary. Regarding the fact that the registration form for the youth participating in the program was not completed appropriately, it is necessary to study it more in-depth. It is important to identify whether the person responsible for the workshops was resistant to perform this task, or if the difficulty was related to characteristics of the program, the public served, and the methodology used. The cases of Ceapa and FV reinforce the need to consolidate a monitoring culture at PSPC.

The actors involved in the monitoring activities demonstrated knowledge of the instrument and recognized its operational and strategic advantages. However, they mentioned some disadvantages such as the difficulty of creating results and impact indicators for the programs, which can more accurately reflect the relationship between the activities performed and the changes in the territories and lives of the beneficiaries. This presupposes the construction of analytical rather than managerial monitoring, according to the typologies of Jannuzzi (2016), in order to explain the cause and effect relationships that this type of monitoring can establish. Overcoming this obstacle leads to the recommendations by Mokate (2000). The author advocates for financial, political, and intellectual investment so that the monitoring team, and other strategic actors involved in the process, have the technical, social, and methodological abilities to build an M&E system that meets the needs and expectations of the PSPC actors in Minas Gerais. This includes the relevance of building a database that allows crosschecking the information of the four programs, a panel of indicators, and infrastructure, equipment, and human resources, able to provide excellent monitoring.

As for the third dimension, there was frequent use of data and information generated by the PSPC monitoring instrument. It was possible to observe the ‘direct’ and ‘political’ categories of use, and the ‘instrumental,’ ‘process,’ ‘persuasive,’ and ‘imposed’ types of use, mentioned in the model proposed by Bechelaine (2013), as shown in Box 1. The box also presents excerpts from the interviews that illustrate the types of uses identified.

BOX 1
TYPE OF DATA AND INFORMATION USE IN THE MONITORING INSTRUMENT OF THE POLICY ON SOCIAL PREVENTION OF CRIME IN THE STATE OF MINAS GERAIS

Source: Elaborated by the authors.

BOX 1 TYPE OF DATA AND INFORMATION USE IN THE MONITORING INSTRUMENT OF THE POLICY ON SOCIAL PREVENTION OF CRIME IN THE STATE OF MINAS GERAIS

Almost all respondents mentioned the instrumental use of evaluation results. The persuasive use was also common, cited by practically all actors, whether referring to the need for institutionalization or policy strengthening, either for arguments to defend budget allocation or obtaining alternative forms of financing, which is justified by the scarcity of budgetary resources for the PSPC. Respondents also repeatedly referred to the imposed use, i.e., to a need to be accountable to public administration bodies and agents (Supec management group, governor, and mayors) and society. When asked if the use of this data and information was able to change the perception of public policy about M&E - evidence of the ‘process’ use - only one respondent noticed some transformation in this sense.

It was not observed the use of evaluation results according to the ‘mechanical,’ ‘legitimative,’ ‘symbolic,’ ‘conceptual,’ and ‘enlightenment’ types. Also, there was no evidence of misuse and abuse, as portrayed by Alkin and Coyle (apud Bechelaine, 2013).

Regarding the disclosure of the information obtained through the PSPC monitoring process, the results were disclosed on the websites of SESP, Seplag, and Instituto Elo¹. For the actors involved, however, this was not enough for the population to know the results of the policy, indicating poor communication of information. Interviewees cited factors that influenced this issue, such as the high social expectation with repression practices, in contrasts with the policy’s prevention actions; the strategic place in government occupied by the Military Police and its traditional role of repression; and the interest of the media in publicizing the evolution of crime rates. It is noteworthy that these facts differ from the reports regarding the ‘imposed use,’ to render accounts to society.

Finally, addressing the challenges regarding monitoring and use of information (fourth dimension), the interviewees mentioned issues referring to resources in general, such as computers, the Internet, and the workforce. They also mentioned operational circumstances, such as failing to do other activities because of the duty to complete the monitoring spreadsheet; and more technical aspects such as a) absence of a tool that meets the need of the actors to create a database and connecting information with other programs; b) difficulty in turning information into knowledge in order to improve the policy; c) failing to use the results of external research; d) difficulty in producing timely and more qualified information.

The study found that the M&E system of the PSPC in Minas Gerais needs to be improved, considering a change to its characteristic as a “one-way street,” in which the units produce data and information and send it to the managerial group. The policy managers have to be able to produce more frequent and prompt analyzes to support the building of increasingly qualified interventions in the field.

It was possible to observe the “paradox of scarcity amidst abundance,” where there is information but it is not synthesized and shared intelligently, as mentioned by Jannuzzi (2016). Many data is not transformed into information and therefore has no direct utility. It is also emphasized that the FV demonstrated weakness regarding the reliability of their data and information, a common conjuncture in the Brazilian public security field, according to Lima (2005).

To enhance the use of data and information, the respondents considered it crucial to constantly demonstrate the importance of the information about the policy to improve the work of the teams. They also argued that increased dialogue between the management group and the units is fundamental, as well as to strengthen ties with universities, and the creation of mechanisms for public participation and evaluation.

Regarding the elements that would help to improve monitoring, there is a need for information exchange with other sectors, the creation of effectiveness indicators, and continuous training.

It was possible to see that some of the challenges and suggestions that emerged from this research are aligned with the referenced literature. The first element that stands out is the need to build monitoring capacity, as observed by Dulci (2012). Also, the necessity of material and human resources for effective monitoring was mentioned, as well as the urgency to create a monitoring system that meets the needs of the actors. The dissemination of results is another essential element, which allows the creation of moments of reflection and discussion of findings, both for internal and external actors. In this sense, respondents mentioned that more profound dialogues would be helpful to approximate the public officials managing the policy to the actors engaged in the daily implementation, also connecting monitoring instruments to the field of action, as suggested by Mokate (2006).

5. FINAL CONSIDERATIONS

The methodological complexity of the programs that make up the PSPC reveals the importance of M&E. It shows that the policy still does not have an M&E system, as described in the literature. The study observed that the monitoring spreadsheets recording data from the various programs do not “communicate” with each other, preventing the systemic analysis of the policy. The programs are, therefore, monitored in a fragmented manner. PSPC would benefit from an M&E system that meets the needs and expectations of program actors and the PSPC goals in general, with specific and common panel data and indicators for the four programs, as well as adequate equipment and human resources. These observations support the conclusion that analytical monitoring is needed to complement the managerial monitoring in place. In the case of public security, this measure is important in designing crime control strategies. Therefore, financial, political, and intellectual investment is indispensable for the monitoring team, and the other strategic actors involved, providing technical, social, and methodological capacities, as suggested in the literature.

Regarding the types of use of the evaluation results from the PSPC monitoring, the ‘instrumental’ was used for decision-making; the ‘persuasive’ to support political or financial debates; the ‘imposed’ to offer accountability; and, finally, the ‘process’ use, to change the perception about the M&E systems in public policies. Moreover, it was noticed that elements of the ‘instrumental,’ ‘persuasive,’ and ‘imposed’ uses were mentioned several times during the interviews, both by the policy management group and the interviewees working in the field. The ‘process’ type of use, however, was mentioned only once, by a technician working in the field. This phenomenon may be justified by the fact that the Minas Gerais PSPC is implemented through Partnership Agreement. These types of implementation contracts are based on indicators and goals, which is a logic aligned with the framework of monitoring explored in this study. It also suggests the need to institutionalize such practices to turn the agenda of crime prevention into a state policy.

However, some aspects lead to underutilization of the information produced by the monitoring instrument. First, the informal and inadequate perception, in the case of the Ceapa program, that monitoring was an activity of inspecting convicted felons in the accomplishment of their sentence. This view was recurrent throughout the research and appears to be a particularity of the program, which is a situation that needs to be reframed among the actors involved.

It was also noted that although the FV is PSPC’s oldest and best-known program in Minas Gerais, it is the one that demonstrates the biggest weakness in monitoring. In addition to presenting, in some cases, low data reliability, a common situation in the field of Brazilian public safety, this program has difficulties in completing the registration of young participants.

In addition, it was found that the production of policy information is mainly concentrated in the UPCs, that is, the monitoring of the PSPC works basically with a single flow: from the field to the management group. The concentrated production of information at the units brings a sense of control exercised by the management group and may jeopardize the meaning of monitoring for all actors involved. That said, the managerial group should produce analyzes more frequently and promptly, to subsidize the construction of quality interventions in the field.

Another relevant factor was the insufficient disclosure of the policy’s results in Minas Gerais. Although the actors understand the usefulness of the information produced, little is invested in the dissemination to civil society and other strategic actors. Therefore, it is relevant to promote the dissemination of results and create moments of reflection and discussion of findings, both for internal and external actors. The appropriation of this information by these individuals gives meaning to M&E and enables greater use of results.

This study intends to offer subsidies to strengthen the PSPC monitoring in Minas Gerais and maximize the use of information obtained through this instrument, producing effective results to benefit the program users and the population in general.

REFERÊNCIAS

• Bechelaine, C. (2013). As avaliações vão para a gaveta? um estudo da utilização das avaliações de impacto no Governo de Minas Gerais (Dissertação de Mestrado). Universidade Federal de Minas Gerais, Belo Horizonte, MG.
• Boullosa, R. F., & Araújo, E. T. (2009). Avaliação e monitoramento de projetos sociais Curitiba, PR: IESDE.
• Cano, I., & Rojido, E. (2016). Mapeo de programas de prevención de homicidios en América Latina y el Caribe (Informe Final). Rio de Janeiro, RJ: Universidade do Estado do Rio de Janeiro.
• Cardoso, J.Jr . (2014). Monitoramento estratégico de políticas públicas: requisitos tecnopolíticos, proposta metodológica e implicações práticas para alta administração pública. Revista Brasileira de Monitoramento e Avaliação, 7, 4-29.
• Decreto Estadual n. 43.295, de 29 de abril de 2003 (2003). Dispõe sobre a organização da Secretaria de Estado de Defesa Social. Belo Horizonte, MG.
• Decreto Estadual n. 43.334, de 20 de maio de 2003 (2003). Cria o Programa de Controle de Homicídios do Estado de Minas Gerais. Belo Horizonte, MG .
• Dias, G. (2008). A informação como instrumento de planejamento de políticas públicas aplicadas à segurança pública. In A. P. M. Miranda, & L. L. Lima (Eds.), Políticas públicas de segurança, informação e análise criminal (pp. 159-191). Niterói, RJ: Universidade Federal Fluminense.
• Dulci, O. (2012). Avaliação de programas sociais: desafios e potenciais na construção de um sistema de informações. Cadernos de Estudos - Desenvolvimento Social em Debate, 13, 221-237.
• Estado de Minas Gerais. (2009). Prevenção social à criminalidade. A experiência de Minas Gerais Belo Horizonte, MG : Superintendência de Prevenção à Criminalidade.
• Estado de Minas Gerais. (2017). Portfólio da política de prevenção social à criminalidade Recuperado de http://www.seguranca.mg.gov.br/images/seds_docs/Prevencao/Portflio%20-%20Preveno%20Social%20%20Criminalidade%202017.pdf
  » http://www.seguranca.mg.gov.br/images/seds_docs/Prevencao/Portflio%20-%20Preveno%20Social%20%20Criminalidade%202017.pdf
• Faria, C. (2005). A política da avaliação de políticas públicas. Revista Brasileira de Ciências Sociais, 20(59), 97-110.
• Garcia, R. (2001). Subsídios para organizar avaliações de ação governamental (Textos para Discussão IPEA, n. 776). Brasília, DF: Instituto de Pesquisa Econômica Aplicada.
• Jannuzzi, P. (2011a). Avaliação de programas sociais no Brasil: repensando práticas e metodologias das pesquisas avaliativas. Planejamento e Políticas Públicas, 16(36), 251-275.
• Jannuzzi, P. (2011b). Monitoramento analítico como ferramenta para aprimoramento da gestão de programas sociais. Revista Brasileira de Monitoramento e Avaliação, 1, 36-65.
• Jannuzzi, P. (2014). Avaliação de programas sociais: conceitos e referências de quem a realiza. Estudos em Avaliação Educacional, 25(58), 22-42.
• Jannuzzi, P. (2016). Monitoramento e avaliação de programas sociais: uma introdução aos conceitos e técnicas Campinas, SP: Alínea.
• Jannuzzi, P., Resende, L., Silva, M., & Sousa, M. (2009). Estruturação de sistemas de monitoramento e especificação de pesquisas de avaliação, os problemas dos programas públicos no Brasil. Reflexões para Ibero-América: avaliação de programas sociais (Cadernos EIAPP, pp. 101-138). Brasília, DF: Escola Nacional de Administração Pública.
• Lima, R. (2005). Contando crimes e criminosos em São Paulo: uma sociologia das estatísticas produzidas e utilizadas entre 1871 e 2000 (Tese de Doutorado). Universidade de São Paulo, São Paulo, SP.
• Marques, B. (2015). Parcerias entre Estado e terceiro setor: um estudo de caso da política de prevenção social à criminalidade do Estado de Minas Gerais (Monografia de Graduação). Fundação João Pinheiro, Belo Horizonte, MG .
• Miranda, D. (2015). O papel da política de prevenção à criminalidade para o alcance da segurança cidadã: estudo de caso dos programas de base local do governo do Estado de Minas Gerais (Dissertação de Mestrado). Belo Horizonte, MG : Fundação João Pinheiro.
• Mokate, K. (2000). Convirtiendo el “monstro” en aliado: la evaluación como herramienta de la gerencia social. Revista do Serviço Público, 53(1), 89-131.
• Miguel Vera (2006). Uso de la información generada por la evaluación de programas sociales. In Evaluación para el desarrollo social: aportes para un debate aberto en América Latina (pp. 21-50). Ciudad de Guatemala, Guatemala: Magna Terra. 157
• Oliveira, L. (2017). Parceirização entre Estado e terceiro setor: uma análise institucional do modelo de parcerias com organizações da sociedade civil de interesse público (OSCIPs) em Minas Gerais e da execução da política de prevenção social à criminalidade via termo de parceria (Dissertação de Mestrado). Fundação João Pinheiro, Belo Horizonte, MG .
• Patton, M. (2008). Utilization-focused evaluation Thousand Oaks, CA: SAGE.
• Ramos, M., & Schabbach, L. (2012). O estado da arte da avaliação de políticas públicas: conceituação e exemplos de avaliação no Brasil. Revista de Administração Pública, 46(5), 1271-1294.
• Rua, M. (2004). A avaliação no ciclo da gestão pública (Manuscrito não publicado). Brasília, DF: Escola Nacional de Administração Pública.
• Serpa, S. (2010). Para que avaliar? Identificando a tipologia, os propósitos e a utilização das avaliações de programas governamentais no Brasil (Dissertação de Mestrado). Universidade de Brasília, Brasília, DF.
• Silva, L. (2017). Políticas públicas e números. Analisando a construção e os usos da informação na segurança pública no território do Rio de Janeiro (Tese de Doutorado). Universidade Federal do Rio de Janeiro, Rio de Janeiro, RJ.
• Souza, T. (2016). A implementação da política estadual de prevenção social à criminalidade de Minas Gerais: paradigmas, coalizões e incrementalismo (Dissertação de Mestrado). Fundação João Pinheiro, Belo Horizonte, MG .
• Weiss, C. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-33.

• 1
  Civil Society Organization of Public Interest (OSCIP), responsible for the implementation of the PSPC in Minas Gerais, via the Partnership Agreement 44, signed in 2017 and valid until 2020.

• [Translated version] Note: All quotes in English translated by this article’s translator.

Publication Dates

• Publication in this collection
  26 June 2020
• Date of issue
  May-Jun 2020

History

• Received
  12 Feb 2019
• Accepted
  08 Sept 2019