Composite indicator to evaluate quality of municipal management of primary health care

Abstracts

OBJETIVO: Desenvolver um indicador sintético para avaliar a qualidade da gestão municipal da atenção básica à saúde. MÉTODOS: O modelo de avaliação baseia-se em aspectos da gestão do sistema de saúde. Foram utilizados 55 indicadores de desempenho classificados sob os critérios de relevância, efetividade, eficácia e eficiência e suas medidas agregadas por meio de aplicação de análise envoltória de dados de modelo aditivo, em medidas de valor, mérito e qualidade. A aplicação foi feita a 36 municípios catarinenses com população entre 10 mil e 50 mil habitantes em 2006. RESULTADOS: Os resultados da aplicação foram apresentados em medidas monóticas no intervalo [0, 1] (medidas = 1: eficientes; demais: ineficientes). Cinco municípios apresentaram medida = 1 na qualidade da gestão das ações de acesso, enquanto oito obtiveram medida = 1 na qualidade da gestão das ações de provimento. Os demais municípios, para ambas as dimensões, foram classificados como ineficientes (medidas < 1). CONCLUSÕES: A qualidade da gestão municipal da atenção básica à saúde pode ser avaliada com indicador sintético, construído por técnicas de programação linear, que contempla simultaneamente os critérios de relevância, de efetividade, de eficácia e de eficiência agregados em medidas de valor, mérito e qualidade.

Indicadores de Qualidade em Assistência à Saúde; Atenção Primária à Saúde; Gestão em Saúde; Administração Municipal

ORIGINAL ARTICLES

Composite indicator to evaluate quality of municipal management of primary health care

Actividad física en adultos y ancianos evaluados por acelerometría

Dirceu Scaratti^I; Maria Cristina Marino Calvo^II

^IÁrea de Ciências Exatas e da Terra. Universidade do Oeste de Santa Catarina. Joaçaba, SC, Brasil

^IIDepartamento de Saúde Pública. Centro de Ciências da Saúde. Universidade Federal de Santa Catarina. Florianópolis, SC, Brasil

^{Correspondence}

ABSTRACT

OBJECTIVE: To develop a composite indicator to evaluate the quality of municipal management of primary health care.

METHODS: The evaluation model focuses on aspects of health system management. Fifty-five performance indicators were used and classified according to the criteria of relevance, effectiveness, efficacy and efficiency. The measures were aggregated through an additive data envelopment analysis model for measures of value, merit and quality. Data was utilized from 36 municipalities in Santa Catarina State (Southern Brazil), with populations between 10 thousand and 50 thousand residents in 2006.

RESULTS: The results are presented as monotonic measures over the interval [0, 1] (score = 1: efficient; other values: inefficient). Five municipalities had a score of 1 in the quality of management for actions promoting access, while eight municipalities received a score of 1 in the quality of management of actions for service provision; the other municipalities were classified as inefficient (score < 1) for both dimensions.

CONCLUSIONS: The quality of municipal management in primary health care can be evaluated with a composite indicator, constructed through linear programming techniques, which simultaneously considers the criteria of relevance, effectiveness, efficacy and efficiency and expresses them as measures of value, merit and quality.

Descriptors: Quality Indicators Health Care. Primary Health Care. Health Management. Municipal Management.

RESUMEN

OBJETIVO: Analizar el uso de la acelerometría como medida objetiva de actividad física en adultos y ancianos.

MÉTODOS: Revisión sistemática en las bases PubMed, Web of Knowledge, EBSCO y Medline, del 29 de marzo al 15 de abril de 2010. Se utilizaron en la búsqueda las palabras-clave: "accelerometry", "accelerometer", "physical activity", "PA", "patterns", "levels", "adults", "older adults" y "elderly", de forma aislada o combinadas usando "and" o "or". Las listas de referencias de los artículos recuperados fueron examinadas para encontrar artículos potenciales. De los 899 estudios localizados, 18 fueron revisados integralmente, y sus datos extraídos y analizados.

RESULTADOS: Once estudios se realizaron en los Estados Unidos, cinco en Europa, uno en Camerún y otro en Australia. Pocos involucraron ancianos, y sólo uno se refirió a la estación o período del año en que transcurrió la colecta de datos. Los métodos, análisis y resultados discreparon entre los estudios, imposibilitando un análisis más a fondo.

CONCLUSIONES: Se debe promover la estandarización de procedimientos que permitan comparar resultados entre estudios y monitorear alteraciones en una población. Estos datos contribuyen adecuación de las estrategias de monitoreo y promoción de la actividad física.

Descriptores: Indicadores de Calidad de la Atención de Salud. Atención Primaria de Salud. Gestión en Salud. Administración Municipal

INTRODUCTION

Evaluations have existed since early civilization.⁶ Their application in public programs increased with the Second World War, due to a need to control the spending of scarce national resources. In Brazil, the field began to be developed in the 1980s.¹⁹

It is a challenge to transform the concept of quality evaluation into criteria, indicators and standards that assure validity.¹⁰ Sander¹⁷ contributed considerably with evaluation studies applied to the quality of management. He utilized the historical retrospective of administration theory and his influence in education in Latin America to highlight the constructs of administration based on efficiency, efficacy, effectiveness and relevance. These four constructs point to four criteria to evaluate and guide administrative performance. Its theoretical essence is intimately connected to the nature of each construct, which corresponds to the economic, institutional, political and cultural dimensions associated with the respective criteria. Scriven¹⁸ described the concept of an object's quality as dimensions of value and merit: an object has quality when it has value and merit, be it a system, a process or a program. It has value when its resources are well applied to meet the needs of stakeholders; and it has merit when it performs well what it intends to do. An object can have merit and not have value when the manager does not meet the needs of the population of interest. Therefore, all objects without merit do not have value, since resources should not have been spent with efficacy and efficiency in order to meet the needs of interested parties.

The combination of the proposals by Sander¹⁷ and Scriven¹⁸ may explain the concept of quality, considering value and merit. These conditions are sufficient for systems, processes, projects and programs to exhibit quality, and the criteria of efficiency, efficacy, effectiveness and relevance are necessary to exhibit quality.⁷

One of the challenges to evaluate management in health in a deterministic fashion, instead of probabilistic, is finding techniques that allow for simultaneous analysis of all the aspects involved.

Data envelopment analysis (DEA) is a widely used method in the study of productivity and technical efficiency or organizations that utilize multiple inputs and generate multiple products. It allows for the identification of improved practices through the empirical identification of frontiers using linear programming. In recent years, there has been a substantial increase in international publications using DEA for health evaluations.^{11,14,15,16,20} In Brazil, articles report the use of DEA in economic studies of education and health.^4,8,9,13

The quality of municipal management can be expressed by the ability of the manager to take actions that reduce the risk of disease and other harms and that make access universal and equitable for all municipal residents to the actions and services necessary for health promotion, prevention and rehabilitation. This study sought to develop a composite indicator to evaluate the quality of municipal management in primary health care.

METHODS

A methodological study was undertaken to develop an evaluation model focused on the management of the health system through use of DEA and indicators for efficiency, efficacy, effectiveness and relevance, consolidated into a composite indicator of quality. The model was tested in small municipalities (from 10 to 50 thousand residents) of Santa Catarina State (Southern Brazil) in the year 2006.

The evaluation matrix considered two dimensions: the management of actions promoting access (intersectoral activities; popular participation; human resources and infrastructure) and the type of actions (external; internal); and the management of actions for service provision (child; adolescent; adult; older adults) and the type of actions (promotion and prevention; diagnosis and treatment). This resulted in 12 sub-areas of analysis, for which indicators were selected that reflected the criteria used for quality: relevance, effectiveness, efficacy and efficiency. The selection was performed based on a search of the literature, workshops to create consensus among specialists and technical experts from the Santa Catarina State Secretary of Health.

Indicators and measures were not selected for the criteria of efficiency in actions of service provision, since this is a fundamental preoccupation of management and not service provision. Indicators were selected for external actions of management for intersectoral activities and popular participation. Eight indicators for each type of action (internal and measured for the four criteria) were selected for the management of human resources and infrastructure (Table 1).

The model evaluated the relative quality of municipal management in three stages: in the first, measures of relevance and effectiveness of management were used to generate a measure of value; in the second, measures of efficacy and efficiency were used to generate measure of merit; and in the third, the measures of value and merit were used to generate a measure of quality. A mathematical algorithm for linear programming was developed to evaluate the performance of the municipal manager compared to the performance of other managers through use of the function-impact performance of the most relevant factors, from the view of the manager. The mathematical algorithm produced variable relative measures in accordance with the manager evaluated. The algorithm used for the aggregation of the measures was applied with Lingo© software (Lindo Systems, Chicago, USA).

The resulting curves for excellent performance were defined by the best combinations of value and merit. The curves were denominated "frontiers of observed quality" and considered as excellent the quality observed in municipalities represented by points on the frontier and considered other municipalities as inefficient. The algorithm calculated the distance from each point to the frontier of observed quality, and associated a measure that was inversely proportional to the distance for each point, in order to obtain a monotonic and increasing measure for quality over the interval [0,1]. The same principal was assumed in the aggregation of measures of relevance and effectiveness to generate the measure of value, as well as in the aggregation of measures of efficacy and efficiency to generate the measure of merit. Value rankings were produced to position the municipalities from the sample ("good" management for the 25% best positioned, "poor" for the 25% worst positioned and "normal" for other municipalities between positions from 25% and 75%).

An additive model was utilized for the analysis.⁵ In the algorithm developed, a municipality was designated as Munº whose management was simultaneously evaluated for various criteria of performance (Cj,J=1,2,...,J) and the municipalities were associated with measures(Mj,J=1,2,...,J) that were monotonic and increasing over the interval [0,1].

Observed values of 0 < m_j< 1 were considered for the measures (Mj,J=1,2,...,J). The management of Munº can be evaluated in an absolute and relative manner with these values. In the first case, the standards for excellent performance are recognized (mj*,J=1,2,...,J j), and management is considered efficient when (mj*=mj*j); in other cases, management is considered inefficient. In the second case, excellent standards m_j* do not exist or are not recognized, and the management of Munº is evaluated relative to the management of municipalities similar to Munⁿ, (n=1,2,...,N) considering the combination of the measures (Mj,J=1,2,...,J).

In the mathematical models that utilize DEA to verify of the management of a municipality Munº is efficient of inefficient, it is assumes that the measures M1,...,Mk assume values m1,...mk such that:

Always that:

Therefore, the problem in verifying if there exists a Munⁿ better than Munº can be resolved by verifying of there exist numbers such that:

To verify the existence of such z_n, the linear programming problem is solved

Sk > 0, k = 1, 2,..., K e Zn > 0, n = 0, 1, 2,..., N;

Which maximizes

Such that

When S* > 0, the manager of Munº is inefficient, since s*_k > 0 for some k, the observed data demonstrate the possibility that managers can increase the performance of the municipality in one of the criteria without harming performance in another criteria. On the other hand, when S* = 0, management can be considered excellent, since s*_k= 0 for all k indicates that the managers cannot increase the performance of this organization in any of the criteria without harming performance in another criteria. The 55 indicators were aggregated by the algorithm developed (Figure).

RESULTS

The number of reports generated by applying the mathematical algorithm depended on the desired characteristics of the evaluation. Reports were generated for each aggregation of measures in each type of action, focus and dimension, in addition to partial reports for performance in relevance, effectiveness, efficacy, efficiency, value and merit. Table 2 presents measures of relevance, effectiveness, efficacy, efficiency, value and merit for each of the types of actions in each focus, as well as the quality measures of municipal management in primary health care, its dimensions and their respective focus, for each municipality in the sample.

The values were presented for each evaluation criteria (relevance, effectiveness, efficacy and efficiency) relative to all the other municipalities. The criteria of value resulted from aggregating the measures of effectiveness and relevance; the criteria of merit resulted from aggregating the measures of efficiency and efficacy. Value = 1 indicated that the municipality was in the observed frontier for that measure (efficiency), and the smaller the value, the farther the municipality was from the ideal value of that measure. The quality of actions was expressed as the performance resulting from aggregating value and merit for each action taken - external action, internal action, promotion and prevention, diagnosis and treatment - for each of the four focuses of the two dimensions. Quality indicates the performance resulting from the aggregation of the four focuses of each dimension, allowing for measurement of management performance in actions promoting access and actions for the provision of health services. Aggregation of the latter two measures resulted in the measure of performance for the management of primary health care.

The measures (1.0), (0.0) and (0.5) indicated levels of quality for the municipal management of primary health care, according to the standard of quality adopted for study. The quality of a municipality's management was considered: (i) Good, when represented by the measure (1.0); (ii) Poor for the measure (0.0), and (iii) Normal for the measure (0.5). Measures with (*) indicate that a ranking was not produced for this management action, since it was not included in the analysis (Table 3).

The classification was proposed as a summary alternative for the rankings. In the example presented, the municipality can have good management for intersectoral activities and infrastructure and normal management for popular participation and human resources in the dimension of management for actions promoting access to health services. It also had good management for children and poor management for adolescents, adults and older adults. In comparison with the 35 other municipalities evaluated, this municipality had a normal quality of municipal management in primary health care. Reports were generated for each municipality analyzed.

Excellent values (1.000) were obtained by five municipalities in the quality of management for actions promoting access (Q_SMS) and by eight municipalities in the management of action for provisioning services (Q_PROV); one municipality obtained a value (1.000) for the quality of management in primary health care (Q_GABS) and tem (28%) obtained values above 0.900. The lowest value obtained was for human resources (0.219 in municipality 31). The lowest mean for values was observed for popular participation and the highest for children (Table 4).

DISCUSSION

This is one of few studies that focuses on the evaluation of the performance and provision of primary health care. The proposed model uses a mathematical algorithm as an alternative to construct a composite indicator that allows for identification of potential areas for improvement in the overall performance of municipal management in primary health care (Sint_G_ABS). The evaluation model and its results point to large differences in the quality of services performed in the health sector. Studies² also applied DEA to measure the technical efficiency of 351 primary health care units in Portugal, divided into 12 geographic regions, and concluded that evidence exists for large variations in the access to health services, in technical efficiency and in the quality of services provided.

Some of the composite indicators presented low minimum values: popular participation (0.250) and human resources (0.219), and four municipalities presented excellent values for more than one composite indicator evaluated (Table 4). This suggests that management in health in these municipalities occurs through the prioritization of some types of actions in detriment to others, and this prioritization varies within the municipalities analyzed.

Evaluation studies of management efficiency in health care through the services of general surgery, ophthalmology and orthopedic trauma surgery in 22 Valencian hospitals³ (Eastern Spain) utilized a non-parametric DEA approach and discriminant analyses to show the effectiveness of the DEA model to classify health services as efficient or inefficient. The study adopted a scale of 0 to 1, with a value of 1 considered efficient and a value less than 1 as inefficient. The same procedure was adopted in the model presented here and resulted in five municipalities with an excellent value (1) in the management of actions promoting access, eight in the management of actions of service provision and only one municipality when considering the two dimensions simultaneously. The result suggests that actions of service provisioning continue to be prioritized in the municipalities analyzed, which does not necessarily signify a management that guarantees improved services. All other occurrences where management actions promoting access had a composite indicator of 1, the value generated for management actions of service provisioning are below the third quartile. The same can be observed in the opposite direction, except for municipality 20 (efficient); all municipalities with a composite indicator of 1 in actions of service provisioning have measures below the third quartile of management actions for access. This indicates that the manager makes decisions between prioritizing actions that guarantee access and/or service provisioning, and the situation results where only one municipality has a value of one for the composite quality indicator.

Another possibility of the model developed by this study is the identification of inefficient areas relative to other municipalities, similar to other studies.⁹ The distance between the values for each municipality relative to the reference point for excellence allows for verification of where and how much the municipality could improve its situation relative to other similar municipalities. For the 36 municipalities, the distances to the frontier were 8,472 (0.235/municipality) for actions promoting access, 3,382 (0.094/municipality) for actions of service provision and 5.92 (0.164/municipality) for the management of primary care. Again the analysis identified prioritization of actions of service provision by municipalities.

The use of the model resulted in 22.5% of composite indicators that measured the quality of items evaluated with values above 0.900, a level similar to the evaluation of technical efficiency of 89 health centers in Ghana.¹

The DEA approach was utilized in the evaluation of productive efficiency in Brazilian hospitals^4,12 and of public spending in health.⁸ This indicates the potential of this approach in health evaluations, but also demonstrates that greater importance has been given to the identification of technical inefficiency.

The results indicated one municipality with an excellent quality of management in primary health care; 27.8% of municipalities presented values > 0.900 in performance and none presented performance <0.678. The item with the best average performance was "children" with an average of 0.870, traditionally a priority in all health systems, and the worst performance was for popular participation with an average of 0.553, which appears to be less prioritized in small municipalities.

Two other types of studies could be conducted: one including factors not controlled by municipal managers (social, economic and environmental factors), which impact the results of primary health care and affect municipal management; and the other can utilize more complex DEA models (invariant models, with two phases), which allow for more robust comparison between the management of primary health care in municipalities with different characteristics.

The study is supported by other international applications and publications in the field. The study allowed for evaluation of management quality in primary health care in municipalities with small populations, through use of DEA approach. The evaluation used multiple performance indicators also utilized by the Ministry of Health and aggregated them by dimensions, types of actions and focus of activities in the primary health care of municipalities. The results of this evaluation process were grouped according to multiple criteria of performance that reflect the capacity of the municipal health manager to allocate resources to meet the needs of health promotion, prevention and rehabilitation in their municipalities.

REFERENCES

Received: 6/17/2011

Approved: 1/30/2012

Article based on the doctoral thesis by Scaratti D presented to the Universidade Federal de Santa Catarina in 2008.

The authors declare no conflict of interests.

• 1. Akazili J, Adjuik M, Jehu-Appiah C, Zere E. Using data envelopment analysis to measure the extent of technical efficiency of public health centres in Ghana. BMC Int Health Hum Rights. 2008;8:11. DOI:10.1186/1472-698X-8-11
• 2. Amado CAEF, Santos SP. Challenges for performance assessment and improvement in primary health care: the case of the Portuguese health centres. Health Policy. 2009;91(1):43-56. DOI:10.1016/j.healthpol.2008.11.008
• 3. Caballer-Tarazona M, Moya-Clemente I, Vivas-Consuelo D, Barrachina-Martínez I. A model to measure the efficiency of hospital performance. Math Comput Model 2010;52:1095-102. DOI:10.1016/j.mcm.2010.03.006
• 4. Cesconetto A, Lapa JS, Calvo MCM. Avaliação da eficiência produtiva de hospitais do SUS de Santa Catarina, Brasil. Cad Saude Publica 2008;24(10):2407-17. DOI:10.1590/S0102-311X2008001000021
• 5. Charnes A, Cooper WW, Golany B, Seyford L. Foundations of data evelopment analysis for Pareto-Koopmans efficient empirical production functions. J Econom 1985;30(1/2):91-107. DOI:10.1016/0304-4076(85)90133-2
• 6. Contandriopoulus AP, Champagne F, Denis JL, Pineault R. A avaliação da área da saúde: conceitos e métodos. In: Harz ZMA. Avaliação em saúde: dos modelos conceituais à pratica na análise da implantação de programas. Rio de Janeiro: Fiocruz; 1997. p.29-47.
• 7. Davok DF. Qualidade em Educação. Aval (Campinas). 2007;12(3):505-13. DOI:10.1590/S1414-40772007000300007
• 8. Faria FP, Jannuzzi PM, Silva SJ. Eficiência dos gastos municipais em saúde e educação: uma investigação através da análise envoltória no estado do Rio de Janeiro. Rev Adm Publica. 2008;42(1):155-77. DOI:10.1590/S0034-76122008000100008
• 9. Gonçalves AC, Noronha CP, Lins MPE, Almeida RMVR. Análise Envoltória de Dados na avaliação de hospitais públicos nas capitais brasileiras. Rev Saude Publica 2007;41(3):427-35. DOI:10.1590/S0034-89102006005000023
• 10. Hartz ZMA. Avaliação em Saúde: dos modelos conceituais à prática na análise da implantação de programas. Rio de Janeiro: Fiocruz; 1997.
• 11. Kirigia JM. Emrouznejad A, Cassoma B, Asbu EZ, Barry S. A performance assessment method for hospitals: the case of municipal hospitals in Angola. J Med Syst 2008;32(6):509-19. DOI:10.1007/s10916-008-9157-5
• 12. Lins ME, Lobo MSC, Silva ACM, Fiszman R, Ribeiro VJP. O uso da Análise Envoltória de Dados (DEA) para avaliação de hospitais universitários brasileiros. Cienc Saude Coletiva 2007;12(4):985-98. DOI:10.1590/S1413-81232007000400020
• 13. Lobo MSC, Lins MPE, Silva ACM, Fiszman R. Avaliação de desempenho e integração docente-assistencial nos hospitais universitários. Rev Saude Publica. 2010;44(4):581-90. DOI:10.1590/S0034-89102010000400001
• 14. Murray CJ, Lopez AD, Wibulpolprasert S. Monitoring global health: time for new solutions. BMJ. 2004;329(7474):1096-100. DOI:10.1136/bmj.329.7474.1096
• 15. Pinillos M, Antoñanzas F. La Atención Primaria de Salud: descentralización y eficiencia. Gac Sanit. 2002;16(5):401-7.
• 16. Retzlaff-Roberts D, Chang CF, Rubin RM. Technical efficiency in the use of health care resources: a comparison of OECD countries. Health Policy 2004;69(1):55-72. DOI:10.1016/j.healthpol.2003.12.002
• 17. Sander B. Educational management in Latin America: construction and reconstruction of knowledge. Washington: Organization of Amer State; 1996.
• 18. Scriven M. Evaluation Thesaurus. 4. ed. Thousand Oaks: Sage; 1991.
• 19. Uchimura KY, Bosi MLM. Qualidade e subjetividade na avaliação de programas e serviços em saúde. Cad Saude Publica 2002;18(6):1561-9. DOI:10.1590/S0102-311X2002000600009
• 20. Zavras AI, Tsakos G, Economou C, Kyriopoulos J. Using DEA to evaluate efficiency and formulate policy within a Greek national primary health care network. Data Envelopment Analysis. J Med Syst. 2002;26(4):285-92. DOI:10.1023/A:1015860318972

Publication Dates

• Publication in this collection
  24 Apr 2012
• Date of issue
  June 2012

History

• Received
  17 June 2011
• Accepted
  30 Jan 2012