Record of hospitalizations for ambulatory care sensitive conditions: validation of the hospital information system

Abstracts

OBJECTIVE:

to estimate the sensitivity, specificity and positive and negative predictive values of the Unified Health System's Hospital Information System for the appropriate recording of hospitalizations for ambulatory care-sensitive conditions.

METHOD:

the hospital information system records for conditions which are sensitive to ambulatory care, and for those which are not, were considered for analysis, taking the medical records as the gold standard. Through simple random sampling, a sample of 816 medical records was defined and selected by means of a list of random numbers using the Statistical Package for Social Sciences.

RESULT:

the sensitivity was 81.89%, specificity was 95.19%, the positive predictive value was 77.61% and the negative predictive value was 96.27%. In the study setting, the Hospital Information System (SIH) was more specific than sensitive, with nearly 20% of care sensitive conditions not detected.

CONCLUSION:

there are no validation studies in Brazil of the Hospital Information System records for the hospitalizations which are sensitive to primary health care. These results are relevant when one considers that this system is one of the bases for assessment of the effectiveness of primary health care.

Primary Health Care; Validation Studies; Hospitalization

OBJETIVO:

estimar a sensibilidade, a especificidade e os valores preditivos positivo e negativo do Sistema de Informação Hospitalar do Sistema Único de Saúde, para o registro adequado de internações por condições sensíveis à atenção primária.

MÉTODO:

foram considerados para análise os diagnósticos por condições sensíveis e condições não sensíveis à atenção primária, tendo o prontuário como padrão-ouro. Por meio de amostragem aleatória simples, foi definida amostra de 816 prontuários, selecionados por lista de números randômicos, utilizando-se o Statistical Package for Social Sciences.

RESULTADO:

a sensibilidade foi de 81,89%, a especificidade de 95,19%, o valor preditivo positivo de 77,61% e o valor preditivo negativo de 96,27%. No cenário do estudo, o Sistema de Informações Hospitalares foi mais específico do que sensível, com quase 20% de internações sensíveis não detectadas.

CONCLUSÃO:

não existem estudos no Brasil para validação do Sistema de Informação Hospitalar usando as internações sensíveis à atenção primária. Estes resultados são relevantes quando se considera que esse sistema é uma das bases de avaliação da efetividade da atenção primária à saúde.

Atenção Primária à Saúde; Estudos de Validação; Hospitalização

OBJETIVO:

estimar la sensibilidad, la especificidad y los valores predictivos positivo y negativo del Sistema de Información Hospitalaria del Sistema Único de Salud para el registro adecuado de Internaciones por Condiciones Sensibles a la Atención Primaria.

MÉTODO:

fueron considerados para el análisis los diagnósticos por condiciones sensibles y condiciones no sensibles a la atención primaria, teniendo la ficha médica como estándar oro. Por medio de muestreo aleatorio simple fue definida una muestra de 816 fichas, seleccionadas con una lista de números aleatorios utilizando el programa Statistical Package for Social Sciences.

RESULTADO:

la sensibilidad fue de 81,89%, la especificidad de 95,19, el valor predictivo positivo de 77,61% y el valor predictivo negativo de 96,27%. En el escenario del estudio, el Sistema de Informaciones Hospitalarios fue más específico que sensible, con casi 20% de Internaciones Sensibles no detectadas.

CONCLUSIÓN:

no existen estudios en Brasil para validación del Sistema de Información Hospitalario usando las internaciones sensibles a la atención primaria. Estos resultados son relevantes cuando se considera que ese sistema es una de las bases de evaluación de la efectividad de la atención primaria a la salud.

Atención Primaria de Salud; Estudios de Validación; Hospitalización

Introduction

The implantation of the Unified Health System (SUS) has been marked by many advances; however, there are still challenges to be overcome if its principles and directives are to be truly realized. Among the challenges, one can cite the consolidation of a quality Primary Health Care (PHC) which, by ensuring access, is effectively constituted as the population's first choice for the resolution of its health problems.

From the perspective of overcoming this and other challenges, the forming of health care networks is being discussed in Brazil, the PHC having a fundamental role as the arranger of flows and counter-flows of service users within the health system, with the exception of emergency cases^{( 1 )}.

In relation to these flows, in many situations in which the PHC does not resolve the cases, the demand for hospitalization probably includes a proportion of cases with diagnoses which are sensitive to this level of care and which, therefore, could have been resolved there^{( 2 )}.

In various countries, the evaluation of access to and the effectiveness of the PHC is obtained through the analysis of the Hospitalizations for Ambulatory Care-Sensitive Conditions^{( 3 - 5 )}. This is an indicator which originated in the United States of America (USA)^{( 3 )} since when adaptations have been made, so as to cover local characteristics, depending on the context to be analyzed. This may be observed in the study where the American list was applied for validation in the European context^{( 6 )}.

In Spain, among the strategies proposed to improve the utility of this indicator, are the search for its greater specificity and analyses by the municipalities^{( 5 )}. In this regard, the prudent use of this indicator can help to increase the PHC's resolutive capacity, by identifying areas which are clearly possible to improve, in particular by evidencing health problems which need better monitoring and better coordination between the assistential levels^{( 7 - 8 )}.

In Brazil, this indicator was initially adopted by some State Health Departments; however, a more systematized discussion was started in 2007, by a group made up of professionals from the Ministry of Health (MS), managers, researchers and specialists in this issue. As a product of this work group, the Brazilian list of Hospitalizations for Ambulatory Care-Sensitive Conditions (HACSC) was developed, having as its conceptual mark a model proposed by authors from other countries where for some health conditions, opportune and good quality primary care can avoid hospitalization or reduce its frequency^{( 9 )}.

The list was published by the MS, through the Ordinance SAS n. 221 of April 17th 2008, covering 19 groups of diagnoses, totaling 120 ICD categories - 10 (with three digits) and 15 subcategories (with four digits), to be used as an assessment instrument for the evaluation of primary care and/or for hospital use, potentially being applied to evaluate the performance of the health system in the national, state and municipal ambits^{( 10 )}.

Bearing in mind that the measuring of the HACSC will be accomplished prioritarily through the analysis of the hospitalizations of the SUS, recorded on the Hospital Information System (SIH/SUS) - implanted by the MS^{( 11 )} - by means of the analysis of the Authorizations for Hospitalization (AIH), it is justifiable to investigate the degree to which the information found in the AIH reproduces that recorded in the patients' medical records, as incorrect diagnoses present in the SIH-SUS lead to mistaken evaluations of the primary care. It is emphasized that at the time of writing, there are no studies published in Brazil validating the system considering the indicator.

The SIH/SUS is a system which covers Brazil, fed by the AIH, which - among other actions - makes it possible to store data from hospitalizations; to present and process monthly the AIH of the public health establishments, private health establishments, and those contracted; to make available to the managers reports with information for payment for service providers and to monitor the performance of hospitals regarding the goals established in the contracts between manager and hospitals^{( 11 )}. Although this system was conceived, primarily, for the management of the hospital system, its data can contribute in an important way for the investigation of, or construction of, the profile of hospital morbidity and mortality and for the assessment of the quality of the health care offered to a population^{( 11 )}.

It is asked if the SIH/SUS is a valid source of information for assessing primary care based on the HACSC records. This study aimed to estimate the sensitivity, specificity and the predictive values of the SIH-SUS for the recording of the HACSC set as an aggregate indicator.

Methodology

The study was undertaken in a general hospital in the city of São Paulo, under the management of a Social Organization (SO), whose financial resources are passed on by means of a management contract with the State Health Department, for the achieving of agreed goals. It has 266 beds distributed among the following specialities: general practice, surgery, pediatrics, orthopedics and obstetrics. This hospital has a teaching and research center as well as medical residency.

An epidemiological method of validation of criteria was undertaken, considering for analysis the diagnoses covered in the Brazilian HACSC List^{( 10 )} and the other causes of hospitalization, with the medical records as the gold standard.

The sample was based on the hospitalized population in 2008, totaling 10,616 medical records and had as its parameters: p = 0.50 for sensitivity and specificity; alpha of 5% and sampling error of 0.05. Confidence Intervals (CI) were calculated at 95% for all the estimates.

In the sample calculation the following formula for a finite population was used:

Where:

n = sample number

N= finite population = 10,616

z = 1.96 (alpha of 5%)

e = sampling error - assumed 0.05

p = proportion expected, assumed as 0.50.

1-p = p complement = 0.50.

The sample number (n) was 371 medical notes and considering 10% losses, the final sample n was 408 medical records for the calculation of sensitivity. An equal number of medical records was estimated for the calculation of specificity, 816 records having been evaluated.

The 816 records were selected randomly, using the Statistical Package for Social Sciences (SPSS) 11^(r) software, for generation of the list of random numbers, through the AIH number.

For the identification of the medical records and their respective pair in the SIH/SUS, the AIH number and the variables 'sex' and 'date of birth' were considered, manually. When there was agreement between all the identification variables, the medical records were selected for study along with their corresponding pair in the AIH. A database was developed using the Access application for entering the data from the AIH and the medical records.

The data was collected by a single observer, the project's principal researcher. The information collected was the complete diagnosis on discharge written by the physician in the discharge report, and the ICD 10 when present. After the inputting of the information in the database, an assessment of the quality of the information from the medical records was made, with a view to check if the given ICD 10 corresponded to the diagnosis established by the physician. The verification of the ICD 10 in relation to the written diagnosis was undertaken by two of the project's researchers, item by item. In some situations, when there was no correspondence, advice from a professional with greater experience in coding of underlying causes was necessary, so as to preserve the diagnosis written by the physician, avoiding an excessive number of losses in the study and decoding the diagnosis recorded by the physician without altering it.

The following situations were defined as losses in the study: disagreement of SEX considering the AIH/medical records; disagreement of DATE OF BIRTH considering the AIH/medical records; medical records not found; principal diagnosis queried; principal diagnosis written in pencil; more than one principal diagnosis, different from each other; principal diagnosis not identified in the ICD 10; diagnosis on admission assumed to be the principal diagnosis; only the ICD as principal diagnosis; diagnosis registered by an acronym not identified at the time of collection.

Of the 816 diagnoses/ICD collected in the medical notes, after checking of the ICD/ diagnosis written in full, it was necessary to adjust the ICD for 203 records (26.10%) and in 53 records (7.23%) without ICD, the corresponding ICD was added next to the diagnosis written in full.

It is emphasized that both the adjustment made in the ICD and the ICD designation next to the diagnosis written in full were made respecting, rigorously, the diagnosis written in full established in the medical records by the physician, considered to be the gold standard.

The Sensitivity, Specificity, Positive Predictive Value and Negative Predictive Value were estimated, and their respective 95% Confidence Intervals were calculated.

The calculation of sensitivity was made considering the causes defined as HACSC by the medical records which was estimated by the proportion of true positives detected by the SIH/SUS among all the HACSC defined by the medical records. The complementary estimate for sensitivity represents the false negatives by the SIH/SUS.

The calculation of the specificity was made considering the causes defined as non-HACSC by the medical records, and was estimated by the proportion of true negatives detected by the SIH/SUS, among all the non-HACSC medical records. The complementary estimate for specificity represents the false positives by the SIH/SUS.

The positive predictive value (PPV) was estimated by the proportion of positive diagnoses of the SIH/SUS among all the positive diagnoses.

The negative predictive value (NPV) was estimated by the proportion of negative diagnoses of the SIH-SUS among all the negative diagnoses.

The study was approved by the Research Ethics Committees of the University of São Paulo School of Nursing (process nº 860/2009/CEP - EEUSP) and of the Hospital (CEP Record: 01/1/ P) meeting the requirements of the National Health Council's Resolution n. 196, of 10/10/96(12).

Results

There were 65 losses, which corresponds to 8.0% of the total, with the most frequent causes being: principal diagnosis queried (26.1%); disagreement regarding sex in the medical notes and the AIH (20.0%); principal diagnosis not identified in the ICD 10 and the ICD alone as the principal diagnosis (both with 13.8%); and disagreement in relation to the date of birth in the medical notes and in the AIH (9.2%). 751 records were analyzed (AIH/medical records pair), HACSC and non-HACSC, compared with the Brazilian List, considering the diagnosis recorded in the AIH and in the medical records. The gold standard (medical records) identified 127 hospitalizations as HACSC (17% - CI 95%: 13.1%-20.6%) and 624 as non-HACSC. In 104 records, the diagnosis was HACSC both in the AIH and in the medical notes, while in 594 records, the diagnosis was NON-HACSC in the AIH and NON-HACSC in the medical notes (Table 1).

Table 1
Classification of the diagnoses regarding HACSC and NON-HACSC in the AIH and in the medical records. São Paulo, Brazil, 2008

Result		Medical records		Total
		HACSC	Non-HACSC
AIH
	HACSC	104	30	134
	NON-HACSC	23	594	617
Total		127	624	751

• Source: SIH/SUS and medical records

The sensitivity of the SIH/SUS, with the medical records held as the gold standard, was 81.9% (CI 95%: 75.2%-88.6%), the specificity was 95.2% (CI 95%: 93.5%-96.9%), the positive predictive value was 77.6% (CI 95%: 70.5%-84.7%) and the negative predictive value was 96.3% (CI 95%: 94.8%-97.8%).

With sensitivity of 81.9%, the SIH/SUS fails to capture about 18% of the true occurrences of HACSC. Regarding the probability of predicting a diagnosis by HACSC (PPV), the result was median because of being dependent on the frequency of hospitalizations by HACSC in the area studied, which is 17%.

Discussion

The SIH/SUS has the largest database, in Brazil, of data available on hospitalizations, which has been used as much for the undertaking of epidemiological studies and evaluation as for guiding the adoption of steps in public health management, including serving as a base for the transference of financial resources. This study's undertaking brings a contribution to these two fields in verifying the proportion of valid information between the medical notes and the AIH, estimating the degree of accuracy of the SIH and the possibility of its use as a basis for the production of indicators.

The greater problems of validity for the SIH/SUS data are related to the hospitalization diagnosis, because of the unreliable quality of the information in the patient's medical records; of the problems inherent to the coding of the diagnosis by the ICD and to the frauds committed to increase the financial reimbursement of the health institutions contracted^{( 13 )}.

Agreeing with the authors above this study reveals problems related as much to the hospitalization diagnosis as to the coding of the diagnosis by ICD (data not presented). The problems with the medical records' diagnosis constituted the study's highest percentage of losses, because there were failures in the note-taking, such as for example the querying of the principal diagnosis or its non-identification when paired with the ICD 10. It is known, further, that because of the system's economic bias there is the possibility of distortions of classification in the recording of the causes of hospitalization. The hospital studied, whose management is by management contract with the State Health Department, does not have its financial resources coupled to the individual procedures recorded, but rather to the set of agreed-upon goals, which could minimize classification bias, without necessarily avoiding them. This study opens the possibility for future research which could, in addition to the validation of the SIH-SUS, quantify the procedures' costs and which could describe the possible differential errors of classification in relation to these costs.

A study involving applications of SIH/SUS data in Collective Health located 76 studies. Of these, only 3.9% dealt with the quality of the SIH/SUS information, while 34.2% of the studies dealt with the description of the standard of hospital morbidity/mortality and of the medical assistance given^{( 13 )}.

This fact deserves to be highlighted, as prior to using a source of information, it is necessary to know how valid it is. Such a precaution takes on greater importance, considering that one is dealing with a database relevant for, among other aspects, guiding conduct and supporting decision-making, including that related to financing.

It follows that the quality of information must be the object of study so as to reveal the inconsistencies, so that adjustments may be made, for the instances responsible, conferring greater validity in the use of the data. In this regard, it falls to the hospital's technical management to ensure appropriate training for the professionals, as well as adopting quality control measures, including undertaking audits on the quality of the records in the medical records and their corresponding records in the AIH.

The SIH/SUS presented a higher probability of making accurate records of non-HACSC diagnoses in comparison with the medical records, than of capturing these diagnoses when present. That is, the present study demonstrates that the system was more specific than sensitive in the hospital studied.

One study undertaken to estimate the sensitivity, specificity, predictive values and the agreement of the Information System on Maternal Mortality (SIM), in comparison to the investigation of maternal death revealed a sensitivity of 75%^{( 14 )}. In the search for the explanation for the fact that SIM had not recorded all the maternal deaths declared, various hypotheses were raised, such as coding errors and the selection of underlying cause, or typing errors^{( 14 )}.

One other study, using a distinct methodology - without the objective of validation - to verify the main problems resulting from the comparative analysis between the deaths recorded in the SIH/SUS and those recorded in the SIM, observed that the data had a great similarity considering the two systems^{( 15 )}. However, the same problems were identified mentioned in the validation of the SIM, in relation to diagnosis errors in the AIH: incorrect classification of the illness's code and the low importance given to information on morbidity by the professionals responsible.

The failure to capture 18% of the occurrences of HACSC in this validation clearly indicates the need for the introduction of steps, as much by public administration as by the hospital administrations, for the improvement of their quality, as an accuracy equal to or close to 100% would be expected, given that the system should reflect the hospitalizations.

Conclusion

In Brazil there are no studies of the SIH/SUS regarding its validity for HACSC records; this is the first to validate the SIH/SUS, based in a sample of medical records from a hospital which has its own administrative characteristics. It is believed that the same should be replicated in other hospitals, where they have distinct ways of managing the hospital system. This study's results must not be extrapolated to other settings, as different estimates of sensitivity and specificity could be estimated in different management settings.

In the area of nursing, the results are important in terms of knowledge of an information system's quality, it being emphasized that nurses are professionals who manipulate, use and analyze health information systems and can make up a team for audits in hospitals or, especially, act in the field of primary care, undertaking technical work with a direct relationship with the HACSC.

The results of the validation of the SIH/SUS are extremely relevant when one considers that this system is one of the databases for the evaluation of the effectiveness of primary care. If, for a given scenario, the SIH/SUS should not reflect the hospitalizations in the most accurate way, it should not be used for this purpose, and other, more valid, methods for verification of HACSC must be discussed.

References

• ¹
  Ministério da Saúde (BR). Secretaria Executiva. Departamento de Apoio à Descentralização. Diretrizes operacionais dos pactos pela vida, em defesa do SUS e de gestão . Brasília (DF): Ministério da Saúde; 2006. 69 p.
• ²
  Ministério da Saúde (BR). Portaria GM n. 648, de 28 de março de 2006. Aprova a Política Nacional de Atenção Básica, estabelecendo a revisão de diretrizes e normas para a organização da atenção básica para o Programa Saúde da Família (PSF) e o Programa Agentes Comunitários de Saúde (PACS). In: Ministério da Saúde (BR). Política Nacional de Atenção Básica. Brasília (DF): Ministério da Saúde; 2006. p. 7-51.
• ³
  Bermúdez-Tamayo C, Márquez-Calderón S, Rodríguez del Aguila MM, Perea-Milla LE, Ortiz Espinosa J. Características organizativas de la atención primaria y hospitalización por los principales ambulatory care sensitive conditions. Aten Primária. 2004;33(6):305-11.
• ⁴
  Campos AZ, Theme-Filha MM. Internações por condições sensíveis à atenção primária em Campo Grande, Mato Grosso
do Sul, Brasil, 2000 a 2009. Cad Saúde Pública. 2012;28(5)845-55.
• ⁵
  Ricketts TC, Randolph R, Howard HA, Pathman D, Carey T. Hospitalization rates as indicators of access to primary care. Health Place. 2001;7(1):27-38.
• ⁶
  Caminal Homal J, Starfield B, Sánchez Ruiz E, Casanova Matutano C, Morales M. The role of primary care in preventing ambulatory care sensitive conditions. Eur J Public Health. 2004;14(3):246-51.
• ⁷
  Gervas J, Homar JC. Hospitalizations by ambulatory care sensitive conditions (ACSC) from the general practitioner/family physician's point of view. Rev Esp Salud Publica. 2007;81(1):7-13.
• ⁸
  Nedel FB, Facchini LA, Bastos JL, Martín-Mateo M. Conceptual and methodological aspects in the study of hospitalizations for ambulatory care sensitive conditions. Cienc Saúde Coletiva. 2011;16 Suppl 1:1145-54.
• ⁹
  Alfradique ME, Bonolo PF, Dourado I, Costa-Lima MF, Macinko J, Mendonça CS, et al. Internações por condições sensíveis à atenção primária: a construção da lista brasileira como ferramenta para medir o desempenho do sistema de saúde (Projeto ICSAP Brasil). Cad Saúde Pública. 2009;25(6):1337-49.
• ¹⁰
  Ministério da Saúde (BR). Portaria n. 221, de 17 abril de 2008. Publica em forma do anexo a lista brasileira de internações por condições sensíveis à atenção primária. Diário Oficial da União, Poder Executivo, Brasília. 21 set. 2008. Seção 1:50.
• ¹¹
  Ministério da Saúde (BR). Manual técnico operacional do sistema de informações hospitalares. 2011. [Internet]. [acesso 13 jul 2011]. Disponível em: http://w3.datasus.gov.br/sih/Manuais_SIH_SETEMBRO_2010.pdf.
  » Disponível em: http://w3.datasus.gov.br/sih/Manuais_SIH_SETEMBRO_2010.pdf
• ¹²
  Ministério da Saúde (BR). Normas para pesquisa envolvendo seres humanos: Res. CNS n. 196/96 e outras. 2ª ed. Brasília: Ministério da Saúde; 2003. 101 p.
• ¹³
  Bittencourt SA, Camacho LAB, Leal MC. O Sistema de Informação Hospitalar e sua aplicação na saúde coletiva. Cad Saúde Pública. 2006;22(1):19-30.
• ¹⁴
  Mota SMM, Gama SGN, Theme Filha MM. A investigação do óbito de mulher em idade fértil para estimar a mortalidade materna no Município de Belém, Estado do Pará, Brasil. Epidemiol Serv Saúde. 2009;18(1):55-64.
• ¹⁵
  Amaral TCL. Mortalidade hospitalar na rede SUS: espelho dos óbitos ocorridos na população brasileira? [dissertação]. Rio de Janeiro (RJ): Instituto de Medicina Social da Universidade do Estado do Rio de Janeiro; 2002. 89 p.

Publication Dates

• Publication in this collection
  Sept-Oct 2013

History

• Received
  08 Aug 2012
• Accepted
  14 June 2013