Concurrent Validation of Nursing Scores (NEMS and TISS-28) in pediatric intensive care

Abstracts

OBJETIVO: Examinar a validade concorrente do escore Nine Equivalents of Nursing Manpower Use Score (NEMS) em comparação ao Therapeutic Intervention Scoring System-28 (TISS-28) em uma Unidade de Terapia Intensiva Pediátrica (UTIP). MÉTODOS: Estudo de coorte prospectivo observacional, realizado na UTIP de um hospital universitário brasileiro, no período de dois anos, com uma amostra de 816 pacientes. Foram realizadas 7.702 observações de cada um dos escores. RESULTADOS: A média da pontuação máxima do NEMS foi 26,6±9,2 e do TISS-28 21,3±8,2. Em todas as médias, o TISS-28 foi inferior ao NEMS (p<0,001). Houve uma boa correlação entre eles (r²=0,704 para todas as observações). A concordância entre o TISS-28 e o NEMS foi boa, apresentando apenas 6,2% de diferença entre os escores. CONCLUSÃO: Os resultados mostraram boa correlação e concordância entre o TISS-28 e o NEMS, permitindo validar o NEMS nessa população de pacientes pediátricos.

Unidades de terapia intensiva; Enfermagem; Enfermagem pediátrica; Indicadores; Enfermagem prática

ORIGINAL ARTICLE

Concurrent validation of nursing scores (NEMS and TISS-28) in pediatric intensive care

Simone Travi Canabarro^I; Kelly Dayane Stochero Velozo^II; Olga Rosária Eidt^II; Jefferson Pedro Piva^II; Pedro Celiny Ramos Garcia^II

^IUniversidade Federal de Ciências da Saúde de Porto Alegre, Porto Alegre, RS, Brazil

^IIPontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre, RS, Brazil

^{Corresponding author}

ABSTRACT

OBJECTIVE: Examine the concurrent validity of the Nine Equivalents of Nursing Manpower Use Score (NEMS) in comparison to the Therapeutic Intervention Scoring System-28 (TISS-28) in a Pediatric Intensive Care Unit (PICU).

METHODS: Prospective observational cohort study conducted in a PICU of a Brazilian university hospital over a period of two years with a sample of 816 patients. A total of 7,702 observations were obtained for each of the scores.

RESULTS: The average maximum score of the NEMS was 26.6±9.2 and for the TISS-28 it was 21.3±8.2. The TISS-28 was lower than the NEMS (p<0.001) for all the averages. A good correlation was observed between them (r2=0.704) for all observations. Agreement between the TISS-28 and the NEMS was good, presenting only a 6.2% difference between the scores.

CONCLUSION: The results show good correlation and agreement between the TISS-28 and the NEMS, enabling the NEMS validation in this population of pediatric patients.

Keywords: Intensive care units; Nursing; Pediatric nursing; Indicators; Practical nursing

Introduction

Intensive Care Units (ICUs) are seen as critical care areas that demand highly specialized professionals, advanced technology and the organization of work processes, which result in increased concern related to costs and operationalization. Therefore, they require proper documentation and a preview of measurable parameters that qualify and quantify care delivery that is essential for children hospitalized in Pediatric Intensive Care Units (PICUs).

Nurses can use tools to identify the severity of patient conditions, therapeutic interventions and the requirements of nursing care in intensive therapy. The use of scores enables the assessment of certain characteristics presented by patients, contributing to decision-making and evidence-based practice. The Therapeutic Intervention Scoring System was originally presented in 1974. Some changes were implemented to it over time and its simplified version is currently the most disseminated of its versions.^(1,2) The Therapeutic Intervention Scoring System-28 (TISS-28), presented in 1996, is composed of 28 items designed to measure the severity of the disease and nursing workload.^(2,3) The creation of the Nine Equivalents of Nursing Use Manpower (NEMS)^(2,4) was based on the TISS-28.

Work processes within the dynamics of ICUs require optimization of time and feasibility of implementation; the NEMS is an agile tool because it presents only nine items as variables.^(1,5,6) The NEMS is also appropriate for the management of nursing professionals who work in intensive therapy and for the evaluation of ICUs.^(7-9)

Studies validating NEMS were implemented with clinical and surgical patients hospitalized in adult ICUs,^(4,10) however, such studies are rare and seldom implemented in pediatric units.⁽⁵⁾ The primary objective of this study is to examine the concurrent validity of the NEMS in comparison to the TISS-28 in a Pediatric Intensive Therapy Unit.

Methods

This is a prospective observational cohort study conducted in a level III Pediatric Intensive Care Unit at São Lucas Hospital, Pontifícia Universidade Católica do Rio Grande do Sul. The study was conducted between October 1^st, 2006 and September 30^th, 2008. The sample was composed of patients aged between 28 days and 18 years old who were hospitalized in the PICU. All the children who remained in the unit for more than eight hours, regardless of the severity of their condition, and for a period of four hours or longer in the case of death, were included. Patients who were readmitted to the PICU after being discharged from other Units were considered new patients.

Sample size was computed based on an average population of 400 patients hospitalized in the PICU per year. A total of 800 pediatric hospitalizations in the PICU were estimated over a period of two years. The sample power was computed with a level of significance set at 5% to detect the main associations of interest. Thus, the sample presents a power of 100% to evaluate the association between a cut off point of 50% of the categories of the NEMS and TISS-28, with mortality estimated to be 6%.

Data were collected from the medical charts of patients hospitalized in the PICU and the instrument was composed of two parts: the first part was composed of therapeutic interventions from the TISS-28 and NEMS(4) adapted for follow-up until discharge or death; the second part addressed socio-demographic data in addition to the Pediatric Risk of Mortality (PRISM).⁽¹¹⁾

The TISS-28's therapeutic interventions include seven categories that correspond to: basic activities, ventilator support, cardiovascular support, renal support, neurological support, metabolic support, and specific interventions. Each of these parameters is composed of items with scores that range from one to eight, totaling 28 measures.⁽³⁾ The NEMS includes nine items: basic monitoring, intravenous medication, mechanical ventilator support, supplementary ventilatory care, single vasoactive medication, multiple vasoactive medication, hemofiltration and/or dialysis techniques, specific intervention in the ICU, and specific interventions outside the ICU.⁽⁴⁾

Data were collected daily by four RNs during the child's entire hospitalization from 12pm to 2pm using the records on the patient's chart concerning the last 24 hours of hospitalization in the Unit. The team of nurses collecting data was previously trained. After this stage, the Kappa test was applied to verify inter-rater agreement. Agreement at 0.85 was obtained, which indicates strong agreement. Collected data were reviewed by the nurse researcher and stored in a database in a Microsoft Office Excel^® spreadsheet to be analyzed later using the Statistical Package for the Social Sciences^® (SPSS), version 17.0.

The results were considered significant when p<0.05. Continuous variables with normal distribution were presented in averages (± standard deviation) and categorical variables in percentages. Continuous variables with non-normal distribution were expressed in medians and interquartile intervals (CI95%). When indicated, the categorical variables were compared using the Chi-square test or Fisher's Exact test, while Student's t test was used to compare the averages.

The population's mortality was reviewed with the Standard Mortality Ratio (SMR) computation, which is based on the PRISM. It is a reliable indicator of severity that was validated by the authors⁽¹²⁾ and used in the institution during the study. SMR corresponds to the ratio between observed and expected mortality and its variation, assessed according to standard deviation, the values of which confirm the hypothesis that observed mortality is equal to the expected when ±1.96. The Area Under Curve Receiver Operating Characteristic (AUROC) was used to assess sensitivity (correct prediction of death) and specificity (correct prediction of survival).

The correlation of results between the two scores, the NEMS and the TISS-28 (continuous variables), was tested using Pearson's linear correlation, analyzing the degree of association between both, and customization was performed through binary logistic regression analysis. To interpret the results of linear correlation,⁽¹³⁾ we considered "r" between 0.0-0.3 to be weak, 0.3-0.6 to be moderate and >0.6 to be strong correlation.

Bland & Altman plotting⁽¹⁴⁾ was used in the analysis of agreement to verify the variation of scores. We considered the analysis from this plotting to be representative of good agreement when more than 95% of the sample was within its limits (± 1.96 standard deviation in relation to the average).⁽¹⁴⁾

The study met the national and international standards concerning ethics in research involving human subjects.

Results

A total of 830 new hospitalizations were observed in the PICU during the study's period. There were 13 admissions concerning newborns aged less than 28 days old that were not included: 12 in the postoperative period of cardiac surgery and one on mechanical ventilation due to bronchiolitis. Hence, 817 admissions were eligible. Data from one patient (0.12%) were lost. A total of 816 hospitalizations composed the sample and generated 7,702 observations for the measures.

The median age was 23.47 (5.7-72.2) months; most were males (56.9%) and remained hospitalized less than seven days (65.4%). A total of 608 patients (74.3%) presented one or more organic dysfunctions during the hospitalization. The most prevalent dysfunctions were respiratory (45.6%), followed by neurological (19.4%) and cardiologic dysfunctions (17.2%).

In regard to their origin, 56.1% came from the study's hospital (surgical center and nursing ward) and 43.9% came from the emergency department or from another hospital; 58% were clinical patients and 46% required mechanical ventilation.

As shown in Table 1, the scores obtained on the TISS-28 during hospitalization ranged from six to 52, with an average of 19.2 ± 7.4 and a median of 18. On the day of the highest score (the maximum TISS-28), the TISS-28 ranged from six to 59, with an average of 21.3±8.2 and a median of 23. For all the averages of observations, TISS-28 was below NEMS (p<0.001). The NEMS scores at admission ranged from six to 48, with an average of 24.7 ± 8.2 and median of 23. The maximum NEMS score ranged from six to 51, with an average of 26.6 ± 9.2 and a median of 25. PRISM had a good performance with expected mortality of 6.9% while the observed rate of mortality was 6.6%. The SMR, ratio of the observed by the predicted mortality was 0.96 (CI95%).

The NEMS and TISS-28 showed good discrimination of mortality when applied at admission [AUROC of 0.71 (CI95% 0.63 - 0.78) and 0.68 (CI95% 0.60 - 0.75), respectively]; and maximum scores [AUROC of 0.80 (CI95% 0.74 - 0.85) and 0.76 (CI95% 0.70 - 0.82), respectively]. Correlation among the indexes was good, both at admission (r2=0.70) and in relation to the maximum score (r2=0.74) (p<0.01).

The 816 studied patients were hospitalized from one to 277 days, with a median of five (three-ten) days, and totaling 7,702 observations. Including all the measures, the TISS-28 ranged from two to 59, with an average of 19.3 ± 6.6 and a median of 19. The NEMS ranged from zero to 51, with an average of 24.3 ± 8.2 and a median of 27.

Comparison between the NEMS and TISS-28 showed that the difference between the scores was 5 ± 4.45 (CI95% 4.9 - 5.1). The limit of agreement for two standard deviations was from -3.9 to +13.9 (Figure 1). The difference between the scores that were larger than two standard deviations (>8.9 DP) was 6.2%.

Good correlation was found between the NEMS and TISS-28 despite the difference between the two. The correlation between the analyzed scores was linear and positive (r=0.825; r2=0.704, p<0.001). When customization was performed using binary logistic regression, the relationship between the two systems was NEMS = 4.25 + (1.04 x TISS-28).

When the sample was stratified, we observed that the difference between the NEMS and TISS-28 persists within a small interval of 3.6 points (2.4 to six), which would not justify a new customization (Table 2).

Discussion

This independent study was conducted in a Pediatric Intensive Care Unit to compare the scores obtained by children and adolescents in the application of the NEMS and TISS-28. The data collected enabled customization for the computation, based on the NEMS, of the score obtained in the TISS-29.

This study presents a limitation due to the fact that data were collected in a single PICU, though it favors the uniformity of data. It is also important to consider that data were collected only once, between the morning and the afternoon. Some studies collect data on three shifts but choose the highest value or the average.⁽¹⁰⁾ Additionally, the TISS-28 and NEMS do not consider the time nurses spend with care provided to the family (assistance and guidance). In this context, other scores such as the Nursing Activities Scores should be verified.⁽¹⁵⁾The median age found in this study was children younger than two years old while most were male. Other studies report a higher percentage at an older age in relation to the age of the sample in intensive care, that is, 44.3 months⁽¹²⁾ and 8.5 years old in international study.⁽¹⁶⁾ Similar results concerning gender were also verified in an epidemiological study conducted in a PICU.⁽¹⁷⁾

Aiming to compare the diagnoses that resulted in hospitalization in a PICU according to organic dysfunction, we verified in the literature differences with a greater proportion of cardio-circulatory dysfunctions (30%) followed by respiratory (27%) and neurological (22%) dysfunctions.⁽¹²⁾ Respiratory dysfunctions predominated in this study.

Considering the progression of patients over the course of their hospitalization in the Pediatric Intensive Care Unit, the scores obtained on both the TISS-28 and the NEMS by the patients who died were always higher than those obtained by the survivors. More severe patients require a greater number of therapeutic interventions, which is also related to a heavier nursing workload.⁽¹⁸⁾ The finding that patients who do not survive obtain higher scores has also been verified in other studies.^(18-21)

When the PRISM was higher than ten, mortality was 25.9%, and when the PRISM was lower than ten, the mortality observed was 3.4%. The averages of the NEMS and TISS-28 for patients who obtained a PRISM>10 was also higher when related to length of hospitalization.

The AUROC was 0.80% for the score obtained on the NEMS. This means that a patient who ends up dying obtains higher scores on the NEMS than a survivor 80% of the time, considering the maximum score obtained on the NEMS. Therefore, as already observed with the TISS-28,⁽²²⁾ the NEMS shows a good ability to discriminate mortality during hospitalization and also when the maximum scores of the indicators are considered.

During clinical progression, we observed that 94.4% of the patients were discharged from the PICU and 6.6% died; this finding is close to the mortality indicators reported in international studies conducted in PICUs.⁽²³⁾ This information differs from studies in the pediatric field conducted in Brazil, which report higher mortality rates in pediatric intensive care.⁽²⁴⁾

We observed that the NEMS and the TISS-28 presented good agreement. A series of changes in the TISS-28 have been proposed in the progressive process of these scores that aims to assess the severity of patients through the therapeutic interventions to which they are subject, in addition to assessing the workload in ICUs.^(2,25) Taking into account that one point on the TISS-28 is equivalent to approximately 10.6 minutes^(2,26) of a nurse's work during his/her shift, these scores are appropriate to discuss work processes in order to adapt resources to the needs of intensive care units.

The NEMS overestimated the value of the TISS-28 in all the studied variables. One of the most important contributions of this study was the finding that when one decreases approximately four to five points in the NEMS's score, one finds a result that is very close to the TISS-28's score. The equation found for the customization [NEMS = 4.25 + (1.04 x TISS-28)] was very similar to that of a study conducted with adults.⁽¹⁰⁾ There are few studies in the pediatric field using scores to study therapeutic interventions.⁽⁵⁾

It was possible to customize the NEMS and TISS-28 scores for a Pediatric Intensive Care Unit. In general, the TISS-28 is conceived as reflecting the nursing workload in a broad range of levels of activity. The NEMS, however, has a more attractive performance, as shown in this study, and contains only nine therapeutic interventions, which demands less time for data collection. The use of the NEMS in PICUs is useful for pediatric intensive care nurses, since it helps to measure the severity of patients' conditions and their nursing care needs, in accordance with Resolution 7/2010, National Agency for Sanitary Vigilance.⁽²⁷⁾

Conclusion

This study enabled the validation of the NEMS in a Pediatric Intensive Care Unit of a University Hospital. We observed that the more therapeutic interventions, the higher the scores obtained and, consequently, the more severe the patient's condition. A good correlation was found between the TISS-28 and the NEMS in this population of pediatric patients, and both presented good discriminatory capacity for mortality and good association with the PRISM. However, the NEMS overestimated the TISS-28 values for all the studied variables, which enabled obtaining a customized computation between the scores.

Acknowledgments

To the National Council for Scientific and Technological Development (CNPq; research grant on productivity A1 for PCR Garcia; research grant on productivity A2 for JP Piva) and to the Coordination for the Improvement of Higher Education Personnel (CAPES; Master's scholarship).

Collaborations

Canabarro ST and Garcia PCR participated in the project's conception, analysis and interpretation of data, redaction, critical review of intellectual content and final approval of the version to be published. Velozo KDS contributed to the analysis and interpretation of data, redaction, critical review of the intellectual content and final approval of the version to be published. Eidt OR participated in the project's conception, critical review of intellectual content and final approval of the version to be published. Piva JP collaborated with data analysis, critical review of intellectual content and final approval of the version to be published.

References

1. Canabarro ST, Velozo KD, Eidt OR, Piva JP, Garcia PC. [Nine Equivalents of Nursing Manpower Use Score (NEMS): a study of its historical process]. Rev Gaúcha Enferm. 2010;31(3):584-90.Portuguese.

2. Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14(2):207.

3. Miranda DR, de Rijk A, Schaufeli W. Simplified Therapeutic Intervention Scoring System: the TISS-28 items--results from a multicenter study. Crit Care Med. 1996;24(1):64-73.

4. Reis Miranda D, Moreno R, Iapichino G. Nine equivalents of nursing manpower use score (NEMS). Intensive Care Med. 1997;23(7):760-5.

5. Monroy JC, Hurtado Pardos B. [Utilization of the nine equivalents of nursing manpower use score (NEMS) in a pediatric intensive care unit]. Enferm Intensiva. 2002;13(3):107-12. Spanish

6. Junger A, Hartmann B, Klasen J, Brenck F, Röhrig R, Hempelmann G. Impact of different sampling strategies on score results of the nine equivalents of nursing manpower use score (NEMS). Methods Inf Med. 2007;46(4):410-5.

7. Lucchini A, Chinello V, Lollo V, De Filippis C, Schena M, Elli S, et al. [The implementation of NEMS and NAS systems to assess the nursing staffing levels in a polyvalent intensive care unit]. Assist Inferm Ric. 2008;27(1):18-26. Italian.

8. Iapichino G, Radrizzani D, Ferla L, Pezzi A, Porta F, Zanforlin G, et al. Description of trends in the course of illness of critically ill patients. Markers of intensive care organization and performance. Intensive Care Med. 2002;28(7):985-9.

9. Haagensen R, Jamtli B, Moen H, Stokland O. [Experiences with scoring systems SAPS II and NEMS for registration of activities in an intensive care unit]. Tidsskr Nor Laegeforen. 2001;121(6):687-90. Norwegian.

10. Rothen HU, Küng V, Ryser DH, Zürcher R, Regli B. Validation of "nine equivalents of nursing manpower use score" on an independent data sample. Intensive Care Med. 1999;25(6):606-11.

11. Pollack MM, Ruttimann UE, Getson PR. Pediatric risk of mortality (PRISM) score. Crit Care Med. 1988;16(11):1110-6.

12. Martha VF, Garcia PC, Piva JP, Einloft PR, Bruno F, Rampon V. [Comparison of two prognostic scores (PRISM and PIM) at a pediatric intensive care unit]. J Pediatr (Rio J). 2005;81(3):259-64. Portuguese.

13. Callegari-Jaques SM. Bioestatística: princípios e aplicações. Porto Alegre: Artmed. 2003. p. 90.

14. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Int J Nurs Stud. 2010;47(8):931-6.

15. Miranda DR, Nap R, de Rijk A, Schaufeli W, Iapichino G; TISS Working Group. Therapeutic Intervention Scoring System. Nursing activities score. Crit Care Med. 2003;31(2):374-82.

16. Muranjan MN, Birajdar SB, Shah HR, Sundaraman P, Tullu MS. Psychological consequences in pediatric intensive care unit survivors: the neglected outcome. Indian Pediatr. 2008;45(2):99-103.

17. Einloft PR, Garcia PC, Piva JP, Bruno F, Kipper DJ, Fiori RM. [A sixteen-year epidemiological profile of a pediatric intensive care unit, Brazil]. Rev Saúde Publica. 2002;36(6):728-33. Portuguese.

18. Balsanelli AP, Zanei SS, Whitaker IY. [Relationships among nursing workload, illness severity, and the survival and length of stay of surgical patients in ICUs]. Acta Paul Enferm. 2006;19(1):16-20. Portuguese.

19. Elias AC, Tiemi M, Cardoso LT, Grion CM. [Application of the therapeutic intervention scoring system (TISS 28) at an intensive care unit to evaluate the severity of the patient]. Rev Latinoam Enferm. 2006;14(3):324-9. Portuguese.

20. Garcia PC, Goncalves LA, Ducci AJ, Toffoleto MC, Ribeiro SC, Padilha KG. [Therapeutic interventions in intensive care units: analysis according to therapeutic intervention scoring system-28 (TISS-28)]. Rev Bras Enferm. 2005;58(2):194-9. Portuguese.

21. Muehler N, Oishi J, Specht M, Rissner F, Reinhart K, Sakr Y. Serial measurement of Therapeutic Intervention Scoring System-28 (TISS-28) in a surgical intensive care unit. J Crit Care. 2010;25(4):620-7.

22. Lefering R, Zart M, Neugebauer EA. Retrospective evaluation of the simplified Therapeutic Intervention Scoring System (TISS-28) in a surgical intensive care unit. Intensive Care Med. 2000;26(12):1794-802.

23. Dominguez TE, Chalom R, Costarino AT Jr. The impact of adverse patient occurrences on hospital costs in the pediatric intensive care unit. Crit Care Med. 2001;29(1):169-74.

24. Lago PM, Piva J, Kipper Dl, Garcia PC, Pretto C, Giongo M, et al. [Life support limitation at three pediatric intensive care units in southern Brazil]. J Pediatr (Rio J). 2005;81:111-7. Portuguese.

25. Kwiecien K, Wujtewicz M, Medrzycka-Dabrowska W. Selected methods of measuring workload among intensive care nursing staff. Int J Occup Med Env. 2012;25(3):209-17.

26. Padilha KG, Sousa RM, Kimura M, Miyadahira AM, da Cruz DA, Vattimo Mde F, et al. Nursing workload in intensive care units: a study using the Therapeutic Intervention Scoring System-28 (TISS-28). Intensive Crit Care Nurs. 2007;23(3):162-9.

27. Ministério da Saúde (BR). Agência Nacional de Vigilância Sanitária. Resolução - RDC nº 7, de 24 de fevereiro de 2010. Dispõe sobre os requisitos mínimos para funcionamento de Unidades de Terapia Intensiva e dá outras providências [Internet]. Brasília; 2010 [citado 2012 outubro 15]. Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/anvisa/2010/res0007_24_02_2010.html

Submitted December 26, 2011

Accepted March 27, 2013

Conflict of interest: no conflicts of interest to declare.

Publication Dates

• Publication in this collection
  27 May 2013
• Date of issue
  2013

History

• Received
  26 Dec 2011
• Accepted
  27 Mar 2013