SURGICAL PLANNING IN IDIOPATHIC SCOLIOSIS: APP USABILITY AND UTILITY

PLANEJAMENTO CIRÚRGICO NA ESCOLIOSE IDIOPÁTICA: USABILIDADE E UTILIDADE DE APP

PLANIFICACIÓN QUIRÚRGICA EN ESCOLIOSIS IDIOPÁTICA: USABILIDAD Y UTILIDAD DEL APP

ABSTRACT

Objective:

To evaluate the usability and usefulness of an application developed for surgical planning of scoliosis correction to measure the time spent on planning. Methods: Experimental, controlled, cross-sectional, and applied study. The application underwent a usability test using the SUS questionnaire, in addition to applying a questionnaire to assess its usefulness using the TAM utility perception model. Thirty-six volunteers were divided into three groups: residents, orthopedists from the Brazilian Society of Orthopedics and Traumatology (SBOT), and spine surgeons. Results: By applying the SUS questionnaire, the application obtained a score of 75.69, representing classification C on the Sauro and Lewis scale, the third best. The application’s utility was considered good after evaluating the positive responses of the TAM model. On average, planning through the application was 20% faster than manual planning. Conclusion: The Scoliplan application proved useful and optimizes surgical planning in AIS, as it establishes a step-by-step standardization supported by the literature (Suk classification); it helps the user with uniform and reliable planning. Level of Evidence III; Diagnostic Studies.

Keywords:
Scoliosis; Adolescent; Planning; Smartphone; Mobile Applications

RESUMO

Objetivo:

Avaliar a usabilidade e utilidade de um aplicativo desenvolvido para planejamento cirúrgico de correção de escoliose, mensurar o tempo dispendido para o planejamento. Métodos: Estudo do tipo experimental, controlado, transversal e aplicado. O aplicativo foi submetido ao teste de usabilidade através do questionário SUS, além da aplicação de questionário para avaliar a utilidade deste pelo modelo de percepção de utilidade TAM. Participaram 36 voluntários que foram divididos em 3 grupos: residentes, ortopedistas da Sociedade Brasileira de Ortopedia e Traumatologia (SBOT) e cirurgiões de coluna. Resultados: Pela aplicação do questionário SUS, o aplicativo obteve escore de 75,69, valor que representa classificação C na escala de Sauro e Lewis, a terceira melhor. A utilidade do aplicativo foi considerada boa após avaliação das respostas positivas do modelo TAM. O planejamento através do aplicativo foi em média 20% mais rápido que o planejamento manual. Conclusão: o aplicativo Scoliplan mostrou-se útil e vem otimizar o planejamento cirúrgico em EIA, à medida que estabelece uma padronização de um passo a passo, apoiado na literatura (classificação de Suk), auxilia o usuário para um planejamento uniforme e confiável. Nível de Evidência III; Estudos Diagnósticos.

Descritores:
Escoliose; Adolescente; Planejamento; Smartphone; Aplicativos móveis

RESUMEN

Objetivo:

Evaluar la usabilidad y utilidad de una aplicación desarrollada para la planificación quirúrgica de corrección de escoliosis, medir el tiempo empleado en la planificación. Métodos: Estudio experimental, controlado, transversal y aplicado. La aplicación se sometió a una prueba de usabilidad mediante el cuestionario SUS, además de aplicar un cuestionario para evaluar su utilidad mediante el modelo de percepción de utilidad TAM. 36 voluntarios fueron divididos en 3 grupos: residentes, ortopedistas de la Sociedad Brasileña de Ortopedia y Traumatología (SBOT) y cirujanos de columna. Resultados: Al aplicar el cuestionario SUS, la aplicación obtuvo un puntaje de 75,69, valor que representa la clasificación C en la escala de Sauro y Lewis, la tercera mejor. La utilidad de la aplicación se consideró buena tras evaluar las respuestas positivas del modelo TAM. La planificación a través de la aplicación fue en promedio un 20 % más rápida que la planificación manual. Conclusión: la aplicación Scoliplan demostró ser útil y viene a optimizar la planificación quirúrgica en AIS, ya que establece una estandarización paso a paso, respaldada por la literatura (clasificación de Suk), ayuda al usuario para una planificación uniforme y confiable. Nivel de Evidencia III; Estudios de Diagnósticos.

Descriptores:
Escoliosis; Adolescente; Planificación; Teléfono Inteligente; Aplicaciones Móviles

INTRODUCTION

Adolescent Idiopathic Scoliosis (AIS) affects young people between the ages of 10 and skeletal maturity. It is defined as a three-dimensional deformity of the spine usually associated with vertebral rotation and an angulation of more than 10 degrees on the X-ray.¹

In surgical cases, to define the levels to be fused in the correction of AIS, Suk described a simple and easy-to-learn classification. It encompasses all curve patterns, predicts the limits of deformity, defines the extent of fusion, and has good interand intra-observer reproducibility.²

Preserving more segments in scoliosis correction surgery has led to a better distribution of functional movement through the non-fused levels,³ and could minimize the degenerative effects of overload on discs and joints.⁴

With technological advances in healthcare, it has been possible to develop a smartphone application (Scoliplan) that can provide spine surgeons and resident physicians with a sequential AIS surgical planning system where each step is guided according to the criteria developed by Suk.

OBJECTIVES

Plan AIS surgeries according to the classification described by Suk using the Scoliplan app;

To evaluate the usability and usefulness of the application among spine surgeons, general orthopedic doctors, and resident doctors;

Quantify the time spent during the usability test of the application.

MATERIALS AND METHODS

The Unichristus Research Ethics Committee approved the study. Opinion number: 4.311.351.

The application was subjected to a usability test. It was used to evaluate the planning of a scoliosis correction surgery. The following questions were raised: whether the proposal for preoperative planning via a smartphone app is feasible; whether the time taken to carry out planning via the app is less than when traditional planning was carried out; whether the system developed had good levels of usability and was useful for clinical practice.

Twelve spine surgeons who are members of the Brazilian Spine Society (SBC), 12 orthopedists who are members of the Brazilian Orthopedics and Traumatology Society (SBOT) and who work as preceptors in medical residency programs, and 12 secondand third-year orthopedics and traumatology residents were invited to take part in the study.

The surgery to be planned was an AIS repair, which requires defining the vertebral levels to be addressed.

An evaluation questionnaire was developed to assess the results of this study. It was divided into two parts: Part 1, based on the questionnaire - SUS,⁵ validated in Portuguese in 2011,⁶ which aims to collect information on the usability of the application developed and the simplicity of learning to use it (Ease of Learning); Part 2, based on Davis’ Technology Acceptance Model,⁷ aimed at identifying the level of usefulness of the system as perceived by users, perceived usefulness, during the planning of AIS correction surgery. (Appendix A)

The SUS scale is an easy-to-apply model for checking the usability of systems.⁸-¹¹ The SUS scale questionnaire consists of 10 questions (or items), each with five answer options following a 5-point Likert scale.

The questions that makeup Part 2 of the evaluation tool were based on Davis’ Technology Acceptance Model (TAM), which allows the quantification of the degree of usefulness perceived by users of a given application.

A clinical case was created of a young woman, 16 years old, with menarche at 12 years old, without identification, with a spinal deformity that began in adolescence. The patient shows a double structured curve (thoracic and lumbar) on the panoramic radiograph of the total spine. Panoramic total spine X-rays in AP and right and left inclinations were available to the participants.

During the usability test, the evaluator explained the test step-by-step. The free and informed consent form was read out, any questions about the research were answered, and the participant was asked to sign it.

The participant was given a printed clinical case template, presenting the history of the current illness and the patient’s clinical data. Panoramic radiographs of the spine in the posterior view, right lateral inclination, and left lateral inclination were printed in high quality on separate sheets.

The participant was given a case with a goniometer, a 4b black pencil, and an eraser. The Suk scoliosis classification model and the Nash-Moe vertebral rotation classification model were also made available so the participants could ask questions during the planning.

Surgical planning using the app was done on a Samsung^® A7 Lite tablet with an 8.1-inch screen. The participant received the tablet with the Scoliplan app already accessed. After initially filling in the patient’s details, the stopwatch was started. The radiographic images of the same clinical case were already in the tablet’s gallery, and when prompted by the app, the participant started planning.

The evaluator was responsible for timing the first and second stages of the test.

The evaluator described all the data collected on two data collection sheets so that the data could later be extracted and the results compared.

After the test phase, the participants were invited to answer the SUS and TAM questionnaires with specific adaptations for the context of planning a scoliosis correction surgery.

Statistical Analysis

The data was expressed as absolute frequency and percentage or mean and standard deviation and compared between the professional categories using Pearson’s chi-square test or Fisher’s exact test and the Kruskal-Wallis/Dunn test, respectively. The comparison between App and Manual was made considering the entire sample and each professional category using the McNemar and Wilcoxon tests. All the analyses were conducted using SPSS v20.0 for Windows with a 95% confidence level.

RESULTS

The SUS score for the sample of 36 participants was 75.69. The score ranged from 69.64 to 81.74, with a 95% confidence interval. The results showed that the application had a good level of usability according to the SUS questionnaire.

When the scores of the different groups were evaluated, the results showed no significant difference, as shown in Table 1.

Table 1
Results SUS score.

		Study groups
	Total	Residents	Orthopedists	Spine surgeons	p-Value
SUS	75.69±6.05	75.00±7.15	75.83±6.60	76.25±4.59	0.816a

Source: Prepared by the author.
• ^*
  p<0.05, aKruskal-Wallis/Dunn test (mean±SD).

The second part of the questionnaire, which presented statements related to the application based on the TAM model, elicited responses from questions 11 to 17, as shown in Table 2.

Table 2
TAM model questionnaire results.

		Study groups
	Total	Residents	Orthopedists	Spine surgeons	p-Value
Q11
I disagree	0(0.0%)	0(0.0%)	0(0.0%)	0(0.0%)	1.000b
I agree	36(100.0%)	12(100.0%)	12(100.0%)	12(100.0%)
Q12
I agree	35(97.2%)	12(100.0%)	11(91.7%)	12(100.0%)	0.358b
I can’t answer that	1(2.8%)	0(0.0%)	1(8.3%)	0(0.0%)
Q13
I disagree	0(0.0%)	0(0.0%)	0(0.0%)	0(0.0%)	1.000b
I agree	36(100.0%)	12(100.0%)	12(100.0%)	12(100.0%)
Q14
It contributes a lot	22(61.1%)	11(91.7%)*	8(66.7%)*	3(25.0%)	0.016b
Contributes	13(36.1%)	1(8.3%)	4(33.3%)	8(66.7%)*
Contributes little	1(2.8%)	0(0.0%)	0(0.0%)	1(8.3%)
Q15
It contributes a lot	13(36.1%)	7(58.3%)*	6(50.0%)*	0(0.0%)	0.006b
Contributes	23(63.9%)	5(41.7%)	6(50.0%)	12(100.0%)*
Q16
Indifferent	1(2.8%)	0(0.0%)	0(0.0%)	1(8.3%)	0.007b
I agree	24(66.7%)	4(33.3%)	9(75.0%)*	11(91.7%)*
I totally agree	11(30.6%)	8(66.7%)*	3(25.0%)	0(0.0%)
Q17
Indifferent	4(11.1%)	0(0.0%)	0(0.0%)	4(33.3%)*	0.002b
I agree	18(50.0%)	4(33.3%)	6(50.0%)*	8(66.7%)*
I totally agree	14(38.9%)	8(66.7%)*	6(50.0%)	0(0.0%)

Source: Prepared by the author.
• ^*
  p<0.05, bChi-square test (n, %).

About the time taken to carry out the planning, there was no statistical difference between the groups for the first and second planning stages. This difference was statistically significant in the first stage and the total time taken in manual planning.

When comparing the time between the app and the manual technique, all the results showed a shorter planning time using the app with p<0.05.

DISCUSSION

Traditionally, the surgical planning of an AIS is carried out manually with printed radiographs, using a pencil and a scoliometer.¹² digital radiographs became more common in health centers nationwide only a few years ago. It is still very common to find conventional X-rays in many cities in Brazil. As a result, better identification of radiological parameters to define surgical planning is compromised in some cases.

Technological advances in conjunction with health have promoted discussions and tried to make life easier for those who need to maintain quality when carrying out a surgical procedure as crucial to a patient’s life as AIS correction.¹³

The literature shows that there have been frequent studies in which digital tools tend to match and, in some cases, surpass the reliability of the choices of vertebrae that are part of scoliosis.¹⁴,¹⁵

The time taken to carry out manual planning, especially in the first stage of the test, was lower in the group of spinal surgeons, as expected, given their experience. However, when planning using the app, we observed that all the groups saw a reduction in total planning time of approximately 20%. The second stage of the planning involved answering the question about the LVSC and the rotation of L3 to finish the procedure, which saw an average time reduction of 39%.

Moftian (2022) developed a computer-aided measurement system with a reproducible approach to assess the Cobb angle in patients with idiopathic scoliosis. Despite being restricted to evaluating the time taken to measure only the angle after loading the image into the system, this result is similar to our study regarding the time taken to perform the task, comparing the digital and manual forms.¹⁶

When evaluating the usability of the Scoliplan application, it was possible to reach an average SUS score of 75.69 points. When the score for spinal surgeons was evaluated, the average was 76.25. Evaluating the three groups, there was no significant difference between the results found, defining the usability of the application as uniform between the groups.

Some studies present classifications based on the average SUS score to categorize a system’s usability level. The Bangor, Kortum, and Miller scale has strong interface validity for existing data. A score of 70 traditionally means approval, proposing a set of acceptability ranges that would help professionals determine whether a SUS score indicates an acceptable interface.¹⁷ Using this scale, the Scoliplan application falls into category C, which corresponds to a “good” result, the third best level of usability in all classes (residents, orthopedists, and spine surgeons).

When using the scale proposed by Sauro and Lewis, Scoliplan scored B in evaluating the total sample and separating by group.¹⁸

It is, therefore, possible to consider that the Scoliplan application had a good level of usability according to the scales used for this measurement.

When analyzing the perceived usefulness of the application, Davis’ model⁷ was used to identify the level of usefulness of the system, as perceived by users, during the measurement of angles and interpretation of images using the application. Usability is an important factor, but if the user doesn’t realize how useful the system is, they won’t use it.

When analyzing the questions created based on the TAM model, all the participants in the three groups responded positively about the app’s help in surgical planning, measuring the Cobb angle, and presenting the Nash-Moe vertebral rotation model. Regarding the possibility of tracing the LVSC, 97.2% of the responses were positive; only one was unable to answer, a participant from the orthopedic group, but there was no significant difference between them.

About the other questions, there was a significant difference in the answers between the groups, although negative responses were minimal. When asked about the app’s help for resident doctors, only one answer was negative (a spine surgeon). Only positive responses about assisting spinal surgeons were presented, with the statistical difference between the groups being related to the responses “contributes” and “contributes a lot”. The response from the group of surgeons was 100% supportive. The fact that they were more knowledgeable than the other groups on scoliosis and already had expertise in manual planning for this type of surgery led to this result.

The difference in the groups’ knowledge of the subject also influenced answer 17, when it was stated that the app helped better understand the concepts related to planning scoliosis surgery. Four respondents in the group of surgeons were indifferent, while SBOT orthopedists and residents agreed with the statement.

Despite some differences that were found and that can be justified by expertise, 97.2% of the participants stated that they agree or agree that the technology used in the app is useful for planning scoliosis correction surgery. Similar results have already been found in applications developed for surgical planning in other areas of orthopedics, such as knee arthroplasty,¹⁹ and to aid decision-making based on the measurement of the most common orthopedic angles.²⁰

Study limitations

The application study was carried out only with a double-curved idiopathic scoliosis model. More tests on clinical cases with different curve patterns could make the results more reliable. In addition, applying the tests to more participants would make the sample more expressive.

CONCLUSION

The application developed proved useful for the surgical planning of Adolescent Idiopathic Scoliosis. It showed good usability and excellent accuracy for measuring Cobb’s angles defining the terminal and neutral vertebrae. He reproduces Suk’s classification with the sequence developed, leading to the final planning. In addition, the app was superior to the manual technique in planning time. The study should be continued using other clinical cases with different idiopathic scoliosis patterns to improve further the usability and usefulness standards found.

• Reviewed by: Rodrigo Amaral

Appendix

Appendix A
SUS and TAM questionnaires. Part 1 - SUS Questionnaire (System Usability Scale).

	I strongly disagree	I disagree	Indifferent	I agree	I totally agree
1) I think I'd like to use this application frequently.	1	2	3	4	5
2) I found this application unnecessarily complex.	1	2	3	4	5
3) I found the application easy to use.	1	2	3	4	5
4) I think I would need technical support to use this application.	1	2	3	4	5
5) I thought that the application's various functions were well integrated.	1	2	3	4	5
6) I thought there was a lot of inconsistency in the application.	1	2	3	4	5
7) I imagine most people can learn to use this application very quickly.	1	2	3	4	5
8) I found the application very complicated to use.	1	2	3	4	5
9) I felt very confident in using this application.	1	2	3	4	5
10) I had to learn several things before I could start using this application.	1	2	3	4	5

Appendix A.
Part 2 - TAM model

11) Measuring the Cobb angle using the app can help the surgeon make decisions when planning scoliosis surgery. ( ) Agree ( ) Disagree ( ) Don’t know how to answer	14) I believe that the standardization through a step-by-step proposed by the app can help resident doctors learn about surgical planning in Scoliosis. ( ) Contributes a lot ( ) Contributes ( ) Contributes little ( ) Does not contribute
12) The possibility of tracing the Central Vertical Sacral Line on the X-ray photo collected by the app could help in the planning of scoliosis surgery. ( ) Agree ( ) Disagree ( ) Don’t know how to answer	15) I believe that standardization through a step-by-step process proposed by the app can help spine surgeons plan scoliosis surgery. ( ) Contributes a lot ( ) Contributes ( ) Contributes little ( ) Does not contribute
13) The presentation of the vertebral rotation model proposed by Nash-Moe helps the surgeon to correctly define the type, leading to less chance of error in surgical planning. ( ) Agree ( ) Disagree ( ) Don’t know how to answer	16) I think it’s a useful technology for planning scoliosis correction surgery ( ) Strongly Disagree ( ) Disagree ( ) Indifferent ( ) Agree ( ) Strongly Agree
	17) It helped me to understand better the concepts related to scoliosis planning. ( ) Strongly Disagree ( ) Disagree ( ) Indifferent ( ) Agree ( ) Strongly Agree

REFERENCES

• ¹
  Addai D, Zarkos J, Bowey AJ. Current concepts in the diagnosis and management of adolescent idiopathic scoliosis. Childs Nerv Syst. 2020;36(6):1111-9. doi:10.1007/s00381-020-04608-4.
  » https://doi.org/10.1007/s00381-020-04608-4
• ²
  Suk SI. Pedicle Screw Instrumentation for Adolescent Idiopathic Scoliosis: The Insertion Technique, the Fusion Levels and Direct Vertebral Rotation. Clin Orthop Surg. 2011;3(2):89-100. doi:10.4055/cios.2011.3.2.89.
  » https://doi.org/10.4055/cios.2011.3.2.89
• ³
  Marks M, Newton PO, Petcharaporn M, Bastrom TP, Shah S, Betz R, et al. The postoperative segmental motion of the unfused spine distal to the fusion in 100 patients with adolescent idiopathic scoliosis. Spine (Phila Pa 1976). 2012;37(10):826-32. doi:10.1097/BRS.0b013e31823b4eab.
  » https://doi.org/10.1097/BRS.0b013e31823b4eab
• ⁴
  Lee MC, Õunpuu S, Solomito M, Smith BG, Thomson JD. Loss in spinal motion from inclusion of a single midlumbar level in posterior spinal fusion for adolescent idiopathic scoliosis. Spine (Phila Pa 1976). 2013;38(22):1405-10. doi:10.1097/BRS.0b013e3182a4038b.
  » https://doi.org/10.1097/BRS.0b013e3182a4038b
• ⁵
  Brooke J. SUS - A quick and dirty usability scale. In: Brooke J. Usability Evaluation In Industry. Boca Raton: CRC Press; 1996. p. 189-94.
• ⁶
  Tenório JM, Cohrs FM, Sdepanian VL, Torres Pisa I, De H, Marin F. Desenvolvimento e Avaliação de um Protocolo Eletrônico para Atendimento e Monitoramento do Paciente com Doença Celíaca. RITA. 2010;17(2):210-20.
• ⁷
  Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Q. 1989;13(3):319-49. doi:10.2307/249008.
  » https://doi.org/10.2307/249008
• ⁸
  Zbick J, Nake I, Milrad M, Jansen M. A web-based framework to design and deploy mobile learning activities: Evaluating its usability, learnability and acceptance. In: Proceedings - IEEE 15th International Conference on Advanced Learning Technologies: Advanced Technologies for Supporting Open Access to Formal and Informal Learning; ICALT 2015. Institute of Electrical and Electronics Engineers Inc.; 2015. p. 88-92. doi:10.1109/ICALT.2015.97.
  » https://doi.org/10.1109/ICALT.2015.97
• ⁹
  Tabuenca B, Kalz M, Ternier S, Specht M. Mobile Authoring of Open Educational Resources for Authentic Learning Scenarios. Univers Access Inf Soc. 2016;15:329-43.
• ¹⁰
  Schmitz B, Ternier S, Kalz M, Klemke R, Specht M. Designing a mobile learning game to investigate the impact of role-playing on helping behaviour. In: European Conference on Technology Enhanced Learning; 2013. p. 357-70. doi:10.1007/978-3-642-40814-4_28.
  » https://doi.org/10.1007/978-3-642-40814-4_28
• ¹¹
  Alamer RA, Al-Otaibi HM, Al-Khalifa HS. L3MS: A lightweight language learning management system using mobile web technologies. In: 15th International Conference on Advanced Learning Technologies: Advanced Technologies for Supporting Open Access to Formal and Informal Learning. Institute of Electrical and Electronics Engineers Inc.; 2015. p. 326-7. doi:10.1109/ICALT.2015.13.
  » https://doi.org/10.1109/ICALT.2015.13
• ¹²
  Carman D, Browne R, Birch J. Measurement of scoliosis and kyphosis radiographs. Intraobserver and interobserver variation. J Bone Joint Surg. 1990;72(3):328-33.
• ¹³
  Robertson GAJ, Wong SJ, Brady RR, Subramanian AS. Smartphone apps for spinal surgery: is technology good or evil?. Eur Spine J. 2016;25(5):1355-62. doi:10.1007/s00586-015-3932-z.
  » https://doi.org/10.1007/s00586-015-3932-z
• ¹⁴
  Elfiky T, Patil N, Shawky M, Siam A, Ragab R, Allam Y. Oxford cobbometer versus computer assisted-software for measurement of cobb angle in adolescent idiopathic scoliosis. Neurospine. 2020;17(1):304-11. doi:10.14245/ns.1938260.130.
  » https://doi.org/10.14245/ns.1938260.130
• ¹⁵
  Shaw M, Adam CJ, Izatt MT, Licina P, Askin GN. Use of the iPhone for Cobb angle measurement in scoliosis. Eur Spine J. 2012;21(6):1062-8. doi:10.1007/s00586-011-2059-0.
  » https://doi.org/10.1007/s00586-011-2059-0
• ¹⁶
  Moftian N, Soltani TS, Salahzadeh Z, Pourfeizi HH, Gheibi Y, Fazlollahi A, et al. Computer-aided Measurement System Using Image Processing for Measuring Cobb Angle in Scoliosis. Middle East J Rehab Health Stud. 2022;9(1):e111360. doi:10.5812/mejrh.111360.
  » https://doi.org/10.5812/mejrh.111360
• ¹⁷
  Bangor A, Kortum P, Miller J. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J Usability Stud. 2009;4(3):114-23.
• ¹⁸
  Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. 2nd ed. Amsterdam: Elsevier; 2016. doi:10.1016/C2010-0-65192-3.
  » https://doi.org/10.1016/C2010-0-65192-3
• ¹⁹
  Nogueira JBS. Desenvolvimento e Avaliação de Usabilidade de Aplicativo Para Planejamento de Artroplastias Totais de Joelho [dissertation]. Centro Universitário Christus; 2016. [Accessed June 24, 2022]. Available at: https://repositorio.unichristus.edu.br/jspui/handle/123456789/628
  » https://repositorio.unichristus.edu.br/jspui/handle/123456789/628
• ²⁰
  Macedo FS de. Aplicativo Para Auxílio Na Avaliação de Exames de Imagem e Tomada de Decisão Em Ortopedia [dissertation]. Centro Universitário Christus; 2020. [Accessed June 24, 2022]. Available at: https://repositorio.unichristus.edu.br/jspui/handle/123456789/1012
  » https://repositorio.unichristus.edu.br/jspui/handle/123456789/1012

Publication Dates

• Publication in this collection
  24 Nov 2023
• Date of issue
  2023

History

• Received
  17 Aug 2022
• Accepted
  25 Sept 2023