Development and validation of a tool for competence assessment of the insert of the intrauterine device

Abstract

Objectives:.

to develop and validate an instrument to assess the professional competence of nurses in the insertion of the intrauterine device (IUD)

Methods:

methodological study, developed in three stages: 1) tool development (Theoretical procedures); 2) evaluation of the tool by experts and judgment of the initially proposed items (Appearance validation and content validation); 3) test of the version resulting from the assessment by experts and assessment of internal consistency (Analytical procedures). Ten judges participated in the face and content validation, among obstetric nurses and gynecologists/obstetricians from the Sofa Feldman Hospital in Belo Horizonte (MG), while 38 nursing residency students were evaluated using the test tool, in the last stage of the study. Calculations of the Content Validity Index (CVI) and Cronbach’s alpha coefficient were performed as psychometric measures.

Results:

the initial tool covered 39 items. No item obtained CVI<0.8; however, through suggestions from the judges, items were merged, totaling 34 items. The total Cronbach’s alpha coefficient for this version was 0.828.

Conclusion:

the tool developed is valid and reliable. It is believed that the implementation of this tool will contribute to the training of professionals and the improvement of knowledge, behaviors, and skills in nursing consultations with a focus on reproductive planning with an emphasis on the insertion of the IUD.

Key words:
Nursing; Intrauterine devices; Clinical competence; Competency-based education; Validation study

Resumo

Objetivos:.

desenvolver e validar um instrumento de avaliação da competência profissional do enfermeiro na inserção do dispositivo intrauterino (DIU)

Métodos:

estudo metodológico, desenvolvido em três etapas: 1) desenvolvimento da ferramenta (Procedimentos teóricos); 2) avaliação da ferramenta por especialistas e julgamento dos itens inicialmente propostos (Validação aparente e validação de conteúdo); 3) teste da versão resultante da avaliação pelos especialistas e avaliação da consistência interna (Procedimentos analíticos). Participaram da validação aparente e de conteúdo 10 juízes, dentre enfermeiros obstétricos e médicos ginecologistas/obstetras do Hospital Sofa Feldman em Belo Horizonte (MG), enquanto 38 alunos da residência em enfermagem foram avaliados por meio da ferramenta em teste, na última etapa do estudo. Foram realizados cálculos do Índice de Validade de Conteúdo (IVC) e do coeficiente alfa de Cronbach como medidas psicométricas.

Resultados:

a ferramenta inicial abrangeu 39 itens. Nenhum item obteve IVC<0,8; contudo, através de sugestões dos juízes, itens foram fundidos, totalizando 34 itens. O coeficiente alfa de Cronbach total desta versão foi de 0,828.

Conclusão:

a ferramenta desenvolvida apresenta-se válida e confável. Acredita-se que a implantação dessa ferramenta contribuirá para a formação de profissionais e para o aprimoramento dos conhecimentos, comportamentos e habilidades na consulta de enfermagem com foco no planejamento reprodutivo com ênfase na inserção do DIU.

Palavras-chave:
Enfermagem; Dispositivos intrauterinos; Competência clínica; Educação baseada em competências; Estudo de validação

Introduction

The TCu 380A intrauterine device (IUD) is a small, flexible plastic covered in copper and is inserted into the uterus through the cervix as a contraceptive method. It is the most used method in the world with a prevalence of 8.6% in the United States, 11% in Europe, and 41% in China. In Brazil, the prevalence is less than 2% in women of childbearing age.¹

Health services are having an organizational barrier that hinders Brazilian women to use this contraceptive method. Among these barriers, there is the limitation of the performance of other health professionals, other than the doctor, in the insertion of the IUD, since the multidisciplinary team is not always qualified or available to perform the procedure.²

Currently, the insertion of the IUD is a practice that receives legal support to be performed by other trained and qualified health professionals.³ Studies in different countries show that these professionals (physicians, general nurses, obstetric nurses, obstetricians, and midwives) should have prior knowledge of female pelvic anatomy and that training should consist of theory and practice. Theoretical training should address the method’s indications and contraindications, as well as its mechanism of action, duration of use, impossible side effects, and complications. The practicetraining should address the step by step of inserting the device, requiring from three to 20 IUD insertions under supervision at the end of the training, depending on the evolution of each professional.⁴,⁵ Assessmenttools can be useful in this context for the tracking of existing competences and those that still need to be acquired, representing an important resource for managers and educators.

Thus, these instruments, also called tools, influence organization, decisions about care, and the interventions, being instruments that are integral to clinical practice, health assessment, and research. To have applicability and to be able to present scientifically robust results, the instruments must present good psychometric properties.⁶

The use of instruments that identify existing skills of nursing professionals has been the subject of contemporary studies in the area since this routine assessment can favor a safe, humane professional practice and minimize risks to patients, nurses, the institution, and the community. They can also contribute to academic and organizational settings in health services.⁷, ⁸, ⁹

Despite the relevance of this practice, especially in women’s health area, there is a gap in the knowledge about competences exercised by nurses in the insertion of the IUD.¹,¹⁰ Also, an instrument that directs the teaching of IUD insertion of these professionals was not identified in the literature.

Thus, this work aimed to develop and validate a tool evaluating the competence of professional nurses in the insertion of the IUD, during the nursing consultation.

Methods

This is a methodological study developed from April to September 2019, divided into three stages: 1) tool development (Theoretical procedures); 2) evaluation of the tool by experts and judgment of the items initially proposed (apparent validation and content validation); 3) test stage of the version resulting from the evaluation by specialists and evaluation of the internal consistency (Analytical procedures) (Figure 1). Therefore, in the construction of the proposed instrument, we used the model proposed by Pasquali to evaluate parameters of validity and precision and demonstrate evidence of validity in psychometrics.¹¹,¹²

Figure 1
Stages of elaboration and validation of the tool for competency assessment of the insertion of intrauterine device.

Tool Development (Theoretical Procedures)

The first step wasto defineand measure the competence needed by nurses to insert the IUD. Competence is a theoretical concept that contributes to safe professional practice. It is a complex combination of knowledge, performance, skills, values, and attitudes.¹³ It involves the possession of sufficient knowledge and skills to perform work-related tasks, but it also embodies ethics, values, and the ability to reflect practice.¹⁴

To guide the construction of the tool, as a theoretical base, we used the public health policies adopted in Brazil, the protocols of the Ministry of Health (MH), the World Health Organization (WHO), the Pan American Health Organization (PAHO) and the International Confederation of Midwives (ICM).³,¹⁵, ¹⁶, ¹⁷, ¹⁸

The first version of the tool was created by the researchers and was structured in 3 domains (knowledge, behaviors, and skills). It was composed of 39 verification items, with 5 possibilities of answers using a Likert-type scale, organized as follows: (1) Strongly Disagree; (2) Partially Disagree; (3) I Do not Know/Not Applicable; (4) Partially Agree; (5) Totally Agree.¹⁹,²⁰

Evaluation of the tool by experts and judgment of the items initially proposed (Apparent Validation and Content Validation)

The apparent and content validation was based on the evaluation of the tool by specialized professionals and by the judgment of the items initially proposed. The tool underwent a thorough evaluation and analysis of each domain and items initially proposed to compose the tool, using the Delphi technique.⁶,²¹ This evaluation was carried out independently by 10 obstetric nurses and gynecologists/obstetricians, with experience in the insertion of the IUD, either in teaching (theoretical part) or in preceptorship (practical part), for at least five years, in the residency programs of Hospital Sofia Feldman (HSF), located in Belo Horizonte - Minas Gerais. These professionals were part of the panel of judges who helped in the process of the study,optimizing the tool. Due to the need for professionals with specific qualifications to judge the tool, the researchersintentionally chose the judges. The invitation letter was sent by email. After acceptance, wesent the instrument under development and guidelines on criteria for evaluating the tool. The concept that should be followed to evaluate the instrument in terms of representativeness, simplicity, clarity, relevance, and precision was detailed in the information. We also collected data regarding the professional training of each judge.

The criteria used by the judges to evaluate the items initially proposed were: clarity, relevance, simplicity, representativeness, and precision. Each item of the competency assessment tool was evaluated using a Likert-type scale, with options at four levels of importance and selection of a single response for each criterion: 1 = Item does not include the criterion; 2 = Item unable to meet the criterion without review; 3 = Item meets the criterion but needs minimal change, and 4 = Item meets criterion.⁶ This step was guided by a semi-structured form, delivered to each of the judges, along with an invitation letter and feedback, and we recommended returningit within a period of up to 10 days.

After returning the material to the judges, the answers were tabulated and analyzed qualitatively and quantitatively and all the suggestions were organized in tables. For this, we used the program Microsoft® Excel® 2010 and the data treatment was performed using descriptive statistics.

For quantitative data analysis, we calculated the Content Validity Index (CVI) of each item of the tool by the sum of the agreement of items that were evaluated as “3” or “4” by the judges, dividing the result of this sum by the total number of judges. Items whose CVI was less than 0.8 have been reformulated. Items whose CVI was greater than 0.8 and, preferably, greater than 0.906 remained unchanged.

The qualitative evaluation step of the instrument consisted of the analysis of the items proposed by the judges, with the possibility of making considerations and suggestions about each item for the reformulation of the instrument, in a space reserved for comments and suggestions. This step comprised the semantic evaluation of the instrument as a whole, in addition to clarity and adequacy. At this step, we also offered the possibility of including new relevant questions related to the competence of IUD insertion by nurses.

Test stage of the version resulting from the evaluation byexperts and evaluation of internal consistency (Analytical Procedures)

The version of the tool resulting from the evaluation by judges (second version) was then tested by applying it to the target population (Pre-Test Stage). Thus, six obstetricnurses from the HSF used the tool to assess the competence of resident students in obstetric nursing, from March to August 2019, who had performed at least 10 IUD insertions in an interval, which is the number of IUD insertions required by the institution for the professional to be authorized to perform the procedure without direct supervision. Each obstetric nurse received previous guidance from the researchers on how to complete the tool. In the same sense, they were instructed to carry out a direct assessment of each student, guided by the use of the tool.

In all, 38 nursing residency students were evaluated using the test tool. The data collected in this step were processed and analyzed by SPSS program, version 22.0, for Windows 10.0. Thus, an item-by-item analysis of all components of the tool was carried out.

Based on the results obtained from the individual assessment of each student, we verified the instrument’s internal consistency, which is one of the criteria for assessing the tool’s reliability. Internal consistency indicates whether all domains of the tool measure the same characteristic. A low internal consistency estimate may mean that the items measure different constructs, or the responses to the tool items are inconsistent²². To verify this assumption, the calculation of Cronbach’s alpha coefficient was initially performed. This coefficient reflects the degree of covariance between items on a scale. Thus, the smaller the sum of the items’ variance, the more consistent the tool.²²For this research, Cronbach’s alpha values > 0.70 were recommended.

In addition to Cronbach’s alpha coefficient, we used two other reliability models: the alpha if the item was deleted and the average correlation between the items. Alpha values if the item were deleted allow the researcher to assess whetherthe value of the total Cronbach’s alpha coefficient of the domain increases or decreases when removing an item from a certain domain of the instrument.²²In this way, we verifed whether any item in the tool was individually affecting the value of Cronbach’s alpha. As for the average correlation between items, as the alpha coefficient increases, the average correlation follows this increase. Therefore, if correlations are high, there is evidence that the items measure the same construct, satisfying the reliability assessment.²² In this study, the value of 0.30 was considered the limit for evaluating this criterion.

The study was authorized by the HSF, through a letter of consent, signed by the authority of the body, and was approved by the Research Ethics Committee (CEP) of the UFVJM under opinion nº 3,274,884. Before data collection, each study participant received guidance and clarification on the research objectives and the nature of data collection. Then, all of them officially accepted to participate by signing the informed consent form.

Results

The development of the first version of the tool guide for evaluation of competence of the professional nurses in the insertion of IUD by the researchers was based on the technique of insertion of IUD indicated by the MH and the WHO, in the knowledge and behaviors indicated by the ICM as indispensable to the professional, in public health policies related to reproductive planning in Brazil and nursing legislation. Thus, the initial version had 39 verification items, divided into three domains: knowledge, behavior, and skills.

In the second stage, a panel of 10 judges, composed of eight obstetric nurses and two gynecologists/obstetricians (four with specialist degrees, three master’s students, one master, one doctoral student, and one doctor), performed the analysis of items and the domains of the initially proposed tool. In the quantitative assessment, no item had a CVI lower than 0.8 (Table 1). Therefore, no item was modified considering this criterion. However, through the qualitative assessment performed by the judges, and from the judgment that the items represented similar aspects, they decided to unify items 9, 10, 12, and 13 of the frst version of the tool into a single item, making it Item 9 of the second version of the tool. Items 24, 25, and 33 were also merged into a single item, becoming item 21 of the second version of the tool. Thus, after evaluating and contemplating all the considerations by the judges, the tool was defined as containing 34 items.

Table 1
Content Validity Index of the total of 39 items of the first version of the assessment tool for the insertion of the interval IUD by nurses, evaluated by 10 judges in the criteria of representativeness, simplicity, clarity, relevance, and precision.

	Content Validity Index
	Representativeness	Simplicity	Clarity	Relevance	Precision
DOMAIN 1: KNOWLEDGE
The obstetric nurse has the knowledge and/or understanding of...
1. structure governing reproductive health for women of all ages, including laws, policies, protocols, and professional guidelines	1.0	1.0	1.0	1.0	1.0
2. culture, local beliefs, traditional and modern health practices (beneficial and harmful)	1.0	1.0	1.0	1.0	1.0
3. female anatomy and physiology related to conception and reproduction	1.0	1.0	1.0	1.0	1.0
4. components of health history, family history, and relevant genetic history	1.0	1.0	1.0	1.0	1.0
5. content and investigation of the physical examination and laboratory studies that assess the potential for a pregnancy	1.0	1.0	0.9	0.9	1.0
6. reproductive planning methods, their basic principles, deadlines for use, mode of action, indication and contraindication for use, benefits, and risks	1.0	1.0	1.0	1.0	1.0
7. signs and symptoms of urinary tract infection and STDs	1.0	1.0	1.0	1.0	1.0
8. main screening methods for cervical cancer, (such as visual inspection with acetic acid, Pap smear)	1.0	1.0	1.0	1.0	1.0
DOMAIN 2: BEHAVIORS
The obstetric nurse ...
9. is responsible for the clinical decisions and actions	0.9	0.9	0.9	0.9	0.9
10. acts consistently following the professional ethics, values, and human rights	1.0	0.9	0.9	1.0	0.9
11. uses universal/standard precautions, infection prevention, and control strategies	1.0	1.0	1.0	1.0	1.0
12. behaves in a courteous, non-judgmental, non-discriminatory, and culturally appropriate manner with all patients regardless of their status, origin, or religious belief	0.9	1.0	0.9	0.9	0.9
13. maintains the confidentiality of all information shared by women; communicates essential information among other health providers or family members only with women’s explicit permission and imperative need	1.0	1.0	1.0	1.0	0.9
14. allows women to make informed choices about their health	0.9	0.9	0.9	0.9	0.9
15. refers or transfers to care providers or facilities for continuing care when health care needs exceed their competence	0.9	1.0	0.9	1.0	1.0
16. works collaboratively (teamwork) with other health professionals to improve services for women and families	1.0	1.0	0.9	1.0	1.0

	Content Validity Index
	Representativeness	Simplicity	Clarity	Relevance	Precision
DOMAIN 3: SKILLS
The obstetric nurse has the skill and/or ability to ...
17. education discussions with and for women and their families using appropriate communication and listening skills	1.0	1.0	1.0	1.0	1.0
18. advising women on managing side effects and problems with using different reproductive planning methods	1.0	1.0	1.0	1.0	1.0
19. order, interpret and perform common laboratory tests (such as blood count, pap smear, STD screening, BHCG)	1.0	1.0	1.0	1.0	1.0
20. perform anamnesis addressing the health, obstetric, gynecological, and reproductive health history	1.0	1.0	1.0	1.0	1.0
21. complete and verify that all documentation for performing the IUD insertion is duly completed (appointment form, anamnesis, informed consent form)	1.0	1.0	1.0	1.0	1.0
22. perform physical examination	1.0	1.0	1.0	1.0	1.0
23. provide care, support, and referral or treatment for women with STIs	0.9	1.0	1.0	0.9	1.0
24. guide what the IUD is, its mechanism of action, time of use, its indications and contraindications, and how the procedure will occur	1.0	1.0	1.0	1.0	1.0
25. resolve the woman’s doubts regarding the method and the consent form	1.0	1.0	1.0	1.0	1.0
26. removing the IUD that has visible or not visible threads, including the use of alligator clips	1.0	1.0	1.0	1.0	0.9
27. perform a vaginal examination to assess the position of the uterus	1.0	1.0	1.0	1.0	1.0
28. perform vulva inspection and specular examination using lubricant	1.0	1.0	1.0	1.0	1.0
29. perform disinfection/cleaning of the cervix when necessary	1.0	1.0	1.0	1.0	1.0
30. perform hysterometry and IUD insertion, using Pozzi forceps to straighten the uterus	1.0	1.0	1.0	1.0	1.0
31. cut the threads and leave them outside the cervix by approximately 3 cm	1.0	1.0	1.0	1.0	1.0
32. recognize warning signs after insertion (example: vagal reflex, uterine perforation) and take appropriate action	1.0	1.0	1.0	1.0	1.0
33. guide possible side effects, warning signs, and when to seek help from the health service	1.0	1.0	1.0	1.0	1.0
34. guide the woman to perform the self-examination to assess the location of the IUD	1.0	1.0	1.0	1.0	1.0
35. prescribe symptomatic cramps	0.9	0.9	0.9	0.9	1.0
36. fill in the patient´s record properly explaining the size of the wire left	1.0	1.0	1.0	1.0	1.0
37. schedule review from 42 days	1.0	1.0	1.0	1.0	1.0
38. request US if necessary	0.9	1.0	1.0	0.9	1.0
39. check the size of the wires through speculum examination at the review appointment	1.0	1.0	1.0	1.0	1.0

• STD = Sexually transmitted diseases; BHCG = Human chorionic gonadotropin hormone; IUD = Intrauterine device; STIs = Sexually transmitted infections; US = Ultrasound.

This version showed a total Cronbach’s alpha coefficient of 0.828. This tool also obtained a high level of reliability with Cronbach’s alpha for the knowledge and behavior domains. However, for the skills domain, Cronbach’s alpha was unsatisfactory (<0.70) (Table 2). When evaluating the isolated behavior of each item in this domain, there was an increase in Cronbach’s alpha if items 22, 23, and 25 were eliminated, in addition to a low corrected item-total correlation (<0.30). The results of this analysis suggested the elimination of items 20, 33, and 34 from the second version of the tool, as they have zero variance and do not add information to the assessment.

Table 2
The item-total correlation coefficient, alpha value of the total of 34 items of the second version of the Nurse Interval IUD Insertion Assessment Tool, and alpha values when each of the items was excluded.

Domain	Item	Scale means if the item is excluded	Scale variance if the item is excluded	Cronbach's alpha	Cronbach's Alpha if the item is excluded	Corrected item-total correlation
Knowledge
	2	34.08	4.615		0.960	0.903
	3	34.11	4.421		0.956	0.958
	4	34.11	4.637		0.966	0.789
	5	34.11	4.637		0.966	0.789
	6	34.13	4.388		0.960	0.900
	7	34.08	4.669		0.962	0.857
	8	34.08	4.669		0.962	0.857
Behavior	9	19.61	1.218		0.763	0.879
	10	19.61	1.218		0.763	0.879
	11	19.63	1.212	0.841	0.784	0.852
	12	19.68	1.033		0.949	0.443
	13	19.53	1.385		0.809	0.726
Skills	14	76.18	17.019		0.606	0.493
	15	76.24	16.942		0.607	0.406
	16	76.26	17.605		0.635	0.066
	17	76.21	16.982		0.607	0.440
	18	76.16	17.434		0.616	0.381
	19	76.34	16.610		0.602	0.401
	20	-	-		-	-
	21	76.18	17.289		0.614	0.370
	22	78.03	17.972		0.634	0.034
	23	79.89	16.259		0.634	0.154
	24	76.21	17.576	0.630	0.622	0.204
	25	76.53	16.202		0.680	0.045
	26	76.26	16.253		0.600	0.375
	27	76.21	16.549		0.595	0.616
	28	76.21	17.144		0.611	0.374
	29	77.13	11.090		0.617	0.399
	30	76.37	15.590		0.595	0.366
	31	76.24	16.564		0.597	0.546
	32	76.13	17.577		0.618	0.442
	33	-	-		-	-
	34	-	-		-	-

• IUD = Intrauterine device.

Although the analyses indicated the removal of items 20, 22, 23, 25, 33, and 34 for a domain with greater internal consistency (alpha = 0.704), the theoretical evaluation of items 20, 23, and 25 of the second version of the tool showed that these items were indispensable. Thus, we kept them in the tool. All the aforementioned changes are highlighted in Table 3.

Table 3
Versions of the tool aimed at the assessment by competence of the professional nurse in the insertion of the IUD according to the stage of the research.

Items initially proposed by the researchers	Version after evaluation of the tool by the judges	Tested final version
DOMAIN 1: BASIC KNOWLEDGE The obstetric nurse has the knowledge and/or understanding of...	DOMAIN 1: BASIC KNOWLEDGE The obstetric nurse has the knowledge and/or understanding of...	DOMAIN 1: BASIC KNOWLEDGE The obstetric nurse has the knowledge and/or understanding of...
1. structure governing reproductive health for women of all ages, including laws, policies, protocols, and professional guidelines	1. structure governing reproductive health for women, including laws, policies, protocols, and guidelines	1. structure governing reproductive health for women, including laws, policies, protocols, and guidelines
2. culture, local beliefs, traditional and modern health practices (beneficial and harmful)	2. culture, local beliefs, traditional and modern health practices (beneficial and harmful) related to contraception	2. culture, local beliefs, traditional and modern health practices (beneficial and harmful) related to contraception
3. female anatomy and physiology related to conception and reproduction	3. anatomy and physiology of the female body, related to the processes of conception and reproduction	3. anatomy and physiology of the female body, related to the processes of conception and reproduction
4. components of health history, family history, and relevant genetic history	4. components of women’s health history and family history, relevant for IUD insertion	4. components of the woman’s health history and family history, relevant for IUD insertion
5. content and investigation of the physical examination and laboratory studies that assess the potential for a pregnancy	5. content and investigation of women’s physical examination and laboratory studies that assess the potential for a pregnancy	5. content and investigation of the woman’s physical examination and laboratory studies that assess the potential for a pregnancy
6. reproductive planning methods, their basic principles, deadlines for use, mode of action, indication and contraindication for use, benefits, and risks	6. contraceptive methods, their mechanisms of action, deadlines for their use, indication, contraindication, benefits, and risks	6. contraceptive methods, their mechanisms of action, deadlines for their use, indication, contraindication, benefits, and risks
7. signs and symptoms of urinary tract infection and STDs	7. signs and symptoms of urinary tract infection and sexually transmitted infections	7. signs and symptoms of urinary tract infection and sexually transmitted infections
8. main screening methods for cervical cancer, (such as visual inspection with acetic acid, Pap smear)	8. screening methods for cervical cancer, (such as visual inspection with acetic acid, Pap smear)	8. screening methods for cervical cancer, (such as visual inspection with acetic acid, Pap smear)
DOMAIN 2: BASIC BEHAVIOR The obstetric nurse...	DOMAIN 2: BASIC BEHAVIOR The obstetric nurse...	DOMAIN 2: BASIC BEHAVIOR The obstetric nurse...
9. is responsible for the clinical decisions and actions it performs	9. acts following human rights, ethics, and professional support	9. acts following human rights, ethics, and professional support
10. acts consistently following professional ethics, values, and human rights	-	10. uses universal/standard precautions, infection prevention, and control strategies
11. uses universal/standard precautions, infection prevention, and control strategies	10. uses universal/standard precautions, infection prevention, and control strategies	11. allows women to make informed choices about their health
12. behaves in a courteous, non-judgmental, non-discriminatory, and culturally appropriate manner with all clients regardless of their status, origin, or religious belief	-	12. refers to the doctor and/or transfers to the hospital for continuity of care when these exceed their competence
13. maintains the confidentiality of all information shared by the women; communicates essential information among other health providers or family members only with women’s explicit permission and imperative need	-	13. works collaboratively (teamwork) with other healthcare professionals
14. allows women to make informed choices about their health	11. allows women to make informed choices about their health	DOMAIN 3: BASIC SKILLS The obstetric nurse has the skill and/or ability to...
15. refers or transfers to care providers or facilities for continuing care when health care needs exceed their competence	12. refers to the doctor and/or transfer to the hospital for continuity of care when these exceed their competence	14. promote health education with/and for women and their families using appropriate communication and listening skills
16. works collaboratively (teamwork) with other health professionals to improve services for women and families	13. works collaboratively (teamwork) with other healthcare professionals	15. advise women on the management of side effects and problems arising from the use of different contraceptive methods
DOMAIN 3: BASIC SKILLS The obstetric nurse has the skill and/or ability to...	DOMAIN 3: BASIC SKILLS The obstetric nurse has the skill and/or ability to...	16. order, interpret, and perform common laboratory tests (e.g., blood count, Pap smear, STI screening, BHCG)
17. education discussions with and for women and their families using appropriate communication and listening skills	14. promote health education with/and for women and their families using appropriate communication and listening skills	17. perform anamnesis addressing the health, obstetric, gynecological, and reproductive health history of women
18. advising women on managing side effects and problems with using different reproductive planning methods	15. advise women on the management of side effects and problems arising from the use of different contraceptive methods	18. complete and verify that all documentation for the insertion of the IUD is duly completed and signed (appointment form, anamnesis, informed consent form)
19. order, interpret, and perform common laboratory tests (e.g., blood count, pap smear, STD screening, BHCG)	16. order, interpret, and perform common laboratory tests (e.g., blood count, Pap smear, STI screening, BHCG)	19. perform a physical examination of the woman
20. perform anamnesis addressing the health, obstetric, gynecological, and reproductive health history	17. perform anamnesis addressing the health, obstetric, and gynecological history, and women’s reproductive health	20. provide care, support, and referral for women with a sexually transmitted infection
21. complete and verify that all documentation for performing the IUD insertion is duly completed (appointment form, anamnesis, informed consent form)	18. complete and verify that all documentation for the insertion of the IUD is duly completed and signed (appointment form, anamnesis, informed consent form)	21. guide the woman on what the IUD is, mechanism of action, duration of use, how the procedure will be performed, possible side effects, warning signs of complications when to seek the health service, and clarify women’s doubts regarding the method
22. perform physical examination	19. perform a physical examination of the woman	22. remove the IUD that has visible threads or not, using alligator clips for those that do not have visible threads, at any time, when the woman wishes or the term of use has expired
23. provide care, support, and referral or treatment for women with STIs	20. provide care, support, and referral for women with a sexually transmitted infection	23. perform a bimanual pelvic exam
24. guide what the IUD is, its mechanism of action, time of use, its indications and contraindications, and how the procedure will occur	21. guide women on what the IUD is, mechanism of action, duration of use, how the procedure will be performed, possible side effects, warning signs of complications when to seek the health service, and clarify the woman’s doubts regarding the method	24. perform vulva inspection and speculum examination
25. resolve women’s doubts regarding the method and the consent form	-	25. perform cervical antisepsis
26. remove the IUD that has visible wires or not, including the use of alligator clips	22. remove the IUD that has visible threads or not, using alligator forceps for those that do not have visible threads, at any time, when the woman wishes or when the use period ends	26. perform clamping of the anterior lip of the cervix with Pozzi forceps to straighten the uterus and perform hysterometry
27. perform a vaginal examination to assess the position of the uterus	23. perform a bimanual pelvic exam	27. Insert the IUD rods into the insertion guide conductor, insert the IUD into the bottom of the uterine cavity using the deposition technique, cut the wires and leave them outside the external orifice of the cervix by approximately 2 to 3 cm
28. perform vulva inspection and specular examination using lubricant	24. perform vulva inspection and speculum examination	28. recognize warning signs after IUD insertion (example: vagal reflex, uterine perforation) and take appropriate action
29. perform disinfection/cleaning of the cervix when necessary	25. perform cervical antisepsis	-
30. perform hysterometry and IUD insertion, using Pozzi forceps to straighten the uterus	26. perform clamping of the anterior lip of the cervix with Pozzi forceps to straighten the uterus and perform hysterometry	29. guide the woman to perform the self-examination (vaginal examination) to assess the positioning of the IUD
31. cut the threads and leave them outside the cervix by approximately 3 cm	27. insert the IUD rods into the insertion guide conductor, insert the IUD into the bottom of the uterine cavity using the deposition technique, cut the wires and leave them outside the external orifice of the cervix in approximately 2 to 3 cm	30. prescribe symptomatic cramps according to professional practice law
32. recognize warning signs after insertion (example: vagal reflex, uterine perforation) and take appropriate action	28. recognize warning signs after IUD insertion (example: vagal reflex, uterine perforation) and take appropriate action	31. fill in the patient´s record properly, explaining the size of the wires left
33. guide possible side effects, warning signs, and when to seek help from the health service	-	32. schedule an IUD review between 30 and 40 days after insertion
34. guide the woman to perform the self-examination to assess the location of the IUD	29. guide women to perform the self-examination (vaginal examination) to assess the positioning of the IUD	33. check wire size by speculum examination at the IUD review appointment
35. prescribe symptomatic cramps	30. prescribe symptomatic cramps according to professional practice law	34. request an ultrasound if threads of a different size than the one left on the day of the IUD insertion
36. fill in the patient´s recordproperly explaining the size of the wire left	31. fill in the patient´s recordproperly, explaining the size of the wires left
37. schedule review from 42 days	32. schedule an IUD review between 30 and 40 days after insertion
38. request US if necessary	33. check wire size by speculum examination at the IUD review appointment
39. check the size of the wires through speculum examination at the review appointment	34. request an ultrasound if threads of a different size than the one left on the day of the IUD insertion

• STD = Sexually transmitted diseases; BHCG = Human chorionic gonadotropin hormone;IUD = Intrauterine device; STIs = Sexually transmitted infections; US = Ultrasound.

The version of the tool from the evaluation by the judges was tested from its application by obstetric nurses at the HSF, to verify the competence of 38 residents in obstetric nursing during the application of the IUD. Regarding the characteristics of students evaluated using the tool, most were female (97.3%), with a mean age of 26.7 years old (±3.02), and 50% were up to 26 years old, ranging between 23 and 37 years old. The average time of professional practice before the residency course was 5.6 months (±11.71), and 50% had no previous professional clinical experience in the nursing area. The maximum time of the previous action was 48 months.

The tool, in its tested form composed of 34 items, accepts total scores ranging from 34 (worst case scenario) to 170 (best case scenario) points. Considering the score by domain, the tool accepts the following scenario: basic knowledge (8 to 40 points); basic behaviors (5 to 25 points); and basic skills (21 to 105 points). The average score achieved by professionals from the assessment guided by the tool was 38.97 (±2.43) for the knowledge domain, 24.53 (±1.35) for the behavior domain, and 72.24 (±3.77) for the skills domain.

Only items 20, 22, 33, and 34 were not influenced by the maximum and minimum effect, since less than 15.0% of participants were classified in the highest or lowest possible score on the response scale (Table 4).

Table 4
Descriptive statistics of the values obtained by the second version of the assessment tool for the insertion of the IUD interval by nurses (n=38).

Variable	Strongly Disagree (1)		Partially Disagree (2)		I don’t Know/Not Applicable (3)		Partially agree (4)		I totally agree (5)
	f	(%)	f	(%)	f	(%)	f	(%)	f	(%)
DOMAIN 1: KNOWLEDGE
Does the nurse have the knowledge and/or understanding of...
1. structure governing reproductive health for women, including laws, policies, protocols, and guidelines	0	0	0	0	0	0	6	15.8	32	84.2
2. culture, local beliefs, traditional and modern health practices (beneficial and harmful) related to contraception	0	0	0	0	0	0	4	10.5	34	89.5
3. anatomy and physiology of the female body, related to the processes of conception and reproduction	0	0	0	0	0	0	5	13.2	33	86.8
4. components of women’s health history and family history, relevant for IUD insertion	0	0	0	0	0	0	5	13.2	33	86.8
5. content and investigation of the woman’s physical examination and laboratory studies that assess the potential for a pregnancy	0	0	0	0	0	0	5	13.2	33	86.8
6. contraceptive methods, their mechanisms of action, deadlines for their use, indication, contraindication, benefits, and risks	0	0	0	0	0	0	6	15.8	32	84.2
7. signs and symptoms of urinary tract infection and sexually transmitted infections	0	0	0	0	0	0	4	10.5	34	89.5
8. screening methods for cervical cancer, (such as visual inspection with acetic acid, Pap smear)	0	0	0	0	0	0	4	10.5	34	89.5
DOMAIN 2: BEHAVIOR
The nurse...
9. acts following human rights, ethics, and professional support	0	0	0	0	0	0	3	7.9	35	92.1
10. uses universal/standard precautions, infection prevention and control strategies	0	0	0	0	0	0	3	7.9	35	92.1
11. allows women to make informed choices about their health	0	0	0	0	0	0	4	10.5	34	89.5
12. refers to the doctor and/or transfers to the hospital for continuity of care when these exceed their competence	0	0	1	2.6	0	0	3	7.9	34	89.5
13. works collaboratively (teamwork) with other healthcare professionals	0	0	0	0	0	0	2	5.3	36	94.7
DOMAIN 3: SKILLS
The nurse has the ability and/or ability to...
14. promote health education with/and for women and their families using appropriate communication and listening skills	0	0	0	0	0	0	3	7.9	35	92.1
15. advise women on the management of side effects and problems arising from the use of different contraceptive methods	0	0	0	0	0	0	5	13.2	33	86.8
16. order, interpret, and perform common laboratory tests (e.g., blood count, Pap smear, STI screening, BHCG)	0	0	1	2.6	0	0	3	7.9	34	89.5
17. perform anamnesis addressing the health, obstetric, gynecological, and reproductive health history of the women	0	0	0	0	0	0	4	10.5	34	89.5
18. complete and verify that all documentation for the insertion of the IUD is duly completed and signed (appointment form, anamnesis, informed consent form)	0	0	0	0	0	0	2	5.3	36	94.7
19. perform a physical examination of the women	0	0	0	0	0	0	9	23.7	29	76.3
20. provide care, support and referral for women with a sexually transmitted infection	0	0	0	0	38	100.0	0	0	0	0
21. guide the woman on what the IUD is, mechanism of action, duration of use, how the procedure will be performed, possible side effects, warning signs of complications when to seek the health service, and clarify the women’s doubts regarding the method	0	0	0	0	0	0	3	7.9	35	92.1
22. remove the IUD that has visible threads or not, using alligator forceps for those that do not have visible threads, at any time, when the women wishe or when the use period ends	0	0	0	0	36	94.7	1	2.6	1	2.6
23. perform bimanual pelvic exam	36	94.7	0	0	0	0	0	0	2	5.3
24. perform vulva inspection and speculum examination	0	0	0	0	0	0	4	10.5	34	89.5
25. perform cervical antisepsis	4	10.5	0	0	0	0	0	0	34	89.5
26. perform clamping of the anterior lip of the cervix with Pozzi forceps to straighten the uterus and perform hysterometry	0	0	1	2.6	0	0	3	7.9	34	89.5
27. Insert the IUD rods into the insertion guide conductor, insert the IUD into the bottom of the uterine cavity using the deposition technique, cut the wires and leave them outside the external orifice of the cervix in approximately 2 to 3 cm	0	0	0	0	0	0	4	10.5	34	89.5
28. recognize warning signs after IUD insertion (example: vagal reflex, uterine perforation) and take appropriate action	0	0	0	0	0	0	4	10.5	34	89.5
29. guide the woman to perform the self-examination (vaginal examination) to assess the positioning of the IUD	8	21.1	1	2.6	0	0	4	10.5	25	65.8
30. prescribe symptomatic for cramps according to professional practice law	1	2.6	0	0	0	0	6	15.8	31	81.6
31. fill in the patient´s record properly, explaining the size of the wires left	0	0	0	0	0	0	5	13.2	33	86.8
32. schedule an IUD review between 30 and 40 days after insertion	0	0	0	0	0	0	1	2.6	37	97.4
33. check wire size by speculum examination at the IUD review appointment	0	0	0	0	38	100	0	0	0	0
34. request an ultrasound if threads of a different size than the one left on the day of the IUD insertion	0	0	0	0	38	100	0	0	0	0

• IUD = Intrauterine device;STIs = Sexually transmitted infections; BHCG= Human chorionic gonadotropin hormone.

Discussion

The insertion of the IUD by nurses can improve access to an effective contraceptive method as well as increase the solvability of women’s health care. However, until now, the lack of a validated tool to assess the competence of nursing professionalsfor the insertion of IUD prevented assertive conclusions about the performance of these professionals. This study shows a tool developed for this purpose, which proved to be valid and reliable in the context in which it was tested, and can contribute as a guide tool for the education and training of nursing professionals in the context of IUD insertion. Thus, two major processes had to be involved in the construction of the instrument: the creation of the tool and the demonstration of evidence of validity and reliability.

The initial idea of this study was to develop a tool to be used by preceptor nursing professionals in the assessment of the knowledge, behavior, and skills of the nursing professional in training during the nursing consultation with a focus on reproductive planning and IUD insertion. Thus, knowledge, behavior, and skills were the domains that guided the elaboration of the items.

The learner’s knowledge is measured by the tool and translates the theoretical background acquired in the undergraduate course and the improvement course in reproductive planning. The behavior is related to the professional’s posture, decision-making, ethics, and ability to work in a team. The word skill, on the other hand, refers to the quality of being skillful, intelligent, of demonstrating aptitude, ingenuity, and dexterity,⁷ mainly in the execution of the technique. From this established panorama, the structure and sequencing of the evaluation instrument were initially defined.

Thus, the instrument was structured so that, in a visible and fast way, the teacher can perceive the evolution of the learner during the practical classes, and detail the attributes necessary for the development of competence in each consultation and the expected performance at the end of the practice training.

After defining the most relevant domains for the evaluated construct, considering the vast experience of the authors, questions related to the basic competences to be measured by the instrument were proposed. The choice of the type of scale used was guided by the advantages related to the ease of understanding and presence of quantifable response options. However, the dificulty in establishing precision with the amount of positive and negative responses shows a limitation in the use of Likert-type scale.²³

The tool presented apparent and content validity, and the analyses performed by the judges were important for the adequacy of the items and a better understanding of the instrument by the target population. Some items from the frst version of the tool were unifed into a single item, as they address similar issues. In this case, some were related to the approach of essential guidelines for women who wish to use IUD. These guidelines form the basis of high-quality care and include questions such as: what is the IUD, mechanism of action, duration of use, how the procedure will be performed, possible side effects, warning signs of complications, and when to seek the health service.³ Women must receive all guidelines and clarifications that involve the procedure in advance so that they can consciously decide on their consent,³ and this issue was duly addressed by the tool.

One of the objectives of residency programs in Brazil is to train recently graduated professionals to integrate teaching-service-community and increase the number of qualified professionals in the job market. In the test phase of the instrument, we verifed that the residency students in obstetric nursing involved in this study had little professional clinical experience before the residency program, as they were recent graduates, but still had a good evaluation of the items related to the Knowledge domains, Behaviors, and Basic Skills. Thus, the tool directs the teaching of nursing consultation with a focus on reproductive planning and IUD insertion and can be used by students with different professional experiences.

Specifically, a significant portion of residents in the skills domain (21.1%) failed to guide women regarding the importance of performing self-examinations to confirm the proper placement of the IUD. This step is important for continuity of care and greater safety of the method. This domain of skills was the one that received most indication for deletion or reformulation of items. The content validation phase resulted in the elimination of the item related to the orientation of possible side effects, warning signs, and when to seek help in the health service, as it addresses the moment after the insertion of the IUD.

However, some items were kept due to their relevance in the current clinical-epidemiological context. Among them, the item whose approach was the care, support, and referral or treatment of women with sexually transmitted infections (STIs). STIs have a direct impact on reproductive and child health, leading to infertility and complications in pregnancy and childbirth, causing fetal death and harm to the child’s health. They also have an indirect impact on facilitating the sexual transmission of the human immunodeficiency virus (HIV).²⁴ As the prevention and treatment of STIs interrupt the chain of transmission and prevent other infections and possible complications,²⁵ we considered it an important issue to be evaluated by nurses during the application of the IUD.

Among the important skills for the insertion of IUD, the performance of the bimanual pelvic exam is also highlighted,⁴,⁵ an item duly addressed by the tool for assessment of competences presented by this study. When performing the bimanual pelvic exam, nurses assess the size, position, consistency, and mobility of the uterus and identify tender points that may indicate the existence of an infection. A retroverted uterus requires correction with the traction of the Pozzi forceps during IUD insertion.³,¹⁸ A retrospective study²⁶ showed that uterine orientation may also be related to perforation. The study with 37 cases of uterine perforation showed that 42% of the perforations occurred in a retroverted uterus. Another consideration is not just the orientation of the uterus to the vagina (version), but also the orientation of the uterine body to the cervix (flexion). If an unexpected acute angle is present at any of these points, there may be an increased likelihood of perforation, especially if the Pozzi Forceps are not placed on the cervix to straighten the axis of insertion.

The item related to performing cervical antisepsis was also kept in the tool, as it is a skill that involves patient safety, being essential for the insertion of IUD. The MH recommends cleaning the cervix with a water-based antiseptic with iodoform or chlorhexidine hydrochloride. This simple procedure minimizes the chances of uterine infection after IUD insertion.³

Items related to performing a bimanual pelvic exam and performing cervical antisepsis had a low Cronbach’s Alpha Coefficient, probably due to the high divergence of conduct among residents. This divergence may be due to the resident’s teaching-learning process. During theoretical classes, the IUD insertion process is taught as recommended by the literature, but in practice, some preceptors do not follow the protocols and recommendations, influencing the residents’ performance. In this way, the value of this tool is because it identifes problems in training, as well as prepares them for teaching and learning.

A limitation of this study is that we tested the instrument with a limited number of obstetrical nursing professionals. Despite the HSF being one of the hospitals with the highest number of residents in obstetric nursing in the country, further research with samples of nurses isan improvement.²⁷ New studies should be conducted to assess the validity and reliability of this tool from self-application, to enable its use in the self-assessment of professionals. Future studies with a longitudinal design may produce additional evidence on the use of the instrument in the context of the assessment by competences of the nurse professional in the dynamic inserting of the IUD.

After the judges’evaluation and a pre-test applied to the obstetric nursing residents, we thinkthat the assessment tool by competence of the professional nurse in the insertion of the IUD is appropriate to the teaching-learning context, an important axis of the evaluative dimension in the scope of training of nurses in gynecological nursing consultations with IUD insertion. This tool was aimed at helping clearly and objectively both who is being trained and who is training. The tool was validated as to its content and adapted more specifically to its patients. The validity of an instrument requires a continuous process consisting of several studies and covering other criteria for validation and reliability testing, such as stability and equivalence, and can be tested in courses/training as an assessment and self-report assessment tool forpeople in training.

Acknowledgments

This work was carried out with the support of the Coordination for the Improvement of High Education Personnel- Brazil (CAPES – Portuguese acronym) - Financing Code 001.

References

• ¹
  Laporte M, Becerra A, Castro L, Veiga Junior N, Espejo-Arce X, Bahamondes L. Evaluation of clinical performance when intrauterine devicesare inserted by different categories of healthcare professional. Int J Gynecol Obstet. 2020 Feb; 152 (2): 196-201.
• ²
  Gonzaga VAS, Borges ALV, Santos OA, Rosa PLFS, Gonçalves RFS. Barreiras organizacionais para disponibilização e inserção do dispositivo intrauterino nos serviços de atenção básica à saúde. Rev Esc Enferm USP. 2017; 51: e03270.
• ³
  Ministry of Health (BR). Secretaria de Atenção à Saúde. Departamento de Ações Programáticas Estratégicas. Manual técnico para profissionais de saúde: DIU com cobre T Cu 380A. Brasília (DF): Ministry of Health; 2018.
• ⁴
  Nunavut Midwifery Registration Committee. Process for certification for intrauterine contraception. Alberta: IUC Certification Process; 2016.
• ⁵
  United States Agency for International Development (USAID). IUD guidelines for family planning service programs: a problem-solving reference manual. 3^rd ed. Baltimore: USAID; 2008.
• ⁶
  Coluci MZO, Alexandre NMC, Milani D. Construção de instrumentos de medida na área da saúde. Ciênc Saúde Coletiva. 2015; 20 (3): 925-36.
• ⁷
  Soares MI, Leal LA, Resck ZMR, Terra FS, Chaves LDP, Henriques SH. Competence-based performance evaluation in hospital nurses. Rev Latino-Am Enferm. 2019; 27: e3184.
• ⁸
  Holanda FL, Marra CC, Cunha ICKO. Assessment of professional competence of nurses in emergencies: created and validated instrument. Rev Bras Enferm. 2018; 71 (4): 1865-74.
• ⁹
  Miranda FBG, Mazzo A, Pereira Junior GA. Assessment of individual and interprofessional skills of health professionals in simulated clinical activities: a scoping review. Interface (Botucatu). 2018; 22 (67): 1221-34.
• ¹⁰
  Ouyang M, Peng K, Botfield JR, McGeechan K. Intrauterine contraceptive device training and outcomes for healthcare providers in developed countries: a systematic review. PLoS One. 2019 Jul; 14 (7): e0219746.
• ¹¹
  Pasquali L. Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.
• ¹²
  Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clin. 1998; 25 (5): 206-13.
• ¹³
  Fullerton JT, Ghérissi A, Johnson PG, Thompson JB. Competence and competency: core concepts for international midwifery practice. Int J Childbirth. 2011; 1 (1): 4-12.
• ¹⁴
  Cowan DT, Norman IJ, Coopamah VP. A project to establish a skills competency matrix for EU nurses. Br J Nurs. 2005 Jun; 14 (11): 613-7.
• ¹⁵
  International Confederation of Midwives (ICM). Essential competencies for midwifery practice [Internet]. Netherlands: ICM; 2019; [access in 2019 Sep 09]. Available from: https://www.internationalmidwives.org/our-work/policy-and-practice/icm-definitions.html
  » https://www.internationalmidwives.org/our-work/policy-and-practice/icm-definitions.html
• ¹⁶
  Kyei AA, Dennis-Antwi JA, Ibinga KR, Azfar P; Standard ICM. Competency-based list for basic skills training in midwifery schools: a reference guide for practical skills teaching. ICM/UNFPA Investing in midwives and others with midwifery skills to save the lives of mothers and newborns and improve their health. Netherlands: International Confederation of Midwives (ICM)/United Nations Population Fund (UNFPA); 2012.
• ¹⁷
  Organização Pan-Americana de Saúde (OPAS). Conjunto de ferramentas para o fortalecimento da obstetrícia. Montevidéu: Centro Latino-Americano de Perinatologia e Saúde da Mulher e Reprodutiva (CLAP/SMR); 2014.
• ¹⁸
  World Health Organization (WHO). Family planning: a global handbook for providers (2018 update). Geneva: WHO; 2018.
• ¹⁹
  Bermudes WL, Santana BT, Braga JHO, Souza PH. Tipos de escalas utilizadas em pesquisas e suas aplicações. VÉRTICES (Campos dos Goytacazes). 2016; 18 (2): 7-20.
• ²⁰
  Jebb AT, Ng V, Tay L. A review of key likert scale development advances: 1995-2019. Front Psychol. 2021 May; 12: 637547.
• ²¹
  Niederberger M, Spranger J. Delphi technique in health sciences: a map. Front Public Health. 2020; 8: 457.
• ²²
  Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saúde. 2017; 26 (3): 649-59.
• ²³
  Curado MAS, Teles J, Marôco J. Análise de variáveis não diretamente observáveis: influência na tomada de decisão durante o processo de investigação. Rev Esc Enferm USP. 2014 Feb; 48 (1): 149-56.
• ²⁴
  Ministry of Health (BR). Secretaria de Vigilância em Saúde. Boletim Epidemiológico. Sífilis [Internet]. Brasília (DF): Ministry of Health; 2020; [access in 2021 Aug 03]. Available from: file:///C:/Users/Helisamara/Downloads/boletim_sifilis_2020.pdf
  » file:///C:/Users/Helisamara/Downloads/boletim_sifilis_2020.pdf
• ²⁵
  Ministry of Health (BR). Departamento de Doenças de Condições Crônicas e Infecções Sexualmente Transmissíveis. Protocolo clínico e diretrizes terapêuticas para atenção integral às pessoas com infecções sexualmente transmissíveis (IST). Brasília (DF): Ministry of Health; 2020.
• ²⁶
  Kaislasuo J, Suhonen S, Gissler H, Lähteenmäki P, Heikinheimo O. Uterine perforation caused by intrauterine devices: clinical course and treatment. Hum Reprod. 2013 Jun; 28 (6): 1546-1551.
• ²⁷
  Lacerda LD, Arma JC, Paes LG, Siqueira EF, Ferreira LB, Fetzner RR, et al. Inserção de dispositivo intrauterino por enfermeiros da atenção primária à saúde. Enferm Foco. 2021; 12 (Suppl 1): 99-104.

Publication Dates

• Publication in this collection
  05 Aug 2022
• Date of issue
  Apr-Jun 2022

History

• Received
  03 Feb 2021
• Reviewed
  06 Dec 2021
• Accepted
  26 Dec 2021