Contributions of the Delphi technique to the validation of an occupational therapy assessment in the visual impairment field¹

Abstract

Introduction

The Delphi Technique is used to establish consensus on a given subject, in areas that still need construction, validation, revision, or better exploration of knowledge and intervention methods.

Objective

To show the contributions of the Delphi Technique as a validation strategy for an Occupational Therapy assessment for visually impaired persons.

Method

Quali-quantitative research, descriptive and exploratory, using the Delphi Technique as a validation strategy for the assessment. A non-probabilistic accessibility sample was constituted for the composition of a panel of experts. The questionnaires and the Assessment were sent, by e-mail, on the same date to all participants. An electronic device was exclusively used as a mechanism for sending and receiving documents. Data analysis took place during the collection, considering that the next round started only after analyzing the data from the previous questionnaire. To achieve consensus, the result of the round should have CVItotal≥0.90 and CVIitem≥0.78 and stability in the participants' suggestions.

Results

A panel was formed with 8 experts, held three rounds of the Technique, with a total duration of 12 months. In the third and last round, the CVItotal=0.97, and there were no suggestions that would cause changes in the Assessment. Therefore, it was considered valid.

Conclusion

The Delphi Technique proved to be advantageous allowing its online and anonymous performance among the participants, permitting the participation of professionals with heterogeneous expertise, who contributed and granted the establishment of consensus regarding the content, structure, language, organization, and the need for items for an Occupational Therapy assessment in visual impairment field, resulting in its validation.

Keywords:
Occupational Therapy; Visually Impaired Persons; International Classification of Functioning; Disability and Health; Disability Evaluation; Validation Study; Delphi Technique

Resumo

Introdução

A Técnica Delphi é utilizada para estabelecer consenso sobre determinado assunto, em áreas que precisam de construção, validação, revisão ou melhor exploração de conhecimentos e métodos de intervenção.

Objetivo

Apresentar as contribuições da Técnica Delphi como estratégia de validação de um instrumento em Terapia Ocupacional para avaliar adolescentes e adultos com deficiência visual.

Método

Pesquisa quali-quantitativa, exploratória e descritiva, com emprego da Técnica Delphi para validação do instrumento. Foi constituído um painel de especialistas por amostragem não probabilística por acessibilidade. Os questionários de opinião sobre a avaliação e a avaliação propriamente dita foram enviados na mesma data para todos os participantes. Utilizou-se exclusivamente o meio eletrônico como mecanismo de envio e recepção dos documentos. A análise dos dados transcorreu na coleta, tendo em vista que a rodada seguinte tinha início apenas após a análise do questionário anterior. Para obtenção do consenso, o resultado da rodada deveria apresentar IVCtotal≥0,90, IVCitem≥0,78 e estabilidade nas sugestões dos participantes.

Resultados

Foi formado um painel com 8 especialistas, realizadas três rodadas da Técnica, com duração total de 12 meses. Na terceira e última rodada, o IVCtotal=0,97 e não houve sugestões que provocassem alterações na avaliação, sendo considerada válida.

Conclusão

A Técnica Delphi se mostrou vantajosa por permitir sua realização online e anônima entre os participantes, possibilitando a participação de profissionais com expertises heterogêneas, que contribuíram e permitiram o estabelecimento do consenso quanto ao conteúdo, estrutura, linguagem, organização e a necessidade de itens para uma avaliação terapêutica ocupacional na área da deficiência visual, resultando em sua validação.

Palavras-chave:
Terapia Ocupacional; Pessoas com Deficiência Visual; Classificação Internacional de Funcionalidade; Incapacidade e Saúde; Avaliação da Deficiência; Estudo de Validação; Técnica Delfos

Introduction

The Delphi technique was first described during the Cold War, at the time, used for military purposes (Grisham, 2009; Habibi et al., 2014; Waggoner et al., 2016; Massaroli et al., 2017; Marques & Freitas, 2018). It expanded to the scientific universe from 1990 onwards and is currently used in several areas of knowledge, including research in health and education (Faro, 1997; Almeida et al., 2009; Habibi et al., 2014; Marques & Freitas, 2018).

This technique is used to establish consensus on a given topic, and it is a tool in areas where knowledge still needs construction, validation, revision, or better exploration (Powell, 2003; Habibi et al., 2014; Revorêdo et al., 2015; Massaroli et al., 2017). Inspired by the oracle of the Greek temple of Apollo, Delphi, it seeks to consult specialists or experts for future projections or to better understand certain themes, being applied in rounds of questionnaires (Faro, 1997; Kayo & Securato, 1997; Powell, 2003; Almeida et al., 2009; Grisham, 2009; Habibi et al., 2014; Waggoner et al., 2016; Marques & Freitas, 2018). The name of the Delphi Technique has already been criticized for its reference to the mystical universe, although it does not have this character (Massaroli et al., 2017).

With a mixed approach, it uses qualitative and quantitative strategies for data collection and analysis (Revorêdo et al., 2015; Massaroli et al., 2017). Through descriptive statistics, the frequency and percentage of opinions are calculated and the Delphi study is followed until the level of agreement established a priori is obtained, also carrying out a qualitative analysis of the participants' comments and suggestions, incorporating them, when relevant, to the product being validated (Revorêdo et al., 2015; Marques & Freitas, 2018).

The Delphi Technique has some features that are considered advantageous. The anonymity among the participants avoids embarrassment, inhibitions, and intimidation that can occur in face-to-face meetings, allowing the expression of comments from everyone, even those judges whose opinions are a minority (Kayo & Securato, 1997; Waggoner et al., 2016; Marques & Freitas, 2018). Other advantages relate to the possibility of providing participants with feedback for their contributions, presentation by those responsible for the study of what is being built by the group, the possibility of reviewing the experts' answers, as well as the formation of heterogeneous groups when the subject or product to be validated is multidimensional and multidisciplinary (Grant & Davis, 1997; Powell, 2003; Habibi et al., 2014; Revorêdo et al., 2015; Waggoner et al., 2016; Marques & Freitas, 2018). The opportunity to apply the Technique in a virtual environment is identified as another differential and advantage for its use, which allows more time for the participant to reflect on the phenomenon studied, can lead to greater adherence, participation, in addition to lower cost and time spent for its performance (Grisham, 2009; Coutinho et al., 2013; Waggoner et al., 2016; Marques & Freitas, 2018).

The Technique follows a few steps. The first one is the definition of the facilitator, who is responsible for analyzing and synthesizing the answers and generating new consultations with the participants through a new questionnaire. Afterward, criteria for choosing the participants of the expert panel and the information collection strategy are defined, either through a questionnaire, interview script, or poll.

The literature suggests the questionnaire as one of the best formats for performing the Technique, although there is no standardized model for its construction (Faro, 1997; Almeida et al., 2009). Some authors point out that the instrument is carefully designed so that it is not extensive, promoting the participants to leave, nor reduced to the point of generating the possibility of automatic responses, with insufficient reflection on the phenomenon (Faro, 1997; Kayo & Securato, 1997; Marques & Freitas, 2018). Questionnaires should provide spaces to encourage participants to make their comments without being tied to structured answers (Kayo & Securato, 1997).

For the formation of the expert panel, the number of participants is still not a consensus in the literature (Faro, 1997; Grant & Davis, 1997; Powell, 2003; Habibi et al., 2014; Waggoner et al., 2016; Marques & Freitas, 2018). According to Lynn (1986), a minimum of 5 participants is required. The maximum number was not established; however, it is suggested that the participation of 10 experts not be exceeded. According to this author (Lynn, 1986), the number of participating judges will depend on how many experts can be identified, how many are accessible and agree to participate.

Another reference shows a wide variety in the number of participants. Coutinho et al. (2013), who carried out a systematic review of the literature on the technique, suggest that the decision on the number of participating judges will vary with the nature of the object of study, which may indicate greater or lesser availability of people. As a result, they identified surveys with a range of 6 to 305 participating expert judges (Coutinho et al., 2013).

Powell (2003) argues that, although there is no consensus on the number of participants, it is not necessary to organize a panel with a statistically representative number, since the representativeness for the Delphi Technique is in the quality of the formed panel. Coutinho et al. (2013), in their systematic review, found that the rate of return was inversely proportional to the number of research participants who used the Delphi Technique. Thus, the greater the number of participants, as new rounds were carried out, the fewer people maintained their participation. The occurrence of a decrease in participants as the number of rounds increases is also stated in the research by Waggoner et al. (2016).

It begins with the distribution of a broad and exploratory questionnaire on the topic under study. From the compilation of the participants' answers, new questionnaires are constructed and modifications to the product are suggested, seeking to deepen the opinions and improve the items of the evaluation instrument. The new rounds of questionnaire distribution and data analysis should take place until the answers are stable, without new suggestions for changes, low or no divergence, and when consensus is established (Faro, 1997; Kayo & Securato, 1997; Powell, 2003; Almeida, 2004; Almeida et al., 2008, 2009; Grisham, 2009; Habibi et al., 2014; Revorêdo et al., 2015; Marques & Freitas, 2018).

The quantitative determination of consensus in a survey using the Delphi Technique may vary. Almeida et al. (2009) in their research, established the agreement rate of 0.70 or 70%. In a systematic literature review, Coutinho et al. (2013) identified twenty scientific articles that used the Delphi Technique in research related to primary health care. The authors analyzed that the choices of consensus values were different. For example, in five articles the minimum value as an agreement criterion was 75%, in other studies, it was considered 80%, and two did not show the minimum agreement (Coutinho et al., 2013).

Alexandre & Coluci (2011) argue that the number of participants must be considered to establish the Content Validity Index (CVI). For surveys with less than 5 judges, everyone must agree to be considered a consensus; in surveys with more than 6 participants, an agreement rate greater than 0.78 or 78% should be considered (Lynn, 1986; Polit & Beck, 2006; Alexandre & Coluci, 2011).

We can calculate two types of CVI. The CVI per item, which refers to the agreement among the participants regarding each of the items in the round, and the total CVI, which corresponds to the agreement of the questionnaire for the round as a whole. However, not only the index or degree of the agreement must be considered to establish a consensus, but also when there are no more contributions and suggestions from experts that are relevant to the objectives of the study and that generate changes in the product (Faro, 1997; Powell, 2003; Almeida, 2004; Almeida et al., 2008, 2009; Grisham, 2009; Habibi et al., 2014; Revorêdo et al., 2015; Marques & Freitas, 2018).

The integrative review by Revorêdo et al. (2015), regarding the use of the Delphi Technique in health research, identified that most of the publications found referred to studies on the creation and validation of assessment instruments. Corroborating the finding by Revorêdo et al. (2015), this manuscript presents the results of doctoral research carried out to establish the face and content validity of an assessment, the Occupational Therapeutic Assessment (OTA) for Adolescents and Adults with Visual Impairment (Silva, 2020; Silva & Montilha, 2020, 2021). The OTA is a direct interview to identify occupational performance, daily influencers, interests, priorities, expectations, and the performance of functional tasks to observe capabilities in a standard environment and their subsequent qualification in the International Classification of Functioning, Disability, and Health (ICF) (Silva, 2020; Silva & Montilha, 2020, 2021).

To establish the validity of the OTA, the Delphi Technique was used as a tool. Thus, this paper aims to highlight the contributions of the Technique as a validation strategy for a new assessment instrument in Occupational Therapy, in the area of visual impairment.

Methodology

This is a quali-quantitative, exploratory and descriptive research, using the Delphi Technique, in three rounds for the face and content validation of an assessment instrument in Occupational Therapy in the area of visual impairment (Gil, 2008).

Location

We only used electronic means as a mechanism for sending and receiving documents during the validation process, with no need for a specific location and the displacement of the expert panel.

Participants

A non-probabilistic sampling for accessibility or convenience was constituted (Gil, 2008). A panel of expert judges was composed for the validation process, following the face and content validation strategy and the Delphi Technique.

Expert judges included health professionals involved in research and/or teaching and/or clinical practice, working with visual impairment and/or with knowledge in the process of construction and validation of assessment instruments and/or experts on the International Classification of Functioning, Disability, and Health (ICF). We excluded professionals who were involved, at any time, in the construction of the OTA and those with a conflict of interest with the research.

To compose the panel, each participant had to accept and sign the Informed Consent Form (ICF). The ICF was sent by e-mail and should be answered and duly signed also digitally.

Data collection procedure

All data collection procedures were developed electronically. Invitations to health professionals who matched the research criteria were sent by e-mail, together with the ICF. The sending and receiving of all signed documents were carried out exclusively by e-mail, within a previously stipulated period. After the signed consent forms were returned, the 1^st round of the Delphi study began, when the OTA and the 1^st questionnaire were sent, on the same date, to all participants, for consultation with experts on the assessment. The expert had access to the questionnaire, built with the Google Forms tool, through a link. All other rounds of the Technique took place through the same online strategy with submission of the OTA and subsequent questionnaires.

Data analysis procedures

Data analysis of the face and content validation process was performed using descriptive, qualitative, and quantitative strategies to interpret the results of the collections carried out with expert judges in the instrument validation process.

Surveys that use the Delphi Technique have data analysis procedures taking place during collection, considering that the next round starts only after analyzing the data from the previous questionnaire (Massaroli et al., 2017).

The Content Validity Indices (CVI) per item (CVIitem) and total (CVItotal) were calculated using the formulas suggested in the reference literature (Polit & Beck, 2006). For the Content Validity Index per item (IVCitem) the formula used was: CVI = number of positive responses/total number of responses. The calculation of the total CVI can be obtained in three different ways: CVItotal = number of positive answers/(nº judges X nº items); or the sum of each judge's agreement averages divided by the total number of judges; as well as the result of the average of the item CVI values, adding them and dividing by the number of items. The three forms of calculation must result in the same value (Polit & Beck, 2006).

According to the literature, to establish validity in new assessment instruments, the expected default value for the calculations of the total CVI is ≥0.90 or 90% and for the item CVI, it is ≥0.78 or 78% (Lynn, 1986; Polit & Beck, 2006).

Thus, the research reported here considered such default values of item CVI and total CVI to establish the validity of the Occupational Therapeutic Assessment (OTA) for Adolescents and Adults with Visual Impairment. The question that obtained ≤0.75 or 75% of agreement among the expert judges was compulsorily revised and the item corresponding to the OTA was compulsorily modified or excluded (Lynn, 1986). The question with an index between 0.75 (75%) and 0.78 (78%) was revised; however, not necessarily excluded or changed.

Ethical aspects

This research was approved by the Research Ethics Committee of the Faculty of Medical Sciences, State University of Campinas, under opinion nº 2.167.526/17.

Results and Discussion

The results described below aim to highlight the contributions of the Delphi Technique in the face and content validation process of the Occupational Therapeutic Assessment (OTA) for Adolescents and Adults with Visual Impairment².

Figure 1 shows the steps of the validation process using the Delphi Technique and Table 1 shows the rounds, the materials analyzed by the participants, and the summarized results.

Figure 1
Stages of the validation process using the Delphi Technique.

Table 1
Rounds, analyzed materials and summarized results.

Delphi Technique Round		Result
1st Round	OTA1 submission	Adequacy of OTA1 – OTA2
	Judgment of clarity, objectivity, content, language, structure	Construction of a manual
		CVIT=0,892 (89,2%)
2nd Round	OTA2 submission	Adequacy of OTA2 – OTA3
	Judging the adjustments of the 1st Round	CVIT=0,941 (94,1%)
3rd Round	OTA3 submission	Consensus
	Judging the adaptations of the 2nd Round	CVIT=0,979 (97,9%)

For the development of this research, we invited 13 people, including occupational therapists, speech therapists, and ophthalmologists. Potential participants were identified based on the author's knowledge and in scientific articles in related areas that matched the inclusion and exclusion criteria.

Invitations were made by email. Professionals who did not have an electronic contact known to the researcher were asked about their interest in participating via cell phone text messages, briefly explaining the purpose of the research and requesting the email for the formal invitation, and sending the consent form. Invitations took place between September and December 2017. After the start of the 1^st round of the study, new participants would no longer be accepted.

Nine professionals agreed to participate; however, eight returned the emails with signed consent forms. During the entire process, the eight people remained as research participants, answering the questionnaires whenever requested.

According to Grisham (2009), it is very important to select a panel that has a balance between impartiality and interest in the topic, with knowledge in the area and commitment to participate in a process that usually consists of more than one round. For Marques & Freitas (2018), the commitment and willingness of the participants are fundamental, since losing research individuals, when it involves multiple rounds, is considered common.

The findings of the aforementioned surveys show the importance of care that should be taken when choosing the expert panel of judges, which justified the attention given in this OTA validation survey for the selection of its participants, using the convenience sampling method.

In this research, four occupational therapists and four speech therapists participated. During the development of the Technique, one participant declared a master's degree in progress, three with a completed master's degree, three with a doctoral degree in progress, and one with a completed doctoral degree. Regarding knowledge and experience, all reported being familiar with the ICF, five reported being visually impaired and visually impaired, four reported having knowledge and experience in functional vision assessment and Occupational Therapy, three judges reported knowledge of the assessment process in Occupational Therapy, and one in the validation and reliability processes of assessment instruments.

The 1^st round of the Delphi Technique started in December 2017. An email was sent, on the same date, to all participants, with the questionnaire link (built on the Google Forms platform), so that they could give their opinion about the Assessment, and the OTA as an attachment in Portable Document Format – pdf. The deadline established for the return of this round was January 2018.

The questionnaire about the OTA consisted of thirteen questions (Tables 2 and 3). Some questions enabled to issue an opinion on a five-point scale, ranging from Strongly Agree to Completely Disagree (Table 2). Other questions asked for answers on a dichotomous scale (yes/no) (Table 3), in addition to spaces for participant dissertation, providing opinions and comments, allowing the improvement of the OTA and deepening of opinions on the referred assessment by experts in the 2^nd round.

Table 2
1^st round of Delphi Technical: Expert judgment on OTA₁ – Five-point assessment.

Statements	Degree of agreement
	CVI item % n = 8
	CD	PD	NN	PA	CA	IVC item
1. In my opinion, the way OTA is organized is adequate.	-	-	25%	12.5%	62.5%	0.75 or 75%
2. In my opinion, the order of topics in the OTA is adequate.	-	-	12.5%	12.5%	75%	0.875 or 87.5%
3. In my opinion, the OTA is written in an accessible language.	-	-	-	37.5%	62.5%	0.1 or 100%
4. In my opinion, the OTA is clearly written.	-	-	-	25%	75%	0.1 or 100%
5. In my opinion, the OTA is written objectively.	-	-	12.5%	12.5%	75%	0.875 or 87.5%
6. In my opinion, the OTA presents enough information for your understanding.	-	12.5%	12.5%	25%	50%	0.75 or 75%

• CD: Completely disagree; PD: Partially disagree; NN: Neither agree nor disagree; PA: Partially agree; CA: Completely agree. Adapted from Silva (2020).

Table 3
1^st Delphi Technical round: Expert judgment on OTA₁ - Two-point assessment.

Questions	Degree of agreement
	CVI item % n = 8
	Yes	No
7. Would you change the sequence of the OTA items? (current sequence: general instructions, personal data, visual aspects, semi-structured interview, practical activity, instructions and tables for practical activity)	62.5%	37.5%
8. Does the OTA have an adequate extension, considering the aspects it intends to assess?	87.5%	12.5%
9. Does OTA have any words that are difficult to understand?	-	100%
10. Does OTA have any words that have a double meaning or double interpretation?	-	100%
11. I believe that OTA includes:
11.1 Visual aspects	100%	-
11.2 Functionality of the visually impaired person	100%	-
11.3 Daily difficulties of the visually impaired person	100%	-
11.4 Restrictions on the participation of the visually impaired person	75%	25%
11.5 Limitation in carrying out the activities of the visually impaired person	100%	-
11.6 Abilities of the visually impaired person	87.5%	12.5%
11.7 Interests of the visually impaired person	87.5%	12.5%
11.8 Expectations of the visually impaired person	75%	25%
11.9 Environmental barriers for the visually impaired person	100%	-
11.10 Environmental facilitators of the visually impaired person	87.5%	12.5%
12. Has the OTA, as a whole, proved to be feasible to be applied?	100%	-
13. In your opinion, is the name of OTA appropriate? (Occupational Therapeutic Assessment for Adolescents and Adults with Visual Impairment)	100%	-

• Adapted from Silva (2020).

The total Content Validity Index (total CVI) obtained in the 1^st round was 0.892 (89.2%), which still does not allow us to consider the OTA valid. One question had a CVI item = 0.625 (62.5%), requiring mandatory changes, and another four questions had a CVI item of 0.75 (75%), requiring revision and adjustment.

All answers were analyzed considering the CVI; however, even questions with an agreement rate ≥of 0.78 or 78% that contained comments, suggestions, and opinions were considered, allowing for modifications and additions to the Occupational Therapeutic Assessment.

Some considerations from expert judges required individual feedback by email to justify the non-inclusion or adequacy of the material, for example, by suggesting the addition of aspects that are not part of the occupational therapist's professional competences.

We changed the OTA based on the suggestions of the 1^st round, didactically calling it OTA₂. A new questionnaire was built, deepening some questions and anonymously sharing the opinions generated in the 1^st round that led to changes in the Assessment.

The start of the 2^nd round of the Delphi Technique took place in March 2018. The OTA² and the link to the new questionnaire were sent by email. The initial deadline for returns was April 2018; however, there was a request for it to be extended, becoming May 2018.

The questionnaire for this round had seventeen questions with the possibility of answers in a dichotomous scale (yes/no) and an open field, in each question, for the participants to contribute with their opinions (Table 4). At the end of the questionnaire, another open space invited the experts to observe and suggest aspects that were not covered in the structured questions.

Table 4
2^nd round Delphi Technical: Expert judges' judgment on the OTA₂ - Two-point evaluation.

Questions	Degree of agreement
	CVI item % n = 8
	Yes	No
1. Construction of a manual with information on the purpose of the OTA, evaluation structure, application methods, and terminology used. Do you believe that the manual constructed has enough information for the application of the OTA by occupational therapists?	100%	-
2. Inclusion of a brief explanation about the ICF, its objectives, components, and qualifiers. Do you think it is important to have a brief explanation of the ICF in the manual?	75%	25%
3. Inclusion of a brief explanation about the ICF, its objectives, components, and qualifiers. Do you believe that the information provided about the ICF is sufficient for the application of the OTA?	100%	-
4. Question-specific instructions were placed immediately before the question that concerns it. Do you agree with the new layout of the instructions for each question?	87.5%	12.5%
5. Adequacy of the numbering of tables and charts; review of the question numbers. Do you agree with these changes?	100%	-
6. Inclusion, in the personal data item, about religion. Do you agree with the addition of this question?	87.5%	12.5%
7. Better descriptions regarding education and professional activity, in the personal data item. Do you agree with these changes?	87.5%	12.5%
8. Change and adequacy of question 4.4 “Difficulties in basic and instrumental activities of daily living”. Do you agree with the change in the table of activities with the addition of the magnitude of difficulty table?	100%	-
9. Inclusion of “intimate relationships” in the table of activities with possible difficulty (question 4.4). Do you agree with the inclusion of this activity?	100%	-
10. In question 4.5, do you agree with the new format when asking about priority activities for the rehabilitation process?	100%	-
11. Inclusion of general questions about household activities, in question 5. Do you agree with the inclusion of questions regarding activities and the home environment?	100%	-
12. Regarding question 5.1., do you believe that the questions asked are relevant?	87.5%	12.5%
13. Inclusion and adequacy of questions in professional activities, in question 5.3. Do you agree with such changes and adjustments?	87.5%	12.5%
14. Adequacy and inclusion of instructions for applying question 6, “interpersonal relationships”. Do you believe that now, with the instructions present, question 6 is easier to interpret and ask the subject?	100%	-
15. In question 7, “socio-emotional aspect”, questions were included. Do you agree with the addition?	100%	-
16. Inclusion of a final table describing the codes used in the OTA. Do you think it is important to have the picture at the end of the OTA?	87.5%	12.5%
17. Inclusion of suggested recipes to carry out the practical cooking activity. Do you believe it is important for the recipes to be included at the end of the OTA?	100%	-

• Adapted from Silva (2020).

The total CVI was 0.941 (94.1%), an agreement between participants following standardized values of total CVI≥0.90 (90%). After analysis, no question in the 2^nd round received a CVI item <0.75 (75%). One question received a CVI item=0.75 (75%), being compulsorily revised and adequate.

Although the total CVI value, if analyzed in isolation, is sufficient to consider the OTA valid, the expert judges contributed with relevant suggestions for an initial assessment in Occupational Therapy in the area of visual impairment, leading to changes in the OTA and preventing the Assessment was considered valid in face and content.

To consider the consensus, based on the Delphi Technique, it is necessary not only statistical convergence, through the Content Validity Index calculations, but also little or no divergence of opinions and absence of suggestions that are important to the proposal and agreed with the literature in the area (Faro, 1997; Powell, 2003; Almeida, 2004; Almeida et al., 2008; Almeida et al., 2009; Grisham, 2009; Habibi et al., 2014; Revorêdo et al., 2015; Marques & Freitas, 2018). Quantitative and qualitative consensus is necessary, the latter being possible only if open spaces are made available (Kayo & Securato, 1997).

As a result of the 2^nd round, the new version of the OTA was formulated, called OTA₃, with aesthetic adjustments, reduced content, more objective, enabling and encouraging the occupational therapist professional to consult the sources in the literature for theoretical deepening, in addition to the addition and reorganization of items in the Assessment.

The 3^rd round of the Delphi Technique began in early June 2018, with the submission of the OTA₃ and the link to the third questionnaire to the eight participants. The deadline for returning the responses from the round was scheduled for the end of the same month; however, an extension of the return was requested, starting in August 2018.

The 3^rd round questionnaire had six questions with the possibility of answers in a dichotomous scale (yes/no) and a space, after each question, for the experts to lecture (Table 5). This was built based on the suggestions of the participants obtained in the 2^nd round and on the changes that occurred in the Occupational Therapeutic Assessment so that such changes could be evaluated and judged.

Table 5
3^rd round of Delphi Technical: Experts´ judgment on OTA₃ - Two-point assessment.

Questions	Degree of agreement
	CVI item % n = 8
	Yes	No
1. The manual has changed, resulting in a more succinct body text. Do you agree with the changes to the manual as a whole?	100%	-
2. The text on the explanation of the ICF changed, starting to be presented with fewer details about its applicability, structure, and codes. The evaluator is invited to consult the World Health Organization references on the matter. Do you agree with the change made?	100%	-
3. In the item “personal data – education” the information “postgraduate” was added. Do you agree with the insertion?	100%	-
4. The instructions for each item were highlighted in frames for an aesthetic separation. Do you agree with the change and the new presentation?	100%	-
5. Item 5.1 “Household activities” has been reorganized and a question has been added. Do you agree with the changes made?	100%	-
6. After the changes resulting from the 1st and 2nd rounds, do you believe the OTA is ready to be used?	87.5%	12.5%

• Adapted from Silva (2020).

In September 2018, the analyzes of the 3^rd round were concluded. No item received agreement ≤0.78 or 78% and needed to be compulsorily revised or excluded from the Assessment. The total CVI was 0.97 (97.91%), an index higher than the standard to consider the OTA valid. The comments of the participants in this round did not lead to changes in the Assessment that would justify carrying out a new round of the Technique. If the experts presented new contributions or divergences about the changes in the Assessment carried out so far, a new round of questionnaires would be organized and would generate new changes in the OTA for a new opportunity for judgment.

Thus, with agreement indices above the standard established in the literature and with the stability of the suggestions for changes in the OTA, it was allowed to be considered valid.

In total, we carried out three rounds of the Technique, which is following the literature. Although it does not establish a limit, the literature points to the performance of two to four rounds as usual for reaching consensus (Kayo & Securato, 1997; Powell, 2003; Almeida et al., 2009; Waggoner et al., 2016; Massaroli et al., 2017; Marques & Freitas, 2018). The collaborations for the validation were obtained from complementary and multidisciplinary expertise, which allowed for rich and varied suggestions and comments, and the expansion of discussions. The heterogeneity of participating judges is referenced in some scientific articles as being ideal for exploratory research using the technique (Grant & Davis, 1997; Powell, 2003; Habibi et al., 2014; Revorêdo et al., 2015; Waggoner et al., 2016; Marques & Freitas, 2018).

Finally, the possibility of open spaces for judges to speak and have the freedom to offer their opinions was another aspect that favored the deepening of issues and changes (Kayo & Securato, 1997). The opening of the space for comments allowed the three rounds of the Technique since the considerations in the questionnaires were essential for the refinement and validation of the Occupational Therapeutic Assessment (OTA) for Adolescents and Adults with Visual Impairment.

Conclusion

The Delphi Technique proved to be an advantageous tool for establishing content and face validity, since the distance and online strategy on the chosen platform, without the displacement of participants, avoided research costs, which would be generated in meetings face-to-face, facilitated the process of sending and receiving materials, the systematization of the data obtained in each round, the engagement and continuity of participation. In this research, there was no loss of expert judges between the rounds, all participants remained active throughout the process.

The distance and anonymity allowed all judges to express their opinions without being overlapped by the other participants, no discomfort was reported during the process. The Delphi Technique also allowed participants to review their suggestions at each new round and the provision of open spaces favored the deepening of the questionnaires and changes in the Assessment, which, only through statistical analysis with CVI calculations, would not be possible.

The deadlines were made more flexible during the process, due to particular needs expressed by some experts, causing the total execution time of the Technique to be extended. From sending the invitations to the analysis of the 3^rd round, which ruled out the need for a new round, it took twelve uninterrupted months of data collection and analysis using the Delphi Technique as a tool.

The time spent to participate in the research, which includes - in more than one round - the participants receiving the material, critical reading and reflection to answer the questionnaire questions, and the subsequent sending of the answers to the researcher, can be considered a risk for the loss of experts during the validation process. Although the number of participants remained unchanged during this study, the significant drop in experts can impair the validation of an assessment instrument. Therefore, the choice of the expert panel must be attentive and careful by the responsible researchers.

The tool enabled adaptation of the Assessment based on complementary knowledge and expertise, with professionals involved in one or more areas related to the topic, enriching the suggestions and, as a consequence, the Occupational Therapeutic Assessment. A difficulty found in the selection stage of potential participants was the identification and location of professionals who matched the inclusion criteria, willing to participate in research that implies dedication, motivation, interest, and sufficient knowledge to maintain the involvement and availability to contribute in-depth to the material.

Finally, the Delphi Technique proves to be powerful in aiding scientific development and should be a more explored tool for the elaboration, construction, identification of indicators, and validation of health assessments. After the entire validation process of this research has elapsed, the Technique is identified as a possible strategy for establishing consensus on the content, structure, language and organization, and items for an Occupational Therapy assessment, enabling the participation of professionals, with training and heterogeneous expertise, which contributed and allowed the validation of an assessment in Occupational Therapy in the area of visual impairment.

• 1
  This article is part of the doctoral research developed in the Graduate Program in Health, Interdisciplinarity and Rehabilitation/Faculty of Medical Sciences/UNICAMP, entitled “Validation of Occupational Therapy Assessment for Adolescents and Adults with Visual Impairment”. The current ethical procedures were complied with.
• 2
  The questionnaires, modifications and results of the validation process are available in Silva (2020) and Silva & Montilha (2020).
• How to cite: Silva, M. R., & Montilha, R. C. I. (2021). Contributions of the Delphi technique to the validation of an Occupational Therapy assessment in the visual impairment field. Cadernos Brasileiros de Terapia Ocupacional, 29, e2863. https://doi.org/10.1590/2526-8910.ctoAO2163

Referências

• Alexandre, N. M. C., & Coluci, M. Z. O. (2011). Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciência & Saúde Coletiva, 16(7), 3061-3068. http://dx.doi.org/10.1590/S1413-81232011000800006
  » http://dx.doi.org/10.1590/S1413-81232011000800006
• Almeida, M. H. M. (2004). Elaboração e validação do instrumento CICAc: classificação de idosos quanto à capacidade para o autocuidado. Revista de Terapia Ocupacional da Universidade de São Paulo, 15(3), 112-120. http://dx.doi.org/10.11606/issn.2238-6149.v15i3p112-120
  » http://dx.doi.org/10.11606/issn.2238-6149.v15i3p112-120
• Almeida, M. H. M., Spínola, A. W. P., Iwamizu, P. S., Okura, R. I. S., Barroso, L. P., & Lima, A. C. P. (2008). Confiabilidade do instrumento para classificação de idosos quanto à capacidade para o autocuidado. Revista de Saúde Pública, 42(2), 317-323. http://dx.doi.org/10.1590/S0034-89102008000200018
  » http://dx.doi.org/10.1590/S0034-89102008000200018
• Almeida, M. H. M., Spínola, A. W. P., & Lancman, S. (2009). Técnica Delphi: validação de um instrumento para uso do terapeuta ocupacional em gerontologia. Revista de Terapia Ocupacional da Universidade de São Paulo, 20(1), 49-58. http://dx.doi.org/10.11606/issn.2238-6149.v20i1p49-58
  » http://dx.doi.org/10.11606/issn.2238-6149.v20i1p49-58
• Coutinho, S. S., Freitas, M. A., Pereira, M. J. B., Veiga, T. B., Ferreira, M., & Mishima, S. M. (2013). O uso da técnica Delphi na pesquisa em atenção primária à saúde: revisão integrativa. Revista Baiana de Saúde Pública,37(3), 582-596. http://dx.doi.org/10.22278/2318-2660.2013.v37.n3.a398
  » http://dx.doi.org/10.22278/2318-2660.2013.v37.n3.a398
• Faro, A. C. M. (1997). Técnica Delphi na validação das intervenções de enfermagem. Revista da Escola de Enfermagem da USP, 31(2), 259-273. http://dx.doi.org/10.1590/S0080-62341997000200008
  » http://dx.doi.org/10.1590/S0080-62341997000200008
• Gil, A. C. (2008). Métodos e técnicas de pesquisa social. São Paulo: Atlas.
• Grant, J. S., & Davis, L. L. (1997). Selection and use of content experts for instrument development. Research in Nursing & Health, 20(3), 269-274. http://dx.doi.org/10.1002/(SICI)1098-240X(199706)20:3<269::AID-NUR9>3.0.CO;2-G
  » http://dx.doi.org/10.1002/(SICI)1098-240X(199706)20:3<269::AID-NUR9>3.0.CO;2-G
• Grisham, T. (2009). The Delphi technique: a method for testing complex and multifaceted topics. International Journal of Managing Projects in Business, 2(1), 112-130. http://dx.doi.org/10.1108/17538370910930545
  » http://dx.doi.org/10.1108/17538370910930545
• Habibi, A., Sarafrazi, A., & Izadyar, S. (2014). Delphi technique theoretical framework in qualitative research. The IJES, 3(4), 8-13.
• Kayo, E. K., & Securato, J. R. (1997). Método Delphi: fundamentos, críticas e vieses. Cadernos de Pesquisa em Administração, 1(4), 51-61.
• Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382-385. http://dx.doi.org/10.1097/00006199-198611000-00017
  » http://dx.doi.org/10.1097/00006199-198611000-00017
• Marques, J. B. V., & Freitas, D. (2018). Método Delphi: caracterização e potencialidades na pesquisa em Educação. Pro-Posições, 29(2), 389-415. http://dx.doi.org/10.1590/1980-6248-2015-0140
  » http://dx.doi.org/10.1590/1980-6248-2015-0140
• Massaroli, A., Martini, J. G., Lino, M. M., Spenassato, D., & Massaroli, R. (2017). Método Delphi como referencial metodológico para a pesquisa em enfermagem. Texto & Contexto Enfermagem, 26(4), 1-9. http://dx.doi.org/10.1590/0104-07072017001110017.
  » https://doi.org/10.1590/0104-07072017001110017
• Polit, D. F., & Beck, C. T. (2006). The content validity index: are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489-497. http://dx.doi.org/10.1002/nur.20147
  » http://dx.doi.org/10.1002/nur.20147
• Powell, C. (2003). The Delphi technique: myths and realities. Journal of Advanced Nursing, 41(4), 376-382. http://dx.doi.org/10.1046/j.1365-2648.2003.02537.x
  » http://dx.doi.org/10.1046/j.1365-2648.2003.02537.x
• Revorêdo, L. S., Maia, R. S., Torres, G. V., & Maia, E. M. C. (2015). O uso da técnica Delphi em saúde: uma revisão integrativa de estudos brasileiros. Arquivos de Ciências da Saúde, 22(2), 16-21. http://dx.doi.org/10.17696/2318-3691.22.2.2015.136
  » http://dx.doi.org/10.17696/2318-3691.22.2.2015.136
• Silva, M. R. (2020). Validação da Avaliação Terapêutica Ocupacional (ATO) para Adolescentes e adultos com deficiência visual (Tese de doutorado). Universidade Estadual de Campinas, Campinas.
• Silva, M. R., & Montilha, R. C. I. (2020). Validação da Avaliação Terapêutica Ocupacional (ATO) para adolescentes de adultos com deficiência visual por painel de especialistas. Revista de Ocupación Humana, 20(1), 27-48. http://dx.doi.org/10.25214/25907816.947
  » http://dx.doi.org/10.25214/25907816.947
• Silva, M. R., & Montilha, R. C. I. (2021). A avaliação terapêutica ocupacional (ATO) para adolescentes e adultos com deficiência visual. Revista Interinstitucional Brasileira de Terapia Ocupacional, 5(2), 252-265. http://dx.doi.org/10.47222/2526-3544.rbto38127
  » http://dx.doi.org/10.47222/2526-3544.rbto38127
• Waggoner, J., Carline, J., & Durning, S. J. (2016). Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Academic Medicine, 91(5), 663-668. http://dx.doi.org/10.1097/ACM.0000000000001092
  » http://dx.doi.org/10.1097/ACM.0000000000001092

Edited by

Section editor Profa. Dra. Marcia Maria Pires Camargo Novelli

Publication Dates

• Publication in this collection
  01 Oct 2021
• Date of issue
  2021

History

• Received
  25 Sept 2020
• Reviewed
  04 Feb 2021
• Reviewed
  05 Apr 2021
• Accepted
  15 Apr 2021