John et al. (2006)2121. John A, Sell D, Sweeney T, Harding-Bell A, Williams A. The Cleft Audit Protocol for Speech-Augmented: a validated and reliable measure for auditing cleft speech. Cleft Palate Craniofac J. 2006;43(3):272-88. https://doi.org/10.1597/04-141.1. PMID:16681400.
https://doi.org/10.1597/04-141.1...
|
Perceptual assessment protocol with training to use the protocol.
(To develop an assessment tool (CAPS-A) to use in inter-center audit studies in cleft lip and palate and test the reliability and validity of this assessment tool)
|
Training offered to evaluators by experienced speech-language therapists in the use of the new assessment protocol (CAPS-A) with consensual analyses (phase 3 of the study)
Duration: 6 hours
Modality: face-to-face |
Counting, sentence repetition, and spontaneous conversation
Scale:
Equal intervals (5 points) |
Speech-language therapists with experience in speech assessment of cleft lip and palate
(N=4) |
Intra- and inter-rater reliability rates are considered good/very good. The data suggest that after 6 hours of structured training, inter-rater agreement can be achieved. Evaluators considered the CAPS-A protocol to be acceptable and easy to use with appropriate training. |
Sell L et al. (2009) 2222. Sell D, John A, Harding-Bell A, Sweeney T, Hegarty F, Freeman J. Cleft Audit Protocol for Speech (CAPS-A): a comprehensive training package for speech analysis. Int J Lang Commun Disord. 2009;44(4):529-48. https://doi.org/10.1080/13682820802196815. PMID:18821108.
https://doi.org/10.1080/1368282080219681...
|
Perceptual assessment protocol with training to use the protocol.
(To delineate, execute, and evaluate a training program for speech therapists with systematization and reliability in the use of the CAPS-A assessment protocol)
|
Training offered by CAPS-A speech-language therapists for the use of the protocol.
Presentation of the construction and specific aspects of the assessment protocol (protocol development process, review of definitions adopted by the protocol, standardization of numerical scales of speech parameters, review of types of errors in the production of consonants in speech). Offering examples to illustrate the scales used, and exercises to establish consensus.
Afterward, practice with consensus (one case) followed by individual analyses, immediately after training and one month later.
Duration: 4 days (including two days of specific training followed by consensual practice before individual analyses)
Modality: face-to-face |
Counting, phrases, rhymes, and spontaneous sampling
Scale: Equal intervals (4 points) |
Speech-language therapists (N=36) |
In general, there was an increase in the reliability of intra- and inter-rater analyses, including the hypernasality speech parameter. |
Lee, Whitehill, Ciocca (2009) 1616. Lee A, Whitehill TL, Ciocca V. Effect of listener training on perceptual judgement of hypernasality. Clin Linguist Phon. 2009;23(5):319-34. http://dx.doi.org/10.1080/02699200802688596. PMID:19399664.
http://dx.doi.org/10.1080/02699200802688...
|
Training
(To investigate the effect of training and feedback on the intra- and inter-rater reliability of the hypernasality judgment)
|
Calibration (G1, G2, G3)
(Focus on hypernasality and coexisting speech disorders
G1: Exposure to speech samples, exemplifying hypernasality, nasal emission, vocal disorders, and articulatory errors
G2 (practice without feedback)
G3 (practice with feedback)
Practice: training with a hierarchy of difficulty with 4 steps:
1. identification of the presence or absence of hypernasality, nasal emission, vocal disorders, and articulatory errors
2. Identification of types of disorders
3. Hypernasality degrees discrimination
(Identification of the most hypernasal sample between two samples
4. Judgment of hypernasality
Duration: - calibration, 30 minutes; - practice 1 hour/group
Modality: face-to-face |
Phrases
Direct Magnitude Scale |
Speech-Language Therapy Students (N=36) |
A significant difference in inter-rater agreement between the groups with practice (training with and without feedback) and the group without practice (exhibition of samples).
Between the two groups that performed the practice, there was no significant difference.
Both groups with practice showed an increase in the reliability of intra-rater and inter-rater analyses.
Therefore, training (practice with and without feedback) helped improve the reliability of hypernasality ratings. |
Chapman et al. (2016) 1717. Chapman KL, Baylis A, Trost-Cardamone J, Cordero KN, Dixon A, Dobbelsteyn C et al. The Americleft Speech Project: a training and reliability study. Cleft Palate Craniofac J. 2016;53(1):93-108. http://dx.doi.org/10.1597/14-027. PMID:25531738.
http://dx.doi.org/10.1597/14-027...
|
Perceptual assessment protocols with training to use the protocol.
(To describe the reliability indices of the speech assessment of two perceptual assessment protocols CAPS-A and CAPS-A-AM)
*CAPS-A-AM: modifications made to the CAPS-A protocol
|
Training offered using the CAPS-A
(Study 1)
Training included a description of the steps and details of the procedures adopted; phonetic transcription and classification of speech parameters using scales, and practices with
consensus.
Duration: 3 days
Modality: face-to-face
Individual analyses
Study 1 (CAPS-A): before, immediately, one month after training
Study 2 (CAPS-A-AM): analyses on two moments, 5 weeks after study 1 |
Counting, rhymes, and phrases
Scale: Equal intervals (5 points) |
Speech-language therapists with experience in speech assessment of cleft lip and palate
(N=9)
Study 1: N=9
Study 2: N=6, (also included in Study 1) |
Study 1 CAPS-A: Significant improvements in inter-rater reliability indices after training, including hypernasality parameter.
Study 2 CAPS-A-AM: There were no significant differences in results between raters, including for hypernasality.
Inter-rater reliability indices of Study 2 were lower than those of Study 1. After adjusting the data analysis procedures, the findings were similar.
In both studies, there was good intra-rater reliability. |
Butts et al. (2016) 1818. Butts SC, Truong A, Forde C, Stefanov DG, Marrinan E. Perceptual assessment of velopharyngeal dysfunction by otolaryngology residents. Otolaryngol Head Neck Surg. 2016;155(6):1034-9. https://doi.org/10.1177/0194599816662247. PMID:27484234.
https://doi.org/10.1177/0194599816662247...
|
Training
(To assess the ability of Otorhinolaryngology (ENT) residents to assess hypernasality in patients with velopharyngeal dysfunction)
|
Educational Module
(Speech samples, explanations about velopharyngeal dysfunction concerning clinical cases) and
questionnaire about the module).
Educational module offered only for the experimental group,
between the classifications of the speech samples, initial and final, performed by the two groups (control and experimental), with one to two weeks of interval between the classifications.
Duration: 40 minutes
Modality: online |
Speech samples (module) not informed.
Scale: Equal intervals (4 points) |
ORL
Residents
(N=30) |
Improvement in the percentage of agreement (experimental group) after training, but without statistical difference concerning the control group. Greater accuracy in classifications for absent and severe grades |
Oliveira,
et al. (2016) 1414. Oliveira ACASF, Scarmagnani RH, Fukushiro AP, Yamashita RP. The influence of listener training on the perceptual assessment of hypernasality. CoDAS. 2016;28(2):141-8. http://dx.doi.org/10.1590/2317-1782/20162015163. PMID:27191877.
http://dx.doi.org/10.1590/2317-1782/2016...
|
Training
(To investigate the influence of previous training on the agreement between different raters in the perceptual judgment of hypernasality)
|
Definition of criteria and establishment of samples of
reference by experienced speech therapists (consensus)
Duration: unspecified
Modality: face-to-face |
Phrases and counting
Scale: Equal intervals (4 points) |
Speech-language therapists with experience in speech assessment of cleft lip and palate
(N=3) |
Increased intra- and inter-rater agreement rates |
Spruijt et al. (2018) 2323. Spruijt NE, Beenakker M, Verbeek M, Heinze ZCM, Breugem CC, Mink van der Molen AB. Reliability of the Dutch Cleft Speech Evaluation Test and Conversion to the Proposed Universal Scale. J. Craniofac. Surg. 2018;29(2):390-5. https://doi.org/10.1097/SCS.0000000000004261. PMID:29381632.
https://doi.org/10.1097/SCS.000000000000...
|
Perceptual assessment protocols with training to use the protocol.
(To measure intra- and inter-rater reliability using Dutch Cleft Speech Evaluation Test (DCSET) and convert DCSET into universal scales)
|
Training for the use of modifications in the Dutch Cleft Speech Evaluation Test (DCSET) based on video reviews and consensus analysis)
(Phase 2 of the study)
Review of scales for analysis of resonance and speech production errors, after training (Phase 2).
Phase 2 was performed after phase 1 (protocol without reviews)
Duration: Unknown
Modality: face-to-face |
Phrases
Scale: Equal intervals (3 points) for hypernasality |
Speech-language therapists with experience in speech assessment of cleft lip and palate
(N=2) |
Phase 1: did not show the expected results; adjustments were made regarding the use of the scale of some speech parameters.
Level 2:
reliability scores ranged from average to good (all speech parameters)
Hypernasality:
Reliability of analyses between evaluators (oral samples): Average |
Bruneel et al. (2020) 2424. Bruneel L, Alighieri C, D'haeseleer E, Kissel I, Adriaansen A, Sseremba D et al. Reliability results of perceptual ratings of resonance, nasal airflow and speech acceptability in patients with cleft palate by Ugandan speech-language pathologists following a two-day workshop. Int. J. Pediatr. Otorhinolaryngol. 2020;136:110191. https://doi.org/10.1016/j.ijporl.2020.110191. PMID:32593063.
https://doi.org/10.1016/j.ijporl.2020.11...
|
Perceptual assessment protocols with training to use the protocol.
(To develop and validate an instrument in the Belgian language for perceptual assessment in patients with cleft palate)
|
Phase 1: Preliminary study
Introduction of speech variables and description of the structure of the assessment protocol, with a reference sample for each degree and type of speech disorder (two hours)
Consensus practice (1 hour and a half)
Total duration: 3 and a half hours
Modality: face-to-face
Phase 2 (validation):
Description and explanation of the adopted definitions and parameters, description of the structure of the assessment protocol, and classification scales.
Consensus practice.
Total duration: 4 hours |
Counting, phrases, and connected speech
Scale: Equal intervals (4 points) |
Speech-language therapists, (N=2; study 1)
Speech-language therapists (N=4; study 2) |
Phase 1: In general, indices with good inter-rater analysis reliability, including hypernasality.
Phase 2: In general, good intra- and inter-rater reliability, including speech hypernasality.
In both phases, inter-rater reliability was lower than intra-rater reliability. |
Lohmander et al. (2021)1313. Lohmander A, Klintö K, Schalling E, Portela AS, Johansson K, McAllister A. Students take charge of learning - using e-learning in perceptual assessment in speech-language pathology. Scand. J. Educ. Res. 2021;65(3):468-80. http://dx.doi.org/10.1080/00313831.2020.1716064.
http://dx.doi.org/10.1080/00313831.2020....
|
Training
(To evaluate the training result, short and long-term, through an e-learning tool)
|
Training using a platform (PUMA website) with clinical cases and feedback from speech therapists experienced in the assessment of speech disorders in cleft lip and palate.
Training is offered during a structured course (teaching activity, including lectures, seminars, and laboratory activities), in the student’s free time, through the website. Two groups of students from two different universities were involved in the study.
Perceptual training included:
- Listening to speech samples containing examples of different types and degrees of speech disorders in individuals with cleft lip and palate
- Phonetically transcribing a video sample, with the possibility of comparison with analysis by experienced speech-language therapists
- Analyzing speech samples, using a scale of equal intervals, with the possibility of comparison with audio samples with consensual analyses of speech-language therapists
Training carried out individually or jointly.
Number of sessions and time devoted to training: unspecified.
Modality: online
Training performed in week 2.
Week1: instruction/ pre-test; Week 3: post-test |
Isolated words
Scale: Equal intervals
(4 points) |
Speech-Language Therapy Students
(N=45) |
A total of 16.5 hours (transcription activities) and 8.5 (analyses with scales) were used for training by the total number of students.
There was a significant improvement in phonetic transcription, after training, for total students. A significantly higher number of responses agree with the assessment of experienced speech-language therapists, after training, for the hyponasality and weak intraoral pressure variables for the total number of students. An improvement trend in responses to hypernasality was observed for one of the groups of students.
Positive comments from students regarding accessibility and practice time.
The e-learning indicated an improvement in the ability of the students (evaluators without experience) in the auditory-perceptual assessment. |
Bruneel et al. (2022) 2020. Bruneel L, Danhieux A, Van Lierde K. Training speech pathology students in the perceptual evaluation of speech in patients with cleft palate: reliability results and the students' perspective. Int. J. Pediatr. Otorhinolaryngol. 2022;157:111145. https://doi.org/10.1016/j.ijporl.2022.111145. PMID:35468487.
https://doi.org/10.1016/j.ijporl.2022.11...
|
Training
(To assess the immediate and long-term effect of perceptual training on the reliability of intra- and inter-rater analyses)
|
Definition of criteria and presentation of audio and audiovisual speech samples for practice in judging speech parameters (hypernasality, hyponasality, nasal emission, and nasal turbulence), in addition to training for assessing speech intelligibility and acceptability.
Additional samples for training the judgment of hypernasality, hyponasality, nasal emission, and nasal turbulence, first individually followed by group discussion (consensus judgments).
Training speech samples differed from test samples (pre-training, short-term post-training, and long-term post-training - 1 month).
Duration: 2 Hours
Modality: face-to-face |
Spontaneous samples and phrases
Scale: Equal intervals
5 points |
Speech-Language Therapy Students
(N=31) |
In general, the training positively affected the reliability results of the analyses and the students’ confidence in making the judgments (another aspect of interest in the study); however, these findings were dependent on the analyzed speech variable and at the time of the measurement.
Little or no training effect (short and long-term) was observed for hypernasality, air emission, and nasal turbulence variables. |
Thumbnail
