How to determine the quality of a questionnaire according to the CONsensus-based Standards for the selection of health Measurement INstruments? A simplified guide to the measurement properties of assessment instruments - Part II: validity, responsiveness, interpretability and a checklist for characterizing the quality of instruments

Chaves, Thaís Cristina; Lima, Thamiris Costa de; Spavieri, Juliana H. Padilha; Claudio, Ana Carolina de Jacomo; Pereira, Roger Berg Rodrigues; Lira, Mariana Romano de

doi:10.5935/2595-0118.20230092-en

Acessibilidade / Reportar erro

Brasil

Español English

sumário « anterior atual seguinte »

Sumário

REVIEW ARTICLE • BrJP 6 (4) • Oct-Dec 2023 • https://doi.org/10.5935/2595-0118.20230092-en copy

How to determine the quality of a questionnaire according to the CONsensus-based Standards for the selection of health Measurement INstruments? A simplified guide to the measurement properties of assessment instruments - Part II: validity, responsiveness, interpretability and a checklist for characterizing the quality of instruments

Authorship SCIMAGO INSTITUTIONS RANKINGS

ABSTRACT

BACKGROUND AND OBJECTIVES:

The type of questionnaire that aims to capture a patient’s perception/view of an aspect to be measured (e.g. pain intensity) is called Patient Reported Outcome Measure (PROM). One of the biggest challenges that clinicians and researchers often face is making a decision about which PROM to use for the assessment of their patient with pain, especially due to the lack of scientific literacy needed to understand the criteria and terms used in the field of measurement properties. Thus, the objectives of this study (part II) were: (I) to introduce basic concepts about PROMs with a focus on the terminology and criteria defined by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) and (2) to describe the measurement properties of the validity, responsiveness and interpretability domains and propose a checklist for assessing the quality of PROMs’ measurement properties.

METHODS:

This study was produced using a search for articles from the COSMIN initiative. For didactic purposes, the text was divided into two parts.

RESULTS:

This article included a description of the measurement properties of the validity (content, structural, construct), responsiveness (must be assessed through accuracy analyses, AUC≥0.70) and interpretability (which provides the minimum clinically important change) domains. In addition, a checklist was proposed for determining the quality of the measurement properties of assessment instruments.

CONCLUSION:

This study described the measurement properties within the validity and responsiveness domains, and the importance of interpretability for obtaining the minimum clinically important difference. The proposed checklist for evaluating these properties can help clinicians and researchers to determine the quality of an instrument and make a decision about the best option available.

Keywords:
Chronic pain; Psychometrics; Musculoskeletal pain; Reliability; Surveys and questionnaires

HIGHLIGHTS

1. Within the validity domain it is possible to analyze: structural validity and construct validity-hypothesis testing.

2. Responsiveness is the ability of an instrument to detect changes over time.

3. Interpretability is the ability to extract meaning from the results obtained by PROMs.

4. The minimum clinically important change (MIC) is one of the interpretability measures of PROMs.

5. A checklist with 20 items was proposed to assist in determining the PROM with the best quality.

Items	Judging criteria	Classification
Validity of Content	1 - Has the construct measured by PROM or OMI been adequately described/defined?	Yes □ No □ Not described □ Cannot be determined □
	2 - Has a conceptual model of the construct measured by PROM or OMI been described?	Yes □ No □ Not described □ Cannot be determined □
	3 - Is it clearly described to which target population the PROM or OMI applies?	Yes □ No □ Not described □ Cannot be determined □
Relevance	4 - Do the PROM or OMI questions seem relevant to the target population?	Yes □ No □ Not described □ Cannot be determined □
	5 - Is the reporting period for recalling the construct adequate and clearly described?	Yes □ No □ Not described □ Cannot be determined □
Understanding	6 - Are the questions, answer options and instructions in the PROM or OMI easy for the target population to understand?	Yes □ No □ Not described □ Cannot be determined □
Scope	7 - Does the PROM or OMI cover all the fundamental concepts that should be considered when evaluating the construct?	Yes □ No □ Not described □ Cannot be determined □
Structural validity - PROM or OMI domains or sub-scales	8- Does a description through appropriate analysis show that the scale is unidimensional or multidimensional?	Yes □ No □ Not described □ Cannot be determined □
	a - For confirmatory factor analysis the following aspects are described: CFI or TLI > 0.95 or RMSEA or SRMR < 0.06	Yes □ No □ Not described □ Cannot be determined □
	b - For Rasch analysis is described: no violation of unidimensionality, no violation of local independence, no violation of monotocity, and adequate model fit (infit and outfit values between ≥ 0.5 and ≤ 1.5)?
	c -For exploratory factor analysis, Factor loadings > 0.30 and only 10% of items loading on more than 1 factor and explained variance of at least 50% or scree plot results or Kaiser criterion (Eigenvalues > 1) aligned with the PROM or OMI conceptual model?
Internal consistency	9 - Has structural validity been verified?	Yes □ No □ Not described □ Cannot be determined □
	10 - Cronbach's alpha > 0.70?	Yes □ No □ Not described □ Cannot be determined □
Reliability	11 - Did the reliability of the scale show adequate values? Such as ICC or weighted Kappa or r ≥ 0.70?	Yes □ No □ Not described □ Cannot be determined □
	12 - Can the test-retest period be considered adequate?	Yes □ No □ Not described □ Cannot be determined □
	13 - Was a clear description offered that the patients were stable in the test-retest period?	Yes □ No □ Not described □ Cannot be determined □
Error measurement	14 - SDC or LoA < MIC	Yes □ No □ Not described □ Cannot be determined □
Construct validity - hypothesis testing	15 - Have at least 75% of the hypotheses raised been confirmed?	Yes □ No □ Not described □ Cannot be determined □
Criteria Validity	16 - Was a correlation r > 0.70 observed between the score of the long and short version of the PROM or OMI? Or AUC ≥ 0.70?	Yes □ No □ Not described □ Cannot be determined □
Responsivity	17 - Were at least 75% of the hypothetical comparisons confirmed or AUC ≥ 0.70?	Yes □ No □ Not described □ Cannot be determined □
Interpretability	18 - Is data/values described for the PROM or OMI that allow interpretation of the scores obtained? Ex1: Minimum Clinically Important Change (MIC)? Ex2: cut-off value for determining subgroups? Ex3: how to interpret the score: for example, what does a high or low score mean?	Yes □ No □ Not described □ Cannot be determined □
Cross-cultural adaptation	19 - Is there a version of the PROM or OMI available in Brazilian Portuguese that has followed an appropriate method of cross-cultural adaptation?	Yes □ No □ Not described □ Cannot be determined □
	20 - Have the measurement properties of PROM or OMI been tested on a sample of Brazilians?	Yes □ No □ Not described □ Cannot be determined □

Sociedade Brasileira para o Estudo da Dor Av. Conselheiro Rodrigues Alves, 937 Cj2 - Vila Mariana, CEP: 04014-012, São Paulo, SP - Brasil, Telefones: , (55) 11 5904-2881/3959 - São Paulo - SP - Brazil
E-mail: dor@dor.org.br

Acompanhe os números deste periódico no seu leitor de RSS

[1] Correspondence to: Thaís Cristina Chaves E-mail: thaischaves@ufscar.br

1. What construct does each PROM measure?	“Both assess the perception of disability related to low back pain”
2. What is the logic behind the PROMs score?	“The higher the score, the worse the disability”
3. Is a correlation expected between PROMs scores? Why?	“Yes, because they both measure the same construct”
4. What is the expected magnitude of the correlation? Ex: weak, moderate or intense?	“At least r > 0.50”
5. What is the direction of the expected correlation? Positive or negative?	“Positive, because for both instruments the higher the score, the greater the disability”*
So, formulate the complete hypothesis of correlation between the scores of the scales:	“A moderate and positive correlation is expected between the RMDQ and ODI scores. Based on the hypothesis that the instruments measure the same construct”