Validation of a measuring instrument for the perception of oral health in women

Abstract

The aim of this study was to estimate the reliability, validity and factorial invariance of the Portuguese version of the Geriatric/General Oral Health Assessment Index (GOHAI) as applied to Brazilian women. A total 701 women over 18 years of age participated in this study. Telephone interviews were conducted. We evaluated the construct-related validity through factorial, convergent and discriminant validity. We carried out a confirmatory factor analysis using the χ²/df, CFI, GFI and RMSEA indexes. The invariance of the model in a second independent sample was estimated by multi-group analysis and internal consistency using Cronbach’s alpha coefficient. Items 5 and 9 presented factor weights below the adequate value and were removed. The three-dimensional and unifactorial model presented an adequate fit. We observed strong factorial invariance of the models in two independent samples (three-factor: ρλ = 0.62; pCov = 0.89, one-factor: ρλ = 0.81; pCov = 0.68) and weak factorial invariance between users and nonusers of dental prosthetics (three-factor: ρλ = 0.55; pCov = 0.01, one-factor: ρλ = 0.51; pCov = 0.02). The convergent validity was suboptimal. Internal consistency was adequate. The GOHAI applied to the study sample showed adequate reliability, factorial validity and stability in independent samples and between users and nonusers of dental prosthetics in both the three-dimensional and the unifactorial structures.

Reproducibility of Results; Validity of Tests; Scales; Oral Health

Introduction

The assessment of oral health and its impact on the quality of life of individuals is an important step in health care practices, and some measuring instruments have been proposed in the literature to perform this assessment. Among these, one can mention the Oral Health Impact Profile (OHIP)¹ and the Geriatric/General Oral Health Assessment Index (GOHAI).² The purpose of these instruments is to assess the impact of oral health on an individual’s quality of life, measured according to the perception of oral health.

The GOHAI was initially proposed in English by Atchison and Dolan² for the U.S. population, but currently has versions adapted for different countries, namely Sweden,³ Malaysia,⁴ Saudi Arabia,^5,6 Germany,⁷ France,⁸ Romania,⁹ China,¹⁰ India,¹¹ Turkey,¹² Mexico¹³ and Brazil.¹⁴

Despite the several transcultural proposals submitted by different countries, only the studies conducted in Sweden,³ Saudi Arabia,^5,6 India,¹¹ Turkey¹² and Mexico¹³ included the assessment of the oral health perception construct as evidence of factorial validity. However, even these studies did not perform confirmatory factor analysis, which should have been adopted, considering that the theoretical construct had been previously defined by Atchison and Dolan.² It should be noted that the confirmatory factor analysis, based on the theoretical construct defined by the scale’s authors, is required to evaluate the validity of the data gathered. Although this analysis strategy is still unusual in the medical literature, psychometrics states that it is essential in order to clarify the metric characteristics of the scales in different samples, because it is the only way to assess the quality of the information obtained.

Knowing the widespread use of GOHAI in Brazilian studies, and given the lack of information on its psychometric qualities, this study was designed to estimate the reliability, validity and factorial invariance of the Portuguese version of the Geriatric/General Oral Health Assessment Index (GOHAI), as applied to Brazilian women.

Methodology

Study and sample design

This is a cross-sectional study with a probabilistic sampling design conducted in two stages. First, the participants were stratified according to the census sector, and then the sample units were randomly and systematically selected from the phone book.

The inclusion criteria were: living in the city of Araraquara (SP-Brazil), having a telephone landline, being female and over 18 years of age.

The minimum sample size to be used in validation studies is estimated by standard formulas for analyzing structural models¹⁵ that take into consideration the model’s degree of freedom, a of 5%, and a test power of at least 80%, which resulted in a sample size estimation of 144 subjects for this study. However, since the objective was to study the psychometric qualities of the GOHAI in a normative female population, we decided to use a sample large enough to capture the population’s variability adequately. Thus, we chose to use a representative sample of residents from the city of Araraquara - SP, female, and over 18 years of age. Accordingly to the 2010 Census there was a total population of 162,912 residents over the age of 18 years, 50% female, in Araraquara, SP. Assuming an α = 5%, a relative error margin of 7.5% resulted in a minimum sample size of 681 people. Assuming a 10% rate of refusal to participate, the sample size was corrected to n = 757.

Measuring instrument

The instrument used was the Portuguese version of the Geriatric/General Oral Health Assessment Index (GOHAI). The questionnaire consists of 12 items distributed in one dimension, but the items composing the instrument were drawn up considering three aspects, namely, “physical/functional,” “psychosocial/psychological,” and “pain/discomfort.” In this study, we tested a three-factor structure, preserving the theoretical distinction of items. The answers were given in a five-point type-Likert scale.² In this Portuguese version, three of the 12 items were inverted in relation to the remaining items.

To characterize the sample, data was gathered on age, marital status (single, married, widowed, divorced), use of dental prosthetics (yes or no) and type (fixed partial prosthesis, removable partial prosthesis or complete denture), education level and economic level. The level of education and economic development were estimated according to the Brazilian Economic Classification Criteria.¹⁶

Procedures

Given the evidence presented in the literature¹⁷ and the simplicity of the application of the instrument used, we decided to conduct the survey by telephone interviews.

Whenever a phone call was answered, the researcher presented himself and read the Informed and Free Consent Form for Research Subjects. Only people who agreed to its terms participated in the study. Participants were selected from the phone book of the city of Araraquara, using a systematic sampling process. The phone calls lasted approximately 10 minutes and were performed by only one interviewer calibrated in a pilot study (k =0.89). For calibration, the researcher applied the GOHAI on two separate occasions one week apart. Sixty women participated in the pilot study. Telephone calls were made until a total of 757 calls were reached. Thus, we performed 810 calls, of which 757 were answered, and, of these, 56 people refused to participate in the study.

Analysis of the psychometric characteristics

The psychometric sensitivity of each item of the GOHAI was assessed through its measures of shape. Absolute values of skewness (Sk) and kurtosis (Ku) higher than 3 and 7, respectively, were considered indicative of severe deviation from the normal distribution.¹⁸

The factorial validity of the three-dimensional model for the total sample was estimated by confirmatory factor analysis, using the qui-square statistic over degrees of freedom (χ²/df), the comparative fit index (CFI), the goodness of fit index (GFI) and the root mean square error of approximation (RMSEA), as the goodness of fit indices. The fit of the model to the data was considered adequate when factorial item weights of λ ≥ 0.50, χ²/df ≤ 4.0, CFI and GFI ≥ 0.90 and RMSEA ≤ 0.10 were reached.¹⁸

Items with λ < 0.40 were removed from the model, as well as those related to more than one factor, as indicated by the modification indices estimated by the Lagrange Multipliers (LM > 11, p < 0.001). The comparison between the initial model (Model 0-M0) and that obtained after removal of the items (Model 1-M1) was performed using indices based on the information theory (Akaike Information Criterion-AIC, Browne-Cudeck Criterion-BCC, and Bayesian Information Criterion-BIC). The best model was that which had the lowest values in one or more of these indices.¹⁸

The stability of the factor model was assessed in two steps and evaluated using cross-validation. In the first step, the sample was divided into two parts, one containing 70% of the sample (n = 490), called “test sample” (Model 2-M2), and the other, 30% (n = 211), named “sample validation” (Model 3-M3). A multigroup analysis with the chi-square difference test (Δχ²) was performed to test the invariance of the model in the two subsamples. When the hypothesis of factorial invariance of the factorial weights (weak invariance) was accepted, invariance of covariance (strong invariance) analysis was performed.¹⁸

The same procedure was adopted in a second analysis, but now the stability of the model was tested on users (n = 274) (Model 4-M4) and nonusers (n = 427) (Model 5-M5) of dental prosthetics.

For models 1-5, convergent validity, discriminant validity and internal consistency were calculated. Convergent validity was estimated by the average variance extracted (AVE) and composite reliability (CR).¹⁹ Values of AVE ≥ 0.50 and CR ≥ 0.70 were considered indicative of convergent validity.¹⁹ Discriminant validity was accepted when AVE_i and AVE_j for factors i and j was greater than the squared correlations between these two factors (r_ij²).

The internal consistency of the factors was estimated with the standardized Cronbach’s alpha coefficient (a), and was considered appropriate when a ≥ 0.70.²⁰

Analyses were conducted using IBM SPSS^® Amos (v.20, SPSS Inc., Chicago, USA) and IBM SPSS^® Statistics (v.21 SPSS Inc., Chicago, USA).

First-order correlations between factors were calculated on the total sample data. Values close to 1 were considered suggestive of a second-order hierarchical model. The standardized regression weights (β) from the proposed second-order factor to the first-order factors were estimated in order to study the second-order hierarchical model. However, given the strong and highly significant correlation values between first-order factors, we also evaluated the goodness of fit of a unifactorial structural model, and validity and reliability were estimated following the same procedure reported above.

Ethical Aspects

The present study was approved by the Ethics Research Committee of the School of Pharmaceutical Sciences - UNESP (protocol: 52/2009).

Results

The study included 701 women, showing an adherence rate of 92.6%, thus indicating the representativeness of the sample in the study population. The mean age of the participants was 44.4 (SD=16.3) years. Table 1 presents the distribution of participants in regard to sociodemographic and clinical characteristics.

Table 1
Distribution [n(%)] of participants in regard to sociodemographic and clinical characteristics. Araraquara - SP / Brazil, 2011.

Variable	n(%)
Marital status
Single	161(22.97)
Married	407(58.06)
Widowed	78(11.13)
Divorced	55(7.84)
Social class
A	8(1.14)
B	258(36.81)
C	388(55.35)
D/E	47(6.70)
Educational level
Illiterate/Incomplete primary school	86(12.27)
Complete primary school/incomplete middle school	148(21.11)
Complete middle school/incomplete high school	136(19.40)
Complete high school/incomplete university	260(37.09)
Complete university education	71(10.13)
Dental Prosthesis
No	274(39.09)
Yes	427(60.91)
Type of dental prosthesis
Fixed partial prosthesis	87(20.37)
Removable partial prosthesis	179(41.92)
Complete denture	161(37.71)

As observed, most participants were married, belonged to social class C and were users of some type of dental prosthesis.

In all dimensions, the items had skewness (Sk = |0.42-1.92|) and kurtosis (Ku = |0.30-2.36|) values close to those presented in the normal distribution, therefore, posing no considerable sensitivity or normality issues.

The information-theory-derived indices support the better fit of the adjusted model (M1) to the data, as compared to the initial model (M0). All models M1 to M5 presented an adequate goodness of fit to the data. The reliability (CR, α) was adequate for the physical/functional and psychosocial/psychological dimensions in all models. The convergent validity (AVE) had below adequate values for all models, but the 3 factors presented discriminant validity (AVE≥r²) for two of the three factors.

In the cross validation analysis, we found an adequate model fit for both the Test (M2) and the Validation (M3) Samples (χ²/df = 1.92, CFI = 0.97, GFI = 0.96, RMSEA = 0.04). Strong factorial invariance of the model was observed for both samples (Δχ²: λ = 5.33, p = 0.62; Cov = 2.30, p = 0.89). The model of goodness of fit for both users (M4) and nonusers (M5) of dental prosthetics was also adequate (χ²/df = 3.11, CFI = 0.93, GFI = 0.93, RMSEA = 0.06) and there was weak invariance of the model in both users and nonusers (Δχ²: λ = 5.92, p = 0.55; Cov = 21.74, p < 0.01).

Judging from the strong correlations observed between factors, we evaluated the goodness of fit of a second-order factorial model. Figure 1 shows the second-order hierarchical factor of “Perception of oral health” of GOHAI, fitted to the total sample.

Figure 1
Second-order hierarchical model adjusted to the total sample (χ2/df = 3.66, CFI = 0.97, GFI = 0.97, RMSEA = 0.06) for the Geriatric/General Oral Health Assessment Index (GOHAI)

We can confirm through the standardized regression weights that all dimensions of GOHAI are strongly and significantly (p < 0.01) related to the central “Perception of oral health” construct.

Owing to the low convergent validity of the three factors, the lack of discriminant validity between D1 and the other factors (Table 1), and the strong relation between the three factors and the central construct (Figure 1), from a strictly psychometric perspective, the GOHAI would produce results with greater validity and reliability if it were scored as a unifactorial construct. Thus, Table 2 shows the goodness of fit of a unifactorial construct in the different subsamples under study (models M1_u to M5_u).

Table 2
Goodness of fit indices of the confirmatory factor analysis (CFA), convergent validity (AVE), composite reliability (CR) of the models fitted to different samples/models (M0-M5), Pearson correlation between factors (r), squared correlations (r2), and internal consistency (α) for the Geriatric/General Oral Health Assessment Index (GOHAI). Araraquara - SP / Brazil, 2011.

				Model#
	Estimate	M0	M1	M2	M3	M4	M5
CFA	l	-0.14-0.81	0.50-0.80	0.50-0.80	0.50-0.80	0.56-0.79	0.45-0.79
	χ2/df	4.34	3.65	2.95	1.76	2.67	2.21
	CFI	0.93	0.97	0.96	0.97	0.95	0.97
	GFI	0.95	0.97	0.96	0.95	0.95	0.97
	RMSEA	0.07	0.06	0.06	0.06	0.08	0.05
	BIC	398.31	270.65	240.17	182.91	217.35	213.85
	AIC	275.39	161.39	139.51	102.47	130.63	116.49
	BCC	276.41	162.15	140.61	105.12	132.65	117.76
	r	0.82-0.92	0.86-0.99	0.84-0.99	0.90-0.96	0.78-0.97	1.00-1.00
AVE	##D1	-	0.42	0.42	0.46	0.42	0.41
	D2	-	0.48	0.49	0.48	0.49	0.46
	D3	-	0.28	0.30	0.24	0.38	0.21
CR	D1	-	0.74	0.74	0.77	0.74	0.73
	D2	-	0.79	0.79	0.78	0.79	0.77
	D3	-	0.43	0.46	0.38	0.55	0.34
r2	D1 X D2	-	0.51	0.52	0.48	0.48	0.51
	D1 X D3	-	0.24	0.23	0.26	0.26	0.29
	D2 X D3	-	0.28	0.28	0.26	0.28	0.33
a	D1	-	0.77	0.76	0.80	0.78	0.74
	D2	-	0.77	0.78	0.78	0.79	0.75
	D3	-	0.42	0.45	0.37	0.53	0.34

^#M0: Three-factor model, M1: Adjusted three-factor model (excluding items 5 and 9 with correlation between errors 1 and 2), M2: M1 adjusted to the test sample, M3: M1 adjusted to the validation sample, M4: M1 adjusted sample of users of dental prosthetics, M5: M1 adjusted sample of nonuser dental prosthetics; ^##D1: physical/functional, D2: psychosocial/psychological, D3: pain/discomfort

All models from M1_u to M5_u presented an adequate fit to the data. However, taking into account the indices based on the information theory, we noted that there was no relevant difference between the one-factor and the three-factor models (Table 1 and Table 2). The reliability (CR, a) was adequate, and convergent validity was below adequate for all models.

In the cross validation analysis, we found adequate fit of the one-factor model simultaneously in the Test Sample and in the Validation Sample (χ²/df = 1.92, CF I= 0.97, GFI = 0.95, RMSEA = 0.04) with strong invariance (Δχ²: λ = 5.23, p = 0.81; Cov = 20.17, p = 0.68). The simultaneous fit to the sample of users and nonusers of dental prosthetics (χ²/df = 3.04, CFI = 0.93, GFI = 0.92, RMSEA = 0.05) was adequate, and the invariance of the model was weak (Δχ²: λ = 8.24, p = 0.51; Cov = 10.02, p = 0.02).

Discussion

As far as we know, this is the first study that has ever evaluated the psychometric properties of the data gathered with the Geriatric/General Oral Health Assessment Index (GOHAI), applied to Brazilian women. The results suggest acceptable psychometric properties of the GOHAI for Brazilian women, with the same sociodemographics of the probabilistic sample composing this study. In this paper, we have not only given information about the metric quality of the GOHAI for the sample, but have also introduced the steps required for evaluating the quality of the data arising from the use of scales like the GOHAI.

The analysis of factorial validity and invariance in independent samples is a unique condition for determining the best model of data analysis and its stability in different samples. In this study, we noted that the evaluation of the perception of oral health among Brazilian women can be performed either considering the three-factor model (Table 2), preserving the content (tri-factorial) proposal of the items,² or using the one-factor model (Table 3). However, we observed (Figure 1) that the three factors have strong correlations with each other and that the convergent validity (Table 2) was below the desirable values,¹⁹ indicating that the items that reflect these factors did not correlate more strongly with this factor than others. These estimates are strong indications that the unifactorial structure may be more parsimonious in representing the construct, despite adequate fit to the three-factor model. The comparison between the factorial structure obtained and the published research on GOHAI dimensionality has shown mixed results. In Turkey¹² and Sweden,³ the two-factor model was that which presented the best fit to the data; in Saudi Arabia⁶ and in India¹¹ the best fit was the unifactorial model; and in Mexico¹³ the four-factor structure was proposed as offering the best fit to the data gathered. These mixed results also demonstrate a well-known fact: the validity of a scale is not characteristic of an instrument per se, but of the instrument as applied to the sample.²¹ Thus, the assessment of the psychometric properties of the data is the basic premise to be evaluated, prior to presenting any outcomes arising from the use of scales. However, we would like to point out that the assessment of these characteristics in psychometric instruments is essential to performing steps such as sensitivity analysis of psychometric items, as well as factorial, convergent, discriminant, concurrent, discriminant and predictive validity, which are usually neglected. These steps must be performed by all who intend to evaluate latent variables, since their evaluation is complex and requires the use of robust statistical modeling, and usually also large samples.¹⁸

Table 3
Quality of adjustment indices of the confirmatory factor analysis (CFA), convergent validity (AVE), composite reliability (CR) of the unifactorial models adjusted to different samples and internal consistency (a) of the Geriatric/General Oral Health Assessment Index (GOHAI). Araraquara - SP / Brazil, 2011.

	Model#
Estimate	M1u	M2u	M3u	M4u	M5u
CFA*
l	0.44-0.80	0.43-0.80	0.44-0.79	0.48-0.78	0.47-0.79
c2/df	3.50	2.85	1.68	2.72	2.06
CFI	0.97	0.96	0.97	0.94	0.98
GFI	0.97	0.96	0.95	0.94	0.97
RMSEA	0.06	0.06	0.06	0.08	0.05
BIC	256.72	227.19	169.42	210.48	197.37
AIC	161.11	139.10	99.04	134.60	112.18
BCC	161.78	140.07	101.36	136.36	113.29
AVE*	0.40	0.40	0.40	0.41	0.40
CR*	0.87	0.87	0.87	0.87	0.86
a	0.86	0.86	0.86	0.87	0.86

*CFA: confirmatory factor analysis, AVE: average variance extracted, CR: composite reliability; # M1u: adjusted unifactorial model (excluding items 5 and 9 and with correlation between errors 1 and 2), M2u: M1u adjusted to the test sample, m3u: M1u adjusted to the validation sample, M4U: M1u adjusted to the sample of users of dental prosthetics, M5u: M1u adjusted to the sample of nonusers of dental prosthetics.

Despite the stability of the factor weights in the different samples, we observed no invariance in the correlations between factors, when applying the GOHAI in samples of users and nonusers of dental prosthetics. These findings suggest that the samples were associated to oral health perception factors differently. This observation is also supported by the variance explained by the model, which is higher for users (three-factor=74% and one-factor=71%) than for nonusers (three-factor=59% and unifactorial=57%) of dental prosthetics. This characteristic is consistent with the theoretical conception proposed by the GOHAI, designed to capture the perception of oral health in geriatric populations having more compromised oral health.²

Despite the limitations of this study, namely that the sample was representative only of the female population, that it was a cross-sectional and correlational study and that no adjustment of the results was made in the oral clinical condition of the subjects, it should be noted that these limitations are common in validation studies of scales, and that the results attest to the reliability and validity of the Portuguese version of the GOHAI.

Conclusion

The GOHAI applied to the sample of women in this study presented good reliability, factor validity and stability in independent samples and between users and nonusers of dental prosthetics, when using both a three-factor and a one-factor structure.

References

• ¹
  Slade GD, Spencer AJ. Development and evaluation of the oral health impact profile. Community Dent Health. 1994 Mar;11(1):3–11.
• ²
  Atchison KA, Dolan TA. Development of the Geriatric Oral Health Assessment Index. J Dent Educ. 1990 Nov;54(11):680–7.
• ³
  Hägglin C, Berggren U, Lundgren J. A Swedish version of the GOHAI index. Psychometric properties and validation. Swed Dent J. 2005;29(3):113–24.
• ⁴
  Othman WN, Muttalib KA, Bakri R, Doss JG, Jaafar N, Salleh NC, et al. Validation of the Geriatric Oral Health Assessment Index (GOHAI) in the Malay language. J Public Health Dent. 2006 Ago 18;66(3):199-204.
• ⁵
  Atieh MA. Arabic version of the Geriatric Oral Health Assessment Index. Gerodontology. 2008 Mar;25(1):34–41.
• ⁶
  Daradkeh S, Khader YS. Translation and validation of the Arabic version of the Geriatric Oral Health Assessment Index (GOHAI). J Oral Sci. 2008 Dec;50(4):453–9.
• ⁷
  Hassel AJ, Rolko C, Koke U, Leisen J, Rammelsberg P. A German version of the GOHAI. Community Dent Oral Epidemiol. 2008 Feb;36(1):34–42.
• ⁸
  Tubert-Jeannin S, Riordan PJ, Morel-Papernot A, Porcheray S, Saby-Collet S. Validation of an oral health quality of life index (GOHAI) in France. Community Dent Oral Epidemiol. 2003 Aug;31(4):275–84.
• ⁹
  Murariu A, Hanganu C, Bobu L. Evaluation of the reliability of the geriatric oral health assessment index (GOHAI) in institutionalised elderly in Romania: a pilot study. Oral Health Dent Manag. 2010 Mar;9(1):11–5.
• ¹⁰
  A-Dan W, Jun-Qi L. Factors associated with the oral health-related quality of life in elderly persons in dental clinic: validation of a Mandarin Chinese version of GOHAI. Gerodontology. 2011 Sep;28(3):184–91.
• ¹¹
  Deshmukh SP, Radke UM. Translation and validation of the Hindi version of the Geriatric Oral Health Assessment Index. Gerodontology. 2012 Jun;29(2):e1052–8.
• ¹²
  Ergül S, Akar GC. Reliability and validity of the Geriatric Oral Health Assessment Index in Turkey. J Gerontol Nurs. 2008 Sep;34(9):33–9.
• ¹³
  Sánchez-García S, Heredia-Ponce E, Juárez-Cedillo T, Gallegos-Carrillo K, Espinel-Bermúdez C, de la Fuente-Hernández J et al. Psychometric properties of the General Oral Health Assessment Index (GOHAI) and dental status of an elderly Mexican population. J Public Health Dent. 2010;70(4):300–7.
• ¹⁴
  Pinto VG. Saúde Bucal Coletiva. 4th ed. São Paulo: Santos Livraria; 2000. 541 p.
• ¹⁵
  Kim KH. The relation among fit indexes, power and sample size in structural equation modeling. Struct Equ Modeling. 2005 Aug;12:368–90.
• ¹⁶
  ABEP. Associação Brasileira de Empresas de Pesquisa. Critério de Classificação Econômica Brasil - 2008. 2008 [cited 2014 Jul 10]; Available from: abep.org/codigosguias/Criterio_Brasil_2008.pdf
  » abep.org/codigosguias/Criterio_Brasil_2008.pdf
• ¹⁷
  Ferreira AD, César CC, Malta DC, Andrade AC, Ramos CG, Proietti FA, et al. Validity of data collected by telephone survey: a comparison of VIGITEL 2008 and ‘Saúde em Beagá’ survey. Rev Bras Epidemiol. 2011 Sep;14 Suppl 1:16–30.
• ¹⁸
  Maroco J. Análise de equações estruturais. Lisboa: Report Number; 2010. 374 p.
• ¹⁹
  Fornell C, Larcker DF. Evaluating Structural Equation Models With Unobservable Variables And Measurement Error. J Mark Res. 1981 Feb;18(1):39–50.
• ²⁰
  Maroco J, Garcia-Marques T. Qual a fiabilidade do alfa de Cronbach? Questões antigas e soluções modernas?. Lab Psicol. 2006 Aug;4(1):65–90.
• ²¹
  Anastasi A. Psychological testing. 6th ed. New York: Macmillan Publishing Company; 1988.

Publication Dates

• Publication in this collection
  2014

History

• Received
  07 Aug 2013
• Accepted
  21 Mar 2014
• Reviewed
  16 July 2014