How reproducible is self-reported information on exposure to smoking, drinking, and dietary patterns? Evidence among Brazilian adults in the Pró-Saúde Study

ORIGINAL ARTICLE

How reproducible is self-reported information on exposure to smoking, drinking, and dietary patterns? Evidence among Brazilian adults in the Pró-Saúde Study

Dóra Chor; Eduardo Faerstein; Márcia Guimarães Mello Alves; Claudia de Souza Lopes

Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brazil

^{Correspondence}

ABSTRACT

CONTEXT: Epidemiological studies of the validity and reliability of self-reported information on important risk factors for non-communicable chronic diseases are scarce in Brazil.

OBJECTIVE: We evaluated the test-retest reliability of information overall and stratified by gender, age and education on active and passive smoking, alcohol intake and aspects of dietary habits.

TYPE OF STUDY: Test-retest reliability.

SETTING: Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brazil.

PARTICIPANTS: 192 University employees.

PROCEDURES: Self-administered questionnaires were completed on two occasions, two weeks apart.

MAIN MEASUREMENTS: Kappa Statistics; Intraclass Correlation Coefficient.

RESULTS: Information on smoking status and pack-years smoked had almost perfect levels of agreement, respectively, kappa = 0.97 (95% CI, 0.92-1.00), and intraclass correlation coefficient = 0.93 (CI 95%, 0.89-0.96). Characteristics of alcohol intake yielded substantial levels of agreement (kappa ranging from 0.62 to 0.69). The reproducibility of the information on dietary habits varied from 0.67 to 0.79 (kappa). No clear-cut patterns could be identified comparing information by age or gender. There was a slight tendency towards greater reliability among people with higher levels of education.

CONCLUSION: The reproducibility of information on smoking, drinking, and dietary patterns ranged from substantial to excellent, as investigated in the Pró-Saúde Study, a longitudinal investigation recently launched in Rio de Janeiro.

Key words: Brazil. Smoking. Alcohol. Drinking.Diet.

RESUMO

CONTEXTO: Estudos epidemiológicos sobre a validade e confiabilidade de informações auto-referidas sobre fatores de risco para doenças crônicas não-transmissíveis são escassos no Brasil.

OBJETIVO: Avaliar a confiabilidade teste-reteste da informação ¾ total e estratificada por gênero, idade e educação ¾ sobre fumo ativo e passivo, consumo de álcool e aspectos de hábitos dietéticos.

TIPO DE ESTUDO: Confiabilidade teste-reteste.

LOCAL: Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brasil.

PARTICIPANTES: 192 funcionários técnico-administrativos.

PROCEDIMENTOS: Questionários auto-administrados foram completados em duas ocasiões, com duas semanas de intervalo.

ANÁLISE ESTATÍSTICA: Estatística Kappa; Coeficiente de Correlação Intraclasse.

RESULTADOS: Estatística. As informações sobre o hábito de fumar cigarros e a quantidade de cigarros (maços-ano) apresentaram níveis quase perfeitos de concordância, respectivamente, kappa = 0,97 (IC 95%, 0,92 1,00); coeficiente de correlação intraclasse = 0,93 (IC 95%, 0,89 0,96). Características de consumo de álcool apresentaram níveis substanciais de concordância (kappa variando entre 0,62 e 0,69). A confiabilidade da informação sobre hábitos dietéticos variou entre 0,67 e 0,69 (kappa). Nenhum padrão definido pode ser identificado comparando a informação segundo idade ou gênero. Houve uma leve tendência à maior confiabilidade entre pessoas com maiores níveis de escolaridade.

CONCLUSÃO: A confiabilidade das informações sobre o hábito de fumar, consumo de bebidas alcoólicas e padrões de dieta variou entre muito boa a excelente, de acordo com a investigação realizada no Estudo Pró-Saúde, investigação longitudinal, recentemente iniciada no Rio de Janeiro.

Palavras-chave: Brasil. Fumo. Bebida. Alcoólica. Dieta.

INTRODUCTION

There are no published reports of epidemiological studies among Brazilian adults regarding the validity and reliability of self-reported information on smoking, exposure to passive smoking, and alcohol intake, which are important risk factors for non-communicable chronic diseases. A single Brazilian study assessing the quality of information regarding habitual diet has been made.¹

In addition, no information exists about how the quality of self-reported information might vary across population subgroups (defined, for example, by gender, age, or socioeconomic status). Research conducted in the United States, however, has shown that information quality may vary with these and other characteristics of the study population,^2-4 leading to measurement errors that may in part explain discrepant results between epidemiological studies.⁵

Information on health-related behavior, generally obtained by means of interviews or self-administered questionnaires, may be distorted by respondents' recall errors, or by their wish to give responses which would be considered socially desirable.^6,7 Thus, the evaluation of the quality of information obtained from epidemiological research is crucial, and the estimation of validity and reproducibility indicators is the main tool for conducting such evaluations.⁸

This paper reports on the test-retest reliability of information on smoking, exposure to passive smoking, alcohol intake, and aspects of diet (fruit and vegetable intake). It also presents stratum-specific estimates by gender, age and educational attainment. Questions on these and other exposures were subsequently included in a multidimensional, self-administered questionnaire applied to 4,030 university employees in Rio de Janeiro, Brazil. This was the Pró-Saúde Study, launched in 1999 with the objective of investigating relationships between social factors and health outcomes.

Study population

The test-retest study was conducted in June 1999. The eligible participants were 1,120 employees who were at that time working at the University where the Pró-Saúde Study was subsequently launched, but who were not permanent employees (i.e. they were not on the administrative "tenure track" that exists in Brazil), although they were not short-term "temporary" workers either. Their job activities were similar to those performed by permanent employees, but they were considered ineligible for the main study because the prospects for their long-term follow-up were uncertain. A 20% systematic sample of 224 potential respondents (e.g. the 5^th, 10^th, etc.) was drawn from official listings.

Test-retest reliability study

The conditions under which the questionnaire was administered were identical to those of the main study, i.e. during working hours, in University classrooms, and with help available from trained personnel. Among 224 employees sampled, 202 (90.2%) agreed to fill out the questionnaire. After its completion, they were invited to repeat the procedure two weeks later, with the explanation given being that the purpose was to test the adequacy of the questionnaire, rather than the participants' responses. The procedure was repeated by 192 respondents (95.0% of the initial participants), without relevant variations in the proportions of participants in the test and retest according to age, gender, or schooling.

Responses to the following closed-ended questions are analyzed in the present paper: "How often do you eat fresh fruit?"; "How often do you eat vegetables?"; "Have you ever been a cigarette smoker, that is, have you smoked at least 100 cigarettes (five packs) during your lifetime?"; "Do you currently smoke cigarettes?"; "In general, how many cigarettes per day did you smoke or do you currently smoke?"; "In total, for how many years did you smoke or have you been smoking? (do not include time periods when you did not smoke)". The following open-ended questions were also analyzed: "Are there other people who smoke at your workplace or at home?"; "In the last two weeks, did you drink any alcoholic beverage?"; "In the last two weeks, on how many days did you drink any alcoholic beverage?"; "In the last two weeks, on average, how many drinks did you have on each of those days when you drank?"

Statistical analysis

Agreement was measured for the entire sample, and also stratifying for gender, age (18-39 years vs. 40 years or more), and education (high school or less vs. college or more). The responses were analyzed in terms of assessing smoking status (current, former, or never); exposure to passive smoking (at workplace, at home, both or neither); alcohol intake during previous 2 weeks (yes or no), number of days of drinking (1, 2-5, or more than 5), and average number of drink units (1, 2-4, 5-7, or more than 7); usual frequency of fruit and vegetable intake (less than once a week, 1-3 days/week, 4-6 days/week, or daily). Pack-years of cigarettes smoked were calculated by multiplying the average number of cigarettes smoked daily by the duration in years of the smoking habit.

We compared frequencies reported at the test and retest by using kappa statistics (k) for categorical variables (active and passive smoking, and alcohol intake) and weighted kappa (k_w– squared error weighting)⁹ for ordinal variables (frequency and amount of alcohol intake; fruit intake and vegetable intake).¹⁰ The intraclass correlation coefficient (ICC) was utilized to measure agreement for the continuous variable pack-years smoked.¹¹ According to criteria from Landis and Koch,¹² kappa values greater than 0.80 represent "almost perfect" agreement; those between 0.60 and 0.80 show "substantial" agreement; those between 0.40 and 0.60 show "moderate" agreement, and values below 0.40 represent "poor to fair" agreement.

The independent associations of gender, age, and education with test-retest agreement (outcome variable: "agree versus disagree") were investigated by fitting logistic regression models for each health-related behavior or exposure.

RESULTS

Overall, no substantial differences were observed between test and retest prevalence (Table 1, previous page). Information on smoking status and pack-years smoked was shown to be highly reliable: k = 0.97 (95% confidence interval, CI: 0.92-1.00) and ICC = 0.92 (95% CI: 0.88-0.95), respectively. A high level of agreement was also estimated regarding passive smoking (k = 0.73). Nevertheless, it is worth noting that the prevalence of passive smoking "at home only" estimated in the test (7.4%) was nearly double that of the retest (4.2%), which did not occur with passive smoking "at workplace only" or "at home and workplace".

Reliability figures for characteristics of alcohol intake were lower, with the kappa statistic varying between 0.62 (95% CI: 0.48-0.76) and 0.69 (95% CI: 0.58-0.80).

Among diet-related characteristics, information on fruit intake showed the highest agreement (k_w = 0.79), while information on intake of vegetables ranked lower (k_w = 0.67).

No clear-cut patterns could be identified by comparing kappa statistics by gender and age strata (Table 2). However, it is worth noting that information on pack-years smoked showed lower reliability among younger employees. In addition, there was a slight tendency towards higher reliability among people with higher levels of education.

The association between agreement and potential sociodemographic predictors (gender, age, and education) was further investigated -except for smoking, as this yielded reliability values close to 1.0 (data not presented). The responses regarding number of alcohol units, intake of vegetables and fruit showed no association (p < 0.10) with any of these potential sociodemographic predictors. However, women were over three times more likely than men to agree on passive smoking information: crude odds ratio, OR = 3.3; (95% CI: 1.5-7.2); after adjusting for age and education, OR = 3.6 (95% CI: 1.6-8.3) (data not shown in table).

DISCUSSION

Our results indicate good levels of reliability regarding information on important risk factors for major adult chronic diseases in this Brazilian study population. Reliability of the information on smoking, number of cigarettes smoked and exposure to passive smoking (except at home) ranged from substantial to almost perfect. Reproducibility of the other items regarding dietary patterns and alcohol intake ranged from moderate to almost perfect.

Studies conducted in the United States have also considered information on tobacco use and alcohol intake to be reasonably reproducible.^13,14 The reproducibility of information on dietary patterns tends to show more variability depending on the data collection instrument utilized and the frequencies of intake of specific dietary items.^2,4

The estimated measures of reliability have practical implications for future analyses of the Pró-Saúde Study data. We were especially interested in assessing possible differences across the responses given by sociodemographic subgroups. Gender, age, and education were each predictive of some variation in agreement for one or more risk factors, but no consistent effect of any of these characteristics on the reproducibility of information was observed. Therefore, the overall similar prevalence of the risk factors measured in both the test and the retest, and the absence of systematic differences in the information across sociodemographic strata suggest that potential misclassification of results in the main study would result mostly from non-differential errors.

Stein et al.³ found similar results when investigating demographic predictors of agreement in information on chronic disease risk factors: there was no consistent effect of any demographic characteristic across risk factors, although some of them predicted higher levels of concordance for one or more risk factors. In contrast, in two other studies,^2,15 agreement on information about preventive health practices and physical activity variables tended to be lower among non-white than white women.

Although good reliability does not ensure validity, reliability studies are often conducted insofar as they provide information on consistency and stability of the answers.^2-4,13-16 One important limitation of the test-retest method in estimating reproducibility must be considered. Any actual change in the condition or exposure under study, which may occur during the period between the test and the retest, will inevitably result in underestimated reliability. This may have been the case with the moderate reliability estimated in this study for the information on alcohol intake during the previous two weeks.

CONCLUSION

The need for reliability studies built into epidemiological investigations should not be overlooked, as levels of reliability can vary according to characteristics of the population and data collection methods. The reproducibility of information on smoking, drinking, and dietary patterns ranged from substantial to excellent, as investigated in the Pró-Saúde Study, a longitudinal investigation recently launched in Rio de Janeiro.

Sources of funding: Research Support Foundation of Rio de Janeiro (FAPERJ), grant no. E-26/170.714/2001

Conflicts of interest: None

Date of first submission: July 11, 2002

Last received: October 22, 2002

Accepted: November 1, 2002

Publishing information

Dóra Chor, MD, PhD. Escola Nacional de Saúde Pública, Fundação Oswaldo Cruz, Rio de Janeiro, Brazil.

Eduardo Faerstein, MD, PhD.Institute of Social Medicine, Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brazil.

Márcia Guimarães Mello Alves, MD, MPh.Doctoral student, Escola Nacional de Saúde Pública, Fundação Oswaldo Cruz, Rio de Janeiro, Brazil.

Claudia de Souza Lopes,MD, PhD.Institute of Social Medicine, Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brazil.

• 1. Sichieri R. Epidemiologia da obesidade. Rio de Janeiro: Eduerj; 1998.p.25-35.
• 2. Brownson RC, Eyler AA, King AC, Shyu Y-L, Brown DR, Homan SM. Reliability of information on physical activity and other chronic disease risk factors among US women aged 40 years or older. Am J Epidemiol 1999;149(4):379-91.
• 3. Stein AD, Courval JM, Lederman RI, Shea S. Reproducibility of responses to telephone interviews: demographic predictors of discordance in risk factor status. Am J Epidemiol 1995;141(11):1097-105.
• 4. Colditz GA, Willett WC, Stampfer MJ, et al. The influence of age, relative weight, smoking, and alcohol intake on the reproducibility of a dietary questionnaire. Int J Epidem iol 1987;16(3):392-8.
• 5. Castellsagué X, Muñoz N, De Stefani E, et al. Independent and joint effects of tobacco smoking and alcohol drinking on the risk of esophageal cancer in men and women. Int J Cancer 1999;82(5):657-64.
• 6. Aday LA. Designing and conducting health surveys: a comprehensive guide. San Francisco: Jossey-Bass Pub; 1991.p.165
• 7. Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. Oxford: Oxford University; 1989.p.55.
• 8. Szklo M, Javier Nieto F. Epidemiology: beyond the basics. Gaithersburg, Maryland: Aspen Publishers Inc; 2000.p.330.
• 9. Fleiss JL, Cohen J. Equivalence of weighted kappa and intraclass correlation coefficient as measures of reliability. Educ Psychol Meas 1973;33:613-9.
• 10. Bartko JJ, Carpenter WT. On the methods and theory of reliability. J Nerv Ment Dis 1976;163(5):307-17
• 11. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1(8476):307-10.
• 12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-74.
• 13. Bowlin SJ, Morrill BD, Nafziger AN, Lewis C, Pearson TA. Reliability and changes in validity of self-reported cardiovascular disease risk factor using dual response: the behavioral risk factor survey. J Clin Epidemiol 1996;49(5):511-7.
• 14. Brownson RC, Jackson-Thompson J, Wilkerson JC, Kiani F. Reliability of information on chronic disease risk factors collected in the Missouri Behavioral Risk Factor Surveillance System. Epidemiology 1994;5(5):545-9.
• 15. Stein AD, Lederman RI, Shea S. Reproducibility of the women's module of the Behavioral Risk Factor Surveillance System questionnaire. Ann Epidemiol 1996;6(1):47-52.
• 16. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): development, reliability and feasibility. Addiction 2002;97(9):1183-94.

Publication Dates

• Publication in this collection
  14 July 2003
• Date of issue
  2003

History

• Received
  11 July 2002
• Reviewed
  22 Oct 2002
• Accepted
  11 Nov 2002