Applying the RE-AIM conceptual framework for the promotion of physical activity in low- and middle-income countries

ABSTRACT

Objective:

the RE-AIM framework has been widely used to evaluate internal and external validity of interventions aimed to promote physical activity, helping to provide comprehensive evaluation of the reach, efficacy, adoption, implementation and maintenance of research and programming. Despite this progress, the RE-AIM framework has not been used widely in Latin America. The purpose of this manuscript is to describe the RE-AIM framework, the process and materials developed for a one-day workshop in Guadalajara, and the acceptability and satisfaction of participants that attended the workshop.

Methods:

lecture, interactive examples and an agenda were developed for a one-day RE-AIM workshop over a three month period.

Results:

thirty two health care practitioners (M age = 30.6, SD=9.9 years) attended the workshop. Most (100%) rated the workshop as credible, useful (100%) and intended to apply it in current or future research (95%).

Conclusion:

results suggest intuitive appeal of the RE-AIM framework, and provide a strategy for introducing the utility and practical application of the framework in practice settings in Mexico and Latin America.

Descriptors:
Latin America; Strategies; Evaluation of Programs; Exercise; Health Plan Implementation

RESUMO

Objetivo:

o modelo RE-AIM tem sido amplamente utilizado para avaliar a validade interna e externa de intervenções que objetivam promover atividade física, auxiliando a fornecer uma avaliação abrangente do alcance, eficácia, adoção, implementação e manutenção de pesquisas e programas. Apesar deste progresso, o modelo RE-AIM não tem sido amplamente utilizado na América Latina. O objetivo deste artigo é descrever o modelo RE-AIM, o processo e os materiais desenvolvidos para uma oficina de um dia de duração em Guadalajara e a aceitabilidade e satisfação dos participantes que compareceram na oficina.

Métodos:

durante o período de três meses, palestras e exercícios interativos foram desenvolvidos para uma oficina de um dia sobre o modelo RE-AIM.

Resultados:

trinta e dois profissionais de saúde (Média de idade=30,6±9,9 anos) participaram da oficina. A maioria dos participantes classificou a oficina como credível (100%), útil (100%) e pretende aplicar o que foi aprendido em investigações atuais ou futuras (95%).

Conclusão:

os resultados sugerem que o modelo RE-AIM é um recurso intuitivo e fornecem uma estratégia para a apresentação da utilidade e aplicação prática do modelo em cenários práticos no México e na América Latina.

Descritores:
América Latina; Estratégias; Avaliação de Programas; Exercício; Implementação de Plano de Saúde

RESUMEN

Objetivo:

el marco RE-AIM ha sido ampliamente utilizado para evaluar la validez interna y externa de intervenciones destinadas a promover la actividad física, ayudando a proporcionar una evaluación exhaustiva de la cobertura, eficacia/efectividad, adopción, implementación y mantenimiento de programas sobre actividad física. A pesar de este progreso, el marco RE-AIM no se ha utilizado ampliamente en América Latina. El objetivo de este artículo es presentar el marco conceptual RE-AIM, describir el proceso y los materiales desarrollados para un taller en Guadalajara; y reportar la aceptabilidad y satisfacción de los participantes que acudieron al taller.

Métodos:

durante un período de tres meses, se desarrolló un taller sobre el marco RE-AIM. El taller duró un día e incluyó una conferencia y ejercicios interactivos.

Resultados:

treinta y dos profesionales de la salud (edad media = 30.6 [±9.9 años]) participaron en el taller. La mayoría calificó el taller como creíble, útil (100%) y manifestó la intención de aplicar lo aprendido en investigaciones actuales o futuras (95%).

Conclusión:

los resultados sugieren que el marco RE-AIM es intuitivamente atractivo y que el curso es una estrategia adecuada para introducir la utilidad y aplicación práctica del marco en espacios de práctica en América Latina.

Descriptores:
América Latina; Estrategias; Evaluación de Programas; Ejercicio; Implementación de Plan de Salud

Introduction

Regularly performed physical activity (PA) is an international health priority^{(1, 2)}. Physical inactivity is endemic in Mexico, and a majority of Mexican children (58.6 %) and many adults (19.4%) fail to meet physical activity recommendations^3-6. Like other low and middle income countries (LMIC), the Mexican population is at high risk for developing health compromising conditions related to physical inactivity^7-9. Implementing research and practice in LMICs provides an opportunity to understand and investigate the application of research techniques such as the RE-AIM framework, that have, to date, almost entirely focused on high income countries.

The RE-AIM framework has been broadly applied in the US as well as other high -income countries across a wide array of PA-related research and programming; however, use of the framework has not been established in LMICs. The RE-AIM framework provides a model to inform the design, implementation and evaluation of physical activity, so its introduction in a LMIC country, like Mexico, is timely and promising. RE-AIM components of reach, efficacy/effectiveness, adoption, implementation and maintenance (individual and organizational) have been used to review internal and external validity of PA interventions using behavior change theories, school-based strategies, telephone-delivered strategies, workplace interventions, and interventions targeting cancer survivors^10-14.

Similar to other LMIC, infrastructure and public health needs in Mexico have historically focused on the prevention and treatment of infectious diseases. More recently, behavioral research has started to focus on preventing chronic conditions. In the case of Mexico, national statistics describing high rates of obesity and type 2 diabetes have led to recent sweeping policy changes. These commenced in 2012 with a change in political leadership, along with international supports, leading to activities throughout Mexico to improve cardiometabolic health, with a focus on increasing PA among all Mexicans². The Mexican health statistics and policy changes have provided a favorable macro-level context to plan and evaluate current research and evidence-based interventions to increase PA in Mexico. The use of frameworks like RE-AIM allow for the possibility of achieving a public health impact by focusing on a range of outcomes such as the reach, efficacy, adoption, implementation and maintenance of these strategies.

Like many LMIC, health promotion efforts in Mexico tend to rely on both clinic-based and community public health programming^15-19. Programs that have a broad reach into the population while demonstrating robust effectiveness across subgroups within the population can have a strong public health impact and may be considered for broad dissemination to other communities, systems and regions^20-21. Despite their promise, a review of PA public health programs in Mexico showed that programs might report reach and adoption, but there was poor monitoring and evaluation of factors related to effectiveness, implementation and maintenance²². The evidence supporting these programs is insufficient for determining public health impact, limiting the ability to implement these programs on a broader state or national scale^9,15,19,23. Current methods of evaluation and reporting exclude key areas that would facilitate dissemination about the expertise of those delivering the program, the program components, implementation activities and costs, the long-term sustainability of the programs and health and behavior outcomes for participants¹⁶.

Despite the ability of the RE-AIM framework to help researchers and practitioners evaluate and assess the public health impact, there has been little use of the framework in Latin American countries such as Mexico, in part driven by a lack of knowledge and expertise. There is a strong need to develop capacity among public health practitioners and health promoters so that efforts can be systematically evaluated in order to disseminate successful programs across Mexico and to other LMICs with sizeable Hispanic populations. In this manuscript, we present the development and outcomes of a RE-AIM training workshop delivered in Guadalajara, Mexico including examples of planning and evaluation across the RE-AIM framework.

Method

The work described herein is the result of nearly a decade of multinational collaboration that developed through a participatory process involving researchers from Canada, the United States and Mexico. The primary goals of the partnership have been to increase scientific capacity and infrastructure in México with the express objective of discovering, enhancing and implementing strategies across multiple settings to increase PA among Hispanics or Latinos throughout North America. The workshop was conceived as a strategy to meet both goals by improving the quality of evaluation of public health programming for PA in Mexico using culturally relevant and interactive examples and was presented as a pre-congress session to the Congreso Internacional de Avances en Medicina de “Hospitales Civiles de Guadalajara” in 2014.

The RE-AIM Framework

RE-AIM is comprised of five indicators: Reach, Efficacy/Effectiveness, Adoption, Implementation and Maintenance²⁴. These indicators can be used in the evaluation of programs, procedures, policies or scientific studies. Reach is defined as the number or percentage of the population and the representativeness of those included in the program or study. Efficacy and effectiveness measure change in the variable of interest as well as impact on quality of life and adverse outcomes. Adoption measures the number, percentage, and representativeness of staff and settings involved. Implementation assesses the extent to which a program or policy is delivered consistently, and the time and costs of the program. Maintenance assesses the long-term effects and attrition in the project, both of individuals and organizations. This includes the extent of discontinuation, modification, or sustainability of program.

Although there are other strategies for measuring process factors related to implementation of interventions that can describe internal and external validity, the RE-AIM framework has the advantages of being contextual, practical and having robust evidence of its applicability across a wide array of interventions, populations, settings and health behaviors. RE-AIM offers a systematic framework for expanding beyond the usual measures of efficacy and effectiveness, to the broader criteria of internal and external validity. RE-AIM moves away from a paradigm that focuses on the magnitude of effect as a key indicator for program/intervention impact towards a broader conceptualization of public health impact that includes reach, organizational adoption, and sustainability. RE-AIM attends to the characteristics of programs and interventions that ensure these can be readily adopted, widely implemented, and sustained. RE-AIM has been used to plan health interventions, evaluate health interventions, evaluate health policy impact, assess the literature, and to compute composite metrics to estimate intervention impact^25-29.

Participants

Thirty two health practitioners (M age = 30.6, SD=9.9 years) participated in an eight hour workshop in Guadalajara. Participants represented a broad array of health professions including, PA trainers (28.1%, N=9), physicians (28.1%, N=9), teachers (15.6%, N=5), nutritionists (9.4%, N=3), nurses (6.3%, N=2), community workers (6.3%, N=2), a psychologist (3.1%, N=1) and one student (3.1%, N=1). Cost of attendance was included as part of the overall Congress fees, and attendees were able to apply for continuing education credits by virtue of their participation.

Measures

Before the workshop commenced, all participants completed items on an anonymous, pre-workshop, simple paper and pencil survey and indicated their age and health professions. Participants also indicated whether they had heard of RE-AIM before, and whether they evaluated their PA programming in their work settings, and how they hoped to use the skills that they gained in the workshop.

Post-workshop, participants completed the remaining items on the survey. They were asked to rate the amount of new information that they learned in the workshop on a scale of 1 (learned no new information) to 7 (a lot of new information), how credible they found the information on a scale of 1 (not credible) to 7 (very credible). Participants also rated how likely it was that they would use the information gained either in their current profession or in the next six months on a scale of 1 (not very likely) to 7 (very likely). Last, participants indicated how interested they were in learning more about the issues presented in the workshop in a short course in the future on a scale of 1 (not interested) to 7 (very interested).

Surveys were distributed by the team, and later returned in a single file folder. Both surveys were developed for use with this workshop.

Development of the workshop

Classroom style lecture

The first half of the workshop (~4 hours) included classroom style lecture, where one of the co-authors provided information and examples of the RE-AIM framework using PowerPoint slides. This initial approach was based on information from previous literature reviews suggesting little awareness of the RE-AIM framework in México and from key information gleaned from the Mexican members of the multinational collaboration (described above). First, well-known examples from studies done in Latin American countries were presented to show that, although efficacy or effectiveness are widely reported in the published literature, information was insufficient to determine which interventions worked, for whom and under which conditions. For example, participants were polled during the lecture to consider examples which illustrated how to report factors related to scalability, such as cost. Another example was how retention and sustainability might make a difference in which program was more likely to be adopted across organizations. Polls were based on items from the RE-AIM measure presented in Figure 1, previously developed by Glasgow et al. and later expanded by Allen et al.^28-29.

Figure 1
RE-AIM components used in the development and implementation of the workshop.

Interactive examples

Materials were developed over a two month period involving three teleconference calls among the authors. Each example was developed by a team involving one Mexican partner and one Canadian/US partner. Examples were then reviewed by the group, and inconsistencies or quandaries were discussed and resolved via teleconference. Materials were developed to have plausibility within a Mexican context and were translated and back translated to Spanish by bilingual native Mexican Spanish speakers. Examples were developed around three general content areas: Policy and environmental changes, prevention and public health, self- management of chronic diseases. Subgroups discussed each content example independently and then regrouped as a large group to talk through the examples together.

Results

Workshop Agenda and Materials

The resulting workshop agenda featured classroom style learning for the entire group in the first half of the session, followed by a small group, interactive activities. After the small groups exercise, the entire group reconvened to discuss the activities and answer questions. Activity 1 presented two programs with information about the reach, efficacy/effectiveness, adoption, implementation and maintenance of each example. A summary is presented in Figure 2. Participants were asked to review the examples and then rate the two programs on a single grid, using a five-point rating system, where 1 equaled poor and 5 equaled excellent. As presented in Figure 3, this provided a way to visually compare each of the programs and evaluate which might be a better fit to meet the organizations goals.

Figure 2
Program examples used for interactive activities to demonstrate RE-AIM constructs

Figure 3
Sample of a completed rating form to compare programs in Activity 1.

In Activity 2, participants were asked to describe their own intervention idea, indicate which RE-AIM dimensions were targeted for intervention, and which might be described, but would not be targeted for intervention. Participants then had the opportunity to describe challenges that they would face in their organization.

Participant outcomes

Thirty two researchers and practitioners attended the RE-AIM training. Before the workshop, only five (15.6%) attendees had previously heard of or used the RE-AIM framework. About one fourth (N=8, 25%) used program evaluation tools in their work. Twenty-three (72%) indicated that they hoped to use the framework to guide their work in evaluating current and planning future projects, two (6%) in future academic training (e.g., thesis), and seven (22%) did not respond to the question of how they hoped to use the skills gained in the workshop.

Twenty-one participants completed the post workshop survey. After attending the training, 85.7% (N=18, M=6.33, SD=0.97) ranked a 6 or 7 indicating that they had learned a lot of new information in the workshop, and all ranked the information gleaned as credible (100%, N=21, M=6.71, SD=.46). Nearly all rated a 6 or 7 indicating that they would likely use the information acquired in the workshop in the current position (95%, N=20, M=6.71, SD=.56) or hope to use it in the next six months (95%, N=20, M=6.62, SD=.60). All participants indicated that they would be interested in participating in a short course to learn more about the RE-AIM framework (100%, N=21, M=6.85, SD=.36).

Discussion

This manuscript describes the development and reactions to the first RE-AIM workshop for public health practitioners delivered in Mexico. Relying on empirical evidence, we carefully constructed a culturally relevant workshop that produced favorable knowledge acquisition among a group of public health practitioners in Mexico. It was clear from the response to the workshop that there was intuitive appeal for RE-AIM to this audience of practitioners who rated the information presented as useful and credible, and desired to learn more about it for current or future use in their work.

In a time where PA programming has reached the public health agenda, opportunities for improving current programs, disseminating successful strategies and informing future public health initiatives are numerous. RE-AIM can be used for evaluating the reach, impact and implementation of current PA initiatives at the participant, organization and policy levels. Further, by addressing cost, adoption and implementation factors, RE-AIM can guide the expansion and sustainability of successful programs across Mexico. By focusing on factors associated with the reach, real-world implementation, dissemination, and sustainability of successful PA programs, such initiatives could reach broader populations, a wide range of organizations and inform decision makers.

Public health programming in Mexico is to some extent driven by political priorities; thus, when administrations change, so do public health promotion priorities. For example, in 2012, the change in administrations and health promotion strategy emphasized greater promotion of health behaviors related to obesity and diabetes in Mexico. This laudable change in priorities had effects throughout the country, increasing policies and programming at the national and state level that focused on PA and nutrition. Rapid shifts in priorities on a national level may leave little time to plan careful evaluation, even in the case of very positive changes as have been seen in Mexico. In the context of a politically driven public health system, individual practitioners, although motivated and well trained, may have little control over planning, implementation and evaluation of programming. RE-AIM framework can be helpful to show how these political priorities put emphasis on some aspects of programs such as reach, while other dimension are neglected including implementation or maintenance. During the workshop, participants made additional system considerations, concerning the environment and specific policies for Mexico, and focused on how to apply the RE-AIM framework in such contexts and political conditions.

Strengths of the reported experience included the dissemination of a well-researched and validated evaluation framework, careful crafting of relevant examples and interactive exercises, development and delivery by an experienced, multilingual team of researchers and practitioners and a very positive reception from a group of inexperienced participants. This study relied on a relatively small sample size, insufficient for more elaborate statistical modeling. Although pre- and post-workshop surveys were completed anonymously, self-report measures can suffer from response bias. Future research and development in this area should emphasize continued development of locally relevant examples and interactive activities and streamlining of RE-AIM measures to help aid adoption on a broader scale. From a translational science perspective, public health programming evaluation derived from the RE-AIM framework can help practitioners and policy makers predict the behavior of organizations and key stakeholders who are instrumental in the wide-spread adoption and successful implementation of evidence-based programs. Understanding where programming is both successful and challenging in the process of adoption and implementation can in turn drive their potential for sustainability, needed adaption for scaling up, and areas ripe for expansion.

Despite the strengths and potential for gain that the RE-AIM framework offers, there are areas of additional development and future research, particularly in Mexico. For example, most attendees had a clear and immediate understanding of efficacy/effectiveness, but struggled with some of the other concepts such as adoption. It was very helpful to have carefully constructed and clearly translated definitions and examples for exercises and discussion to help illustrate how the RE-AIM could be applied in the local context. Examples based on real life programs would have helped to anchor constructs even more clearly; however, we struggled to find programs from Mexico that reported enough RE-AIM indicators to use as examples²². It was a great strength to have an experienced team of presenters that included native speakers to help explain constructs to the diverse audience and overcome barriers to understanding. Last, the RE-AIM framework may help enhance teamwork by providing clear definitions and real-world understanding of health outcomes. Workshops such as this one can promote understanding, communication, and planning across multiple disciplines, enhancing interprofessionalism and successful teamwork³⁰.

Conclusion

Although the RE-AIM framework was initially conceptualized as a model to evaluate research, in reality, most interventions are tempered by the community in which they are administered. Programs can operationalize processes that involve local organizations and the intervention community itself. For example, in health care settings, interventions and programming must be designed to integrate within the existing organizational processes as well as the abilities of the practitioners and the reach of the clinic. There is an important role for partnerships between practitioners and researchers so that practice realities can inform research ideals. There is also room for simplification and streamlining of measures of the RE-AIM constructs for use under real-world conditions with limited resources. Next steps for translation of the RE-AIM framework include adapting the RE-AIM strategy to evaluate programming that builds on existing resources, requires little advance planning, and that is reasonably easy to accomplish given existing resources.

Acknowledgements

The authors would like to acknowledge and thank Bruna Gonçalves C. da Silva for her contribution of translating the manuscript to Portuguese.

References

• ¹
  National Physical Activity Plan Alliance. Proceedings; 2015 February 23-24; Washington, DC: National Physical Activity Plan Congress. Available from: http://www.physicalactivityplan.org/index.html
  » http://www.physicalactivityplan.org/index.html
• ²
  World Health Organization. Interim report of the Commission on Ending Childhood Obesity: Open for comment. March-June 2015. Available from: http://www.who.int/end-childhood-obesity/interim-report-for-comment/en/
  » http://www.who.int/end-childhood-obesity/interim-report-for-comment/en
• ³
  Medina C, Janssen I, Campos I, Barquera S. Physical inactivity prevalence and trends among Mexican adults: Results from the National Health and Nutrition Survey (ENSANUT) 2006 and 2012. BMC Public Health. 2013;13:1063. doi: 10.1186/1471-2458-13-1063.
  » https://doi.org/10.1186/1471-2458-13-1063
• ⁴
  Romero-Martinez M, Shamah-Levy T, Franco-Nunez A, Villalpando S, Cuevas-Nasu L, Gutierrez JP, et al. National Health and Nutrition Survey 2012: Design and coverage. Salud Publica Mex. [Internet]. 2013;55(Suppl 2):S332-40. Available from: http://ref.scielo.org/tmvbtr
  » http://ref.scielo.org/tmvbtr
• ⁵
  Morales-Ruán MC, Méndez-Gómez I, Shamah-Levy T, Valderrama-Álvarez Z, Melgar-Quiñonez H. La inseguridad slimentaria esta asociada con obesidad en mujeres adultas en México. Salud Publica Mex. [Internet]. 2014 [Acceso 13 julio 2016];56:54-61. Disponible en: http://ref.scielo.org/bszqjy
  » http://ref.scielo.org/bszqjy
• ⁶
  Rodriguez Martinez M, Galaviz KI, Ulloa EJ, Gonzalez-Casanova I, Lopez y Taylor JR. Results from Mexico’s 2014 report card on physical activity for children and youth. J Phys Act Health. 2014;11(Suppl 1):S74-8. doi: 10.1123/jpah.2014-0172.
  » https://doi.org/10.1123/jpah.2014-0172
• ⁷
  Acosta-Cazares B, Escobedo-de la Pena J. High burden of cardiovascular disease risk factors in Mexico: An epidemic of ischemic heart disease that may be on its way? Am Heart J. 2010;160(2):230-6. doi: 10.1016/j.ahj.2010.05.016.
  » https://doi.org/10.1016/j.ahj.2010.05.016
• ⁸
  Villalpando S, de la Cruz V, Rojas R, Shamah-Levy T, Avila MA, Gaona B, et al. Prevalence and distribution of type 2 diabetes mellitus in Mexican adult population: A probabilistic survey. Salud Publica Mex. [Internet]. 2010 [Access Jan 11, 2017];52(Suppl 1):S19-26. Available from: http://ref.scielo.org/85g77m
  » http://ref.scielo.org/85g77m
• ⁹
  Jauregui A, Medina C, Salvo D, Barquera S, Rivera-Dommarco JA. Active commuting to school in Mexican adolescents: Evidence from the Mexican National Nutrition and Health Survey. J Phys Act Health. 2015;12(8):1088-95. doi: 10.1123/jpah.2014-0103
  » https://doi.org/10.1123/jpah.2014-0103
• ¹⁰
  Antikainen I, Ellis R. A RE-AIM evaluation of theory-based physical activity interventions. J Sport Exerc Psychol. [Internet]. 2011 [Access Jan 11, 2017];33(2):198-214. Available from: http://journals.humankinetics.com/doi/abs/10.1123/jsep.33.2.198
  » http://journals.humankinetics.com/doi/abs/10.1123/jsep.33.2.198
• ¹¹
  Estabrooks P, Dzewaltowski DA, Glasgow RE, Klesges LM. Reporting of validity from school health promotion studies published in 12 leading journals, 1996-2000. J Sch Health. 2003;73(1):21-8. doi: 10.1111/j.1746-1561.2003.tb06554.x.
  » https://doi.org/10.1111/j.1746-1561.2003.tb06554.x
• ¹²
  Goode AD, Reeves MM, Eakin EG. Telephone-delivered interventions for physical activity and dietary behavior change: an updated systematic review. Am J Prev Med. 2012;42(1):81-8. doi: 10.1016/j.amepre.2011.08.025.
  » https://doi.org/10.1016/j.amepre.2011.08.025
• ¹³
  Vuillemin A, Rostami C, Maes L, Van Cauwenberghe E, Van Lenthe FJ, Brug J, et al. Worksite physical activity interventions and obesity: a review of European studies (the HOPE project). Obes Facts. 2011;4(6):479-88. doi: 10.1159/000335255.
  » https://doi.org/10.1159/000335255
• ¹⁴
  White SM, McAuley E, Estabrooks PA, Courneya KS. Translating physical activity interventions for breast cancer survivors into practice: An evaluation of randomized controlled trials. Ann Behav Med. 2009;37(1):10-9. doi: 10.1007/s12160-009-9084-9.
  » https://doi.org/10.1007/s12160-009-9084-9
• ¹⁵
  Staten LK, Cutshaw CA, Davidson C, Reinschmidt K, Stewart R, Roe DJ. Effectiveness of the Pasos Adelante chronic disease prevention and control program in a US-Mexico border community, 2005-2008. Prev Chronic Dis. [Internet]. 2012 [Access Jan 11, 2017];9:E08. Available from: https://www.cdc.gov/pcd/issues/2012/10_0301.htm
  » https://www.cdc.gov/pcd/issues/2012/10_0301.htm
• ¹⁶
  Carvajal SC, Miesfeld N, Chang J, Reinschmidt KM, de Zapien JG, Fernandez ML, et al. Evidence for long-term impact of Pasos Adelante: Using a community-wide survey to evaluate chronic disease risk modification in prior program participants. Int J Environ Res Public Health. 2013;10(10):4701-17. doi: 10.3390/ijerph10104701.
  » https://doi.org/10.3390/ijerph10104701
• ¹⁷
  Balcazar HG, de Heer H, Rosenthal L, Aguirre M, Flores L, Puentes FA, et al. A promotores de salud intervention to reduce cardiovascular disease risk in a high-risk Hispanic border population, 2005-2008. Prev Chronic Dis. [Internet]. 2010 [Access Jan 11, 2017];7(2):A28. Available from: https://www.cdc.gov/pcd/issues/2010/Mar/09_0106.htm
  » https://www.cdc.gov/pcd/issues/2010/Mar/09_0106.htm
• ¹⁸
  Azevedo KJ, Mendoza S, Fernandez M, Haydel KF, Fujimoto M, Tirumalai EC, et al. Turn off the TV and dance! Participation in culturally tailored health interventions: Implications for obesity prevention among Mexican American girls. Ethn Dis. [Internet]. 2013 [Access Jan 19, 2017];23(4):452-61. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3940265/
  » https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3940265
• ¹⁹
  Sarmiento O, Torres A, Jacoby E, Pratt M, Schmid TL, Stierling G. The Ciclovia-Recreativa: A mass-recreational program with public health potential. J Phys Act Health. 2010;7(Suppl 2):S163-80. Available from: http://journals.humankinetics.com/doi/abs/10.1123/jpah.7.s2.s163
  » http://journals.humankinetics.com/doi/abs/10.1123/jpah.7.s2.s163
• ²⁰
  Abrams DB, Orleans CT, Niaura RS, Goldstein MG, Prochaska JO, Velicer W. Integrating individual and public health perspectives for treatment of tobacco dependence under managed health care: A combined stepped-care and matching model. Ann Behav Med. [Internet]. 1996 [Access Jan 19, 2017];18(4):290-304. Available from: https://link.springer.com/article/10.1007/BF02895291
  » https://link.springer.com/article/10.1007/BF02895291
• ²¹
  Estabrooks PA, Gyurcsik NC. Evaluating the impact of behavioral interventions that target physical activity: Issues of generalizability and public health. Psychol Sport Exerc. 2003;4(1):41-55. Available from: http://www.sciencedirect.com/science/article/pii/S146902920200016X
  » http://www.sciencedirect.com/science/article/pii/S146902920200016X
• ²²
  Jauregui E, Pacheco AM, Soltero EG, O’Connor TM, Castro CM, Estabrooks PA, et al. Using the RE-AIM framework to evaluate physical activity public health programs in Mexico. BMC Public Health. 2015;15:162. doi: 10.1186/s12889-015-1474-2.
  » https://doi.org/10.1186/s12889-015-1474-2
• ²³
  Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am J Public Health. [Internet]. 1999 [Access Jan 18, 2017];89(9):1322-7. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1508772/
  » https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1508772
• ²⁴
  Reach Effectiveness Adoption Implementation Maintenance (RE-AIM). [Internet]. [Access Feb 13, 2017]. Available from: http://www.re-aim.hnfe.vt.edu/.
  » http://www.re-aim.hnfe.vt.edu
• ²⁵
  Klesges LM, Estabrooks PA, Dzewaltowski DA, Bull SS, Glasgow RE. Beginning with the application in mind: Designing and planning health behavior change interventions to enhance dissemination. Ann Behav Med. 2005;29(2):66-75. doi:10.1207/s15324796abm2902s_10.
  » https://doi.org/10.1207/s15324796abm2902s_10
• ²⁶
  Eakin EG, Lawler SP, Vandelanotte C, Owen N. Telephone interventions for physical activity and dietary behavior change: A systematic review. Am J Prev Med. [Internet]. 2007 [Access Jun 19, 2016];32(5):419-34. Available from: http://dx.doi.org/10.1016/j.amepre.2007.01.004
  » http://dx.doi.org/10.1016/j.amepre.2007.01.004
• ²⁷
  Jilcott S, Ammerman A, Sommers J, Glasgow RE. Applying the RE-AIM framework to assess the public health impact of policy change. Ann Behav Med. 2007;34(2):105-14. doi:10.1007/BF02872666.
  » https://doi.org/10.1007/BF02872666
• ²⁸
  Allen K, Zoellner J, Motley M, Estabrooks PA. Understanding the internal and external validity of health literacy interventions: A systematic literature review using the RE-AIM framework. J Health Commun. 2011;16 Suppl 3:55-72. doi: 10.1080/10810730.2011.604381.
  » https://doi.org/10.1080/10810730.2011.604381
• ²⁹
  Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: What is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27(1):3-12. doi:10.1207/s15324796abm2701_2.
  » https://doi.org/10.1207/s15324796abm2701_2
• ³⁰
  Dufrene C. Health care partnerships: A literature review of interdisciplinary education. Journal Nurs Educ. 2012;51(4):212-6. doi: 10.3928/01484834-20120224-01.
  » https://doi.org/10.3928/01484834-20120224-01

Publication Dates

• Publication in this collection
  2017

History

• Received
  10 Nov 2016
• Accepted
  11 May 2017