Evaluation practices in the field of Food and Nutrition

Práticas avaliativas no campo da Alimentação e Nutrição

ABSTRACT

The objective of this study was to characterize and analyze the different existing methods for the evaluation of food and nutrition programs and services in Brazil, through a systematic review of the literature focused on complete articles published in national indexed journals. We searched the PubMed, MedLine and LILACS databases using the following search terms and Boolean operators: “evaluation and program”; “project”; “intervention”; “servisse”; “actions and nutrition”; “nutritional”. The research was restricted to articles written in Portuguese, English and Spanish and published between 2001 and 2015. Twenty-two studies were selected and the analysis indicates; most were carried out through quantitative approaches and external evaluations based on epidemiological theory; participatory evaluation strategies are still uncommon; Impact assessments and implementation were predominant; there is little diversity in terms of references to the theoretical framework in the field of evaluation of health care programs in the planning and execution of the evaluation processes analyzed. The results of this study indicate the need for a more comprehensive evaluation considering the complexity of the interventions evaluated using the theoretical-methodological apparatus available in the literature to understand the importance of the different perspectives of the agents involved in the evaluation processes.

Keywords:
Program evaluation; Nutrition policy; Nutrition, public health.

RESUMO

O objetivo deste estudo foi caracterizar e analisar os diferentes métodos existentes para a avaliação de programas e serviços de alimentação e nutrição no Brasil, por meio de revisão sistemática da literatura focada em artigos completos publicados em revistas indexadas nacionais. Foi realizada uma pesquisa nas bases de dados PubMed, MedLine e LILACS utilizando os seguintes termos de busca e operadores booleanos: “avaliação e programa”; “projeto”; “intervenção”; “serviço”; “ações e nutrição”; “nutricional”. A pesquisa foi restrita a artigos escritos em português, inglês e espanhol e publicados entre 2001 e 2015. Vinte e dois estudos foram selecionados e a análise indica; a maioria foi realizada por meio de abordagens quantitativas e avaliações externas baseadas na teoria epidemiológica; as estratégias de avaliação participativa ainda são incomuns; as avaliações de impacto e implementação foram predominantes; há pouca diversidade em termos de referências ao arcabouço teórico no campo da avaliação de programas de atenção à saúde no planejamento e execução dos processos de avaliação analisados. Os resultados deste estudo indicam a necessidade de uma avaliação mais abrangente, considerando a complexidade das intervenções avaliadas utilizando o aparato teórico-metodológico disponível na literatura para compreender a importância das diferentes perspectivas dos agentes envolvidos nos processos de avaliação.

Palavras-chave:
Avaliação de programas e projetos de Saúde; Nutrição em saúde pública; Política nutricional.

INTRODUCTION

The evaluation of social projects and programs in Brazil has developed since the mid-1990s, especially in the third and health sectors [¹]. Three factors influenced the progress of health assessment in Brazil: the development of Collective Health, including its planning and management sub-area; the establishment of the Sistema Único de Saúde (SUS, Unified Health System), which required greater knowledge and improvement of the initiatives being implemented; and, finally, the support provided by international financial institutions to carry out evaluations, usually following the procedures adopted in powerful countries [²].

Initiatives of the Ministry of Health, such as the Projeto de Expansão e Consolidação do Programa de Saúde da Família (PROESF, Project for the Expansion and Consolidation of the Family Health Program) and the Projeto Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica (PMAQ, National Project for Improvement of Access and Quality of Primary Health Care), and special notice given by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq, The National Council for Scientific and Technological Development), encouraged the expansion and consolidation of research centers in several Brazilian universities and state and municipal health secretariats, promoting the so-called institutionalization of evaluation [³].

Health assessment has developed within Collective Health, as shown by the presence of more than 200 research groups working on this topic scattered across more than 80 higher education institutions affiliated to CNPq [²]. There has also been an increase in the number of articles on evaluation in national journals and their special thematic issues, as reported by Santos [⁴]. Since 2006, the Associação Brasileira de Saúde Coletiva (Brazilian Association of Collective Health) has had a thematic working group addressing monitoring and evaluation. Several health system subsectors have been designing and carrying out evaluation studies, such as primary care [⁵], sexually transmitted diseases/Aids [⁶], mental health [⁷], and oral health [⁸] among others.

The field of food and nutrition started developing after the 1970s, with emphasis on the protection of the human right to food, in the context of the Brazilian sanitary reform and the establishment of the SUS [⁹]. The development of policies on food and nutrition and food and nutrition security, as well as the consequent recognition of food as a constitutional right guaranteed by the Organic Law for Food and Nutrition Security, have demanded more accountability and transparency regarding the results achieved through the innovations.

The second version of the National Food and Nutrition Policy, implemented in 2012, recommends the continuous monitoring and evaluation of food and nutrition actions within the SUS. It also emphasizes the need to invest in research on the design and evaluation of the programs proposed in this policy to ensure the planning of nutritional care provided by the SUS [¹⁰].

Among the principles of the Marco de Referência de Educação Alimentar e Nutricional para as Políticas Públicas (Food and Nutrition Education Framework for Public Policies) is the importance given to the evaluation of the actions within the Educação Alimentar e Nutricional (EAN, Food and Nutrition Education). Therefore, according to what is described in page (p.32) [¹¹], the EAN should be perceived based on a methodological framework that includes a participatory planning, monitoring, and evaluation process.

The potential of learning about programs and interventions on food and nutrition has been recognized; therefore, if the lack of impact evaluation methods is overcome, decision makers can focus only on the long-term effects of the interventions evaluated [⁵].

The complex, intersectoral and participatory characteristics of the actions that guide food and nutrition security justifies changing the traditional evaluations focusing on the impact of the food and nutrition programs. This focus on inputs and outputs has disregarded many aspects and factors of an intervention, which, in this case, have the characteristic of a “black box” or an independent variable [¹²].

According to studies on trends in the evaluation of public interventions in this field, a well-established and fully developed food and nutrition evaluation is a goal to be reached [¹³,¹⁴]. In order to deepen the discussion about evaluation in this field, our aim is to investigate in a systematic way how the evaluation studies in the field of Food and Nutrition have been applied and carried out, based on indexed scientific articles, characterizing their current scenario and analyzing their focuses, theoretical and methodological perspectives, and possible limitations.

METHODS

A systematic literature review [¹⁵-¹⁸] was conducted in order to answer the following question: how have evaluation studies been carried out in the field of food and nutrition? Therefore, a search of scientific articles published between 2001 and 2015 addressing this topic was conducted. Literature reports such as theses, dissertations, conference proceedings, etc. were excluded.

Search terms were defined based on the descriptors found in the Descritores em Ciências da Saúde (DeCs, Health Sciences Descriptors) of the Biblioteca Virtual em Saúde (BVS, Virtual Health Library). The terms defined were searched in titles, abstracts, and topics. The following search terms and Boolean operators were defined: “evaluation and program”; “project”; “intervention”; “service”; “actions and nutrition”; “nutritional”, which resulted in 6,809 articles.

The first search for readily available articles was based on the following inclusion criteria: articles indexed in the Pubmed, MedLine, and LILACS databases; articles published between the 2001 and 2015; articles in Portuguese, English, and Spanish; and articles having Brazil as the publication country. After this refinement, 561 articles were selected for screening of titles and abstracts. The search was conducted between August and September 2015.

The exclusion criteria were as follows: a) studies that did not address the topic of nutrition and human consumption; b) duplicate copies; c) studies focusing on isolated aspects of an intervention without a systematic evaluation; d) studies focusing on specific and isolated interventions that did not include planned actions; and e) studies in which the term ‘evaluation’ referred to the technical domain of anthropometric evaluation rather than evaluation of programs and services provided. A total of 22 articles were finally selected, which were carefully read and categorized according to the following analysis guide:

• - Initiative evaluated

• - Program/service purpose

• - Analysis focus

• - Evaluation approach

• - Type of evaluators (internal and external) and affiliation

• - Methodology adopted

• - Sources of funding/partnerships

• - Evaluation criteria

• - Judgments

Two researchers conducted the search individually in the chosen databases comparing the articles initially retrieved in order to select the articles for analysis. In the event of any discrepancy, a third researcher was appointed to settle any issue and help decide on the final selection of the articles retrieved.

The analysis of the articles selected (Chart 1) showed that most of them used the term ‘evaluation’ referring to clinical and epidemiological studies, focusing on the measurement, description, and identification of biological aspects of health and the analysis of individual nutritional indicators. In some cases, some programs were mentioned. The aspects they analyzed were somehow related to their objectives. However, in those articles, the programs themselves were not the focus of the research. Some studies evaluated the actions of nutritional interventions on individuals and population groups using some parameters as the anthropometric evaluation [¹⁹,²⁰], and the evaluation of the nutritional intervention was focused on a specific population group [²¹,²²], among others.

Chart 1
Studies evaluating food and nutrition programs and policies according to the program, affiliation, methodology, and objective of the evaluation (2001-2015).

Author	Year	Program Evaluated	Affiliation	Type of evaluator	Methodology	Evaluation objective
Castro & Monteiro [19]	2002	Leite é Saúde (Milk means heath)	UERJ; USP	External	Quantitative	Evaluate the impact of the program “Leite é Saúde” on the nutritional recovery of malnourished children receiving care in the municipal health care system of Rio de Janeiro
Augusto & de Souza [20]	2010	Vivaleite (Milk Program)	USP	External	Quantitative	Evaluate the effectiveness of a public supplementary feeding program in promoting weight gain in children
Gabriel et al. [21]	2008	Programa deIntervenção (Intervention Program)	UFSC	Internal	Quantitative	Evaluate the results of a nutritional program for the promotion of healthy eating in elementary school students
Lana et al. [22]	2004	Programa 17 Passos (Seventeen Steps Program)	UFMG	External	Quantitative	Evaluate the impact of the 17 Passos program on the continuation of breastfeeding beyond 2 months
Amorim, et al. [23]	2012	Cantina Escolar Saudável (Healthy School Cafeteria)	UnB; UFSC	External	Quantitative	To present the development and adoption of a method to evaluate the implementation of a Healthy School Cafeteria
Faleiros et al. [24]	2005	ProgramaPuericultura (Child Well-being Program)	UFPEL	External	Quantitative	Evaluate the impact of a Child Well-being Program on the promotion of exclusive breastfeeding
Vidal et al. [25]	2011	Programa de Atenção Integral à Saúde da Criança (Comprehensive Children Health Care Program)	IMIP; UFPE	External	Quantitative	Evaluate the level of implementation of the Comprehensive Children Health Care Program in Pernambuco
Carvalho et al. [26]	2009	Programa de Incentivo ao Combate às Carências Nutricionais (Incentive Program to Overcome Nutritional Deficiencies)	IS/SES- São Paulo; SMS-Assis	External	Quantitative	Evaluate the impact of the Programa de Incentivo ao Combate às Carências Nutricionais in the city of Assis, São Paulo State
Bezerra et al. [27]	2007	Programa Saúde da Família (Family Health Program)	Fiocruz, IMP	External	Quantitative	Evaluate The Level of Implementation of The Breastfeeding Promotion Program in Public Health Care Facilities in Recife, Pernambuco
Veloso & Santana [28]	2002	Programa de Alimen-tação do Trabalhador (Worker Feeding Program)	UFBA	External	Quantitative	Evaluate the nutrition impact of the program
Veloso et al. [29]	2007	Programa de Alimen-tação do Trabalhador (Worker Feeding Program)	UFBA	External	Quantitative	Evaluate the impact of worker feeding programs on weight gain and overweight
Almeida et al. [30]	2010	Programa de Incentivo ao Combate às Carências Nutricionais (Incentive Program to Overcome Nutritional Deficiencies)	UFPB	External	Quantitative and Qualitative	Evaluate the program in a municipality in the Northeastern region in Brazil with parents or guardians of children less than five years of age
Martins et al. [31]	2007	Programa de Incentivo ao Combate às Carências Nutricionais (Incentive Program to Overcome Nutritional Deficiencies)	MS; UnB; UFBA; UNIFACS; FATEC	External	Quantitative	Evaluate food security and fight against hunger programs in the state of Bahia, including the Programa Nacional de Combate às Deficiências da Vitamina A (National Program to Overcome Vitamin A Deficiencies)
Vargas et al. [32]	2011	Programa de Intervenção Intervention Program	UFRJ; UERJ	External	Quantitative	Evaluate the effects of a obesity prevention program on the dietary practices in adolescents attending public schools
Spinelli & Canesqui [33]	2004	Programa Nacional de Alimentação Escolar (National School Feeding Program)	UFTM; UNICAMP	External	Quantitative and Qualitative	Evaluate the implementation of the municipal national school Feeding Program in Cuiabá, Mato Grosso
Dubeux et al. [34]	2004	Programa Saúde da Família (Family Health Program)	IMIP	External	Quantitative	Evaluate the level of implementation of Breastfeeding Promotion Program
Santos et al. [35]	2005	Vivaleite (Milk Program)	UnB; UFPEL	External	Quantitative	Evaluate the impact of the Milk Program on child growth and body composition and on the level of maternal compliance with the recommendations regarding the use of milk as a nutritional supplement
Sá & Szarfarc [36]	2009	Vivaleite (Milk Program)	USP	External	Quantitative	Compare the prevalence of anaemia in children aged less and older than 6 months consuming iron-fortified milk supplied by the Vivaleite Program and receiving nutritional guidance
Bandoni et al. [37]	2010	Programa Cozinhas Comunitárias Community (Cafeteria Program)	USP	External	Quantitative	Describe and evaluate the Community Cafeterias supported by the Ministério do Desenvolvimento Social (Brazilian Ministry of Social Development) in 2006
Almeida et al. [38]	2010	Cegonha Feliz (Happy Stork Program)	UEL	External	Quantitative	Investigate the prevalence of breastfeeding at sixth months in children who were born before and after the implementation of a children’s morbid mortality reduction program
Paiva et al. [39]	2011	Programa de Incentivo ao Combate às Carências Nutricionais (Incentive Program to Overcome Nutritional Deficiencies)	UFPI; UEPB	Internal	Quantitative	Carry out a descriptive analysis of the Programa Nacional de Combate às Deficiências da Vitamina A (National Program to Overcome Vitamin A Deficiencies) in the state of Paraíba
Cardoso et al. [40]	2007	Iniciativa Unidade Básica Amiga da Amamentação (Breastfeeding-Friendly Primary Care Unit Initiative)	Fiocruz; UGF	External	Quantitative	Compare the prevalence of breastfeeding and the main reasons for seeking care in a neonatal clinic of a basic care unit in Rio de Janeiro

• Nota: UnB: Universidade de Brasília; UFSCar: Universidade Federal de São Carlos; UERJ: Universidade Estadual do Rio de Janeiro; USP: Universidade de São Paulo; UFPEL: Universidade Federal de Pelotas; UFPE: Universidade Federal de Pernambuco; IS/SES: Instituto de Saúde/Secretaria do Estado da Saúde (Health Institute, State Health Secretariat); SMS: Secretaria Municipal de Saúde (Municipal Health Secretariat); IMIP: Instituto Materno Infantil de Pernambuco (Maternal and infant Institute of Pernanbuco); UFBA: Universidade Federal da Bahia; UFPB: Universidade Federal da Paraíba; MS: Ministério da Saúde (Ministry of Health); UNIFACS: Universidade Salvador; FATEC: Faculdade de Tecnologia; UFRJ: Universidade Federal do Rio de Janeiro; UFTM: Universidade Federal do Triângulo Mineiro; UNICAMP: Universidade Estadual de Campinas; UFMG: Universidade Federal de Minas Gerais; UEL: Universidade Estadual de Londrina; UFPI: Universidade Federal do Piauí; UEPB: Universidade do Estado da Paraíba; Fiocruz: Fundação Oswaldo Cruz; UGF: Universidade Gama Filho.

The most commonly used term in these scientific studies to designate the actions evaluated was “intervention”, and the majority was used to designate professional practices aimed to target groups or certain individuals through specific actions. They did not refer to programs or services, which are here understood as organized and planned efforts to mobilize resources and workers and develop a techno-assistance model with the aim of solving social problems, improving certain social conditions, and promoting collective well-being [⁴¹]. Moreover, the notion of program and/or service refers to intervention in social reality, inserted in the context of a public policy [⁴²]. Therefore, the programs vary according to broader political, economic, and social scenarios, and government support is essential for their planning, implementation, and evaluation.

Type of evaluators and their affiliation or connection

External evaluation was defined by Scriven [⁴³] as the evaluation performed by individuals that were not involved in any stage of program planning or execution, i.e., those who do not have any connection or responsibilities with the program prior to the evaluation. External evaluators’ greater credibility may come because of their supposed neutrality or impartiality and their expertise [²¹]. On the other hand, internal evaluation refers to the evaluation conducted by evaluators that were involved in the execution or planning of the program [⁴⁴] and therefore have comprehensive knowledge of its characteristics and problems, such as its organizational matrix [²¹].

In addition to the debate over external versus internal evaluations, there is a greater issue regarding the participation of interest groups in the conception, design, and use of evaluation practices. House [⁴⁵] highlights the role of evaluators in a context in which evaluations are increasingly subjected to political interests. The author adds that the independence of an evaluation process is related to the ability to identify and deal with different interests involved, which is essential to develop more democratic evaluations [²⁵]. The so-called Deliberative Democratic Evaluation is characterized by three principles: 1) inclusion of stakeholders’ relevant information and opinions and their values and interests in the evaluation; 2) extensive dialogue between evaluators and other stakeholders seeking to foster understanding among them; and, 3) deliberation by all parties to help reaching evaluative conclusions [²⁵].

The choice of external, internal, or a mixed team of evaluators, depends on the context of each reality; thus, such decision requires careful consideration [⁴⁶]. Unsuitable internal or external evaluators can lead to the deterioration of the relationships between the subjects evaluated and those who occupy leadership positions [⁴⁷]. Since one of the most important aspects of an evaluation is pronouncing judgment, the decentralization of power and delegation of authority to evaluators require receptivity and accessibility by those who are managing the evaluations and those who are subject to the evaluation process [¹⁹].

External evaluations were predominant in the articles reviewed, which seems to indicate the following: a need for the development and adoption of methodological strategies that can guarantee the inclusion of non-specialists in the evaluation processes and other interest groups in the field under study here; institutional environments more conducive to interactive exchanges and democratic debate required by participatory evaluations. In fact, according to previous reports, an effective participation requires overcoming several obstacles and unawareness of the real challenges and difficulties is the biggest obstacle [⁴⁸]. These are issues that have been identified in food and nutrition evaluations, and they are in accordance with the characteristics of evaluations in the field of Social Services [⁴⁹].

The studies analyzed also showed that the great majority of the external evaluators were affiliated to federal public universities and were not part of the teams responsible for the programs evaluated. In the empirical domain of evaluation initiatives in the field of evaluation and nutrition, there are characteristics similar to those of a more general health evaluation, such as: the relationship between agents of the scientific field (universities and research groups) and agents of the bureaucratic field (usually working in municipal and state health secretariats). In the present study, however, the academic representatives play a prominent role in the process, characterizing it as an external evaluation.

This association between the bureaucratic and scientific fields is also evidenced by the fact that among the articles analyzed that revealed the source of resources used in the evaluation, three were carried out with the financial support from the Ministry of Health [¹⁴,²³,³⁴], and three others received support from international agencies (Food and Agriculture Organization of the United Nations, Inter-American Development Bank, and International Atomic Energy Agency [²⁶,²⁹,³⁰].

Methodological strategy in the articles reviewed

In the field of evaluations and health care programs and services, like in a more general scientific field, there are still unresolved debates over qualitative and quantitative approaches [²³]. Based on positivist and post-positivist theories, quantitative approaches are advocated due to their objectivity in data collection, analysis, and interpretation. In terms of evaluations, this logic is based on the assumption that the evaluator should focus only on supposedly more objective facts produced by the program [²¹]. Guba & Lincoln [⁵⁰] address the fundamental epistemological issues of the positivist paradigm in scientific inquiry, and the belief in an objective reality external to the subject and indifferent towards their interests. According to these authors, the predominance of the positivist approach in evaluations has led to the exaggerated dependence of the formal quantitative measurements based on the greater precision of the data collected and the reliability of the evaluation instruments that can generate indisputable truths, as well as the lack of openness to other alternative methodologies for understanding the reality [³⁵].

The predominant emphasis on the methodological strategies disregarding the reality to be investigated has led authors such as Demo [⁵¹] to criticize what he called the ‘method dictatorship’ to refer to excessively quantitative and objectivist approaches. The qualitative perspective, on the other hand, is characterized by the recognition of subjectivity and symbolic dimension as integral parts of the social reality, aspects that are hardly covered using closed ended instruments [⁵²].

Based on the assumption that reality is constituted based on an extensive dimension - which can be captured through the standardization and frequency of occurrence of certain actions promoting linearity and certain predictability to the phenomena - and an intensive dimension that provides deep meanings and conceptions and is accessed through dialogue and direct human relationship, it is possible to assert that there is complementarity between qualitative and quantitative approaches and not dichotomy [²⁸].

The analysis of the articles demonstrated the predominance of epidemiological and quantitative approaches in the evaluation processes in the food and nutrition field. This trend seems to indicate an emphasis on biological aspects resulting from the influence of the traditional medical field on the field of nutrition, as discussed by Vasconcelos [⁵³]. Like the articles available in scientific literature in general [⁵⁴], the articles reviewed revealed prevalence of quantitative methods. On the other hand, the urgent need to transcend the traditional biomedical model, still used in the field of food and nutrition, has been expressed by researchers, especially since the after the early 2000s and the formation of the Thematic Group on Food and Nutrition in Collective of Associação Brasileira de Saúde Coletiva (Abrasco), in 2008 [⁵⁵].

Participation as a methodological strategy

The evaluation of programs and services aims, among other things, to compare what was planned with the reality, which certainly will offer some resistance to the initial plan and will be guided by the questions addressed to the intervention evaluated. It is believed that many programs can benefit from an evaluation [⁵⁶] if it is part of public policy programs. The method used to carry out an evaluation, however, directly influences its comprehensiveness and the types of results obtained.

Influenced by the constructivist paradigm, the philosophical and epistemological movements that emerged between the nineteenth and twenty-first centuries have gained increasing recognition and space in different methods, fields of research, and social actions. In the same period and based on Constructivist Epistemologies [⁵⁷], the evaluation of programs started, exploring democratic [³⁵,⁵⁸] and participatory approaches [²].

These approaches are different from the others because they consider that the results of an evaluation are directly influenced by the relationship between stakeholders and evaluators, making them a major aspect in the development and carrying out of the evaluation. Democratizing the evaluation process means considering the thoughts and opinions of those involved during the processes of planning and carrying out the evaluation [³⁷]. Therefore, more democratic and participative strategies can contribute to participants’ learning and transformation, bringing, at the same time, dynamism and benefits to the program [³⁵].

Among the articles reviewed, only one reported using a participatory strategy in the development of the evaluation process [³⁷]. Its objective was to evaluate the Programa Nacional de Combate às Deficiências da Vitamina A (National Program to Overcome Vitamin A Deficiencies) in a municipality in the Northeastern Region in Brazil. The authors referred to the participation of the parents and guardians of the children evaluated - the focus of the evaluation - as a participatory strategy. However, a more careful analysis indicated that their participation was limited to the provision of information for data collection, barely exploiting their participation potential.

Criteria adopted and analysis focus

The development of an evaluation includes, among other aspects, previously established criteria [⁵⁹], and based on the choice of these criteria, the evaluator should formulate a value judgment to be applied to the object being evaluated [⁶⁰]. Judgments are statements about the merit worth, or significance of the intervention, whereas criteria are requirements set out to enable judgment of changes in a given situation, which is a necessary condition to carry out the evaluation [⁶¹].

Although being an essential part of program evaluation, the criteria used for analysis were not clearly described in the articles reviewed. However, they were identified in the present study after a careful reading of the selected papers. Several authors did not define the criteria they used to carry out the evaluation in the methodological description; they only mentioned them during data analysis [²³]. Not describing the pre-established criteria adopted can be explained by the lack of use of methodological references in the field of program evaluation to develop the methodological model. This suggests that the authors did not use specific frameworks in the evaluation of health programs and services, prioritizing the traditional scientific method to draw conclusions about the programs and services evaluated. This can be explained by the characteristics of Nutrition as an area of knowledge and the consequent training and education of professionals, which are associated with market demands and are based on medical knowledge [⁶²,⁶³].

With regard to the analyses carried out by the articles reviewed, we will adopt the classification proposed by Contandriopoulos [⁶⁴], through which the author identifies different analysis focuses in the evaluation studies. These analyses can be characterized as: intervention analysis, which considers the relationships between the objective and the methods employed. Productivity analysis, which investigates how the resources are used; Impact analysis, which investigates the influence of services on the program issues; outcome analysis, which relates the resources used with the effects obtained; Implementation analysis, which investigates how much of the design and the planning were put into practice; and Strategic analysis, which investigates the relevance of the evaluation [³⁹]. These definitions of analysis are in agreement with the studies carried out by Vieira-da-Silva [65], who addresses the main characteristics of policies, programs, or practices to be evaluated, and by Draibe [⁴²], who discusses the uses of different indicators and methodological strategies to draw conclusions in evaluations.

Chart 2 shows that the impact and implementation analyses were predominant in the articles reviewed. The strong presence of impact analyses suggests that the evaluators seek to investigate the effectiveness of the actions carried out by the institutions evaluated. However, this particular model of analysis usually transforms its variables into dichotomous variables, and thus the evaluations have the characteristics of “black-box” evaluations. The “black-box” model, although simple to apply, affects the quality of the conclusions derived from its use. The implementation analysis, however, explores the multifaceted reality in which the programs are implemented, indicating the effect of the intervention based on the level of implementation of the program. This analytical method helps the evaluator investigate the different contexts encompassed by the programs [³⁹].

Chart 2
Studies evaluating food and nutrition programs and policies according to the program, affiliation, methodology, and objective of the evaluation (2001-2015).

Author	Analysis focus	Evaluative criteria	Judgments
Castro & Monteiro et al. [19]	Impact; Implementation	Improvement of nutrition status	- Did not hinder the success of the intervention
Augusto & de Souza [20]	Impact	Weight-for-age Z-score	- Effective in promoting weight gain - Participation in food supplementation program leads to the improvement of anthropometric indicators
Gabriel et al. [21]	Impact	Food consumption	- Improvement of some behaviors and healthy eating practices after the nutritional intervention
Lana et al. [22]	Impact	Breastfeeding	- The program had a positive impact
Amorim et al. [23]	Implementation	Experience in the field; integration with the school community; nutrition education; Improvement and monitoring of the school cafeteria; products sold in the cafeteria	- Actions are in the initial stage of implementation
Faleiros et al. [24]	Impact	Prevalence of exclusive breastfeeding	- Exerted positive impact on exclusive breastfeeding rates - Adequacy of the program to the promotion of exclusive breastfeeding
Vidal et al. [25]	Implementation	Program Evaluation Questionnaire	- The action [...] had the lowest level of implementation - The action [...] had the highest level of implementatio - Low level of implementation
Carvalho et al. [26]	Impact; Implementation	Weight-for-age Z-score	- Relative importance of the implementation of food supplementation programs was confirmed
Bezerra et al. [27]	Implementation	Program Evaluation Questionnaire	- The action [...] cannot yet be considered implemented since the group of professionals [...] do not carry out all the activities, evidencing low adherence to the recommended standards
Veloso & Santana [28]	Impact	Weight gain	- Undesirable effects - Contribution to increased number of comorbidities - lack of nutritional education as suggested in the guidelines - Actions need revision
Veloso et al. [29]	Strategy; Productivity; Impact	Risk of developing chronic diseases; weight gain and overweight	- Undesirable effects - Nutritional problems persist
Almeida et al. [30]	Implementation	Knowledge about the program by those responsible for it; knowledge about the Vitamin A by the responsible for the program; accessibility	- The population has access - Poor knowledge about the program
Martins et al. [31]	Logic; Implementation; Effectiveness	Infrastructure; implementation strategy; results obtained	- The recently created Programa Nacional de Suplementação de Vitamina A (PNVITA, National Vitamin A Supplementation Program) has not yet been successfully and effectively implemented
Vargas et al. [32]	Impact	Food practices; anthropometry	- Favorable changes [...] stimulated the continuous effort towards the implementation of similar programs
Spinelli & Canesqui [33]	Implementation	Organizational structure; human resources; infrastructure; routine; program beneficiaries’ opinions	- Unquestionable positive effects of decentralization
Dubeux et al. [34]	Logic; Implementation	Instituto Materno Infantil de Pernambuco (Mother and infant Institute of Pernanbuco) Questionnaire	- Need for greater participation in the activities for the implementation of actions - Great need for discussions and professionals’ awareness
Santos et al. [35]	Impact	Socioeconomic questionnaire; child health questionnaire; anthropometric data; double-marked water	- Lack of impact [...] due to implementation problems
Sá & Szarfare [36]	Impact; Implementation	Hemoglobin concentration; socioeconomic questionnaire and medical anamnesis	- Proven effectiveness
Bandoni et al. [37]	Effect; Implementation	Food handling; hygiene; equipment and utensils; preparation area; nutritive value of meals	- Some results are worrisome
Almeida et al. [38]	Impact; Implementation	Breastfeeding	Clear low impact of the “Cegonha feliz” program
Paiva et al. [39]	Logic; Strategy; Implementation	Structure; processes; coverage	- Need for greater attention to the actions related to professional training proposed by the managers - Need for greater efforts to meet coverage goals
Cardoso et al. [40]	Impact; Implementation	Breastfeeding; health outcomes	- Proven important - It contributed to improve breastfeeding outcomes and the follow-up of nursing mothers - It had significant impact

The predominant focus on the quantitative methods identified in the articles analyzed hinders full understanding of the problems that may occur in programs and services, especially when dealing with groups and populations with subjective reality [⁹,¹²]. Historically, in the field of evaluation, this characteristic has led to processes that disregard the context into which the interventions are inserted, lacking to address aspects related to the role of the subjects involved in the evaluations [³⁵].

Evaluation approaches’ references

There are several possibilities of methodological approaches to program evaluation. There is no methodological framework that can be used with any type of evaluation; there are only guidelines available [¹³,²¹,⁶⁶], which help guiding evaluators’ choice of methods that are most suitable for each evaluation. Therefore, it can be inferred that each evaluator “construct” his/her own evaluation [⁴³].

In the present study, ten articles made references to authors in the field of program evaluation; however, only one of them used more than two references of prominent authors in the field of evaluation program and services to carry out the evaluation [³²]. The most frequently cited study was “Avaliação em saúde: dos modelos conceituais à pratica na análise da implantação de programas”, an article published in 1997 by Zulmira Maria de Araújo Hartz.

The lack of program evaluation references may indicate that the evaluators may not be aware of their existence. Once they are immersed in epidemiological interpretations of quantitative studies, it is possible that the authors extrapolate this methodological approach to the evaluation of programs, without recognizing, however, the great number of interpretations and conclusions available in the literature regarding evaluation that can contribute to their study.

In addition, in the field of food and nutrition, studies using Social and Human Sciences approaches are fairly recent. More specifically, studies making interdisciplinary references have been conducted due to the link between food and nutritional issues and Collective Health [⁶⁷,⁶⁸].

The meaning of the words evaluation and epidemiology should also be taken into consideration in studies on food and nutrition programs and policies. The word epidemiology has a quite singular meaning. It refers to a set of methods belonging to the branch of science responsible for understanding the health-disease process that affects population groups in terms of its frequency and spread.

On the other hand, the word evaluation has a more extensive and widespread use. In the Social Sciences, it has a meaning similar to that previously presented in the beginning of this study. In the health sciences, it is present on a daily basis but in several different ways. Any service provided by a health care professional involves some type of evaluation in order to guide the professional to find a starting point for reaching conclusions.

Judgments in the articles reviewed

As previously stated, one of the main roles of a program evaluation is to make judgments about the merit and worth of an intervention [²¹,²⁹,⁴⁴]. Together with other procedures, it is an essential characteristic of evaluations [⁴²].

In the present study, all the articles selected for analysis had a value judgment, which was, however, formulated in different ways. As shown in Chart 2, there were clear and detailed descriptions of the programs, including the use of qualitative adjectives such as “highest”, “lowest”, “positive”, and “effective”, to characterize the interventions evaluated. However, most of the time, the judgments were expressed indirectly, implicit in the text, and they did not meet the standards and criteria considered as ideal by the authors and in their final recommendations.

Since there are different methods and ways to plan and carry out an evaluation, there is a variety of ways to show and report the results [³⁷]. Less democratic evaluation models include the report of the results only to management team members, i.e., those capable of transforming the conclusions derived from the results into actions. However, more recent methodologies have encouraged reporting or transferring the evaluation results to all people involved in order to use the evaluation process as a catalyst for changes and learning [⁶⁹].

None of the articles reviewed stated the report of the evaluation directly to the population or agents involved in the program evaluated. Transferring or dissemination of information and results only through scientific journals is one of the lowest interactive potential means.

FINAL CONSIDERATIONS

It can be said that the evaluations of programs in the field of nutrition and food in Brazil are predominantly characterized by the use of quantitative research approach strategies and are carried out by external evaluators, i.e., evaluators that were not involved in the initiative being evaluated. Moreover, it was observed a limited use of frameworks developed within the specific field of evaluation of health care programs and services in Brazil and abroad, demonstrating little interaction between the evaluations that are actually carried out in the field of nutrition and the instruments available for this type of approach. One of the consequences of these facts is the risk of uncritical transposition of research techniques and the efforts towards objectivity in the evaluation of interventions. This situation refers to the attempts to objectify social programs in the 1960s in the United States, when the emerging field of evaluation sought to establish itself through the use of the methods and legitimacy of the natural sciences for the assessment of social policies that were increasing at that time.

Based on the impact analysis, the emphasis on the effect of the initiatives evaluated reveals the predominant interest in several areas in the so-called net effects of the program or service under consideration. These long-term effects on society as a whole would be the most relevant effect in terms of transparency interventions and accountability to the general public. However, the program and service evaluation scope encompasses a set of methods and perspectives that allow addressing different aspects of the object being evaluated, such as its presupposed theories, issues related to its relevance to the problem under study, and issues related to the level of implementation and costs. On the other hand, there are perspectives that go beyond the focus of the evaluation; they consider the various agents involved and initiatives based on the greater involvement of parties that are potentially interested and/or affected by evaluation processes.

In view of these theoretical, methodological, and ethical aspects, it is very important that the evaluation of programs and services in the field of nutrition and food in Brazil use more comprehensive approaches, i.e., they should consider the complexity of the interventions evaluated by making use of the theoretical-methodological apparatus available in the literature. Furthermore, it is necessary to seek diversified approaches to evaluation processes in order to understand the importance of the different perspectives of the agents involved in the intervention for the evaluation processes.

REFERENCES

• ¹
  Furtado JP. Avaliação participativa. In: Otero MR, organizador. Contexto e prática da avaliação de iniciativas sociais no Brasil: temas atuais. São Paulo: Peirópolis; 2012. p.21-41.
• ²
  Furtado JP, Vieira-da-Silva LM. A avaliação de programas e serviços de saúde no Brasil enquanto espaço de saberes e práticas. Cad Saúde Pública. 2014;30(12):2643-55.
• ³
  Felisberto E, Freese E, Bezerra LC, Alves CK, Samico I. Análise da sustentabilidade de uma política de avaliação: o caso da atenção básica no Brasil. Cad Saúde Pública. 2010;26(6):1079-95.
• ⁴
  Santos ID. Avaliação do impacto de programas nutricionais. Rev Nutr. 2009;22(1):141-50. https://doi.org/10.1590/S1415-52732009000100013
  » https://doi.org/10.1590/S1415-52732009000100013
• ⁵
  Almeida PF, Giovanella L. Avaliação em Atenção Básica à Saúde no Brasil: mapeamento e análise das pesquisas realizadas e/ou financiadas pelo Ministério da Saúde entre os anos de 2000 e 2006. Cad Saúde Pública. 2008;24(8):1727-42.
• ⁶
  Souza Ferraz DA, Nemes MI. Avaliação da implan-tação de atividades de prevenção das DST/AIDS na atenção básica: um estudo de caso na Região Metropolitana de São Paulo, Brasil. Cad Saúde Pública. 2009;25(Supl.2):S240-50.
• ⁷
  Dantas CR, O AMGR. Cartografia das pesquisas avaliativas de serviços de saúde mental no Brasil (2004-2013). Physis. 2014;24(4):1127-79.
• ⁸
  Claudia Flemming C, Calvo MC. Avaliação da Atenção em Saúde Bucal no Brasil: uma revisão da literatura. Saúde Transform Soc. 2012;3(1):92-100.
• ⁹
  Vasconcelos FAG, Batista Filho M. História do cam-po da alimentação e nutrição em saúde coletiva. Ciênc Saúde Coletiva. 2011;6(1):81-90.
• ¹⁰
  Ministério da Saúde (Brasil). Política Nacional de Alimentação e Nutrição. Brasília: Ministério da Saúde; 2011 [acesso 2013 fev 25]. Disponível em: http://189.28.128.100/nutricao/docs/geral/pnan 2011.pdf
• ¹¹
  Ministério do Desenvolvimento Social e Combate à Fome (Brasil). Marco de referência de educação alimentar e nutricional para as políticas públicas. Brasília: Ministério do Desenvolvimento Social e Combate à Fome; 2012 [acesso 2013 abr 10]. Disponível em: https://www.nestle.com.br/nestle nutrisaude/Conteudo/diretriz/Marco_Referencia_de_Educacao_Nutricional_Alimentar.pdf
• ¹²
  Magalhães R. Avaliação de políticas e iniciativas públicas de segurança alimentar e nutricional: dile-mas e perspectivas metodológicas. Ciênc Saúde Coletiva. 2014;19(5):1339-46.
• ¹³
  Henrique FCS, Lira PI, Santos SM, Andrade SL. Tendência do campo de avaliação de intervenções públicas de alimentação e nutrição em progra-mas de pós-graduação no Brasil: 1980-2004. 2007;23(12):2972-81.
• ¹⁴
  Vasconcelos FD. The construction of scientific knowledge in Food and Nutrition: Analysis of dissertations and theses in the Brazilian post-graduation programs in Nutrition. Rev Nutr. 2015;28(1):5-16. https://doi.org/10.1590/1415-52 732015000100001
  » https://doi.org/10.1590/1415-52
• ¹⁵
  Costa AB, Zoltowski APC, Koller SV, Teixeira MAP. Construção de uma escala para avaliar a qualidade metodológica de revisões sistemáticas. Ciênc Saúde Coletiva. 2015;20(8):2441-52.
• ¹⁶
  Sampaio RF, Mancini MC. Estudos de revisão sistemática: um guia para síntese criteriosa da evidência científica. Rev Bras Fisioter. 2007;11(1):83-9.
• ¹⁷
  Atallah AN, Castro AA. Revisão sistemática e metanálise. In: Atallah AN, Castro AA. Evidências para melhores decisões clínicas. São Paulo: Lemos-Editorial; 1998;
• ¹⁸
  Gonçalo CS, Castro CM, Bonon MZ, Motta PMR, Dahdal AB, Batista JC, et al. Planejamento e execução de revisões sistemáticas da literatura. Brasília Med. 2012;49(2):104-10.
• ¹⁹
  Castro IRR, Monteiro CA. Avaliação do Impacto do programa 'Leite é Saúde' na recuperação de crianças desnutridas no Município do Rio de Janeiro. Rev Bras Epidemiol. 2002;5(1):52-62.
• ²⁰
  Augusto RA, de Souza JM. Efetividade de pro-grama de suplementação alimentar no ganho ponderal de crianças. Rev Saúde Pública. 2010;44(5):793-801.
• ²¹
  Gabriel CG, Santos MV, Vasconcelos FD. Ava-liação de um programa para promoção de há-bitos alimentares saudáveis em escolares de Flo-rianópolis, Santa Catarina, Brasil. Rev Bras Saúde Matern Infant. 2008;8(3):299-308.
• ²²
  Lana AP, Lamounier JA, César CC. Impacto de um programa para promoção da amamentação em um centro de saúde. J Pediatr. 2004;80(3):235-40.
• ²³
  Amorim NF, Schmit BD, Rodrigues MD, Recine EG, Gabriel CG. Implantação da cantina escolar saudável em escolas do Distrito Federal, Brasil. Rev Nutr. 2012;25(2):203-17. https://doi.org/10.1590/S1415-52732012000200003
  » https://doi.org/10.1590/S1415-52732012000200003
• ²⁴
  Faleiros JJ, Kalil G, Casarin DP, Laque Jr PA, Santos IS. Avaliação do impacto de um programa de puericultura na promoção da amamentação exclusiva Impact of a well baby care program on the promotion of exclusive breastfeeding. Cad Saúde Pública. 2005;21(2):482-9.
• ²⁵
  Vidal SA, Frias PG, Marques NM. Avaliaçäo normativa das açöes do Programa de Atençäo Integral à Saúde da Criança (PAISC) em Pernam-buco. Rev Bras Saúde Mater Infant. 2001;1(2):129-35.
• ²⁶
  Carvalho LG, Saldiva SR, Rosa TE, Lei DL. Evolução do estado nutricional de crianças submetidas a um programa de suplementação alimentar em município do Estado de São Paulo. Rev Nutr. 2009;22(2):207-17. https://doi.org/10.1590/S14 15-52732009000200003
  » https://doi.org/10.1590/S14
• ²⁷
  Bezerra LC, Frias PG, Vidal SA, Macedo VC, Vanderlei LC. Aleitamento materno: avaliação da implantação do programa em unidades básicas de saúde do Recife, Pernambuco (2002). Ciênc Saúde Coletiva. 2007;12(5):1309-7.
• ²⁸
  Veloso IS, Santana VS. Impacto nutricional do programa de alimentação do trabalhador no Bra-sil. Rev Panam Salud Publica. 2002;11(1):24-31.
• ²⁹
  Veloso IS, Santana VS, Oliveira NF. Programas de alimentação para o trabalhador e seu impacto sobre ganho de peso e sobrepeso. Rev Saúde Pública. 2007;41(5):769-76.
• ³⁰
  Almeida ER, Carvalho AT, Nilson EA, Coutinho JG, Ubarana JA. Avaliação participativa do Programa Nacional de Suplementação de Vitamina A em um município da Região Nordeste do Brasil. Cad Saúde Pública. 2010;26(5):949-60.
• ³¹
  Martins MC, Santos LM, Santos SM, Araújo MD, Lima AM, Santana LA. Evaluation of public policies to promote food security and the fight against hunger, 1995-2002. 3-the Brazilian National Program to Control Vitamin A Deficiency. Cad Saúde Pública. 2007;23(9):2081-93.
• ³²
  Vargas ICS, Sichieri R, Sandre-Pereira G, Veiga GV. Avaliação de programa de prevenção de obesidade em adolescentes de escolas públicas. Rev Saúde Publica. 2011;45(1):59-68.
• ³³
  Spinelli MA, Canesqui AM. Descentralização do programa de alimentação escolar em Cuiabá: 1993-1996. Rev Nutr. 2004;17(2):151-65.
• ³⁴
  Dubeux LS, Frias PG, Vidal SA, Santos DM. Incen-tivo ao aleitamento materno: uma avaliação das equipes de saúde da família do município de Olin-da, Pernambuco. Rev Bras Saúde Matern Infant. 2004;4(4):399-404.
• ³⁵
  Santos IS, Gigante DP, Coitinho DC, Haisma H, Valle NC, Valente G. Evaluation of the impact of a nutritional program for undernourished children in Brazil. Cad Saúde Pública. 2005;21(3):776-85.
• ³⁶
  Sá AC, Szarfarc SC. Prevalência de anemia em crianças, antes e durante a participação em pro-grama de fortificação alimentar com ferro. Nutrire. 2009;34(2):115-26.
• ³⁷
  Bandoni DH, Marchioni DM, Brasil BG, Figueiredo IC, Sarti FM. O programa de incentivo à instalação de cozinhas comunitárias: avaliação de uma polí-tica pública brasileira. Nutrire. 2010;35(1):15-27.
• ³⁸
  Almeida CC, Scochi MJ, Souza RK, Carvalho WO. Prevalência de aleitamento materno antes e após a implantação de um programa de redução de morbimortalidade infantil, no município de Campo Mourão (PR). Ciênc Saúde Coletiva. 2010;15(2):575-83.
• ³⁹
  Paiva AD, Cagliari MP, Queiroz DD, Souto RA, Brito VR, França IS. Programa Nacional de Suplementação de Vitamina A no Estado da Paraíba: uma análise a partir do relato de profissionais da Equipe de Saúde da Família. Epidemiol Serv Saúde. 2011;20(3):373-83.
• ⁴⁰
  Cardoso LO, Vicente AS, Damião JJ, Rito RV. Impacto da implementação da Iniciativa Unidade Básica Amiga da Amamentação nas prevalências de aleitamento materno e nos motivos de consulta em uma unidade básica de saúde. J Pediatr. 2008;84(2):147-53.
• ⁴¹
  Rossi PH, Lipsey MW, Freeman HE. Evaluation: A systematic approach. Thousand Oaks (CA): Sage Publications; 2003.
• ⁴²
  Draibe SM. Avaliação de implementação: esboço de uma metodologia de trabalho em políticas públicas. Tendências e perspectivas na avaliação de políticas e programas sociais. São Paulo: PUC-SP; 2001.
• ⁴³
  Scriven M. Evaluation thesaurus. Beverly Hills (CA): Sage; 1991.
• ⁴⁴
  Love AJ. Internal evaluation: Building organizations from within. Beverly Hills (CA): Sage; 1991.
• ⁴⁵
  House ER. The role of the evaluator in a political world. Canadian J Program Evaluation. 2004;19(2):1-16.
• ⁴⁶
  Marino E, Locatelli C. Avaliação externa e avaliação interna: distinções, aplicações, competências dos avaliados, passos e cuidados para a execução. In: Otero MR, organizador. Contexto e prática da avaliação de iniciativas sociais no Brasil: temas atuais. São Paulo: Peirópolis; 2012. p.107-21.
• ⁴⁷
  Fitzpatrick JL, Sanders JR, Worthen BR. Program evaluation: Alternative approaches and practical guidelines. New York: Longman Publishers; 1997.
• ⁴⁸
  Furtado JP. A avaliação participativa. In: Otero MR, organizador. Contexto e prática da avaliação de iniciativas sociais no Brasil: temas atuais. São Paulo: Peirópolis; 2012. p.21-41.
• ⁴⁹
  Gasparini MFV, Furtado JP. Avaliação de Pro-gramas e Serviços Sociais no Brasil: uma análise das práticas no contexto atual. Serv Soc Soc. 2014;(117):122-41.
• ⁵⁰
  Guba EG, Lincoln YS. Avaliação de quarta geração. Campinas: Unicamp; 2011.
• ⁵¹
  Demo P. Pobreza política: a pobreza mais intensa da pobreza brasileira. Campinas: Autores As-sociados; 2006.
• ⁵²
  Minayo MCS. O desafio do conhecimento: pes-quisa qualitativa em saúde. São Paulo: Hucitec; 2006.
• ⁵³
  Vasconcelos FD. Os arquivos brasileiros de nutri-ção: uma revisão sobre produção científica em nutrição no Brasil (1944 a 1968). Cad Saúde Pública. 1999;15(2):303-16.
• ⁵⁴
  Bosi MML, Mercado, FJ, organizadores. Avaliação qualitativa de programas de saúde: enfoques emergentes. Rio de Janeiro: Vozes; 2006.
• ⁵⁵
  Medeiros MAT, Prado S, Bosi MLM. Contributions for the capacity development in Food and Nutrition in Public Health. Rev Nutr. 2014;27(6):645-52.
• ⁵⁶
  O'Sullivan RG. Practicing evaluation: A collaborative approach. Thousand Oaks (CA): Sage Publications; 2004.
• ⁵⁷
  Vedung E. Four waves of evaluation diffusion. Evaluation. 2010;16(3):263-77.
• ⁵⁸
  Stufflebeam D. Evaluation models: New directions for evaluation. 2001;2001(89):7-98.
• ⁵⁹
  Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z. Avaliação: conceitos e métodos. In: Avaliação: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.
• ⁶⁰
  Scriven M. New frontiers of evaluation: Evaluation practice. 1986;7(1):7-44.
• ⁶¹
  Alves CK, Natal S, Felisberto E, Samico I. Inter-pretação e análise das informações: o uso de matrizes, critérios, indicadores e padrões. In: Samico I, Felisberto E, Figueiró AC, Frias PG, orga-nizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: Medbook; 2010.
• ⁶²
  Furtado JP, Minayo MC, Akerman M, Drumond Júnior M, Carvalho YM. Avaliação de programas e serviços. In: Saúde em debate. Rio de Janeiro: Hucitec; 2006. p.715-39. v.170.
• ⁶³
  Costa N. Revisitando os estudos e eventos sobre a formação do nutricionista no Brasil. Rev Nutr. 1999;12(1):5-19.
• ⁶⁴
  Contandriopoulos AP, Champagne F, Denis JL, Pineault R. A avaliação na área da saúde: conceitos e métodos. Avaliação em saúde: dos modelos conceituais à prática na análise da implantação de programas. Rio de Janeiro: Fiocruz; 1997.
• ⁶⁵
  Viera-da-Silva, LM. Avaliação de políticas e pro-gramas de saúde. Rio de Janeiro: Fiocruz; 2014.
• ⁶⁶
  Medeiros MAT, Amparo-Santos L, Domene SMA. Contributions for the capacity development in Food and Nutrition in Public Health. Rev Nutr. 2013;26(5):583-93. https://doi.org/10.1590/1415-52 73201400060013e
  » https://doi.org/10.1590/1415-52
• ⁶⁷
  Chen HT. Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks (CA): Sage Publications; 2005.
• ⁶⁸
  Bosi MLM, Prado SD. Alimentação e nutrição em saúde coletiva: constituição, contornos e estatuto científico. Ciênc Saúde Coletiva. 2011;16(1):7-17.
• ⁶⁹
  Amparo-Santos L, Soares MD. Challenges of academic and scientific output in the interface between Social and Human Sciences and Food and Nutrition Sciences. Rev Nutr. 2015;28(1):89-98. https://doi.org/10.1590/1415-52732015000100008
  » https://doi.org/10.1590/1415-52732015000100008

Publication Dates

• Publication in this collection
  May-Jun 2017

History

• Received
  02 May 2016
• Reviewed
  01 Dec 2016
• Accepted
  03 Jan 2017