Halo Effect in Peer Review: Exploring the Possibility of Bias Associated with the Feeling of Belonging to a Group

Efeito Halo na Revisão por Pares: Explorando a Possibilidade de Viés Associado ao Sentimento de Pertencimento de Grupo

Abstract

In this study, we explore the possibility of bias in the article evaluation process during peer review, due to homogeneous groups of reviewers, most of whom belong to the same professional field. The conditions of the context of journals considered for the manifestation of bias are: a) articles written in a regional language other than English, which restricts the group of reviewers; b) specific theme, which requires reviewers who specialize in the theme and predominantly belong to the same professional group; and c) publication of articles associated with the interests of the professional group to which the reviewers belong. Our proposition is that under these conditions, articles containing messages in keeping with the concerns and demands of the group to which the reviewers belong, could be viewed more favorably reviewers, thereby benefiting from the halo effect. We use the case study as a research strategy, identifying of a journal that meets the described conditions of context. Content analysis techniques were applied to the texts of articles pertaining to the research context. The analyses indicate that the interests of professional groups can prevail over scientific interests, demonstrating the need to manage and control the necessary heterogeneity of the group of reviewers.

Keywords:
halo effect; reviewer; peer review; public servant; scientific article

Resumo

Exploramos nesta pesquisa a possibilidade de viés no processo de avaliação de artigos, durante a revisão por pares, decorrente de grupo homogêneo de avaliadores pertencentes em sua maioria a uma mesma categoria profissional. As condições de contexto da revista consideradas para manifestação do viés são: a) artigos redigidos em idioma regional, distinto do inglês, que restrinja o grupo de avaliadores; b) temática específica, que demande avaliadores especializados no tema e pertencentes predominantemente a mesma categoria profissional; e c) publicação de artigos associados aos interesses da categoria profissional de pertencimento dos avaliadores. Nossa proposição é que nessas condições, artigos contendo mensagens alinhadas aos anseios e reivindicações do grupo de pertencimento dos avaliadores, podem obter maior empatia dos avaliadores, beneficiando-se assim do efeito halo. Utilizamos o estudo de caso como estratégia de pesquisa a partir da identificação de uma revista que atendesse as condições de contexto descritas. Aplicaram-se as técnicas de análise de conteúdo, aos textos de artigos pertinentes ao contexto da pesquisa. As análises indicam que interesses de grupos profissionais podem prevalecer sobre os interesses científicos, evidenciando a demanda por gestão e controle da necessária heterogeneidade do corpo de avaliadores.

Palavras-chaves:
halo effect; parecerista; revisão por pares; servidor público; artigo científico

1 Introduction

The rejection rate of articles submitted to top tier journals is over 90% (DAY, 2011). This rejection rate has risen in recent decades (STRAUB, 2008), creating major stress and concerns among authors involved in the review of scientific articles. The most widespread review process comprises two phases: the editor review and peer reviews. Biases may occur in any decision-making process (TVERSKY; KAHNEMAN, 1974; WHITE; POLDRACK, 2014). Therefore, there is also the risk of wrong decisions in double-blind peer reviews (LEE; SUGIMOTO; ZHANG; CRONIN, 2013). The literature indicates the occurrence of bias in article reviews especially in the editor review due to the perception of the quality of the article (halo effect) due to the characteristics of the authors, especially the author’s name and affiliation (PETERS; CECI, 1982). This study discusses the possibility of occurrence of the halo effect in the subsequent phase of the process, the peer review phase. This study is based on the assumption that articles aligned with the concerns and demands of the group to which the reviewers belong are viewed more favorably by the reviewers, encouraging the halo effect.

The possible analysis bias investigated in this study is highly specific in terms of the necessary characteristics of the context of the environment for it to occur, making it very rare and, thus, difficult to observe. A combination of the following characteristics of the journal is required: a) texts of articles written in a regional language other than English, restricting the group of reviewers to those that master the language; b) a highly specific theme, requiring reviewers who specialize in the theme and predominantly belong to the same professional field; and c) the journal publishing articles that address topics associated with the interests of the professional field to which most of the reviewers belong. The overlapping of these characteristics could mean that reviewers assess articles addressing themes closely linked to the interests of their own professional field. This situation could result in bias in favor, in other words, the halo effect in the review of these articles. Identifying a journal with all the characteristics of context described, we structured the present study with the following objective: to analyze articles with content closely linked to the interests of the professional group of the reviewers, verifying whether this content was developed within the precepts of scientific research; in other words, whether they respect the canons of scientific research.

The importance attributed to journals and their articles for the evolution of science and society is demonstrative of the relevance of the double-blind review (BORNMANN; 2012; THYER, 2008) and serves as the justification for the present study. The interference of the halo effect in peer reviewing, even sporadically, interfering in the publication of few articles, can be very harmful to the image, reputation and credibility of the journal. The work of many decades and hundreds of articles may be called into question if the community perceives that articles are published that do not respect the canons of scientific research to serve the interests of a certain professional group.

2 Halo Effect in article reviews

Studies on double-blind peer review have grown for three reasons: a) the perception of their importance to the production, evaluation, and consumption of science; b) the results of the studies remain inconclusive; and c) because they are a source of multiple criticisms that have encouraged debate among researchers (SABAJ MERUANE; GONZÁLEZ VERGARA; PINA-STRANGER, 2016). One of the aspects that have come under debate is the elimination of bias and the drive for impartiality in the review process. “Impartiality is a recognized source of procedural justice in meritocratic systems because it demonstrates that decision-making processes are objective and unbiased” (CLAIR, 2015, p. 162). As the peer review is a decision-making process, it falls to journal editors and managers to ensure impartiality in the process, combatting the various forms of bias that emerge in double-blind peer review.

Among the most discussed and accepted biases on the part of academia in the double-blind peer review are those associated with the halo effect and the horn effect during the editor review (PETERS; CECI, 1982). The halo effect is a cognitive decision-making bias, defined as an aspect of the individual or entity under analysis, perceived by the reviewer, as something positive that can have a positive effect on the judgment of all the other important aspects to be analyzed, thus distorting the result of the decision. The horn effect is similar, but involves an aspect perceived by the reviewer as something negative, having a negative impact on the result. Thus, the halo effect can aid an article with little scientific merit due to the authorship being associated with a prestigious author or institution, while the horn effect can harm the chances of an article that is scientifically valuable because of the authorship being associated with an unknown author or institution (SMITH, 2006). As the metadata of the authors is required upon submission, the influence of the author’s name or affiliation affects only the editors or the editor review phase.

Several other forms of bias in the double-blind peer review of articles have been discussed in academic circles in recent decades. Lee, Sugimoto, Zhang and Cronin (2013) developed a taxonomy of these biases:

Category “Content-Based Bias”, broken down into: “Confirmation bias”, reviewer bias against manuscripts that describe results that are not consistent with his theoretical perspective (JELICIC; MERCKELBACH, 2002); “Conservatism bias”, bias resulting from the conservatism of the reviewer who tends to be against groundbreaking and innovative research (BRABEN; 2004); “Publication bias”, bias associated with the tendency for journals to publish research demonstrating a positive rather than negative outcome (BARDY; 1998); and “Bias against interdisciplinary research”, bias associated with the understanding that disciplinary reviewers prefer mainstream research (TRAVIS; COLLINS, 1991);

Category “Bias as Deviation from ‘True Quality’ Value” - showing that reviewers can fail to interpret and apply evaluative criteria in consistent ways (BLACKBURN; HAKAL, 2006). Broken down into: “Bias as low inter-rater reliability”, agreement between reviewers is very low, the inter-rater reliability for two reviewers on a single submission should be about 0.8 − 0.9 (MARSH; JAYASINGHE; BOND., 2008); and “Bias as deviation from proxy measures for true quality”, encompassing studies that seek to assess the construct validity of peer review as a test/process by comparing its outcomes to proxy measures for manuscript quality, e.g., studies have investigated the citation patterns of “rejected-then-published-elsewhere” articles (BORNMANN; DANIEL, 2008);

Category “Bias as a Function of Author Characteristics”, broken down into prestige, affiliation, nationality, language and gender;

Category: “Bias as a Function of Reviewer Characteristics”, the authors do not divide the content into subcategories, but cite possible biases regarding gender, nationality and disciplinary affiliation, e.g., philosophers or psychologists (LEE; SCHUNN, 2011).

In this study, a new subcategory of bias associated with “Bias as a Function of Reviewer Characteristics” is presented and discussed: the professional group to which reviewers belong. The bias associated with the characteristics of a group, in the case of this study the group of reviewers, is based on two theories: Social Identity Theory (SIT) and Self-categorization Theory (SCT). Social identification is the central element of SIT and refers to the degree to which an individual perceives himself as a member of a social group or category (TAJFEl, 1981). According to SIT, too strong a social identification can have a considerable impact on actions and beliefs, which may provoke favoritism towards those within the group and derogation towards those outside the group (TAJFEL, 1982). Similarly, SCT states that positive feelings and attitudes toward a group are likely to manifest when the social categorization process is activated (TURNER, 1999). This study discusses the possibility that the categorization process of a group of reviewers of a journal, composed predominantly by public servants, is activated when analyzing the content of articles directly associated with topics of interest of this group.

To determine whether an article might have been unduly favored during the peer review process, we needed to verify whether the content most attractive to the group of reviewers, in other words, the demands associated with the group to which they belong, are duly advocated in accordance with scientific research canons. “Scientific canons include significance, theory-observation compatibility, generalizability, consistency, reproducibility, precision, and verification.” (CORBIN; STRAUSS, 1990, p. 4). In this study, we will use two canons as analysis units for the articles in the sample, generalizability and consistency. Definitions of these two canons are given below:

generalizability is “partly achieved through a process of abstraction that takes place over the entire course of the research. The more abstract the concepts, especially the core category, the wider the theory's applicability” (CORBIN; STRAUSS, 1990, p. 15);

“consistency in data collection, the investigator should watch for indications of all important concepts in every observation -- ones carried over from previous analyses as well as ones that emerge in the situation. All of the observations would be qualified by noting the conditions under which the phenomena occur, the action/interactional form they take, the consequences that result, and so forth. Careful noting of qualifiers gives specificity to concepts” (CORBIN; STRAUSS, 1990, p. 9).

3 Research method

As it is difficult to find journals combining all the characteristics of the context of the environment necessary for the occurrence of this possible event, as described in the introduction, we will work with the single case study as a research strategy, a journal, and with the identification of an intentional sample of extreme cases (YIN 2003). Extreme cases are understood as articles in a context with great potential for conflict of interest due to the direct interests of the author and reviewers with regard to the object of study. In this subsection, we will first describe the criteria that characterize the selected journal within the context of the environment and the criteria used to identify articles with great potential for conflict of interest. We will then describe the dimensions (registration units) used in the content analysis technique to verify possible violations of canons associated with good scientific research practices in messages supporting the demands of the group to which both the authors and reviewers belong.

3.1 Case selection: the thematic field and journals chosen for analysis

Selection of the thematic field of the articles. To analyze the halo effect in question, we had to identify a thematic field whose journal have a high probability of homogeneity, with the authors and reviewers belonging to a common professional group. The purpose was to analyze articles with a greater potential for conflicts of interest and higher risks of biased research due to the halo effect associated with the interests of the professional group of both the authors and reviewers. Therefore, we opted to analyze journal in the field of Public Administration, as their authors and reviewers are concentrated in public research and teaching institutions. In other words, most of the researchers and authors in this field are public servants. Another advantage of this field is the possibility of easily identifying themes that are prone to conflicts of interest. Every article on the subtheme of people management in public organizations has this potential. This subtheme involves discussing subjects such as wages and salaries, benefits, pensions, quality of life at work, training, evaluation of performance, laws that outline the group’s rights and duties and other themes of interest to public servants.

Selection of journals. Due to the need for a journal with language restrictions, we selected a Brazilian journal in the field of public administration, which has been publishing articles since 1967. Brazil has 36 institutions that offer Master’s or Doctorate Degree courses in Public Administration. Of these, 32 are in the public sector (Registration Data, 2016). The predominance of texts in Portuguese (92.5%) and Brazilian reviewers (93.5%) is shown in Appendix A, with a chart showing the numbers regarding the language of the published articles and the origins of the authors and reviewers. To demonstrate the heavy concentration of authors and reviewers in the regional journal, using a language other than English, we also show in Appendix A the tabulation of these data, for the same period, from one of the most prominent international journals in the field, the Public Administration Review. From here on, we will identify the articles of Brazilian journal as BR-J (acronym for Brazilian journal) and the articles of Public Administration Review as PAR-J (acronym for Public Administration Review journal).

Appendix A shows that the vast majority of articles (92.5%) published in the BR-J are in Portuguese, the official language of Brazil, and most reviewers and authors are associated with Brazilian institutions, 98.9% and 93.5%, respectively. Of the 443 reviewers that analyzed articles in the period in question, only 5 did not belong to Brazilian institutions. In the case of the PAR-J, there is a better distribution in terms of the country of origin of the first authors of the articles and the reviewers. The USA is the country with the highest concentration for the PAR-J: 64.7% of authors and 74.2% of reviewers. Using principles of probability, we can state that of the 201 articles published by the Brazilian Journal from 2013 to 2015, there is a 92.47% (0.935*0.989) chance of an article having been written by a first author with links to a Brazilian institution and of this article being evaluated by a reviewer who is also associated with a Brazilian institution. The chance of an overlap between a group of reviewers and authors from the same country in the case of the PAR-J is 48.01% (0.647*0.742), considering the data of authors and reviewers from the USA. Applying the two-binomial proportion test, these differences are statistically significant, at the level of significance of 0.05, for the difference in concentration of country associated with the institution of the first author, 64.7% American for the PAR-J and 93.5% Brazilian for the BR-J, and also for the concentration of country associated with the institution of the reviewer, 74.2% American for the PAR-J and 98.9% Brazilian for the BR-J.

The concentration of the BR-J’s reviewers linked to the same group is characterized in Table 1, which shows the origin of the reviewers identified in the annual lists of reviewers (nominees) of the articles published in the specialized journal between 2010 and 2015. As shown, 64% of the reviewers are public servants. Of the other reviewers, we should highlight those connected to foundations that specialize in the management of public companies. These foundations provide consultancy services to public agencies, often without requiring a bidding process due to their competence in the sector. Therefore, this group is also naturally motivated to be more sensitive to messages associated with the problems and needs of public servants. Considering both groups of reviewers subject to the bias of perception provoked by the halo effect, public servants and employees of foundations that specialize in public administration account for 81.5% of the total number of the reviewers of journal from 2010 to 2015, and prone to the halo effect.

Table 1
Profile of the reviewers of the BR-J

Reviewer Profile	2010	2011	2012	2013	2014	2015	Total	%
Public servant	58	67	81	78	103	117	504	64.0	81.5
Private sector (foundations)	21	20	27	24	20	26	138	17.5
Private sector (others)	19	29	23	25	23	27	146	18.5	18.5
Total	98	116	131	127	146	170	788	100	100

• Fonte: dados da pesquisa.

3.2 Identification of articles with a high potential for conflicts of interest

In February 2016, a search was conducted on the website of both journals and in repositories that publish the journals. As the PAR-J and BR-J have been publishing articles since the 1940s and 1960s, respectively, some articles are digitally stored as images. Therefore, the search mechanisms were replicated in repositories that divulge the journals’ articles in order to ensure a more accurate and wide-ranging search. In the first stage of identification, articles on the theme of people management were sought. For this purpose, a search was made in the strings of texts associated with the Title and Abstract metadata. At least one of these had to include one of the following terms: “public servant(s)”, “public employee(s)”, for the PAR-J; “servidor(es) público(s)”, “funcionário(s) público(s)” for the BR-J. The abstracts of the articles resulting from this first activity were read in order to gauge their suitability for the study with some of the themes of people management. In the second part of the process of identifying articles, it was necessary to determine whether at least one of the authors was a public servant operating in the same context of the public sphere (local, state or federal) involved in the analyses of the article. For this analysis of whether the authors were connected to public agencies, the content of two metadata of all the authors was analyzed: name of the institution and the supplied e-mail address. In the third stage of the article identification process, the content analysis technique was used to determine whether the texts of the article contained any demands for the public servant community. Table 2 shows the number of articles that resulted from each of the three stages of the article identification process. Thirteen articles with a potential conflict of interests were identified, with 12 in the BR-J and 1 from the PAR-J.

Table 2
Articles resulting from each stage of the process of identifying articles with potential conflicts of interest

	BR-J[a]	PAR-J[b]
1ª) Title or Abstract: containing the term “public servant(s)” or “public employee(s)” for the PAR-J, “servidor(es) público(s)” or “funcionário(s) público(s)” for the BR-J; in addition to one of the themes of people management	26	32
2ª) Authors: at least one must be a public servant working in the same context of the public sphere (local, state or federal) involved in the analyses of the article	14	16
3ª) Content: presenting a demand for public servants in the context debated in the article	12[c]	1[d]
[a] Research conducted on the journal website and in the SciELO repository [b] Research conducted on the journal website and in the EBSCO repository [c] References of the articles described in Appendix B [d] References of the articles described in Appendix C

• Fonte: dados da pesquisa.

Content analysis procedures and techniques for the articles in question

Content analysis (MILES; HUBERMAN 1994) was the main technique applied in this study. In the process of identifying articles with potential conflicts of interest, it was used to identify articles that presented an explicit demand in favor of the community of public servants addressed in the study. For this purpose, two dimensions were used: report and demand. A report includes texts that describe a problem or potential problem. Texts associated with a demand present the action recommended by the authors of the article in the sample to resolve the problem or avoid the occurrence of a problem identified in the report. The results of this stage of the selection process of articles with a potential conflict of interest are shown in Figure 1. Each article has two lines of description, the first presenting the texts of the report and the second with the texts of the demand. In the last column of Figure 1, we present an interpretative summary, with a more direct version of the essence of the texts of the articles in terms of Report and Demand.

Figure 1
Results of the content analysis of the articles in the sample

To determine whether the demands identified in the articles respected the canons of scientific research, the content analysis technique was then applied, working with two types of dimensions: generalizability and consistency. Thus, the aim was to gauge: a) whether the specific issues of a public agency or public location were not being improperly generalized for a wider range of public servants, in other words, respect for the canon of generalizability; b) whether the data collected and analyzed hold up the main claims that support the demand, in other words, respect for the canon of consistency. The logical foundations for defining these aspects to be analyzed were respectively: a) the wider the scope of public servants that benefit from the demand, the greater the possibility of the reviewers feeling that they are included in the demand and will thus benefit from the demands made in the article; b) favorable demands are always likely to be appreciated by those set to benefit from them, and the anxiety to achieve the demand to benefit their own group will lead authors to write messages making demands that do not necessarily reflect the results of their fieldwork.

4 Findings

We begin this section by presenting the findings of the content analysis of the 13 articles in the sample with regard to two scientific research canons: a) consistency in communication of the “public servant report” and communicating the “demand” in terms of the collection or availability of information to support these declarations; and b) the generalizability of the demand in relation to the collectivity studied in each of the articles.

4.1 Generalizability of the demand

The more generalizable the context of the demand in terms of public servants who will benefit from it, the more interesting and attractive the article will be to the whole public servant community, including the reviewers and authors that are members of this professional group. This situation is more likely to attract the interest of reviewers and, consequently, manifest the halo effect. It is also the most laborious and arduous task for the researchers. A scientifically integral item of information in this broad context would require many efforts for collecting information from the three public spheres (federal, state and municipal) and different segments in which public servants are employed (public safety, education, health, transport, etc.). Non-compliance with this requirement could create two situations of risk associated with the quality of the study, with penalties for the generalizability canon of scientific knowledge:

Risk A - Excessive generalizability from limited samples: the authors generalize and include all public servants of the country in question, despite the restrictions on data collection in terms of the sphere and/or segment of the public employees in the sample. In this situation, findings are overestimated and the results achieved configure a loss of focus and the aims of the study;

Risk B - Idiosyncratic findings due to limited samples: discussions are very restricted, e.g., involve only one public company, which makes the results largely unattractive to most readers of the journal and diminishes the scientific importance because of their non-generalizability.

The content analysis technique was applied to the 13 texts in the sample to verify the occurrence of the two situations of risk. For this purpose, the units of analysis were defined as “public institution” analyzed, the “operational segment” of the institution and its “level within the public sphere”. The values observed for these units of analysis are described, respectively, in the third, fourth and fifth columns of Figure 2. Another important aspect was the term used to highlight the group associated with the demand stated in each article (as indicated in the underlined excerpts of the transcript in Figure 1). This was configured as another unit of analysis, shown in the sixth column of Figure 2, “term used for demand”.

Figure 2
Analysis of Demand regarding Generalizability canon

Of the 12 articles of BR-J, 10 showed problems related to one of the two risks associated with the generalizability canon. Risk A was identified in 8 articles (A1, A2, A3, A5, A8, A9, A11, and A12) and Risk B in 2 articles (A4 and A7). The two articles that showed no problems in terms of generalizability were A6 and A10. For the only PAR-J article in the sample (B1) we did not find problems regarding the risks associated with the generalizability canon.

As shown in the second column of Figure 2, the most used research strategy in the thirteen articles in the sample was Case Study Research. This strategy has several purposes that are useful to science. However, it is not recommended when the purpose is the generalizability of theories, axioms and other levels of abstractions, as is the case in articles that present demands. The risk of idiosyncrasy associated with the application of the case study research strategy is highlighted by Eisenhardt (1989, p.547):

Another weakness is that building theory from cases may result in narrow and idiosyncratic theory. Case study theory building is a bottom up approach such that the specifics of data produce the generalizations of theory. The risks are that the theory describes a very idiosyncratic phenomenon or that the theorist is unable to raise the level of generality of the theory.

4.2 Consistency of the report and demand

The consistency canon became a theme for the content analysis of the thirteen articles in the sample. The origin of the argumentative logic employed by the authors to issue their report to public servants and present their demand was observed. The analysis unit “method” was created, to which the following values were assigned:

Referenced - if based on texts from other authors or sources;

Implied - from the analysis of the data collected in the field;

Opinion - if only the opinion of the authors is given without being based on field data or external references.

Figure 3 shows the classification attributed to the method employed to draft the report and demand. The two columns headed “Analysis of evidence in the text” (third and fifth columns) contain excerpts from the texts of the article, with commentary and analysis, explaining the assigned classification, respectively, to the Report and Demand. The most critical situation in terms of the consistency of the statement is when there is no reflection of the literature and when there is no basis in the field data, in other words, it is merely an opinion. For the Report unit of analysis, the classification “Opinion” was assigned to 3 articles (A4, A6 and A8). For the Demand unit of analysis, the classification “Opinion” was assigned to 12 articles of BR-J.

Figure 3
Analysis of the Reports and Demands regarding the Consistency canon

In a cross analysis of the 13 articles in the sample regarding the two canons (generalizability and consistency), problems related to both canons were identified in 10 articles. Articles A6 and A10 had problems restricted to the consistency canon, A6 in relation to the texts of the Report and Demand, and A10 only in relation to the Demand. In short, all the articles of the BR-J sample had problems in relation to the texts of the Report and Demand. The reviewers should have recommended the exclusion of these texts or greater detail and depth of the studies to support their claims. On the other hand, no problems were detected in the article from the PAR-J.

4.3 Further analyses of the demands identified in the articles

The 13 articles identified as having a potential conflict of interest were submitted to the second stage of the content analysis: Repackaging and aggregating the data (MILES; HUBERMAN 1994). In accordance with the precepts of content analysis, during the content analysis, all 13 articles from the sample were read and analyzed by more than one researcher. During the reading, an effort was made to “identify themes and trends in the data overall” and “find out where the emphases and gaps in the data are” (MILES; HUBERMAN 1994, p. 92). For this purpose, during the reading, the researchers introduced labels identifying excerpts of texts associated with emphases: characteristics of communication that emphasize and highlight the demand associated with the interests and concerns of the public servants, making it more attractive to this community, including the reviewers. The analysis and consolidation of the set of excerpts of texts marked with labels for emphasis, led us to find two more analysis dimensions: “Diffusion and use of the demand throughout the article”, and “Discussion centered on public servants”.

4.3.1 Diffusion and use of the demand throughout the article

To gauge the level of importance and emphasis given by the authors to the Demand, as well as the probability of the reviewers having noted this content, we conducted analyses to locate the Demand within the structure of the text. Figure 4 shows the location of the texts in the article in which the authors present their demands. In three articles, the demand is already highlighted in the abstract. This is the case of articles A4, A7 and A9. Three articles state the demand in more than one section of the text (A1, A4 and A7). Considering that there are two articles common to both these groups, it is shown that four articles used at least one of the resources to highlight and emphasize their demands.

Figure 4
Location of the Demand within the structure of the article

Highlighting the demand, including declaring it in the abstract, would be justified, for instance, in a participant study in which demands were made as this would be the major objective of the study. This is not the case of the articles in the sample, as shown in the second column of Figure 2, which indicates the type of research strategy. Highlighting and emphasizing the demand may be considered a means of ensuring the perception of the message and, consequently, increasing the chances of sensitizing the reviewers the occurrence of the halo effect when they analyze the article.

4.3.2 Discussion centered on public servants

“The primary role of the public servant is to help citizens articulate and meet their shared interests” (DENHARDT; DENHARDT 2000, p.549). Articles with discussions that focus exclusively on the interests of public servants within the context of people management in public organizations have another characteristic resulting from this bias. They discuss only the rights of public servants in a way that is not associated with their duties. In practice, this means discussing aspects of people management independent of their objectives and goals. In the case of public servants, this discussion should include the beneficiaries of their actions: society and citizens.

One way of identifying egocentric articles, i.e., articles focused on public servants, was to seek articles that do not include the terms “citizen” and the term “client”. For this purpose, the Word Cruncher function of Atlas.Ti qualitative analysis support was used to count the frequency of all the words in each of the thirteen articles. This was followed by a reflective reading of the articles to gauge the context of the use of the terms citizen and client, discarding the occurrence of words out of context. For a better understanding of the discourse that was strongly focused on public servants, we included in Figure 5 the total for the word “servant” in the article. Figure 5 shows the final result of these actions and indicates that seven of the articles in the sample have egocentric characteristics.

Figure 5
Frequency of the words Citizen, Client and Servant in the articles

4.4 Summary of the analyses of the articles in the sample

Figure 6 summarizes the results of the seven aspects analyzed for the 13 articles in the sample. The most critical dimensions in terms of characterizing biased analysis are those associated with the canons of good practices in scientific research, in other words, the four items associated with the two topics of “generalizability of demand” and the two topics of “consistency of the report and demand”. All of the 12 articles of BR-J violate one of the four aspects associated with these two topics. When all the other aspects are considered, focusing on the contextualization of the message to the interests of public servants, five articles of BR-J sample make use of four or more of the resources analyzed. The only one article of PAR-J making demands in favor of public servants did not violate the canons associated with good scientific research practices.

Figure 6
Aspects analyzed and identified in the articles from the sample

5 Discussion

The results of the analyses indicate that the halo effect could occur in a peer review in the context of language restrictions as observed in BR-J. The high probability of texts in the BR-J written by authors associated with Brazilian public institutions also being evaluated by reviewers associated with Brazilian public institutions is explained by the language restriction of the articles. We can say nothing about the horn effect because we did not have access to the articles on public servant management rejected by the journals. Thus, what we can highlight is the possible occurrence of bias, in other words, a moderate risk of articles being favored in peer reviews due to the feeling of belonging in the group of reviewers. This possibility implies a need for precautions in terms of the accountability and quality assurance in the peer review process of their articles. Issues also arise here regarding the authors, the submission of articles and the reviewers concerning the acceptance of articles for evaluation, possible personal conflicts of interest due to the theme addressed in the article and the professional career of the reviewer. When no reviewers are available from other professional fields other than the one addressed in or related to the article, the journal should expand its team of reviewers. If it is not possible to do this with professionals from other fields, the subtheme should be excluded from the themes addressed by the journal.

Regional journal editors with limitations in terms of language and devoted to a very specific theme, as in the case of the present study, are exposed to many difficulties. To help overcome these problems, researchers should reflect on how important it would be to the evolution of academia and science in countries with less scientific tradition to aid the evaluation process of these peripheral journals. It is necessary to sensitize researchers to commit themselves not only to major journals that confer prestige and status, but also to cooperate with journals with less prestige and recognition. The creation and valuing of social performance indicators for researchers, encompassing these collaborative actions, could be the way to sensitize and act to meet this demand.

In terms of supraeditorial oversight, associated with the metadata of articles from the journals and the repositories of journals often used as sources of information for studies in the field of epistemology, library science, scientometrics and information science, it would be important to add one more category of metadata in addition to language of publication: the language of evaluation. This is relevant because many local and national journals allow peer analysis of articles written in the local language despite the fact that they will later be published in English. This mechanism is used by these journals to increase their readership and to ensure that investments are made in translation only for authors whose texts are approved following peer review. This practice was adopted recently by a number of journals in Brazil and other countries that do not use English as an official language. With all articles published in English, it is more difficult to perceive bias associated with the restrictions of less widespread languages, as observed in this study. This accounts for the importance of registering the metadata protocols of articles, such as the Dublin Core Metadata Initiative (2009), which adds another term before “language”, resulting, for instance, in “publication language” and “peer-review language”.

Limitations and continuity of the study. This study did not analyze the bias of homogeneity in the group of researchers regarding the analysis of texts that carry messages contrary to the concerns demands of the group. In the theoretical perspective, the horn effect in the context in question was not considered, i.e., the rejection of quality scientific articles because they include information that is contrary to the interests of the group of reviewers. This type of bias in the double-blind peer review of articles is as important and critical as the context analyzed in this study. The result of the halo effect is a tendency to publish articles that violate scientific precepts. The result of the horn effect is that, because of the homogeneity of the reviewers regarding a characteristic of the group, there is a risk that important scientific knowledge will not be published in order to preserve the interests of the group. These two possible biases on the part of reviewers of scientific articles, due to group interests, are portrayed in Figure 7: Type 1, bias due to the halo effect, addressed in the present study; Type 2, bias due to the horn effect, not addressed in this study. Thus, the Type 2 bias is an opportunity for continuing the present research project.

Figure 7
Types of bias in double-blind review of articles according to the group to which the reviewers belong

References

• BARDY, A. H. Bias in reporting clinical trials. British Journal of Clinical Pharmacology, v.46, n.2, p. 147-150, 1998.
• BLACKBURN, J. L.D.; HAKEL, M. D. An examination of sources of peer-review bias. Psychological Science, v.17, n.5, p.378-382, 2006.
• BORNMANN, L. The Hawthorne Effect in Journal Peer Review. Scientometrics, v.91, n.3, p.857-862, 2012.
• BORNMANN, L.; DANIEL, H. D. Selecting manuscripts for a high impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, v.59, n.11, p.1841-1852, 2008.
• BRABEN, D. W. Pioneering research: a risk worth taking. Hoboken, NJ: Wiley-Interscience, 2004.
• CLAIR, J. A. Procedural Injustice in the System of Peer Review and Scientific Misconduct. Academy of Management Learning & Education, v.14, n.2, p.159-172, 2015.
• CORBIN, J.; STRAUSS, A. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology, v.13, n.1, p.3-21, 1990.
• DAY, N. E. The Silent Majority: Manuscript Rejection and Its Impact on Scholars. Academy of Management Learning & Education, v.10, n.4, p.704-718, 2011.
• DENHARDT, R. B.; DENHARDT, J. V. The New Public Service: Serving Rather than Steering. Public Administration Review, v.60, n.6, p.549-559, 2000.
• DUBLIN CORE METADATA INITIATIVE. Dublin Core Metadata Element Set, Version 1.1. 2009. Retrieved October 10, 2016, from http://dublincore.org/documents/dces/
  » http://dublincore.org/documents/dces
• EISENHARDT, K. M. Building Theories from Case Study Research. The Academy of Management Review, v.14, n.4, p.532-550, 1989.
• JELICIC, M., AND MERCKELBAch, H. Peer-review: Let’s imitate the lawyers! Cortex, v.38, p.406-407, 2002.
• LEE, C. J., AND SCHUNN, C.D. Social biases and solutions for procedural objectivity. Hypatia: A Journal of Feminist Philosophy, v.26, n.2, p.352-373, 2011.
• LEE, C. J.; SUGIMOTO, C. R.; ZHANG, G.; CRONIN, B. Bias in peer review. Journal of The American Society for Information Science & Technology, v.64, n.1, p.2-17 2013.
• MARSH, H. W., JAYASINGHE, U. W., AND BOND, N. W. Improving the peer-review process for grant applications: Reliability, validity, bias, and generalizability. American Psychologist, v.63, n.3, p.160-168, 2008.
• MILES, M. B., HUBERMAN, A. M. Qualitative data analysis: an expanded sourcebook. 2 ed. Thousand Oaks: Sage, 1994.
• PETERS, D. P.; CECI, S. J. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, v.5, n.2, p.187-195, 1982.
• REGISTRATION DATA OF BRAZILIAN MASTER AND PH.D. PROGRAMS. (2016, May 22). Retrieved from:https://sucupira.capes.gov.br/sucupira/public/consultas/coleta/programa/listaPrograma.jsf
  » https://sucupira.capes.gov.br/sucupira/public/consultas/coleta/programa/listaPrograma.jsf
• SABAJ MERUANE, O.; GONZÁLEZ VERGARA, C.; PINA-STRANGER, D. A. What we still don't know about peer review. Journal of Scholarly Publishing, v.47, n.2, p.180-212, 2016.
• SMITH, R. Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, v.99, n.4, p.178-182, 2006.
• STRAUB, D. W. Editor's comments. MIS Quarterly, v.32, n.1, p.iii-viii, 2008.
• TAJFEL, H. Human groups and social categories: Studies in social psychology. Cambridge (UK): Cambridge University Press, 1981.
• TAJFEL, H. Social identity and intergroup relations. Cambridge (UK): Cambridge University Press, 1982.
• THYEL, B. A. Preparing research articles. New York: Oxford University Press, 2008.
• TRAVIS, G. D. L.; COLLINS, H. M. New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values, v.16, n.3, p.322-341, 1991.
• TURNER, J. C. Some current issues in research on social identity and self-categorization theories. In: ELLEMERS, n.; SPEARS, R.; DOOSJE, B. (Eds), Social identity. Oxford (UK): Blackwell, 1999. p.6-34.
• TVERSKY, A.; KAHNEMAN, D. Judgment under uncertainty: heuristics and biases. Science, v.185, n.4157, p.1124-1131, 1974.
• WHITE, C. N.; POLDRACK, R. A. Decomposing bias in different types of simple decisions. Journal of Experimental Psychology, v.40, n.2, p.385-398, 2014.
• YIN, R. K. Case study research: design and methods. Thousand Oaks, MA: Sage, 2003.

Appendix A

Consolidation of the articles, authors’ and reviewers’ origins for public administration journals: one regional, Revista de Administração Pública, with language limitations, and the other prominent in the international arena, Public Administration Review for 2013-2015.

Public Administration Review Revista de Administração Pública Language of Articles Articles Language of Articles Articles Total % Total % English 167 100.0 English 10 5.0 Portuguese 186 92.5 Spanish 5 2.5 TOTAL: 167 100.0 TOTAL: 201 100.0 Country of 1st Author’s Institution Authors Country of 1st Author’s Institution Author Total % Total % USA 108 64.7 Brazil 188 93.5 Netherlands 14 8.4 Portugal 3 1.5 Denmark 8 4.8 Chile 2 1 England 6 3.6 Spain 2 1 China 5 3.0 Germany 1 0.5 South Korea 5 3.0 Argentina 1 0.5 Wales 5 3.0 Colombia 1 0.5 Sweden 3 1.8 USA 1 0.5 Australia 2 1.2 England 1 0.5 Hong Kong 2 1.2 Mozambique 1 0.5 Germany 1 0.6 Belgium 1 0.6 Brazil 1 0.6 Spain 1 0.6 Estonia 1 0.6 Italy 1 0.6 Portugal 1 0.6 Singapura 1 0.6 Taiwan 1 0.6 TOTAL: 167 100.0 TOTAL: 201 100.0 Country of the Reviewers’ Institution Reviewers Country of the Reviewers’ Institution Reviewers Total % Total % USA 1,327 74.2 Brazil 438 98.9 England 68 3.8 Spain 3 0.7 Netherlands 53 3 Portugal 2 0.5 South Korea 51 2.9 Denmark 36 2 Australia 30 1.7 Hong Kong 21 1.2 Spain 19 1.1 Canada 18 1 Belgium 14 0.8 Wales 14 0.8 China 13 0.7 Singapore 12 0.7 Switzerland 12 0.7 Germany 11 0.6 Italy 11 0.6 Norway 10 0.6 Sweden 9 0.5 Israel 8 0.4 South Africa 7 0.4 New Zealand 6 0.3 Portugal 6 0.3 Scotland 6 0.3 Mexico 4 0.2 Taiwan 4 0.2 Ireland 3 0.2 Malta 3 0.2 Turkey 3 0.2 Brazil 2 0.1 Emirates 1 0.1 Estonia 1 0.1 France 1 0.1 Jamaica 1 0.1 Japan 1 0.1 Malaysia 1 0.1 Pakistan 1 0.1 Thailand 1 0.1 TOTAL: 1,789 100 TOTAL: 443 100.0

Public Administration Review			Revista de Administração Pública
Language of Articles	Articles		Language of Articles	Articles
	Total	%		Total	%
English	167	100.0	English	10	5.0
			Portuguese	186	92.5
			Spanish	5	2.5
TOTAL:	167	100.0	TOTAL:	201	100.0
Country of 1st Author’s Institution	Authors		Country of 1st Author’s Institution	Author
	Total	%		Total	%
USA	108	64.7	Brazil	188	93.5
Netherlands	14	8.4	Portugal	3	1.5
Denmark	8	4.8	Chile	2	1
England	6	3.6	Spain	2	1
China	5	3.0	Germany	1	0.5
South Korea	5	3.0	Argentina	1	0.5
Wales	5	3.0	Colombia	1	0.5
Sweden	3	1.8	USA	1	0.5
Australia	2	1.2	England	1	0.5
Hong Kong	2	1.2	Mozambique	1	0.5
Germany	1	0.6
Belgium	1	0.6
Brazil	1	0.6
Spain	1	0.6
Estonia	1	0.6
Italy	1	0.6
Portugal	1	0.6
Singapura	1	0.6
Taiwan	1	0.6
TOTAL:	167	100.0	TOTAL:	201	100.0
Country of the Reviewers’ Institution	Reviewers		Country of the Reviewers’ Institution	Reviewers
	Total	%		Total	%
USA	1,327	74.2	Brazil	438	98.9
England	68	3.8	Spain	3	0.7
Netherlands	53	3	Portugal	2	0.5
South Korea	51	2.9
Denmark	36	2
Australia	30	1.7
Hong Kong	21	1.2
Spain	19	1.1
Canada	18	1
Belgium	14	0.8
Wales	14	0.8
China	13	0.7
Singapore	12	0.7
Switzerland	12	0.7
Germany	11	0.6
Italy	11	0.6
Norway	10	0.6
Sweden	9	0.5
Israel	8	0.4
South Africa	7	0.4
New Zealand	6	0.3
Portugal	6	0.3
Scotland	6	0.3
Mexico	4	0.2
Taiwan	4	0.2
Ireland	3	0.2
Malta	3	0.2
Turkey	3	0.2
Brazil	2	0.1
Emirates	1	0.1
Estonia	1	0.1
France	1	0.1
Jamaica	1	0.1
Japan	1	0.1
Malaysia	1	0.1
Pakistan	1	0.1
Thailand	1	0.1
TOTAL:	1,789	100	TOTAL:	443	100.0

Appendix B

Articles from the journal with demands for public servants

A1	Borges, R. Are public officials really less satisfied than private sector workers? A comparative study in Brazil. Revista de Administração Pública, v.47, n.6, p.1477-1496, 2013.
A2	Corrêa, A. M. H., and Carrieri, A. P. Moral harassment degrading work relations: a case study in the Judiciary. Revista de Administração Pública, v.38, n.6, p.1065-1084, 2004.
A3	Ibañez, N., Basso, W., Viana, A. L. D., and Shirassu, M. Healthcare systems for city and state civil workers: characteristics, conditioning factors and trends. Revista de Administração Pública, v.37, n.2, p.379-407, 2003.
A4	Magalhães, E. M., Oliveira, A. R., Lima, A. A. T. F. C., Campos, D. C. S., and Cunha, N. R. S. Training policy for the technical-administrative staff at the Federal University of Viçosa (UFV) as perceived by trained employees and managers. Revista de Administração Pública, v.44, n.1, p.55-86, 2010.
A5	Marques, A. L., Borges, R., and Reis, I. C. Organizational change and work satisfaction: a study of civil servants in Minas Gerais state. Revista de Administração Pública, v.50, n.1, p.41-58, 2016.
A6	Nogueira, M. A. The hard change: notes on State reform and human resources. Revista de Administração Pública, v.33, n.6, p.61-88, 1999.
A7	Paiva, K. C. M., and Couto, J. H. Quality of life and `post-management shock¿ managerial stress: the Copasa-MG case. Revista de Administração Pública, v.42, n.6, p.1189-1211, 2008.
A8	Rocha, C. B., and Silva, J. R. G. Employees’ identification with a state-owned company in a changing context: the case of Finep. Revista de Administração Pública, v.41, n.4, p.685-703, 2007.
A9	Rodrigues, D. D., and Afonso, L. E. The impact of the creation of Funpresp on the pension benefits of federal civil servants. Revista de Administração Pública, v.49, n.6, p.1479-1505, 2015.
A10	Silva, P. V. The public servant in the federal constitutions and the Brazilian national constituent assembly design. Revista de Administração Pública, v.22, n.2, p.106-116, 1988.
A11	Vaitsman, J. Managerialism, culture, and expectations among healthcare civil servants. Revista de Administração Pública, v.35, n.1, p.29-47, 2001.
A12	Zaina, N. M. C. Perception and analysis of individual and organizational variables in public - hopes and realities. Revista de Administração Pública, v.21, n.3, p.2-34, 1987.

Appendix C

Articles from the PAR-J that present a demand from public servants

B1	Nigro, L. G., and Waugh Jr., W. L. Violence in the American workplace: Challenges to the public employer. Public Administration Review, v.56, n.4, p.326-333, 1996.

Publication Dates

• Publication in this collection
  10 Feb 2020
• Date of issue
  Jul-Sep 2019

History

• Received
  14 Feb 2019
• Accepted
  12 Sept 2019