Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil

Atitudes perante a revisão por pares aberta entre as partes interessadas em uma revista acadêmica no Brasil

Abstract

Scholarly journals should consider the attitudes of their communities before adopting any of the seven traits of open peer review. Unfortunately, surveys from the Global North might not apply to the Global South, where double-blind peer review is commonplace even among natural sciences and medicine journals. This paper reports the findings of a survey on attitudes toward open peer review among four stakeholder groups of a scholar-led medical journal in Brazil: society members, journal readers, authors, and reviewers. Compared to a previous survey, which mostly recruited natural sciences researchers from Europe, this survey found similar support for open peer review in general and for most of its traits. One important exception was open identities, which were considered detrimental by most participants, even more in this survey than in the previous one. Interestingly, participants were more dismissive of open identities as a whole than of statements about its specific consequences. Because preprints are increasingly popular but incompatible with double-blind review, future research should examine the effects of transitioning from double-blind to open identities, especially on gender bias. Meanwhile, scholarly journals with double-blind review might prefer to begin by adopting other traits of open review or to make open identities optional at first.

Keywords:
Scholarly communication; Academic communities; Scholarly journals; Self-publishing; Medical science

Resumo

Periódicos científicos deveriam considerar as atitudes de suas comunidades antes de adotar qualquer um dos sete traços da revisão por pares aberta. Infelizmente, inquéritos do Norte Global podem não se aplicar ao Sul Global, onde a revisão por pares duplo-cega é comum mesmo entre periódicos de ciências naturais e de medicina. Este artigo relata os achados de um inquérito sobre as atitudes perante a revisão por pares aberta em quatro grupos de partes interessadas em um periódico médico no Brasil: membros da associação, além de leitores, autores e revisores do periódico. Em comparação a um inquérito prévio, o qual recrutou principalmente pesquisadores de ciências naturais da Europa, este inquérito encontrou suporte semelhante à revisão por pares em geral e à maioria de seus traços. Uma importante exceção foram as identidades abertas, traço considerado prejudicial pela maioria dos participantes, neste inquérito ainda mais do que no anterior. Curiosamente, os participantes foram mais desfavoráveis às identidades abertas como um todo do que às assertivas sobre as consequências específicas desse traço de revisão aberta. Uma vez que os preprints são crescentemente populares, mas incompatíveis com a revisão duplo-cega, pesquisas futuras deveriam examinar os efeitos de uma transição da revisão duplo-cega para identidades abertas, especialmente sobre o viés de gênero. Enquanto isso, periódicos científicos com revisão duplo-cega podem preferir adotar outros traços de revisão aberta, ou tornar as identidades abertas inicialmente opcionais.

Palavras-chave:
Comunicação científica; Comunidades científicas; Periódicos científicos; Autopublicação; Ciências médicas

Introduction

Peer review became a core feature of journal publishing in the second half of the 20^th century, amidst other significant transformations of scholarly communication (Baldwin, 2018; Tennant et al., 2017; Zuckerman; Merton, 1971). In this now-called traditional form of peer review, editors elicit reports from reviewers of their choice to improve the authors’ manuscripts and inform the editors’ decisions; the entire process is concealed from the public. Peer review is usually double-blind: authors’ and reviewers’ identities are concealed from each other. Single-blind peer review, in which reviewers know the authors’ identities, is usual for journals from the Global North on natural sciences and medicine (Pontille; Torny, 2014).

While peer review is usually trusted and considered beneficial (Jubb, 2016; Mulligan; Hall; Raphael, 2013), direct evidence of its benefits is scarce (Jefferson et al., 2007; Smith, 2006, 2010), and traditional peer review has received strong criticism and calls for reform over the centuries (Csiszar, 2016; Ross-Hellauer, 2017; Tennant et al., 2017). For instance, former medical editor Smith (2006, 2010) denounced traditional peer review as inconsistent, biased, prone to abuse, wasteful, resistant to innovative research and inaccurate, missing most errors. His former journal was one of the first to reform peer review (Schroter; Loder; Godlee, 2020) and, since then, the movement grew into a “peer review revolution” (Tennant et al., 2017).

“Open peer review” is an umbrella term for peer review innovations in the spirit of open science (Ross-Hellauer, 2017; Tennant et al., 2017; Wolfram et al., 2020). Ross-Hellauer (2017) found 122 definitions of open peer review, consisting of various combinations of seven “traits”: open identities was defined as when “authors and reviewers are aware of each other’s identity”; open reports was defined as when “review reports are published alongside the relevant article”; open participation was defined as when “the wider community are able to contribute to the review process”; open interaction was defined as when “direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged”; open pre-review manuscripts was defined as when “manuscripts are made immediately available (e.g., via pre-print servers like ArXiv) in advance of any formal peer review procedures”; open final version commenting was defined as “review or commenting on final ‘version of record’ publications”; and open platforms (“decoupled review”) was defined as when “review is facilitated by a different organizational entity than the venue of publication.” Open identities and open reports are included in most definitions of open peer review and are thus considered “core traits” (Ross-Hellauer, 2017). Refer to Ross-Hellauer (2017) for an analysis of which traits respond to which shortcomings of traditional peer review and fit into which school of open science, and to Ross-Hellauer (2017), Tennant et al. (2017) and Bruce et al. (2016) for reviews of the evidence base for different traits of open peer review. In the Global North, “born open” natural sciences or medicine journals account for much of the open peer review adoption (Tennant et al., 2017; Wolfram et al., 2020). In the Global South, the Scientific Electronic Library Online (SciELO) Network is probably one of the main organizations driving the adoption of open peer review.

Implementing open peer review is not straightforward, though. Different traits of open peer review serve different editorial goals, and their acceptability may vary from one scholarly community to another (Ross-Hellauer; Görögh, 2019). For example, communities traditionally adopting single-blind peer review (medicine and natural sciences in the Global North) (Pontille; Torny, 2014) are also those adopting open identities (Wolfram et al., 2020), suggesting the benefit might be less obvious for communities adopting double-blind peer review. Furthermore, open participation seems more popular in the social sciences and humanities than in the natural sciences, technology and medicine (Ross-Hellauer, 2017).

One proposed strategy is for editors to survey their communities’ attitudes toward open peer review (Ross-Hellauer; Görögh, 2019). In information science, for example, most editors of Brazilian journals are willing to implement some trait of open peer review in their journals, but believe no trait would be compatible with the national reality of the discipline (Garcia; Targino, 2017). Previous surveys found open identities to be the trait scholars thought less likely to improve peer review (Ross-Hellauer; Deppe; Schmidt, 2017), and double-blind peer review to be preferred over single-blind or open identities (Bernal; Román-Molina, 2018; Mulligan; Hall; Raphael, 2013). To the best of our knowledge, however, no survey about open peer review has targeted a wider scholarly community in the Global South yet.

This study reports an online survey on attitudes toward open peer review among stakeholders of a medical journal in Brazil: Revista Brasileira de Medicina de Família e Comunidade (ISSN 2179-7994). This open access journal is fully sponsored by the national association on family and community medicine, Sociedade Brasileira de Medicina de Família e Comunidade (SBMFC). Revista Brasileira de Medicina de Família e Comunidade (RBMFC) publishes articles about family and community medicine, primary health care and their interfaces with public health, health services research and medical education. Each year sees about 60 articles in continuous publication, including not only research articles but also quality improvement reports, clinical reviews, case reports and opinion pieces, among other article types. It is the leading national journal on the modestly-sized medical specialty, with most readers, authors, reviewers and editors coming from Brazil. The journal in question receives more articles from postgraduate programs in collective health than from other knowledge areas (such as medicine), possibly because that’s the area most family and community physicians in the country earn their master’s and PhD degrees in (Fontenelle et al., 2020). The journal is indexed in Literatura Latinoamericana y del Caribe en Ciencias de la Salud (LILACS), Latindex, Directory of Open Access Journals (DOAJ) and Dimensions, among other databases (Fontenelle; Sarti, 2020d). As for ranking, RBMFC was last categorized as B4 in Qualis Periódicos 2013-2016, and in 2021 its Google Scholar h5-index was 17.

Methods

This electronic survey was administered through the Web from April 6 to May 10, 2020, using formr 0.18.3 (Arslan; Tata, 2019), an open-source survey framework (Arslan; Walther; Tata, 2020). The full formr “run” (questionnaires and their interconnections) used in this survey is openly available at the Open Science Framework (Fontenelle; Sarti, 2020a). The participants belonged to four groups of RBMFC stakeholders:

• RBMFC members were physicians (that is, not medical students) with a currently non-expired membership in SBMFC;

• Readers were anyone who had read one or more of articles from RBMFC within the last twelve months;

• Authors were anyone who had published an article in RBMFC within the last five years;

• Reviewers were anyone who had peer-reviewed a submission for RBMFC within the last five years.

The survey was approved by the Research Ethics Committee of Universidade Vila Velha (CAAE nº 28912719.0.0000.5064, report nº 3.846.811). Participants had to give informed consent before proceeding to the questionnaire items. The information provided to prospective participants included the study objectives in neutral terms (“We would like to hear the opinion of readers, authors and reviewers of RBMFC and members of SBMFC about some ways RBMFC might conduct peer review differently”) to avoid biasing the sample to a more positive or negative attitude toward the current system or open peer review. Participants were also informed about the survey’s anonymity, expected duration (“only 15 minutes”, which were also used to inform about duration and harms), benefits (“Results will inform the editorial policies of RBMFC and other scholarly journals in general”), authorship, ethical approval, and means of contact. Participating in the survey was completely voluntary, that is, it was not a requirement for people to continue interacting with RBMFC or SBMFC in any way, and no direct incentives were offered to prospective study participants. Because of the strict anonymity of the survey, there was no way to prevent people from participating multiple times.

Participants were invited through four different advertisements: SBMFC mailed an invitation to its eligible members on April 7 and on May 8, 2020; RBMFC mailed an announcement about the survey to its registered users (readers, authors, reviewers and others) on April 12 and on May 5, 2020; one of the survey authors forwarded the mailed announcement to SBMFC’s email discussion list on May 6, 2020; and RBMFC displayed throughout the survey period an announcement in its rightmost lateral column, above the fold (visible without scrolling down, at least with screen sizes larger than those of smartphones). As in the informed consent page, the advertisements expressed the survey objectives neutrally and listed who was eligible to participate. Because it was anticipated that there would be some overlap between the stakeholder groups, all advertisements linked to the same landing page, which did not require a username and/or password, but was accessible only for those who knew its URL (uniform resource locator, the network “address”).

The questionnaire was adapted from Ross-Hellauer, Schmidt and Deppe (2017) to the study context and translated to Portuguese and Spanish by the authors of this survey. Because the questionnaire defines its key terms (traditional peer review and each trait of open peer review) and had already been pilot tested and administered by Ross-Hellauer, Deppe and Schmidt (2017), the authors of this paper opted for pilot testing only the questionnaire translation and functionality by reading and filling it in themselves, and also by asking a few colleagues to do so.

In total, the questionnaire had five pages. After a landing page for language selection (Portuguese, Spanish, English) and another one for informed consent, two main pages comprised 45 items, and a fifth and last page thanked participants and provided the URL where the results would eventually be made available: <https://osf.io/u9p4n/>. The first main page comprised 18 items: seven about participant characteristics (age, gender, schooling, geographic region, stakeholder groups, satisfaction with peer review in RBMFC, personal experience with peer review); seven about the opinion on whether each trait would improve peer review; and four about agreement on scholarly communication currently working well and the desirability of three open science aspects (open access, open data, and open peer review) being common practice. The second main page comprised 27 items: three about the experience as author and/or reviewer with open identities, open reports, and open participation; and 24 about the agreement with statements about the seven traits of open peer review. Items about satisfaction, desirability (“… should be common practice”) and agreement had an ordinal scale with five levels, plus a sixth “don’t know”. All items (except stakeholder group) had to be filled before proceeding to the next page, and participants were not allowed to revise their answers before submitting them. The questionnaire items were not randomized in any way; the only adaptive aspect of the survey was the language selection.

The survey results were described for each stakeholder group using absolute and relative frequencies. Attitudes toward open peer review, as well as satisfaction with the current system and attitudes toward open access and open data, were described by the combined frequency of answers “agree” and “strongly agree” (or “satisfied” and “very satisfied”, “better” and “much better”). To contrast stakeholder groups despite their overlap, attitudes were also estimated through proportional odds logistic regression with multi-membership (Bürkner, 2018), using packages brms (Bürkner, 2017), version 2.13.5, and RStan, version 2.21.2, for the R environment for statistical computing, version 4.0.2. The model allowed for the possibility that the attitudes of authors and/or reviewers varied differently than attitudes of other participants. The estimates and their 95% uncertainty intervals (UI) were calculated with only weakly informative prior distributions, which are fully described in the analytic code (Fontenelle, 2020a) and were preregistered (Fontenelle; Sarti, 2020b). Answers “don’t know” were considered missing data and excluded from the estimation for the corresponding items. Participants who did not complete the second main page of the questionnaire were not excluded from the analysis of the items in the first main page. Furthermore, their answers on the second page were not imputed, because such data missingness did not correlate with attitudes in the first main page (Kendall’s tau ranging from -0.08, for open peer-review manuscripts, to +0.01, for open participation). Survey participants were not weighted, except for the exclusion of participants who did not belong to any stakeholder group. There was no need to handle atypical timestamps. The analysis plan included in the preregistration (Fontenelle; Sarti, 2020b) and the final analytical code (Fontenelle, 2020a) are available alongside the open data (Fontenelle; Sarti, 2020c). There was no substantial deviation from the analysis plan, other than correcting a misnamed variable.

Results

The survey’s landing page was reached by 402 people, of which 191 (48%) consented to participate, 151 (38%) completed the first of two main pages, and 134 (33%) completed the whole questionnaire. The survey starting times were evenly distributed through the study period (median 2020-04-17, interquartile range [IQR] 2020-04-12 to 2020-05-03) and the survey had a median duration time of 9.4 minutes (IQR, 6.8 to 14.9). The 151 participants who completed the first main page comprised 86 (5%) of 1774 eligible SBMFC members, 82 readers (out of approximately 200 thousand annual visits), 42 authors (out of 290 articles with 992 unique authors), and 50 (22%) of 226 eligible reviewers.

Table 1 describes the 151 survey participants who completed the first main page of the questionnaire. The majority of the participants were 35-44 years old (38%) and male (55%); most were from Brazil (97%) and, in general, their most advanced degree was undergraduate or medical school (40%). All of them belonged to at least one stakeholder group: 57% were SBMFC members, 54% were readers, 28% were authors, and 33% were reviewers. Most participants did not have any experience with open identity (83%), open reports (81%), or open participation (90%) as authors or reviewers, even though 60 (39%) reported having experience both as authors and reviewers with open peer review in general. Two-thirds of the survey participants were satisfied with peer review in RBMFC.

Half of the study participants agreed the current system of scholarly communications worked well (Table 2). Most of them agreed it should be common scholarly practice to make research publications and data available under open access, but only a little over half agreed that open peer review should be a common scholarly routine.

One core trait, open identities, received the least support among all traits, with few participants believing it would improve peer review (Table 3), even though nearly half the participants agreed that open identities would improve the quality of reviewer report and that reviewers and authors should have the option of revealing their own identities (Table 4). While many participants worried open identities would inhibit reviewers from agreeing to review, fewer thought it would inhibit authors from submitting. Furthermore, few participants agreed open identities were fairer than traditional peer review.

The other core trait, open reports, received support more closely matching agreement with open peer review in general, both in terms of stakeholders believing it would improve peer review (Table 3) and of them agreeing it would provide useful information to readers and improve the quality of the reviewer reports (Table 4). On the other hand, a similar proportion of participants worried open reports would inhibit reviewers from making strong criticism or from agreeing to review.

Open interaction was the trait that received the most support, with most participants believing it would improve peer review (Table 3) and agreeing it would result in better publications (Table 4). While almost as many participants believed open final-version commenting would also improve peer review (Table 3), few agreed with post-publication peer review in blog articles, online journal clubs, and social media (Table 4). Support for open participation, open pre-review manuscripts and open platforms was also lower than for open peer review in general, but not as small as support for open identities (Tables 3, 4).

Support for open peer review varied according to the stakeholder group participants belonged to. Readers and SBMFC members seemed to be more supportive (than authors and reviewers) of open peer review in general (Table 2) and trait by trait (Table 3), as well as when considering more specific statements (Table 4).

Table 1 -
Characteristics of the RBMFC stakeholders participating in the survey (Brazil, 2020).

Characteristic	SBMFC members	Readers	Authors	Reviewers	Total
Age
Under 24	1 (1%)	4 (5%)	-	-	5 (3%)
25-34	24 (28%)	25 (30%)	6 (14%)	7 (14%)	43 (28%)
35-44	36 (42%)	31 (38%)	20 (48%)	21 (42%)	58 (38%)
45-54	12 (14%)	8 (10%)	8 (19%)	10 (20%)	21 (14%)
55-64	11 (13%)	11 (13%)	8 (19%)	11 (22%)	20 (13%)
Over 65	2 (2%)	3 (4%)	-	1 (2%)	4 (3%)
Gender
Female	36 (42%)	32 (39%)	19 (45%)	23 (46%)	68 (45%)
Male	50 (58%)	50 (61%)	23 (55%)	27 (54%)	83 (55%)
Non-binary	-	-	-	-	-
Geographic region
North	2 (2%)	4 (5%)	2 (5%)	1 (2%)	8 (5%)
Northeast	8 (9%)	10 (12%)	3 (7%)	6 (12%)	17 (11%)
Southeast	40 (47%)	36 (44%)	17 (40%)	23 (46%)	67 (44%)
South	25 (29%)	25 (30%)	19 (45%)	15 (30%)	42 (28%)
Central-West	9 (10%)	5 (6%)	-	2 (4%)	12 (8%)
Outside Brazil	2 (2%)	2 (2%)	1 (2%)	3 (6%)	5 (3%)
Schooling
High school	-	3 (4%)	-	-	3 (2%)
Undergraduate or medical school	40 (47%)	33 (40%)	5 (12%)	5 (10%)	60 (40%)
Master’s	33 (38%)	24 (29%)	17 (40%)	20 (40%)	48 (32%)
PhD	13 (15%)	22 (27%)	20 (48%)	25 (50%)	40 (26%)
Involvement with RBMFC
SBMFC members	86 (100%)	47 (57%)	22 (52%)	26 (52%)	86 (57%)
Readers	47 (55%)	82 (100%)	26 (62%)	29 (58%)	82 (54%)
Authors	22 (26%)	26 (32%)	42 (100%)	21 (42%)	42 (28%)
Reviewers	26 (30%)	29 (35%)	21 (50%)	50 (100%)	50 (33%)
Overall satisfaction with peer review in RBMFC
Very dissatisfied	5 (7%)	4 (7%)	4 (10%)	3 (6%)	11 (9%)
Dissatisfied	3 (4%)	3 (5%)	-	-	3 (2%)
Neither satisfied nor dissatisfied	16 (23%)	15 (25%)	8 (21%)	7 (15%)	28 (23%)
Satisfied	35 (50%)	29 (48%)	19 (49%)	24 (51%)	59 (49%)
Very satisfied	11 (16%)	10 (16%)	8 (21%)	13 (28%)	19 (16%)
Experience with open peer review
Neither	43 (50%)	32 (39%)	8 (19%)	6 (12%)	60 (40%)
Author	13 (15%)	12 (15%)	5 (12%)	-	22 (15%)
Reviewer	3 (3%)	4 (5%)	2 (5%)	8 (16%)	10 (7%)
Both	27 (31%)	34 (41%)	27 (64%)	36 (72%)	59 (39%)
Experience with open identities
Neither	67 (88%)	62 (85%)	31 (82%)	34 (76%)	111 (83%)
Author	6 (8%)	5 (7%)	5 (13%)	3 (7%)	9 (7%)
Reviewer	-	1 (1%)	-	4 (9%)	5 (4%)
Both	3 (4%)	5 (7%)	2 (5%)	4 (9%)	9 (7%)
Experience with open reports
Neither	66 (87%)	61 (84%)	29 (76%)	36 (80%)	108 (81%)
Author	8 (11%)	9 (12%)	8 (21%)	1 (2%)	14 (10%)
Reviewer	-	-	-	5 (11%)	5 (4%)
Both	2 (3%)	3 (4%)	1 (3%)	3 (7%)	7 (5%)
Experience with open participation
Neither	69 (91%)	67 (92%)	35 (92%)	41 (91%)	121 (90%)
Author	5 (7%)	4 (5%)	2 (5%)	-	6 (4%)
Reviewer	1 (1%)	-	-	2 (4%)	3 (2%)
Both	1 (1%)	2 (3%)	1 (3%)	2 (4%)	4 (3%)

• Source: Elaborated by the authors (2020).
• Note: RBMFC: Revista Brasileira de Medicina de Família e Comunidade; SBMFC: Sociedade Brasileira de Medicina de Família e Comunidade.

Table 2 -
Agreement with statements about aspects of open science among RBMFC stakeholders (Brazil, 2020).

Statement	SBMFC members		Readers		Authors		Reviewers
	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)
The overall current system of scholarly communications works well	43	52	51 (39-63)	37	48	45 (28-57)	23	56	54 (40-73)	30	60	54 (40-71)
Making research publications open access should be common scholarly practice	74	86	85 (73-92)	75	91	92 (84-97)	36	86	79 (63-90)	42	84	84 (71-93)
Making research data open access should be common scholarly practice	70	81	82 (71-89)	71	87	86 (77-93)	35	85	75 (58-86)	40	82	79 (67-91)
Open peer review should be common scholarly practice	52	64	62 (50-74)	51	66	63 (51-76)	21	55	50 (25-63)	26	55	55 (39-69)

• Source: Elaborated by the authors (2020).
• Note: RBMFC: Revista Brasileira de Medicina de Família e Comunidade; SBMFC: Sociedade Brasileira de Medicina de Família e Comunidade; UI: Uncertainty Interval.

Table 3 -
RBMFC stakeholders believing open peer review traits would improve peer review in the journal (Brazil, 2020).

Open peer review trait	SBMFC members		Readers		Authors		Reviewers
	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)
Open identity	14	18	23 (13-35)	10	13	10 (5-19)	3	8	7 (2-18)	4	9	9 (3-20)
Open reports	36	50	58 (44-73)	30	42	46 (32-61)	8	24	29 (11-50)	15	35	40 (23-58)
Open participation	45	59	61 (46-75)	36	49	44 (29-57)	16	43	30 (12-52)	21	48	46 (29-63)
Open Interaction	56	69	70 (57-82)	52	68	70 (57-83)	22	56	45 (21-66)	28	61	62 (45-78)
Open peer review trait	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)
Open pre-review manuscripts	35	46	43 (30-58)	32	44	39 (26-54)	10	27	28 (12-45)	14	32	30 (14-46)
Open final-version commenting	53	67	64 (53-76)	47	63	61 (47-72)	25	68	58 (42-70)	23	51	58 (43-69)
Open platforms	31	48	57 (42-71)	29	45	45 (31-61)	7	26	17 (5-42)	10	30	34 (16-53)

• Source: Elaborated by the authors (2020).
• Note: The model-based estimates refer to each group if their participants did not also participate in other groups. RBMFC: Revista Brasileira de Medicina de Família e Comunidade; SBMFC: Sociedade Brasileira de Medicina de Família e Comunidade; UI: Uncertainty Interval.

Table 4 -
Agreement with statements about open peer review among RBMFC stakeholders (Brazil, 2020).

Statement	SBMFC members		Readers		Authors		Reviewers
	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)
Making reviewer identities open will make reviewers less likely to make strong criticisms	38	53	52 (41-63)	37	52	52 (41-65)	21	57	52 (38-67)	22	52	50 (34-61)
Making reviewer identities open will increase the quality of reviews	30	42	45 (32-60)	24	34	34 (20-48)	10	28	23 (8-42)	14	33	40 (25-61)
Reviewers should be allowed to choose whether or not to make their identities open	41	59	56 (40-71)	40	58	65 (50-80)	21	58	53 (30-70)	31	72	71 (54-87)
Authors should be allowed to choose whether or not to make their identities open	31	44	39 (25-54)	33	48	49 (34-66)	17	47	46 (27-63)	26	59	60 (44-80)
Making reviewer identities open is fairer to authors	22	31	35 (22-51)	18	26	24 (13-39)	6	17	11 (3-31)	6	15	20 (9-36)
Potential reviewers are less likely to agree to review for journals that make reviewer identities open	28	43	43 (28-57)	29	48	47 (30-64)	21	62	60 (42-84)	14	38	47 (29-64)
Potential authors are less likely to submit to journals that make reviewer identities open	11	16	18 (9-31)	13	20	20 (11-34)	15	42	59 (31-82)	9	22	30 (15-49)
Potential authors are less likely to submit to journals that make author identities open	14	21	29 (18-41)	18	29	31 (19-44)	12	34	36 (24-54)	12	31	36 (24-52)
Published review reports provide useful information for the reader	44	63	66 (53-79)	40	59	62 (47-75)	16	46	49 (22-64)	20	50	56 (39-70)
Statement	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)	n	%	Estimate (%) (95% UI)
Publishing review reports will make reviewers less likely to make strong criticisms	25	36	38 (26-49)	27	40	39 (27-52)	14	40	41 (27-53)	18	44	42 (31-55)
Publishing review reports will increase the quality of reviews	43	61	62 (48-76)	34	49	50 (33-64)	15	43	46 (23-60)	23	53	53 (39-69)
Potential reviewers are less likely to agree to review for journals that publish review reports	25	38	42 (30-53)	26	42	42 (30-55)	17	50	44 (32-59)	14	36	43 (29-56)
Potential authors are less likely to submit to journals that publish review reports	17	25	31 (20-43)	21	33	33 (21-47)	15	47	45 (29-75)	12	32	36 (21-52)
Everybody with sufficient knowledge should be able to participate in the review process, regardless of their formal qualifications or area of work	29	41	38 (26-53)	30	43	44 (29-61)	9	25	22 (8-43)	14	33	37 (23-54)
Close circles of reviewers and editors hold back innovative research	41	57	53 (40-67)	44	62	57 (43-73)	16	44	46 (23-58)	20	44	47 (29-59)
Reviewers are more likely to review if they are invited	53	80	77 (64-87)	52	76	77 (63-87)	31	84	79 (64-91)	37	84	83 (71-94)
Increased interaction between authors & reviewers will result in better publications	54	74	69 (56-79)	54	75	74 (63-86)	28	78	67 (42-80)	33	77	73 (60-86)
Manuscripts should be made openly accessible before peer review begins	30	43	45 (30-61)	24	35	29 (17-44)	8	24	20 (7-40)	13	33	29 (14-47)
Blog articles, online journal clubs and social media commentary on final-version publications are part of peer review	26	39	34 (21-51)	24	35	27 (16-42)	11	30	24 (9-41)	10	24	26 (12-41)

• Source: Elaborated by the authors (2020).
• Note: The model-based estimates refer to each group if their participants did not also participate in other groups. RBMFC: Revista Brasileira de Medicina de Família e Comunidade; SBMFC: Sociedade Brasileira de Medicina de Família e Comunidade; UI: Uncertainty Interval.

Discussion

This article reported an online survey with each of four stakeholder groups of RBMFC, a scholar-led journal from Brazil, on their attitudes toward open peer review. While few eligible stakeholders participated in the survey, the neutral language of the advertisements means participants should have similar attitudes to non-participants. One way in which participants might differ from non-participants is the extent to which they concern themselves with peer review in RBMFC. This is indicated by a larger proportion of reviewers participating (in comparison to the proportion of SBMFC members participating) and by the fact that the participants (even among readers and SBMFC members) are more likely to hold a master’s or PhD degree than family and community physicians in general (Fontenelle et al., 2020). Furthermore, while the small sample size precludes precise estimates (as disclosed through the uncertainty intervals), the survey closely reproduced the methods of Ross-Hellauer, Depper and Schimdt (2017). Therefore, the confidence in this survey’s findings increases to the extent that they are similar to theirs, and differences between the surveys hint at the possibility of contextual effects.

This survey’s participants were markedly less experienced with open peer review than those in Ross-Hellauer, Depper and Schimdt (2017), besides being somewhat younger. Additionally, they were from a small medical specialty in a middle-income country in Latin America and are arguably more used to open access journals and double-blind peer review. Meanwhile, participants in Ross-Hellauer, Depper and Schimdt (2017) were generally from the natural sciences in Europe, and many of them are arguably more used to single-blind peer review. This survey had probably fewer or no participants from the publishing industry, but this information was not captured in order to avoid breaking anonymity.

With these differences in mind, it is encouraging that both groups had similarly positive attitudes toward the current system of scholarly communication, open access, open data and open peer review in general. Although attitudes toward most individual traits of open peer review were also similar, open identities deserve further scrutiny.

Stakeholders of RBMFC were even less supportive of open identities than participants in Ross-Hellauer, Depper and Schimdt (2017), and this was the least supported trait in that survey, even though it is considered one of the core traits of open peer review (Ross-Hellauer, 2017). In fact, most RBMFC stakeholders believed opening identities would make peer review worse or much worse (see Supplemental Table 2 in [Fontenelle, 2020b]). Interestingly, when confronted with statements about the consequences of open identities, participants in both surveys were much less dismissive. Stakeholders of RBMFC (including reviewers) agreed less than participants in Ross-Hellauer, Depper and Schimdt (2017) that opening reviewers’ identities would inhibit them from accepting an invitation, and only a third of RBMFC stakeholders agreed opening authors’ identities would inhibit them from submitting. The last statement was not part of the other survey’s questionnaire, possibly because authors’ identities are already open in single-blind peer review.

Authors and reviewers of RBMFC were much more likely to disagree than to agree with open identities being fairer to authors (see Supplemental Table 2 [Fontenelle, 2020b]), whereas participants of Ross-Hellauer, Depper and Schimdt (2017) were as likely to agree as to disagree. The difference is probably due to RBMFC adopting double-blind review, which has been consistently reported as the preferred form of peer review in researcher surveys (Bernal; Román-Molina, 2018; Mulligan; Hall; Raphael, 2013; Pontille; Torny, 2014; Tennant et al., 2017). Switching a natural science journal from single-blind to double-blind was found to attenuate bias in peer review against female authors and might benefit other author demographics as well (Pontille; Torny, 2014; Tennant et al., 2017). On the other hand, blinding reviewers to authors’ identities cannot avoid bias when the blinding fails, nor can it avoid bias against the manuscripts’ contents, such as the conclusions or the theoretical approach (Pontille; Torny, 2014; Tennant et al., 2017). Perhaps more importantly, there is no evidence on the effect of open identities on bias in peer review.

In line with Ross-Hellauer, Depper and Schimdt (2017), most RBMFC readers, authors, and reviewers did not agree manuscripts should be made freely available before peer review, and most authors and reviewers did not believe preprints would improve peer review. This lack of enthusiasm contrasts with the proliferation of preprint servers in the last year, such as bioRxiv, medRxiv, OSF Preprints, SciELO Preprints and EmeRI (Emerging Research Information). Furthermore, RBMFC did not receive a single protest when the journal started to explicitly accept manuscripts already available as preprints, in December 2018. One explanation might be that authors have no intention of depositing their own manuscripts in preprint servers, but do not object to other authors doing so.

Unfortunately, double-blind review is essentially incompatible with preprints (Pedri; Araújo, 2021), especially journal-led preprints, when scholarly journals routinely deposit their pre-review manuscripts in preprint servers, such as SciELO Preprints or EmeRI. The incompatibility is even more pronounced with EmeRI, as it encourages preprint readers to volunteer to review the manuscripts for corresponding journals (open participation). Other traits of open peer review are not so incompatible with double-blind review: open reports need not be signed; open interaction can be anonymized by editorial platforms, such as Open Journal Systems (OJS) 3; nothing stops decoupled peer review from being double-blind; and open commenting on the final version complements rather than substitutes traditional peer review.

Conclusion

Stakeholders of RBMFC, a medical journal in Brazil, were generally as supportive to open peer review as participants of a large-scale survey with an over-representation of researchers from the Global North and the natural sciences. This suggests the results of the previous survey are generalizable to other settings and encourages replication in the arts, humanities and social sciences, which were underrepresented in the previous survey and not included in the present one.

To the best of our knowledge, this is the first survey to actively recruit and separately describe the attitudes of readers or members of a journal’s learned society. Both stakeholder groups were found to be even more supportive of open peer review than the journal’s authors and reviewers, providing assurance for editors of society journals to experiment with open peer review. Stakeholders of scholarly journals published by university departments might be more uniform in their attitudes.

Always a paragon of open science, SciELO Brazil requires its journals to implement some form of “open peer review”: open identities, open reports, or crediting associate editors. Based on this survey and the discussed literature, journal editors should be extremely cautious of open identities, as most researchers believe it would make more harm than good. If open identities are to be adopted, scholarly journals should consider making them optional at first, and monitor the adherence.

The increasing popularity of preprints might make double-blind review unfeasible, and open identities unavoidable. Unfortunately, no experimental study to this date has examined the effects of transitioning from double-blind to open identities. Scholarly journals currently using double-blind peer review should ideally participate in randomized trials to examine the effects of such a transition on the quality and bias of peer review.

References

• Arslan, R. C.; Tata, C. Chain simple forms / surveys into longer runs using the power of R to generate pretty feedback and complex designs https://formr.org. Zenodo, 2019. Available from: Available from: https://zenodo.org/record/3229668#.XxtPXiVv-V4 Cited: Jul. 23, 2021.
  » https://zenodo.org/record/3229668#.XxtPXiVv-V4
• Arslan, R. C.; Walther, M. P.; Tata, C. S. formr: a study framework allowing for automated feedback generation and complex longitudinal experience-sampling studies using R. Behavior Research Methods, v. 52, n. 1, p. 376-387, 2020.
• Baldwin, M. Scientific autonomy, public accountability, and the rise of “peer review” in the Cold War United States. Isis, v. 109, n. 3, p. 538-558, 2018.
• Bernal, I.; Román-Molina, J. Informe de la encuesta sobre evaluación por pares y el módulo “Open Peer Review” de Digital.CSIC. Madrid: Digital.CSIC, 2018. Disponible en: Disponible en: https://digital.csic.es/handle/10261/167425 Acceso en: 7 abr. 2021.
  » https://digital.csic.es/handle/10261/167425
• Bruce, R. et al Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Medicine, v. 14, n. 1, p. 85, 2016.
• Bürkner, P.-C. Brms: an R package for bayesian multilevel models using Stan. Journal of Statistical Software, v. 80, n. 1, p. 1-28, 2017.
• Bürkner, P.-C. Advanced bayesian multilevel modeling with the R package brms. The R Journal, v. 10, n. 1, p. 395-411, 2018.
• Csiszar, A. Peer review: troubled from the start. Nature News, v. 532, n. 7599, p. 306, 2016.
• Fontenelle, L. F. Analytic code for “Attitudes to open peer review among RBMFC stakeholders”. [S. l]: Open Science Framework, 2020a. Available from: Available from: https://osf.io/avs5c/ Cited: Aug 26, 2020.
  » https://osf.io/avs5c/
• Fontenelle, L. F. Data analysis for “Attitudes to open peer review among RBMFC stakeholders”. [S. l]: Open Science Framework, 2020b. Available from: Available from: https://osf.io/5ntep/ Cited: Aug 26, 2020.
  » https://osf.io/5ntep/
• Fontenelle, L. F. et al Postgraduate education among family and community physicians in Brazil: the Trajetórias MFC project. Family Medicine and Community Health, v. 8, n. 3, e000321, 2020.
• Fontenelle, L. F.; Sarti, T. D. Questionnaire for “Attitudes to open peer review among RBMFC stakeholders”. [S.l]: Open Science Framework, 2020a. Available from: Available from: https://osf.io/78gw2/ Cited: Aug 26, 2020.
  » https://osf.io/78gw2/
• Fontenelle, L. F.; Sarti, T. D. Attitudes to open peer review among RBMFC stakeholders [S. l]: Open Science Framework, 2020b. Available from: Available from: https://osf.io/mvc98/ Cited: Aug 10, 2020.
  » https://osf.io/mvc98/
• Fontenelle, L. F.; Sarti, T. D. Data for “Attitudes to open peer review among RBMFC stakeholders”. [S. l.]: Open Science Framework, 2020c. Available from: Available from: https://osf.io/gh73a/ Cited: Aug 28, 2020.
  » https://osf.io/gh73a/
• Fontenelle, L. F.; Sarti, T. D. Open access to articles, data and research materials on RBMFC. Revista Brasileira de Medicina de Família e Comunidade, v. 15, n. 42, p. 2671-2671, 2020d.
• Garcia, J. C. R.; Targino, M. G. Open peer review sob a ótica de editores das revistas brasileiras da ciência da informação. In: Encontro Nacional De Pesquisa Em Ciência Da Informação, 18., 2017, Marília. Anais eletrônicos [...]. Londrina: Associação Nacional de Pesquisa e Pós-Graduação em Ciência da Informação, 2017. Disponível em: Disponível em: http://enancib.marilia.unesp.br/index.php/XVIII_ENANCIB/ENANCIB/paper/view/19 Acesso em: 3 mar. 2021.
  » http://enancib.marilia.unesp.br/index.php/XVIII_ENANCIB/ENANCIB/paper/view/19
• Jefferson, T. et al Editorial peer review for improving the quality of reports of biomedical studies. The Cochrane Database of Systematic Reviews, n. 2, MR000016, 2007.
• Jubb, M. Peer review: the current landscape and future trends. Learned Publishing, v. 29, n. 1, p. 13-21, 2016.
• Mulligan, A.; Hall, L.; Raphael, E. Peer review in a changing world: an international study measuring the attitudes of researchers. Journal of the American Society for Information Science and Technology, v. 64, n. 1, p. 132-161, 2013.
• Pedri, P.; Araújo, R. F. Revisão por pares aberta. AtoZ: Novas Práticas em Informação e Conhecimento, v. 10, n. 1, p. 5-8, 2021.
• Pontille, D.; Torny, D. The blind shall see! The question of anonymity in journal peer review. Ada: Journal of Gender, New Media, and Technology, v. 4, 2014.
• Ross-Hellauer, T. What is open peer review? A systematic review. F1000Research, v. 6, p. 588, 2017.
• Ross-Hellauer, T.; Deppe, A.; Schmidt, B. Survey on open peer review: attitudes and experience amongst editors, authors and reviewers. Plos One, v. 12, n. 12, e0189311, 2017.
• Ross-Hellauer, T.; Görögh, E. Guidelines for open peer review implementation. Research Integrity and Peer Review, v. 4, n. 1, p. 4, 2019.
• Ross-Hellauer, T.; Schmidt, B.; Deppe, A. Open AIRE Open Peer Review Survey 2016. Zenodo, 2017. Available from: Available from: https://zenodo.org/record/439531#.XJAUIChKiM8 Cited: Mar. 18, 2019.
  » https://zenodo.org/record/439531#.XJAUIChKiM8
• Schroter, S.; Loder, E.; Godlee, F. Research on peer review and biomedical publication. BMJ, v. 368, m661, 2020.
• Smith, R. Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, v. 99, n. 4, p. 178-182, 2006.
• Smith, R. Classical peer review: an empty gun. Breast Cancer Research, v. 12, n. 4, S13, 2010.
• Tennant, J. P. et al A multi-disciplinary perspective on emergent and future innovations in peer review. F1000Research, v. 6, p. 1151, 2017.
• Wolfram, D. et al Open peer review: promoting transparency in open science. Scientometrics, v. 125, p. 1033-1051, 2020.
• Zuckerman, H.; Merton, R. K. Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, v. 9, n. 1, p. 66-100, 1971.

Contributors

• 1
  L. F. FONTENELLE was responsible for the conceptualization; data curation; formal analysis; investigation; methodology; project administration; software; visualization; writing - original draft. T. D. SARTI was responsible for the conceptualization; methodology; writing - review & editing.
• 2
  Correspondência para/Correspondence to: L. F. FONTENELLE.

Publication Dates

• Publication in this collection
  29 Oct 2021
• Date of issue
  2021

History

• Received
  17 Dec 2020
• Reviewed
  13 Apr 2021
• Accepted
  21 July 2021