Construction and validation of reputation metrics for human resource management in Brazilian public organizations

Abstract

This study deals with developing a reputation measure for the human resource units of the Brazilian public administration. The measurement model consists of 61 items distributed across three dimensions - trust, credibility, and quality. The measure was submitted to a process of theoretical validation - with the participation of six evaluators - and empirical validation - carried out with a sample of 308 respondents from governmental agencies. Structural analyses suggested an impasse between two models, one with one factor, the other with two, then resolved through a bifactor model, whose results indicated a unifactorial structure composed of 19 items theoretically associated with the quality of services provided by the human resource unit to employees and the credibility and trust attributed by employees toward the human resource unit. The findings supported the reputation measure obtained theoretical and empirical support and can be used as a reliable indicator in initiatives to evaluate the effectiveness of human resource areas in the context of public organizations.

Keywords:
human resource management; human resource management in public organizations; human resources reputation; departmental reputation; organizational reputation

Resumo

Este estudo trata do desenvolvimento de uma medida de reputação dos setores de gestão de pessoas da administração pública brasileira. Constituída de 61 itens distribuídos em 3 dimensões - confiança, credibilidade e qualidade -, a medida foi submetida a um processo de validação teórica, com participação de 6 juízes, e empírica, etapa que envolveu uma amostra de 308 respondentes de órgãos dos diversos poderes públicos. As análises estruturais sugeriram um impasse entre 2 modelos fatoriais - um deles com 1 fator e o outro, com 2 -, resolvido por meio de um modelo bifator, cujos resultados indicaram uma estrutura unifatorial composta por 19 itens, associados teoricamente à qualidade dos serviços prestados pelos setores de gestão de pessoas aos seus públicos internos e à credibilidade e confiança atribuídas por tais públicos àquele setor. Enfim, a medida de reputação obteve sustento teórico e empírico, podendo ser usada como indicador fidedigno e confiável em iniciativas de avaliação sobre a efetividade das unidades de gestão de pessoas no contexto de organizações públicas.

Palavras-chave:
gestão de pessoas; gestão de pessoas em organizações públicas; reputação de gestão de pessoas; reputação departamental; reputação organizacional

Abstract

Este estudio trata sobre el desarrollo de una medida de reputación para los sectores de gestión de personas de la administración pública brasileña. Constituida por 61 ítems distribuidos en tres dimensiones ‒confianza, credibilidad y calidad‒, la medida fue sometida a un proceso de validación teórica, con la participación de seis jueces, y empírica, esta etapa realizada con una muestra de 308 encuestados de organismos de diversos poderes públicos. Los análisis estructurales sugirieron un impasse entre dos modelos factoriales ‒uno con un factor, el otro con dos‒, resuelto a través de un modelo bifactorial, cuyos resultados indicaron una estructura unifactorial compuesta por 19 ítems teóricamente asociados a la calidad de los servicios prestados por los sectores de gestión de personas a sus públicos internos y a la credibilidad y confianza atribuidas por dichos públicos a ese sector. Finalmente, la medida de reputación obtuvo sustento teórico y empírico, pudiendo ser utilizada como un indicador fidedigno y confiable en iniciativas de evaluación de la efectividad de las unidades de gestión de personas en el contexto de las organizaciones públicas.

Palabras clave:
gestión de personas; gestión de personas en organizaciones públicas; reputación en gestión de personas; reputación departamental; reputación organizacional.

1. INTRODUCTION

Despite the initiatives undertaken in the field of Strategic Human Resources Management (SHRM) in the context of Brazilian public administration since the 1990s, several reasons prevent sectors responsible for this matter from achieving a strategic level in organizations, among which the following factors stand out: institutional, such as strict rules and legislation and numerous external control mechanisms; political, such as lack of support from senior management, influence of groups on HR actions and administrative discontinuity; organizational, such as resistance to change, distance from power and centralization and low engagement of managers; and sectoral, such as lack of resources in the Human Resources (HR) unit, high volume of administrative personnel activities and low reputation (Araújo, 2019; Côrtes, 2016; Fonseca & Meneses, 2016).

Regarding the last factor, specific to HR units, especially about the reputation of these sectors, studies have identified that, in Brazilian Executive public organizations, reputation is positively related to the demand for and adherence to training initiatives and training offered to employees, with the sector’s actions oriented towards organizational results, obtaining financial resources, and increasing the unit’s autonomy to decide and act (Fonseca, 2013).

Similarly, in the Legislative organizations, there is evidence that the greater reputation of the HR unit results in higher decision-making support, greater availability of resources, increased levels of autonomy of the unit, and increased insertion in the strategic decisions of the legislative house (Côrtes, 2016).

Although the subject of reputation at the level of organizational units has been little researched (Newburry, 2017; Veh, Göbel, & Vogel, 2018), it is clear from the studies above that reputation is a condition for HR sectors to be able to operate strategically, planning or implementing actions guided by the results sought by public organizations (Camões, 2013). Thus, some studies suggest that HR sectors and their various target audiences tend to disagree about the effectiveness of human resources services, with this discrepancy being a supposed reason why competent sectors, confident of their excellent reputation, do not seek to improve services that they offer (Bartram, Stanton, Leggat, Casimir, & Fraser, 2007; Guest & Conway, 2011).

In these studies, the premise is that the policies and practices of HR sectors are, in part, responses to the demands and expectations of various constituents, both internal and external to the organization. The sector’s effectiveness would result from how much the unit continually meets the expectations of its constituents (Tsui, 1984), which would lead to a stable perception of effectiveness over time. From that point on, these constituents, or stakeholders, could predict the actions of the HR units. That is the phenomenon of reputation (Ferris et al., 2007; Tsui, 1990).

Thus, understanding the HR units’ reputation can be a way to gain legitimacy and facilitate the institutionalization of their policies and practices (Ferris et al., 2007). Therefore, measuring it becomes essential for its management. Furthermore, as it affects the stakeholders’ behavior and the organization itself, reputation is seen as an intangible asset that is difficult to copy, as it is something that takes time to build, is of great value to the personnel sectors responsible for administrative management of the organization’s main assets (Vlašić & Langer, 2012).

Considering that there are almost no specific studies and theoretical works about human resources reputation (Ferris et al., 2007; Trullen, Stirpe, Bonache, & Valverde, 2016; Ulrich & Grochowski, 2018), this article communicates the process results of construction and validation for the HR units reputation measure of Brazilian public organizations.

2. THEORETICAL FOUNDATIONS

At the organizational level, reputation is one of the most strategic resources of organizations, as it guides decisions about investments, attracting investors, differentiated pricing practices, and access to the capital market (Boyd, Bergh, & Ketchen, 2010; Fombrun & Shanley, 1990), and is also relevant to individuals, departments, and groups (Coyne, 2010). It is not, however, a given asset (Vlašić & Langer, 2012) but rather a set of perceptions about a given entity that then influences the stakeholders behavior with whom it interacts internally and externally (Coyne, 2010), so it is essential to measure it.

In a bibliographical survey carried out in several databases - for example, Annual Reviews, Oxford Journals, SciELO, SpringerLink, Science Direct, Emerald Insight, Sage Journals, and Google Scholar - with the unit descriptors level reputation, department reputation, and multilevel reputation, with filter in the title, only three theoretical models on the phenomenon of reputation were identified.

One of these studies treats the phenomenon in a one-dimensional and sectorized way when theorizing the reputation of organizational marketing units (Gök, Peker, & Hacioglu, 2015). Another, more comprehensive because it deals with individual and organizational reputation and the relationship with various antecedents, does not consider the intermediate level of organizational analysis (Ferris et al., 2014). In a doctoral thesis presented by Coyne (2010) at Ohio State University, reputation was theorized as a multilevel construct. We understood theorizing as constructing theoretical-deductive models about reputation at more than one level of analysis (micro, meso, and macro).¹

Empirical production is also minimal, considering the specific context of human resources units. Based on the descriptors HR reputation or human resources reputation, Google Scholar presents 669 results without considering filters. Considering the descriptors in the title (allintitle), only ten studies focused on the topic. These results reinforce the need for more publications in the area, not being sufficient for a theoretical-conceptual systematization.

It is also noticeable in studies dealing with the phenomenon at the organizational and individual levels. There have been many attempts to describe and integrate definitions of reputation in use (Clardy, 2012; Dowling, 2016; Lange, Lee, & Dai, 2011; Walker, 2010). To demonstrate the dimension of this problem in understanding what reputation is, Dowling (2016) lists 50 different definitions of organizational reputation since 1983, which is also highlighted in the case of studies on organizational reputation in public administration (Bustos, 2021).

Of these definitions, the most cited are 3: valuable political assets that are used to generate public support, obtain delegated authority and discretion from politicians, protect the agency from political attacks, and recruit and retain valuable employees; a set of symbolic beliefs about an organization’s capabilities, roles and obligations (Carpenter, 2010); and a set of symbolic beliefs about an organization’s actual performance, as well as its capabilities, roles and obligations to fulfill its organizational mission (Maor, 2010).

These and many other definitions treat reputation as individual and collective beliefs or evaluations established by different audiences about organizational products and services or as signals emitted by organizations to their stakeholders. While beliefs reflect characteristics or behaviors that distinguish the organization, affective evaluations reflect its goodness or badness (Lange et al., 2011; Veh et al., 2018). Signals, in turn, of economic origin, refer to the idea that a company can suggest that it has specific unknown characteristics by communicating with others that are easily visible (Bustos, 2021; Joo & Mclean, 2006; Swoboda, Huber, Schuster, & Hirschmann, 2017; Veh et al., 2018). Finally, status refers to definitions of reputation that compare an organization to others.

Although fragile constitutively and operationally, to the point of making it difficult to systematize knowledge about reputation at this single organizational level of analysis - the reason may be the fact that several scientific fields dedicated to the subject, including economics, marketing, administration and psychology -, the 50 definitions listed by Dowling (2016), in some way, shed light on the topic, given the knowledge that, in the theoretical development of reputation, it is essential to separate it from similar constructs. All the latest literature reviews on organizational reputation indicate that it is essential to differentiate it, for example, from identity and image (Bustos, 2021; Clardy, 2012; Gardberg, 2017; Walker, 2010; Wartick, 2002). It is even more critical considering the absence of theoretical proposals on reputation at the unit level, the target of this article.

Beginning with the concept of identity, which serves as a reference for defining both image and reputation (Whetten & Mackey, 2002), organizations use this concept to describe their irrational responses to insignificant situations or those without a logical reason. In this context, Albert and Whetten (1985) define reputation as what is most central, durable, and distinctive about an organization. That is the dominant ontological line in research on identity, which assumes that organizations are social aggregates so that the questions raised deal with how participants see the organization and which individuals are relevant in defining or interpreting the organizations’ identity (Whetten, 2006).

Given the high degree of abstraction inherent in this concept of identity, studies tend to utilize the notion of image. Image comprises messages congruent with identity and disseminated by organizational agents in their communications with external parties. Reputation would be reciprocal to image, in other words, it’s the feedback from stakeholders about the credibility of the messages communicated by the organization’s agents (Whetten & Mackey, 2002). Thus, reputation is a socially constructed set of meanings applied to an organization, which can be measured based on interpretations, reports (stories or anecdotes), or classifications (Clardy, 2012; Vlašić & Langer, 2012).

In this way, it is possible to better understand the phenomenon of reputation based on what organizations communicate to their stakeholders to generate a good image of them. For this reason, it is essential to determine how they evaluate communications about what organizations do and how they decide, use their resources, sign contracts, and obtain properties (Scott, 2003).

According to the definition of Whetten and Mackey (2002), a concept of reputation must encompass the following elements: the object evaluated, the stakeholders who will evaluate this object, their expectations about the messages propagated by the object about its intentions and made, as well as the judgments made by stakeholders based on the satisfaction of their expectations. Before formulating a concept according to these theoretical elements, however, at the risk of a proposal that would be difficult to operationalize, it is worth comparing the debate on the subject.

Using a review of 210 articles, Walker (2010) indicates five attributes common to definitions of reputation: based on perceptions, an aggregate of perceptions from various stakeholders, comparable, positive, or negative, and stable and lasting. Another literature review (Lange et al., 2011) points out that the concept falls into three dimensions: “being known,” which deals with the company’s visibility or collective perception of its prominence; “being known for something,” which refers to perceived predictability of organizational results and relevant behaviors of interest to the public; and “generalized favorability,” which concerns perceptions or judgments of the organization as good, attractive and appropriate.

Clardy (2012), when systematizing the known measures of reputation and the possible interpretations of each one, identified five distinct measurement approaches: reputation as general knowledge or beliefs, reputation as evaluation or judgment, reputation as a brand, reputation as personality, and reputation as a financial asset.

The first approach recognizes reputation as a belief or generalized knowledge about the organization, while the second treats the concept as an assessment of the organization’s prestige, excellence, and quality or value. These two are consistent with the ontological bias of organizations as social actors, as well as the dimensions “being known for something” and “generalized favorability,” elaborated by Lange et al. (2011), from which there is a tendency to use the second approach - reputation as evaluation or judgment - because it allows comparing organizations under similar aspects (Clardy, 2012; Veh et al., 2018).

In addition to this potential for replication in different scenarios, from a conceptual point of view, this approach points to the multidimensionality of reputation, such as prestige, excellence, and quality or organizational value. Another point of attention in this approach is the attitudinal nature of the concept. Reputation would be a set of cognitions based on attitudes towards an organization, an individual, or an organizational unit (or sector) that results from the interaction of stakeholders with it. Therefore, reputation will differ between these stakeholders based on their identification with the target they evaluate (Páez, 2018), and this observation should also be incorporated into reputation definitions.

The most recent theoretical review on the subject was that of Dowling (2016), who listed five essential elements for any concept of reputation: object, attributes of judgment, evaluating entity, central concept of the construct, and stability. The object refers to the evaluated organization. Judgment attributes are beliefs and evaluations of the organization. The evaluating entity refers to the organization’s stakeholders, who will judge. The central theme refers to the dimensions of “being known for something” and “generalized favorability” (Lange et al., 2011). Stability refers to the theory that suggests that reputation varies between different stakeholders, contexts (such as countries or industrial sectors), objects (such as types of companies), and time (before and after an organizational crisis). Regarding the last element, reputation would be relatively stable over time, as it is socially constructed based on a history of corporate performance, culminating in stakeholder expectations (Albert & Whetten, 1985; Ferris et al., 2007, 2014; Veh et al., 2018).

Souza (2016) developed a constitutive definition for the Human Resources unit’s reputation in the federal public administration in Brazil. This definition was created based on the content analysis of 26 data collection events, including 15 interviews and 11 focus groups, covering 11 Brazilian federal public agencies: seven from the Executive, one from the Legislature, one from the Judiciary, and two independents. Data was collected from 53 federal public employees, totaling more than 21 recording hours.

In this research, Souza (2016) identified three human resources reputation factors: trust, credibility, and quality of HR services. The author assessed the quality of HR services from the perspective of accessibility to HR, process efficiency, and transparency in communication. Three stakeholders relevant to HR units were also identified (members of senior management, managers, and employees).

Taking into account the considerations of the systematic reviews presented and the constitutive definition of a reputation for people management areas in public administration proposed by Souza (2016), it is possible to establish a proposal for a more robust constitutive definition for the human resources units reputation: evaluations and judgments issued by employees, managers, and members of senior management about their expectations regarding the quality of the services provided by the human resources units in terms of efficiency, transparency, and accessibility (Souza, 2016). These judgments, stable over time, are formed by their own experiences with the HR unit or from their organizations’ colleagues, influencing how employees expect HR to act and determining the unit’s actions with employees.

Box 1 describes the components considered in this definition according to the theoretical basis developed up to this point.

Box 1
Criteria adopted in defining human resources unit’s reputation

With this definition, it becomes possible to proceed to the development stage of the measure itself. In general, these are the assumptions assumed: HR units, as well as organizations, are treated as social actors and subject to comparison; reputation is an attitudinal construct that determines the predictability of the actions of the observed entity; reputation is a multidimensional construct; and HR unit stakeholders evaluate the area under the exact dimensions.

3. RESEARCH METHODS AND TECHNIQUES

In the first research stage, evidence of the theoretical validity of the HR reputation measure was gathered. According to the definition set out at the end of the previous section, the dimensions of reputation are trust, credibility, and quality of human resources services, which are evaluated in terms of service, efficiency, transparency, and accessibility. Box 2 explains the operational definitions of GP reputation and its three facets.

Box 2
GP reputation operational definitions

The items were prepared with these definitions, considering the following psychometric criteria: behavioral, objectivity, simplicity, clarity, relevance, precision, variety, modality, typicality, and credibility. The instrument as a whole followed, in turn, the amplitude and balance criteria (Cohen, Swerdlick, & Sturman, 2014; Hutz, Bandeira, & Trentini, 2015; Pasquali, 2013).

The input for preparing the items was gathered in 15 interviews and 11 focus groups in 11 Brazilian public organizations at the federal level, seven from the Executive, one from the Legislative, one from the Judiciary, and two independent, carried out with 53 participants, 34 from the Executive, 11 from the Legislature, three from the Judiciary and seven from independent organizations, totaling more than 21 hours of recorded and transcribed content. This sampling is classified as intentional with maximum variation (Flick, 2013), including only managers from HR units, as they have a more comprehensive view of the area and its relationships with different groups of influence. This survey of opinions ended when theoretical saturation was found (Fontanella et al., 2011).

Six judges evaluated the questionnaire. All were Ph.Ds. and had published articles about strategic human resources management in the last three years, four of whom are professors at federal educational institutions. The task assigned to them consisted of classifying the 61 items in the three dimensions considered - trust, credibility, and quality - and evaluating the clarity of the items. The responses were analyzed using the Kappa index and the Content Validity Coefficient (CVC). For the semantic analysis, eight more people participated in the research: four students from postgraduate courses (dedicated to studying SHRM in public organizations) and four public employees who were working at HR units.

After the theoretical and semantic analyses, a second stage began with developing the research questionnaire to submit items to Exploratory Factor Analysis (EFA). To this end, we adopted a five-point Likert type response scale. The only difference was the anchoring only with numbers and at the extremes of the scale, as opposed to determining the nomination for each answer.

Five researchers from a group dedicated to studying strategic human resources management in the context of public organizations and five public employees from Executive and Judiciary organizations participated in a pre-test. At this stage, information about completion time, identified flaws, and suggestions were requested. Mainly, typing errors and suggestions for improvement in the questionnaire fields’ descriptions and the instrument’s initial description were identified.

The final application of the questionnaire took place electronically for a month, through contact with respondents via direct mail, and ended with 308 respondents, only federal public employees who consented to the terms of the study. The EFA technique determined the sample size necessary to guarantee the quality of this research, for which samples of at least 300 participants are recommended (Field, 2020). We confirmed the adequacy with the following analyses: outliers, correlation matrix (KMO) adequacy, and factor loadings.

In addition to the quantitative criteria, sampling followed the requirements of diversity and variability. Public organizations employees (n=95) from all three branches of government were covered. Also, at the level of respondents, these criteria were met: employees (n=203), managers (n=97), and members of senior management (n=8) participated in the study.

As for the analysis itself, through the software Factor, EFA was applied to determine the factorial structure of the reputation measure. We decided the number of factors retained by comparing three methods: parallel analysis, scree plot, and Kaiser criterion (Field, 2020). EFA was implemented using a polychoric matrix, and the extraction method Robust Diagonally Weighted Least Squares (RDWLS) (Asparouhov & Muthén, 2010).

In this study, EFA was used to determine the factor structure of the HR reputation measure. We determined the number of factors to retain after comparing three methods: parallel analysis, scree plot diagram, and Kaiser criterion, based on eigenvalues (Field, 2020). The parallel analysis technique was implemented with random permutation of the observed data (Timmerman & Lorenzo-Seva, 2011), considering that the random permutation of data allows the creation of confidence intervals for the indices used by the replication of the original database of the research (Lorenzo-Seva & Ferrando, 2006). This technique is called optimized data permutation, which differs from Horn’s classical optimization proposed by Monte Carlo simulation (Timmerman & Lorenzo-Seva, 2011).

Promax oblique rotation was used. The H index defined the stability of the factors. In addition, we analyzed three unidimensionality indicators of the HR reputation measure: Unidimensional Congruence (UniCo), Explained Common Variance (ECV), and Mean of Item Residual Absolute Loadings (MIREAL).

The adequacy of the correlation matrix was analyzed using the Kaiser-Meyer-Olkin (KMO) test and Bartlett’s test of sphericity. We also evaluated the adequacy of the model using the fit indices Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI), and Tucker-Lewis Index (TLI). These indices, generally obtained in confirmatory factor analyses, are made available in EFA through the Factor software, making this technique robust for instrument validation. Finally, the reliability of the factors was analyzed using Cronbach’s alpha.

4. RESULTS AND DISCUSSIONS

Of the 61 items prepared, we excluded 24 items because they did not reach Kappa indices more significant than 80% (n=5) or because the CVCs did not exceed 0.75 (n=19), leaving, at the end of this analysis, 37 items, of which 14 were semantically modified upon the judges’ recommendation.

When verifying the adequacy of the sample data and defining the reputation factors, according to the principal axes factoring method (Principal Axis Factoring - PAF), another ten variables with correlations greater than 0.8 were eliminated; otherwise, they would affect the multicollinearity and made it impossible to estimate the contribution of variables to the corresponding factor. There were no variables with correlations lower than 0.3.

After excluding these variables, we used the KMO as the last criterion for sample adequacy for the EFA, with the value obtained equaling 0.971. We performed EFA with the RDWLS as a method of extraction and Promax oblique rotation. The correlation matrix of the 27 items proved to be interpretable as it presented Bartlett’s sphericity test as significant (8083.457, df =351, p<0.001).

The parallel analysis indicated a unifactorial structure with all loadings above 0.3 in this configuration. However, Kaiser’s criterion (eigenvalues above 1) and the scree plot diagram directed the interpretation towards a two-factor structure. Kaiser’s criterion indicated two factors, but the eigenvalues differed significantly - the first was 18.38, and the second was 1.41.

We carried out the analysis with two factors. We eliminated seven variables because they had cross-loadings above 0.3, and this bifactor structure remained stable. The first factor comprised six items associated with the theoretical quality dimension (QUAL), and the second included 13 items from the trust and credibility dimensions (CONF and CRED). We excluded one item due to its association with a theoretical factor different from what we initially proposed.

Table 1
Factor loadings explained variance and reliability of the HR reputation measure

Items	Factor 1	Factor 2
CONF01 - The HR unit applies management tools to improve employees’ work.	-0.119	0.855
CONF02 - After some action by the HR unit, such as performance evaluation and climate survey, there will be improvements in employees’ work.	-0.086	0.829
CONF03 - The practices proposed by the HR unit improve the work performed in the organization.	0.009	0.892
CONF05 - The HR unit looks after the interests of the employees.	0.143	0.802
CONF06 - The actions carried out by the HR unit are good for employees.	0.105	0.821
CONF10 - I am not harmed by the actions carried out by the HR unit.	0.176	0.659
CONF11 - The HR unit defends the interests of the employees.	-0.057	0.933
CONF12 - I know that I will not be harmed if they review any policy or practice of the HR unit.	0.147	0.500
CONF13 - The HR unit implements the actions it proposes.	0.120	0.784
CRED02 - The information disclosed in plans (intention) and reports (execution) by the HR unit represents reality.	0.015	0.819
CRED03 - The HR unit has reliable information about the work reality of the professionals.	0.200	0.586
CRED07 - My co-workers say the HR unit does what it promises to employees.	0.151	0.737
CRED08 - The HR unit delivers what it promises to employees.	0.203	0.746
QUAL05 - My co-workers receive prompt assistance when requesting services from the HR unit.	0.679	0.277
QUAL10 - I have a close relationship with the HR unit.	0.646	0.090
QUAL13 - I have access to the HR unit.	0.988	-0.107
QUAL14 - Access to the HR unit is easy.	0.959	-0.107
QUAL15 - I can easily obtain information about personnel issues from the HR unit.	0.801	0.096
QUAL16 - The HR unit is accessible to all employees.	0.630	0.238
% of variance	7.39	66.76
α	0.915	0.956
Composite Reliability	0.893	0.950
H- Latent	0.951	0.971
H- Observed	0.885	0.940
Factor determination index (FDI)	0.975	0.986
Orion	0.951	0.971

• Notes: These are translations suggestions for these items because it was originally written in Portuguese.
• Source: Elaborated by the authors.

We adopted EFA with Bias Corrected and accelerated (BCa) - bootstrap 95% confidence intervals for loading values - and with Promax oblique rotation, which resulted in a reputation measure with 19 items distributed across two factors. The KMO measurement for these 19 items demonstrated the adequacy of the sample for the analysis (KMO [95% CI BCa] = 0.947 [0.944 - 0.959]).

Together, the factors explained 73.91% of the variance, with 66.76% of the proportion of explained variance attributed to factor 2 and 7.15% to factor 1. Both factors also explained more than 3% of the total variance, being relevant to the measure. The correlation presented between factors was 0.759. Both factors also presented four or more factor loadings above 0.6.

In short, the indices obtained (χ² = 43.239, df = 134, χ²/ df = 0.323; RMSEA [95% IC BCa] = 0.000 [0.010 - 0.0288]; TLI [95% IC BCa] = 1.1011 [ 1.010 - 1.016]; CFI [95% CI BCa] = 1.008 [1.007 - 1.012]) indicated that a two-factor structure was adequate and reliable. Cronbach’s alpha presented values above 0.70 for both factors. The composite reliability index exceeded 0.7 for both factors. The H- Latent and H- Observed measures indicated that both factors could be replicable in future studies (H > 0.80). The FDI and Orion, above 0.9, pointed to the quality and effectiveness of the factor score estimates.

Despite this evidence suggesting the suitability of a two-factor structure, as well as the high levels of reliability and replicability, two indications pointed to a one-dimensional structure: the high shared correlation between the factors (0.759) and the result of the parallel analysis, corroborated by the indices described in Table 2.

Table 2
One-dimensional approximation evaluation

Index	General evaluation	Overall rating (95% CI)	Interpretation
single	0.987	(0.980 0.993)	> 0.95
ECV	0.909	(0.888 0.929)	> 0.85
MIREAL	0.222	(0.193 0.249)	< 0.30

• Source: Elaborated by the authors.

As noted, the indices indicated that the measure should be interpreted as unidimensional, being endorsed by the confidence interval. Therefore, we submitted the 19 items to a single-factor structure (Table 2). This factorial structure also presented adequate fit indices (χ² = 163.0, df = 152, χ²/ df = 1.07; RMSEA [95% CI BCa] = 0.066 [0.010 - 0.0879]; TLI [95% CI BCa] = 0.989 [0.980 - 1.001]; CFI [95% CI BCa] = 0.990 [0.982 - 1.001]). The indices, in general, remained stable and portrayed as expected: reliability and replicability indices increased, as did the residuals, due to the reduction in factors and the explained variance of the model. Except for RMSEA, which is still acceptable, the other fit indices (CFI and TLI) remained adequate.

Ultimately, the data presented indicated different analysis options. Despite initially suggesting a unidimensional structure, the factors’ reliability, quality, and replicability indices pointed to a two-factor structure, indicating that the latent traits would probably be maintained in new samples. Therefore, we compared the fit indices to summarize the factorial structures obtained (Table 3).

Table 3
Factor structures comparison of the HR reputation measure

Indexes	Two-factor structure (95% CI)	One-factor structure (95% CI)
RMSEA	0.000 (0.010 - 0.0288)	0.066 (0.000 - 0.0879)
NNFI (TLI)	1.011 (1.010 - 1.016)	0.989 (0.980 - 1.001)
CFI	1.008 (1.007 - 1.012)	0.990 (0.983 - 1.001)
BIC	283,580 (231,722 - 288,923)	571,160 (355,505 - 730,029)

• Source: Elaborated by the authors.

Both models presented excellent fit indices, with the single-factor structure demonstrating a higher RMSEA (0.063), but still adequate. However, among these indices, when prioritizing the Bayesian Information Criterion (BIC) - with models with a lower BIC being more appropriate (Brown, 2015) - the two-factor structure stood out, although, in general, this and the one-factor structure are adequate. With this impasse, we decided to specify a two-factor model.

The two-factor model structure also presented adequate fit indices (χ² = 25.347, df = 117, χ²/ df = 0.217; RMSEA [95% IC BCa] = 0.000 [0.010 - 0.2182]; TLI [95% IC BCa] = 1.017 [1.017 - 1.020]; CFI [95% IC BCa] = 1.012 [1.011 - 1.014]), better than the best presented before, even though the BIC was located between the one and two-factor models (BIC [95% BCa] = 305.535 [289.367 - 303.298)]. The correlation matrix between the factors was 0.564.

The OmegaS coefficient, analogous to Cronbach’s alpha, achieved results above 0.9. However, their values did not indicate the unidimensionality of the data. Regarding replicability, the H index did not present an appropriate score only for the “quality” dimension. In the case of FDI, for the two factors - quality; trust, and credibility - the index was lower (as expected) due to a smaller number of items, but still close to adequate. The Item Explained Common Variance (I-ECV) indicated that only two items in the measure suggested unidimensional treatment. The ECV of the general factor, in this two-factor model, showed significant variation (ECV [IC 95% BCa] = 0.589 [0.555 - 0.611]) to the ECV of the two-factor measure, resulting in the data not being treated as unidimensional. The Percent Uncontaminated Correlations (PUC) were very low (PUC = 0.456), confirming the non-unidimensionality of the instrument.

These results converge with the discussed multidimensionality (Clardy, 2012; Fombrun, Gardberg, & Sever, 2000; Fombrun & Shanley, 1990; Helm, 2007, 2011; Lange et al., 2011; Money, Saraeva, Garnelo-Gomez, Pain, & Hillenbrand, 2017; Páez, 2018; Swoboda et al., 2017; Veh et al., 2018; Walker, 2010; Wartick, 2002) of the reputation construct, theoretically assumed in this study during the original proposition of the items.

According to the theoretical perspective of Lange et al. (2011), the dimension “being known for something” refers to a perceived predictability of results and behaviors relevant to stakeholders. Items that represent expectations - and their fulfillment - about the results of the HR unit clarify these perspectives. “The HR unit is close to me,” “I can get information about personnel issues from the HR unit easily,” and “the HR unit is accessible to all employees” are good examples of the “being known for something” dimension. These items highlight that the HR unit is recognized for its quality. They also show that the HR unit demonstrates and is perceived as being accessible, transparent, and efficient.

The “generalized favorability” dimension refers to generalized perceptions, opinions, and evaluations of the HR reputation as a suitable and appropriate sector in the view of its stakeholders. This dimension represents the attitudinal and evaluative component of HR. “After some action by the HR unit (performance evaluation, climate survey, among others), there will be an improvement in the work of employees.” “I know that I will not be harmed if they review any policy or practice of the HR unit” and “The HR unit delivers what it promises to employees” are good examples of the “generalized favorability” dimension. From this perspective, stakeholders perceive the benefits of the HR unit for the work they perform in the organization.

It is also necessary to portray the fine line between trust and credibility. While the first deals with positive expectations associated with the acceptance of a risk or vulnerability in the GP unit, the second refers to the perception of the quality of a source, media, or message, which may or may not result in trustworthy behaviors (Rieh & Danielson, 2007). Uses such as “Trust the information,” “Accept the advice,” and “Believe in the result” are understood as references to credibility (Fogg & Tseng, 1999).

This distinction becomes more complicated when the authors define credibility into four types: assumed, experienced, superficial, and reputed (or renowned). Reputed credibility refers to how a person perceives another, or something based on what others have said. Of course, trust does not strictly refer to how expectations were created, but they derive from their own experiences or those of third parties (Páez, 2018), so much so that much of the reputation management literature has a strong focus on communication (Carroll, 2013; Evans, 2014; Luoma-aho, 2013; Melewar, 2008). Therefore, there is no practical distance when analyzing reputation, separating trust, and credibility into two factors.

Both trust and credibility were demonstrated as stakeholders’ expectations regarding the HR unit. Empirical portrayal does not differentiate between expectations about unit behavior and information. In other words, trust and credibility are part of a single latent trait, and based on the adjustment indices presented, this pattern will likely be maintained.

5. FINAL CONSIDERATIONS

This article demonstrated valid evidence of a reputation measure for human resources management areas of Brazilian federal public administration organizations. This study is valuable in filling the research gap that proposed operationalizing reputation at the departmental level and applied to a specific context. The measure developed can increase predictive performance models for the human resources units and organizations. Therefore, based on the results presented and discussed, the purpose of the research was achieved, as evidence of theoretical and empirical validity was communicated.

First, this study’s purpose arose from the interest in understanding the numerous failures in attempts to implement strategic human resources management in public administration (Appel & Bitencourt, 2008; Camões & Meneses, 2016). Secondly, identifying this construct as relevant in the strategic process of human resources units in public administration (Côrtes, 2016; Souza, 2016) made the operationalization of this complex phenomenon indispensable for the empirical analysis of this field of research.

The complexity of working on a multifaceted phenomenon like reputation is highlighted when there are no approaches at the same level of analysis. Studies show the relevance of the construct but still discuss measurement according to indices such as “best companies to work for,” created 15 years ago (Ferris et al., 2007; Joo & Mclean, 2006) and which confuse departmental reputation with organizational reputation. Unsurprisingly, despite all the investment in the theoretical and empirical understanding of the phenomenon, it was noted that it took much work to deal with the subject in the development of the measure.

Thus, with a view to a definitive theoretical-empirical solution, it is suggested that this study be completed by gathering new evidence of the validity of the HR reputation measure, which can be done by conducting applications on new samples and carrying out Confirmatory Factor Analyses (CFA), with variance analysis and linear regression, in which models reputation is associated with measures that relate to it, such as individual commitment, turnover, and organizational citizenship. Such tests aim to attribute signs of convergent and discriminant validity to the measure developed.

Furthermore, we recommend that future research seek to determine whether the following hypotheses theoretically supported in the literature on organizational reputation hold in the case of human resources units: that the HR reputation differs between members of HR unit itself and other employees of the public agencies; that HR reputation differs between employees who have contact with the HR unit and those who do not; that stakeholders differ in their assessment of the HR unit’s reputation; and that reputation varies depending on the set of policies and processes developed in each public organization.

REFERÊNCIAS

• Albert, S., & Whetten, D. A. (1985). Organizational identity. Research in Organizational Behavior, 7, 263-295.
• Appel, H., & Bitencourt, C. C. (2008). Gestão de pessoas por competência: institucionalização, possibilidades e dificuldades implícitas nas relações trabalhistas brasileiras. Organizações & Sociedade, 15(46), 175-193. Recuperado de https://doi.org/10.1590/S1984-92302008000300009
  » https://doi.org/10.1590/S1984-92302008000300009
• Araújo, M. L. (2019). Iniciativas de indução à implementação da gestão de pessoas por competências no Executivo federal (Dissertação de Mestrado). Universidade de Brasília, Brasília, DF.
• Asparouhov, T., & Muthén, B. (2010, maio 03). Simple second order chi-square correction Recuperado de https://www.statmodel.com/download/WLSMV_new_chi21.pdf
  » https://www.statmodel.com/download/WLSMV_new_chi21.pdf
• Bartram, T., Stanton, P., Leggat, S., Casimir, G., & Fraser, B. (2007). Lost in translation: exploring the link between HRM and performance in healthcare. Human Resource Management Journal, 17(1), 21-41. Recuperado de https://doi.org/10.1111/j.1748-8583.2007.00018.x
  » https://doi.org/10.1111/j.1748-8583.2007.00018.x
• Boyd, B. K., Bergh, D. D., & Ketchen, D. J. (2010). Reconsidering the reputation -performance relationship: a resource-based view. Journal of Management, 36(3), 588-609. Recuperado de https://doi.org/10.1177/0149206308328507
  » https://doi.org/10.1177/0149206308328507
• Brown, T. A. (2015). Confirmatory factor analysis for applied research (2a ed.). New York, NY: Guilford Publications.
• Bustos, E. O. (2021). Organizational reputation in the public administration: a systematic literature review. Public Administration Review, 81(4), 731-751. Recuperado de https://doi.org/10.1111/PUAR.13363
  » https://doi.org/10.1111/PUAR.13363
• Camões, M. R. S. (2013). Análise do processo de implementação da Política Nacional de Desenvolvimento de Pessoal(Dissertação de Mestrado). Universidade de Brasília, DF.
• Camões, M. R. S., & Meneses, P. P. M. (2016). Gestão de pessoas no governo federal: análise da implementação da Política Nacional de Desenvolvimento de Pessoal. Cadernos Enap, 45, 1-104. Recuperado de https://repositorio.enap.gov.br/handle/1/2560
  » https://repositorio.enap.gov.br/handle/1/2560
• Carpenter, D. P. (2010). Reputation and power: organizational image and pharmaceutical regulation at the FDA Princeton, NJ: Princeton University Press.
• Carroll, C. E. (2013). The handbook of communication and corporate reputation Hoboken, NJ: John Wiley & Sons.
• Clardy, A. (2012). Organizational reputation: issues in conceptualization and measurement. Corporate Reputation Review, 15(4), 285-303. Recuperado de https://doi.org/10.1057/crr.2012.17
  » https://doi.org/10.1057/crr.2012.17
• Cohen, R. J., Swerdlick, M. E., & Sturman, E. D. (2014). Testagem e avaliação psicológica: introdução a testes e medidas(8a ed.). Porto Alegre, RS: AMGH.
• Côrtes, F. G. (2016). Gestão estratégica de pessoas no Legislativo federal: desafios e possibilidades Brasília, DF: Editora UnB.
• Coyne, E. (2010). Reputation as information: a multilevel approach to reputation in organizations Columbus, OH: Ohio State University. Rcuperado dehttps://etd.ohiolink.edu/pg_10?0::NO:10:P10_ACCESSION_NUM:osu1276696814
  » https://etd.ohiolink.edu/pg_10?0::NO:10:P10_ACCESSION_NUM:osu1276696814
• Dowling, G. R. (2016). Defining and measuring corporate reputations. European Management Review, 13(3), 207-223. Recuperado de https://doi.org/10.1111/emre.12081
  » https://doi.org/10.1111/emre.12081
• Evans, K. (2014). The handbook of communication and corporate reputation. Journal of Business & Finance Librarianship, 19(2), 166-168. Recuperado de https://doi.org/10.1080/08963568.2014.884867
  » https://doi.org/10.1080/08963568.2014.884867
• Ferris, G. R., Harris, J. N., Russell, Z. A., Blass, F. R., Ellen, B. P., & Martinez, A. D. (2014). The role of reputation in the organizational sciences: a multilevel review, construct assessment, and research directions. Research in Personnel and Human Resources Management, 32, 241-303. Recuperado de https://doi.org/10.1108/S0742-730120140000032005
  » https://doi.org/10.1108/S0742-730120140000032005
• Ferris, G. R., Perrewé, P. L., Ranft, A. L., Zinko, R., Stoner, J. S., Brouer, R. L., … Laird, M. D. (2007). Human resources reputation and effectiveness. Human Resource Management Review, 17(2), 117-130. Recuperado de https://doi.org/10.1016/j.hrmr.2007.03.003
  » https://doi.org/10.1016/j.hrmr.2007.03.003
• Field, A. (2020). Descobrindo a estatística usando o SPSS(5a ed.). Porto Alegre, RS: Penso.
• Flick, U. (2013). Introdução à metodologia de pesquisa: um guia para iniciantes Porto Alegre, RS: Penso .
• Fogg, B. J., & Tseng, H. (1999). The elements of computer credibility. In Proceedings of the 99º SIGCHI conference on human factors in computing systems the CHI is the limit, Pittsburgh, PA. Recuperado de https://doi.org/10.1145/302979.303001
  » https://doi.org/10.1145/302979.303001
• Fombrun, C. J., Gardberg, N. A., & Sever, J. M. (2000). The reputation quotientSM: a multi-stakeholder measure of corporate reputation. Journal of Brand Management, 7(4), 241-255. Recuperado de https://doi.org/10.1057/bm.2000.10
  » https://doi.org/10.1057/bm.2000.10
• Fombrun, C. J., & Shanley, M. (1990). What’s in a name? Reputation building and corporate strategy. Academy of Management Journal, 33(2), 233-258. Recuperado de https://doi.org/10.5465/256324
  » https://doi.org/10.5465/256324
• Fonseca, D. R. (2013). Autonomia de unidades de gestão de pessoas para o desempenho das atividades estratégicas de capacitação na administração pública federal (Dissertação de Mestrado). Universidade de Brasília, Brasília, DF.
• Fonseca, D. R., & Meneses, P. P. M. (2016). Fatores para implantação e desenvolvimento da gestão por competências em agências reguladoras federais. Revista Eletrônica Científica da UERGS, 2(2), 117-133. Recuperado de https://doi.org/10.21674/2448-0479.22.117-133
  » https://doi.org/10.21674/2448-0479.22.117-133
• Fontanella, B. J. B., Luchesi, B. M., Saidel, M. G. B., Ricas, J., Turato, E. R., & Melo, D. G. (2011). Amostragem em pesquisas qualitativas: proposta de procedimentos para constatar saturação teórica. Cadernos de Saúde Pública, 27(2), 388-394. Recuperado de https://doi.org/10.1590/S0102-311X2011000200020
  » https://doi.org/10.1590/S0102-311X2011000200020
• Gardberg, N. A. (2017). Corporate reputation: fashion, fad, or phenomenon? Corporate Reputation Review, 20(3-4), 177-180. Recuperado de https://doi.org/10.1057/s41299-017-0033-4
  » https://doi.org/10.1057/s41299-017-0033-4
• Gök, O., Peker, S., & Hacioglu, G. (2015). The marketing department’s reputation in the firm. European Management Journal, 33(5), 366-380. Recuperado de https://doi.org/10.1016/j.emj.2015.05.002
  » https://doi.org/10.1016/j.emj.2015.05.002
• Guest, D., & Conway, N. (2011). The impact of HR practices, HR effectiveness and a “strong HR system” on organisational outcomes: a stakeholder perspective. The International Journal of Human Resource Management, 22(8), 1686-1702. Recuperado de https://doi.org/10.1080/09585192.2011.565657
  » https://doi.org/10.1080/09585192.2011.565657
• Helm, S. (2007). One reputation or many? Comparing stakeholders’ perceptions of corporate reputation. Corporate Communications, 12(3), 238-254. Recuperado de https://doi.org/10.1108/13563280710776842
  » https://doi.org/10.1108/13563280710776842
• Helm, S. (2011). Corporate reputation: an introduction to a complex construct. In S. Helm (Ed.), Reputation management (pp. 3-16). Heidelber, Germany: Springer Berlin Heidelberg.
• Hutz, C. S., Bandeira, D. R., & Trentini, C. M. (2015). Psicometria Porto Alegre, RS: Artmed.
• Joo, B. K. B., & Mclean, G. N. (2006). Best employer studies: a conceptual model from a literature review and a case study. Human Resource Development Review, 5(2), 228-257. Recuperado de https://doi.org/10.1177/1534484306287515
  » https://doi.org/10.1177/1534484306287515
• Lange, D., Lee, P. M., & Dai, Y. (2011). Organizational reputation: a review. Journal of Management, 37(1), 153-184. Recuperado de https://doi.org/10.1177/0149206310390963
  » https://doi.org/10.1177/0149206310390963
• Lorenzo-Seva, U., & Ferrando, P. J. (2006). Factor: a computer program to fit the exploratory factor analysis model. Behavior Research Methods, 38(1), 88-91. Recuperado de https://doi.org/10.3758/BF03192753
  » https://doi.org/10.3758/BF03192753
• Luoma-aho, V. (2013). Corporate reputation and the theory of social capital. In C. E. Carroll (Org.), The handbook of communication and corporate reputation (pp. 279-290). Hoboken, NJ: Blackwell Publishing Ltd.
• Maor, M. (2010). Organizational reputation and jurisdictional claims: the case of the U.S. Food and Drug Administration. Governance, 23(1), 133-159. Recuperado de https://doi.org/10.1111/j.1468-0491.2009.01470.x
  » https://doi.org/10.1111/j.1468-0491.2009.01470.x
• Melewar, T. C. (2008). Facets of corporate identity, communication, and reputation London, UK: Routledge.
• Money, K., Saraeva, A., Garnelo-Gomez, I., Pain, S., & Hillenbrand, C. (2017). Corporate reputation past and future: a review and integration of existing literature and a framework for future research. Corporate Reputation Review, 20(3-4), 193-211. Recuperado de https://doi.org/10.1057/s41299-017-0034-3
  » https://doi.org/10.1057/s41299-017-0034-3
• Newburry, W. (2017). Dimensions, contexts, and levels: a flourishing reputation field with further advancement to come. Corporate Reputation Review, 20(3-4), 183-185. Recuperado de https://doi.org/10.1057/s41299-017-0026-3
  » https://doi.org/10.1057/s41299-017-0026-3
• Páez, A. (2018). Reputation and group dispositions. Review of Philosophy and Psychology, 9, 469-484. Recuperado de https://doi.org/10.1007/s13164-018-0399-y
  » https://doi.org/10.1007/s13164-018-0399-y
• Pasquali, L. (2013). Psicometria: teoria dos testes na psicologia e na educação (5a ed.). Petrópolis, RJ: Vozes.
• Rieh, S. Y., & Danielson, D. R. (2007). Credibility: a multidisciplinary framework. Annual Review of Information Science and Technology, 41(1), 307-364. Recuperado de https://doi.org/10.1002/aris.2007.1440410114
  » https://doi.org/10.1002/aris.2007.1440410114
• Scott, W. R. (2003). Organizations: rational, natural, and open systems Hoboken, NJ: Prentice Hall.
• Souza, I. G. L. (2016). Reputação das áreas de recursos humanos Brasília, DF: Editora UnB .
• Swoboda, B., Huber, C., Schuster, T., & Hirschmann, J. (2017). Corporate reputation effects across nations: the impact of country distances and firm-specific resources. Management International Review, 57(5), 717-748. Recuperado de https://doi.org/10.1007/s11575-017-0313-3
  » https://doi.org/10.1007/s11575-017-0313-3
• Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2), 209-220. Recuperado de https://doi.org/10.1037/a0023353
  » https://doi.org/10.1037/a0023353
• Trullen, J., Stirpe, L., Bonache, J., & Valverde, M. (2016). The HR department’s contribution to line managers’ effective implementation of HR practices. Human Resource Management Journal, 26(4), 449-470. Recuperado de https://doi.org/10.1111/1748-8583.12116
  » https://doi.org/10.1111/1748-8583.12116
• Tsui, A. S. (1984). Personnel department effectiveness: a tripartite approach. Industrial Relations - A Journal of Economy and Society, 23(2), 184-197. Recuperado de https://doi.org/10.1111/j.1468-232X.1984.tb00896.x
  » https://doi.org/10.1111/j.1468-232X.1984.tb00896.x
• Tsui, A. S. (1990). A multiple-constituency model of effectiveness: an empirical examination at the human resource subunit level. Administrative Science Quarterly, 35(3), 458-483. Recuperado de https://doi.org/10.2307/2393313
  » https://doi.org/10.2307/2393313
• Ulrich, D., & Grochowski, J. (2018). Building a world class HR department. Strategic HR Review, 17(4), 181-185. Recuperado de https://doi.org/10.1108/shr-05-2018-0046
  » https://doi.org/10.1108/shr-05-2018-0046
• Veh, A., Göbel, M., & Vogel, R. (2018). Corporate reputation in management research: a review of the literature and assessment of the concept. Business Research, 12, 315-353. Recuperado de https://doi.org/10.1007/s40685-018-0080-4
  » https://doi.org/10.1007/s40685-018-0080-4
• Vlašić, G., & Langer, J. (2012). Concept of reputation: different perspectives and robust empirical understandings. Tržište, 24(2), 219-244. Recuperado de https://hrcak.srce.hr/en/file/140725
  » https://hrcak.srce.hr/en/file/140725
• Walker, K. (2010). A systematic review of the corporate reputation literature: definition, measurement, and theory. Corporate Reputation Review, 12(4), 357-387. Recuperado de https://doi.org/10.1057/crr.2009.26
  » https://doi.org/10.1057/crr.2009.26
• Wartick, S. L. (2002). Measuring corporate reputation. Business & Society, 41(4), 371-392. Recuperado de https://doi.org/10.1177/0007650302238774
  » https://doi.org/10.1177/0007650302238774
• Whetten, D. A. (2006). Albert and Whetten revisited: strengthening the concept of organizational identity. Journal of Management Inquiry, 15(3), 219-234. Recuperado de https://doi.org/10.1177/1056492606291200
  » https://doi.org/10.1177/1056492606291200
• Whetten, D. A., & Mackey, A. (2002). A social actor conception of organizational identity and its implications for the study of organizational reputation. Business & Society, 41(4), 393-414. Recuperado de https://doi.org/10.1177/0007650302238775
  » https://doi.org/10.1177/0007650302238775

• 1
  Despite the large number of scientific studies that address this variable at a macro (organizational) and micro (individual) level, none have addressed reputation as a concept that interacts at different levels. It is worth highlighting studies, mainly in universities, that address the reputation of departments, generally as a proxy for the characteristics of professors (publications, status, among others) and the department’s budget. However, the relationship between the different levels of reputation - from the individual to the organizational level - was only portrayed in Coyne’s thesis (2010).

Reviewers:

• 12
  Leonardo Henrique Lima de Pilla (Fundação Getulio Vargas, Rio de Janeiro / RJ - Brasil; Tribunal de Contas da União, Brasília / DF - Brasil) https://orcid.org/0009-0005-0378-9611https://orcid.org/0009-0005-0378-9611

• 13
  Two of the reviewers did not authorize the disclosure of their identities.

• 14
  Peer review report: the peer review report is available at this URL: https://periodicos.fgv.br/rap/article/view/90372/84847

• 15
  [Translated version] Note: All quotes in English translated by this article’s translator.

• DATA AVAILABILITY

  The entire dataset supporting the results of this study is available upon request to the corresponding author Igor Guevara Loyola de Souza. The dataset is not publicly available due to contain information that compromises the privacy of the research participants.

Alketa Peci (Fundação Getulio Vargas, Rio de Janeiro / RJ - Brasil) https://orcid.org/0000-0002-0488-1744https://orcid.org/0000-0002-0488-1744

Gabriela Spanghero Lotta (Fundação Getulio Vargas, São Paulo / SP - Brasil) https://orcid.org/0000-0003-2801-1628https://orcid.org/0000-0003-2801-1628

Data availability

The entire dataset supporting the results of this study is available upon request to the corresponding author Igor Guevara Loyola de Souza. The dataset is not publicly available due to contain information that compromises the privacy of the research participants.

Publication Dates

• Publication in this collection
  05 Jan 2024
• Date of issue
  Nov-Dec 2023

History

• Received
  21 Oct 2022
• Accepted
  14 Aug 2023