CONSENSUS BUILDING IN A GROUP DECISION-MAKING PROCESS

ABSTRACT

Expert agreement is a key issue for consensus building process; appropriate methods and tools are needed to support an efficient group decision-making process. This study presents a combination of Analytic Hierarchy Process (AHP) with the Delphi method as a useful support for group decision-making processes aimed at consensus building. A new methodology that includes stability and exclusion analysis; as well as new coefficients of concordance and consistency with a statistical approach are proposed through a case study to explore the consensus building in the group decision-making process.

Keywords:
consensus building; consistency analysis; exclusion analysis; stability; MCDM

1 INTRODUCTION

In many cases evaluations of alternatives against these parameters may be obtained only from experts (or the stakeholder him/her self); an overall evaluation of alternatives quality may be obtained only through subjective preferences of the decision maker (^{Figueira et al., 2005}). In such case, stakeholder or experts agreement is a key issue.

When a group of n experts, decision makers or both is involved such that $n\ge 2$ is considered a group decision-making problem (^{Liu et al., 2017}; ^{Paramita et al., 2018}). In this case, the Delphi method (^{Dalkey & Helmer, 1963}) is a practice for the convergence of judgements, which is generally used to make experts converge on shared judgements; is a useful support for participatory decision-making processes aimed at consensus building (^{Le Pira, Inturri, Ignaccolo, & Pluchino, 2017}). Delphi surveys use more than one round of a questionnaire, with feedback after the first round, to elicit experts judgements. A strength of the Delphi method is that because the questionnaires are completed by participants anonymously, they are not prone to social influences such as dominant personalities or pressure to conform to the majority, or to agree with perceived experts (^{Le Pira, Inturri, Ignaccolo, Pluchino, et al., 2017}; ^{RoSSmann et al., 2017}; ^{von der Gracht, 2012}); yet still give participants an opportunity to consider and revise their own judgements in light of what other participants think (^{Maclennan et al., 2018}).

At each round of anonymous interaction, the members of the group are asked to align their judgements, these iterations are aimed at mitigating strong positions and finding a collective decision.

In principle, it is used to elicit experts judgements about criteria, alternatives, attributes, etc. onwards “elements”; but it can also be used to explore consensus building in a group decision-making process. In the Delphi method the degree of consensus, concordance or agreement of the group can be determined by the expression (1).

$C=1-\left[\frac{V_n}{V_t}\right]\times 100$ (1)

Where

C: Coefficient of concordance

V_n: Negative votes

V_t: Total votes

However, the Delphi method may affect the degree of achieved consensus towards the final decision (^{Le Pira, Inturri, Ignaccolo, & Pluchino, 2017}). In this respect, there are multiple ways to measure consensus derived from the Delphi procedure, some of them based on qualitative analysis and some others based on descriptive and inferential statistics (^{von der Gracht, 2012}).

The Delphi method is generally used when multiple-criteria to analyze are involved, for that reason is thus considered a Multi-Criteria Decision Analysis (MCDA) method or Multi-Criteria Decision Making (MCDM) method. The main focus of the MCDM field is to introduce procedures, methods as well as tools for solving problems and support decision-makers (DM) to make better decisions (^{Aires & Ferreira, 2018}). MCDM problems generally comprises of five components which are: goal, criterias, alternatives, decision-makers and outcomes; these can be classified in Multi Attribute Decision Making (MADM) and Multi Objective Decision Making (MODM); else both share similar characteristics (^{Kumar et al., 2017}).

In the process of building consensus, the most of the consulted literature use the Analytic Hierarchy Process (AHP) method by ^{Saaty (1990}) and Delphi method by ^{Dalkey & Helmer (1963}). However, the MCDM methods has combined with simulation (^{Justo Mendes et al., 2020}; ^{Pergher & De Almeida, 2018}), Decisions Support Systems (DSS) (^{Mannina et al., 2019}; ^{Zanazzi et al., 2012}), and fuzzy logic (^{Crispim et al., 2020}; de ^{Souza & Krohling, 2012}; ^{Dinçer & Yüksel, 2019}) among others in a quest to complement the weaknesses of this methods and techniques. Besides the MCDM methods previously cited, other methods that use a single decision maker exist such as: ELECTRE (Elimination et Choice Traduisant la Realité), TOPSIS (Techniques for Order Preference by Similarity to Identical Solution) and PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluations) (^{Bevilacqua Leoneti, 2016}; ^{Paramita et al., 2018}) among others, that they usually have a software to carry out them.

^{Von der Gracht (2012)} mentions 15 methods to measure the consensus and stability of the group when the Delphi method is used; among them the Kendall’s coefficient of concordance (Kendall W), Spearman’s test, the standard deviation, among others. Recently other ways of measuring the degree of consensus have been proposed. For example, the overlap coefficient by ^{Le Pira, Inturri, Ignaccolo, Pluchino, et al. (2017)} where the concept of overlap represent the average similarity between the collective list c and the n individual ones, i.e. as a measure of the average degree of consensus of the group (^{Le Pira, Inturri, Ignaccolo, & Pluchino, 2017}).

Appropriate methods and tools are needed to support an efficient group decision-making process aimed at consensus building. This paper describes a group decision-making process based on AHP was used to structure the problem and to elicit experts judgements and a Delphi method was set up to promote consensus building.

The remainder of the paper is organized as follows: section 2 introduces the materials and methods used in the study; section 3 illustrates the case study; in section 4 the results are discussed and some general conclusions are provided in the section 5.

2 MATERIALS AND METHODS

Develop an efficient group decision-making process can be considered the main goal of the Delphi method, and the only stopping criteria of the rounds is the degree of consensus of the group. ^{Dajani et al. (1979}) remarked that consensus defined as “general agreement of a substantial majority” by ^{Kanniyapan et al. (2019}) is meaningless if group stability has not been reached; group stability is thus considered the necessary criterion (^{von der Gracht, 2012}). The authors ^{Dajani et al. (1979)} defined stability as “the consistency of responses between successive rounds of a study”. A certain level of agreement, e.g. convergence of opinions toward consensus, may in turn also be found in an unstable situation. However, this definition of stability does not guarantee that when three or more rounds of surveys are conducted, the judgment issued in the last round is consistent with the judgment issued in the first round. The individual judgments cannot change in a significant way (^{Zanazzi & Monteiro Gomes, 2009}); for that reason, in the present study stability refers to the consistency of the responses between all rounds of the study.

The need to determine the degree of consistency in the opinion of each expert (stability) is that, even if there is agreement in the group of experts, the results obtained may not be "reliable" due to the existence of "leadership", be a "non-homogeneous" group of experts or other causes. Therefore, the consensus reached by experts is reliable only if they are consistent.

Through a combination of the AHP and Delphi methods aimed at: Structuring the decision-making problem, elicit experts judgements and make them converge towards shared judgements. To explore consensus building in a group decision-making process the following methodology is proposed in the Figure 1.

Figure 1
Proposed methodology.

In the section 2.1 we introduce some basic assumptions and axioms necessary to understand the reasoning behind the study. In section 2.2, new coefficients will be proposed to measure the degree of agreement of the group and the consistency in the judgments issued by each expert; and in section 2.3 a new exclusion analysis will be proposed whose main objective is to determine whether excluding an expert can reach a consensus.

2.1 Assumptions and axioms

Some basic assumptions and axioms are necessary to understand the reasoning behind the study:

1. It is here assumed that the preferences of an individual (e.g. expert, stakeholder) are represented by an ordered list (ranking) of a set of prefixed elements; where the least important element equals 1, the next to 2 and so on (e.g. for three elements A, B and C, a possible order is $A\succ B\succ C$ and $A=3,B=2,C=1$ );

2. The collective preference ranking must be transitive, meaning that, if element A is preferred to B and B to C, then A is preferred to C;

3. In a consensus building process, the final collective preference ranking is assumed to be accepted, meaning that it reflects the individual preferences at a reasonable level (or a good degree of consensus and consistency).

Axiom 1: If a person is asked the same question at different times and answers approximately the same; the variability (expressed by its standard deviation) in their response will be small and their judgment consistent. On the contrary, if a person is asked the same question at different times and answers significantly differently; The variability in their response will be significant and your judgment will be inconsistent. Therefore, consistency = 1 − inconsistency.

Axiom 2: If several people are asked the same question and answer approximately the same; the variability (expressed by the standard deviation) in their answers will be small and judgment concordant. On the contrary, if several people are asked the same question and respond significantly differently; the variability in their answers will be significant and their judgment will be discordant. Therefore, the concordance = 1 − discordance.

Based on the Axioms 1 and 2, it should be noted that when we refer to consistency, variability is analyzed in the judgment of an expert; while when we refer to concordance, the variability in the judgment of several experts is analyzed. Therefore, from the mathematical point of view, we refer to the same analysis in different contexts; Therefore, the principles used to calculate concordance can be applied to calculate consistency.

Axiom 3: Being “n” the number of experts and “m” the number of criteria, alternatives, etc. (elements); The probability that experts have issued their judgment randomly and agree with each other is: $p={\left[\frac{1}{m!}\right]}^n$ as shown in Figure 2.

Figure 2
Probability of agreement.

It is considered a problem of group decision-making when there are at least two experts or decision makers; However, it is unusual to address a problem that needs more than one expert with two experts. As the probability of concordance in judgments issued randomly when using three or more experts is very small; It is assumed in the first round of surveys that experts are consistent if they agree with each other. If the experts do not reach a consensus in the first round, the next round of surveys is carried out, executing the methodology proposed in Figure 1 until the coefficient of concordance of the group and the coefficient of consistency of each expert are greater than 50%.

2.2 Degree of agreement and consistency of the group Degree of agreement or consensus

Step 1: Assuming that there are (m) number of elements, (n) number of experts and (x) rounds of surveys conducted $\forall \left(i=1,2,3,\dots ,m\right)\wedge \left(j=1,2,3,\dots ,n\right)\wedge \left(h=1,2,3,\dots ,x\right)$. To determine the degree of consensus of the experts in the round x, a matrix with the elements k _ij is elaborated and they are normalized by means of the expression (2).

${\hat{k}}_{\mathit{ij}}=\frac{k_{\mathit{ij}}}{{\sum }_{i=1}^m\left(k_{\mathit{ij}}\right)}$ (2)

By doing this we are transforming k _ij into a continuous random variable where ${\hat{k}}_{\mathit{ij}}\in \left[0,1\right]$.

Step 2: Calculate the disagreement of the experts (σ_t ) using the expression (3) proposed by ^{Hillier & Lieberman, (2010}).

${\sigma }_t=\sqrt{\sum _{i=1}^m\left[{\left(\frac{b_i-a_i}{6}\right)}^2\right]}$ (3)

Where

$\left(b_i=\mathit{max}{\hat{k}}_{\mathit{ij}}\right)\wedge \left(a_i=\mathit{min}{\hat{k}}_{\mathit{ij}}\right)$

σ_t: Total standard deviation in expert judgments

To carry out this approach, since ${\hat{k}}_{\mathit{ij}}$ is a continuous random variable that takes values in the range of 0 to 1, the most associated probability function for this purpose is the beta distribution (^{Freund et al., 1996}). So, it is assumed that ${\hat{k}}_{\mathit{ij}}$ follows a beta distribution.

Step 3: Determine the Total Concordance Percentage (C _t ) of the experts by means of the expression (4).

$C_t=\left(1-{\sigma }_t\right)\times 100$ (4)

The reasoning for this approach is done considering Axiom 2.

To be concordant, σ_t must be (at least) less than 0, 05, guaranteeing that $C_t>95$. This approach is based on the fact that, when performing the proposed calculation procedure when preparing a ranking or order of preference, the maximum disagreement that a group of experts can reach is 90%. As consensus by simple majority is reached when concordance is above 50%, in this case it would be above 95% since 90% equals 0% and 100% equals 100%.

However, to have a more rigorous reference value, it would be necessary to determine the total maximum discrepancy that may exist when n experts assess m elements, assuming that the standard deviation of each element is the maximum possible. For this to happen, each element that is assessed must have the maximum (R ⁺) and minimum (R ⁻) possible range score; and the number of times that can occur (q), is given by the number of experts involved. So that: given a number of n experts in total disagreement, there will only until n elements with the maximum range of disagreement. But when the number of elements is greater than the number of experts then there will be n elements with the maximum range of disagreement $\left(R_1^{\pm }\right)$, up to n elements with the following maximum range of disagreement $\left(R_2^{\pm }\right)$ and so on $\left(R_e^{\pm }\right)$; such that the number of ranges (r) is given by the expression (5) rounding to the upper integer.

$r=\frac{m}{n}$ (5)

The number of times the range e (qe) can occur is given by the expression (6)

$\left(q_e=n\right)\forall e=\left(1,2,3,\dots r-1\right);\left(q_r<n\right);\sum _{i=1}^r\left(q_e\right)=m$ (6)

The maximum $\left(R_e^{+}\right)$ and minimum $\left(R_e^{-}\right)$ of the range e will be given by the expression (7).

$\left(R_e^{+}={\hat{k}}_{m-\left(i-1\right)}\right)\wedge \left(R_e^{-}={\hat{k}}_i\right)\forall i=\left(1,2,3,\dots ,m\right);\left({\hat{k}}_m\succ {\hat{k}}_{m-1}\succ \dots \succ {\hat{k}}_1\right)$ (7)

And the total maximum discordance (σ_máx ) that can reach (n) experts analyzing (m) elements is given by the expression (8)

${\sigma }_{m\mathit{\text{á}}x}=\sqrt{\sum _{e=1}^r\left[q_e\times {\left(R_e^{+}-\frac{R_e^{-}}{6}\right)}^2\right]}$ (8)

Where

r: Number of Ranges

$R_e^{+}$: Maximum Range e Score

$R_e^{-}$: Minimum Range e Score

q_e: Number of occurrences of Rank e

The percentage of total minimum Concordance (C _mín ) will be given by the expression (9).

$C_{m\mathit{\text{í}}n}=\left(1-{\sigma }_{m\mathit{\text{á}}x}\right)\times 100$ (9)

By applying the proposed expressions (5, 6, 7, 8 and 9) to determine the minimum concordance, the results shown in Table 1 are obtained.

Table 1
Minimum concordance percentage (Cmín).

Elements	Experts
	2	3	4	5	6	7	8	9	10
2	92,14	92,14	92,14	92,14	92,14	92,14	92,14	92,14	92,14
3	92,14	90,38	90,38	90,38	90,38	90,38	90,38	90,38	90,38
4	92,55	91,18	90,00	90,00	90,00	90,00	90,00	90,00	90,00
5	92,97	91,69	90,84	90,06	90,06	90,06	90,06	90,06	90,06
6	93,36	91,98	91,38	90,81	90,28	90,28	90,28	90,28	90,28
7	93,70	92,47	91,75	91,33	90,93	90,55	90,55	90,55	90,55
8	94,00	92,83	92,03	91,72	91,41	91,12	90,83	90,83	90,83
9	94,26	93,09	92,45	92,02	91,78	91,55	91,33	91,11	91,11
10	94,50	93,40	92,64	92,27	92,09	91,96	91,72	91,55	91,38

Based on the results in Table 1, the maximum total discordance (σ_t ) decreases when the number of experts is less than the number of elements and when the number of experts tends to four. When the number of experts is greater than the number of elements $(n>m)$; The maximum discordance remains constant because at least two experts will agree on the score or assigned position of at least one element.

Considering the above stated, the percentage of total normalized Concordance $\left({\hat{C}}_t\right)$ is calculated by the expression (10) and must be greater than 50%.

${\hat{C}}_t=\frac{C_t-C_{m\mathit{\text{í}}n}}{100-C_{m\mathit{\text{í}}n}}\times 100$ (10)

Degree of consistency

In order to determine whether the result achieved is reliable or not, consistency in the judgment of each expert is calculated by analyzing the variability in the judgments issued by the same expert in each round of surveys from the second round. So that, the judgment issued by the same in all rounds is taken into account. A matrix of x rounds of surveys and m elements is developed; generating a consistency matrix for each expert, to which the following procedure is applied.

1. Normalize by (2)

2. Calculate the inconsistency by (3)

3. Calculate the Expert Consistency Percent $\left(C_j\right)\forall \left(j=1,2,3,\dots ,n\right)$ by (4)

4. Calculate the Consistency Percentage of the expert j normalized $\left({\hat{C}}_j\right)\forall \left(j=1,2,3,\dots ,n\right)$ by (10).

5. Calculate the Average Consistency Percentage in round $h\left({\overline{C}}_h\right)\forall \left(h=2,3,\dots ,x\right)$ by (11).

${\overline{C}}_h=\frac{1}{n}\times \sum _{j=1}^n{\hat{C}}_j$ (11)

To ensure acceptable reliability in the calculation of the concordance ${\overline{C}}_h$ must be greater than 50%.

2.3 Exclusion analysis

The main objective of this new analysis not approached in the bibliography, is to determine whether excluding an expert can reach a consensus. Three forms of exclusion are proposed:

1. Calculating the Concordance Percentage of the group excluding each expert (one at a time) in each round of surveys. If there is any significant increase in the agreement of the group when the expert j is excluded, you can take the decision to exclude that expert from the analysis.

2. Calculating the percentage of agreement between each pair of experts. If there is any expert ( j) whose Percentage of Average Concordance between each pair of experts in the last round is significantly lower compared to the rest, you can take the decision to exclude that expert from the analysis.

3. Analyzing the consistency of expert j in the last round; if ${\hat{C}}_j$ is less than 50%, you can take the decision to exclude that expert from the analysis.

Collective preference ranking

It has been proved that the geometric mean is the unique way to combine the judgments of several individuals to obtain a judgment for a group (^{Figueira et al., 2005}). Determining the collective ranking by the expression (12).

$w_i=\sqrt[n]{\prod _{j=1}^n{\hat{k}}_{\mathit{ij}}}$ (12)

3 CASE STUDY

This case study is based on the decision-making process of the group of experts and decision makers involved in the identification of the best business strategy that allows Mariel Logistics Services to assimilate a new market segment; This company is located in the Mariel Special Development Zone (ZEDM) in the province of Artemisa, Cuba.

The group of experts and decision makers developed a brainstorming session to analyze the problem of how to assimilate a new market segment with higher quality standards. As shown in Figure 3, this problem was structured as a multi-criteria decision-making problem.

Figure 3
Hierarchy of the problem.

Subsequently, using different methods and techniques to help decision-making such as simulation (^{Kabashkin, 2016}; ^{Vitorino et al., 2017}), estimated values were obtained for each criterion to implement each alternative shown in the Table 2.

Table 2
Decision matrix.

Criterio	A1	A2	A3	A4	A5
(C1) Economic benefit (USD)	916626	492813	527313	985626	596313
(C2) Customers Served (Uf)	2	3	3	4	4
(C3) Tons handled (Ton)	1467	1767	1767	1767	2067
(C4) Use of refrigerated chambers (%)	46,8	46,8	57,8	57,8	75
(C5) Delayed merchandise (Ton)	1350	1350	1350	1350	1350

Once these estimated values are obtained, to extract the experts’ judgment, experts were asked to prepare a ranking of the alternatives from the best to the worst. The results obtained in the first round of surveys are shown in Table 3.

Table 3
Results of the first round of applied surveys.

Round 1
	E1	E2	E3	E4	E5
A1	5	4	1	2	2
A2	4	1	2	5	1
A3	2	3	4	1	3
A4	1	5	3	3	5
A5	3	2	5	4	4

When developing the methodology proposed in Figure 1, it was obtained that the degree of consensus or agreement of the experts is 9.15%. The subsequent exclusion analysis indicates that Expert 1 is the one that most affects the consensus of the group; however, excluding it does not significantly increase the consensus of the group, since it increases to 16.8%; so it is necessary to conduct another round of surveys. The collective ranking shown to experts is: $A\mathit{5}\mathit{\succ }A\mathit{4}\mathit{\succ }A\mathit{1}\mathit{\succ }A\mathit{3}\mathit{\succ }A\mathit{2}$. The results of the second round of surveys are shown in Table 4.

In the second round, the degree of consensus of the experts is 58.15% higher than 50% and the average consistency of the experts is 55.2% higher than 50% as well, indicating that the collective ranking $A\mathit{4}\mathit{\succ }A\mathit{5}\mathit{\succ }A\mathit{3}\mathit{\succ }A\mathit{2}\mathit{\succ }A\mathit{1}$ is a reliable result. If the analysis of exclusion corresponding to round 2 was developed, it would have been determined that Expert 1 is the one that most affects the consensus of the group, excluding it, the consensus of the group would increase to 65, 7% and the average consistency of the experts would be increase to 64, 4%; however, the collective ranking would be the same.

Table 4
Results of the second round of surveys.

Round 2
	E1	E2	E3	E4	E5
A1	3	1	1	1	2
A2	2	2	2	3	1
A3	1	3	3	2	3
A4	5	5	5	4	5
A5	4	4	4	5	4

4 DISCUSSION

The results obtained are compared with other tests proposed by (^{Kamnitui et al., 2019}; ^{Mao, 2017}; ^{Puth & Neuh, 2015}; ^{von der Gracht, 2012}). The concordance of the group in each round is compared with the Kendall W. And the consistency of each expert is compared to Spearman’s rank-order correlation coefficient (Spearman’s rho); using in all cases the statistical software Minitab 17. The above mentioned tests were selected considering the characteristics of the data (two or more ordinal variables, dependent when analyzing the consistency of an expert and independent when analyzing the agreement of the group).

With five decision alternatives, each expert has $m!=120$ possible answers, as a whole the five experts have $m{!}^n=24883200000$ possible answers, of which only in 120 possible answers can totally disagree or totally agree. Figure 4 compares the results for the concordance of the group in the first and second round; two fictitious rounds are added where experts totally disagree (Round I) and totally agree (Round F).

Figure 4
Results of concordance coefficient.

The results shown in Figure 4 indicate that the same expected values are obtained by the two coefficients when the experts disagree and agree completely. In round 1 both coefficients are equal, not so in round 2; however, the same conclusion is reached: experts have reached a consensus in Round 2.

When analyzing the consistency of each expert, the results shown in Figure 5 were obtained. Since the results of the Spearman’s rho correlation coefficient are on a scale of −1 to +1 it was necessary to apply a transformation on the scale so that the values obtained could be compared.

Figure 5
Results of consistency coefficient.

The results shown in Figure 5 indicate that, considering the reference values set for each test, the same conclusion is reached: experts 1 and 2 are inconsistent while experts 3, 4 and 5 are consistent. It should be noted that the similarity in the results achieved by the ${\overline{C}}_h$ coefficient is 72% with respect to Spearmans rho.

5 CONCLUSIONS

The proposed methodology differs from the rest because it develops a broader consistency (stability) analysis than that proposed by ^{Dajani et al. (1979}) and incorporates an exclusion analysis that allows the group decision-making process to be thoroughly analyzed. In addition, new coefficients of concordance and consistency with a statistical approach are proposed.

^{Dajani et al. (1979}) and ^{von der Gracht (2012)} have analyzed the possibility of subgroups of individuals with discordant or divergent judgments. However, they have not analyzed the possibility of excluding from the analysis those atypical individuals that affect the analysis of the average behavior of the group, the proposed exclusion analysis solves this problem.

The results obtained with the concordance and consistency coefficients proposed are similar with the obtained with Kendall W and Spearman’s rho test respectively.

In future research, it will be analyzed whether it is possible after the first round of surveys to determine whether or not the group of experts selected can reach a consensus on the condition that all experts are consistent. What will allow “a priori” to determine if the selected group is homogeneous or not.

References

• ¹
  AIRES RFF & FERREIRA L. 2018. The rank reversal problem in multi-criteria decision making: A literature review. Pesquisa Operacional, 38(2): 331-362. https://doi.org/10. 1590/0101-7438.2018.038.02.0331
  » https://doi.org/10. 1590/0101-7438.2018.038.02.0331
• ²
  BEVILACQUA LEONETI A. 2016. Considerations regarding the choice of ranking multiple criteria decision making methods. Pesquisa Operacional , 36: 259-277. https://doi.org/ 10.1590/0101-7438.2016.036.02.0259
  » https://doi.org/ 10.1590/0101-7438.2016.036.02.0259
• ³
  CRISPIM J, FERNANDES J & REGO N. 2020. Customized risk assessment in military shipbuilding. Reliability Engineering and System Safety. https://doi.org/10.1016/j.ress. 2020.106809
  » https://doi.org/10.1016/j.ress. 2020.106809
• ⁴
  DAJANI JS, SINCOFF MZ & TALLEY WK. 1979. Stability and Agreement Criteria for the Termination of Delphi Studies. Technological Forecasting & Social Change, 13: 83-90.
• ⁵
  DALKEY N, & HELMER O. 1963. An Experimental Application of the Delphi Method to the Use of Experts. Management Science: 458-467.
• ⁶
  DE SOUZA TTM, & KROHLING RA. 2012. Fuzzy TODIM for group decision making. Congreso Latino-Iberoamericano de Investigación Operativa.
• ⁷
  DINÇER H, & YÜKSEL S. 2019. An integrated stochastic fuzzy MCDM approach to the balanced scorecard-based service evaluation. Mathematics and Computers in Simulation, 166: 93-112. https://doi.org/10.1016/j.matcom.2019.04.008
  » https://doi.org/10.1016/j.matcom.2019.04.008
• ⁸
  FIGUEIRA J, GRECO S, & EHRGOTT M (EDS.). 2005. Multiple Criteria Decision Analysis State of the Art Surveys. Springer.
• ⁹
  FREUND JE, MILLER IR, & JOHNSON R. 1996. Probabilidad y estadística para ingenieros. Prentice-Hall.
• ¹⁰
  HILLIER FS, & LIEBERMAN GJ. 2010. Introduction to Operations Research (Ninth). McGraw-Hill.
• ¹¹
  JUSTO MENDES JA, ASFORA FREJ E, TEIXEIRA DE ALMEIDA A, & ARAÚJO DE ALMEIDA J. 2020. Evaluation of flexible and interactive tradeoff method based on numerical simulation experiments. Pesquisa Operacional , 40: e231191. https://doi.org/10. 1590/0101-7438.2020.040.00231191
  » https://doi.org/10. 1590/0101-7438.2020.040.00231191
• ¹²
  KABASHKIN I. 2016. Heuristic Based Decision Support System for Choice of Alternative Routes in the Large-Scale Transportation Transit System on the Base of Petri Net Model. Procedia Engineering, 134: 359-364. https://doi.org/10.1016/j.proeng.2016.01. 020
  » https://doi.org/10.1016/j.proeng.2016.01. 020
• ¹³
  KAMNITUI N, GENEST C, JAWORSKI P, & TRUTSCHNIG W. 2019. On the size of the class of bivariate extreme-value copulas with a fixed value of Spearman s rho or Kendall s tau. Journal of Mathematical Analysis and Applications, 472(1): 920-936.
• ¹⁴
  KANNIYAPAN G, JAWAHAR L, SARRAZIN I, SAY T, & PONNIAH V. 2019. Selection criteria of building material for optimising maintainability. Construction and Building Materials, 221: 651-660.
• ¹⁵
  KUMAR A, SAH B, SINGH AR, DENG Y, HE X, KUMAR P, & BANSAL RC. 2017. A review of multi criteria decision making ( MCDM ) towards sustainable renewable energy development. Renewable and Sustainable Energy Reviews, 69(October 2016): 596-609. https://doi.org/10.1016/j.rser.2016.11.191
  » https://doi.org/10.1016/j.rser.2016.11.191
• ¹⁶
  LE PIRA M, INTURRI G, IGNACCOLO M, & PLUCHINO A. 2017. Modelling consensus building in Delphi practices for participated transport planning. Transportation Research Procedia, 25: 3725-3735. https://doi.org/10.1016/j.trpro.2017.05.226
  » https://doi.org/10.1016/j.trpro.2017.05.226
• ¹⁷
  LE PIRA M, INTURRI G, IGNACCOLO M, PLUCHINO A, & RAPISARDA A. 2017. Finding shared decisions in stakeholder networks: An agent-based approach. Physica A, 466: 277-287. https://doi.org/10.1016/j.physa.2016.09.015
  » https://doi.org/10.1016/j.physa.2016.09.015
• ¹⁸
  LIU J, ZHAO H, LI Z, & LIU S. 2017. Decision process in MCDM with large number of criteria and heterogeneous risk preferences. Operations Research Perspectives, 4: 106-112. https://doi.org/10.1016/j.orp.2017.07.001
  » https://doi.org/10.1016/j.orp.2017.07.001
• ¹⁹
  MACLENNAN S, KIRKHAM J, LAM TBL, & WILLIAMSON PR. 2018. A randomized trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement. Journal of Clinical Epidemiology, 93: 1-8. https://doi.org/10.1016/j.jclinepi.2017.09.024
  » https://doi.org/10.1016/j.jclinepi.2017.09.024
• ²⁰
  MANNINA G, FERREIRA T, COSENZA A, SÀNCHEZ-MARRÈ M, & GIBERT K. 2019. Decision support systems ( DSS ) for wastewater treatment plants - A review of the state of the art. Bioresource Technology, 290(May): 121814. https://doi.org/10.1016/j.biortech. 2019.121814
  » https://doi.org/10.1016/j.biortech. 2019.121814
• ²¹
  MAO G. 2017. Testing independence in high dimensions using Kendall’s tau. Computational Statistics & Data Analysis, 117: 1-10. https://doi.org/10.1016/j.csda.2017.07. 012
  » https://doi.org/10.1016/j.csda.2017.07. 012
• ²²
  PARAMITA P, MAHAPATRA SS, MOHANTY A, & STHITAPRAGYAN . 2018. A novel multi-attribute decision making approach for selection of appropriate product conforming ergonomic considerations. Operations Research Perspectives , 5: 82-93. https://doi.org/10. 1016/j.orp.2018.01.004
  » https://doi.org/10. 1016/j.orp.2018.01.004
• ²³
  PERGHER I, & DE ALMEIDA AT. 2018. Determining production and inventory parameters: An integrated simulation and mavt approach with tradeoff elicitation. Pesquisa Operacional , 38(1): 87-97. https://doi.org/10.1590/0101-7438.2018.038.01.0087
  » https://doi.org/10.1590/0101-7438.2018.038.01.0087
• ²⁴
  PUTH M, & NEUH M. 2015. Effective use of Spearman s and Kendall s correlation coef fi cients for association between two measured traits. 102: 3-10. https://doi.org/10.1016/ j.anbehav.2015.01.010
  » https://doi.org/10.1016/ j.anbehav.2015.01.010
• ²⁵
  ROSSMANN B, CANZANIELLO A, GRACHT HVD, & HARTMANN E. 2017. Technological Forecasting & Social Change The future and social impact of Big Data Analytics in Supply Chain Management: Results from a Delphi study. Technological Forecasting & Social Change , October: 0-1. https://doi.org/10.1016/j.techfore.2017.10.005
  » https://doi.org/10.1016/j.techfore.2017.10.005
• ²⁶
  SAATY TL. 1990. How to make a decision: The Analytic Hierarchy Process. 48.
• ²⁷
  VITORINO A, MONTENEGRO S, FACELI K, & CASADEI V. 2017. Technologies and decision support systems to aid solid-waste management: a systematic review. Waste Management, 59: 567-584. https://doi.org/10.1016/j.wasman.2016.10.045
  » https://doi.org/10.1016/j.wasman.2016.10.045
• ²⁸
  VON DER GRACHT HA. 2012. Consensus measurement in Delphi studies Review and implications for future quality assurance. Technological Forecasting & Social Change , 79(8): 1525-1536. https://doi.org/10.1016/j.techfore.2012.04.013
  » https://doi.org/10.1016/j.techfore.2012.04.013
• ²⁹
  ZANAZZI JL & MONTEIRO GOMES LFA. 2009. La búsqueda de acuerdos en equipos de trabajo: El método decisión con reducción de la variabilidad (DRV). Pesquisa Operacional , 29(1): 195-221.
• ³⁰
  ZANAZZI JL, MONTEIRO GOMES LFA, & DIMITROFF M. 2012. Toma de decisiones en grupo aplicada a sistemas de mantenimiento preventivo. Congreso Latino-Iberoamericano de Investigación Operativa.

Publication Dates

• Publication in this collection
  07 Dec 2020
• Date of issue
  2020

History

• Received
  19 Mar 2019
• Accepted
  16 Oct 2020