A HYBRID MULTICRITERIA METHODOLOGY TOPSIS-MACBETH-2N APPLIED IN THE ORDERING OF TECHNOLOGY TRANSFER OFFICES

ABSTRACT

This paper aims to analyse the ordering of Technology Transfer Offices’ (TTO) positions regarding innovation indicators using multicriteria analysis as a tool. The purpose is to identify and analyse the TTOs’ strengths and weaknesses in a specific higher education institution in order to understand their performance via ranking, for obtaining revenues from intellectual property assets possibilities developed by them. The use of a methodology for the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), using two normalizing methods, was preceded by a treatment of this qualitative data with the Measuring Attractiveness by a Category Based Evaluation Technique (MACBETH). It is based on an applied questionnaire on TTO’s for recognizing the status of the technology transfer condition to developing nations, focusing on an approach to the institutionalization of TTO’s within a national context. The results of these technical applications bring together novel approaches to the processing of data using both methods to support multicultural decisions (using qualitative and quantitative data). These are initially arranged in different ways and present better analytical conditions in respect to the presented technological development. The first combination between TOPSIS and MACBETHS, with two normalized methods, and associated with a Pearson analysis between the criteria.

Keywords:
multicriteria decision analysis; decision making/process; MACBETH; TOPSIS; technology transfer offices

1 INTRODUCTION

Social innovation involves the shedding of companies that turn to developing innovative products and services which generate revenue in and out of its economic sector. With this appears a new kind of company, informally called spin-offs. These companies diffuse innovative products and services to the population in order to solve social unrest, such as assistive technology oriented to support the empowerment of senior citizens. There is a social technology that is largely operated by non-governmental organizations (NGOs) or other companies that coordinate across business sector exchanges.

In this regard, observing innovation and how it is institutionalized becomes believable to the understanding of how technological development is spread so that the entire society benefit. Within the scope of Higher Education Institutions (in Portuguese IES - Instituição de Ensino Superior) and the creation and regulation of Technology Transfer Offices (TTOs).

Brazilian Higher Education Institutions (IES) need scientific basis for obtaining revenues, making laboratories benchmark nucleus for developing innovative technologies (as products or services in any economic sector) profitable for sustaining institutions, without government grant intensively (considering Brazilian educational policies have low government budget).

The decision analysis process usually considers a variety of alternatives, which must be carefully evaluated so that the “best” decision can be chosen. The aim of decision analysis is to provide logical methods to improve the decision-making process of individuals and companies that are focusing on the development of models designed for decision making, under conditions of uncertainty and that could consider multiple decision criteria, for supporting outranking decisions when there is subjectivity (either qualitative perspective) for analysing criteria. As decisions are dynamic, decision-makers (DM) must be convinced that the analysis process is conducted properly and thoroughly, in order to enable the DM to estimate the potential outcome of their decision. This justifies the swift growth in demand for approaches based Multicriteria Decision Aid (MCDA) methods, and explains the rise in adoption of models to support decisions that occurs in situations of uncertainty or even risk (^{Kadzinski & Tervonen, 2013}; ^{Climaco et al., 2009}). Some authors prefer the name multiple criteria decision aiding while others use the name multiple criteria decision analysis (^{Pujadas et al., 2017}).

The purpose of this paper is to identify and analyse the strengths and weaknesses of TTOs in a specific higher education institution in order to understand their performance for promoting investments in TTOs with weak profile and also obtaining a better revenue from strong TTOs. This paper intends to address the following research question: “What are the potentialities and weaknesses of these TTOs and how can they improve, in regards to the higher education institution to which they belong?”; “Which TTO’s have strong potential for developing innovative researches to became products and services, considering their infrastructure, human resources qualification and scientifically productivity?”; “Can MCDA combination help to understand better the IES how to support these TTO’s through a questionnaire results, considering subjectivity from those qualitative questions?”

In this way, the aim of this study is to analyze - in a multicriteria method - results obtained from a TTOs’ questionnaire containing both qualitative and quantitative closed questions, and thus, observe the grip on the multicriteria method TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) after the transformation of qualitative responses into quantitative using MACBETH (Measuring Attractiveness by a Category Based Evaluation Technique) method as its base reference.

In this paper, the authors propose a “new TOPSIS” using two normalization methods, in order make sensibly analyse the results (TOPSIS-MACBETH-2N). They are present in equation 9 and 10. This TOPSIS-MACBETH-2N hybrid method permits a sensitive analysis of alternatives; comparing these two TOPSIS’ rankings with MACBETH. ^{Behzadian et al. (2012}) review categorized 266 scholarly papers from 103 journals since the year 2000, and there is no hybrid methodology with MACBETH and TOPSIS. The authors did other research, and did not find this combination.

For understanding if there are considerable differences among the MCDA combined methods for supporting the research or not; for understanding TTO’s that can be better substituted for improving revenues for Higher Educational Institutions through new patents, copyrights and trademarks by innovative products and services (Table 4); this paper applied these tools in a different way, observing better TTO’s profiles for investments, using those criteria.

To answer this research question, a survey of 145 TTOs was conducted using the database of an institutional questionnaire applied for this higher education institution. It is based on an applied questionnaire given to TTOs for recognizing the status of the technology transfer condition, focusing on an approach to the institutionalization of TTOs within a regional context.

The remainder of this paper comprises five sections: section 1 introductory, section 2 presents the innovation context being institutionalized in the IES, the section 3 describes the methodology (TOPSIS-MACBETH-2N) applied, section 4 observes the perspectives of the tools used in the qualitative and quantitative context, and the results; and section 5 is devoted to the final considerations of the results.

2 THEORETICAL FOUNDATIONS

This section presents

• Innovation Concepts;

• Considerations about Technology Transfer Offices in Brazil;

• Some considerations about Multicriteria;

• Pearson Correlation;

• MACBETH and TOPSIS;

2.1 Innovation

Institutions are turning to the production of intellectual assets and the copyrights, patents and trademark registrations, in addition to the production of exploratory scientific papers vis-a` -vis the innovations practiced.

The innovation process needs different sources: companies, universities, laboratories, research and development institutes, among others. Technological innovation in companies is a key factor in the maintenance of its activities and for the growth and development of the country (^{Pereira et al., 2015}). ^{O’Kane (2016, paper in press)} also understands that although functionally separate, there is a high level of interdependence between the established role identity of scientists and the emerging role identity of TTO executives. This is because scientists require TTO assistance to realize the commercialization ambitions of the research, as obtaining registration of the intellectual property is what increases the IES’ visibility, incomes and revenues as well; for investing in newer researches, infrastructure for laboratories and other institutional demands.

Technological innovation in companies is a key factor in the maintenance of its activities and for the growth and development of the country (^{Pereira et al., 2015}). ^{O’Kane (2016, paper in press)} also understands that although functionally separate, there is a high level of interdependence between the established role identity of scientists and the emerging role identity of TTO executives. This is because scientists require TTO assistance to realize the commercialization ambitions of the research, as obtaining registration of the intellectual property is what increases the IES’ visibility, incomes and revenues as well; for investing in newer researches, infrastructure for laboratories and other institutional demands.

Innovation concepts observed extensively until the mid-1970s were the epistemological construction for mass production could increase while keeping sizeable revenue amounts.

However, with a quality management trend on production, discussion arose of bringing the conflicting issues of the production chain to academia, thus keeping high rates of economic development without environmental commitment (^{Martins et al., 2015}). By bringing this vision to academia, business and society began to gain knowledge of the development of research that met the issue of constant social demands with the scientific perception of innovation registered as a common good.

Innovation brings increased competitiveness in the country: disruptions to innovative or provocative factors lead to economic development with added value in the production chain of the various economic sectors.

Innovation can be seen as market experiments seeking broad and extensive changes, which fundamentally structure industries and markets. Thus, this is the fundamental basis of capitalism; hence, this is a production system that needs constant renewal and reinvigoration of its consumer goods and capital. (^{Pereira et al., 2015}).

Because of this new role assigned to universities, issues of intellectual property and technology transfer began to occupy policy maker’s discussion agendas in universities. This was particularly true in public universities, where innovation culture development and a greater research results protecting importance, awareness resulted in the economic benefits sharing with academic researchers (^{Ritter dos Santos et al., 2013}).

2.2 TTOs in Brazilian context

The concept and the identification for something new, aims to analyse and document the adaptations and innovations (even if they are benchmarking), in order that the best practices in innovative production processes raise the national intellectual property (^{Cornell University et al., 2015}).

The enforcement of Innovation Brazilian ^{Law number 10.973 in 2004} became one of the most important milestones in technological innovation promotion in Brazil, because it realigned universities with the knowledge economy upon sharing their knowledge with society by promoting inventions and intellectual property. Apart from the volume of patents originating from universities and research institutes have increased significantly over the last few years, it is important to observe what it is happening inside the IES to understand their performance (^{Ritter dos Santos et al, 2013}).

Considering the TTOs’ objective is to promote innovation and technology transferred from universities to the society in general, this particular IES analysed opportunities to institutionalize TTOs in their campi for opening investments to this institution in its several centres. These TTOs were created to promote registries of intellectual properties regarding the academia innovation, as well the shared innovation with companies and society in general, which searched the institution for supporting their innovative ideas.

Hence, these TTOs produce innovative goods and services can support the launch of the startups inside this IES. Relatively speaking, the increasing number of startups and business incubators are small; due to these circumstances inside this IES, less than 20 startups were registered.

2.3 Multiple criteria decision - concepts

Real-world decision problems are rarely mono-criterion based. They generally incorporate a variety of criteria, often contradictory. In many practical situations, alternatives must be ranked given multiple, conflicting criteria of preference. Such intervention in the decision-making process might occur; and the set of criteria can comprise quantitative as well as qualitative criteria (^{Gomes et al, 1997}). The methods are needed for supporting the evolution of the process.

Decision-making is sometimes a complex and difficult task, especially when it involves finding best alternative, an option, a choice, either an action, e.g., or ranking all alternatives in the presence of multiple, usually conflicting criteria (^{Kuo, 2017}). The vision of providing decision support through the development of knowledge of the context even before the MCDA exists as an established area of knowledge (^{Ensslin et al., 2010}).

^{Spak (2012}) observes multicriteria methods in real problems where a predetermined number of alternatives - in the majority of time with uncertain preferences - with divergences of opinions (inside a group of DM) are utilized.

Multicriteria decision making method plays an important role to select the nondominate one(s) among a number of feasible alternatives evaluated according to multiple criteria in real-life decision-making problems, including uncertainty (^{Sahin, 2016}; ^{Mishra, Chatterjee, 2017}).

Research devoted to such problems is most often referred to as multicriteria decision making or multiple criteria decision making (MCDM). MCDM is a usual human activity which helps making decisions mainly in terms of choosing, ranking or sorting alternatives. (^{Zhang et al, 2017}). A number of multi-criteria methodologies have been developed over the elapsing of the time with the aim of providing a systematic framework that considers the multidimensional nature of the real-world problem (^{Pujadas et al., 2017}).

The family of criteria is supposed to satisfy the following consistency conditions (^{Greco, Mousseau & Słowinski, 2008}):

• (a) Exhaustively - any two alternatives having the same evaluations on all criteria from G should be considered indifferent.

• (b) Cohesion - imposing correct distinctions of those criteria that are maximizing and thus minimizing (Naciff de Andrade et al., 2016).

• (c) Monotonicity - when comparing two alternatives, an improvement of one of them on at least one criterion should not deteriorate its comparison to the other alternative.

• (d) Non-redundancy - excluding criteria whose valued characteristics are just done by other criteria.

Regarding decisors’ profile; the DM ranks alternatives (AI) described by the criteria (Cj), with a weight criterion (W) and grades (classification) that are given for each alternative, in each criterion. Assuming the knowledge of the preferences of the decision-makers and the quality of the evaluation, it can be admitted that one action is as good, better or worse than another, that is, to rank the alternatives (^{Cardoso et al., 2009}).

In the multiple criteria sorting problem, alternatives need to be assigned to one or more classes, based on their evaluations on multiple and potentially conflicting criteria. (^{Kadzinski et al, 2014}). In multiple criteria ranking problems alternatives are compared pairwise, and the results express preferences with the use of comparative notions (^{Kadzinski et al, 2012}).

Multicriteria methods are generally classified in two groups: compensatory methods and noncompensatory methods (when strength in one of the criteria doesn’t compensate weaknesses in others) (^{Yunna et al., 2016}). The first group, assumes the decision-maker is able to explain his preferences rationally; there is a global preference function which aims to synthesize the multiple criteria into a single criterion. Non-compensatory methods adopt different procedures and do not consider a single synthesis criterion (^{Lopes & Almeida, 2013}). Non-compensatory MCDA methods are largely based on the (binary relation) outranking relation. MCDA methods deal with three major types of problem: choice, ranking and sorting (^{Greco & Słowinski, 2016}).

The selection of a multicriteria method depends on the decision-maker’s preferences being aggregated, and not being restricted to a single criterion, namely, reducing everything to just one measure. In many cases, the decision-maker has great difficulty in dealing with the tradeoffs of a problem (^{Lopes & Almeida, 2013}).

Some authors use MCDA for non-compensatory methods (outranking methods) and MCDM to compensatory methods (unique criterion of synthesis methods) (^{Govindan & Jepsen, 2016}). MCDA can also be defined as a set of techniques which are designed to search for a number of alternatives within multiple criteria and conflicting objectives (^{Gomes et al., 2017}).

MCDA is a general term for methods, providing a quantitative approach to support decision making in problems involving several criteria and choices (alternatives or actions) (^{Botti & Peypoch, 2013}; Bailey et al., 2014).

Nowadays, MCDA is facing three important methodological challenges: handling a complex structure of criteria, dealing with interactions between criteria and reducing the cognitive effort of the DMs in interaction with MCDA methods. These challenges are usually handled separately; however, they often concern the same decision problem (^{Angilella et al., 2016}).

MCDM/A is a field of study that originated in Operations Research (OR) and focuses its efforts on solving multiple criteria decision problems (^{Zaka & Kruszynskib, 2015}). These problems are complex decision situations in which several, often contradictory, points of view must be considered. The MCDM/A methodology clearly identifies the major participants of the decision making/aiding process such as: the DM, the analyst and the interveners (stakeholders) and their roles in this process (^{Kadzinski et al., 2016}).

Sometimes the mutual preference independence in a criteria set can be violated. For example, an excellent (comfortable) car with high speed has a price which is associated with comfort and high speed (^{Corrente et al., 2016}). ^{Angilella, Greco and Matarazzo (2010}) consider another point, about mutual preference independence: the comprehensive importance of criteria is greater than the sum of the importance of the two criteria or more considered separately. They propose that same decision problem, namely that very often there is positive interaction (synergy) of criteria instead of mutual preference independence problem.

DM commonly uses linguistic variables to evaluate the importance weights of criteria and the ratings of alternatives with respect to various criteria. The concept of linguistic variables is useful in the case of complex or ill-defined situations (^{Chen, 2015}).

As decisions are dynamic, the DM must be convinced that the analysis process is conducted properly and thoroughly, in order to enable the DM to estimate the potential outcome of their decision. This justifies the swift growth in demand for approaches based MCDA methods, and explains the rising in adoption of models to support decision that occurs in situations of uncertainty or even risk (^{Gomes et al., 2017}).

2.4 Pearson’s correlation

Pearson’s correlation coefficient when applied to a population is commonly represented by R(x, y) and may be referred to as the population correlation coefficient or the population Pearson correlation coefficient; verifying statistical independence between variables. The formula 1 (where x and y are variables) represents Pearson’s correlation:

$R(x,y)=\frac{\sum _{i=1}^n\left(x_i-X\right)\left(y_i-Y\right)}{\sqrt{\sum _{i=1}^n{\left(x_i-\overline{X}\right)}^2}\sqrt{\sum _{i=1}^n{\left(y_i-\overline{Y}\right)}^2}},$ (1)

where

$\overline{X}=\frac{\sum _{i=1}^n{x}_i}{n}$ (2)

$\overline{Y}=\frac{\sum _{i=1}^n{y}_i}{n},$ (3)

and −1 ≤ R(x, y) ≤ 1.

2.5 MACBETH

MACBETH helps to evaluate options by making a qualitative representation in attractiveness preferences terms. The method helps to make a quantitative scale through attractiveness qualitative information. Thus, it is easy to explain the same to the decision-makers, as well as preferences explanation (^{Roubens et al., 2006}). The Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE) was also considered for conducting this research, considering the sensibility of using a set of weightings (^{Leoneti, 2016}). However, as PROMETHEE has a quantitative value for being analysed before the judgments for partial rankings; this particularity was against the data collected from the questionnaire because it was necessary to build a scale for answer options, in those questions where only it was possible to answer the professor or student profiles. Such uniqueness also happened at ^{Gomes et al. (2008}) when farmers had a difficult judgment for giving a quantitative opinion among intra and inter criteria.

MACBETH method is a method of semantic judgment. With it, value functions are obtained through semantic judgments made by comparing the attractiveness difference between any two lines of action, always in pairs; after elaborating the judgments on the differences of attractiveness and constructed a cardinal scale of value on the set of alternatives. Then, any inconsistencies are verified and, subsequently, a cardinal value scale is determined that represents the value judgments of the decision maker. The scale obtained and normalized, providing the values of the weights for the alternatives under evaluation, which allows the use of an aggregation model, in general additive.

MACBETH is a support decision method which allows evaluating options considering multiple criteria, using qualitative judgments over attractive differences for weighing up criteria and evaluate options (^{Bana e Costa et al., 2012}).

MACBETH is an interactive approach that uses semantic judgments about the differences in several stimuli attractiveness to help a decision maker quantify the relative attractiveness of each. The proposed method is designed for ranking problem. Thus, what distinguishes MACBETH from the other multicriteria models is that it needs only qualitative judgments about the difference of attractiveness regarding elements at a time, in order to generate numerical scores for the options in each criterion and to weigh the criteria using - according to the method - the base-scale (^{Dhouib, 2014}).

Otherwise, the advantage of an ordinal preference scale transformation to a cardinal scale allows differing the distance among the alternatives: identifying the weak, very weak, strong, very strong etc. The DM compares options for informing the indifference or defining the correlation among the alternative judgments. This is a MACBETH’s great advantage regarding other multiple criteria support decision methods. From a technical point of view, MACBETH enables the construction of a multicriteria additive value model whose benefits of actions are represented by their overall scores computed by adding the products of the scores of the actions to the benefit dimensions by the respective weights (^{Bana e Costa et al., 2012}).

Scales of value are common in several complex problem areas involving multiple criteria identified when presenting conflicting criteria among them which are not understandable or subjective. Conflicting criteria could also be not quantifiable due to not needing the application of a scale of value, that is to say, to attribute value to such criteria as a way of approaching them in the decision process. By using these methods, there are great interactions between the qualitative and quantitative data for making decisions (^{Spak, 2012}).

Per Naciff de Andrade et al. (2016), in order to be possible to ordinally measure the attractiveness of options “x” from a finite set “X” - it is necessary to associate a real number v(x) to each of “x” that satisfies the conditions of: stricter preference (Equation 4) and indifference (Equation 5).

$x,y\in X:xPy\iff v\left(x\right)>v\left(y\right)$ (4)

$x,y\in X:xIy\iff v\left(x\right)=v\left(y\right)$ (5)

Where: P - preference and I - indifference.

w, x, y, z ∈ X with x more attractive than y and w more attractive than z: so, the quotient (v(x) − v(y))/(v(w) − v(z)) measures the attractiveness difference between x and y when the attractiveness difference between w and z is the unit of measure.

Respecting such conditions, a numeric scale of intervals is established (v: X → R: x → v(x)). The transition of ordinal information at MACBETH to cardinal is done by comparing options in pairs in a qualitative way. The M-MACBETH software deploys the model, as far as the evaluations are inserted at the trials matrix automatically observing the consistency. If there is any inconsistence, possible solutions are shown.

2.6 TOPSIS

One of the most widely used multi-criteria decision analysis methods is the TOPSIS method, which was proposed by Hwang and Yoon in 1981, and extended by Yoon in 1987, as well as by Lai Hwang, Jou Lai, and Yun Liu in 1993. In the TOPSIS method, the optimal alternative is nearest to the positive ideal solution (PIS) and farthest from the negative ideal solution (NIS), measuring the distance to PIS and to the NIS (^{Walczak & Rutkowska, 2017}; ^{Roszkowska & Wachowicz, 2015}). TOPSIS performance attracted attention in terms of accuracy, and TOPSIS had good performance in descriptive analysis (^{Leoneti, 2016}).

The multicriteria support for decision making consists of a group of techniques for supporting a DM to make decisions about a complex problem, evaluating and choosing alternatives for solving it, following different criteria and points of view (^{Gomes et al., 2017}).

This technique allows the decision to be regulated on a weighted criteria basis relevant to the problem to be answered by the DM agents. Here the importance of the criteria is defined by them, in an interactive process with other technical-political actors (^{Sarrazin & de Smet, 2015}).

Regarding the MCDA in Brazil, ^{Longaray et al. (2015}) observe within the multicriteria methods most applied from 2004 to 2013 in scientifically studies, bibliometric researches showed the difficulty of identifying in the literature the use of MCDA. Because there are more than ten thousand possibilities of papers to be analysed inside the Brazilian scientifically production and on his (Longaray’s) research; he restricted it to observing only the most cited authors.

Conversely, TOPSIS enhances by being more intuitive and simple for the use of mathematical procedures that contribute to the ease of implementation and application, allowing it to evaluate a non-limited quantity of alternatives and differ from the comparative approaches (^{Junior & Carpinetti, 2015}; ^{Kluczek & Gladysz, 2015}).

One of the most widely used multi-criteria decision analysis methods are the TOPSIS method, which was proposed by Hwang and Yoon in 1981, and extended by Yoon in 1987, as well as by Lai Hwang, Jou Lai, and Yun Liu in 1993. (^{Walczak & Rutkowska, 2017}; ^{Roszkowska & Wachowicz, 2015}).

The PIS is the one maximizes the most “advantageous” classification each criterion, while the NIS is the one minimizes the classification each criterion. According to ^{Bilbao-Terol et al. (2014)}, the PIS is made up of all the best possible values based on the benefit criteria, and the NIS consists of all the worst possible values of the cost criteria in order to classify a finite set of alternatives. TOPSIS, therefore, is a method that results in more balance in the evaluation, placing the alternatives in relation to two points of reference: the PIS or ideal point (the method minimizes the distance), and the NIS or the anti-ideal point (method maximizes the distance) (^{Walczak & Rutkowska, 2017}).

According to ^{Kuo (2017}), TOPSIS has been widely applied in the past decades, considerable efforts have been made to modify, either extend TOPSIS, and even though to compare, or hybridize it with other MCDM methods. TOPSIS stands out as an intuitive method, characterized by the simplicity of its mathematical procedures, ease of deployment, and because it permits the evaluation of an unlimited quantity of alternatives, unlike methods that use comparative approaches.

According to TOPSIS, the best alternative would be which is next to the positive ideal solution and the most distant to the negative ideal solution (also known as anti-ideal). The positive ideal solution is that which maximizes the criteria of benefits and minimizes the criteria of costs; in other words, the negative ideal solution maximizes the criteria of costs and minimizes criteria of benefits. The positive ideal solution is composed of all the best values achievable from the benefits criteria and the negative ideal solution consists of all the worst values achievable from cost criteria, in order to rank a finite set of alternatives e.g. (^{Bhutia & Phipon, 2012}; ^{Bilbao-Terol et al., 2014}).

Since using only MACBETH, the DM is already able to rank the alternatives. However, the use of TOPSIS allows the DM to locate the alternatives between the best and the worst. And the use of two normalizations (formulae present in equation 9 and 10) allowed a sensitivity analysis.

^{Lima-Junior and Carpinetti (2016)} propose the use of fuzzy set; however, for this problem, it was proposed the use of MACBETH to transform a semantic scale into numerical scale. It’s only 4 questions. Where each question has a maximum of five options. Faced with this, this modelling is theoretically simple. It is easier for the decision maker to understand, and more usual for continued applicability at the questionnaire, serving for this paper’s purpose.

The Euclidean calculated distances between A _i and A ⁺ (benefits) and between A _i and A ⁻ (costs) is realized by the equations (6) and (7):

$D_i^{+}=\sqrt{\sum _{j=1}^n{\left(p_{ij}-p_j^{+}\right)}^2}$ (6)

$D_i^{-}=\sqrt{\sum _{j=1}^n{\left(p_{ij}-p_j^{-}\right)}^2}$ (7)

while the relative proximity calculations Ci for each alternative Ai in relation to positive ideal solution A+ is done by the equation (8):

$C_i=\frac{D_i^{-}}{D_i^{+}-D_i^{-}}$ (8)

where i = 1, ..., m and the index value C _i ranges between 0 and 1.

First Normalization method (equation 9):

$y_{ij}=\frac{x_{ij}}{\sqrt{\sum _{i=1}^n{x}^{2}ij}},i=1,2,\dots ,m;j=1,2,\dots ,n$ (9)

Second Normalization Method (equation 10):

$y_{ij}=\frac{x_{ij}}{max{X}_i}$ (10)

3 MACBETH-TOPSIS-2N APPROACH

Different approaches are proposed from various schools of thinking. There are nor better neither worse techniques; only techniques that fit better to a certain situation. It is not easy to say which MCDM approach is more reasonable and reliable for a given decision-making problem, as the selection of MCDM methods itself is a complicated MCDM process. Many DMs apply several MCDM approaches to the same problem, compare their results, and then make the final decision. This approach is difficult to comprehend and complex to implement because it requires extensive technical knowledge in MCDM fields. A combination of different MCDM evaluation techniques to construct a hybrid model may be the correct choice in solving this problem (^{Wang et al., 2016}).

The reasons for integrating two methods together are:

• MACBETH is used to treat qualitative data (in the questionnaire).

• TOPSIS uses the criteria to order the alternatives (including the qualitative data previously treated by MACBETH).

• MACBETH and TOPSIS use different methods to normalize; the aggregation uses the two normalized methods (allowing different arrays associated with each normalization method to rise, thus allowing a sensitivity analysis of the results).

• TOPSIS e MACBETH are based on additive aggregation of criteria.

3.1 The questionnaire

This research purpose is to understand the innovation evolution as an opportunity for promoting revenue to invest at IES infrastructure; considering TTOs’ profiles by their strength and weakness. Those profiles were shown on the questionnaire (Table 1) by the collected answers; restricting this research to TTOs of science, technology and biomedicine areas. The questions were developed to know the TTOs profiles at the beginning of their creation and for understanding in which they are strong and what kind of investment it is necessary to offer for improving royalties obtained from intellectual properties, agreements and partnerships; when questions observed any sort of weakness for ranking those TTOs (^{Martins et al., 2015}).

Table 1
The questionnaire applied to TTOs.

Questions	Codes
Among the level of degree in the offered courses, in which of them is the TTO attached with a higher degree of certification?	q 1
Are there research projects?	q 2
What is the TTO highest professors’ coordinators’ degree?	q 3
Among TTOs collaborators, what is the highest education degree?	q 4
Numbers of praticated activities	q 5
Numbers of published articles/papers	q 6
Numbers of equipment’s obtained	q 7
Numbers of promotions obtained	q 8
Number of published books	q 9
Number of completed collaborations	q 10
Number of elaborated products	q 11
Number of completed social projects	q 12
Number of completed services	q 13
Number of published thesis/essays	q 14

This questionnaire was a new one formulated by a strategic department as IES’ scientifically census for understanding which TTO in science, technology and biomedicine areas would have the infrastructure for offering better innovative products and services patents, trademarks and copyrights to IES obtains revenues. TTO’s coordinators answered with numbers the quantity of assets, human resources, scientifically production that their TTO’s have produced since the very beginning of their existence. These questions were answered via an electronic institutional database with confidentiality among them for any TTO knows the answer of the other (only the strategic department knows the answer of each TTO). This scientifically census result was codified in codes for each TTO (e.g. “A17” means TTO “A” number “17”) in order to decide which of them could be offered investment for their productivity and potential productivity (a decision which could be taken by the Rectory and the Provost which those TTO’s responsibilities belong to).

Questionnaire questions were adapted from a government application form made by the Ministry of Science and Technology answered as a scientifically census annually for all higher educational institutions in Brazil.

The methodology for observing this questionnaire data was decision analysis, because they are done in many ways, regardless of the checked methodology or the mental model applied: all the before structuring of the best choice is built based on the same elements. It is possible to share in three principal elements: the DM, alternatives and criteria (^{Gomes et al., 2017}).

The decision makers were Rectory and Provost; the alternatives were TTOs from science, technology and biomedicine areas and criteria adopted were the questionnaire answers transformed into quantitative analyses in MACBETH combined with TOPSIS for ranking TTOs that could be supported in a short term by IES in order to boost revenues for the institution.

After collecting all the answers, the strategic department observed the collected data behaviour in a statistics perspective, for understanding the data quality produced by the TTOs’ answers in the questionnaire; as an analogous thought in observing decision support management analysed by ^{Chaves et al. (2012}). The first attempt was to understand the question which was better correlated. In this regard, as Figure 1 shows, using the R Language Program through the command pairs.panels (x) where x is the list with all the answers of the 145 TTOs for each one of the 14 questions: as observed the correlations are small and sometimes negative, visualized also in the plots of dispersion. 0.78 was the best correlation between the question number 14 (q ₁4) and the question number 6 (q ₆); which means articles and papers produced by professors and students who work at these TTOs are good outputs from thesis and dissertations related to the researched areas.

Figure 1
The correlation and dispersion of the questions using the command pairs.panels (x) using the R Language Program.

Although one of the correlations is 0.78 between the questions (q ₁₄ and q ₆) in the questionnaire, the Kaiser-Meyer-Olkin factor adequacy was 0.77 showing a good range. The kurtosis was 4.683885 (leptokurtic: higher than the normal distribution) and the Bartlett’s Test had a p-value < 0 (1.138693e-75).

The skewness shows -2.346562 which means the distribution has a left tail (values below average - positive skewness). After considering the complex behaviour of the questionnaire using Multivariate Normality Tests, it was proposed to observe TTOs in the best position of a rank using MCDA, according to the answers obtained from the questionnaire. ^{Han and Shin (2014}) observe that quantitative criteria, methods can be used in a better analysis when there is consistent quantitative data. Thus, the statistic step was concluded from computing MCDA tools results for ranking in a compensatory way the alignment between MACBETH and TOPSIS normalized twice.

It is usually the DM who quantifies and qualifies alternatives and criteria for structuring the decision problem. The DM and the analyst are the same person in most of the cases. Regarding the decision-making analysis; it is recommended to be different individuals, so as to not submit their desires to the decisor and in this way, compromise the results. ^{Chiu et al. (2006}) observe the need for supporting DM because the perception, criteria in their process of evolution may sometimes be vague.

Following the trend of sustainable development where TTOs must aim to promote innovation (^{Elkington, 1999}) regarding the demand from social technology transfer, the questionnaire was produced considering the quality of knowledge from the team that works with TTOs and their outputs (e.g.: articles, projects, books, services, social projects, essays, patent registration). With these perspectives in mind, 14 questions were formulated. Four questions were qualitative questions whose answers were transformed into quantitative via the use of MACBETH in the “base” scale: the transformation using the program named M-MACBETH for changing the ordinal scale into cardinal scale (questions: q ₁, q ₂, q ₃ and q ₄ - Table 1 and Table 2).

Table 2
The qualitative answers transformed at M-MACBETH base-scale.

Questions coded	Options of answer	Codification of the answers at M-MACBETH	M-MACBETH “de base” scale
q 1	Doctor	D	17
	Master	M	14
	Specialist	E	11
	Undergraduate	G	7
	Extension	Ext	0
q 2	Yes, with student’s participation	S	8
	Yes, without student’s participation	WS	4
	No, there aren’t.	N	0
q 3	Doctor	D	12
	Master	M	8
	Specialist	E	4
	Undergraduate	G	0
q 4	Doctor	D	11
	Master	M	8
	Specialist	E	5
	Undergraduate	G	2
	Intern	Estag	0

The rest of the 10 questions were quantitative and were simply grouped with these transformed 4 questions computed at M-MACBETH software. Table 1 shows the questionnaire applied to TTOs because these 10 questions were cardinals and quantitative.

Issues are treated as criteria. The answers to questions are scored on the criteria. The projects are the alternatives, formed by the scores in the criteria. For this it is necessary to use the multicriteria to order the projects.

3.2 MACBETH’s application

Considering the weights for each answer option at MACBETH “de base” scale, in this Brazilian IES context, it’s important to observe that human resources who work at TTOs without high degree level education (Doctor e.g. of higher degree), receive a lower grade/weight because it means less opportunity for achieving government subsidies in researches, because some scientifically programs launched by Ministry of Science and Technology demands a coordinator and a Doctor research as part of the TTOs’ team, as well as some international universities only admit agreement cooperation signature if the TTO’s responsible is a Doctor coordinator. Infrastructure is other impact factor because if TTO has enough supportive assets for developing products and services, there will be revenue in short term for university being made by tradeable agreement between companies and the university. Conversely, scientifically production shows TTOs’ productivity profile also showing which TTOs could bring to university revenue in a short term considering intellectual properties tradeable with companies and inside the government as well.

3.3 TOPSIS’ application

Considering that the identification of the most innovative and developed TTOs’ searches Multicriteria Decision Aid depends on the alignment of the qualitative to be transformed into quantitative questions, it was decided - after observing several Multicriteria Decision Aids - using MACBETH and TOPSIS. After deciding the aids, the possibility of testing TOPSIS was observed in a different system from just calculating electronic sheets. It was decided to use the programming language R for this purpose.

Programming language ^{R was organized in 2002} as “The R Foundation for Statistical Computing”, abbreviated as “R Foundation” seated in Vienna, Austria and is active worldwide (^{R FOUNDATION, 2002}). It is considered a free software foundation because everyone can add new packages (new contents of calculations programming) and it is possible for a person without any experience with this software to read the manuals of the packages and the help contents for understanding and calculating any desirable output (if this output has not been available yet, it is possible to add this output offering to the software the package with the input formulas).

4 APPLYING THE PROPOSAL AND RESULTS

Founded values will be in MACBETH’s base-scale that can be normalized and represented in percentiles. Using the M-MACBETH base-scale was displayed at Table 2 where the qualitative answers were transformed into quantitative.

Finally, after getting the transformation needed using M-MACBETH, there are conditions to accept cardinal values from the four questions applied at the questionnaire. This is the reason MACBETH was used: for supporting the TOPSIS method used in the entire questionnaire for ranking TTOs.

R was chosen because of the simple way data is input into the system and how it obtains the rank as output, after inserting action commands in its console from the package named MCDM (Table 3).

Table 3
Applying the MCDM package as basis of calculation the TOPSIS via R.

> x < − read.csv (“rstudio2.csv”, header = TRUE, sep = “;”)
> d < − as.matrix(x[,1:14])
> library(topsis)
> w < − c(0.081, 0.090, 0.065, 0.026, 0.048, 0.138, 0.053, 0.081, 0.090, 0.065, 0.026, 0.048, 0.138, 0.053)
> i < − rep(“+”,14)
> output < −topsis(d,w,i)
> output
alt.	row	score	rank	alt.	row	score	rank
1	1	0.23323150	37.0	40	40	0.51270187	2.0
2	2	0.14144521	84.0	41	41	0.12878584	96.0
3	3	0.25030013	30.0	42	42	0.11472777	106.0
4	4	0.13031608	93.5	43	43	0.10003992	118.0
5	5	0.24346407	31.0	44	44	0.23383440	36.0
6	6	0.16835934	63.0	45	45	0.35947005	13.0
7	7	0.08327374	131.0	46	46	0.11665346	103.0
8	8	0.44200323	4.0	47	47	0.15526012	74.0
9	9	0.25060285	29.0	48	48	0.16829886	64.0
10	10	0.36619584	11.0	49	49	0.16377551	68.0
11	11	0.26294366	23.0	50	50	0.18183857	54.0
12	12	0.37987110	10.0	51	51	0.16478733	66.0
13	13	0.12878584	96.0	52	52	0.16415691	67.0
14	14	0.18865359	53.0	53	53	0.25895098	25.0
15	15	0.13599546	88.0	54	54	0.16988663	62.0
16	16	0.49778185	3.0	55	55	0.30532252	15.0
17	17	0.38678157	8.0	56	56	0.06907011	139.0
18	18	0.11591935	104.0	57	57	0.29261896	18.0
19	19	0.28973731	19.0	58	58	0.29797531	16.0
20	20	0.21644781	41.0	59	59	0.08657962	123.5
21	21	0.27117633	21.0	60	60	0.22251728	40.0
22	22	0.21120322	44.0	61	61	0.07741437	134.5
23	23	0.12266571	102.0	62	62	0.13767696	85.0
24	24	0.39713673	5.0	63	63	0.08395204	128.0
25	25	0.13042475	92.0	64	64	0.27027553	22.0
26	26	0.12439997	100.0	65	65	0.17432236	59.0
27	27	0.25808288	27.0	66	66	0.08395204	128.0
28	28	0.38884870	7.0	67	67	0.14905041	77.0
29	29	0.13692193	86.0	68	68	0.13654440	87.0
30	30	0.08715050	122.0	69	69	0.07437783	137.0
31	31	0.07684338	136.0	70	70	0.14283694	78.0
32	32	0.31071386	14.0	71	71	0.09776193	119.0
33	33	0.13031608	93.5	72	72	0.39465251	6.0
34	34	0.17285858	61.0	73	73	0.38402681	9.0
35	35	0.12878584	96.0	74	74	0.11335110	108.0
36	36	0.12567601	99.0	75	75	0.14227038	81.0
37	37	0.36155858	12.0	76	76	0.23659404	35.0
38	38	0.15325858	75.0	77	77	0.16026527	71.0
39	39	0.14226222	82.0	78	78	0.18026568	58.0

Using the R Language Program, it can be seen that the TTO number 98 took the first place in the rank, using the MACBETH-TOPSIS method with entropic weights. The next section analyses the MACBETH-TOPSIS-2 using normalization into two different ways of assessment basis.

4.1 MACBETH-TOPSIS-2N: Second normalized method

The first assessment basis used in this normalized method of MACBETH-TOPSIS-2N was the normalization using just the split between the greater number inside the question column, by the number of the row analysed. For example: in the first question, the greater number was 17 split by the first row, answering precisely 17 and giving as a result, the number 1 - this calculation was done for all the answers in all the rows and columns.

The second assessment basis used in normalized method MACBETH-TOPSIS-2N was the normalization using the sum of all the numbers from a question of a column; for example: the first column question has the sum 2,194 that was split by the number of the first row answered in this column-question - 17 - resulting in 0.007748405, where this calculation was also done for all the answers in all the rows and columns. Table 4 shows a comparison among results used in this paper for ranking TTOs.

Table 4
Comparison among MACBETH-TOPSIS-2N and R.

General TTOs’ rank	Rank by splitting the greater number by the others	Rank by splitting the sum of the number by the others	Rank from R Language - assessment by entropy
1	A98	A9	A98
2	A16	A98	A40
3	A40	A58	A16
4	A24	A40	A8
5	A12	A16	A24
6	A8	A24	A72
7	A28	A73	A28
8	A45	A72	A17
9	A17	A17	A73
10	A53	A12	A12
11	A11	A8	A10
12	A72	A11	A37
13	A19	A28	A45
14	A118	A118	A32
15	A21	A76	A55
16	A22	A130	A58
17	A55	A21	A100
18	A73	A55	A57
19	A32	A45	A19
20	A44	A19	A86
21	A90	A53	A21
22	A57	A60	A64
23	A5	A10	A11
24	A14	A126	A109
25	A113	A37	A53
26	A20	A1	A102
27	A50	A27	A27
28	A109	A36	A113
29	A9	A57	A9
30	A1	A86	A3
31	A34	A100	A5
32	A52	A44	A90
33	A6	A142	A118
34	A119	A97	A108
35	A10	A32	A76
36	A142	A113	A44
37	A48	A82	A1
38	A27	A128	A135
39	A3	A140	A128
40	A130	A64	A60
41	A64	A3	A20
42	A47	A23	A142
43	A60	A90	A138
44	A39	A22	A22
45	A128	A102	A104
46	A67	A108	A126
47	A78	A84	A80
48	A76	A109	A84
49	A58	A133	A133
50	A29	A80	A97
51	A70	A20	A119
52	A108	A5	A140
53	A81	A50	A14
54	A126	A51	A50
55	A104	A34	A81
56	A25	A52	A83
57	A96	A104	A130
58	A4	A127	A78
59	A33	A115	A65
60	A51	A14	A127
61	A80	A81	A34
62	A139	A6	A54
63	A138	A78	A6
64	A133	A87	A48
65	A87	A62	A96
66	A137	A48	A51
67	A135	A96	A52
68	A77	A83	A49
69	A127	A54	A144
70	A115	A119	A106
71	A145	A39	A77
72	A114	A138	A132
73	A102	A143	A111
74	A143	A77	A47
75	A26	A65	A38
76	A74	A135	A87
77	A111	A46	A67
78	A49	A49	A70
79	A54	A120	A122
80	A68	A145	A145
81	A106	A47	A75
82	A121	A122	A39
83	A116	A75	A115
84	A107	A114	A2
85	A65	A107	A62
86	A83	A137	A29
87	A140	A38	A68
88	A38	A18	A15
89	A132	A43	A143
90	A91	A111	A139
91	A86	A67	A137
92	A79	A2	A25
93	A15	A106	A4
94	A129	A29	A33
95	A85	A139	A13
96	A95	A144	A35
97	A71	A70	A41
98	A110	A68	A114
99	A30	A15	A36
100	A2	A129	A26
101	A59	A26	A120
102	A93	A25	A23
103	A13	A4	A46
104	A35	A33	A18
105	A41	A74	A107
106	A37	A132	A42
107	A88	A94	A125
108	A120	A125	A74
109	A97	A13	A88
110	A61	A35	A82
111	A134	A41	A116
112	A46	A116	A91
113	A125	A88	A129
114	A92	A42	A85
115	A62	A79	A95
116	A82	A91	A79
117	A94	A95	A121
118	A112	A85	A43
119	A136	A121	A71
120	A99	A71	A94
121	A23	A89	A110
122	A63	A110	A30
123	A66	A30	A59
124	A101	A59	A93
125	A42	A93	A112
126	A89	A92	A136
127	A144	A61	A63
128	A100	A134	A66
129	A36	A131	A101
130	A43	A112	A89
131	A69	A99	A7
132	A75	A31	A131
133	A122	A136	A124
134	A7	A7	A61
135	A84	A63	A134
136	A141	A66	A31
137	A117	A101	A69
138	A18	A124	A99
139	A131	A56	A56
140	A56	A141	A92
141	A124	A69	A117
142	A123	A117	A123
143	A103	A105	A103
144	A31	A123	A141
145	A105	A103	A105

Different rankings with the same data are noted. It shows that the result is influenced by modeling. Model influences choices suggest possible decisions. In view of this, it was decided to compare them with the purpose of allowing the decision-maker to discern which options would be presented as the highest frequency among the most highly rated.

4.2 TTOs analysis

The domino set of alternatives are {A₁; A₈; A₉; A₁₀; A₁₁; A₁₂; A₁₄₂; A₁₆; A₁₇; A₁₉; A₂₁; A₂₂; A₂₄; A₂₇; A₂₈; A₃; A₃₂; A₄₀; A₄₄; A₄₅; A₅₃; A₅₅; A₅₇}.

Only four criteria need a minimum evaluation:

• (q ₁) - undergraduate;

• (q ₃) - specialist;

• (q ₅) - number 1;

• (q ₇) - number 1.

The Pearson’s correlation obtains (R(x, y)> 0.6) observed at Table 5:

Table 5
Pearson’s analysis.

Question	Question	Pearson’s correlation
Numbers of promotions obtained (q 8)	Numbers of published articles/papers (q 6)	0.64
Numbers of promotions obtained (q 8)	Numbers of equipment obtained (q 7)	0.61
Numbers of promotions obtained (q 8)	Number of published thesis/essays (q 14)	0.68

The set of domain alternatives, obtained by the questionnaire results in MACBETH-TOPSIS-2N showing that:

• (a) When there are coordinators with a doctorate in Brazil, they facilitate the production of books, articles, goods and services without resources. Promoting the desirable products of TTOs for technological innovation to occur through technology transfer.

• (b) To succeed in calls for funding, the minimum required is: (q ₁) undergraduate; (q ₃) specialist; (q ₅) 1; (q ₇) 1. However, this minimum entails a need for a great score in the other criteria. Public calls only with professors with a Doctoral Degree (it is a restrictive item from Brazilian funding agencies).

• (c) Numbers of promotions obtained is the criteria with more synergy with other criteria.

Some of them had more published papers than others that could be a good output; however, without the infrastructure these TTOs will fail without the support of skilled labour. Since happens this profile, they aren’t a good alternative set for achieving any subsidies from university, because although they have a good scientifically production, the lack of infrastructure and the lack of doctorate coordinators could promote a production constraint for improving university’s revenue.

Another observation is the lesser demand from companies and society in the using of the TTOs from IES as an opportunity of innovation dissemination for promoting partnerships.

This is due mainly to the use of the Innovation Law in Brazil, regarding actions to transfer the knowledge produced to society.

This can also be observed because there is not a routine inside this particular IES in promoting these TTOs as a mechanism of introduction of intellectual property and technology transfer management inside the IES for professors conduct partnerships among their TTOs. For example, medical and chemical TTOs do not have partnerships when producing their products, services or software. By working together, their research would be sped up compared to working separately.

Conversely, the use of different ways of calculating MACBETH-TOPSIS-2N has shown that the position may likely change in two places or more from one method of calculating to another. Such process variability occurs when there are big quantities of data for ranking and the numbers do not have the same behaviour for the survey respondents. For example, some TTOs produced 66 published articles/papers while others did not produce any during a whole production year (considering the type of research spending years for being developed). None of the ranks had the same TTOs at the same place according to the assessment basis, but in the majority of the TTOs, they had the same place in two basic assessment processes.

These three rankings of assessment basis showed how a DM can conduct the decision of qualifying investments at TTOs, considering the quantities or the behavior of these quantities of produced works through the production year. Otherwise, productive TTOs might support financially other TTOs not so productive (because the deployment stage at the research) with their royalties, granting these TTOs requesting more support for their researches; provoking in a middle term better conditions for developing researches at this educational institution.

CONCLUSIONS AND FINAL REMARKS

This paper objective was to identify innovative TTOs from a specific IES in Brazil using MACBETH and TOPSIS MCDAs for supporting the transformation of a qualitative questionnaire’s answers and transforming four questions into quantitative cardinal number questions by being treated with the TOPSIS method in order to obtain TTOs’ rank in three different ways of assessment basis.

Observations showed that TTOs in Brazil have few actions to transfer knowledge to society because there are not partnerships in high volume among universities, society and companies, despite the government instituting an Innovation Law in 2004 for promoting such scientific development.

According to these results, it is suggested that TTOs have to increase the degree of their collaborators for giving better opportunities of investments in the production of articles, projects and intangible assets, with less expected bureaucracy from the norm via opening companies and developing products in Brazil.

By using observed available data from TTOs, this paper has endeavoured to identify how to improve the partnerships between those agents of society which seek the best quality in products and services, in terms of new features and processes, via several economic sectors.

There should also be a likely association in future applications between MACBETH and TOPSIS as a hybrid method for supporting multicriteria decisions. This is because of the assessment basis behaviour that did not have the same position at the three ranks for the same TTOs and this is important to DM to understand how to evaluate the best TTOs production or the TTOs need more investment for transferring social technology.

REFERENCES

• ¹
  ANGILELLA S, CORRENTE S, GRECO S & SŁOWINSK R. 2016. Robust Ordinal Regression and Stochastic Multiobjective Acceptability Analysis in multiple criteria hierarchy process for the Choquet integral preference model. Omega, 63: 154-169. DOI: 10.1016/j.omega.2015.10.010
  » https://doi.org/10.1016/j.omega.2015.10.010
• ²
  ANGILELLA S, GRECO S & MATARAZZO B. 2010. Non-additive robust ordinal regression: A multiple criteria decision model based on the Choquet integral. European Journal of Operational Research, 201: 277-288. DOI: 10.1016/j.ejor.2009.02.023
  » https://doi.org/10.1016/j.ejor.2009.02.023
• ³
  BANA E COSTA CA, DE CORTE J-M & VANSNICK J-C. 2012. MACBETH. International Journal of Information Technology and Decision Making, 11(2): 359-387. doi.org/10.1142/ S0219622012400068
  » https://doi.org/10.1142/ S0219622012400068
• ⁴
  BEHZADIAN M, OTAGHSARA SK, YAZDANI M & IGNATIUS J. 2012. A state-of the-art survey of TOPSIS applications. Expert Systems with Applications, 39: 13051-13069.
• ⁵
  BHUTIA PW & PHIPON R. 2012. Application of AHP and TOPSIS method for supplier selection problem. IOSR Journal of Engineering, 2(10): 43-50.
• ⁶
  BILBAO-TEROL A ET AL. 2014. Using TOPSIS for assessing the sustainability of government bond funds. Omega, 49: 1-17.
• ⁷
  BOLY V ET AL. 2014. Evaluating innovative processes in french firms: Methodological proposition for firm innovation capacity evaluation. Research Policy, 43: 608-622.
• ⁸
  BOTTI L & PEYPOCH N. 2013. Multicriteria ELECTRE method and destination competitiveness. Tourism Management Perspectives, 6: 108-113.
• ⁹
  BRASIL. 2004. Lei 10973: Lei de Inovação Tecnológica. Planalto, Casa Civil.
• ¹⁰
  CARDOSO RS, XAVIER LC, GOMES CFS & ADISSI PJ. 2009. Uso de SAD no apoio à decisão na destinação de resíduos plásticos e gestão de materiais. Pesqui. Oper., 29(1): 67-95. DOI: http: //dx.doi.org/10.1590/S0101-74382009000100004
  » http: //dx.doi.org/10.1590/S0101-74382009000100004
• ¹¹
  CHAVES LC, ENSSLIN L & ENSSLIN S. 2012. Mapeamento do tema gestão do apoio à decisão quando analisado sob a ótica de seus resultados. Revista Eletrônica Sistemas & Gestão, 7(3): 336- 348. DOI: 10.7177/sg.2012.v7.n3.a4
  » https://doi.org/10.7177/sg.2012.v7.n3.a4
• ¹²
  CHEN TING-YU. 2015. An interval type-2 fuzzy PROMETHEE method using a likelihood-based outranking comparison approach. Information Fusion, 25: 105-120.
• ¹³
  CHIU Y-C ET AL. 2006. An evaluation model of new product launch strategy. Technovation, 26: 1244-1252.
• ¹⁴
  CLÍMACO JCN, COSTA JA, DIAS LC & MELO P. 2009. Supporting collaborative multicriteria evaluation: the VIP Analysis plug-In for decision deck. International Journal of Decision Support System Technology, 1(4): 1-15.
• ¹⁵
  CORNELL UNIVERSITY, INSEAD & WIPO. 2015. The Global Innovation Index 2015: Effective Innovation Policies for Development. Genebra, Suíça.
• ¹⁶
  CORRENTE S, GRECO S& ISHIZAKA A. 2016. Combining analytical hierarchy process and Choquet integral within non-additive robust ordinal regression. Omega, 61: 2-18.
• ¹⁷
  DHOUIB D. 2014. An extension of MACBETH method for a fuzzy environment to analyze alternatives in reverse logistics for automobile tire wastes. Omega, 42: 25-32.
• ¹⁸
  ELKINGTON J. 1999. Cannibals with forks: the triple bottom line of the 21st century business. Capstone, Mankato.
• ¹⁹
  ENSSLIN L, GIFFHORN E, ENSSLIN SR, PETRI SM & VIANNA WB. 2010. Avaliação do Desempenho de Empresas Terceirizadas com o Uso da Metodologia Multicritério de Apoio a` Decisão - Construtivista. Pesquisa Operacional, 30(1): 125-152.
• ²⁰
  GOMES CFS, COSTA HG & BARROS AP. 2017. Sensibility analysis of MCDA using prospective in Brazilian energy sector. Journal of Modelling in Management. DOI:10.1108/JM2-01-2016-0005
  » https://doi.org/10.1108/JM2-01-2016-0005
• ²¹
  GOMES LFAM, MURY AR & GOMES CFS. 1997. Multicriteria ranking with ordinal data. System Analysis Modelling Simulation, 27(2): 139-145.
• ²²
  GOMES EG, MELLO JCCBS & MANGABEIRA JAC. 2008. Índice multicritério de bem estar social rural em um município da Região Amazônica. Pesquisa Operacional, 28(1): 141-160. DOI: 10.1590/S0101-74382008000100008.
  » https://doi.org/10.1590/S0101-74382008000100008.
• ²³
  GOVINDAN K & JEPSEN MB. 2016. ELECTRE: A comprehensive literature review on methodologies and applications. European Journal of Operational Research, 250: 1-29.
• ²⁴
  GRECO S & SŁOWINSKI R. 2016. Multiple Criteria Hierarchy Process for ELECTRE Tri Methods. European Journal of Operational Research, 252: 191-203.
• ²⁵
  GRECO S, MOUSSEAU V & SŁOWINSKI R. 2008. Ordinal regression revisited: Multiple criteria ranking using a set of additive value functions. European Journal of Operational Research, 191: 416-436.
• ²⁶
  HAN K & SHIN J. 2014. A systematic way of identifying and forecasting technological reverse salients using QFD, bibliometrics, and trend impact analysis: A carbon nanotube biosensor case. Technovation, 34: 559-570.
• ²⁷
  JUNIOR FRL & CARPINETTI LCR. 2015. Comparação entre os métodos Fuzzy TOPSIS e Fuzzy AHP no apoio à tomada de decisão para seleção de fornecedores. Gestão & Produção, 22(1): 17-34.
• ²⁸
  KADZINSKI M & TERVONEN T. 2013. Robust multicriteria ranking with additive value models and holistic pair-wise preference statements. European Journal of Operational Research, 228(1): 169- 180.
• ²⁹
  KADZINSKI M & CIOMEK K. 2016. Integrated framework for preference modelling and robustness analysis for outranking-based multiple criteria sorting with ELECTRE and PROMETHEE. Information Sciences, 352-353: 167-187.
• ³⁰
  KADZINSKI M, GRECO S & SŁOWINSKI R. 2014. Robust Ordinal Regression for Dominance-based Rough Set. Approach to multiple criteria sorting. Information Sciences, 283: 211-228.
• ³¹
  KADZINSKI M, GRECO S & SŁOWINSKI R. 2012. Extreme ranking analysis in robust ordinal regression Original Research Article. Omega, 40(4): 488-501.
• ³²
  KLUCZEK A & GLADYSZ B. 2015. AHP/TOPSIS-based approach to the generation of environmental improvement options for painting process - Results from an industrial case study. Journal of Cleaner Production, 101: 360-367.
• ³³
  KUO T. 2017. A modified TOPSIS with a different ranking index. European Journal of Operational Research, 260: 152-160.
• ³⁴
  LEONETI AB. 2016. Considerations regarding the choice of ranking multiple criteria decision making methods. Pesquisa Operacional, 36(2): 259-277. DOI: 10.1590/0101-7438.2016.036.02.0259
  » https://doi.org/10.1590/0101-7438.2016.036.02.0259
• ³⁵
  LIMA-JUNIOR FR & CARPINETTI LCR. 2016. Combining SCOR@ model and fuzzy TOPSIS for supplier evaluation. Int. J. Production Economics, 174: 128-141.
• ³⁶
  LONGARAY AA ET AL. 2015. Caracterização da Produção Científica Brasileira sobre a Aplicação de Métodos Multicritério de Apoio à Decisão: uma Análise das Publicações entre 2004-2013. XXXV Encontro Nacional de Engenharia de Produção. ENEGEP. Fortaleza, Ceará, Brasil.
• ³⁷
  LOPES YG & DE ALMEIDA AT. 2013. Multicriteria Decision Model For Selecting A Portfolio Of Oil And Gas Exploration Projects. Pesquisa Operacional, 33(3): 417-441.
• ³⁸
  MARTINS EF, LIMA GBA & GOMES HC. 2015. Análise Estratégica Multicritério Sócio-Econômico-Ambiental como Ferramenta de Apoio à Decisão em uma Empresa de Distribuição de Energia. Revista Eletrônica Sistemas & Gestão, 10(1): 96-106.
• ³⁹
  MISHRA M & CHATTERJEE S. 2017. Application of Analytical Hierarchy Process (AHP) algorithm to income insecurity susceptibility mapping e A study in the district of Purulia, India. Socio-Economic Planning Sciences, 1-19.
• ⁴⁰
  O’KANE C. 2016. Technology transfer executives’ backwards integration: An examination of interactions between university technology transfer executives and principal investigators. Technovation. DOI:10.1016/j.technovation.2016.08.001
  » https://doi.org/10.1016/j.technovation.2016.08.001
• ⁴¹
  PEREIRA FC, VEROCAI HD, CORDEIRO VR, GOMES CFS & COSTA HG. 2015. Bibliometric Analysis of Information Systems Related to Innovation. Procedia Computer Science, 55: 298-307.
• ⁴²
  PUJADAS P, PARDO-BOSCH F, AGUADO-RENTER A & AGUADO A. 2017. MIVES multi-criteria approach for the evaluation, prioritization, and selection of public investment projects. A case study in the city of Barcelona. Land Use Policy, 64: 29-37. DOI: 10.1016/j.landusepol.2017.02.014
  » https://doi.org/10.1016/j.landusepol.2017.02.014
• ⁴³
  R FOUNDATION. 2002. What is R?. Accessed in September the 25th, 2016 from Accessed in September the 25th, 2016 from https://www.r-project.org/about.html
  » https://www.r-project.org/about.html
• ⁴⁴
  RITTER DOS SANTOS ME & TORKOMIAN ALV. 2013. Technology transfer and innovation: The role of the Brazilian TTOs. International Journal of Technology Management & Sustainable Development, 12(1): 89-111.
• ⁴⁵
  ROSZKOWSKA E & WACHOWICZ T. 2015. Application of fuzzy TOPSIS to scoring the negotiation offers in ill-structured negotiation problems. European Journal of Operational Research, 242(3): 920-932.
• ⁴⁶
  ROUBENS M, RUSINOWSKA A & DE SWART H. 2006. Using MACBETH to determine utilities of governments to parties in coalition formation. European Journal of Operational Research, 172(2): 588-603.
• ⁴⁷
  SARRAZIN R & DE SMET Y. 2015. Applying multicriteria decision analysis to design safe road projects. European Journal of Transport and Infrastructure Research, 15(4): 613-634.
• ⁴⁸
  SAHIN R. 2016. Fuzzy multicriteria decision making method based on the improved accuracy function for interval-valued intuitionistic fuzzy sets. Soft Computing, 20(7): 2557-2563. DOI 10.1007/s00500-015-1657-x
  » https://doi.org/10.1007/s00500-015-1657-x
• ⁴⁹
  SPAK MDS ET AL. 2012. Produção científica brasileira em métodos multicritérios de tomada de decisão: cienciometria de publicações do ENEGEP entre os anos de 2007 a 2011. XXXII Encontro Nacional de Engenharia de Produção. Rio Grande do Sul.
• ⁵⁰
  WALCZAK D & RUTKOWSKA A. 2017. Project rankings for participatory budget based on the fuzzy TOPSIS method. European Journal of Operational Research, 260: 706-714.
• ⁵¹
  WANG P, ZHU Z & WANG Y. 2016. A novel hybrid MCDM model combining the SAW, TOPSIS and GRA methods based on experimental design. Information Sciences, 345: 27-45.
• ⁵²
  YUNNA W, JINYING Z, JIANPING Y, SHUAI G & HAOBO Z. 2016. Study of decision framework of offshore wind power station site selection based on ELECTRE-III under intuitionistic fuzzy environment - A case of China Energy. Conversion and Management, 113: 66-81.
• ⁵³
  ZAKA J & KRUSZYNSKIB MIROSLAW. 2015. Application of AHP and ELECTRE III/IV methods to multiple level, multiple criteria evaluation of urban transportation projects. Transportation Research Procedia, 10: 820-830.
• ⁵⁴
  ZHANG W, JU1 Y & LIU X. 2017. Interval-valued intuitionistic fuzzy programming technique for multicriteria group decision making based on Shapley values and incomplete preference information. Soft Computing, 21(19): 5787-5804. DOI 10.1007/s00500-016-2157-3
  » https://doi.org/10.1007/s00500-016-2157-3

Publication Dates

• Publication in this collection
  Sep-Dec 2018

History

• Received
  01 Sept 2017
• Accepted
  29 June 2018