Evaluation of efficiency of Brazilian federal universities: an approach through Data Envelopment Analysis

Abstract

Higher education, in addition to playing a fundamental social role, has its performance intrinsically linked to the economic development of a country. The recent global crisis caused by the coronavirus pandemic has had a strong impact on the educational system, such as the 8.61% reduction in the public budget allocated to education in Brazil for the year 2021 when compared to 2020 figures. This fact further aggravated an already existing scenario of dismantling public education. The identification of inefficient institutions, with problems in the allocation of public resources, is presented as an alternative to start an improvement process for the development of the public sector. This article proposes to evaluate the efficiency of Brazilian federal universities, from 2017 to 2021, through the application of performance indicators defined by the Federal Court of Auditors for Higher Education Institutions in the DEA methodology with a focus on output indicators, establish a a of the evaluated units, and to determine the efficient and the inefficient ones. Through this study, it is possible to conjecture that the COVID-19 pandemic impacted the efficiency of Brazilian federal universities, which presented a reduction of 0.95% in its average between the years 2019 and 2021, in addition, another important perception is the performance of universities in the North region of the country, which has the best average efficiency coefficient among all regions.

Keywords:
higher education; efficiency; DEA; data envelopment

Resumo

A educação superior, além de desempenhar um fundamental papel social, tem seu desempenho intrinsecamente ligado ao desenvolvimento econômico de um país. A recente crise mundial provocada pela pandemia de coronavírus trouxe fortes impactos para o sistema educacional, como a redução de 8,61% do orçamento público destinado à educação no Brasil para o ano de 2021 quando comparado aos valores de 2020. Fato que agravou ainda mais um cenário já existente de desmantelamento da educação pública. A identificação de instituições ineficientes, com problemas de alocação de recursos públicos, se apresenta como alternativa de início de um processo de melhoria para o desenvolvimento do setor público. Este trabalho se propõe a avaliar a eficiência das universidades federais brasileiras, no período de 2017 a 2021, através da aplicação de indicadores de performance definidos pelo Tribunal de Contas da União para Instituições de Ensino Superior na metodologia DEA com foco nos indicadores de saída, além de estabelecer uma classificação das unidades avaliadas, e determinar as eficientes e as ineficientes. Através do estudo realizado, é possível conjecturar que a pandemia de COVID-19 impactou a eficiência das universidades federais brasileiras, pois estas apresentaram uma redução de 0,95% em sua média entre os anos de 2019 e 2021; além disso, outra percepção importante é o desempenho das universidades da região Norte do país, que possuem a melhor média de coeficiente de eficiência dentre todas as regiões.

Palavras-chave:
educação superior; eficiência; DEA; envoltória de dados

Resumen

La educación superior, además de desempeñar un papel social fundamental, está intrínsecamente ligada al desarrollo económico de un país. La reciente crisis mundial causada por la pandemia del coronavirus ha tenido un fuerte impacto en el sistema educativo, como se refleja en la reducción del 8,61% del presupuesto público para la educación en Brasil para el año 2021 en comparación con los valores de 2020. Este hecho ha agravado aún más un escenario ya existente de desmantelamiento de la educación pública. La identificación de instituciones ineficientes, con problemas en la asignación de recursos públicos, se presenta como una alternativa para iniciar un proceso de mejora del sector público. Este trabajo propone evaluar la eficiencia de las universidades federales brasileñas en el período de 2017 a 2021 a través de la aplicación de indicadores de desempeño definidos por el Tribunal de Cuentas de la Unión para las instituciones de enseñanza superior, utilizando la metodología DEA con enfoque en los indicadores de output, establecer una clasificación de las unidades evaluadas y determinar las eficientes e ineficientes. A través del estudio realizado, es posible conjeturar que la pandemia del COVID-19 ha impactado en la eficiencia de las universidades federales brasileñas, que han presentado una reducción del 0,95% en su promedio entre los años 2019 y 2021. Además, otra percepción importante es el desempeño de las universidades de la región Norte del país, que tienen el mejor coeficiente de eficiencia promedio entre todas las regiones.

Palavras clave:
enseñanza superior; eficiência; DEA; envolvimiento de datos

1 Introduction

Higher education, besides fulfilling a fundamental social role, by representing a nation's highest ideals through the creation and dissemination of knowledge, has its performance intrinsically connected to the economic development, such that its funding attracts the attention of economists, researchers, and government decision makers from around the world (DAULTANI; DWIVEDI; PRATAP, 2021; KAUR, 2021; MONCAYO–MARTÍNEZ; RAMÍREZ–NAFARRATE; HERNÁNDEZ–BALDERRAMA, 2020; ZHANG; WU; ZHU, 2020).

In most countries, including Brazil, the federal government is responsible for maintaining the main higher education facilities (HEIs) (LEE; JOHNES, 2022), which face a scenario of great and growing demand (NAVAS et al., 2020). In Brazil, the private network accounts for more than 95% of college places, while about one-third of students are enrolled in public institutions (BRASIL, 2022).

In addition, the recent global crisis caused by the coronavirus pandemic had a strong impact on the education system. In April 2020, one month after the World Health Organization (WHO) declared COVID -19 a pandemic, 166 countries have already implemented national closure policies for educational institutions to contain the spread of the virus, affecting 84.5% of all students worldwide (BENTO et al., 2021).

One of the impacts of the COVID-19 pandemic in the Brazilian education system has been the reduction of public funding for 2021 by 8.61% compared to 2020, a fact that exacerbates an already existing scenario of cuts in public education, since in the period from 2014 to 2020 there was already a 28.5% decrease in funding dedicated to education (WOICOLESCO; MOROSINI; MARCELINO, 2022).

In light of this context, identifying inefficient institutions, with problems in the allocation of public resources, presents itself as an alternative to initiate an improvement process for public sector development (WHEELOCK; WILSON, 2008). More efficient public policies reduce constraints on the public budget and achieve the same results with fewer resources or even enhance outcomes with current investments (DUFRECHOU, 2016).

In this way, the managers of public HEIs seem themselves pressured to optimize the allocation of financial resources in order to increase efficiency, while facing even more scarce resources (DUAN, 2019; NOJAVAN; HEIDARI; MOHAMMADITABAR, 2021; TRAN; VILLANO, 2018).

The efficiency of higher education institutions can be defined as the ability to obtain the maximum value from the output indicators for a group of input indicators (VISBAL-CADAVID; MENDOZA; HOYOS, 2019). One of the most commonly used methods in the literature to assess the efficiency of HEIs is data envelopment analysis (DEA) (WITTE; LÓPEZ-TORRES, 2017), because it has favorable characteristics for this type of organizational analysis, it is an oriented method for evaluating a group of decision-making units (DMUs), which convert input indicators into output indicators, it can work with multiple input and output indicators, it provides an overview of the DMUs' strengths and weaknesses, and it offers the possibility of evaluating multiple aspects of the educational sector, through which is possible to seek an increase in it’s efficiency (WU et al., 2020).

In this manner, this paper aims to evaluate the efficiency of Brazilian federal universities from 2017 to 2021, through the application of performance indicators defined by the Federal Court of Accounts - Brazil (TCU) for HEIs in the methodology of Data Envelopment Analysis (DEA), focusing on output indicators, in order to elaborate a classification of the evaluated units and determine efficient and inefficient units, in order to answer the following research question: Which Brazilian federal universities displayed greater technical efficiency in the use of public funds between 2017 and 2021?

Some studies in this sense have already been carried out in Brazil, for example, a study that evaluated the efficiency of federal institutions of education, science and technology (IFs) (PARENTE et al., 2021), and another study in which the authors evaluated the efficiency of public resources in 59 Brazilian federal universities between 2013 and 2017 using 2 input and 2 output indicators (HAMMES JUNIOR; FLACH; MATTOS, 2020). However, the study most similar to the one presented here evaluates, by region, the efficiency of 56 Brazilian public universities between 2010 to 2016 using a total of 7 indicators (LETTI; BITTENCOURT; VILA, 2020).

The present study aims to contribute and expand the literature on the management of public funds in education. To this end, it expands the group of indicators for performance analysis and the number of universities evaluated. In this way, it aims to better embrace the diversity of Brazilian universities and, most importantly, to analyze the likely impact of the COVID-19 pandemic. In this way, the present study can collaborate providing indications for the review of the management models of HEIs, through the maximization of investments in public resources and thus shed light on the efficient management of public policies in this sector.

2 Literature review

2.1 Higher Education in Brazil

Nowadays, higher education in Brazil is provided by universities, college centers, colleges, higher education institutes and technological education centers, being public or private, for-profit or not-for-profit (GOMES; MACHADO-TAYLOR; SARAIVA, 2018). However, universities are the most representative institutions among them. In 2020, 4.7 million students were enrolled in universities, which corresponds to more than half (54.3%) of the total. In the Brazilian federal education network, 82.2% of students are enrolled in universities (BRASIL, 2022).

In the last 50 years, the Brazilian higher education system has undergone a significant expansion, quantitatively increasing its network of operation, in addition to offering new qualification paths (BARBOSA, 2019). Between 2010 and 2020, enrollment in higher education increased by 35.5%, from 6.40 million to 8.68 million students (BRASIL, 2022).

Regarding the public sector, this expansion is mainly justified by the intensification of some public policies in the 2000s in Brazil, which allowed access to social classes previously excluded from this educational system (BARBOSA, 2019; CARVALHAES; MEDEIROS; TAGLIARI, 2021). Among them we can mention the Program to Support the Restructuring and Expansion Plans of Federal Universities (REUNI), which had the purpose of funding the structural expansion of the higher education network and improving the use of existing facilities (GOMES; MACHADO-TAYLOR; SARAIVA, 2018).

However, this growth movement in higher education in Brazil was largely supported by the growth of the private sector (BARBOSA, 2019) to meet the demand of students who could not obtain a place in the public network and questioned the government about the lack of opportunities in higher education (COLOMBO; RODRIGUES, 2011). When comparing the years 2010 to 2020, it can be observed an increase of 42% in the number of enrollments in the private network and 19.1% in the public network. In 2020, private HEIs accounted for 77.5% of total enrollment in graduation courses (BRASIL, 2022).

As for the public policies that supported students' access to the private sector, we can mention scholarships under the "College for All" Program (PROUNI) and funding from the Brazilian Student Financing Fund (FIES). Slightly more than half of students in the private higher education sector rely on some type of public funding (BARBOSA, 2019).

Regarding the quality of Brazilian higher education, public universities are seen with greater prestige, employ more qualified professors, due to the strong focus on research and the selective nature of the public education sector, while private institutions offer students low barriers to selection ( MONT’ALVÃO NETO, 2016). In the 2019 National Examination for Student Performance (ENADE), a test to evaluate higher education students by program of study, more than 81% of the highest-scoring courses were from public institutions, and 66% were from the federal network (BRASIL, 2022).

This scenario, in which the quality of higher education is higher in public institutions, and in which there is a movement towards the growth of vacancies in private institutions, motivated by public policies, sheds light on the question of the efficiency of resource use in higher education institutions.

2.2 Efficiency in Higher Education Institutions

Efficiency consists of optimizing a combination of inputs and methods of the production process in order to achieve an optimal level of outputs. In other words, efficiency is the ability to perform tasks correctly, by minimizing the ratio between inputs and outputs and optimizing the use of resources (HAMMES JUNIOR; FLACH; MATTOS, 2020).

Efficiency level is considered an important indicator for higher education institutions and is currently one of the most important public policy objectives, since evaluating the performance of HEIs is a key factor in allocating scarce public resources. If HEIs operate efficiently, this justifies the public funds invested in the sector (AGASISTI et al., 2021; HAMMES JUNIOR; FLACH; MATTOS, 2020).

The education sector provides an excellent context for efficiency evaluation studies, as it presents several challenges, such as the fact that its institutions are not-for-profit, produce multiple outputs, and have difficulty converting their inputs and outputs into monetary values (WITTE; LÓPEZ-TORRES, 2017).

Currently, there is considerable literature on the technical efficiency of HEIs, and this topic has been extensively researched (PAPADIMITRIOU; JOHNES, 2019). Studies focusing on universities' outcomes can be approached in a variety of ways, with most studies choosing indicators based on publications or research funding (GRALKA; WOHLRABE; BORNMANN, 2019).

In Brazil, most studies that assess the efficiency of the education sector have been developed at the municipal level, using financial expenditure indicators as inputs and institutional assessment measures as outputs (PARENTE et al., 2021).

Among the studies that have proposed to address Brazilian public higher education through DEA, we can mention an evaluation of the efficiency of teaching in undergraduate courses at the Fluminense Federal University through the assessment of the National Examination for Student Performance (ENADE) as an input and output parameter (TAVARES; MEZA, 2020); the application of two input indicators and two output indicators for the evaluation of efficiency in the use of public expenditures by 59 Brazilian federal universities in the period from 2013 to 2017 (HAMMES JUNIOR; FLACH; MATTOS, 2020), and the evaluation of the efficiency of 38 federal institutions of education, science and technology (IFs) in the period from 2010 to 2017, using the indicators formulated by the Federal Court of Accounts - Brazil (TCU) to evaluate this type of institutions (PARENTE et al., 2021).

Finally, a study with a similar objective to this work evaluated the efficiency of Brazilian federal universities in the period from 2010 to 2016, taking into account not the indicators defined by the TCU for the evaluation of HEIs, but the component values of these indicators, such as the number of university students, the number of professors and the running costs (LETTI; BITTENCOURT; VILA, 2020). In this sense, several studies already use Data Envelopment Analysis (DEA) as a method for evaluating efficiency in the education sector, which is one of the most common and powerful methods for analyzing public and private educational institutions (LEE; JOHNES, 2022; NAVAS et al., 2020; NOJAVAN; HEIDARI; MOHAMMADITABAR, 2021; VISBAL-CADAVID; MENDOZA; HOYOS, 2019).

2.3 DEA as a Method for Measuring Efficiency in HEIs

Data envelopment analysis, through an approach aimed at evaluating the performance of a group of entities called decision-making units (DMUs), which convert multiple inputs into multiple outputs, has several advantages that make it ideal for efficiency analysis in higher education (WU et al., 2020).

DEA does not require information on how the process of converting inputs into outputs works, and it can not only identify areas for improvement but also describe opportunities for future development. In addition, DEA is able to answer questions about the strengths and weaknesses of DMUs, thus identifying the best volume of resources to be made available to the education sector (WU et al., 2020).

Existing efficiency studies with the application of DEA on HEIs focus mainly on two aspects: Performance evaluation and resource allocation (WU et al., 2020) and can be divided into two large groups: basic education unit evaluation and higher education efficiency analysis (MONCAYO–MARTÍNEZ; RAMÍREZ–NAFARRATE; HERNÁNDEZ– BALDERRAMA, 2020). Within this second group, we can cite some works involving different countries, as shown in Table 1:

Table 1
Studies on the efficiency of educational institutions

Country of Study	Title	Author(s)	Year of Publication
England and Colombia	Using network DEA to inform policy: The case of the teaching quality of higher education in England.	LEE; JOHNES	2022
Russia	Efficiency of regional higher education systems and regional economic short-run growth: empirical evidence from Russia.	AGASISTI et al.	2021
Spain	Efficiency in higher education. Empirical study in public universities of Colombia and Spain.	RAMÍREZ-GUTIÉRREZ; BARRACHINA-PALANCA; RIPOLL-FELIU	2020
China	Performance evaluation and enrollment quota allocation for higher education institutions in China.	ZHANG; WU; ZHU	2020
Colombia	Colombian higher education institutions evaluation.	NAVAS et al.	2020
Mexico	Evaluation of public HEI on teaching, research, and knowledge dissemination by Data Envelopment Analysis.	MONCAYO–MARTÍNEZ; RAMÍREZ–NAFARRATE; HERNÁNDEZ– BALDERRAMA	2020
Italy	Performance-based university funding and the drive towards ‘institutional meritocracy’in Italy.	MATEOS-GONZÁLEZ; BOLIVER	2019
England	Does merging improve efficiency? A study of English universities.	PAPADIMITRIOU; JOHNES	2019
Argentina	Efficiency in public higher education on Argentina 2004–2013: Institutional decisions and university-specific effects.	QUIROGA-MARTÍNEZ; FERNÁNDEZ-VÁZQUEZ; ALBERTO	2018

• Source: Prepared by the authors.

3 Methodological processes

3.1 Data Envelopment Analysis (DEA)

DEA consists of a data-oriented methodology that applies linear programming techniques to evaluate the efficiency of a group of decision-making units (DMUs). DEA calculates the relative efficiency of a DMU through the ratio of the weighted sum of several inputs and several outputs, thus identifying inefficient DMUs as well as the source of their inefficiency (CHARNES; COOPER; RHODES, 1978; LEE; JOHNES, 2022). A DMU is considered relatively efficient if this ratio equals one, and not efficient otherwise (ZHANG; WU; ZHU, 2020).

A DMU considered efficient is not able to reduce the volume of inputs without reducing the value of outputs. Similarly, a DMU considered efficient is not able to increase the volume of outputs without increasing the volume of inputs (QUIROGA-MARTÍNEZ; FERNÁNDEZ-VÁZQUEZ; ALBERTO, 2018).

The analysis orientation of DEA can be divided into input-oriented or output-oriented. The input orientation examines how to minimize the volume of inputs while maintaining the volume of outputs, while the output orientation examines how to maximize the volume of outputs while maintaining the same volume of inputs (MONCAYO–MARTÍNEZ; RAMÍREZ–NAFARRATE; HERNÁNDEZ–BALDERRAMA, 2020).

As for returns to scale, DEA can be divided into models of constant returns to scale, called CCR (CHARNES; COOPER; RHODES, 1978), and variable returns to scale, called BCC models (BANKER; CHARNES; COOPER, 1984).

The CCR model evaluates overall efficiency by analyzing scale and technical efficiencies simultaneously, while in the BCC model scale efficiency is separated from technical efficiency (ZHANG; WU; ZHU, 2020).

For the development of this research, the output orientation was chosen for the DEA model, in order to highlight the DMUs with better results, guiding inefficient DMUs to improve results with the same amounts of inputs. The BCC model type was also selected, in order to allow the analysis of DMUs' gains in scale.

In DEA modeling, with k DMUs, and each one of them using m Inputs to produce n outputs, let X_jk be input j and Y_ik be output i of DMU j, the output-oriented BCC model can be described as follows (BANKER; CHARNES; COOPER, 1984):

(1) $Min{\displaystyle {\sum }_{j=1}^n{X}_{j0}*v_j+w_0}$

Subject to:

(2) ${\displaystyle {\sum }_{i=1}^m{Y}_{i0}*u_i=1,parai=1,\dots ,m}$

(3) $\begin{array}{c}{\displaystyle {\sum }_{i=1}^m{Y}_{ik}*u_i-{\displaystyle {\sum }_{j=1}^n{X}_{jk}*v_j-w_0\le 0,parak=1,\dots ,z}}\\ \begin{array}{cc}{v}_{j}e{u}_i\ge 0;& \begin{array}{cc}i=1,\dots m;& j=1,\dots n\end{array}\end{array}\\ w\in \Re \end{array}$

Where u_i and v_j are the weights of outputs and inputs respectively, m the number of outputs, n the number of inputs, z the number of DMUs and w is the scale factor, which indicates the direction of scale returns, if w is positive, the DMU operates with increasing returns to scale, if w is negative, the DMU operates with decreasing returns to scale, and if w is equal to zero, the DMU operates with constant returns to scale (MEZA; BIONDI NETO; RIBEIRO, 2005).

This way, DEA allows the obtained weights, through the resolution of the model, to be more favorable for the calculation of efficiency, however these weights must guarantee that the efficiency of a DMU is not greater than one (ZHANG; WU; ZHU, 2020).

This methodology for calculating efficiency is focused on determining an optimal production frontier. Points located on this frontier are defined as efficient. Points in the region inside this frontier are defined as inefficient, that is: with an efficiency value less than one. Every inefficient DMU can have its input and/or output values adjusted so that it reaches efficiency by projecting its efficiency value onto the optimal production frontier (ZHANG; WU; ZHU, 2020).

The study performs a panel data analysis, to minimize the effect of time on the DEA, performing a window analysis (DE CASTRO CAMIOTO; MARIANO; DO NASCIMENTO REBELATTO, 2014; FERREIRA; GOMES, 2020). Window analysis is performed by separating the sample periods into different groups (windows) (DE CASTRO CAMIOTO; MARIANO; DO NASCIMENTO REBELATTO, 2014). For this, the following equations were used to determine the windows:

(3) $TamanhodaJanela\left(p\right)=\frac{\left(b+1\right)}{2}$

(4) $N\text{ú}merodeJanelas=b-p+1$

Where b is the number of periods.

This way, in this study the number of periods (b = 5) refers to the years between 2017 to 2021. Therefore, 3 windows were obtained with the size of 3 as follows: window 1 (2017 – 2019); window 2 (2018 – 2020) and window 3 (2019 – 2021).

3.2 Data Collection

This study considers as DMU each Brazilian federal university. Today, Brazil has 68 federal public universities recognized by the MEC (Ministry of Education) with foundations dated between 1910 and 2018, all of which offer face-to-face and/or distance-learning higher education courses (MEC, 2023).

For data collection, indicators defined by the Federal Court of Accounts - Brazil (TCU) were used. The TCU determined through Normative Decision 408/2002 a set of nine management and performance indicators for the Federal Institutions of Higher Education (IFES), known as “TCU Indicators”, they are a set of metrics that aim to enable the evaluation of the operational performance of the institutions (BRASIL, 2002).

The selection and examination of these indicators by the Federal Court of Accounts was based on an operational audit carried out at the University of Brasília, which sought to portray the relevant aspects of the performance of higher education institutions. This set was tested in five other institutions (FUA, UFPE, UFGO, UFRJ and UFRGS) (BRASIL, 2002).

Also, according to Court Ruling 1.043/2006, IFESs must present the result of these indicators annually in their account management reports, which facilitates public access to this information (BRASIL, 2006). Thus, the research used these indicators, which are considered important for the TCU in analyzing the performance of IFESs, with the intention of obtaining the results of the model used, which are aligned with the court's perspectives on the efficiency of institutions.

Considering that half of the mapped universities operate without teaching hospitals, in order to maintain the homogeneity of the DMUs, it was decided to use only the indicators that disregard the management of teaching hospitals.

For data envelopment analysis, the set of TCU indicators can be divided into input and output variables, as shown in Table 2 below. The classification, into inputs and outputs, was based on the relationship of each variable with its role as an applied resource and the result obtained from the application of these resources. Thus, it was understood that the CAPES/MEC Concept for Post-Graduate courses and the Success Rate in Graduate courses are results of the efficient use of resources represented in the variables classified as inputs.

Table 2
Input and output variables used in the study

Type	Variable	Description
Input	Current Cost without Teaching Hospital / Equivalent Student (CC/AE)	Ratio between all the institution's current expenses, excluding expenses with university teaching hospitals and maternities, and the number of equivalent students (number of equivalent undergraduate students, number of full-time postgraduate and medical residency students).
	Full-time Student / Equivalent Teacher (AI/PE)	Ratio between the total number of full-time students enrolled at the university and the number of equivalent professors with or without exclusive dedication.
	Full-time Student / Equivalent Employee without Teaching Hospital (AI/FE)	Ratio between the total number of full-time students enrolled at the university and the number of equivalent employees at the institution, disregarding university teaching hospital workers.
	Equivalent Employee without Teaching Hospital / Equivalent Teacher (FE/PE)	Ratio between the total number of equivalent professors at the university, with or without exclusive dedication and the number of equivalent employees at the institution, disregarding university teaching hospital workers.
	Degree of Student Participation (GPE)	Ratio between the number of students enrolled full-time in the institution and the total number of students.
	Degree of Student Engagement with Postgraduate Courses (GEPG)	Ratio between the number of students enrolled in postgraduate, master's and doctoral programs at the institution and the total number of students.
	Faculty Qualification Index (IQCD)	Degree of qualification of the institution's teachers with the attribution of scores to the training levels (graduate, specialist, master and doctor).
Output	CAPES/MEC Concept for Postgraduate programs (CAPES)	Ratio between the average Capes evaluation scores of master's and doctoral courses and the number of postgraduate programs at the university.
	Graduate Programs Success Rate (TSG)	Ratio between the number of students graduating from the institution by the number of new students.

• Source: (BRASIL, 2004)

The values for each of the indicators will be sourced through research in publicly available databases from the federal government and through consultation with the universities' account management reports. In order to work with the most recent data available, cover the most critical period of the COVID-19 pandemic, and provide a significant time frame that allows an analysis of the efficiency evolution of each DMU, without overloading the data collection process, the indicators will be studied for a 5-year period, from 2017 to 2021.

For the implementation and resolution of the efficiency analysis model, the R Benchmarking package was employed as a computational tool through R Studio.

4 Results and Discussion

After collecting the research data, descriptive statistics were obtained for each variable utilized in the Data Envelopment Analysis (DEA) model. The statistics values are presented in Table 3.

Table 3
Technical efficiency indices by the window analysis approach

Window 1 2017-2019	CC/AE	AI/PE	AI/FE	FE/PE	GPE	GEPG	IQCD	CAPES	TSG
Average	21.414,76	11,81	9,34	1,34	5,77	0,12	3,81	4,37	45,61
Standard Deviation	7.665,57	2,87	3,32	0,34	2,56	0,07	0,67	0,46	14,92
Minimum	3.924,13	3,23	2,18	0,41	0,00	0,01	0,00	0,00	0,00
Maximum	75.231,51	18,49	24,90	3,00	19,04	0,30	5,32	5,00	79,74
Count	189	189	189	189	189	189	189	189	189
Window 2 2018-2020	CC/AE	AI/PE	AI/FE	FE/PE	GPE	GEPG	IQCD	CAPES	TSG
Average	21.407,94	16,78	9,77	1,98	5,70	0,12	3,81	4,43	43,84
Standard Deviation	5.666,32	67,19	4,57	9,29	2,55	0,07	0,66	0,44	15,31
Minimum	3.642,30	5,32	4,35	0,24	0,00	0,01	0,00	0,00	0,00
Maximum	39.392,61	934,77	53,35	128,98	19,04	0,30	5,30	5,00	86,00
Count	189	189	189	189	189	189	189	189	189
Window 3 2019-2021	CC/AE	AI/PE	AI/FE	FE/PE	GPE	GEPG	IQCD	CAPES	TSG
Average	22.084,71	16,65	9,99	1,94	5,55	0,13	3,84	4,50	41,65
Standard Deviation	5.822,14	67,20	4,95	9,30	3,02	0,07	0,61	0,43	15,18
Minimum	3.642,30	0,00	0,00	0,00	0,00	0,00	0,00	0,00	0,00
Maximum	40.634,51	934,77	53,35	128,98	27,08	0,41	5,30	5,00	89,00
Count	189	189	189	189	189	189	189	189	189

• Source: Prepared by the authors.

According to Table 3, it can be observed that the current cost per equivalent student, on average, remains consistent throughout the analyzed period. Thus, on average, the cost per equivalent student for the 2019-2021 period remained similar to the cost for the 2017-2019 period. However, when we consider the absolute value applied in the periods, the cost per equivalent student showed a significant reduction, as can be seen from the maximum value.

The efficiency levels of the universities are detailed in Table 4, which presents the efficiency indicators by time window where each row (W1, W2, and W3) shows the efficiencies found through the output-oriented DEA-BCC model. It is possible to observe in the table the average efficiency of each window, as well as the average efficiency of each DMU in the evaluated windows. Through this, one can observe the trend of efficiency behavior over the evaluated period.

Table 4
Technical efficiency indices by window analysis approach

DMUs	Windows	Periods					Average	Average of the Windows
		2017	2018	2019	2020	2021
FURG	W1	0,9044	0,9359	0,9293			0,9232	0,9243
	W2		0,9171	0,8989	0,9296		0,9152
	W3			0,8989	0,9312	0,9734	0,9345
UFABC	J1	1,0000	1,0000	1,0000			1,0000	1,0000
	J2		1,0000	1,0000	1,0000		1,0000
	J3			1,0000	1,0000	1,0000	1,0000
UFAC	W1	0,8957	0,8952	0,8251			0,8720	0,8686
	W2		1,0000	0,8197	0,8451		0,8883
	W3			0,8131	0,8432	0,8798	0,8453
UFAL	W1	1,0000	1,0000	1,0000			1,0000	0,9975
	W2		0,9775	1,0000	1,0000		0,9925
	W3			1,0000	1,0000	1,0000	1,0000
UFAM	W1	0,8505	0,7983	0,8326			0,8271	0,8389
	W2		0,7788	0,8147	0,8957		0,8298
	W3			0,8131	0,9004	0,8656	0,8597
UFBA	W1	0,9217	0,9170	0,9173			0,9187	0,9093
	W2		0,8978	0,9048	0,9138		0,9055
	W3			0,9033	0,9120	0,8964	0,9039
UFC	W1	0,9267	0,9205	0,9226			0,9233	0,9224
	W2		0,9021	0,9115	0,9330		0,9155
	W3			0,9087	0,9318	0,9443	0,9283
UFCA	W1	1,0000	1,0000	0,9974			0,9991	0,9997
	W2		1,0000	1,0000	1,0000		1,0000
	W3			1,0000	1,0000	1,0000	1,0000
UFCG	W1	0,7668	0,7929	0,8132			0,7910	0,8213
	W2		0,7990	0,8163	0,8527		0,8227
	W3			0,8159	0,8527	0,8821	0,8502
UFCSPA	W1	1,0000	1,0000	1,0000			1,0000	0,9969
	W2		1,0000	0,9948	0,9909		0,9952
	W3			0,9952	0,9912	1,0000	0,9955
UFERSA	W1	0,9300	0,8644	0,9174			0,9039	0,9038
	W2		0,8484	0,9072	0,9270		0,8942
	W3			0,8857	0,9309	0,9237	0,9134
UFES	W1	0,9644	0,9961	0,9833			0,9813	0,9806
	W2		0,9625	0,9684	1,0000		0,9770
	W3			0,9681	1,0000	0,9829	0,9837
UFF	W1	0,9319	0,9421	0,9383			0,9374	0,9408
	W2		0,9435	0,9283	0,9534		0,9417
	W3			0,9263	0,9527	0,9511	0,9434
UFFS	W1	0,9496	0,9502	0,9389			0,9463	0,9314
	W2		0,9280	0,9023	0,9333		0,9212
	W3			0,8987	0,9328	0,9488	0,9268
UFG	W1	0,9109	0,9567	0,9477			0,9384	0,9330
	W2		0,9389	0,9325	0,9230		0,9315
	W3			0,9317	0,9214	0,9343	0,9292
UFGD	W1	0,9054	0,9223	0,9284			0,9187	0,9137
	W2		0,8909	0,9043	0,9252		0,9068
	W3			0,9037	0,9244	0,9189	0,9156
UFJF	W1	0,9651	0,9583	0,9702			0,9645	0,9661
	W2		0,9574	0,9680	0,9713		0,9656
	W3			0,9687	0,9723	0,9638	0,9682
UFLA	W1	0,9916	0,9519	0,9807			0,9747	0,9643
	W2		0,9317	0,9556	0,9676		0,9516
	W3			0,9556	0,9670	0,9769	0,9665
UFMA	W1	0,9497	0,9151	0,9255			0,9301	0,9101
	W2		0,8802	0,8949	0,9021		0,8924
	W3			0,8965	0,9041	0,9231	0,9079
UFMG	W1	0,9933	1,0000	1,0000			0,9978	0,9968
	W2		1,0000	0,9958	1,0000		0,9986
	W3			0,9912	1,0000	0,9907	0,9940
UFMS	W1	0,8744	0,9277	0,9206			0,9075	0,9175
	W2		0,9167	0,9095	0,9313		0,9192
	W3			0,9098	0,9303	0,9369	0,9256
UFMT	W1	0,8722	0,9103	1,0000			0,9275	0,9343
	W2		0,8783	1,0000	0,9543		0,9442
	W3			0,9217	0,9292	0,9429	0,9312
UFOB	W1	1,0000	1,0000	0,9198			0,9733	0,9732
	W2		1,0000	0,9268	1,0000		0,9756
	W3			0,9119	1,0000	1,0000	0,9706
UFOP	W1	0,9091	0,9055	0,9204			0,9117	0,9307
	W2		0,9051	0,9221	0,9606		0,9293
	W3			0,9249	0,9634	0,9651	0,9512
UFOPA	W1	1,0000	1,0000	0,9460			0,9820	0,9720
	W2		1,0000	0,9194	0,9742		0,9645
	W3			0,9087	1,0000	1,0000	0,9696
UFPA	W1	1,0000	0,9875	0,9558			0,9811	0,9461
	W2		0,9182	0,9219	0,9393		0,9265
	W3			0,9219	0,9393	0,9313	0,9308
UFPB	W1	0,9172	0,9241	0,9259			0,9224	0,9234
	W2		0,9197	0,9186	0,9287		0,9223
	W3			0,9178	0,9293	0,9292	0,9254
UFPE	W1	0,9310	0,9720	0,9654			0,9561	0,9505
	W2		0,9327	0,9383	0,9716		0,9475
	W3			0,9383	0,9707	0,9344	0,9478
UFPEL	W1	1,0000	1,0000	1,0000			1,0000	0,9986
	W2		1,0000	0,9997	1,0000		0,9999
	W3			0,9875	1,0000	1,0000	0,9958
UFPI	W1	0,8579	0,8624	0,8910			0,8704	0,8692
	W2		0,8507	0,8796	0,8679		0,8660
	W3			0,8733	0,8673	0,8727	0,8711
UFPR	W1	1,0000	1,0000	0,9735			0,9912	0,9817
	W2		1,0000	0,9694	0,9781		0,9825
	W3			0,9703	0,9781	0,9661	0,9715
UFRA	W1	1,0000	0,9752	1,0000			0,9917	0,9962
	W2		1,0000	1,0000	1,0000		1,0000
	W3			1,0000	1,0000	0,9902	0,9967
UFRB	W1	0,8707	0,9351	0,9593			0,9217	0,9704
	W2		1,0000	0,9911	1,0000		0,9970
	W3			0,9772	1,0000	1,0000	0,9924
UFRGS	W1	0,9853	0,9888	0,9880			0,9874	0,9843
	W2		0,9789	0,9821	0,9855		0,9822
	W3			0,9821	0,9855	0,9826	0,9834
UFRJ	W1	0,9631	0,9750	0,9750			0,9710	0,9500
	W2		0,9242	0,9579	0,9490		0,9437
	W3			0,9358	0,9478	0,9220	0,9352
UFRN	W1	0,9064	0,9078	0,9108			0,9083	0,9069
	W2		0,8955	0,8928	0,9183		0,9022
	W3			0,8919	0,9182	0,9205	0,9102
UFRPE	W1	0,9689	0,9799	0,9779			0,9756	0,9800
	W2		0,9741	0,9696	1,0000		0,9812
	W3			0,9706	1,0000	0,9792	0,9833
UFRR	W1	1,0000	0,9205	0,9073			0,9426	0,8947
	W2		0,9189	0,9089	0,8246		0,8842
	W3			0,9006	0,8264	0,8450	0,8573
UFRRJ	W1	0,9927	0,9559	0,9375			0,9620	0,9541
	W2		0,9297	0,9455	0,9569		0,9440
	W3			0,9424	0,9564	0,9701	0,9563
UFS	W1	0,9418	0,9125	0,9093			0,9212	0,9242
	W2		0,9092	0,9151	0,9433		0,9225
	W3			0,9139	0,9424	0,9300	0,9288
UFSB	W1	1,0000	1,0000	1,0000			1,0000	0,9953
	W2		1,0000	1,0000	0,9668		0,9889
	W3			1,0000	0,9909	1,0000	0,9970
UFSC	W1	0,9608	0,9724	0,9759			0,9697	0,9721
	W2		0,9663	0,9659	0,9801		0,9707
	W3			0,9642	0,9798	0,9835	0,9758
UFSCar	W1	0,9820	1,0000	0,9856			0,9892	0,9907
	W2		0,9874	0,9890	0,9983		0,9916
	W3			0,9876	0,9983	0,9884	0,9914
UFSJ	W1	0,9996	1,0000	0,9745			0,9914	0,9870
	W2		0,9995	0,9645	1,0000		0,9880
	W3			0,9631	1,0000		0,9815
UFSM	W1	0,9733	0,9887	0,9726			0,9782	0,9641
	W2		0,9788	0,9439	0,9644		0,9624
	W3			0,9426	0,9564	0,9561	0,9517
UFT	W1	0,9130	1,0000	0,9395			0,9509	0,9401
	W2		1,0000	0,9201	0,9085		0,9429
	W3			0,9282	0,9127	0,9386	0,9265
UFTM	W1	1,0000	1,0000	1,0000			1,0000	0,9988
	W2		1,0000	1,0000	0,9944		0,9981
	W3			1,0000	0,9943	1,0000	0,9981
UFU	W1	1,0000	0,9657	0,9752			0,9803	0,9754
	W2		0,9580	0,9701	0,9849		0,9710
	W3			0,9701	0,9793	0,9758	0,9751
UFV	W1	1,0000	0,9703	0,9623			0,9775	0,9733
	W2		0,9894	0,9708	0,9675		0,9759
	W3			0,9676	0,9675	0,9642	0,9664
UFVJM	W1	1,0000	0,9764	0,9543			0,9769	0,9859
	W2		0,9910	0,9854	1,0000		0,9921
	W3			0,9696	0,9963	1,0000	0,9887
UnB	W1	0,9403	0,9164	0,9990			0,9519	0,9600
	W2		0,8921	0,9873	0,9744		0,9513
	W3			0,9857	0,9737	0,9707	0,9767
UNIFAL	W1	1,0000	1,0000	1,0000			1,0000	0,9981
	W2		0,9828	1,0000	1,0000		0,9943
	W3			1,0000	1,0000	1,0000	1,0000
UNIFAP	W1	0,9286	1,0000	1,0000			0,9762	0,9921
	W2		1,0000	1,0000	1,0000		1,0000
	W3			1,0000	1,0000	1,0000	1,0000
UNIFEI	W1	0,9530	0,9567	1,0000			0,9699	0,9806
	W2		0,9334	0,9895	1,0000		0,9743
	W3			0,9925	1,0000	1,0000	0,9975
UNIFESP	J1	1,0000		1,0000			1,0000	1,0000
	J2			1,0000	1,0000		1,0000
	J3			1,0000	1,0000	1,0000	1,0000
UNIFESSPA	J1	1,0000	1,0000	1,0000			1,0000	1,0000
	J2		1,0000	1,0000	1,0000		1,0000
	J3			1,0000	1,0000	1,0000	1,0000
UNILA	W1	1,0000	1,0000	1,0000			1,0000	0,9803
	W2		1,0000	0,9633	0,9730		0,9788
	W3			0,9621	0,9805	0,9439	0,9622
UNILAB	J1	1,0000	1,0000	1,0000			1,0000	1,0000
	J2		1,0000	1,0000	1,0000		1,0000
	J3			1,0000	1,0000	1,0000	1,0000
UNIPAMPA	W1	1,0000	1,0000	1,0000			1,0000	0,9996
	W2		1,0000	1,0000	1,0000		1,0000
	W3			0,9981	0,9982	1,0000	0,9988
UNIR	W1	0,7947	1,0000	0,8959			0,8969	0,9179
	W2		1,0000	0,8857	0,9145		0,9334
	W3			0,8948	0,9090	0,9662	0,9233
UNIRIO	W1	0,9668	1,0000	0,9610			0,9759	0,9433
	W2		0,9667	0,8839	0,9411		0,9305
	W3			0,8850	0,9401	0,9453	0,9234
UNIVASF	W1	0,9121	0,9088	0,9206			0,9138	0,9169
	W2		0,8840	0,9192	0,8948		0,8993
	W3			0,9183	0,8970	0,9973	0,9376
UTFPR	J1	1,0000	1,0000	1,0000			1,0000	1,0000
	J2		1,0000	1,0000	1,0000		1,0000
	J3			1,0000	1,0000	1,0000	1,0000
Standard Deviation of Means								0,0423

• Source: Prepared by the authors.

It can be seen that, of the 63 universities analyzed, only 5 of them (UNIFESP, UTFPR, UNILAB, UNIFESSPA and UFABC) maintained efficiency over time. This is evident by considering the average efficiency of the windows, with an indicator of 1.0000. It was also observed, DMUs that showed efficiency in window 1 and a tendency to reduce the level of efficiency along windows 2 and 3, from the analyzed data: 39.68% showed this tendency. This is believed to stem from the effects of the COVID-19 pandemic, as the average Graduation Success Rate displayed a decline during this period. On the other hand, other units showed the opposite movement, showing a trend of increasing efficiency over time. However, the variation indicating these upward and downward trends in the level of technical efficiency reveals that the segment does not exhibit significant variation, showcasing a certain degree of stability, as indicated by the standard deviation (0.0423).

As a way of obtaining a general perception of the performance of universities, we can analyze the average efficiency coefficient of the DMUs for each year, as depicted in Graph 1.

Graph 1
Average efficiency trend over the period

It can be observed that the historical series followed an upward trend between the years 2018 and 2020, displaying a downward trend starting from 2020. This trend appears to be influenced by the decrease in the average Graduate Success Rate, which shifted from 41.3 in 2020 to an average of 37.6.

In order to have a more detailed view of the efficiency of the DMUs, we can divide the universities by geographic region (North, Northeast, Midwest, Southeast and South). Additionally, in accordance with the literature on DEA, a substantial quantity of DMUs could diminish the homogeneity within the analyzed group, and the results may be affected by factors that were not considered in the model (GOLANY; ROLL, 1989).

In Graph 2 below we have the average efficiency coefficient of universities by region, in each year from 2017 to 2021.

Graph 2
Average efficiency trend over the period

Universities in the southern and midwestern regions exhibited the highest average efficiency coefficient, with the southeast region showing negative variations in 2018 and 2021. These results are similar to the findings of Letti, Bittencourt and Vila (2020) who found the Midwest region with the highest efficiency indicators. There was a drop in the efficiency indicator in the period of 2021 in the Southeast region, likely attributable to a significant reduction in the Undergraduate Success Rate, even with higher spending on equivalent students.

Another highlight to be observed in the segmented analysis by region, the number of efficient DMUs (index equal to 1) was higher, which can be attributed to the limited number of units assessed in regions like the Midwest, which can reduce the discriminatory power of the DEA model (GOLANY; ROLL, 1989). However, in the analysis by region, the efficiency indicators showed a slight increase in all regions, which can be explained by the homogeneous grouping in relation to the resources used in the evaluated units, which may present differences between the regions. As an example, the Degree of Student Engagement with Postgraduate Diploma (GEPG) and the Teacher Qualification Index (IQCD) showed significant differences between regions.

Through the composite efficiency analysis, we can infer which university was the most efficient in each year, accordingly, the UFPEL and UFAC institutions emerged as the most efficient in two out of the five years under examination. UFAC was the most efficient university in the application of its resources during the years of the COVID-19 pandemic. Table 5 outlines the universities that demonstrated the utmost efficiency in each year of the study.

Table 5
Most efficient HEIs in the year

2017	2018	2019	2020	2021
UFPEL	UFT	UFPEL	UFAC	UFAC

• Source: Prepared by the authors.

By analyzing the inefficient DMUs in the year 2021 (the most recent period analyzed and the foundation for formulating new strategies) taking as an example the five least efficient units (FURG, UFFS, UFRN, UNILA, UNIRIO), we can determine the target values for their output variables (CAPES/MEC concept for graduate studies and graduation success rate), considering fixed values for the input variables, so that these universities become efficient. The target values are detailed in Table 6.

Tabela 6
Target values for outputs

DMU	CAPES		TSG
	Current	Target	Current	Target
FURG	3,73	4,51	35,36	42,77
UFFS	3,18	3,65	41,64	47,84
UFRN	3,87	4,69	42,89	51,94
UNILA	3,11	4,27	26,27	36,08
UNIRIO	3,62	4,31	32,01	38,13

• Source: Prepared by the authors.

Finally, it is possible to determine which units better represent models to be followed, in other words, benchmarks for the inefficient units. This result is presented in Table 7.

Table 7
Benchmarks of inefficient units

Not efficient university	Benchmarks
FURG	UFCA e UFMG
UFFS	UFRA
UFRN	UFRGS
UNILA	UFCA
UNIRIO	UFRGS

• Source: Prepared by the authors.

Final considerations

The aim of this study was to assess the efficiency of public universities during the period from 2017 to 2021. To accomplish this, the Data Envelopment Analysis (DEA) approach was employed, utilizing panel data, to establish the efficiency score.

Through the conducted study, it is possible to conjecture that the COVID-19 pandemic, which peaked in the years 2020 and 2021, and compulsorily altered the teaching approach, rendering traditional in-person classroom teaching impossible, impacted the efficiency of Brazilian federal universities, which presented a reduction of 0.95% in its average. This sudden transition forced universities to swiftly adapt to online education (EAD) or suspend activities.

Another significant insight from the study is the performance of universities by region, the North region of the country has the best average efficiency coefficient among all regions, and therefore, one should study what are the common management practices in universities in this region and, if they make sense, they could potentially be implemented in institutions from other regions.

This study has succeeded in indicating target values for performance indicators for inefficient federal universities and in identifying which efficient units can serve as the most fitting models for them to follow.

Future studies may explore a comparison between the performance evaluation based on the TCU indicators and an evaluation grounded in other indicators found in the literature.

Acknowledgments

Thanks to the Federal Institute of Espírito Santo for encouraging and supporting the research.

• Translation and review: Caio Moreira Salles Martins E-mail: [e-mail]
  Victor Luís Silva Santos E-mail: [e-mail]

Referências

• AGASISTI, Tommaso et al. Efficiency of regional higher education systems and regional economic short-run growth: empirical evidence from Russia. Industry and Innovation, USA, v. 28, n. 4, p. 507–534, 2021. Disponível em: https://doi.org/10.1080/13662716.2020.1738914. Acesso em: 14 fev. 2023.
  » https://doi.org/10.1080/13662716.2020.1738914.
• BANKER, Rajiv D.; CHARNES, Abraham; COOPER, William Wager. Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management science, USA, v. 30, n. 9, p. 1078–1092, 1984. Disponível em: https://doi.org/https://doi.org/10.1287/mnsc.30.9.1078. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.1287/mnsc.30.9.1078.
• BARBOSA, Maria Ligia de Oliveira. Democratização ou massificação do Ensino Superior no Brasil? Revista de Educação PUC-Campinas, Campinas, v. 24, n. 2, p. 240–253, 2019. Disponível em: https://doi.org/https://doi.org/10.24220/2318-0870v24n2a4324. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.24220/2318-0870v24n2a4324.
• BENTO, Fabio et al. Resilience in higher education: a complex perspective to lecturers’ adaptive processes in response to the COVID-19 pandemic. Education Sciences, USA, v. 11, n. 9, p. 492, 2021. Disponível em: https://doi.org/https://doi.org/10.3390/educsci11090492. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.3390/educsci11090492.
• BRASIL. TRIBUNAL DE CONTAS DA UNIÃO. Acórdão n^o 408 Brasília: TCU, 2002. Disponível em: https://pesquisa.apps.tcu.gov.br/documento/acordao-completo/1622919991.PROC/%2520/DTRELEVANCIA%2520desc%252C%2520NUMACORDAOINT%2520desc/2/%2520?uuid=dc7cb700-1eca-11eb-92c6-0f3f4797229c. Acesso em: 14 fev. 2023.
  » https://pesquisa.apps.tcu.gov.br/documento/acordao-completo/1622919991.PROC/%2520/DTRELEVANCIA%2520desc%252C%2520NUMACORDAOINT%2520desc/2/%2520?uuid=dc7cb700-1eca-11eb-92c6-0f3f4797229c.
• BRASIL. TRIBUNAL DE CONTAS DA UNIÃO. Orientações para o cálculo dos indicadores de gestão. Brasília: SESU, 2004. p. 9. Disponível em: http://portal.mec.gov.br/sesu/arquivos/pdf/indicadores.pdf. Acesso em: 14 fev. 2023.
  » http://portal.mec.gov.br/sesu/arquivos/pdf/indicadores.pdf.
• BRASIL. TRIBUNAL DE CONTAS DA UNIÃO. Acórdão n^o 1.043 Brasília: TCU, 2006. Disponível em: https://pesquisa.apps.tcu.gov.br/documento/acordao-completo/1043%252F2006/%2520/DTRELEVANCIA%2520desc%252C%2520NUMACORDAOINT%2520desc/0/%2520?uuid=dc7cb700-1eca-11eb-92c6-0f3f4797229c. Acesso em: 14 fev. 2023.
  » https://pesquisa.apps.tcu.gov.br/documento/acordao-completo/1043%252F2006/%2520/DTRELEVANCIA%2520desc%252C%2520NUMACORDAOINT%2520desc/0/%2520?uuid=dc7cb700-1eca-11eb-92c6-0f3f4797229c.
• BRASIL. Censo da Educação Superior 2020: notas estatísticas. Brasília: INEP, 2022. Disponível em: https://download.inep.gov.br/publicacoes/institucionais/estatisticas_e_indicadores/notas_estatisticas_censo_da_educacao_superior_2020.pdf. Acesso em: 14 fev. 2023.
  » https://download.inep.gov.br/publicacoes/institucionais/estatisticas_e_indicadores/notas_estatisticas_censo_da_educacao_superior_2020.pdf.
• CARVALHAES, Flavio; MEDEIROS, Marcelo; TAGLIARI, Clarissa. Expansão e diversificação do ensino superior: privatização, educação a distância e concentração de mercado no Brasil, 2002-2016. Available at SSRN 3892300, 2021. Disponível em: https://doi.org/https://dx.doi.org/10.2139/ssrn.3892300. Acesso em: 14 fev. 2023.
  » https://doi.org/https://dx.doi.org/10.2139/ssrn.3892300.
• CHARNES, A.; COOPER, W. W.; RHODES, E. Measuring the efficiency of decision making units. European Journal of Operational Research, USA, v. 2, n. 6, p. 429–444, 1978. Disponível em: https://doi.org/10.1016/0377-2217(78)90138-8. Acesso em: 14 fev. 2023.
  » https://doi.org/10.1016/0377-2217(78)90138-8.
• COLOMBO, Sonia Simões; RODRIGUES, Gabriel Mario. Desafios da gestão universitária contemporânea Porto Alegre: Artmed, 2011. E-book Disponível em: https://books.google.com.br/books?id=57FRpVuaJS0C&lpg=PA43&ots=Gun1fHHR63&dq=ensino%20privado%3A%20Desafios%20da%20gest%C3%A3o%20universit%C3%A1ria%20contempor%C3%A2nea&lr&hl=pt-BR&pg=PR1#v=onepage&q&f=false. Acesso em: 14 fev. 2023.
  » https://books.google.com.br/books?id=57FRpVuaJS0C&lpg=PA43&ots=Gun1fHHR63&dq=ensino%20privado%3A%20Desafios%20da%20gest%C3%A3o%20universit%C3%A1ria%20contempor%C3%A2nea&lr&hl=pt-BR&pg=PR1#v=onepage&q&f=false.
• DAULTANI, Yash; DWIVEDI, Ashish; PRATAP, Saurabh. Benchmarking higher education institutes using data envelopment analysis: capturing perceptions of prospective engineering students. Opsearch, USA, v. 58, n. 4, p. 773–789, 2021. Disponível em: https://doi.org/https://doi.org/10.1007/s12597-020-00501-5. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.1007/s12597-020-00501-5.
• DE CASTRO CAMIOTO, Flávia; MARIANO, Enzo Barberio; DO NASCIMENTO REBELATTO, Daisy Aparecida. Efficiency in Brazil’s industrial sectors in terms of energy and sustainable development. Environmental Science & Policy, USA, v. 37, p. 50–60, 2014. Disponível em: https://doi.org/https://doi.org/10.1016/j.envsci.2013.08.007. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1016/j.envsci.2013.08.007.
• DUAN, Sophia Xiaoxia. Measuring university efficiency: An application of data envelopment analysis and strategic group analysis to Australian universities. Benchmarking: an international journal, USA, v. 26, n. 4, p. 1161–1173, 2019. Disponível em: https://doi.org/https://doi.org/10.1108/BIJ-10-2017-0274. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.1108/BIJ-10-2017-0274.
• DUFRECHOU, Paola Azar. The efficiency of public education spending in Latin America: A comparison to high-income countries. International Journal of Educational Development, USA, v. 49, p. 188–203, 2016. Disponível em: https://doi.org/https://doi.org/10.1016/j.ijedudev.2016.03.005. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1016/j.ijedudev.2016.03.005.
• FERREIRA, Carlos Maurício de Carvalho; GOMES, Adriano Provezano. Introdução à análise envoltória de dados: teoria, modelos e aplicações. 2. ed. Viçosa: UFV, 2020.
• GOLANY, Boaz; ROLL, Yaakov. An application procedure for DEA. Omega, USA, v. 17, n. 3, p. 237–250, 1989. Disponível em: https://doi.org/https://doi.org/10.1016/0305-0483(89)90029-7. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.1016/0305-0483(89)90029-7.
• GOMES, Válter; MACHADO-TAYLOR, Maria de Lourdes; SARAIVA, Ernani Viana. O ensino superior no brasil-breve histórico e caracterização. Ciência & Trópico, Recife, v. 42, n. 1, 2018. Disponível em: https://periodicos.fundaj.gov.br/CIC/article/view/1647/1395. Acesso em: 14 fev. 2023.
  » https://periodicos.fundaj.gov.br/CIC/article/view/1647/1395.
• GRALKA, Sabine; WOHLRABE, Klaus; BORNMANN, Lutz. How to measure research efficiency in higher education? Research grants vs. publication output. Journal of Higher Education Policy and Management, USA, v. 41, n. 3, p. 322–341, 2019. Disponível em: https://doi.org/https://doi.org/10.1080/1360080X.2019.1588492. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.1080/1360080X.2019.1588492.
• HAMMES JUNIOR, David Daniel; FLACH, Leonardo; MATTOS, Luísa Karam de. The efficiency of public expenditure on Higher Education: a study with Brazilian Federal Universities. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 28, p. 1076–1097, 2020. Disponível em: https://doi.org/https://doi.org/10.1590/S0104-40362020002802573. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.1590/S0104-40362020002802573.
• KAUR, Harpreet. Assessing Technical Efficiency of the Indian Higher Education: An Application of Data Envelopment Analysis Approach. Higher Education for the Future, USA, v. 8, n. 2, p. 197–218, 2021. Disponível em: https://doi.org/https://doi.org/10.1177/23476311211011932. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1177/23476311211011932.
• LEE, Boon L.; JOHNES, Jill. Using network DEA to inform policy: The case of the teaching quality of higher education in England. Higher Education Quarterly, USA, v. 76, n. 2, p. 399– 421, 2022. Disponível em: https://doi.org/https://doi.org/10.1111/hequ.12307. Acesso em: 14 fev. 2023.
  » https://doi.org/https://doi.org/10.1111/hequ.12307.
• LETTI, Ariel Gustavo; BITTENCOURT, Mauricio Vaz Lobo; VILA, Luis E. A Comparative Analysis of Federal University Efficiency Across Brazilian Regions (2010-2016). Revista Brasileira de Gestão e Desenvolvimento Regional, São Paulo, v. 16, n. 1, 2020. Disponível em: https://www.rbgdr.net/revista/index.php/rbgdr/article/view/5385. Acesso em: 14 fev. 2023.
  » https://www.rbgdr.net/revista/index.php/rbgdr/article/view/5385.
• MEC - Ministério da Educação. Cadastro Nacional de Cursos e Instituições de Educação Superior Cadastro e-MEC Brasília, 2023. Disponível em: https://emec.mec.gov.br/emec/nova. Acesso em: 14 fev. 2023.
  » https://emec.mec.gov.br/emec/nova.
• MEZA, Lidia Angulo; BIONDI NETO, Luiz; RIBEIRO, Paulo Guilherme. SIAD V.2.0. – sistema integrado de apoio à decisão: uma implementação de modelos de análise envoltória de dados e um método multicritério. In: Simposio Brasileiro de Pesquisa Operacional, 37., 2005, Gramado. Anais [...]. Gramado: SOBRAPO, 2005. p. 406–414. Disponível em: http://www.din.uem.br/sbpo/sbpo2005/pdf/arq0175.pdf. Acesso em: 28 ago. 2023.
  » http://www.din.uem.br/sbpo/sbpo2005/pdf/arq0175.pdf.
• MONCAYO–MARTÍNEZ, Luis A.; RAMÍREZ–NAFARRATE, Adrián; HERNÁNDEZ–BALDERRAMA, María Guadalupe. Evaluation of public HEI on teaching, research, and knowledge dissemination by Data Envelopment Analysis. Socio-Economic Planning Sciences, USA, v. 69, p. 100718, 2020. Disponível em: https://doi.org/https://doi.org/10.1016/j.seps.2019.06.003. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1016/j.seps.2019.06.003.
• MONT’ALVÃO NETO, A. L. More inclusion than diversion: expansion, differentiation, and market structure in higher education. Revista Brasileira de Ciências Sociais, São Paulo, v. 31, n. 92, 2016. Disponível em: https://doi.org/https://doi.org/10.7666/319211/2016. Acesso em: 11 mar. 2023.
  » https://doi.org/https://doi.org/10.7666/319211/2016.
• NAVAS, Lina P. et al. Colombian higher education institutions evaluation. Socio-Economic Planning Sciences, USA, v. 71, p. 100801, 2020. Disponível em: https://doi.org/https://doi.org/10.1016/j.seps.2020.100801. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1016/j.seps.2020.100801.
• NOJAVAN, Majid; HEIDARI, Atefeh; MOHAMMADITABAR, Davood. A fuzzy service quality based approach for performance evaluation of educational units. Socio-Economic Planning Sciences, USA, v. 73, p. 100816, 2021. Disponível em: https://doi.org/https://doi.org/10.1016/j.seps.2020.100816. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1016/j.seps.2020.100816.
• PAPADIMITRIOU, Maria; JOHNES, Jill. Does merging improve efficiency? A study of English universities. Studies in Higher Education, USA, v. 44, n. 8, p. 1454–1474, 2019. Disponível em: https://doi.org/https://doi.org/10.1080/03075079.2018.1450851. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1080/03075079.2018.1450851.
• PARENTE, Paulo Henrique Nobre et al. Eficiência e produtividade nos Institutos Federais de Educação, Ciência e Tecnologia do Brasil. Administração Pública e Gestão Social, [S. l.], v. 13, n. 1, 2021. Disponível em: https://www.redalyc.org/journal/3515/351564966006/html/. Acesso em: 28 ago. 2023.
  » https://www.redalyc.org/journal/3515/351564966006/html/.
• QUIROGA-MARTÍNEZ, Facundo; FERNÁNDEZ-VÁZQUEZ, Esteban; ALBERTO, Catalina Lucía. Efficiency in public higher education on Argentina 2004–2013: Institutional decisions and university-specific effects. Latin American Economic Review, USA, v. 27, n. 1, p. 1–18, 2018. Disponível em: https://doi.org/https://doi.org/10.1186/s40503-018-0062-0. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1186/s40503-018-0062-0.
• TAVARES, Rafael Santos; MEZA, Lidia Angulo. Performance evaluation of undergraduate courses at a Brazilian Federal University. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 29, p. 206–233, 2020. Disponível em: https://doi.org/https://doi.org/10.1590/S0104-40362020002802223. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1590/S0104-40362020002802223.
• TRAN, Carolyn-Dung T. T.; VILLANO, Renato A. Measuring efficiency of Vietnamese public colleges: an application of the DEA-based dynamic network approach. International Transactions in Operational Research, USA, v. 25, n. 2, p. 683–703, 2018. Disponível em: https://doi.org/https://doi.org/10.1111/itor.12212. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1111/itor.12212.
• VISBAL-CADAVID, Delimiro; MENDOZA, Adel Mendoza; HOYOS, Iván Quintero. Prediction of efficiency in Colombian higher education institutions with data envelopment analysis and neural networks. Pesquisa Operacional, Rio de Janeiro, v. 39, p. 261–275, 2019. Disponível em: https://doi.org/https://doi.org/10.1590/0101-7438.2019.039.02.0261. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1590/0101-7438.2019.039.02.0261.
• WHEELOCK, David C.; WILSON, Paul W. Non-parametric, unconditional quantile estimation for efficiency analysis with an application to Federal Reserve check processing operations. Journal of Econometrics, USA, v. 145, n. 1–2, p. 209–225, 2008. Disponível em: https://doi.org/https://doi.org/10.1016/j.jeconom.2008.05.007. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1016/j.jeconom.2008.05.007.
• WITTE, Kristof De; LÓPEZ-TORRES, Laura. Efficiency in education: A review of literature and a way forward. Journal of the operational research society, USA, v. 68, n. 4, p. 339–363, 2017. Disponível em: https://doi.org/https://doi.org/10.1057/jors.2015.92. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1057/jors.2015.92.
• WOICOLESCO, Vanessa G.; MOROSINI, Marilia; MARCELINO, Jocélia M. COVID-19 and the crisis in the internationalization of higher education in emerging contexts. Policy futures in Education, USA, v. 20, n. 4, p. 433–442, 2022. Disponível em: https://doi.org/https://doi.org/10.1177/14782103211040913. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1177/14782103211040913.
• WU, Jie et al. An efficiency analysis of higher education institutions in China from a regional perspective considering the external environmental impact. Scientometrics, USA, v. 122, p. 57–70, 2020. Disponível em: https://doi.org/https://doi.org/10.1007/s11192-019-03296-5. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1007/s11192-019-03296-5.
• ZHANG, Ganggang; WU, Jie; ZHU, Qingyuan. Performance evaluation and enrollment quota allocation for higher education institutions in China. Evaluation and Program Planning, USA, v. 81, p. 101821, 2020. Disponível em: https://doi.org/https://doi.org/10.1016/j.evalprogplan.2020.101821. Acesso em: 28 ago. 2023.
  » https://doi.org/https://doi.org/10.1016/j.evalprogplan.2020.101821.

Publication Dates

• Publication in this collection
  13 Oct 2023
• Date of issue
  2023

History

• Received
  07 June 2023
• Accepted
  31 July 2023
• Reviewed
  30 Aug 2023