SCIENTIFIC PRODUCTION ASSESSMENT IN A COMPUTER SCIENCE DEPARTMENT: A CASE STUDY

AVALIAÇÃO DA PRODUÇÃO CIENTÍFICA EM UM DEPARTAMENTO DE COMPUTAÇÃO: UM ESTUDO DE CASO

ABSTRACT

Bibliometric evaluation is a crucial research topic that finds various applications ranging from faculty productivity evaluation and assessing researcher career trajectory to analyzing trends and emerging themes in a research area. This study examined the CVs of 36 researchers and professors from a Brazilian university’s computer science faculty. The publication data obtained from Plataforma Lattes, an online repository of Brazilian researchers' CVs, was used. The objective of this study was to evaluate the productivity of each professor in quantity and quality, over a period of five years, along with analyzing the co-authorship network. In addition, the study examined the correlation between researcher's productivity and the Research Productivity Grants (Bolsas de Produtividade em Pesquisa - PQ grant) since the Brazilian National Council for Scientific and Technological Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) considers the professor's curriculum as one of the criteria for granting the grant.

Key words:
Bibliometrics; Data analysis; Co-authorship network; Pattern recognition

RESUMO

A avaliação bibliométrica é um tema de pesquisa importante e com múltiplas aplicações, desde a avaliação da produtividade do corpo docente e da trajetória de carreira do pesquisador, até a análise de tendências e temas emergentes em uma área de pesquisa. Neste estudo analisamos os currículos de 36 pesquisadores brasileiros e professores de uma faculdade de computação. Foram utilizados dados de publicação extraídos da Plataforma Lattes, um repositório online de currículos de pesquisadores brasileiros. Nosso objetivo foi avaliar a produtividade de cada professor, tanto em quantidade quanto em qualidade, ao longo de cinco anos, bem como a rede de coautoria produzida. Adicionalmente, investigamos se a produtividade do pesquisador é um bom preditor para a Bolsas de Produtividade em Pesquisa - PQ, pois, segundo o Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), um dos critérios para a concessão desta bolsa é o currículo do professor.

Palavras chaves:
Bibliometria; Análise de dados; Rede de coautoria; Reconhecimento de padrões

1 INTRODUCTION

Cañibano and Bozeman (2009) categorized curriculum vitae (CVs) studies into three major areas: career trajectories, mobility, and collective work analysis. The studies assist in understanding how knowledge and research themes develop in a research field, how topics fade when a new relevant subject emerges, the productivity of researchers throughout their careers and trajectories, and the co-authorship dynamics across different departments (Mathew; Agrawal; Menzies, 2017; Vasilescu et al., 2014).

The Lattes Platform is an online repository of résumés belonging to Brazilian researchers. This freely accessible platform contains details related to the education, research, career, and advisory role of over six million scientists, including professors, researchers, undergraduates, graduates, and Philosophy Doctor (Ph.D). students. Numerous institutions rely on this data for recruitment, researcher evaluation and distribution of financial research support, amongst other intended purposes. This encourages users to keep their CVs updated and thorough. As a result, it offers a substantial amount of reliable data about research in Brazil.

For this study, we collected and selected a list of research papers published in scientific journals and conference proceedings (complete works) based on the Lattes Résumés of 36 professors from a computer science department. The Qualis index, which classifies journals and conferences based on relevance, was obtained for each paper. We evaluated the productivity of each professor, in terms of both quantity and quality, over a period of five years using data from each paper such as the author's list, title, publication venue (journal or conference), and year of publication. We examined the inequality of scientific production among researchers and their preferences for publication venues (journals or conferences). The co-authorship network was analyzed in two scenarios: internal, which examines the frequency of collaboration among professors within the same department to publish a new paper, and external, which looks at collaborations that may involve professors and students from other institutions and academic levels - including undergraduates, graduate students, and PhD candidates. Moreover, we examined whether a researcher's productivity can accurately predict the awarding of a Productivity Scholarships (PQ). It is worth noting that the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), which is the Brazilian national council for Scientific and Technological development, cites the professor's curriculum among the criteria used to award this scholarship.

The structure of this paper is as follows: In the “related work” section, we explain the method we used to collect, select, and structure the data on professors' productivity. In the “results and discussion” section, we present our productivity analysis for each professor. Lastly, the “conclusion” section provides the concluding remarks for the paper.

2 RELATED WORK

Bibliometric assessment is a broad research topic. It has a wide range of applications, including faculty productivity, the analysis of emerging trends and themes, among others. In (Wong et al., 2021), a bibliometric assessment of software engineering was present as in this study which examined various topics, researchers, and institutes by utilizing a systematic mapping technique on reputable software engineering resources. The comparison of different years allowed the authors to identify emerging trends and themes, thereby offering more insight into the field of software engineering. Mathew, Agrawal and Menzies (2017) employed topic analysis to summarize scientific papers in the field of software engineering. As a result, potential trends were detected in the research community.

Way et al. (2017) analyzed faculty productivity in the North American Computer Science market. Individual trajectories are typically described in the conventional narrative as an obsolescent function. Nevertheless, the authors concluded that most trajectories do not conform to the conventional narrative and that a piecewise linear model consisting of two linear functions is better suited to this task. A similar analysis was also conducted using data from the Brazilian Computer Science community (Albertini; Backes; Sá, 2019). In addition to analyzing the individual trajectories of senior researchers, the authors investigated the impact of institutional prestige ranks on annual publication rates.

Bordin, Gonçalves and Todesco (2014) utilized data from a postgraduate department to construct a collaboration network, wherein, they analyzed various metrics from the network, including the average distance between collaborators, the most collaborative authors, the network's density, and the number of components. The authors argued that this information could assist decision-making at both organizational and individual levels. Silva et al. (2020) examined the co-authorship networks of Brazilian academic graduate programs in computer science. They analyzed the network obtained by linking researchers through common publications in terms of topology by using complex network measurements. Regarding program characterization, the authors were able to indicate the most relevant topological measures of the network for this task.

Another line of research concerns the productivity profile of scholarship researchers, such as those receiving scientific PQ. Fagundes (2020) examined PQ researchers in Physical Education over a five-year span from 2015 to 2019. Their findings reveal that most researchers come from the southeast region, which boasts the highest Human Development Index (HDI) in Brazil, and that the PQ-2 scholarship level has a higher prevalence among males. Thus, this scholarship level is the one with the highest number of researchers. Similarly, a relevant study by Sacco et al. (2016) concentrated on the field of Psychology where the authors examined the characteristics of 338 PQ scholarships over a period of three years, from 2012 to 2014. As found in the field of Physical Education, most PQ grants (55.3%) are awarded to scholars in the southeastern region. The authors of the study also found that only ten universities account for 56.7% of PQ grants awarded to researchers, among whom the majority are women.

The study presented in (Castioni; Melo; Afonso, 2020) focuses on Education. The analysis of PQ grants distribution shows that most grant recipients are in the Southeast and South regions. Geographically, these regions concentrate most of the federal and public universities in Brazil. The genealogy of PQ scholarship researchers is examined in (Oliveira et al., 2018). Thus, these authors aimed at mapping the dissemination of knowledge through a network of co-authorship, and the contributions of the researchers to the training of human resources.

3 DATA PREPARATION

In this section, the data collection process of our study will be described in detail. Initially, we collected Lattes Résumés of 65 professors from a computer science department in June 2019. Professors in Brazilian universities are responsible for various tasks, primarily teaching, research, and administration. Due to this, the professors allocate more time to certain areas than others. Since we are concerned with the scientific production of professors, it would not be prudent to compare a professor dedicated to research with one whose work is primarily focused on other tasks. To achieve this goal, we excluded five professors whose Lattes Résumés were outdated, i.e., résumés that were last updated before 2018. We excluded nine professors who did not have a Ph.D., which is an important requirement to supervise graduate students in Brazilian universities.

We also excluded 15 recently hired professors, i.e., professors who were hired after 2014. There were two main reasons why we opted to exclude them: i) we want to analyze a five-year research period of professors, an attribute that they lack, and ii) part of their scientific production was produced during their Ph.D. and not as a consolidated professor and researcher.

Lattes Résumés contain various academic information such as scientific research, artistic production, and articles in newspapers/magazines. Therefore, for this study, we extracted the list of papers published in scientific journals and conference proceedings (complete works) from the 36 remaining Lattes Résumés. The paper is characterized by its author list, title, journal or conference, year of publication, and Qualis index. The latter refers to the Qualis index, which is an official Brazilian classification of journals and conferences maintained by the Coordenadoria de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), a government agency affiliated with the Brazilian Ministry of Education. This classification system groups journals and conferences into nine different levels according to their relevance: These levels are: S (no classification), C, B5, B4, B3, B2, B1, A2, and A1 (highest classification).

4 RESULTS AND DISCUSSION

Data from 629 papers published between 2014 and 2018 (a five-year span) was analyzed. The distribution of papers by each researcher over the years is presented in Figure 1. Significant differences exist in the publication records of researchers, particularly in terms of journal articles. Publishing at conferences is a more agile approach preferred by most researchers. Journal articles may require multiple revisions before publication, while conference papers usually undergo a single review step to determine acceptance. As a result, many researchers present a high number of papers at conferences over the years, while only a few have consistent productivity in journals. Thus, between 2014-2018, researchers had an average of 4.83 and 12.17 publications in journals and conferences, respectively. Figure 2 shows that, on average, 2.52 papers were published in conferences for each paper published in a journal. In general, conferences tend to have higher numbers of papers published than journals; 77.78% of researchers present fewer papers published in journals compared to conferences.

Figure 1
Number of papers published by researchers from 2014 to 2018.

Figure 2
Proportion of papers published in journals and conferences by each researcher from 2014 to 2018.

We also assessed the inequality of scientific output among researchers. Figure 3 indicates that the distribution of publications follows a similar pattern over time, with approximately 80% of the publications being produced by half of the researchers. When evaluated on a yearly basis, these findings are somewhat consistent with Price’s Law, which states that half of the publications are contributed by the square root of the total number of authors. In our case, six authors represent 16.66% of the total number of authors, and they are responsible for approximately 35% of the publications produced in the period studied. Over the five-year period studied, this results in a Gini coefficient of 0.4593, with zero indicating perfect equality.

Figure 3
Distribution of publications over the five years period. Gini coefficient = 0.4593.

Figure 4
Number of papers published by researchers in each Qualis level from 2014 to 2018.

Figure 5
Proportion of papers published by researchers in the top-3 Qualis levels (A1-A2-B1) from 2014 to 2018.

Researchers are expected to publish their findings in academic journals and conferences throughout their academic careers. While the number of published papers is a significant measure of a researcher's performance, the quality of research is equally important. Successful researchers are expected to have a significant number of publications, and these papers should be accepted in highly regarded academic journals and conferences. To evaluate the papers, we utilized the Qualis index, which classifies academic journals and conferences according to their relevance.

The distribution of papers in the Qualis index of each researcher over the years is depicted in Figure 4. It was observed that only a small number of researchers manage to publish in highly regarded journals. One would expect authors with a low journal production to be those who aim to publish in top-ranked journals, which typically have a more rigorous review process. Thus, it was observed that many authors with a low production volume chose to publish their work in journals with lower Qualis classifications. In terms of conference publications, it is worth noting that there has been an improvement in the quality, as there is now a significant proportion of papers being published in highly regarded conferences.

Nevertheless, when accounting for both journals and conferences, most researchers still exhibit a low performance at a higher Qualis level. Figure 5 indicates that less than a third of the researchers have over 50% of their papers accepted in the top Three Qualis levels (A1-A2-B1). The complete distribution of papers for each Qualis level is presented in Figure 6. The distribution across the Qualis levels is significantly imbalanced among the researchers, with several individuals publishing a considerable portion of their scientific research at lower Qualis levels. Figure 7 demonstrates that the distribution in the top five Qualis levels displays a similar tendency over the years, with approximately 90% of publications generated by only 45% of the researchers. An annual analysis reveals that this distribution adheres to Price's Law, which means only six authors are accountable for half of the publications in the top five Qualis levels. During the examined five-year period, this results in a Gini coefficient of 0.6455 (with zero signifying absolute equality), which is a more inequitable outcome than that obtained for the number of published papers.

The number of co-authors over the years was evaluated as demonstrated in Figure 8. The term “co-author” in this study refers to any person who collaborates on a research paper, whether it is a fellow researcher, an undergraduate, a graduate, or a Ph.D. student. The majority of professors have maintained a consistent number of co-authors over the years. Generally speaking, two-thirds of the professors have a maximum of ten co-authors per year, although one professor extrapolated to 100 co-authors due to research in genome analysis.

Figure 6
Proportion of papers published by researchers in each Qualis level from 2014 to 2018.

Figure 7
Distribution of paper in the top-5 Qualis levels over the five years period. Gini coefficient = 0.6455.

Figure 8:
Number of co-authors over the years. 2/3 of the professors have a maximum of 10 co-authors per year.

We also studied the frequency of collaborations among professors from the same department to publish a unique paper. As depicted in Figure 9, there has been a rise in collaboration over the years. As time unwinds, not only does the number of collaborations increase, but the collaborations also change from previous years. This suggests that collaboration is more diverse, as there are researchers with multiple collaborations across different research topics, rather than just thematic research groups publishing together. However, understanding how these researchers are connected is the first step towards improving departmental collaboration. Two authors who are not otherwise connected, but share co-authors, likely have similar research interests. This data may aid in identifying common research areas and facilitating new collaborations between researchers.

Figure 9
Internal collaboration network.

The co-authorship network was also analyzed, as shown in Figure 10. Building this network required identifying authors with the same name across different curriculums. We first applied the Levenshtein distance method to identify papers with identical titles. Out of the 629 papers, 504 unique papers were identified over the course of five years. The next step involved identifying and extracting the names of researchers in each unique paper. Due to the possibility of errors in names, such as variations in the spelling or accents, we also employed Levenshtein distance to identify unique names and the existence of homonyms (different researchers with similar or identical names). This was later rectified manually. Consequently, we identified a total of 1079 co-authors in 504 unique papers. Based on this data, we created a co-author network. In this network, two researchers (or nodes) are connected by an edge if they have shared publications.

Table 1 shows several measurements computed from the co-author network. As time passes, collaboration among researchers increases, resulting in fewer but more extensive connected components. Despite the number of nodes being stable over the years, the total number of nodes observed in the five-year period demonstrates that the network has a dynamic structure, with edges forming and breaking over time. This can be partially explained by the network's inclusion of undergraduate, graduate, and doctorate students who participate in a specific research topic. Nodes are removed upon completion of the graduate course or research topic. The network's dynamic is also influenced by researchers collaborating in multiple universities.

As with department collaboration, understanding the connections among researchers is crucial for decision making. Researchers from Brazil and other countries are part of the co-authorship network. As mentioned earlier, two authors who are not directly linked, but have common co-authors, are likely to have similar research interests. Identifying these shared research topics enables proposing new collaborations among authors from different universities, which can enhance the national and international performance of the department and, in turn, the university.

Figure 10
Co-author network computed for the five years period (2014-2018). In red: authors whose Lattes Résumés were collected.

Table 1
Co-author network measurements.

Year	# nodes	# of components	Largest component	% largest component	Average clustering	Max degree centrality
2014	277	21	77	27.80	0.88	0.16
2015	347	18	143	41.21	0.91	0.41
2016	257	18	35	13.62	0.88	0.12
2017	371	19	164	44.20	0.88	0.44
2018	304	13	156	51.32	0.84	0.28
2014-2018	1079	2	1072	99.35	0.89	0.34

• Source: by the autor (2022).

In 2007, the Brazilian government implemented the Programa de Apoio a Planos de Reestruturação e Expansão das Universidades Federais (REUNI) to expand federal universities in Brazil. One directive of this program aimed to expand undergraduate courses and increase access to them for students. To achieve this, universities hired more professors to provide instruction for newly created courses. As noted in our study, this resulted in faculty renewal in some departments. As shown in Figure 11, only nine of the 36 selected professors were employed before the implementation of this program (highlighted in red). Longevity as a professor allows for the creation of a collaborative network that can positively impact the number of publications. Additionally, faculties require professors to affiliate with a graduate program to serve as an advisor. Subsequently, as an advisor, he is expected to have a higher publication rate. Among the 36 professors in our sample, nine have never affiliated with a graduate program (black), while five have affiliated for less than three years (green). The remaining professors have had a minimum of five years of affiliation, except for professor #35. As anticipated, a professor's publication rate can be positively impacted even by a brief period of serving as an advisor.

Figure 11
The number of papers over the five-year period. In red: hired prior to 2008; in black: never affiliated with a graduate program; in green: less than 3 years affiliation with a graduate program; in blue: hired since 2008 with at least 5 years affiliation with a graduate program.

Figure 12:
Decision tree obtained for the PQ scholarship (class 2, 95.3125% accuracy).

The promotion of scientific and technological research in Brazil is driven by the Brazilian national council for Scientific and Technological development a.k.a CNPq. Among the various forms of research incentives, the PQ grant is awarded by the CNPq to the most productive researchers. One of the criteria for the PQ grant is the professor's curriculum. Therefore, we evaluated if it is possible to predict a professor with the propensity to be eligible for the PQ grant based on the Qualis index of publications in the last five years. Hence, we represented each professor as a feature vector consisting of nine descriptors, each of which denotes the number of papers published by the professor in the respective Qualis level.

In 2019, four out of the 36 professors analyzed were awarded this grant. To address the class imbalance, we applied the Synthetic Minority Over-sampling Technique (SMOTE) (BOWYER et al, 2011) by selecting the five nearest neighbors for each minority class sample. We evaluated our data using the C4.5 algorithm in a 10-fold cross-validation scheme (Quinlan, 1993). We opted to use a decision tree as it provides a more interpretable model while selecting the most discriminating attributes for the problem. Figure 12 shows the decision tree obtained and the two attributes used to classify the data, both related to higher classification levels (first and third levels). This model can predict scholarship grants (class 2) with 95.3125% accuracy, thus indicating that publications in a high-quality venue, according to the Brazilian official classification of journals and conferences, are a strong indicator for obtaining this type of scholarship. Figure 13 presents the scatterplot of the two most significant attributes (Qualis levels).

Although a small number of samples (i.e., a single department) limit this analysis, its fundamental principles may be applicable to other scenarios. Employing an interpretable model makes it possible to identify potential candidates for a PQ scholarship and assists researchers to publish in high-quality venues, thereby increasing their chances of obtaining a grant. In a way, the PQ grant is a measure of prestige for both the researcher and the department/university. Therefore, it represents a form of recognition that could enhance future collaborations and the overall perception of the researcher and the department.

Figure 13
Scatterplot of the two most significant attributes (Qualis levels) to decide for the PQ scholarship.

5 CONCLUSIONS

This paper evaluates the résumés of professors from a computer science department in Brazil, authored by Brazilian researchers. The study utilized data from Lattes Platform, an online database of Brazilian researchers' résumés, and Qualis index, a classification system of journals and conferences based on their relevance, to assess the productivity of professors in terms of both quality and quantity over a period of five years. The findings suggest that researchers have a preference to publish their research work at a conference instead of a journal. On average, for every paper published in a journal, 2.52 papers were published by researchers presented at conferences. Typically, researchers have a lower number of papers published in academic journals compared to conferences, with 77.78% falling below the average. The quality and quantity of scientific production among researchers are uneven, with 80% of publications generated by only half of the researchers. The majority, or two-thirds, of professors have a maximum of 10 co-authors per year. However, recent analysis of the co-authorship network indicates the increase in internal collaboration, accompanied by changes in connections in comparison to prior years. This suggests that collaboration is a partnership aimed at achieving specific research goals. Understanding the connections between these researchers is the first step towards improving collaboration. If two authors are not connected but share co-authors, they may have similar research interests. Furthermore, we explored utilizing a decision tree, which is an interpretable machine learning model, to predict the granting of PQ scholarships (class 2). Obtaining a high accuracy of 95.3125% suggests a promising field for future exploration. As a future work, this analysis will be expanded to other computer science departments to compare the obtained results.

ACKNOWLEDGMENT

André R. Backes acknowledges the financial support of the Brazilian National Council of Scientific and Technological Development (CNPq) (Grant #307100/2021-9). This study was partly financed by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brazil (CAPES) - Finance Code 001.

REFERENCES

• ALBERTINI, M. K.; BACKES, A. R.; SÁ, A. L. A study of publication trajectories of the Brazilian computer science community. Anais da Academia Brasileira de Ciências, [s.l.], v. 91, n. 3, p. 1-12. 2019.
• BORDIN, A. S.; GONÇALVES, A. L.; TODESCO, J. L. Departmental scientific collaboration analysis through coauthorship networks. Perspectivas em Ciência da Informacão, Belo Horizonte, v. 19, n. 2, p. 37-52, 2014.
• BOWYER, K. W. et al. SMOTE: synthetic minority over-sampling technique. Journal Of Artificial Intelligence Research, [s.l.], v. 16, [s.n.], p. 321-357, 2011. Retrieved from: http://arxiv.org/abs/1106.1813 Accessed on: Apr. 2022.
  » http://arxiv.org/abs/1106.1813
• CAÑIBANO, C.; BOZEMAN, B. Curriculum vitae method in science policy and research evaluation: the state-of-the-art. Research Evaluation, [s.l.], v. 18, n. 2, p. 86-94, 2009.
• CASTIONI, R.; MELO, A. A. S.; AFONSO, M. C. L. Cnpq’s productivity scholarship in the area of education: an analysis focusing on basic education. Educação e Pesquisa, [s.l.], v. 46. [s.n.], 2020.
• FAGUNDES, L. C. et al. Productivity profile of CNPq scholarship researchers in Physical Education. Motriz: rev. educ. fis. [s.l.], v. 26, n. 2, 2020.
• MATHEW, G.; AGRAWAL, A.; MENZIES, T. Trends in topics at se conferences (1993-2013). In: INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING COMPANION (ICSE-C), 39., [s.l.]. Proceedings… [s.l.]: IEEE: ACM, 2018. p. 397-398. DOI 10.1109/ICSE-C.2017.52 2017.
  » https://doi.org/10.1109/ICSE-C.2017.52 2017.
• OLIVEIRA, C. A. et al. Academic genealogy in the field of information science: a study on researchers with productivity grants (PQ-CNPq). Em Questão, Porto Alegre, v. 24, n. 2, [Edição Especial 6 EBBC], p. 298-278, 2018.
• QUINLAN, J. R. C4.5: programs for machine learning. San Francisco: Morgan Kaufmann Publishers Inc., 1993. Retrieved from: http://portal.acm.org/citation.cfm?id=152181 Accessed on: Apr. 2022.
  » http://portal.acm.org/citation.cfm?id=152181
• SACCO, A. M. et al. Profile of “CNPq research productivity scholarship” active in psychology in the triennium 2012-2014. Psicologia: Ciência e Profissão, [s.l.], v. 36, n. 2, 2016.
• SILVA, A. J. N. et al. Analysis of co-authorship networks among Brazilian graduate programs in computer science. [s.n.], [s.l.], [s.n.], [s.n.], Dec. 2020 Retrieved from: https://arxiv.org/abs/2012.12439 Accessed on: Apr. 2022.
  » https://arxiv.org/abs/2012.12439
• VASILESCU, B. et al. How healthy are software engineering conferences? Science of Computer Programming, [s.l.], v. 89, [s.n.], p. 251-272, 2014
• WAY, S. F. et al. The misleading narrative of the canonical faculty productivity trajectory. Proceedings of the National Academy of Sciences of the United States of America, [s.l.], v. 114, n. 44, p. 1-8, Oct. 2017.
• WONG, W. E. et al. A bibliometric assessment of software engineering themes, scholars and institutions (2013-2020). Journal of Systems and Software, [s.l.], v. 180, [s.n.], Oct. 2021.

Publication Dates

• Publication in this collection
  22 Dec 2023
• Date of issue
  2023

History

• Received
  18 Apr 2022
• Accepted
  08 Nov 2023