Acessibilidade / Reportar erro

THE VIDEO ASSISTANT REFEREE (VAR) IN SOCCER: WHERE IS THE BIOMECHANICS?

ÁRBITRO ASSISTENTE DE VÍDEO (VAR) NO FUTEBOL: ONDE ESTÁ A BIOMECÂNICA?

ABSTRACT

The aim of this article is to explore the basic principles of Biomechanics in relation to the objective measurements carried out by the Video Assistant Referee (VAR) operators and systems in soccer. It was presented an overall framework about the determination of kinematical variables, the definition and the sources of error, the understanding of factors contributing to errors in VAR decisions, and prospects for future solutions. To exemplify these concepts, the offside rule was chosen as it offers numerous instances of tools aiding for accurate measurements. Based on consolidated procedures of literature, a discussion about the actual challenges and an suggestions for a fully comprehension and improvement of VAR were provided.

Keywords:
Kinematics; Refereeing; Measurement error; Video technology

RESUMO

O objetivo deste artigo foi explorar os princípios básicos da Biomecânica em relação às medições objetivas realizadas pelos operadores e sistemas do Árbitro Assistente de Vídeo (VAR) no futebol. Foi apresentado uma estrutura geral sobre a determinação das variáveis ​​cinemáticas, a definição e as fontes de erro, a compreensão dos fatores que contribuem para os erros nas decisões do VAR e as perspectivas de soluções futuras. Para exemplificar esses conceitos, a regra do impedimento foi escolhida, pois oferece inúmeras instâncias de ferramentas que requerem medições acuradas. Com base em procedimentos consolidados da literatura, foram apresentadas discussões sobre os atuais desafios e sugestões para compreensão e aprimoramento completos do VAR.

Palavras-chave:
Cinemática; Arbitragem; Erro de medida; Tecnologia de vídeo

Introduction

On 3rd March 2007, in Dubai, during a tennis match between Nadal and Youzhny, challenge made by Youzhny was supported by Hawk-Eye. In tennis competition, Hawk-Eye is one of the most important commercial systems that check ball in-or-out during questionable shots, supporting tennis referees during official matches. In that day, Hawk-Eye showed the ball had skimmed the line. The problem became bigger: Nadal claimed that the mark of the ball was still on the court, showing that the ball was out. In that moment, Nadal was World #2 in the ATP ranking and his voice resounded. Nadal questioned the efficiency of the new Hawk Eye line calling technology. Other important players also questioned Hawk-Eye. During 2007 Wimbledon men’s final between against Nadal, Roger Federer asked chair referee to turn off the Hawk-Eye machine, but the request was declined. Besides their fantastic career, Nadal and Federer also contributed to sport development putting in the spotlight a public discussion: does the technology make mistakes too?

Video-assistant referee (VAR) may be defined as the set of tools and procedures that help field referees in difficult decision-making situations. VAR began to be used around the beginning of the 2000s, in sports like tennis and American Football. In tennis, its main objective is to remove the doubt about if the ball hit in or out the court lines. From the discussion about its efficiency, as briefly described in the first paragraph based on Collins and Evans publication 11. Collins H, Evans R. You cannot be serious! Public understanding of technology with special reference to “Hawk-Eye”. Public Underst Sci. 2008;17(3):283-308. DOI: 10.1177/0963662508093370
https://doi.org/10.1177/0963662508093370...
, VAR in tennis evolved for many years to reach the tool we have nowadays. The main gyms where important championships take place have a very large set of high frequency cameras that together, visualizing the same phenomenon, manage to reconstruct in three dimensions the entire trajectory of the ball and then obviously clarify to the public and the players whether the ball had gone in or not. Nowadays, VAR efficiency in tennis is rarely questioned. If in the begging famous players criticized the technology, nowadays it is far approved. Of course, the evolution of tool guaranteed the acceptance of VAR interventions, but the public knowledge about its error (around 3.6 mm, according to Hawk-Eye website) as well.

VAR started to be used in football more than one decade after tennis. In 2014, FIFA World Cup male championship adopted VAR to check goal/non-goal circumstances. In 2016, several championships around the world implemented VAR to help field referee in decision-making actions and since 2018 FIFA World Cup male championship, the system assumed several other responsibilities besides checking doubted goals.

In football, VAR basically exists to respond to three situations. The first one is related to attention and perception. In these situations, VAR tries to detect phenomena and events that happen very quickly that is impossible to be seen by unaided eye. Goal/non-goal and offside checking are included in this class. The second situation is concerning the problems of information processing and categorization. For example, in this class VAR helps the referee suggesting review of foul events when the operators think that a given player deserved red card instead yellow card, or regarding penalty circumstances when a given player touches the ball with upper limbs. Finally, the third situation is regard to behaviour response, when field referee attributes a penalization (red or yellow card) to the wrong player.

Most of the time, second and third classes of VAR’s intervention involve subjective opinions. On the other hand, the first class of situations deal with very objective metrics that require the proper measurements procedures that directly affect on the data interpretation. All these procedures have Biomechanics fundamentals that, if neglected, compromises the data analysis and, consequently, put at stake the technology efficiency. In this sense, it is imperative to elucidate that all measurements, with or without technology, have error. In science, error is part of the research methodology and is constantly reported. In this context, for the sport development, it is time to discuss about VAR errors and how science would treat polemic circumstances.

Therefore, the purpose of this article is to discuss the fundamentals of Biomechanics closed related to objective measures performed by VAR operators and system. The definition of error, the conceptualization of sources of error applied to VAR decisions, scientific evidence, possible solutions and future perspectives were presented. To illustrate all these concepts, offside rule was selected once it promotes several examples about the aids for an accurate measure. Finally, conclusions were addressed about the relevance of the scientific community for the development the VAR tools as well as public knowledge and trust about technology in Sport.

The problem: position determination as function of time

When we talk about VAR interventions to define goal/non-goal or to define if offside rule was violated, we are talking about the determination of elements position in a given space. In the former situation, ball position needs to be determined to check whether it passed through the goal line. The last, players positions (and their limbs) need to be determined to check if the attacker is behind the penultimate opponent. Thus, kinematic information needs to be identified. Kinematics is the study of bodies in motion without regard to the causes of the movement. It is concerned with describing and quantifying positions of bodies and their time derivatives. Usually, in Kinematics, position is determined in relation to a global coordinate system. This reference system is constructed from orthogonal axes with fixed orientation. Thus, every point in the space has its position defined regarding its coordinates for each axis. If the global coordinate system is defined with two axes (X and Y-axis, for instance), we represent elements positions in a 2D space environment; if the global coordinate system is defined with three axes (i.e. X, Y-axis and Z-axis), then elements positions are represented in an 3D environment 22. Robertson DGE, Caldwell GE, Hamill J, Kamen G, Whittlesey SN. Research methods in biomechanics. Second edition. ed. Champaign, Illinois: Human Kinetics; 2014. xii, 428p..

Stereophotogrammetric methods are used to reconstruct 3-D coordinates from photographs, radiographs, and video images 33. Cappozzo A, Della Croce U, Leardini A, Chiari L. Human movement analysis using stereophotogrammetry. Part 1: theoretical background. Gait Posture. 2005;21(2):186-96. DOI: 10.1016/j.gaitpost.2004.01.010
https://doi.org/10.1016/j.gaitpost.2004....
. Considering the video images the alternative found by VAR system, at least two cameras visualizing the same element are necessary to reconstruct 3D position. For 2D reconstruction, a single camera may be enough. From a calibration frame, intrinsic and extrinsic information of each camera are determined and the parameters for reconstruction are calculated. So, in each time stamp, elements identified in the video have their coordinates transformed from pixels to real-world coordinates. However, with all these procedures, errors are built-in, with different characteristics.

There are two classes of errors 44. Chiari L, Della Croce U, Leardini A, Cappozzo A. Human movement analysis using stereophotogrammetry. Part 2: instrumental errors. Gait Posture. 2005;21(2):197-211. DOI: 10.1016/j.gaitpost.2004.04.004
https://doi.org/10.1016/j.gaitpost.2004....
:

  1. Systematic error (or bias): tendency of the instrument to overestimate or underestimate the measurement. They can be estimated and corrected using proper procedures;

  2. Random errors (precision): unpredictable errors, resulting from fluctuations in the measurement itself. We can know their magnitude and distribution, and they must be taken into account to assess the degree of confidence of a measure.

Systematic errors, in general, may be result of calibration inaccuracies (i.e. bad estimation of model parameters) such as errors associated to the measures in the calibration frame or image distortion 55. Burton K. Biomechanics of human movement: applications in rehabilitation, sports and ergonomics. Clin Biomech. 1992;7(4):251. DOI: 10.1016/0268-0033(92)90010-2
https://doi.org/10.1016/0268-0033(92)900...
. On the other hand, the random errors usually are associated with the variation in the determination of the limb/individual position. For instance, when dealing with video-based tracking, the pixel contour of the players is identified, the center of gravity of this image is calculated and the coordinates are projected in the pitch plane 66.Figueroa PJ, Leite NJ, Barros RML. Tracking soccer players aiming their kinematical motion analysis. Comput Vis Image Underst. 2006;101(2):122-35.. Thus, the variation of the contour shape may reflect in random variation of the element position determination, even when the player is not moving.

Taking into consideration these two sources of error, the next section will discuss the VAR actions during offside plays and some important steps that need (or should need) be addressed for an improved intervention.

VAR during offside

Using simple terms, a given player is offside if he/she is in the attacking half and closer to the opposing team's goal-line than both the ball and the second-last opponent. In this sense, it is important to emphasize that:

  1. To define the line that enables or disallows the attacker to play, the part of the defender's and attacker’s body that is taken into account is the one that is closest to the goal-line line and that can be used to touch the ball, i.e., hands and arms do not count, but shoulders and all the other parts do.

  2. The position is considered in the last moment of the attacker’s teammate touches the ball.

Thus, we have here two important measures that have errors and must be checked to guarantee the quality of the evaluation: the right frame that represent the last touch in the ball, and a good position measurement.

The identification of the correct frame is related to the temporal resolution of the cameras. When dealing with videos, temporal resolution is defined by the acquisition frequency (or frame rate). Fast movements require high frame rate cameras to avoid aliasing effect. Aliasing is the overlapping of frequency components resulting from a sample rate below the minimum frequency required to properly identify a phenomenon. In other words, if the movement is fast, low frame rate will not, necessarily, allow the identification of the last contact of ball with the player body. Thus, with broadcast cameras used on some occasions by VAR, it seems highly possible that, in a given frame, the ball is still in contact with the body, and in the next frame the ball is completely far from the player. Thus, high frequency cameras are imperative for an acceptable frame detection. For instance, studies with kinematical analysis of soccer kick use cameras operating, at least, at 240 Hz 77. Palucci Vieira LH, Santiago PRP, Pinto A, Aquino R, Torres RDS, Barbieri FA. Automatic Markerless Motion Detector Method against Traditional Digitisation for 3-Dimensional Movement Kinematic Analysis of Ball Kicking in Soccer Field Context. Int J Environ Res Public Health. 2022;19(3).10.3390/ijerph19031179
https://doi.org/10.3390/ijerph19031179...
. Therefore, it would be highly recommended that VAR operates similarly to avoid aliasing effect.

The second point is regarding the accurate measure of player and/or body limbs position. Once is quite hard to determine all body limbs and, additionally, identify the one closer to the goal line, one alternative would be to consider player position, once literature offers several methods for player tracking. Video-based tracking is very promising once does not require that the player wear any device. Also, performing detailed camera calibration and optical distortion correction, error may vary from 9 to 30 cm 88. Barros RML, Misuta MS, Menezes RP, Figueroa PJ, Moura FA, Cunha SA, et al. Analysis of the distances covered by first division Brazilian soccer players obtained with an automatic tracking method. J Sports Sci Med. 2007;6:10.),(99. Lara JPR, Vieira CLR, Misuta MS, Moura FA, Barros RMLd. Validation of a video-based system for automatic tracking of tennis players. Int J Perform Anal Sport. 2018;18(1):137-50. DOI: 10.1080/24748668.2018.1456886
https://doi.org/10.1080/24748668.2018.14...
),(1010. Vieira LHP, Pagnoca EA, Milioni F, Barbieri RA, Menezes RP, Alvarez L, et al. Tracking futsal players with a wide-angle lens camera: accuracy analysis of the radial distortion correction based on an improved Hough transform algorithm. Comput Methods Biomech Biomed Eng: Imaging Vis. 2017;5(3):221-31. DOI: 10.1080/21681163.2015.1072055
https://doi.org/10.1080/21681163.2015.10...
. Other wearables systems may be used 1111. Rico-González M, Pino-Ortega J, Nakamura FY, Arruda Moura F, Rojas-Valverde D, Los Arcos A. Past, present, and future of the technological tracking methods to assess tactical variables in team sports: A systematic review. Proc Inst Mech Eng P: J Sports Eng Technol. 2020;234(4):281-90. DOI: 10.1177/1754337120932023
https://doi.org/10.1177/1754337120932023...
, but generally involve high-cost equipment installation (in the case of RFID sensors systems) or preset lower accuracy, such as GPS devices.

Independently of the system used, error estimate is imperative. From this estimate, it is possible to make assumptions whether player is offside or not. For example, if VAR identify that measurement error is 10 cm and the distance between defender and attacker is lower than that, it is not possible to know who is closer to the goal line. In this case, VAR may prioritize the attack, giving the idea of “if there is doubt, let’s favor the goal scoring possibility”.

Premier League adopted similar strategy some years ago 1212. Globo.com.[Internet] Rio de Janeiro: Globo; 2021[cited 29/05/2023]. Available from:Available from:https://ge.globo.com/futebol/futebol-internacional/futebol-ingles/noticia/premier-league-muda-linhas-do-var-para-beneficiar-atacantes-no-impedimento.ghtml
https://ge.globo.com/futebol/futebol-int...
. When the positions lines are defined for each player, if the lines are overlapped, the VAR assumes as regular play. The first division of the male Brazilian Championship adopted the same idea since 2023 1313. Globo.com.[Internet] Rio de Janeiro: Globo; 2023[cited 29/05/2023]. Available from:Available from:https://ge.globo.com/rj/futebol/brasileirao-serie-a/noticia/2023/04/05/comunicacao-no-telao-e-linhas-de-impedimento-que-beneficiam-ataque-as-novidades-da-arbitragem-para-o-brasileirao-2023.ghtml
https://ge.globo.com/rj/futebol/brasilei...
. Although it seems like an advance in the way measures are interpreted giving the idea of error range, this modification requires special care. Normally, VAR uses a single camera to define the lines. The lines drawn by VAR have the same thickness from beginning to end, but, as the camera image is not perpendicular to the plane of the field, the same distance on the other side of the field represents less pixels in the image. Therefore, each pixel far from the camera represents a measure, in meters, greater than each pixel closer to the camera. Thus, for fair measure, the line thickness drawn by VAR would need to change as function of the distance from the camera, i.e., lines need to be wider as close as they are from the camera. Finally, it is imperative the same parameters in each match, so, calibration parameter need to be determined to reference possible solutions regarding lines thickness.

The problem with the line’s determination is also closely related to the procedure used by VAR to determine player limbs position. As described before, with a single camera, only 2D data can be determined. However, body parts move in a 3D environment, showing that the present VAR procedure is limited. To reduce errors associated to such measurements, great championships (e.g. FIFA World Cup, UEFA Champions League) adopted a new sophisticated semi-automatic system to properly deal with offside moments 1414. Fifa.com.[Internet ] 2022 [cited 29/05/2023]. Available from:https://www.fifa.com/technical/football-technology/football-technologies-and-innovations-at-the-fifa-world-cup-2022/semi-automated-offside-technology
https://www.fifa.com/technical/football-...
. At least 12 synchronized cameras (frame rate of 50 Hz) are fixed at the stadium. The videos are analyzed by deep learning algorithms allowing the makerless identification of 29 key-points of players’ limbs. To reduce the issue related to temporal resolution, an inertial measurement unit (IMU) at 500 Hz is placed inside the ball, allowing a very precise detection of the kick point.

Markerless systems are very promising solutions for uncontrolled sites where marker-based data collection is not possible. Compared to gold-standard measure, in laboratory environments, markerless systems may present error lower than 3 cm 1515. Lahkar BK, Muller A, Dumas R, Reveret L, Robert T. Accuracy of a markerless motion capture system in estimating upper extremity kinematics during boxing. Front Sports Act Living. 2022;4:939980. DOI: 10.3389/fspor.2022.939980
https://doi.org/10.3389/fspor.2022.93998...
. However, both FIFA and companies responsible for the VAR technology did not present the mean error considering the huge volume area around the soccer pitch.

Such technology requires high-cost complex installation and operation, being still restricted to the most relevant championships. Unfortunately, there is no predictions about when it will be completely adopted for all professional championships around the world. While this does not happen, it becomes necessary to seek improvements in the methods currently available, even considering their limitations. The next section will present a workflow suggestion to improve the quality of the information collected by VAR, possible interpretations and how to deal with situations where the magnitude of the measured information in lower than the error range.

Adjustments to VAR procedures during offside

In order to improve the accuracy of the position information collected, to reduce discrepancies between VAR systems installed in different stadiums during different matches and to guarantee isonomic decision-making by the referee board, special attention during calibration and measurement procedures must be taken. Part of these steps may be performed before the match starts, while other steps require organized team work to provide information as fast as (and as good as) the event requires.

Figure 1 presents a workflow to show the steps needed before the match starts for system calibration and accuracy evaluation. The first one is to install cameras with the view parallel to the pitch sideline. The cameras need to remain fixed during the entire match. The second step is to precisely measure pitch reference points position. Even expecting that pitch dimensions are defined by the official rules, it is possible to exist small variations when the lines are drawn. This information will serve as input data for the calibration frame. At this moment, it is important to define the magnitude of the error that will be accepted. For instance, it is possible to assume the width of the pitch lines as the magnitude of error.

The next step is to perform the calibration of each camera, preferably performing the cameras radial distortion correction 1010. Vieira LHP, Pagnoca EA, Milioni F, Barbieri RA, Menezes RP, Alvarez L, et al. Tracking futsal players with a wide-angle lens camera: accuracy analysis of the radial distortion correction based on an improved Hough transform algorithm. Comput Methods Biomech Biomed Eng: Imaging Vis. 2017;5(3):221-31. DOI: 10.1080/21681163.2015.1072055
https://doi.org/10.1080/21681163.2015.10...
. Then, randomly select several points of the pitch for a 2D reconstruction. With the information of several points position, the next step consists of calculating the error of these measurements for each camera. If the error is lower than the acceptable error previously defined, the system is ready to be used during the match. If the error is higher than the error previously defined as acceptable, it is necessary to repeat all the steps until the desired accuracy is achieved.

Figure 1
Workflow of the steps needed before the match starts for system calibration and accuracy evaluation during offside plays

Figure 2 presents steps that need to be (better) accomplished during the match by the VAR operators. It is very important to emphasize that, even if accurate parameters were achieved during calibration procedures, an inaccurate measure during the match will compromise the interpretation of the results. Besides that, all the information needs to be generated in few seconds, requiring a very organized, controlled, and synchronized teamwork. In this sense, it is suggestive that operators have preestablished responsibilities and act accordingly with their functions with maximal attention. When an offside play happens, if possible, automatic optical distortion corrections are applied for the few frames of the video sequence. After identifying the players (the defender and the attacker) involved, if possible, two independent measures each should be accomplished. Then, points 2D reconstruction, projected on the pitch plane, is performed, and the distance between players is determined. If the distance is smaller than the error estimated previously, it assumes that VAR cannot affirm if attacker is offside. In this case, for instance, VAR may prioritize the attack, like the idea of the lines overlapping presented before. On the other hand, if the distance between the players is bigger than the error estimate, it is safe to say that VAR intervention defines if the play must be confirmed or cancelled.

Figure 2
Workflow of the steps that is expected to be accomplished during the match by the VAR operators

Final considerations

The idea of this report was to present the determining scientific considerations that should be taken into account when the role of VAR is discussed. While a significant body of supporters, players, teams’ staff and sports journalists argue about the mistakes done by VAR operations, relevant scientific evidence shows the advantages of having VAR or other complementary information to avoid injustices in sport 1616. Buraimo B, Forrest D, Simmons R. The 12th man?: refereeing bias in English and German soccer. J R Stat Soc, A: Stat Soc. 2010;173(2):431-49. DOI: 10.1111/j.1467-985X.2009.00604.x
https://doi.org/10.1111/j.1467-985X.2009...
),(1717. Buraimo B, Simmons ROB, Maciaszczyk M. Favoritism and referee bias in European soccer: evidence from the Spanish League and the UEFA Champions League. Contemp Econ Policy. 2012;30(3):329-43. DOI: 10.1111/j.1465-7287.2011.00295.x
https://doi.org/10.1111/j.1465-7287.2011...
),(1818. Dawson P, Dobson S. The influence of social pressure and nationality on individual decisions: Evidence from the behaviour of referees. J Econ Psychol. 2010;31(2):181-91. DOI: 10.1016/j.joep.2009.06.001
https://doi.org/10.1016/j.joep.2009.06.0...
),(1919. Kranjec A, Lehet M, Bromberger B, Chatterjee A. A sinister bias for calling fouls in soccer. PLoS One. 2010;5(7):e11667. DOI: 10.1371/journal.pone.0011667
https://doi.org/10.1371/journal.pone.001...
),(2020. Nevill AM, Newell SM, Gale S. Factors associated with home advantage in English and Scottish soccer matches. J Sports Sci. 1996;14(2):181-6. DOI: 10.1080/02640419608727700
https://doi.org/10.1080/0264041960872770...
),(2121. Unkelbach C, Memmert D. Crowd noise as a cue in referee decisions contributes to the home advantage. J Sport Exerc Psychol. (0895-2779 (Print)). DOI: 10.1123/jsep.32.4.4832010
https://doi.org/10.1123/jsep.32.4.483...
),(2222. Spitz J, Wagemans J, Memmert D, Williams AM, Helsen WF. Video assistant referees (VAR): The impact of technology on decision making in association football referees. J Sports Sci. 2021;39(2):147-53. DOI: 10.1080/02640414.2020.1809163
https://doi.org/10.1080/02640414.2020.18...
. For instance, a recent study 2222. Spitz J, Wagemans J, Memmert D, Williams AM, Helsen WF. Video assistant referees (VAR): The impact of technology on decision making in association football referees. J Sports Sci. 2021;39(2):147-53. DOI: 10.1080/02640414.2020.1809163
https://doi.org/10.1080/02640414.2020.18...
analyzed more than 9000 situations in soccer and reported that the referee’s initial decision was correct in 92.1%. After VAR intervention, the accurate decisions improved to 98.3%. Another study showed 2323. Albanese A, Baert S, Verstraeten O. Twelve eyes see more than eight. Referee bias and the introduction of additional assistant referees in soccer. PLoS One. 2020;15(2):e0227758. DOI: 10.1371/journal.pone.0227758
https://doi.org/10.1371/journal.pone.022...
that the more referees are acting during the match, lower is the bias in favor of the home and big teams.

Therefore, in this opinion article is out of discussion the importance of VAR when a fairer and more democratic sport is sought. We focused on showing that improving VAR accuracy would help to its acceptance by the players, by the clubs’ staff and by the public, in general. On the other hand, some important aspects may not be neglected. The first one is regarding referees and VAR operators’ capacitation in biomechanics. These professionals need to understand all the steps required for an accurate measure, including cameras parameters, calibration, and variables calculations. The professionals need to understand how measurement error is related to accuracy and to properly interpretation the data collected. Besides Biomechanics, other important scientific discussion in several research field discuss about the factor that directly influences on referees’ performance, such as sleep quality, nutrition, attention, and decision-making trainings. With this great variety of points that requires referees’ total dedication and efforts to improve their performance, is not time to discuss, mainly in Brazil, about referees’ professionalization?

The second point is to seek for a more scientific stance by the journalistic community providing the correct interpretation for the VAR intervention, without fanatism and polemic. Journalists need to explain to the society that VAR procedures have errors, but without them is worst. These professionals may explain what measurement error is, just like it is already reported in several subjects, such as voting intention polls, pandemic data, climate change, etc.

Third, it is imperative that the company responsible by VAR public declares the measurement error. The acceptance and credibility of the system will not be achieved with unsupported arguments that the system is perfect. Just like in science, the problem is not to have errors in the measurements, but not knowing them. If the measurement error is known, proper decisions are drawn, mainly in complex situations that will always exist.

Finally, the system needs to be continuously improved, seeking for reduced errors and biased interventions. In the last World Conference in Science and Football, in 2023, a session was dedicated to the major future challenges for VAR, such as communication issues and erroneous penalties attributed to handball rules in moments that the players clearly need to use their arms to run, jump, change of direction and other sports movements. In this sense, the discussion with the scientific community is essential to achieve effective solutions in a shorter period.

Acknowledgements:

This work was supported by the CNPq under grants 401004/2022-8, 200290/2022-3 and 305997/2022-0.

References

Edited by

Editor: Ademar Avelar

Publication Dates

  • Publication in this collection
    15 Dec 2023
  • Date of issue
    2023

History

  • Received
    01 June 2023
  • Reviewed
    28 Aug 2023
  • Accepted
    28 Aug 2023
Universidade Estadual de Maringá Avenida Colombo, 5790 - cep: 87020-900 - tel: 44 3011 4315 - Maringá - PR - Brazil
E-mail: revdef@uem.br