Assessment of the usability of a digital learning technology prototype for monitoring intracranial pressure¹

ABSTRACT

Objective:

to assess the usability of a digital learning technology prototype as a new method for minimally invasive monitoring of intracranial pressure.

Method:

descriptive study using a quantitative approach on assessing the usability of a prototype based on Nielsen's ten heuristics. Four experts in the area of Human-Computer interaction participated in the study.

Results:

the evaluation delivered eight violated heuristics and 31 usability problems in the 32 screens of the prototype.

Conclusion:

the suggestions of the evaluators were critical for developing an intuitive, user-friendly interface and will be included in the final version of the digital learning technology.

Descriptors:
Educational Technology; Health Technology; Nursing; Evaluation

RESUMO

Objetivo:

avaliar a usabilidade de um protótipo educacional digital sobre um novo método para monitoração da pressão intracraniana de forma minimamente invasivo para enfermeiros e médicos.

Método:

estudo descritivo com abordagem quantitativa sobre a avaliação de usabilidade de um protótipo com base nas dez Heurísticas de Nielsen. Participaram quatro especialistas da área de Interação Humano Computador.

Resultados:

a avaliação resultou em oito heurísticas violadas e 31 problemas de usabilidade nas 32 telas do protótipo.

Conclusão:

as sugestões dos avaliadores foram cruciais para o desenvolvimento de uma interface amigável e intuitiva e serão consideradas na versão final da tecnologia educacional digital.

Descritores:
Tecnologia Educacional; Tecnologia em Saúde; Enfermagem; Avaliação

RESUMEN

Objetivo:

evaluar la usabilidad de un prototipo educacional digital sobre un nuevo método para monitorización de la presión intracraneal, de manera mínimamente invasiva.

Método:

estudio descriptivo con abordaje cuantitativo sobre la evaluación de usabilidad de un prototipo con base en las diez reglas Heurísticas de Nielsen. Participaron cuatro especialistas del área de Interacción Humana Computador.

Resultados:

la evaluación resultó en ocho reglas heurísticas violadas y 31 problemas de usabilidad en las 32 pantallas del prototipo.

Conclusión:

las sugestiones de los evaluadores fueron cruciales para el desarrollo de una interfaz amigable e intuitiva y éstas serán consideradas en la versión final de la tecnología educacional digital.

Descriptores:
Tecnología Educacional; Tecnología en Salud; Enfermería; Evaluación

Introduction

Technology has a major influence on current society. Information emerges rapidly from all over the world, and can be accessed via computers and mobile devices connected to the Internet. It is fair to say that technology and society have become inseparable¹.

This vast demand for information brings with it professional competition and the demand for increasingly qualified and well educated professionals in all areas of knowledge. In healthcare, continued professional education enables developing competences and skills to make decisions on behalf of patient safety².

However, lack of time due to overload and an excessive number of daily tasks is often an obstacle for continued education. On the other hand, the working environment can facilitate Distance Learning (DL)^2-5, offering flexibility in terms of learning pace and hours^6-7.

DL is expanding in this country, and is considered an efficient tool as it enables anyone to seek knowledge unhindered by national or other borders⁵. Among the tools available, DL is a digital learning technology that has been increasingly used to promote learning in healthcare, especially for nurses^6,8-10.

Using digital learning technology could be an efficient strategy in the teaching/learning process. However, developing this tool requires rich and dynamic content, and a well-designed interface that is intuitive to the users¹¹. Thus it is important that these resources be evaluated by professionals or users, regardless of the method used.

Numerous methods have been used to evaluate digital learning tools^4,11-12, however regarding the quality of the interface, meaning a product that satisfies and meets user demands, the method most often used is usability^13-16.

Usability assessment normally refers to the ease with which users can perform specific tasks when interacting with the tool or object using the appropriate interface. It is related to five attributes: ease of learning, memorization, mistake prevention, efficiency and user satisfaction¹⁷.

A systematic review shows that one of the most often used approaches to assess usability is a heuristic evaluation¹⁸. Among them are Nielsen's heuristics¹⁷, a simple and low-cost method¹⁹ that can detect a number of usability problems with a small number of evaluators^15,19-20 in a relatively short period of time²¹. This approach provides reliable results and suggestions to improve the interface^16,19-20.

The earlier the assessment, the lower the cost of making changes¹⁷. Thus, creating prototypes has proven an efficient and economic strategy, as potential problems can be tested for and detected, then corrected before the final product version is launched^11,19-20,22.

In this context, to assess the usability of a digital learning technology prototype for a new method for minimally invasive monitoring of intracranial pressure.

Method

This is a descriptive study using a quantitative approach to assess the usability of a digital learning technology prototype.

Assessment of usability used an analytical approach where expert usability evaluators inspected the interface in search of problems, and suggested ways to improve the interface¹⁷. While there are numerous approaches for analytical assessment, in this study we chose heuristic evaluation. ¹⁷

This method is based on using a set of usability principles that guide evaluators as they use an interface in search of problems and shortcomings¹⁷.

The guidelines offer a range of heuristics available, however two experts in usability grouped these heuristics into ten, to facilitate evaluations in practice. These are known as Nielsen's Heuristics¹⁷.

A heuristic evaluation should be done by a small number of evaluators, normally from three to five. Fewer than three is not enough for reliable results, and more than five are not necessary, as the usability problems found become recurring¹⁷.

Thus, this evaluation was conducted by IT professionals enrolled in the Graduate Program of the Computer Science Department of the Federal University of São Carlos, who met the following inclusion criteria: specialize in Human-Computer interaction and have participated in previous heuristic evaluations.

Participants were selected using an intentional method, meaning the researcher chose participants based on inclusion criteria, bearing in mind researcher knowledge of the population and its characteristics, providing the conditions for including individuals in the sample²³. Thus, four potential participants were formally invited by e-mail. Following confirmation that they were willing to participant, a date and place for the first phase of the evaluation was set.

A heuristic evaluation should include three phases: pre-evaluation, evaluation and a session with the evaluators¹⁷, all of which are described below.

Pre-evaluation

In this phase, the interface is presented to evaluator, along with information about functionality, objectives and standard terminology.

The prototype was introduced to the evaluators using a Power Point presentation and multimedia projector. At this meeting, which lasted two and a half hours, the researchers and evaluators discussed the standard application of Nielsen heuristics, and the usability problems associated with each one of them, providing everyone with a better understanding of what to evaluate. We distributed a print-out with questions related to each heuristic, as shown in Figure 1. We also asked the evaluators for suggestions of how to address the problems found, which in this type of evaluation is not required. Finally, a date and place for the next meeting was agreed.

Figure 1
Nielsen's Heuristics¹⁷ and the questions asked for each one. São Carlos, SP, Brazil, 2013

Evaluation

This phase is done individually and participants may choose where they want to perform the evaluation. Each evaluator should run through and analyze his/her prototype at least twice. The first run is to make the evaluator familiar with the interface, while the second allows the evaluator to focus on specific elements in search of usability issues using Nielsen's Heuristics¹⁷, analyzing severity on a scale of zero (0) to four (4), where a higher score is associated with a larger usability problem (Figure 2).

Figure 2
Severity scale based on a usability evaluation using Nielsen's ten Heuristics. São Carlos, SP, Brazil, 2013

The evaluators chose to use their own computer for individual evaluations. Each one was provided with a CD-ROM containing a prototype and the data collection instrument with the following items: heuristic violated, usability problems, location (prototype screen), severity and suggested solutions.

Session with evaluators and project developer.

In this phase the evaluators met with the researcher at the agreed location and date for a meeting lasting three and a half hours. The goal of this meeting was discussion of the evaluations among the evaluators themselves - the heuristics violated, the usability problems found, their severity and location, and suggested solutions for each problem. Evaluators chose do discuss screens in the order in which they appear. Thus each screen was discussed by each evaluator, commenting on the heuristics violated and usability problems found. Where there was disagreement, evaluators discussed until arriving at a formal consensus, which the researcher then entered into the same data gathering tool used for individual assessments. The end result of this phase was a single list of heuristics that had been violated, the usability problems found and the associated severity, the location of the problems and suggestions on how to solve them.

The project was approved by the UNICEP - Central Paulista University Center Ethics Committee for Research on Human Beings, and abides by domestic and international standards of ethics in research involving human beings. Approval protocol 027/2011.

Results

Validation resulted in eight of the ten Nielsen¹⁷ heuristics violated, and 31 problems in the 32 screens of the prototype (Table 1).

Table 1
Violated heuristics, usability problems and their severity in the prototype screens. São Carlos, SP, Brazil, 2013

Heuristic violated	Usability problem		Severity				Total
	Percent		0	1	2	3	4
1 - System status visibility	5 (16.13%)				3	2		5
2 - Match between system interface and the real world	7 (22.58%)			3	1	2		6
3 - User control and freedom	7 (22.58%)		1	1	3	2		7
4 - Consistency and standards	5 (16.13%)				2	4		6
5 - Error prevention	3 (9.68%)			1	1	1		3
6 - Recognition rather than recall	1 (3.23%)			1				1
8 - Esthetics and minimalist design	2 (6.46%)			1	1			2
10 - Help and documentation	1 (3.23%)					1		1
Total by heuristic	31 (100%)		1	7	11	12	0	31

The results show that "Correspondence between the system interface and the real world" and "User control and freedom" were the most often violated, with 7 (22.58%) usability problems found in each one, followed by "System status visibility" and "Consistency and standard", totaling 5 (16.13%).

Among the main comments made by the evaluators regarding the violated heuristics, we point out the following: improve the design, with different colors at the header and footer; overly populated screens with too much content; no standard for links; no warning when a user clicks on a link, informing him/her that he/she is on a web page, and proceed if the user does not want to watch the video; no button to turn off sound, which would give users more control over their actions.

The evaluation also found grammar mistakes, limited explanation of the charts and the analogy of a balloon filling up to simulate increased intracranial pressure.

Discussion

Technology has been an important tool for healthcare professionals - for their training^3,7,11,24, the care they give^3-4,14 and for research^5,18. In this context, it is necessary to assess these systems, otherwise technology may cease to perform its role as a facilitator, and resources will be under-used.

They heuristic assessment of this study was performed by four evaluators. While 3 to 5 are recommended for studies of this nature, who may find up to 75% usability problems^15,17,20-22. We chose to use Nielsen's heuristics¹⁷, considering them to be the best suited to evaluate software(15,19,22), healthcare information systems²² and digital learning tools¹⁶.

It is believed that a combination of two or more types of assessment will enable identifying more usability¹⁸ problems, however Nielsen's heuristics are always present^15-16,20. The authors recommend heuristic evaluations and user testing. Heuristic evaluation may be used the first time to identify and correct the more evident problems, and once changes have been made, will also be used for user testing¹⁶. Thus, as future work in this effort, we intend to evaluate a final version of the prototype with users.

The results shown show the importance of evaluating a system before it is made available to users. Of the ten heuristics proposed by Nielsen, eight were violated, creating 31 usability problems in the 32 screens of the prototype. Correspondence between system interface and the real world, user freedom and control, system status and consistency and standard accounted for over 77% of the violated heuristics. Studies have shown similar results using the same assessment method as that used in this study^15-16,21-22.

Usability problems related to these heuristics can reduce the interaction between the system and its users. Therefore, designers must pay attention when designing interfaces, making sure they are intuitive and easy for even an inexperienced user, avoiding additional effort in learning to use the system²⁵.

The project team and the digital learning technology developer quickly understood the suggestions made by the evaluators, as these were solidly based on the heuristics.

Interfaces and content must be simple, and expressed in clear and objective language that is easy to understand^11,25. In one study, the evaluators suggested less content on each page, and that content be arranged by topic. They also suggested using more objective language and a different visual presentation⁷. Excessive content¹³ e and better content organization¹² were mentioned in other studies.

Thus, one of the important contributions made by the evaluators was to point out usability problems at the interfaces, which could confuse or distract users, suggesting significant changes to address the problems found. Other studies revealed similar suggestions, such as: insert explanations of charts and images that may not be entirely clear^15,25, a scroll-bar for long texts²⁵, changing the color of the interface^9,15 to light green, standardized links²⁵, including an error message when users do not correctly fill out the registration screen, stopping wherever the error occurred¹⁶, and a page describing the icons²⁵. In one study evaluating learning technology, evaluators had problems identifying some of the icons⁷, demonstrating how important it is that they be fully intuitive.

A user-friendly interface may include buttons to make it easier to use. Buttons are used to select items or actions, and should be described using verbs²⁵. Here the evaluators suggested a button for users to increase or decrease font size, a button at the footnote to turn off the sound, and a button for printing. Another study led to similar suggestions¹⁶.

Another comment made by the evaluators involved user control as they use the interface, which is one of the usability principles. A suggestion was made to use a breadcrumb at the top of the screen to track user actions, showing where the user is and how he/she got there, corroborating another finding^21)*.

They "flexibility and efficiency when using" and "help users recognize, diagnose and recover errors" heuristics were not violated. It is believed that these heuristics were not violated as the prototype is not fully functional, and some actions could not be evaluated. Nevertheless, the evaluators made suggestions for developing the final version of the learning technology, such as: "if the browser is configured not to download images and videos, will content and browsing remain intact, with no loss for the user?" "Can the user configure fonts and font size?"

The least violated heuristics were "recognition rather than recall" no error message on the registration page and document support. There is no help item on the pages, and violations related to these heuristics may lead users to an incorrect interpretation of the actual action desired, leading to useless effort, cognitive overload, frustration and dissatisfaction ^{₍20-21,25_).}

Severity ranged from 0 (not a usability problem) to 3 (major usability problems), suggesting that some violations are sufficiently serious to warrant attention. None of the violations in this study were considered grade 4 (catastrophic). Each usability problem found in this study, and the associated severity, were taken into consideration, believing they might negatively affect the effectiveness of the final version of the digital learning technology.

The results of this study show the importance of evaluating digital learning technologies to make them more efficient and user friendly, fostering the teaching and learning process. In nursing, the researchers involved in developing learning systems are concerned with correcting and changing their products based on feedback received from evaluators before making them available to the end user^{7-10,12-13,24}.

One of the limitations in this study was that the prototype was not fully functional, some aspects could not be evaluated. We also had trouble finding five evaluators fulfilling the inclusion criteria. For that reason we used only four, still within the acceptable for this type of evaluation¹⁷.

Conclusion

Given the findings of this study, we believe the goal was achieved as we were able to identify a large number of usability problems with minimal effort, and using a single type of evaluation.

The advantage of heuristic evaluation is that evaluators who are experts in the method were able to make suggestions for the problems found, highlight the strengths and weaknesses of the project. All of the suggestions were considered in the final version of the digital learning technology.

As future work we intent to evaluate the new version of the digital learning technology with users, as in our view evaluations are essential for providing a quality product.

References

• ¹
  Veraszto EV, Silva D, Barros J Filho, Miranda NA, García FG, Amaral SF, et al. Influência da sociedade no desenvolvimento tecnológico: um estudo das concepções de graduandos brasileiros do Estado de São Paulo. Rev CTS [Internet]. 2011 [Acesso 13 out 2015]; 17(6):179-211. Disponível em: http://revistacts.net/files/Volumen%206%20-%20N%C3%BAmero%2017/Veraszto.pdf
  » http://revistacts.net/files/Volumen%206%20-%20N%C3%BAmero%2017/Veraszto.pdf
• ²
  Silva AN, Santos AMG, Cortez EA, Cordeiro BC. Limites e possibilidades do ensino à distância (EaD): na educação permanente em saúde: revisão integrativa. Ciênc Saúde Coletiva [Internet]. 2015 [Acesso 13 out 2015];20(4):1099-1107. Disponível em: http://www.scielosp.org/pdf/csc/v20n4/1413-8123-csc-20-04-01099.pdf
  » http://www.scielosp.org/pdf/csc/v20n4/1413-8123-csc-20-04-01099.pdf
• ³
  Prado C, Silva IA, Soares AVN, Aragaki IMM, Shimoda GT, Zaniboni VF, et al. Teleamamentação no Programa Nacional de Telessaúde no Brasil: a experiência da Telenfermagem. Rev Esc Enferm USP. [Internet]. 2013 [Acesso 9 abr 2014];47(4):990-6. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0080-62342013000400990&lng=pt&nrm=iso
  » http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0080-62342013000400990&lng=pt&nrm=iso
• ⁴
  Grossi MG, Kobayashi RM. A construção de um ambiente virtual de aprendizagem para educação a distância: uma estratégia educativa em serviço. Rev Esc Enferm USP. [Internet]. 2013 [Acesso 9 abr 2014];47(3):756-60. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0080-62342013000300756&lng=pt&nrm=iso
  » http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0080-62342013000300756&lng=pt&nrm=iso
• ⁵
  Rojo PT, Vieira SS, Zem-Mascarenhas SH, Sandor ER, Vieira CRSP. Panorama of nursing distance education in Brazil. Rev Esc Enferm USP. [Internet]. 2011 [Acesso 9 abr 2014];45(6):1476-80. Disponível em: http://www.scielo.br/pdf/reeusp/v45n6/v45n6a28.pdf
  » http://www.scielo.br/pdf/reeusp/v45n6/v45n6a28.pdf
• ⁶
  Fonseca LMMI, Aredes NDA, Dias DMV, Scochi CGS, Martins JCA, Rodrigues MA. Serious game e-Baby: percepção dos estudantes de enfermagem sobre a aprendizagem da avaliação clínica do bebê prematuro. Rev Bras Enferm. [Internet]. 2015 [Acesso 13 out 2015];68(1):13-9. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0034-71672015000100013
  » http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0034-71672015000100013
• ⁷
  Fonseca LMM, Aredes NDA, Leite AM, Santos CB, Lima RAG, Scochi CGS. Evaluation of an educational technology regarding clinical evaluation of preterm newborns. Rev. Latino-Am. Enfermagem. [Internet]. 2013 [Acesso 13 out 2015];21(1):[08 telas]. Disponível em: http://www.scielo.br/pdf/rlae/v21n1/v21n1a11.pdf
  » http://www.scielo.br/pdf/rlae/v21n1/v21n1a11.pdf
• ⁸
  Castro FSF, Dias DMV, Higarashi IH, Scochi CGS, Fonseca LMM. Evaluation of digital educational student-technology interaction in neonatal nursing. Rev Esc Enferm USP. [Internet]. 2015 [Acesso 13 out 2015];49(1):114-121. Disponível em: http://www.scielo.br/pdf/reeusp/v49n1/pt_0080-6234-reeusp-49-01-0114.pdf
  » http://www.scielo.br/pdf/reeusp/v49n1/pt_0080-6234-reeusp-49-01-0114.pdf
• ⁹
  Góes FSN, Camargo RAA, Fonseca LMM, Oliveira GF, Hara CYN, Felipe HR, Caldas NB. Avaliação de tecnologia digital educacional "sinais vitais e anatomia" por estudantes da educação profissionalizante em enfermagem. Reme - Rev Min Enferm. [Internet]. 2015 [Acesso 13 out 2015];19(2):37-43. Disponível em: http://www.reme.org.br/artigo/detalhes/1004
  » http://www.reme.org.br/artigo/detalhes/1004
• ¹⁰
  Silveira DT, Catalan VM, Neutzling AL, Martinato LHM. Digital learning objects in nursing consultation: technology assessment by undergraduate students. Rev. Latino-Am. Enfermagem. [Internet]. 2010 [Acesso 13 out 2015];18(5):[09 telas]. Disponível em: http://www.scielo.br/pdf/rlae/v18n5/pt_23
  » http://www.scielo.br/pdf/rlae/v18n5/pt_23
• ¹¹
  Rodrigues RCV, Peres HHC. An educational software development proposal for nursing in neonatal cardiopulmonary resuscitation. Rev Esc Enferm USP. [Internet]. 2013 [Acesso 9 abr 2014];47(1):235-41. Disponível em: http://www.scielo.br/scielo.php?pid=S0080-62342013000100030&script=sci_arttext
  » http://www.scielo.br/scielo.php?pid=S0080-62342013000100030&script=sci_arttext
• ¹²
  Mori S, Whitaker I Y, Marin H F. Avaliação do website educacional em Primeiros Socorros. Rev Esc Enferm USP. [Internet]. 2013 [Acesso 9 abr 2014];47(4):950-7. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0080-62342013000400950&lng=pt&nrm=iso
  » http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0080-62342013000400950&lng=pt&nrm=iso
• ¹³
  Góes FSN, Fonseca LMM, Furtado MCC, Leite AM, Scochi CGS. Evaluation of the virtual learning object "Diagnostic reasoning in nursing applied to preterm newborns". Rev. Latino-Am. Enfermagem. [Internet]. 2011 [Acesso 9 abr 2013];19(4):894-901. Disponível em: http://www.scielo.br/scielo.php?pid=S0104-11692011000400007&script=sci_arttext&tlng=pt
  » http://www.scielo.br/scielo.php?pid=S0104-11692011000400007&script=sci_arttext&tlng=pt
• ¹⁴
  Rangel AL, Évora YDM, Oliveira MMB. O processo de avaliação do software de geração automática de escala de Trabalho da enfermagem e da escala por ele gerada. J Health Inform. [Internet]. 2012 [Acesso 5 mar 2013];4(n.esp):200-4. Disponível em: http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/208/148
  » http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/208/148
• ¹⁵
  Choi J, Bakken S. Web-based education for low-literate parents in neonatal intensive care unit: development of a website and heuristic evaluation and usability testing. Rev Int J Med Inform. [Internet]. 2010 [Acesso 25 jul 2014];79(8):565-575. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2956000/
  » http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2956000/
• ¹⁶
  Davids MR, Chikte UME, Halperin ML. An efficient approach to improve the usability of e-learning resources: the role of heuristic evaluation. Rev Adv Physiol Educ. [Internet]. 2013 [Acesso 25 jul 2014];7(sn):242-24. Disponível em: http://advan.physiology.org/content/37/3/242.full-text.pdf+html
  » http://advan.physiology.org/content/37/3/242.full-text.pdf+html
• ¹⁷
  Nielsen J. Usability Engineering. San Francisco: Academic Press; 1993. 362 p.
• ¹⁸
  Martins AI, Queirós A, Rocha NP, Santos BS. Avaliação de usabilidade: uma revisão sistemática da literatura. Rev Ibérica de Sistemas Tecnol Informação. [Internet]. 2013 [Acesso 25 jul 2014];11(6):31-43. Disponível em: http://ojs.academypublisher.com/index.php/risti/article/view/risti113144
  » http://ojs.academypublisher.com/index.php/risti/article/view/risti113144
• ¹⁹
  Alsumait A, Al-Osaimi A. Usability heuristics evaluation for child learning applications. J Software. [Internet]. 2010 [Acesso 25 jul 2014];5(6):654-661. Disponível em: http://dl.acm.org/citation.cfm?id=1806417
  » http://dl.acm.org/citation.cfm?id=1806417
• ²⁰
  Weegen S, Verwey R, Tange HJ, Spreeuwenberg MD, Witte LP. Usability testing of a monitoring and feedback tool to stimulate physical activity. Patient Prefer Adherence [Internet]. 2014 [Acesso 25 jul 2014];8(sn):311-322. Disponível em: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3962312/?tool=pubmed
  » http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3962312/?tool=pubmed
• ²¹
  Nabovati E, Vakili-Arki H, Eslami S, Khajouei R. Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst. [Internet]. 2014 [Acesso 25 jul 2014];38(4):35. Disponível em: http://www.ncbi.nlm.nih.gov/pubmed/24682671
  » http://www.ncbi.nlm.nih.gov/pubmed/24682671
• ²²
  Yuan MJ, Finley GM, Long J, Mills C, Johnson RK. Evaluation of user Interface and workflow design of a bedside nursing clinical decision support system. Rev Interact J Med Res. [Internet]. 2013 [Acesso 25 jul 2014];2(1):e4. Disponível em: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3628119/
  » http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3628119/
• ²³
  Polit DF, Beck CT, Hungler BP. Fundamentos de pesquisa em enfermagem: avaliação de evidências para a prática da enfermagem. Porto Alegre: Artmed; 2011. 670 p.
• ²⁴
  Corradi MI, Silva SH, Scalabrin EE. Objetos virtuais para apoio ao processo ensino-aprendizagem do exame físico em enfermagem. Acta Paul Enferm. [Internet]. 2011 [Acesso 5 mar 2013];24(3): 348-53. Disponível em: http://www.scielo.br/pdf/ape/v24n3/07pdf
  » http://www.scielo.br/pdf/ape/v24n3/07pdf
• ²⁵
  Passos JE, Silva TLK. Design de interface do ambiente virtual hyperCAL. Rev Design Tecnol. [Internet]. 2010 [Acesso 13 out 2015];1(sn):88-100. Disponível em: http://www.pgdesign.ufrgs.br
  » http://www.pgdesign.ufrgs.br

• 1
  Paper extrated from Master's Thesis "Educação a distância sobre o método minimamente invasivo para monitoração da pressão intracraniana l", presented to Universidade Federal de São Carlos, São Carlos, SP, Brazil.
• *
  This resources shows the path of pages visited, shows the user where he/she is, and makes it easier to go back to the home page

Publication Dates

• Publication in this collection
  2016

History

• Received
  26 July 2015
• Accepted
  20 Jan 2016