Validation of a questionnaire on the use of Interactive Response System in Higher Education

Abstracts

Objetivo:

o presente estudo visa conceber e validar um instrumento para medir a percepção dos estudantes universitários sobre a utilização dos Dispositivos Móveis Interativos (MIR) como recurso tecnológico-pedagógico na sala de aula.

Método:

foi aplicado o questionário “Mandos Interactivos de Respuesta para la Mejora del Proceso de Enseñanza-Aprendizaje”, criado ad hoc para esta investigação, composto por 24 itens e aplicado a 142 estudantes universitários.

Resultados:

tanto a análise fatorial exploratória quanto a confirmatória desembocam em 3 dimensões: ambiente de sala de aula, processos de ensino-aprendizagem e avaliação. Os resultados obtidos tanto na confiabilidade (alfa de Cronbach 0,955) como na análise fatorial confirmatória (χ²/gl=1,944, CFI=0,97; GFI=0,78; RMR=0,077; RMSEA=0,08) produzem índices altamente satisfatórios.

Conclusão:

análises estatísticas mostram que este instrumento é válido, confiável e fácil de aplicar para os professores avaliarem a experiência estudantil de aprendizagem centrada no aluno.

Descritores:
Educação; Ensino Superior; Aprendizagem; Estudos de Validação; Pesquisas e Questionários; Tecnologia Educacional

Objetivo:

el presente estudio tiene como objetivo diseñar y validar un cuestionario que permita medir la percepción de los estudiantes universitarios del uso de los MIR como recurso tecnopedagógico en el aula.

Método:

se aplicó el cuestionario Mandos Interactivos de Respuesta para la Mejora del Proceso de Enseñanza-Aprendizaje, creado ad hoc para esta investigación, que está formado por 24 ítems, aplicado a 142 estudiantes universitarios.

Resultados:

tanto el análisis factorial exploratorio como el confirmatorio arrojan tres dimensiones: ambiente de clase, los procesos de enseñanza-aprendizaje y la evaluación. Los resultados obtenidos tanto en la fiabilidad (alfa de Cronbach 0.955) como en el análisis factorial confirmatorio (χ²/gl=1.944, CFI=0.97; GFI=0.78; RMR=0.077; RMSEA=0.08) arrojan índices altamente satisfactorios.

Conclusión:

los análisis estadísticos muestran que este instrumento es una herramienta válida, fiable y de fácil aplicación para el docente que le permitirá evaluar la experiencia del alumnado sobre el aprendizaje centrado en el estudiante.

Descriptores:
Educación; Educación Superior; Aprendizaje; Estudios de Validación; Encuestas y Cuestionarios; Tecnología Educacional

Introduction

Higher Education must establish a space for the exchange of practical experiences that promotes knowledge and research within the common framework of the European and Ibero-American Higher Education Area^(1-2).

The implementation of student-centered learning is oriented to establish a model that effectively integrates technology with knowledge of didactic mediation, evolving towards the Technological Pedagogical and Content Knowledge Model (TPACK). This system combines disciplinary, pedagogical and technological knowledge, but always considering the context in which it intervenes^(3-4), and increasing the interaction between professors and students within a critical dialogical approach⁽²⁾.

Some authors consider necessary to establish didactic knowledge through the relationship between different types of knowledge (coming from the own discipline, general pedagogy and students) and the professor’s biography⁽⁵⁾. Thus, during the nursing initial training, both the pedagogical aspects, including the implementation of Interactive Response System (hereinafter IRS) with remote answering devices to monitor students’ progress, and the involvement of expert professors lead to high-quality teaching, among other issues⁽⁶⁾.

In this regard, encouraging professors to integrate technology into the classroom is crucial, as highlighted in the Horizon Report⁽⁷⁾, since it will significantly impact on education in the coming years. To do this, university professors must use the technological tools they are familiar with, as well as access new technological resources to improve teaching processes⁽⁸⁾.

Similarly, technological changes in university professors follow a tendency and are not radical; they introduce those that are consistent with their teaching practices into the learning activities they normally carry out⁽⁹⁾.

In order to face these technological challenges, an inversion of learning process is required; students should be provided with materials in various formats, so that they can carry out preliminary work before arriving at the classroom, incorporating IRS to verify the improvements in the student centered-learning process^(10-12). IRS has been already integrates in some university classes ⁽¹³⁾. Thus, the available literature about Higher Education on the use of this technology in recent years focuses on the fields of Science, Technology, Engineering and Mathematics, Sociology, Humanities, Health (Medicine and Nursing), Business Administration and English language^(13-14).

The reviewed studies indicate that integrating IRS in university classrooms improves three main areas: the classroom environment, the teaching-learning processes and learning assessment^(15-20). Thus, defining the possibilities and limitations of this tool is increasingly important for the improvement of the quality of Higher Education⁽⁶⁾.

Concerning the classroom environment factor, IRS increases attendance⁽²¹⁾ and student participation^(22-25), resulting in a higher level of involvement during classes in comparison to the traditional methodology^(23,26-27). Within the learning factor, some studies establish that frequent and positive interaction makes classes more dynamic when using IRS^(13,28-29), promoting active learning^(30-32). In addition, attention^(33-34), concentration⁽³⁵⁾ and memory^(36-37) are encouraged during the learning process. Extensive research suggests that a better performance is the result of the use of IRS, as some studies indicate^{(31-32,38-41)}, although other studies do not find such an effect⁽⁴²⁾. In relation to the assessment factor, the findings of the literature support the capacity of IRS as a tool for assessment and feedback^(43-44). It is considered that both students and professors benefit from the feedback they receive with the use of this educational technology^(20,27). All of this leads to a key learning process for the interaction of knowledge and know-how⁽⁴⁵⁾.

The aim of this study is to design and validate this study aims to design and validate a questionnaire to measure the students’ perception of the use of IRS as a technopedagogical resource in the classroom.

Method

The design was transectional and descriptive, as the data were collected in a single time in order to describe the phenomenon and analyze it at a certain time.

The research was carried out at Melilla Campus of the University of Granada (Spain), located in North Africa, whose students attend to the Health Sciences, Social and Legal Sciences, and Education and Sports Sciences schools. For this purpose, an intentional non-probabilistic sample was carried out. The selection criteria have been: firstly, professors who use technology in their classrooms, specifically interactive response devices. And secondly, the willingness of the students to participate in this study. Therefore, the sample comprises 142 students: 110 women (77.5%) and 32 men (22.5%). In relation to the academic year, 17 students are in first year (12%), 95 in second year (66.9%) and 30 in third year (21.1%).

To carry out this study, an ad hoc questionnaire was designed for this research, “Interactive Response System for the Improvement of the Teaching-Learning Process (IRS-ITLP)”.

In relation to the items of the IRS-ITLP, they were written after an extensive bibliographic review on the three factors highlighted above^(21,46-47). Despite of the absence of experts on this field, this process provides validity to the questionnaire items.

We started with 65 items grouped into categories⁽⁴⁸⁾. After the analysis of the items, we selected 35 items that were grouped into the three dimensions: learning environment, process and assessment. Regarding the questionnaire response format, a Likert type scale was used, with 5 response alternatives, ranging from 1, totally disagree, to 5, totally agree.

The research was oriented to different degrees of the University of Granada (Spain), all of them being subjects concerning the basic formation of the students. For this purpose, the collaboration of the teaching staff was requested to participate as volunteer in this project and to integrate IRS in their classes.

The use of the IRS in these basic training subjects was carried out throughout the semester of the 2016-17 academic year, before, during and at the end of the classes. At the end of the semester, the IRS-ITLP questionnaire was applied the last week of the semester, with a duration of approximately 15 minutes, to find out the perception of the experience. Students were asked to agree to participate in this experience voluntarily and anonymously, following the rules of the Committee on Publication Ethics (COPE).

The statistical software SPSS version 20.0 has been used for the statistical processing of the data. To know the reliability of each group of items, Cronbach’s alpha was used, and for the validity of the questionnaire, an Exploratory Factor Analysis was carried. For the Confirmatory Factor Analysis, the program LISREL 8.8 was used.

Results

Firstly, the reliability of the IRS-ITLP questionnaire consisting of 35 elements was analyzed using Cronbach’s alpha internal consistency coefficient, which was 0.965. Although this index was high, we proceeded to eliminate those items whose item-total correlation was inferior to 0.20. Finally, the questionnaire was made up of 24 items with a α=0.955, showing homogeneity indexes ranging from 0.42 to 0.85.

Subsequently, the means, standard deviations, asymmetry, and item-total correlations of each of the items were obtained. As can be seen in Table 1, the asymmetry is negative in all items, which shows a greater concentration of responses corresponding to the high scores in those items.

Table 1
Descriptive values of the items in the IRS-ITLP questionnaire. Granada, Spain, 2017

Nº	Items	M*	SD†	Asymmetry	Correl. item-total
1	I am more focused during the classes since the implementation of IRS	3.40	1.15	-0.477	0.689
2	Thanks to IRS, I measure if I am following correctly the contents of the subject during the classes	3.81	1.01	-0.658	0.619
5	During my experience with the IRS I have a good time learning	3.52	1.18	-0.559	0.566
9	IRS is used to find out the initial knowledge of the students	3.54	1.258	-0.595	0.488
10	The use of IRS is carried out by experienced professors to provide good feedback	3.97	0.891	-0.616	0.610
14	The use of IRS helps me to develop my comprehension on the contents I am working on	3.59	1.162	-0.666	0.639
15	The use of the IRSs makes the classes enjoyable and dynamic	3.82	1.119	-0.877	0.656
18	The use of IRS improves my learning performance	3.76	1.129	-0.653	0.762
19	The continuous use of the IRS increases my class attendance	3.70	1.266	-0.793	0.579
20	The use of IRS allows you to know and compare your colleagues' answers with your own answers	3.42	1.234	-0.646	0.441
21	The use of the IRS allows to correct mistakes or misunderstandings about the subject contents during the classes	3.71	1.121	-1.032	0.558
22	I am more interested in classes when using IRS	3.65	1.066	-0.698	0.768
24	I like the use of IRS as an attendance control	3.65	1.005	-0.605	0.699
29	The use of IRS improves motivation during classes	3.83	1.111	-0.825	0.753
30	The use of IRS allows active discussion of misconceptions to build knowledge	3.80	1.168	-0.771	0.792
35	The use of the IRS evaluates my comprehensive knowledge of the contents in each of the topics covered during the classes	3.99	0.971	-1.211	0.788
36	The use of IRS promotes regular study of the subject to be better prepared for classes	3.64	1.094	-0.762	0.715
42	The use of IRS allows you to be more confident when asking questions during classes	3.76	1.254	-0.739	0.787
46	The use of the IRS is done at the end of the classes to review the contents explained during the session	3.80	1.100	-0.736	0.772
47	The use of IRS makes the classes more pleasant and interactive compared to traditional classes	3.86	1.082	-0.804	0.769
48	The use of IRS improves your participation in classes behind anonymity	3.87	1.104	-0.773	0.712
53	The answers provided through the IRS increase my confidence in the classes after verifying that I answered correctly	3.44	1.164	-0.420	0.700
59	IRS provides valuable information to improve your learning process	4.09	1.017	-1.007	0.687
65	The use of IRS improves the understanding of the contents explained in class	3.87	1.160	-0.911	0.727

• ^*
  M = Mean;
• ^†
  SD = Standard deviation

Since any study about the factor analysis of the IRS-ITLP questionnaire had previously been published, before performing a Confirmatory Factor Analysis it was convenient to carry out an Exploratory Factor Analysis (EFA) to explore how the items are grouped into factors. To ensure that the data fit a factor analysis model, the data were subjected to the Kaiser-Meyer-Olkin test (KMO= 0.941) and to the Bartlett’s Test for Sphericity (c²= 2446.206; df= 300; p<0.001). The values indicate that a factor analysis is a suitable technique to structure the information contained in the matrix. The EFA reveals the existence of 3 factors that explain 61.61% of the total variance, being this proportion acceptable. In addition, the item communalities are above h²=0.40, ranging from 0.421 “The continuous use of IRS increases my class attendance” to 0.791 “The use of IRS allows you to know and compare your colleagues’ answers with your own answers”.

Table 2 shows the factors, items, factor loadings and reliability of each dimension, as well as the interpretation of these factors. To determine the dimensions, the factorial loadings criterion has been followed, being the cutoff value 0.30⁽⁴⁹⁾.

Table 2
Factors, items and loadings obtained in the Exploratory Factor Analysis of the IRS-ITLP. Granada, Spain, 2017

Alpha, Factor loadings	Items,factors and variance explained	1	2	3		h2*	α†
FACTOR 1: Learning environment
F1 51.07%	15.The use of IRS makes the classes enjoyable and dynamic	0.759		0.625			0.926
	30. The use of IRS allows active discussion of misconceptions to build knowledge	0.748	0.303	0.736
	48. The use of IRS improves your participation in classes behind anonymity	0.731		0.650
	42. The use of IRS allows you to be more confident when asking questions during classes	0.685	0.403	0.673
	53. The answers provided through IRS increase my confidence in the classes after verifying that I answered correctly	0.640		0.537
	5.During my experience with IRS I have a good time learning	0.602	0.347	0.484
	22. I am more interested in classes when using IRS	0.577	0.497	0.641
	47. The use of IRS makes the classes more pleasant and interactive compared to traditional classes	0.562	0.491	0.644
	24. I like the use of IRS as an attendance control	0.538	0.480	0.590
	1. I am more focused during the classes since the implementation of IRS	0.483	0.560	0.564
FACTOR 2: Teaching-learning process
F2 5.47%	18. The use of IRS improves my learning performance	0.404	0.726	0.740			0.869
	9. The IRS is used to find out the initial knowledge of the students		0.725	0.550
	2. Thanks to the IRS I measure if I am following correctly the contents of the subject during the classes		0.702	0.599
	59. IRS provides valuable information to improve your learning process		0.628	0.512
	10. The use of IRS is carried out by experienced professors to provide good feedback	0.324	0.576	0.483
	46.The use of the IRS is done at the end of the classes to review the contents explained during the session		0.402	0.687
	19. The continuous use of the IRS increases my class attendance		0.516	0.422
FACTOR 3: Learning Assessment
F3 5.07%	20. The use of IRS allows you to know and compare your colleagues' answers with your own answers			0.864		0.755	0.871
	36.The use of IRS promotes regular study of the subject to be better prepared for classes	0.403		0.678		0.686
	21. The use of the IRS allows to correct mistakes or misunderstandings about the subject contents during the classes		0.361	0.641		0.576
	35. The use of the IRS evaluates my comprehensive knowledge of the contents in each of the topics covered during the classes	0.604		0.523		0.715
	29. The use of IRS improves motivation during classes	0.391		0.491		0.680
	65. The use of IRS improves the understanding of the contents explained in class		0.332	0.486		0.602

• ^*
  h2 = Communality;
• ^†
  α= Cronbach's alpha

Subsequently, the Confirmatory Factor Analysis was performed, and the 3-factors model was tested. The maximum likelihood estimation method was used to analyze the correlation matrix. The Goodness-of-Fit of the proposed model was evaluated using various indicators. The χ²/df (484.13/249) scores 1,944, a value that is within the acceptable standards. Moreover, the Root Mean Square Residual (RMR) is 0.077 and the Root Mean Square Error of Approximation (RMSEA) is 0.080, being both indexes considered acceptable since they are between 0.5 and .08⁽⁴⁹⁾. The Goodness-of-Fit Index (GFI) and the Comparative Fit Index (CFI) (with values of 0.78 and 0.97 respectively) are within the tolerance limits. These results confirm that the 3-factor model fits the data, so the model can be maintained as a plausible explanation for the proposed dimensional structure.

Figure 1
Confirmatory factor analysis of the questionnaire “Interactive Response System for the Improvement of the Teaching-Learning Process”

To test the reliability of the instrument, a Cronbach’s alpha test is carried out, obtaining a total value with a α= 0.955 and the dimensions that make it up, obtaining values that range from α=0.922 for factor 1, “Environment”, to α= 0.869 in factor 2 “Teaching-learning process”. These data show that the reliability of the questionnaire is good in all the factors, being lower in factor 3, “Assessment”. Although this statistic has been widely used in social research, it should be complemented with other analysis, such as the Composite Reliability Index (CF) and the Mean Extracted Variance (MEV). The results obtained are shown in Table 3, being in all cases acceptable.

Table 3
Composite Reliability and Mean Extracted Variance from the factors in the IRS-ITLP questionnaire. Granada, Spain, 2017

Factors	CF*	MEV†
F1: Learning Environment	0.955	0.544
F2: Teaching-learning process	0.930	0.508
F3: Learning assessment	0.863	0.540

• ^*
  CF= Composite Reliability;
• ^†
  MEV = Mean Extracted Variance

Discussion

The aim of this study is the construction of a valid and reliable questionnaire to measure the use of IRS in university student-centred learning. The results provide empirical evidence of the validity of this intervention model using IRS, which allows university professors in general, and health sciences professors in particular, to transform the teaching-learning process. This model encourages and involves students in this process through a more active approach for 21^st century professors, who can measure students’ perception on the use of IRSs, an advance that really makes a difference.

Moreover, apart from a technopedagogical resource, it turns into a playful activity for students in a non-game context, being this model included in an emerging educational methodology called Gamification. In summary, this model provides the opportunity to drastically transform traditional classrooms so that it improves the classroom environment, the learning process and their academic performance in a playful and enjoyable way.

The IRS-ITLP questionnaire consists of 24 items that are grouped into 3 factors: Learning Environment, Teaching-Learning Process and Assessment. In relation to the quality of the items, which was measured through item-to-total correlation, the data indicates high rates ranging from 0.488 to 0.787. These values show a high internal consistency supporting the ideas that the items are correlated, and the scale is accurate. Furthermore, a descriptive analysis on the items show that there is negative asymmetry, which reveals that university students tend to agree with the questionnaire statements.

As for the reliability of the scale, the value for Cronbach ‘s alpha is 0.965, indicating a high reliability. These data are in line with the composite reliability indexes of the 3 factors comprising IRS-ITLP, which reach optimum levels: 0.955, 0.930 and 0.863 for Environment, Teaching-Learning Process and Assessment, respectively, being the minimum acceptable value 0.70.

Furthermore, the validity of the construct was tested through an Exploratory Factorial Analysis (EFA) and a Confirmatory Factorial Analysis (CFA). Firstly, the EFA was carried out since there were no similar validated instruments that measure the perception of university students towards learning; we only knew the dimensions that make up the construct according to the literature consulted.

To do so, firstly the data were assessed obtaining significant values both in the Kaiser-Meyer-Olkin test, (KMO= 0.941) and the Bartlett’s sphericity test (c²= 2446.206; df= 300; p < 0.001). These values indicate that a factorial analysis was an adequate technique for interpreting the information contained in this matrix. This analysis yielded three clearly defined factors that explain 61.61% of the total variance. The results of the CFA confirm the three-factor model. The indexes of goodness used were χ²/df =1.944 (which is within the established standards), the RMR is 0.077 and the RMSEA is 0.080, which are considered acceptable as they are between 0.5 and 0.08⁽⁴⁹⁾. In addition, the GFI representing the joint adjustment is 0.78 and CFI is 0.97, so both values are within the tolerance limits. These results confirm that the data fit to the 3-factor model, supporting the proposed dimensional structure.

For all these reasons, these results allow us to be confident in the reliability and validity of this instrument. Therefore, the construction and validation of this questionnaire make possible the application of this instrument to measure the student’s perception in the use of IRS in the learning process.

Regarding the factor “Learning environment”, reliability was measured using the Cronbach’s alpha coefficient, which scores 0.926 and suggests a high internal consistency. The EFA explains 51.7% of the total variance, being the factor with the highest punctuation in this questionnaire. Among the items comprising this dimension are: item 15: the use of IRS makes the classes enjoyable and dynamic, item 48: the use of IRS improves my participation in the classes behind anonymity, item 47: the use of IRS makes the classes more enjoyable and interactive compared to traditional classes, among others. This factor is decisive since university students with an adequate class environment increase class attendance⁽²²⁾ and improves their participation^(13,17,44). It also comes on interaction between professors and students⁽¹³⁾, and has a positive influence on attention^(33-34) and concentration⁽³⁵⁾, as the studies analyzed show.

The second dimension, which corresponds to the “teaching-learning process” factor, shows an internal consistency index of 0.869 and explains 5.4% of the total variance, significantly lowering the weight of the factor. This dimension refers to those elements that are basic to acquire knowledge, such as debates and interaction between professors and students, which positively affects their learning process since it helps students to review and understand the contents. Among the items that make up this factor are: item 47 the use of IRS makes the classes more pleasant and interactive in comparison with traditional classes, item 59 IRS provides valuable information to improve my learning process, item 46 the use of IRS is done at the end of the classes to review the contents explained during the session.

These elements coincide with studies supporting that this working methodology improves performance^{(31-32,38-41)} thanks to the paradigm shift that implies active learning through the connection between knowledge and know-how⁽⁴⁵⁾ and the frequent and positive interaction, which resulted into more dynamic classes when IRS is used^(13,28). This method promotes material comprehension by acquiring a deeper knowledge, helping to review and understand the contents and improving long-term retention^(36-37); the final result is an improvement in the learning process.

In relation to the third dimension, “Learning Assessment” presents a Cronbach’s alpha of 0.871, and the EFA explains 5.07% of the total variance, showing a similar percentage to the previous factor. This dimension is related to feedback and formative evaluation, which help to correct errors or misunderstandings about the contents of the subject worked on^(4,20,40). The items included in this factor are: item 20 the use of IRS allows you to know and compare your colleagues’ answers with your own answers; item 36 the use of IRS promotes regular study of the subject to be better prepared for classes; item 35, The use of IRS evaluates my comprehensive knowledge of the contents in each of the topics covered during the classes. The immediate feedback has a positive impact on both the students in their learning process and the professor in their teaching process, driving the students’ formative evaluation.

In this way, in our global context with no academic frontiers, the European and Latin American Higher Education areas meet the premise that university system must be in a continuous transformation towards active student learning and lifelong learning. In addition, it is necessary to establish evaluation and accreditation of the work carried out in universities, promoting the transference of knowledge between successive researches among different fields of knowledge. This is the way an interdisciplinary understanding can be developed to assess the reality and to question the traditional consideration of the fields of knowledge as isolated compartments separated by disciplinary boundaries.

Among the limitations of this research, we list the following: firstly, it is necessary to continue increasing the number of participants, since the sample is not excessively large, but it does provide a starting point for transferring it to other fields of knowledge. The reason for this is the limited number of professors at the Melilla Campus of the University of Granada who use technopedagogical resources such as IRS in their classrooms and integrate them to processes of active gamification methodology, which is why it is necessary to promote and explore these resources. It is therefore essential to raise awareness and train university professors in the TPACK (Technological Pedagogical and Content Knowledge) model as a conceptual framework that can guide professors when integrating technology into students’ learning processes, which will promote emerging methodologies such as gamification.

Secondly, improving the instrument is crucial so that it can be used by professors in any field of knowledge and in any situation (as has been done during the COVID-19 confinement). The objective is launching an instrument for future periods, according to the Government and university approaches, that can be used in face-to-face training, B-Learning or E-Learning. Therefore, the instrument needs to be developed by including new items that assess this new virtual education models. In addition, we must deepen the influence of this instrument depending on the different disciplines of the Health Sciences, a research that we intend to carry out in the future. The application of technological advances in the university field comes on the active learning process of students, and it is necessary to know their opinion to achieve progress and improvement in teaching. Thirdly, it is necessary further quasi-experimental research in the several fields of university teaching to analyze the influence of gamified active methodology through these technopedagogical resources and traditional methodology, bearing in mind its scope or relationship with academic performance.

Conclusion

The contribution of this study to knowledge is the design, development and dissemination of a new instrument that allows the measurement and assessment of students’ perception on the use of IRS during university training by using a technopedagogical resource in any field of knowledge (whether Health Sciences, Social and Legal Sciences, Arts and Humanities, or Engineering and Architecture). The instrument considers three fundamental factors: learning environment, teaching-learning process and assessment. This questionnaire can be applied in the classroom during the training, helping to improve the teaching-learning processes. In this sense, Health Sciences professors must establish and explore innovative ways to involve students and stimulate active learning. It is important to incorporate active methods by using interactive response commands in disciplines such as nursing, medicine, pharmacy, paramedical education, psychology, dentistry, physiotherapy, speech therapy, biotechnology, epidemiology, genetics, biochemistry, occupational therapy, human nutrition and dietetics, among others. This instrument provides a positive pedagogical approach in the teaching-learning process for Health Sciences professors and students, improving the academic performance with the purpose of acquiring a deep knowledge of the subjects.

References

• ¹
  Sotillo JA, Rodrígue I, Echart E, Ojeda T. El Espacio Iberoamericano de Educación Superior. Diagnóstico y propuestas institucionales. Madrid: Fundación Carolina, CeALCI; 2009.
• ²
  Mingorance AC, Granda J, Rojas G, Alemany I. Flipped Classroom to Improve University Student Centered Learning and Academic Performance. Soc Sci. 2019;8(11):315-28. doi: https://doi.org/10.3390/socsci8110315
  » https://doi.org/10.3390/socsci8110315
• ³
  Harris J, Mishra P, Koehler M. Teachers' technological pedagogical content knowledge and learning activity types: curriculum-based technology integration refrained. J Res Technol Educ. [Internet]. 2009 [cited Oct 21, 2018];41(4):393-416. Available from: https://files.eric.ed.gov/fulltext/EJ844273.pdf
  » https://files.eric.ed.gov/fulltext/EJ844273.pdf
• ⁴
  Cheung G, Wan K, Chan K. Efficient Use of Clickers: A Mixed-Method Inquiry with University Teachers. Educ. Sci. 2018;8(31):1-15. doi: https://doi.org/10.3390/educsci8010031
  » https://doi.org/10.3390/educsci8010031
• ⁵
  Schubert V, Medina J, Lenise M. Proceso de construcción del conocimiento pedagógico del docente universitario de enfermería1. Rev. Latino-Am. Enfermagem. [Internet]. 2011 [cited Oct 22, 2018];19(2). Available from: http://www.scielo.br/pdf/rlae/v19n2/es_26.pdf
  » http://www.scielo.br/pdf/rlae/v19n2/es_26.pdf
• ⁶
  Cheung G, Wan K, Chan K. Efficient Use of Clickers: A Mixed-Method Inquiry with University Teachers. Educ. Sci. 2018;8(31):1-15. doi: https://doi.org/10.3390/educsci8010031
  » https://doi.org/10.3390/educsci8010031
• ⁷
  Johnson L, Adams S, Estrada V, Freeman A. NMC Horizon Report: 2016 Higher Education Edition. [Internet]. Austin: The New Media Consortium; 2016 [cited Sep 15, 2018]. Available from: http://www.aprendevirtual.org/centro-documentacion-pdf/2016-nmc-horizon-report-HE-ES.pdf
  » http://www.aprendevirtual.org/centro-documentacion-pdf/2016-nmc-horizon-report-HE-ES.pdf
• ⁸
  Kapsalis G, Ferrari A, Punie Y, Conrads J, Collado A, Hotulainen R, et al. Evidence of Innovative Assessment: Literature Review and Case Studies, EUR 29882. Luxembourg: European Commission; 2019. doi: https://doi.org/10.2760/552774
  » https://doi.org/10.2760/552774
• ⁹
  Marcelo C, Yot C, Mayor C. Enseñar con tecnologías digitales en la Universidad. Comunicar. 2015;45:117-24. doi: https://doi.org/10.3916/C45-2015-12
  » https://doi.org/10.3916/C45-2015-12
• ¹⁰
  Hinojo FJ, Mingorance AC, Trujillo JM, Aznar I, Cáceres MP. Incidence of the Flipped Classroom in the Physical Education Students' Academic Performance in University Contexts. Sustainability. 2018;10(5):1334-46. doi: https://doi.org/10.3390/su10051334
  » https://doi.org/10.3390/su10051334
• ¹¹
  Njie-Carr VP, Ludeman E, Lee MC, Dordunoo D, Trocky NM, Jenkins LS. An Integrative Review of Flipped Classroom Teaching Models in Nursing Education. J Prof Nurs. 2017;33(2):133-44. doi: 10.1016/j.profnurs.2016.07.001
  » https://doi.org/10.1016/j.profnurs.2016.07.001
• ¹²
  Wilson J, Walker S. Turning a crisis into an interactive drama: the introduction of a 'clickers theatre' in nurse education. Nurse Educ Today. 2017;51:109-11. doi: 10.1016/j.nedt.2016.11.023
  » https://doi.org/10.1016/j.nedt.2016.11.023
• ¹³
  Carrino SS. Digital connection in a physical classroom: Clickers and the student-teacher relationship [thesis]. Greensboro: University of North Carolina; 2015 [cited Dec 12, 2019]. Available from: https://libres.uncg.edu/ir/uncg/f/Carrino_uncg_0154D_11791.pdf
  » https://libres.uncg.edu/ir/uncg/f/Carrino_uncg_0154D_11791.pdf
• ¹⁴
  Zainuddin Z, Halili SH. Flipped classroom research and trends from different fields of study. Int Rev Res Open Distance Learn. [Internet]. 2016 [cited Jul 18, 2018];17(3):313-40. Available from: https://files.eric.ed.gov/fulltext/EJ1102721.pdf
  » https://files.eric.ed.gov/fulltext/EJ1102721.pdf
• ¹⁵
  Ali RA, Alnatour A, Alnuaimi K, Alzoubi F, Almomani M, Othman A. Effects of interactive teaching on university students' knowledge and attitude toward reproductive health: a pilot study in Jordan. J Multidiscip Healthc. 2018;11:211-21. doi: https://doi.org/10.2147/JMDH.S160135
  » https://doi.org/10.2147/JMDH.S160135
• ¹⁶
  Barcelo JM. Medical laboratory science and nursing students' perception of the academic learning environment at a Philippine university using the Dundee Ready Education Environment Measure. J Educ Eval Health Prof. 2016;13(33):1-7. doi: https://doi.org/10.3352/jeehp.2016.13.33
  » https://doi.org/10.3352/jeehp.2016.13.33
• ¹⁷
  Brady M, Seli H, Rosenthal J. "Clickers" and metacognition: a quasi-experimental comparative study about metacognitive self-regulation and use of electronic feedback devices. Comput Educ. 2013;65:56-63. doi: https://doi.org/10.1016/j.compedu.2013.02.001
  » https://doi.org/10.1016/j.compedu.2013.02.001
• ¹⁸
  Castro MJ, López M, Cao MJ, Fernández-Castro M, García S, Frutos M, et al. Impact of educational games on academic outcomes of students in the Degree in Nursing. PLoS One. 2019;14(7):e0220388. doi: https://doi.org/10.1371/journal.pone.0220388
  » https://doi.org/10.1371/journal.pone.0220388
• ¹⁹
  Nishimura A. Effects of different methods of reflection on nurses' gaze and judgement in a task using a touch panel. J Clin Nurs. 2018;27:569-77. doi: https://doi.org/10.1111/jocn.14096
  » https://doi.org/10.1111/jocn.14096
• ²⁰
  Toothaker R. Millennial's perspective of clicker technology in a nursing classroom: A mixed methods research study. Nurse Educ Today. 2018;62:80-4. doi: https://doi.org/10.1016/j.nedt.2017.12.027
  » https://doi.org/10.1016/j.nedt.2017.12.027
• ²¹
  Chen TL, Lan YL. Using a personal response system as an in-class assessment tool in the teaching of basic college chemistry. Australas J Educ Technol. 2013;29(1):32-40. doi: https://doi.org/10.14742/ajet.95
  » https://doi.org/10.14742/ajet.95
• ²²
  Aktekin NÇ, Çelebi H, Aktekin M. Let's kahoot! Anatomy. Int. J. Morphol. 2018;36(2):716-21. doi: http://dx.doi.org/10.4067/S0717-95022018000200716
  » http://dx.doi.org/10.4067/S0717-95022018000200716
• ²³
  Barr ML. Encouraging college student active engagement in learning: The influence of response methods. Innov High Educ. 2013;39(4):307-19. doi: https://doi.org/10.1111/jcal.12205
  » https://doi.org/10.1111/jcal.12205
• ²⁴
  Iskander M. Systematic review of the implementation of audience response systems and their impact on participation and engagement in the education of healthcare professionals. BMJ Simulation & Technology Enhanced Learning. 2018;4(2):47-50. doi: http://dx.doi.org/10.1136/bmjstel-2017-000245
  » http://dx.doi.org/10.1136/bmjstel-2017-000245
• ²⁵
  Licorish SA, Owen HE, Daniel, B, George, JL. Students' perception of Kahoot!'s in uence on teaching and learning. RPTEL. 2018;13(9):1-23. doi: https://doi.org/10.1186/s41039-018-0078-8
  » https://doi.org/10.1186/s41039-018-0078-8
• ²⁶
  Oliveira C, Tirapelli C, Rodrigues CT, Domaneschi C, Caldeira SA. Interactive audience response systems in oral and maxillofacial radiology undergraduate lectures. Eur J Dent Educ. [Internet]. 2017;22:63-9. doi: https://doi.org/10.1111/eje.12258
  » https://doi.org/10.1111/eje.12258
• ²⁷
  Mohan R. Enhancing student engagement and immediate feedback with clickers and response cards. Int J Innov Learn. 2018;24(1):81-97. doi: https://doi.org/10.1504/IJIL.2018.092924
  » https://doi.org/10.1504/IJIL.2018.092924
• ²⁸
  Banks D. Reflections on the use of ARS with small groups. In: ______. Audience Response Systems in Higher Education: Applications and Cases. Pennsylvania: IGI Global; 2006. p. 373-86.
• ²⁹
  Wood A. Nurse Perceptions of Interactivity during Their Onboarding Orientation: Effect of an Audience Response System [thesis]. Boiling Springs: Hunt School of Nursing; 2017 [cited Dec 12, 2019]. Available from: https://digitalcommons.gardner-webb.edu/cgi/viewcontent.cgi?article=1268&context=nursing_etd
  » https://digitalcommons.gardner-webb.edu/cgi/viewcontent.cgi?article=1268&context=nursing_etd
• ³⁰
  Ismaile S, Alhosban F, Hawamdeh S. Making learning fun to increase nursing students' success: Formative feedback in communication learning. Australas Med J. 2017;10(12):1014-21. doi: https://doi.org/10.21767/AMJ.2017.3228
  » https://doi.org/10.21767/AMJ.2017.3228
• ³¹
  Castro MJ, López M, Cao MJ, Fernández-Castro M, García S, Frutos M, et al. Impact of educational games on academic outcomes of students in the Degree in Nursing. PLoS One 2019;14(7):e0220388. doi: https://doi.org/10.1371/journal.pone.0220388
  » https://doi.org/10.1371/journal.pone.0220388
• ³²
  Corell A, Regueras LM, Verdú E, Verdú MJ, Castro JP. Effects of competitive learning tools on medical students: A case study. PLoS One. 2018;13(3):e0194096. doi: https://doi.org/10.1371/journal.pone.0194158
  » https://doi.org/10.1371/journal.pone.0194158
• ³³
  Oswald KM, Rhoten SE. Improving classroom clicker practices: Effects of incentives and feedback on retention. N Am J Psychol. [Internet]. 2014 [cited Dec 6, 2017];16(1):79-88. Available from: https://www.researchgate.net/publication/285958781_Improving_classroom_clicker_practices_Effects_of_incentives_and_feedback_on_retention
  » https://www.researchgate.net/publication/285958781_Improving_classroom_clicker_practices_Effects_of_incentives_and_feedback_on_retention
• ³⁴
  Marshall LL, Varnon AW. An Empirical Investigation of Clicker Technology in Financial Accounting Principles. J Learn High Educ. [Internet]. 2012 [cited Jun 3, 2018];8(1):7-18. Available from: https://files.eric.ed.gov/fulltext/EJ1144930.pdf
  » https://files.eric.ed.gov/fulltext/EJ1144930.pdf
• ³⁵
  Bojinova ED, Oigara JN. Teaching and learning with Clickers: Are Clickers good for students? Interdiscip J E-Learning Learn Objects. 2011;7:169-83. doi: https://doi.org/10.28945/1506
  » https://doi.org/10.28945/1506
• ³⁶
  Benson JD, Szucs KA, DeIuliis E, Leri A. Impact of Student Response Systems on Initial Learning and Retention of Course Content in Health Sciences Students. J Allied Health. [Internet]. 2017 [cited Feb 12, 2018];46(3):158-63. Available from: https://www.ingentaconnect.com/content/asahp/jah/2017/00000046/00000003/art00008
  » https://www.ingentaconnect.com/content/asahp/jah/2017/00000046/00000003/art00008
• ³⁷
  Njie-Carr VP, Ludeman E, Lee MC, Dordunoo D, Trocky NM, Jenkins LS. An Integrative Review of Flipped Classroom Teaching Models in Nursing Education. J Prof Nurs. 2017;33(2):133-44. doi: 10.1016/j.profnurs.2016.07.001
  » https://doi.org/10.1016/j.profnurs.2016.07.001
• ³⁸
  Rana NP, Dwivedi YK. Can clicking promote learning? measuring student learning performance using clickers in the undergraduate information systems class. J Int Educ Bus. 2017;10(2):201-15. doi: http://doi.org/10.1108/JIEB-06-2016-0010
  » http://doi.org/10.1108/JIEB-06-2016-0010
• ³⁹
  Iwamoto D, Hargis J, Taitano E, Vuong K. Analyzing the efficacy of the testing effect using kahoottm on student performance. Turkish Online J Distance Educ. [Internet]. 2017;18(2):93-80. doi: https://doi.org/10.17718/tojde.306561
  » https://doi.org/10.17718/tojde.306561
• ⁴⁰
  George C, Gallegos C, Tesar AJ, Connor K, Martz K. The use of a game-based learning platform to engage nursing students: A descriptive, qualitative study. Nurse Educ Pract. 2017;27:101-6. doi: https://doi.org/10.1016/j.nepr.2017.08.019
  » https://doi.org/10.1016/j.nepr.2017.08.019
• ⁴¹
  Kalaian SA, Kasim RM. Effectiveness of various innovative learning methods in health science class-rooms: a meta-analysis. Adv Health Sci Educ Theory Pract. [Internet]. 2017;22(5):1151-67. doi: https://doi.org/10.1007/s10459-017-9753-6
  » https://doi.org/10.1007/s10459-017-9753-6
• ⁴²
  Maloney LM, Dilger JP, Werfel PA, Cimino LM. Are Audience Response Systems Worth the Cost ? Comparing Question-Driven Teaching Strategies for Emergency Medical Technician Education. Internet J Allied Health Sci Pract. [Internet]. 2017 [cited May 7, 2018];16(1):1-8. Available from: https://nsuworks.nova.edu/cgi/viewcontent.cgi?article=1704&context=ijahsp
  » https://nsuworks.nova.edu/cgi/viewcontent.cgi?article=1704&context=ijahsp
• ⁴³
  Ismail MA, Mohammad JA. Kahoot: A promising tool for formative assessment in medical education. Educ Med J. 2017;9(2):19-26 doi: https://doi.org/10.21315/eimj2017.9.2.2
  » https://doi.org/10.21315/eimj2017.9.2.2
• ⁴⁴
  Nikou SA, Economides AA. Mobile-based assessment: A literature review of publications in major referred journals from 2009 to 2018. Comput Educ. 2018;125:101-19. doi: https://doi.org/10.1016/j.compedu.2018.06.006
  » https://doi.org/10.1016/j.compedu.2018.06.006
• ⁴⁵
  Montejano-Lozoya R, Gea-Caballero V, Miguel-Montoya I, Juárez-Vela R, Sanjuán-Quiles A, Ferrer-Ferrandiz E. Validation of a questionnaire designed to measure nursing student satisfaction with practical training. Rev. Latino-Am. Enfermagem. 2019;27:e3206. doi: https://doi.org/10.1590/1518-8345.3102.3206
  » https://doi.org/10.1590/1518-8345.3102.3206
• ⁴⁶
  Miles NG, Soares TP. Acceptance of clickers in a large multimodal biochemistry class as determined by student evaluations of teaching: Are they just an annoying distraction for distance students? Biochem Mol Biol Educ. 2016; Oct;44(1):99-108. doi: https://doi.org/10.1002/bmb.20917
  » https://doi.org/10.1002/bmb.20917
• ⁴⁷
  Yeh CR, Tao YH. How benefits and challenges of personal response system impact students' continuance intention? A Taiwanese context. Educ Technol Soc. [Internet]. 2013 [cited Jan 10, 2018];16(2):257-70. Available from: https://pdfs.semanticscholar.org/283d/f5d6cddabfc9abb8d2806efbf809975818cb.pdf
  » https://pdfs.semanticscholar.org/283d/f5d6cddabfc9abb8d2806efbf809975818cb.pdf
• ⁴⁸
  Lin YC, Liu TC, Chu CC. Implementing clickers to assist learning in science lectures: The clicker-assisted conceptual change model. Australas J Educ Technol. 2011;27(6):979-96. doi: https://doi.org/10.14742/ajet.924
  » https://doi.org/10.14742/ajet.924
• ⁴⁹
  Kline R. Principles and Practice of Structural Equation Modeling. New York: The Guilford Press; 2011.

Edited by

Associate Editor: Regina Aparecida Garcia de Lima

Publication Dates

• Publication in this collection
  28 June 2021
• Date of issue
  2021

History

• Received
  12 Dec 2019
• Accepted
  13 Aug 2020