Proposal of statistical analysis to support the assessment method of a Brazilian Industrial Engineering course

Gaudêncio, Juliana Helena Daroz; Scheidegger, Anna Paula Galvão; Pinto, Camila Pereira; Turrioni, João Batista; Turrioni, Ana Maria Silveira

doi:10.1590/0103-6513.218916

Acessibilidade / Reportar erro

Brasil

Español English

sumário « anterior atual seguinte »

Sumário

Research Article • Prod. 27 (spe) • 2017 • https://doi.org/10.1590/0103-6513.218916 copy

Proposal of statistical analysis to support the assessment method of a Brazilian Industrial Engineering course

Authorship SCIMAGO INSTITUTIONS RANKINGS

Abstract

The requirement to adapt the university education to demands of society and professional market is improving the teaching methodologies that try to develop skills and prepare the students to deal with day-to-day situations of a business environment. Therefore, this research presents the assessment method utilized in an Industrial Engineering course carried out by a Brazilian public university in partnership with a multinational company. The course had 37 participants, consisting of: 28 students divided into four groups; 1 teacher; 4 university tutors; and 4 company tutors. The main objective is to assess the consistency of grades assigned to students and their work groups using agreement, variance and correlation analyses. As conclusion, the analyses indicated a possible deficiency in the assessment method application since the values of agreement and correlation coefficients were lower than expected but also provided a positive contribution to the improvement of assessment and the course as a whole.

Keywords
Assessment method; Active learning; Active teaching; Higher education; Engineering education

Assessment	Method of assessment	Periodicity	Grade composition
Peer	1. Questionnaire with the quantitative scale	Unique	20%
University tutor-student	1. Unstructured and non-standardized	Weekly	20%
University tutor-student	2. Questionnaire with the quantitative and the qualitative scale about the presentation	Unique	20%
Professor-student	1. Evaluation of presence and participation in class	Weekly	30%
Professor-student	2. Evaluation of the submitted project reports	Twice	30%
Company tutor-group	1. Unstructured and non-standardized	Monthly	30%
Company tutor-group	2. Questionnaire with the quantitative and the qualitative scale about the presentation	Unique	30%

Criteria
1. Was the exhibition structured clearly?
2. Did the presenter know to conduct the sequence between the topics of slides?
3. Was the main idea of the work presented clearly and understandable?
4. Were examples presented relevant?
5. Were examples presented interest?
6. Was the pace of the presentation adequate?
7. Was the volume of information in the presentation adequate?
8. Was the tone of the presenter's voice adequate?
9. Were the slides efficiently used?
10. Was the presentation well prepared?
11. Was the presentation interesting and motivating?
12. Was the presenter’s eye contact with the audience right and proper?
13. Did the presenter manage to hold the audience's attention?
14. Did the presenter show / summarize the main points of the presentation?
15. Did the presenter demonstrate confidence and knowledge about the topic presented?
16. Was the time spent in the presentation well managed?

Appraiser	Inspected	Matched	Percent	95% CI^a a Confidence interval (CI).
T1	64	14	21.88	(12.51; 33.97)
T2	64	29	45.31	(32.82; 58.25)
T3	64	18	28.13	(17.60; 40.76)
T5	64	21	32.81	(21.59; 45.69)
T6	64	14	21.88	(12.51; 33.97)
T7	64	19	29.69	(18.91; 42.42)

Kappa Statistic	Strength of Agreement
< 0.00	Poor
0.00-0.20	Slight
0.21-0.40	Fair
0.41-0.60	Moderate
0.61-0.80	Substantial
0.81-1.00	Almost perfect

Appraiser	Response	Kappa
T1	Overall	-0.0329
T2	Overall	0.2092
T3	Overall	0.0403
T5	Overall	0.1485
T6	Overall	-0.0147
T7	Overall	0.0890

Criteria
1. Empathy	2. Sanity
3. Communication	4. Initiative
5. Proactivity	6. Flexibility
7. Organization	8. Decision-making
9. Self-development	10. Ethic
11. Committal	12. Responsibility
13. Ecological awareness	14. Interpersonal relationship
15. Team spirit	16. Customer focus
17. Focus on results	18. Guidance for quality

Group	Students	Pvalue	Valid hypothesis	Population’s average
1	S1, S2, S3, S4, S5, S6, S7	0.000	H₁	Different
1^b b Analysis without students with lower or higher grades in relation to the others.	S1, S2, S3, S5, S6, S7	0.574	H₀	Equal
2	S1, S2, S3, S4, S5, S6, S7	0.996	H₀	Equal
3^a a Analysis without the presence of outliers;	S1, S2, S3, S4, S5, S6, S7	0.000	H₁	Different
3^b b Analysis without students with lower or higher grades in relation to the others.	S3, S4, S5, S6, S7	0.000	H₁	Different
4	S1, S2, S3, S4, S5, S6, S7	0.013	H₁	Different
4^b b Analysis without students with lower or higher grades in relation to the others.	S1, S2, S3, S5, S7	0.411	H₀	Equal

Groups	Pvalue	Valid hypothesis	Population average
1, 2, 3, 4	0.000	H₁	Different
3 and 4	0.197	H₀	Equal

	S1	S2	S3	S5	S6
S2	0.253
S3	0.220	0.629^* * Significant p-value (pvalue < 0.05).
S5	0.175	0.207	0.160
S6	0.308	0.413	0.190	0.462
S7	0.056	-0.115	-0.137	-0.159	-0.034

(a)	S2	S3	S4	S5	S6	(b)	S1	S3	S4	S5	S6
S3	0.295					S3	0.250
S4	0.441	0.187				S4	0.159	0.443
S5	0.178	-0.056	-0.055			S5	0.335	0.393	0.283
S6	0.520^* * Significant p-value (pvalue < 0.05).	0.064	0.441	0.112		S6	-0.028	0.618^* * Significant p-value (pvalue < 0.05).	0.292	0.163
S7	0.272	-0.147	0.226	0.053	0.316	S7	0.082	0.612^* * Significant p-value (pvalue < 0.05).	0.222	0.483	0.284

(a)	S1	S2	S3	S5	S6	(b)	S1	S2	S3	S4	S5
S2	-0.008					S2	-0.388
S3	-0.333	-0.332				S3	-0.483	0.262
S5	-0.120	0.286	-0.263			S4	0.182	-0.194	-0.267
S6	0.246	-0.063	0.038	-0.306		S5	0.090	-0.279	-0.160	-0.069
S7	0.279	-0.102	-0.035	-0.193	0.976^* * Significant p-value (pvalue < 0.05).	S7	0.055	0.208	0.228	0.069	-0.037

	T1	T2	T3	T4	T5	T6
T2	0.343
T3	0.365	0.346
T4	0.414	0.290	0.305
T5	0.394	0.510^* * Significant p-value (p-value < 0.05).	0.372	0.405
T6	-0.288	0.237	-0.107	0.010	0.226
T7	0.139	0.357	0.152	0.521^* * Significant p-value (p-value < 0.05).	0.461	0.342

	T1	T2	T3	T4	T5	T6
T2	-0.115
T3	-0.241	0.139
T4	0.423	0.422	0.232
T5	-0.159	-0.092	0.362	-0.071
T6	0.524*	-0.207	0.255	0.377	0.300
T7	0.124	0.645^* * Significant p-value (pvalue < 0.05).	-0.050	0.119	0.099	-0.051

	T1	T2	T3	T4	T5	T6
T2	0.390
T3	0.554^* * Significant p-value (pvalue < 0.05).	0.192
T4	0.221	0.232	0.031
T5	0.074	0.259	0.133	-0.124
T6	0.048	0.218	-0.374	-0.313	0.427
T7	0.222	0.170	0.336	-0.174	0.540^* * Significant p-value (pvalue < 0.05).	0.229

	T1	T2	T3	T4	T5	T6
T2	0.654^* * Significant p-value (pvalue < 0.05).
T3	0.123	-0.338
T4	0.563^* * Significant p-value (pvalue < 0.05).	0.363	0.267
T5	0.330	0.324	-0.122	-0.226
T6	0.523^* * Significant p-value (pvalue < 0.05).	0.692^* * Significant p-value (pvalue < 0.05).	-0.048	0.678^* * Significant p-value (pvalue < 0.05).	0.238
T7	-0.170	-0.145	-0.147	0.021	-0.347	-0.100

Associação Brasileira de Engenharia de Produção Av. Prof. Almeida Prado, Travessa 2, 128 - 2º andar - Room 231, 05508-900 São Paulo - SP - São Paulo - SP - Brazil
E-mail: production@editoracubo.com.br

Acompanhe os números deste periódico no seu leitor de RSS

[1] * juliana_hdg@yahoo.com.br