Higher-order thinking skills (HOTS) mathematical problem-solving was first introduced in the Form-3 assessment (PT3) in 2014. However, to date, there have been no studies of students' ability to solve the mathematical problems in this assessment. Therefore, this study investigated the emerging patterns and problems of HOTS mathematical problem-solving in the PT3. This investigation was a case study and classified under a qualitative research approach. Oral reporting (i.e. thinking aloud protocol) was used to obtain the data. The participants were 10 Form-3 students who were candidates for PT3 in 2015. They were students in a secondary school in a district in Johor Bahru. The results show that students who successfully solved the HOTS mathematical problems produced the same process starting with understanding, followed by phases of planning, implementation, and ending with the final answer. The students who failed to answer the HOTS mathematics questions produced a solution pattern starting with understanding followed by planning and implementation. Based on the patterns, this study also identified the problems that emerged in every step of the HOTS mathematical problems-solving processes and discusses how they could be overcome and improved.

Resumo As competências metacognitivas desempenham um papel importante na resolução de problemas matemáticos. No entanto, existe falta de estudos empíricos sobre o papel das competências metacognitivas na resolução de problemas matemáticos, em particular os não rotineiros. Portanto, este estudo foi realizado para identificar competências metacognitivas dos alunos e o impacto de tais competências na resolução de problemas matemáticos não-rotineiros. Usando um método quantitativo, um total de 304 estudantes no distrito de Johor Bahru foram envolvidos no estudo. Questionário de Monitorização Autónomo (SMQ) e teste de matemática foram utilizados na recolha de dados. Os dados obtidos foram analisados por meio de estatística descritiva e inferencial, como a frequência, porcentagem, média, o teste de Mann-Whitney e Kruskal-Wallis H. Os resultados mostraram que o nível de desempenho dos alunos na resolução de problemas matemáticos não-rotineiros era muito baixa. Conclui-se uma diferença significativa nas competências metacognitivas entre estudantes com diferentes níveis de desempenho na resolução de problemas matemáticos não rotineiros. Concluiu-se que as competências metacognitivas devem ser enfatizadas no processo de resolução de problemas matemáticos não rotineiros.

Abstract Metacognitive skills play an important role in solving mathematical problems. However, there is a lack of empirical studies on the role of metacognitive skills in solving mathematical problems, particularly non-routine ones. Therefore, this study was undertaken to identify students' metacognitive skills and the impact of such skills on non-routine mathematical problem solving. By using a quantitative method, a total of 304 students in Johor Bahru district were involved in the study. A Self-Monitoring Questionnaire (SMQ) and a mathematical test were used in data collection. Data were analysed using descriptive and inferential statistics such as frequency, percentage, mean, the Mann-Whitney U test, and the Kruskal-Wallis H test. Results showed that the level of the students' performance in solving non-routine mathematical problems was very low. There was also a significant difference in the metacognitive skills among students with different performance levels in solving non-routine mathematical problems, and we concluded that these metacognitive skills should be emphasised in this process.