Cognitive Task Analysis as a Tool to Enhance Raters’ Reliability at Scoring a Writing Admission Test
Kaydedildi:
| Yazarlar: | , |
|---|---|
| Materyal Türü: | artículo original |
| Durum: | Versión publicada |
| Yayın Tarihi: | 2026 |
| Diğer Bilgiler: | Cognitive Task Analysis (CTA) has proven useful to design training programs in diverse areas of human professional and academic endeavors. This paper explores its use as a tool to increase levels of validity, reliability, and fairness, among other assessment components, and therefore, project a forthcoming calibration of the rating process of a high-stake writing evaluation, by analyzing the thoughts and process description of expert raters as they graded candidates’ essays. This research implemented CTA data elicitation techniques in the form of recordings, transcripts, and reports provided by subject matter experts (SMEs). First, in stage one, volunteer students were asked to write an essay that had the same input, purpose, and conditions of the real test. Next, four SMEs graded three of these essays and recorded their thoughts in oral and written form based on specific guidelines. The information was analyzed, and six preliminary categories of analysis were identified. In stage two, after a qualitative analysis and systematization of information, three of the six categories were determined to be part of the raters’ scope of action: work operationalization (procedural knowledge), cognitive processes for the rating tasks (declarative knowledge), and the assessment instrument. During this phase, the test takers wrote real essays. Two raters were assigned three specific pieces of writing from this batch and asked to record their thoughts and process as they performed the task. The results of the analysis of the three categories are expected to prove useful in addressing the improvement of internal processes in the long term by including them in future rater training processes. |
| Ülke: | Portal de Revistas UCR |
| Kurum: | Universidad de Costa Rica |
| Repositorio: | Portal de Revistas UCR |
| Dil: | Inglés |
| OAI Identifier: | oai:portal.revistas.ucr.ac.cr:article/5385 |
| Online Erişim: | https://revistas.ucr.ac.cr/index.php/rrlm/article/view/5385 |
| Anahtar Kelime: | Cognitive Task Analysis writing task writing rating task operationalization validity análisis de tareas procesos cognitivos evaluación escritura validación operacionalización de tareas |