Investigating the use of an empirically derived, binary-choice and boundary-definition (EBB) scale for the assessment of english language spoken proficiency

Abstract

The drive towards English medium instruction in modern Turkish higher education has increased the need for valid assessment methodology. However, with ever increasing numbers of test-takers and a limited period of time to carry out language assessment, collecting information about prospective students’ language proficiency can often put a strain on limited resources. The assessment of speaking ability is notoriously difficult and many universities in Turkey do not have the capacity to complete large-scale speaking assessment that is both fair and useful. A common problem with the assessment of spoken ability in this educational context is disagreement between raters with regards to the assessment of spoken proficiency. The current study maps out the procedure for developing an empirically derived rating scale that requires binary choices to distinguish the boundaries between score levels “Empirically derived, Binary-choice, Boundary-definition” (EBB) (Turner & Upshur, 1996) for a test of second language speaking ability. Two groups of trained raters used the EBB scale and an analytical scale to assess speaking samples from 30 English language learners studying in the English preparatory program of a Turkish university. Analysis of the test results was conducted using multifaceted Rasch analysis. Results demonstrated that the use of the EBB scale generated higher levels of inter-rater and intra-rater consistency than the analytical scale. The findings indicate that the use of EBB rating scales may be a viable alternative to traditional rating scales for large-scale testing programs in English medium universities. © 2018, Springer International Publishing AG.