ISCA International Workshop on Speech and Language Technology in Education (SLaTE 2011)
We present an evaluation of a Web-deployed spoken language
CALL system, carried out using crowd-sourcing methods.
The system, Survival Japanese, is a crash course in
tourist Japanese implemented within the platform CALL-SLT.
The evaluation was carried out over one week using the Amazon
Mechanical Turk. Although we found a high proportion of
attempted scammers, there was a core of 23 subjects who used
the system in a responsible manner. The evidence that these
subjects learned from their 111 sessions and 9092 spoken interactions
was significant at P =0.001. Our conclusion is that
crowd-sourcing is a potentially valid method for evaluating spoken
Index Terms. CALL, crowd-sourcing, speech recognition, evaluation, Japanese
Bibliographic reference. Rayner, Manny / Frank, Ian / Chua, Cathy / Tsourakis, Nikos / Bouillon, Pierrette (2011): "For a fistful of dollars: using crowd-sourcing to evaluate a spoken language CALL application", In SLaTE-2011, 117-120.