Assessing the Reliability of an Annotation Scheme for Indefinites

MSc Project January 2011 -- MSc in Logic

This project led to the following paper:

Maria Aloni, Andreas van Cranenburgh, Raquel Fernández, and Marta Sznajder (2012) Building a Corpus of Indefinite Uses Annotated with Fine-grained Semantic Functions. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC'12), Istanbul, Turkey.

Instructors

Maria Aloni and Raquel Fernández

Description

Work in (computational) linguistics and cognitive science often builds on subjective judgements. Traditionally, it was considered sufficient to rely on the researchers' own interpretation of a few examples. In recent years, however, it has become widely accepted that further evidence is required that other individuals besides the researchers can reliably make the judgements underlying the work. Such evidence is typically gathered by measuring the extent to which several annotators agree on their judgements. In this project, we will study methods for assessing the reliability of linguistic judgements. In particular, we will investigate possible ways of evaluating the reliability of a functional map for indefinite expressions (such as "any" and "some") characterising the possible functions that indefinites can fulfill. The project will consist in (1) designing an evaluation experiment for such a map of functions, (2) annotating some data according to the proposed design, and (3) evaluating the results by calculating inter-annotator agreement and by drawing conclusions on the reliability of the functional map under investigation.

Slides

Inter-Annotator Agreement: References and Resources