Sanchay News

संचय समाचार

Posts Tagged ‘Annotation Validation

10th April, 2010

leave a comment »

Some more information about the task mode of operation for the Annotation Interfaces (not yet implemented for the Parallel Corpus Markup):

  • The annotation process starts with one copy of the document to be annotated, say, story-1.
  • When an annotator (say, john) claims this document (opens and saves it in the task mode), another copy is created, with the file name story-1-john. This is the file on which the annotator will be working.
  • However, the task name (e.g. story-1) will apply to all the copies.
  • When another user (e.g. terry) has claimed the same task, another copy will be created (story-1-terry).
  • Now the adjudicators can use the annotation comparison facility to compare the two annotations.
  • The adjudicators can select one of the two annotations and make changes to it directly from the comparison facility.
  • When an adjudicator (say, chapman) saves the selected and modified version of one of the annotations, it gets saved with the original document name (story-1), i.e., the original document is overwritten.
  • An option worth considering is whether, instead of overwriting the original document, another copy should be created with the name of the adjudicator (story-1-chapman). But what if the adjudicator is also one of the annotators? That shouldn’t be a problem, because it can’t be true for the same document.
  • However, the copies of the work by the two annotators remain available for more work by the annotators or for any other use later, such as calculating inter-annotator agreement.

That brings me to the facility I will try to add soon: calculating inter-annotator agreement for different levels of annotation.

Also, note that the task mode of working is not yet available for sentence alignment and word alignment interfaces. That is another item on the agenda.