Annotator-Machine-Interaction

This internal project involves experiments that investigate annotator behaviour for difficult tasks. Since 2019, the OVGU team designs experiments in the KMD Experiment Lab to study annotator confidence and its association to annotation quality for labeling tasks. This expertise will be used in the ITN to assist in the process of identifying outliers in the data.

In the following we briefly present two current experiments:
We conduct annotation experiments on health data from a population-based study in cooperation with the University Medicine Greifswald. The triplet annotation task is to decide whether an individual was more similar to a healthy one or to one with a given disorder. We use hepatic steatosis as example disorder, and described the individuals with 10 pre-selected characteristics related to this disorder. We record task duration, electro-dermal activity as stress indicator, and uncertainty as stated by the experiment participants (n = 29 non-experts and three experts) for 30 triplets. We built an Artificial Similarity-Based Annotator (ASBA) and compar its correctness and uncertainty to that of the experiment participants.

More information can be found under publications.

This first experiment formed the basis for further ongoing experiments on the influence of configurations on annotator performance.

Technologies plays an important role for the annotation quality of labeling tasks. Therefore we work at the moment on a Systematic Review on Virtual Reality for Medical Annotation Tasks. Our work investigates benefits offered by VR towards better experience and towards better understanding of annotators.