An adjustable error measure for image segmentation evaluation

Abstract

Due to the subjective nature of the segmentation process, quantitative evaluation of image segmentation methods is still a difficult task. Humans perceive image objects in different ways. Consequently, human segmentations may come in different levels of refinement, ie, under- and over-segmentations. Popular segmentation error measures in the literature (Arbelaez and OCE) are supervised methods (also called empirical discrepancy methods), in which error is computed by comparing objects in segmentations with a reference (ground-truth) image produced by humans. Since reference images can be many, the key issue for a segmentation error measure is to be consistent in the presence of both under- and over-segmentation. In general, the term consistency refers to the ability of the error measure to be low, when comparing similar segmentations, or high, when faced with different segmentations, while capturing under- or over-segmentations. In this paper we propose a new object-based empirical discrepancy error measure, called Adjustable Objectbased Measure (AOM). We introduce a penalty parameter which gives the method the ability to be more (or less) responsive in the presence of over-segmentation. Hence, we extend the notion of consistency so as to include the application’s need in the process. Some applications require segmentation to be extremely accurate, hence under- or over-segmentation should be well penalised. Others, do not. By changing the penalty parameter, AOM can deliver more consistent results not only in reference to the under- or over-segmentation issue alone, but also according to the nature of the application. We compare our method with Arbelaez (used as standard measure in the benchmark of Berkeley Segmentation Image Dataset) and OCE. Our results show that AOM not only is more consistent in the presence of over-segmentation, but is also faster to compute. Unlike Arbelaez and OCE, AOM also satisfies the metric axiom of symmetry.