Evaluating edge detection? Don’t use RMSE, PSNR or SSIM.

Empirical and theoretical evidence for why Figure of Merit (FOM) is the best edge-detection evaluation metric

Image segmentation and edge detection are closely related tasks. Take this output from a coastal segmentation model for example:

The model will classify every pixel as either land or ocean (segmentation mask). Then the coastline is the pixels where this classification changes (edge map). In general, edge detection can be done using the boundaries of the output of an image segmentation model.

I wanted to use this relationship in my research to help evaluate coastal image segmentation models. Similar research all use confusion matrix-based metrics like accuracy, precision and recall. These compare all pixels in a predicted segmentation mask to a ground truth mask.

The problem is these might overestimate performance in the most important region — the coastline.

The majority of pixels are in the middle of the ocean or completely surrounded by land. This makes them easier to classify than those close to the coastline. You can see this in Figure 2. Unfortunately, these errors may be shrouded in the sea of correctly classified pixels.