We will have separate measures for detection and accuracy in finding shot boundaries - the transitions between shots. This is to provide more diagnostic information about system performance. (Thanks to Arnon Amir for his work on this.)
Detection for cuts and grandual transitions will be measured by precision and recall. However, the detection criteria will only require a single frame overlap between the submitted transitions and the reference transition. This is to make the detection independent of the accuracy of the detected boundaries. For the purposes of detection, we will consider a submitted abrupt transition to include the last pre-transition and first post-transition frames so that it has an effective length of two frames (rather than zero).
The matching will be restricted to a 1-1 mapping between submitted and reference transition. Assume a matrix with reference transitions in the columns (in time sequence, earliest leftmost) and submitted transitions in the rows (in time sequence, earliest at the top, no overlap). The algorithm for determining the best match will be the following:
For each column (reference transition), earliest to latest . Calculate overlap for all rows (submitted transitions) with no match yet and of the same type . If no row has an overlap > 0 . . No match for this column . Else considering only rows with overlap > 0 . If there is a single maximum . . Choose that row as the best match for the current column . Else there are ties . . Calculate frame precision for each tie . . If there is a single maximum . . . Choose it as the best match . . Else . . . Choose the earliest submitted transition among the ties .End For
As last year, gradual transitions can only match gradual transitions and cuts only cuts, except in the case of very short gradual transitions (5 frames or less), which, whether in the reference or in a submission, will be treated as cuts. NOTE: this treatment of short graduals as cuts happens only at evaluation time. Systems must still label any non-abrupt transition as gradual in their submissions.
Last year we expanded each abrupt reference transition by 5 frames in each direction before matching against submitted transitions - this in order to accommodate differences in frame numbering by different decoders. This will continue though the size of the expansion might be adjusted if differences in decoders are found to be larger.
Accuracy for reference gradual transitions successfully detected will be measured using the 1-1 matching list output of the detection evaluation described above. The accuracy measures will be frame-based precision and recall:
"fr-recall" # frames shared by "detected" submitted transition & matching ref. transition = ------------------------------------ # frames of "detected" reference transition "fr-precision" # frames shared by "detected" submitted transition & matching ref. transition = ------------------------------------ # frames of "detected" submitted transition
The above pair will be calculated for each detected gradual reference transition. Averages per detected gradual reference transition will be calculated for each file and for each submitted run.
Note that a system can be very good in detection and have poor accuracy, or it might miss a lot of transitions but still be very accurate on the ones it finds. Cases of "double penalty" from last year would result here as good detection and low accuracy.