TREC 2001 Video Track: Details of how various measures were calculated --------------------------------------------------------------------- KI matching process Following process is used to determine matching between known-items and system submitted items. For each system result for one particular topic For each item within the current system result For each known-item within the particular topic Overlapping test as defined. End For. If the current system item matched any known-item Update of the sum of precision needed for average precision. End If. End For. Calculation of average precision for the current system result. End For. --------------------------------------------------------------------- Partial Recall and Partial Average Precision Following process is used to determine the set of unique relevant SB reference shots, which is used for the calculation of both partial recall and partial average precision (partial because the SB collection comprises only half of the search collection). In addition it counts the number relevant for a given system result and topic. For each topic For each system result for the current topic For each item within the current system result If the current item is relevant and has a video name within the # list a SB videos For each SB reference shot within this video If the current SB reference shot and the current item are # matching according to the matching criteria (2/3 and 1/3) # don't exit yet, an item may indicate multiple SB reference # shots are relevant but each such reference shot is counted # only once. If the current SB reference shot is matching for the first # time for the current topic numUniqueRelevant++; End If. If the current SB Reference has not already matched with # any of the current system result numUniqueRelevantInCurrentResult++; End If. End If. End For each SB reference shot... End If. End For each item... Store numUniqueRelevantInCurrentResult for the current system and # current topic for further use. End For each system result... Store of numUniqueRelevant for the current topic for further use. End For each topicr... -- The partial recall -- The partial recall for a system result on a given topic is just calculated and stored in the results file as numUniqueRelevantInCurrentResult / numUniqueRelevant during the enumeration of the results by system and topic. -- The partial average precision -- pseudo-algorithm: For each system result and topic For each item within the current system result from highest ranked # (best)to lowest Find the best matching SB reference shot that has not been already # chosen for the current system result and topic. Calculate the precision at the depth of the above item Add calculated precision to the sum of precisions (part of the formula # for the average precision). End For. Calculate of the partial average precision for the # current topic = sum of precisions / numUniqueRelevant End For. --------------------------------------------------------------------- SB matching process calculations -- The reference -- The reference consists in several files (one per video) containing the transitions for one video. A file named shotBoundaryReferencefiles.xml contains the correspondance between the video names and the reference transitions files. -- The matching process -- Following process is used to determine matching of CUT, short GRADUAL, and GRADUAL transitions. (the definition of a segmentation is: set of transitions of one system for one particular video.) For each system result (which is all the segmentations of one system) For each system segmentation (the results of this system for one video) For each system transition within the current system segmentation Determination of a set A of reference transitions having an # intersection with the current system transition. Determination of a set B of reference CUTs or reference short # GRADUALs transitions within a 5 frames limit outside the limits # of the current system transition. If both set A and set B are empty The current transition is considered as an insertion. Else While the current system segmentation has not matched any # reference transition then iterate reference transitions within # the set B If the current reference transition and the current system one # are the same type If one of both is a CUT and and they match according to the # matching criteria for CUTs This is a match. Else If they match considering the overlaping criteria This is a match. End If. End If. End While. If the current system segmentation has not matched with any of set A It matches with all of the set A. End If. End For each system transition... End For each system segmentation... End For each system result...