Results of the 2013 TRECVID MED/MER Evaluation

for the Pre-Specified and AdHoc Event Tests

Release ID MED13_Scores_20130927
Generated Fri Sep 27 07:44:00 EDT 2013

Pre-Specified Event Submissions
Final MED Scores, Initial MER Scores

  PROGAll PROGSub_Subset PROGSub_Only_Submissions
MER-FullSys MED-MAP MED-MAP MED-MAP
Accuracy ObsTextScore PRRT FullSys ASRSys AudioSys OCRSys VisualSys FullSys ASRSys AudioSys OCRSys VisualSys FullSys VisualSys
PS 100Ex AXES 54.18% 1.36 39.17% 34.6%         35.5%            
BBNVISER 64.96% 1.78 50.59% 33.0% 7.6% 12.0% 4.8% 28.2% 34.5% 8.5% 12.8% 5.5% 29.6%    
CERTH-ITI 43.87% 1.06 129.96%           10.5%       10.2% 10.5% 10.2%
CMU 55.51% 1.63 52.83% 30.6% 7.8% 12.6% 3.1% 26.4% 33.4% 9.8% 13.6% 4.2% 28.1%    
Genie 56.44% 0.90 141.23% 23.3% 0.6% 7.8%   19.9% 25.4% 0.4% 8.4%   21.6%    
IBM-Columbia *54.29% *1.90 *16.57% 3.0%   0.3%   3.0% 3.4%   0.3%   3.4%    
MediaMill       28.1%   5.9%   26.0% 29.9%   6.5%   27.6%    
NII       28.2%   7.1%   24.9% 29.7%   8.7%   25.9%    
ORAND                 0.6%         0.6%  
PicSOM 64.34% 1.96 36.39% 6.4%       6.4% 6.7%       6.7%    
SRIAURORA 73.26% 1.58 148.95% 24.7% 3.0% 0.8% 3.7% 22.5% 25.5% 2.9% 0.8% 5.1% 23.3%    
Sesame 64.10% 2.53 41.83% 27.6% 4.0% 5.9% 0.2% 26.1% 29.0% 5.2% 6.5% 0.3% 27.8%    
SiegenKobeMuro       4.1%       4.1% 4.0%       4.0%    
TNO                 10.3%       5.2% 10.3% 5.2%
TokyoTechCanon       24.5%         25.9%            
UMass       13.0%         14.0%            
VIREO **36.91% **2.06 **22.93% 26.5%       25.5% 28.5%       28.3%    
VisQMUL       0.2%         0.2%            
10Ex BBNVISER       16.6% 3.5% 4.4% 3.2% 13.3% 16.6% 3.3% 4.1% 3.8% 13.6%    
CERTH-ITI                 3.0%       3.0% 3.0% 3.0%
CMU       12.6% 2.0% 4.7% 0.8% 11.2% 12.8% 1.8% 4.8% 0.8% 11.6%    
Genie       10.4% 0.3% 2.6%   10.3% 11.0% 0.4% 2.5%   10.4%    
IBM-Columbia       2.2%   0.3%   2.2% 2.6%   0.4%   2.7%    
MediaMill       15.0%   2.6%   14.0% 15.7%   2.6%   14.9%    
PicSOM       3.2%       3.2% 3.3%       3.3%    
SRIAURORA       13.7% 3.0% 0.9% 3.7% 12.4% 13.9% 2.9% 1.1% 5.1% 12.4%    
Sesame       10.3% 1.4% 2.6% 0.2% 11.6% 10.9% 1.7% 2.6% 0.2% 12.3%    
VisQMUL       0.2%         0.2%            
0Ex BBNVISER       5.2% 1.4% 0.5% 2.8% 3.5% 6.1% 1.6% 0.5% 3.8% 4.3%    
CMU       3.7% 1.8% 0.3% 2.1% 2.4% 4.7% 2.1% 0.4% 2.7% 3.0%    
Genie       1.3% 1.7% 1.1%   1.0% 1.9% 1.8% 1.4%   1.1%    
IBM-Columbia       1.6%   0.2%   1.8% 2.0%   0.3%   2.3%    
SRIAURORA       7.0% 3.0% 0.2% 3.7% 6.5% 8.0% 2.9% 0.3% 5.1% 7.3%    
Sesame       2.4% 1.7%   2.3% 1.3% 2.5% 1.9%   3.3% 1.4%    
TNO                 0.4%         0.4%  
UMass       5.6% 2.3%   3.3% 5.1% 6.2% 2.5%   4.5% 5.3%    
VisQMUL       0.2%         0.2%            

Pre-Adjudicated, Ah-Hoc Event Scores

  PROGAll PROGSub_Subset PROGSub_Only_Submissions
MED-MAP MED-MAP MED-MAP
FullSys ASRSys AudioSys OCRSys VisualSys FullSys ASRSys AudioSys OCRSys VisualSys FullSys VisualSys
AH 100Ex AXES 36.6% 1.0% 12.4% 1.1% 29.4% 36.6% 1.0% 14.0% 1.5% 28.8%    
BBNVISER 32.2% 8.0% 15.1% 5.3% 23.4% 31.6% 7.2% 16.7% 6.1% 24.3%    
CERTH-ITI           7.8%       8.2% 7.8% 8.2%
CMU 36.3% 5.7% 16.1% 3.7% 28.4% 36.0% 5.2% 16.8% 5.4% 28.7%    
Genie 20.2% 4.3% 10.1%   16.9% 21.5% 4.0% 10.8%   17.9%    
IBM-Columbia 2.8%   0.2%   2.8% 2.9%   0.2%   3.0%    
MediaMill 25.3%   5.6%   23.8% 25.1%   6.5%   23.7%    
NII 24.9%   8.8%   19.9% 27.4%   10.7%   22.3%    
ORAND 3.8%       3.8% 5.4%       5.4%    
PicSOM 0.6%   0.1%   0.6% 0.8%   0.2%   0.8%    
SRIAURORA 24.2% 3.9% 9.6% 4.3% 20.4% 25.6% 2.5% 10.7% 5.5% 22.4%    
Sesame 25.7% 3.9% 5.6% 0.2% 23.2% 26.9% 2.6% 6.5% 0.2% 24.0%    
TNO           8.2%       5.2% 8.2% 5.2%
TokyoTechCanon 22.3%         23.0%            
UMass 4.9%         5.9%            
VisQMUL 0.2%   0.2%   0.2% 0.2%   0.3%   0.3%    
10Ex BBNVISER 14.3% 4.1% 5.8% 2.3% 10.8% 15.7% 3.4% 6.7% 2.5% 13.3%    
CERTH-ITI           3.0%       3.0% 3.0% 3.0%
CMU 21.2% 2.6% 9.1% 1.1% 16.3% 22.5% 2.0% 8.6% 1.7% 18.4%    
Genie 11.7% 2.5% 3.9%   7.3% 12.8% 2.1% 3.9%   9.2%    
IBM-Columbia 1.5%   0.2%   1.5% 1.3%   0.2%   1.4%    
MediaMill 14.3%   2.7%   13.7% 16.9%   3.4%   15.7%    
PicSOM 2.1%   0.2%   2.2% 3.3%   0.2%   3.4%    
SRIAURORA 14.4% 3.9% 5.4% 4.3% 10.2% 16.1% 2.5% 5.1% 5.5% 12.4%    
Sesame 12.2% 1.4% 2.7% 0.2% 12.9% 14.0% 0.7% 3.4% 0.2% 15.3%    
VisQMUL 0.2%   0.2%   0.4% 0.2%   0.2%   0.5%    
0Ex BBNVISER 8.1% 2.5% 0.6% 3.0% 5.0% 7.4% 2.0% 0.7% 3.0% 5.0%    
CMU 10.1% 3.1% 0.2% 2.8% 5.2% 10.6% 2.1% 0.3% 3.5% 5.5%    
Genie 0.4% 0.4% 0.5%   1.2% 0.6% 0.5% 0.4%   1.3%    
IBM-Columbia 1.1%   0.2%   1.3% 1.0%   0.2%   1.2%    
SRIAURORA 1.4% 3.9% 0.2% 4.3% 0.6% 1.3% 2.5% 0.2% 5.5% 0.6%    
Sesame 2.8% 2.2%   2.2% 1.3% 2.0% 1.1%   2.6% 1.6%    
TNO           0.3%         0.3%  
UMass 1.0% 2.1%   3.9% 0.5% 1.1% 1.7%   4.9% 0.6%    
VisQMUL +0.2%   0.2%   +0.2% +0.2%   0.2%   +0.2%    

Extra Symbols:

* -> MER Results generated from a different set of test clips
** -> MER Results generated from a different set of test clips
+ -> Late Submissions

Notes:

Release History:

2013.09.17 2013.09.27