1. Description

This document describes the data analysis of the system output for the ActEV evaluations. In general, the goals of a data analysis are multifold:

- measuring the accuracy of a system
- understanding of key factors that affect the quality and performance of a system
- characterizing the system performance by the factors that are in the research interest
- providing a research direction for system improvements and optimizations.

For the detailed information of tasks and measures, please download the link from evaluation plan

Cautionary note: This document intends to provide general comparative analysis across systems, thus some terminology may NOT be applicable for your results.

The results are included as follows:

2. System Performance (TRECVID)

2.1 Performance Ranking by System

Table 2-1: The ranked list of system performance (ordered by Team)
team_name system_name team_abbrev submission_target mean_nAUDCAtTfa.2 mean_p_missAtTfa.15 mean_w_p_missAtRfa.15 mean_p_missAtRfa.15 relative_processing_time details
BUPT-MCPRL MCPRL_S3 BUPT-M_3541 TRECVID 0.5240797 0.4328029 0.7491433 0.7355342 NA link
Fraunhofer IOSB Team primary FRAUNH_2716 TRECVID 0.8270244 0.7747523 0.8474849 0.8492391 NA link
Hitachi fourth HITACH_3525 TRECVID 0.5988943 0.5099535 0.8240635 0.8194696 NA link
hsmw_tuc hsmw_tuc HSMW_T_3537 TRECVID 0.9406444 0.9355066 0.9437137 0.9508973 NA link
MKLab SecondarySystem MKLAB_1739 TRECVID 0.9642403 0.9578112 0.9617993 0.9675152 NA link
MUDSML MMVG-INF-Etrol MUDSML_3535 TRECVID 0.4840737 0.3915245 0.7978986 0.7888416 NA link
NTT_CQUPT p-NTT-CQUPT NTT_CQ_3519 TRECVID 0.6005797 0.5112179 0.8725374 0.8782028 NA link
UCF UCF-P UCF_3524 TRECVID 0.4909939 0.3858007 0.7022757 0.6796603 NA link
vireoJD-MM Pipeline1 VIREOJ_2016 TRECVID 0.6011979 0.5482670 0.7284571 0.7142309 NA link
Dateset Parts Processed by Team’s Submissions
## [1] "############ Datsset parts are NOT available for this evaluation ###############"
Table 2-2: Min/Max System Performance
submission_target min_max mean_nAUDCAtTfa.2 mean_p_missAtTfa.15 mean_w_p_missAtRfa.15 mean_p_missAtRfa.15
TRECVID min 0.4840737 0.3858007 0.7022757 0.6796603
max 0.9642403 0.9578112 0.9617993 0.9675152
Q 2-1: What is the processing run time for team’s submission?
## [1] "####### Processing run time is NOT avaialble for this evaluation ########"
Q 2-2: What is the ranked list of system detection performance (based on mean_nAUDCAtTfa.2)?

Q 2-3: What is the performance comparison?

For the plot below, the x-axis is the level of submission_target, and the y-axis is the metric nAUDCAtTfa.2. The bold-line within the box indicates a median value across different settings for the level and the points marked in black are settings (e.g., activities).

2.2 Performance Ranking by Activity

Table 2-3: The ranked list of activities over systems
target TRECVID
Riding 0.3713314
vehicle_u_turn 0.4811582
Pull 0.5394238
Open_Trunk 0.5822078
Talking 0.6375526
vehicle_turning_right 0.6433038
Loading 0.6489995
Transport_HeavyCarry 0.6672550
vehicle_turning_left 0.6812192
Closing_Trunk 0.6821525
Unloading 0.6834055
specialized_talking_phone 0.7135458
Closing 0.7679403
Entering 0.7719692
Opening 0.7948339
activity_carrying 0.7971502
specialized_texting_phone 0.7981548
Exiting 0.8018531
Q 2-4: What is the ranked list of the activities across systems (based on nAUDCAtTfa.2) ?

For the plot below, the x-axis is the activity type, and the y-axis is the metric nAUDCAtTfa.2. The point marked in black indicates a mean value across different systems (marked points in green) and the green error bar indicates its standard deviation.

3 Summary of Activities Difficulty

Disclaimer

Certain commercial equipment, instruments, software, or materials are identified in this evaluation plan to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the equipment, instruments, software or materials are necessarily the best available for the purpose.