The TRECVID ActEV 2019 leaderboard evaluation is starting from May 01 and ending on August 29th. More details about the evaluation is on the ActEV 2019 website https://actev.nist.gov/trecvid19 and the updated ActEV 2019 Evaluation Plan:  https://actev.nist.gov/pub/Draft_ActEV_2019_EvaluationPlan_042519.pdf

 

ActEV Scorer (Metrics):

The ActEV Scorer (https://github.com/usnistgov/ActEV_Scorer) has been updated. The ActEV 2019 evaluation is based on the “ActEV19_AD” Protocol:

ActEV19_AD Protocol

• This combines both the 1-Second time constraint (ActEV18_AD_1SECOL) and the new time-based false alarm metric (ActEV18_AD_TFA protocol).

 

ActEV18_AD_1SECOL

• This protocol relaxes temporal overlap of system/reference activity instances.  They are required to overlap by greater than 1 second (rather than the intersection-over-union minimum in section 6.1.1 of the Eval plan).  Further, the E(IoU)*Temporal (IoU) term in the kernel function was removed.
• There’s a new threshold-based scoring file ‘scores_by_activity_and_threshold.csv’

 

ActEV18_AD_TFA

• This protocol implements the new “Time-Based False Alarm” (TFA) measure for system evaluation. 
• TFA is defined to be the duration of system instances that do not overlap ANY reference instances divided by the time of video for which no reference annotations exist. 
• This protocol generates new measurements as follows:
• New DET Curves: figures/DET_TFA_*
• New TFA lines in the score’s files with the label ‘[email protected]’.  We presently are using the same thresholds as rfa.  We are going to revise these TFA thresholds after the program establishes new performance goals.

 

ActEV 2019 Data:

The TrecVID ActEV 2019 evaluation is only based on the VIRAT V1 and V2 dataset with 18 target activities (see Eval Plan). The Multiview Extended Video with Activities (MEVA) data will be used in a future evaluation, but we would like the participants to help us annotate the training data for the 38 activities as defined in the annotation guide (coming soon).

 

To download the data, complete these steps:

• Get an up-to-date copy of the ActEV Data Repo via GIT. You'll need to either clone the repo (the first time you access it) or updated a previously downloaded repo with 'git pull'. Note: this is the same repo as used for MEVA.

Clone: git clone https://gitlab.kitware.com/actev/actev-data-repo.git 

Update: cd "Your_Directory_For_actev-data-repo"; git pull

• Add VIRAT-V1 and VIRAT-V2 download credentials:
• Change your working directory the the top-level of the repo.

cd "Your_Directory_For_actev-data-repo"

• Follow the steps in the top-level README.
• For Step 2 in the download instructions, use these two commands to add your access credentials. (Please do not email this command!)

 

  python ./scripts/actev-corpora-maint.py --operation summary --corpus VIRAT-V1 --add_credential '{"corpus": "VIRAT-V1", "urls":   {"https://mig.nist.gov/datasets/VIRAT-V1": {"type": "file_store", "user": "VIRATv1", "password": "???????"}}}'

 

Similarly download the VIRAT V2.