The previous Semantic Indexing task (run from 2010-2015) addressed the problem of automatic assignment of predefined semantic tags representing visual or multimodal concepts to video segments. The new Ad-hoc search task started in 2016 goal is to model the end user search use-case, who is looking for segments of video containing persons,objects,activities,locations, etc. and combinations of the former.
In 2017 the task will again support experiments in the no annotation condition. The idea is to promote the development of methods that permit the indexing of concepts in video shots using only data from the Web or archives without the need of additional annotations. The training data could for instance consist of images or videos retrieved by a general purpose search engine (e.g. Google) using only the query definition with only automatic processing of the returned results. This will not be implemented as a new variant of the task but by using additional categories for the training types besides the A to D ones (see below). By "no annotation", we mean here that no annotation should be manually done on the retrieved samples (either images or videos). Any annotation done by somebody else prior to the general search does not count. Methods developed in this context could be used for building indexing tools for any concept starting only from a simple query defined for it.
Given the test collection (IACC.3), master shot reference, and set of Ad-hoc queries (approx. 30 queries) released by NIST, return for each query a list of at most 1000 shot IDs from the test collection ranked according to their likelihood of containing the target query.
The current test data set (IACC.3) is 4593 Internet Archive videos (144GB, 600 total hours) using videos with durations between 6.5min and 9.5min.
The development data set combines the development and test data sets of the:
Examples of previous Ad-hoc queries (used in 2016) can be found here.
There will be 3 types of participation:
The same data IACC.3 will be used by VBS participants. While VBS supports two kind of tasks: Known-item search and Ad-hoc search, participation in any of the two tasks is optional and teams may choose to join both tasks. Interactive systems at VBS joining the Ad-hoc task will be tested real-time on a subset of random selected queries (from the 30 selected for TRECVID 2017). For questions about participation in the next VBS please contact the VBS organizers: Werner Bailer, Cathal Gurrin, or Klaus Schoeffmann.
P l e a s e n o t e t h e s e r e s t r i c t i o n s and this information on training types. The submission types (automatic and manually-assisted) are orthogonal to the training types (A, B, C ...).
Two main submission types will be accepted:
Each team may submit a maximum of 4 prioritized runs, per submission type, with 2 additional if they are of the "no annotation" training type and the others are not. The submission formats are described below.
Please note: Only submissions which are valid when checked against the supplied DTDs will be accepted. You must check your submission before submitting it. NIST reserves the right to reject any submission which does not parse correctly against the provided DTD(s). Various checkers exist, e.g., Xerces-J: java sax.SAXCount -v YourSubmision.xml.
All queries (approx. 30) will be evaluated by assessors at NIST after pooling and sampling.
Please note that NIST uses a number of rules in manual assessment of system output.