TRECVID 2018 guidelines

Instance Search

Task coordinator: George Awad and Wessel Kraaij

System task:

Data:

Submissions:

We will allow teams to submit multiple runs (to be counted only as one against the maximum allowed) as long as those runs differ only in what set of training examples for a topic are used. The sets will be defined as follows (in the DTD):

Each run will also be required to state the source of training data used from the following options (in the DTD):

  • Each team may submit a maximum of 4 prioritized runs per training example set (note the example set exception mentioned above allowing up to 8 runs in one specific case). All runs will be evaluated but not all may be included in the pools for judgment.

  • Submissions will be identified as either fully automatic or interactive. Interactive runs will be limited to 5 elapsed minutes per search and 1 user per system run.

  • Please note: Only submissions which are valid when checked against the supplied DTDs will be accepted. You must check your submission before submitting it. NIST reserves the right to reject any submission which does not parse correctly against the provided DTD(s). Various checkers exist, e.g., Xerces-J: java sax.SAXCount -v YourSubmision.xml.

  • Here for download (right click and choose "display page source" to see the entire file) is the DTD for search results of one run, the container for one run, and a small example of what a site would send to NIST for evaluation. Please check your submission to see that it is well-formed

  • Please submit each run in a separate file, named to make clear which team it is from. EACH file you submit should begin, as in the example submission, with the DOCTYPE statement and a videoSearchResults element even if only one run is included:

    <!DOCTYPE videoSearchResults SYSTEM "https://www-nlpir.nist.gov/projects/tv2018/dtds/videoSearchResults.dtd">

  • Submissions will be transmitted to NIST via a password-protected webpage .

    Evaluation:

    Important notes

  • The BBC requires allINS task participants fill, sign and submit a renewal data License agreement in order to use the Eastenders data. That means that even if a past participant has a copy of the data, the team must submit a renewal License form before any submission runs can be accepted and evaluated.

  • No usage of previous year's ground truth is allowed in order to filter the current year's search results.

  • No human preknowledge to the closed world of the Eastenders dataset is allowed to be used to filter search results. Any filteration methods should all be automatic without fine tuning based on the Eastenders dataset human knowledge.

  • No manual intervention is allowed to modify testing topics example images. Only automatic methods are allowed.

  • The usage of the included xml transcripts' files are limited to only the transcripted text and not to any other metadata (or xml) attributes (e.g. color of text, etc).

  • Interactive systems essentially use humans to filter or rerank search results, but not to modify testing topics in a preprocessing step.

    Issues: