Call for Participation in TRECVID 2018
CALL FOR PARTICIPATION in the
2018 TREC VIDEO RETRIEVAL EVALUATION (TRECVID 2018)
February 2018 - November 2018
Conducted by the National Institute of Standards and Technology (NIST)
with additional funding from other US government agencies
I n t r o d u c t i o n:
The TREC Video Retrieval Evaluation series (trecvid.nist.gov) promotes
progress in content-based analysis of and retrieval from digital video
via open, metrics-based evaluation. TRECVID is a laboratory-style
evaluation that attempts to model real world situations or significant
component tasks involved in such situations. In it's 18th annual evaluation
cycle TRECVID will evaluate participating systems on 6 different video
analysis and retrieval tasks using various types of real world datasets.
D a t a:
In TRECVID 2018 NIST will use at least the following data sets:
* IACC.3
The IACC.3 is a new dataset introduced in 2016 of approximately 4600 Internet Archive
videos (144 GB, 600 h) with Creative Commons licenses in MPEG-4/H.264
format with duration ranging from 6.5 min to 9.5 min and a mean duration
of almost 7.8 min. Most videos will have some metadata provided by the
donor available e.g., title, keywords, and description.
* IACC.2.A, IACC.2.B, and IACC.2.C
Three datasets - totally approximately 7300 Internet Archive
videos (144GB, 600 hours) with Creative Commons licenses in
MPEG-4/H.264 format with duration ranging from 10 seconds to 6.4
minutes and a mean duration of almost 5 minutes. Most videos will have
some metadata provided by the donor available e.g., title, keywords,
and description
* IACC.1.A, IACC.1.B, and IACC.1.C
Three datasets - totaling approximately 8000 Internet Archive videos
(160GB, 600 hours) with Creative Commons licenses in MPEG-4/H.264
format with duration between 10s and 3.5 minutes. Most videos will
have some metadata provided by the donor available e.g., title,
keywords, and description
* IACC.1.tv10.training
Approximately 3200 Internet Archive videos (50GB, 200 hours)
with Creative Commons licenses in MPEG-4/H.264 with durations
between 3.6 and 4.1 minutes. Most videos will have some metadata
provided by the donor available e.g., title, keywords, and
description
* BBC EastEnders
Approximately 244 video files (totally 300GB, 464 hours) with
associated metadata, each containing a week's worth of BBC EastEnders
programs in MPEG-4/H.264 format.
* Twitter Vine videos
Approximately 50,000 video clips URLs have been collected by NIST
from the public Twitter stream of Vine videos. Only a subset of at
least 2,000 videos will be used with their human annotated text descriptions.
* Twitter and Flickr images and videos
Approximately 188000 images and over 44000 videos were collected from
Twitter social media events and over 26500 images from Flickr. These data
will be used to support the social-media video storytelling linking task.
* Gatwick and i-LIDS MCT airport surveillance video
The data consist of about 150 hours obtained from airport
surveillance video data (courtesy of the UK Home Office). The
Linguistic Data Consortium has provided event annotations for
the entire corpus. The corpus was divided into development and
evaluation subsets. Annotations for 2008 development and test
sets are available.
T a s k s:
In TRECVID 2018 NIST will evaluate systems on the following tasks
using the [data] indicated:
* AVS: Ad-hoc Video Search (automatic, manually-assisted, relevance feedback) [IACC.3]
A new Ad-hoc search task started in TRECVID 2016 and will continue in 2018
to model the end user search use-case, who is looking for segments of video containing
persons, objects, activities, locations, etc. and combinations of the
former. Given about 30 multimedia topics created at NIST, return
for each topic all the shots which meet the video need expressed by it,
ranked in order of confidence. Although all evaluated submissions will be
for automatic runs, Interactive systems will have the opportunity to
participate in the Video Browser Showdown (VBS) in 2019 using
the same testing data (IACC.3).
* ActEV: Activities in Extended Video (ActEv)
The ActEV evaluation seeks to evaluate robust automatic activity detection algorithms
for a multi-camera streaming video environment. ActEV is an extension of the annual TRECVID
Surveillance Event Detection (SED) evaluation by adding a large collection of multi-camera
video data, both of simple and complex activities. ActEV will address activity detection for
both forensic applications and for real-time alerting.
* INS: Instance search (interactive, automatic) [BBB EastEnders]
An important need in many situations involving video collections
(archive video search/reuse, personal video organization/search,
surveillance, law enforcement, protection of brand/logo use) is to
find more video segments of a certain specific person, object,
or place, given a visual example. In 2016 new query type was tested
by asking systems to retrieve specific persons in specific locations.
The same query type will continue in 2018 but with new testing topics.
A set of master locations with various video examples will be given and
each topic will include few examples (image and video) of a person and
ask systems to find that person in one of the known locations.
* SMKBP: Streaming Multimedia Knowledge Base Population
Given streams of text, speech, images, video, and their associated metadata,
from a variety of genres, both formal (e.g.,news) and informal (e.g., social media, blogs),
systems are asked to analyze each incoming information item (~100k items in total)
and produce a set of structured representations (knowledge elements) about events,
sub-events or actions, entities, relations, locations, time, and sentiments (beliefs)
that are observable in that information item given an ontology/schema and zero or more
background (context) information.
This is a joint task with the Text Analysis Conference (TAC) run by NIST.
* LNK: Social-media video storytelling linking [Twitter and Flickr images/videos]
The social-media video storytelling linking task seeks to advance the area of
visual summarization, with collaborative videos, images and texts available from
professional media and social-media users. The challenges in creating such visual
timelines are various: alignment of videos and images through sound or timestamps
to smooth transitions or remove duplicates, detection of video intervals of high
interest, caption generation for a group of pictures, among others. The goal of the
task is to illustrate a news story with social-media content. Starting from a news
story topic and a stream of social-media video and images, the goal is to link a
story-segment to image and video material, while also preserving a good flow of the
whole visual story.
* VTT: Video to Text Description [Twitter Vine videos]
Automatic annotation of videos using natural language text descriptions
has been a long-standing goal of computer vision. The task involves
understanding of many concepts such as objects, actions, scenes, person-object
relations, temporal order of events and many others. In recent years there has
been major advances in computer vision techniques which enabled researchers to
start practically to work on solving such problem. A lot of use case application
scenarios can greatly benefit from such technology such as video summarization,
facilitating the search and browsing of video archives using such descriptions,
describing videos to the blind, etc.
Given a set of Vine videos URLs and number of reference sets of text descriptions,
systems are asked to work and submit results for two subtasks. A "Matching and Ranking"
subtask to return for each video URL a ranked list of the most likely text description
that correspond (was annotated) to the video from each of the reference sets. And a
"Description Generation" subtask to automatically generate for each video URL a text
description (1 sentence).
In addition to the data, TRECVID will provide uniform scoring
procedures, and a forum for organizations interested in comparing
their approaches and results.
Participants will be encouraged to share resources and intermediate
system outputs to lower entry barriers and enable analysis of various
components' contributions and interactions.
***************************************************
* You are invited to participate in TRECVID 2018 *
***************************************************
The evaluation is defined by the Guidelines. A draft version is
available: http://www-nlpir.nist.gov/projects/tv2018/index.html and
details will be worked out starting in February based in part on input
from the participants.
You should read the guidelines carefully before applying to participate in one or more tasks:
Guidelines
P l e a s e n o t e:
1) Dissemination of TRECVID work and results other than in the
(publicly available) conference proceedings is welcomed, but the
conditions of participation specifically preclude any advertising claims based on TRECVID results.
2) All system output and results submitted to NIST are published in
the Proceedings or on the public portions of TRECVID web site archive.
3) The workshop is open only to participating groups that submit
results for at least one task and to selected government personnel
from sponsoring agencies and data donors.
4) Each participating group is required to submit before the November
workshop a notebook paper describing their experiments and results.
This is true even for groups who may not be able to attend the
workshop.
5) It is the responsibility of each team contact to make sure that
information distributed via the call for participation and the
tv18.list@nist.gov email list is disseminated to all team members with
a need to know. This includes information about deadlines and
restrictions on use of data.
6) By applying to participate you indicate your acceptance of the
above conditions and obligations.
There is a tentative schedule for the tasks included in the Guidelines
webpage: Schedule
W o r k s h o p f o r m a t
Plans are for a 2 1/2 day workshop at NIST
in Gaithersburg, Maryland - just outside Washington, DC. Confirmation
and details will be provided to participants as soon as available.
The TRECVID workshop is used as a forum both for presentation of
results (including failure analyses and system comparisons), and for
more lengthy system presentations describing retrieval techniques
used, experiments run using the data, and other issues of interest to
researchers in information retrieval. As there is a limited amount of
time for these presentations, the evaluation coordinators and NIST
will determine which groups are asked to speak and which groups will
present in a poster session. Groups that are interested in having a
speaking slot during the workshop will be asked to submit a short
abstract before the workshop describing the experiments they
performed. Speakers will be selected based on these abstracts.
H o w t o r e s p o n d t o t h i s c a l l
Organizations wishing to participate in TRECVID 2018 must respond
to this call for participation by submitting an on-line application by
2. April. Only ONE APPLICATION PER TEAM please, regardless of how
many organizations the team comprises.
*PLEASE* only apply if you are able and fully intend to complete the
work for at least one task. Taking the data but not submitting any
runs threatens the continued operation of the workshop and the
availability of data for the entire community.
Here is the application URL:
http://ir.nist.gov/tv-submit.open/application.html
You will receive an immediate automatic response when your application
is received. NIST will respond with more detail to all applications submitted
before the end of February. At that point you'll be
given the active participant's userid and password, be subscribed to
the tv18.list email discussion list, and can participate in finalizing
the guidelines as well as sign up to get the data, which is controlled
by separate passwords.
T R E C V I D 2 0 1 8 e m a i l d i s c u s s i o n l i s t
The tv18.list email discussion list (tv18.list@nist.gov) will serve as
the main forum for discussion and for dissemination information about
TRECVID 2018. It is each participant's responsibility to monitor the
tv18.list postings. It accepts postings only from the email addresses
used to subscribe to it. At the bottom of the guidelines there is a
link to an archive of past postings available using the active
participant's userid/password.
Q u e s t i o n s
Any administrative questions about conference participation,
application format/content, subscriptions to the tv18.list,
etc. should be sent to george.awad at nist.gov.
Best regards,
TRECVID 2018 organizers team
Last
updated: [an error occurred while processing this directive]
Date created:
Wednesday, 8-Jan-15
For further information contact