CALL FOR PARTICIPATION in the
2006 TREC VIDEO RETRIEVAL EVALUATION (TRECVID)

February 2006 - November 2006

Conducted by the National Institute of Standards and Technology (NIST)
With support from the Disruptive Technology Office (DTO) and NIST

      Introduction

In 2001 and 2002 the Text Retrieval Conference 
series, which encourages and facilitates research in information
retrieval, sponsored a video "track" devoted to research in automatic
segmentation, indexing, and content-based retrieval of digital
video. In 2003, the track became an independent activity with the
workshop taking place just before TREC and began a two-year cycle
testing shot boundary determination, story segmentation, high-level
feature extraction and search against US news (ABC, CNN) video from
1998. In 2005 the data was extended to include Chinese and Arabic news
sources, story segmentation was dropped, a camera-motion task was
added, and a collection of unproduced BBC travel video was explored.

      2006 System tasks and data

TRECVID 2006 will complete a 2-year cycle on English, Arabic, and
Chinese news video. It will focus on evaluating at least 3 main tasks,
each a sub-problem or variant of the content-based video retrieval
task. The tasks will be shot boundary determination, high-level
feature extraction, and search. NIST and the coordinators are also
working with participants to come up with an evaluation using
unproduced video data similar to the BBC rushes explored in 2005.

New this year: 

 1) Although the amount of test data from sources used in 2005 will be
as large or larger than in 2005, there will be significant additional
data from channels and/or programs not included in the data for
2005. This is expected to provide interesting information on the
extent to which detectors generalize.

 2) Participants in the feature task will be required to submit results
for all 39 features in the common annotation data for 2005. A subset
of those will be chosen by NIST and evaluated. This is intended to
encourage generic methods for development of detectors.

 3) Additional effort will be made to ensure that interactive search
runs come from experiments designed to allow comparison of systems
independent of the main effect of the human in the loop. This will be
worked out with the participants before the guidelines are complete.
This may mean fewer interactive search runs can be submitted from a 
site.

 4) Following the VACE III goals, topics seeking video of events
will be much more frequent this year - exploring the limits of
one-keyframe-per-shot approaches for this kind of topic and encouraging
exploration beyond those limits.

 5) Most of the data will be distributed again on hard drives but
the file system will be ReiserFS to help avoid the data corruption
and access problems encountered in 2005.
 
Shot boundary, feature, and search data:

For test data the above tasks will draw from about 160 hours of news
video from about a dozen programs captured in 2005 in English,
Chinese, or Arabic, and transcoded to MPEG-1. We expect to provide the
output of an automatic speech recognition system and a machine
translation (to English) system for the news videos. For continuity's
sake we hope to get the output of the same systems as in 2005, but
arrangements for this are not yet in place.

Groups that participated in 2005 should already have the common
annotation on the 2005 development data as training data and will not
get a new copy. New participants may request a copy of that training
video data and retrieve participant- and NIST-generated annotations
from the Past Data area of the TRECVID website. It is expected that
additional annotation of the 2005 development data will be donated by
the LSCOM workshop and the MediaMill team at the University of
Amsterdam.

Rushes task and data:

TRECVID 2006 will, we expect, also offer a task aimed at exploring how
a system could help a human mine an archive of rushes for reusable
video segments and how one can evaluate such a system. Rushes comprise
the video material, mostly unstaged with only natural sound, little if
any metadata or speech, shot for productions but largely unused in the
final product. We expect the amount of rushes video to be at least as
great as in 2005 (50 hours) We expect more rushes donated by the BBC
will be available for use in this task. Additional data from another
source may be available. An email discussion group has been created to
draft a proposal by 1. March. If you would like to participate in the
drafting, and are not already participating in that email discussion,
please email over@nist.gov and ask to be included after you apply
for participation in TRECVID 2006.

Data distribution:

Distribution of the training and test data, with the exception of the
shot boundary test data, will be handled by the Linguistic Data
Consortium (LDC) using loaner IDE drives with a *ReiserFS* filesystem,
which must be returned or purchased from LDC within 3 weeks of
receipt. The only charge to participants for data will be the cost of
shipping the drive(s) back to LDC. The shot boundary test data will be
express-shipped to participants on DVD+R by NIST.

Much like TREC, TRECVID will provide, in addition to the data, uniform
scoring procedures, and a forum for organizations interested in
comparing their approaches and results.

Participants will be encouraged to share resources and intermediate
system outputs to lower entry barriers and enable analysis of various
components' contributions and interactions.

The details about the predecessor TREC video track and the latest
about TRECVID can be found at the TRECVID web site
(www-nlpir.nist.gov/projects/trecvid). The evaluation is defined by
the guidelines. A draft version is there now and details will be
worked out starting in mid-February based in part on input from the
participants.


     *You are invited to participate in TRECVID 2006*.

Organizations may choose to participate in one or more of the tasks.
TRECVID participants must submit results for at least one task in
order to attend the workshop. *PLEASE* only apply if you are able and
fully intend to complete the work for at least one task. Taking the
data but not submitting any runs threatens the continued operation of
the workshop and the availability of data for the entire community.

*Please note:* Dissemination of TRECVID work and results other than in
the (publicly available) conference proceedings is welcomed, but the
conditions of participation specifically preclude any advertising
claims based on TRECVID results. All retrieval results submitted to
NIST are published in the Proceedings and on the public portions of
TRECVID web site archive. The workshop is open only to participating
groups that submit results for at least one task and to selected
government personnel from sponsoring agencies. By applying to
participate you indicate your acceptance of the above restrictions.

      Tentative schedule

Here is a tentative schedule. It will be revised and made precise as
part of defining the final guidelines.

  1. Feb  NIST sends out Call for Participation in TRECVID 2006 
 20. Feb  Applications for participation in TRECVID 2006 due at NIST
  1. Mar  Rushes task proposal complete
  1. Mar  LDC begins shipping 2005 training data to new participants
  1. Apr  Guidelines complete; LDC begins shipping 2006 hard drives
 14. Jul  Shot boundary test collection DVDs shipped by NIST
 11. Aug  Search topics available from TRECVID website.
 15. Aug  Shot boundary detection submissions due at NIST for evaluation.
 21. Aug  Feature extraction task submissions due at NIST for evaluation.
 25. Aug  Feature extraction donations available for active participants
 25. Aug  Results of shot boundary evaluations returned to participants
 29. Aug - 13. Oct  Search and feature assessment at NIST
 19. Sep  Results of feature extraction evaluations returned to
          participants
 15. Sep  Search task submissions due at NIST for evaluation
 18. Oct  Results of search evaluations returned to participants
 23. Oct  Speaker proposals due at NIST
 29. Oct  Notebook papers due at NIST
 ~7. Nov  Workshop registration closes
  8. Nov  Copyright forms due back at NIST (see Notebook papers for
          instructions)
 13,14 Nov  TRECVID Workshop at NIST in Gaithersburg, MD
 30. Nov  Workshop papers publicly available (slides added as they arrive)
  1. Mar 2007  Final versions of TRECVID 2006 papers due at NIST

      Workshop Format

The workshop itself (November 13-14 at NIST in Gaithersburg, Maryland
near Washington,DC) will be used as a forum both for presentation of
results (including failure analyses and system comparisons), and for
more lengthy system presentations describing retrieval techniques
used, experiments run using the data, and other issues of interest to
researchers in information retrieval. As there is a limited amount of
time for these presentations, the evaluation coordinators and NIST
will determine which groups are asked to speak and which groups will
present in a poster session. Groups that are interested in having a
speaking slot during the workshop will be asked to submit a short
abstract before the workshop describing the experiments they
performed. Speakers will be selected based on these abstracts.

As some organizations may not wish to describe their proprietary
algorithms, TRECVID defines two categories of participation:

 *Category A: Full participation*
 Participants will be expected to present full details of system
 algorithms and various experiments run using the data, either in a talk
 or in a poster session.
 
 *Category C: Evaluation only*
 Participants in this category will be expected to submit results for
 common scoring and tabulation, and present their results in a poster
 session. They will not be expected to describe their systems in detail,
 but will be expected to report on time and effort statistics.

      How to respond to this call

Organizations wishing to participate in TRECVID 2006 should respond to
this call for participation by submitting an application. An
application consists of an email with four parts: contact information,
a one-paragraph description of your technical approach, whether you
will participate as a Category A or a Category C group, and a list of
tasks that you are likely to participate in:

- shot boundary determination
- high-level feature extraction 
- search (please specify: interactive, manually-assisted, fully automatic)
- rushes exploitation

Contact information includes a full regular address, voice and fax
telephone numbers, and an email address of the one person in the
organization who will be the main TRECVID contact. If desired, you may
include a second email address that will be the address to which all
TRECVID-related email will be sent. For example, this second email
address may be the address of a local mailing list at your institution
that distributes TRECVID mail internally to project participants. If
only one email address is supplied, that address will be used for the
TRECVID mailing list. Please note that email is the primary method of
communication in TRECVID. Once you have applied, you will be
subscribed to the trecvid2006 email discussion list, can participate
in finalizing the guidelines, and sign up to get the data. The
trecvid2005 email discussion list will serve as the main forum for
such discussion and for dissemination of other information about
TRECVID 2005. It accepts postings only from the email addresses used
to subscribe to it.

All applications must be submitted by *February 20, 2006* to
Lori.Buckland at nist.gov. Any administrative questions about
conference participation, application format, content, etc. should be
sent to the same address.

Paul Over
Alan Smeaton
Wessel Kraaij

Last updated: Wednesday, 01-Feb-2006 07:52:54 EST
Date created: Monday, 01-Feb-06
For further information contact