Call for Participation in TRECVID 2007 ------------------------------------------------------------------------ CALL FOR PARTICIPATION in the 2007 TREC VIDEO RETRIEVAL EVALUATION (TRECVID) February 2007 - November 2007 Conducted by the National Institute of Standards and Technology (NIST) With support from the Disruptive Technology Office (DTO) and NIST I n t r o d u c t i o n: In 2006 the TREC Video Retrieval Workshop (TRECVID) series (http://trecvid.nist.gov), which encourages and facilitates research in information retrieval from digital video, completed the second two-year cycle devoted to automatic segmentation, indexing, and content-based retrieval of digital video - broadcast news in English, Arabic, and Chinese. It also completed two years of pilot studies on exploitation of unedited video (rushes). Some 70 research groups have been provided with the TRECVID 2005-2006 broadcast news video and many resources created by NIST and the TRECVID community are available for continued research on this data independent of TRECVID. See the "Past data" section of the TRECVID website for pointers. In 2007 TRECVID will explore related but significantly new and difficult territory: - After 4 years on broadcast news, TRECVID 2007 will test its three fundamental tasks (shot boundary determination, high-level feature extraction, and search) on video from a real archive of related but different genres (news magazine, educational, and cultural programming) to see how well the technologies apply to new sorts of data. - TRECVID 2007 will require a run treating the new video as if no metadata, ASR or MT for the languages of the videos (mostly Dutch) existed - as might occur in the case of video in other less well known languages. This will further emphasize the need to understand the visually-encoded information. For an optional second condition, we will, if possible, provide metadata and the output of ASR (Dutch->English) and MT. The University of Twente has offered to provide ASR. We do not yet have a source for machine translation of the Dutch to English. - An effort will be made to emphasize search for events (object+action) not easily captured in a single frame as opposed to searching for static objects. - While mastershots will be defined as units of evaluation, keyframes or annotation of keyframes will not be provided by NIST. This will require groups to look afresh at how best to train their systems - tradeoffs between processing speed, effectiveness, amount of the video processed. As in the past, participants may want to team up to create training resources. The degree to which systems trained on broadcast news generalize with varying amounts of training data to a related but different genre will be a focus of TRECVID 2007. - A new pilot task with common evaluation will be added: video summarization against unedited video (rushes). Systems will attempt to construct a minimally short video clip that includes the major objects and events of the video to be summarized. As with all the other tasks, system speed will be a fundamental measure, i.e., the time taken to generate the summary. If this task and its evaluation prove feasible and interesting to the community, we expect to focus more attention on it in the future. D a t a: The Netherlands Institute for Sound and Vision (http://portal.beeldengeluid.nl/)has generously provided 400 hours of news magazine, science news, news reports, documentaries, educational programming, and archival video in MPEG-1 for use within TRECVID. We may have an additional 200 hours of non-commercial news and news magazine video in time to include. This is plenty for 2 or 3 years of work. We will use ~100 hours of this data in 2007: ~ 6 hours for the shot boundary task ~ 50 hours for development of search feature detection ~ 50 hours for test of search and feature detection Additional related data: Metadata in Dutch for each video including the program title and a free text description (50-100 words) of the content Master shot reference for search/feature development/test (VOLUNTEER NEEDED TO CREATE MASTER SHOT REFERENCE) Existing keyframes for the TRECVID 2003 and 2005 broadcast news we hope will be available on DVD for purchase from LDC Exiting keyframe annotations for the TRECVID 2003 and 2005 keyframes on TRECVID/LSCOM/MediaMill websites Output of automatic speech recognizer (for Dutch) will be provided by the University of Twente. (VOLUNTEER NEEDED FOR MACHINE TRANSLATION OF DUTCH SPEECH AND METADATA TO ENGLISH) The BBC Archive has provided about 100 hours of unedited material in MPEG-1 from about five dramatic series. By the time the guidelines are complete (1 April) we will choose appropriate (small) amounts for development and test. ? hours for development ? hours (<= 50 clips?) for test Additional related data ? sample groundtruth - major object/event lists created by Dublin City University (DCU)/NIST for some development clips. Just examples, not intended as training data D i s t r i b u t i o n: Shot boundary data: express-shipped on DVD+R from NIST Search / feature / master shot reference data: by download from NIST (or mirror servers?) (VOLUNTEERS NEEDED IN US, EUROPE, ASIA,...TO HOST MIRROR SERVERS) Rushes data: by download from NIST (or mirror servers?) (VOLUNTEERS NEEDED IN US, EUROPE, ASIA,...TO HOST MIRROR SERVERS) Other data will be distributed by email from NIST T a s k s: 1) Shot boundary determination on Sound and Vision data Task: - detect all cuts and gradual transitions and identify as such Ground truth: - created at NIST Evaluation: - run at NIST Measures: - same as in 2006 2) High-level feature extraction on Sound and Vision data Task: - detect 39 LSCOM lite features (or some subset to be specified, appropriate to new video and to limit training data) Evaluation: - manual judgments at NIST of 10 to 20 features Measures: - same as in 2006; perhaps some revisions to run sampling 3) Search on Sound and Vision data Task: - find shots meeting need expressed by 24 multimedia topics created at NIST - emphasis on events Evaluation: - manual judgments at NIST Measures: - same as in 2006 4) Pilot summarization of BBC rushes to be proposed as workshop at ACM MM '07 in Augsburg, Germany 23-29 Sept. with results reported at TRECVID 2007 Task: - automatically create an MPEG-1 summary clip of no longer than ? seconds (or ?% of the original video's duration) that shows the main objects and events in the video to be summarized using the minimal number of frames and presenting the information in a way to maximize usability and speed of object/event recognition. Ground truth: - lists of major objects/events created at DCU/NIST for each clip to be summarized Evaluation: - by NIST using ground truth produced by DCU/NIST. A human will view the summary using only the controls: stop, play, fastforward, rewind, pause and will check off the objects/events in the ground truth list that appear in the video summary. This evaluation process will be timed. Measures: - fraction of ground truth objects/events found in summary - time needed to check summary against ground truth - size of summary (# of frames) - elapsed system time to create summary ? usability/learnability of summary (organization*) ? user satisfaction with summary (organization*) * although playback at evaluation will be limited as stated under above Task, summaries can contain picture-in-picture, split screens, and results of other techiniques for organizing the summary that raise quesitions of usability etc. Not sure how to integrate usability/satisfaction questions into the evaluation process. If the summariztion workshop is accepted by ACM MM '07 (we will know by 1. March), the schedule would have to be approximately as follows: 15 Feb testing of annotation feasibility, feedback on whole idea, program committee in place, workshop proposal due 1 Mar summarization guidelines complete 1 Mar sample annotations of some development data complete 1 Mar development data available for download 1 Apr test data available for download 4 May results submitted to NIST 11 May annotations of test data complete 11 May results sent by NIST to participants for (2 wks) judging 25 May judging results returned to NIST 1 Jun evaluation results distributed to participants 22 Jun papers due to ACM 11 Jul acceptance notification If the ACM MM workshop proposal is not accepted, we will adjust the schedule and include the summarization task like the others at the TRECVID workshop in November in Gaithersburg. Much like TREC, TRECVID will provide, in addition to the data, uniform scoring procedures, and a forum for organizations interested in comparing their approaches and results. Participants will be encouraged to share resources and intermediate system outputs to lower entry barriers and enable analysis of various components' contributions and interactions. The details about the predecessor TREC video track (2001/2002)and the latest about TRECVID (2003-2007) can be found at the TRECVID web site: trecvid.nist.gov. The evaluation is defined by the guidelines. A draft version is there now and details will be worked out starting in mid-February based in part on input from the participants. *You are invited to participate in TRECVID 2007*. Organizations may choose to participate in one or more of the tasks. TRECVID participants must submit results for at least one task in order to attend the TRECVID workshop in Gaithersburg in November. Participation in the ACM MM 07 TRECVID Summarization Workshop (if accepted) will be based on normal ACM MM workshop attendance rules. *PLEASE* only apply if you are able and fully intend to complete the work for at least one task. Taking the data but not submitting any runs threatens the continued operation of the workshop and the availability of data for the entire community. P l e a s e n o t e: 1) Dissemination of TRECVID work and results other than in the (publicly available) conference proceedings is welcomed, but the conditions of participation specifically preclude any advertising claims based on TRECVID results. 2) All retrieval results submitted to NIST are published in the Proceedings and on the public portions of TRECVID web site archive. 3) The workshop is open only to participating groups that submit results for at least one task and to selected government personnel from sponsoring agencies. 4) By applying to participate you indicate your acceptance of the above restrictions. T e n t a t i v e s c h e d u l e Here is a tentative schedule for the tasks - excluding the summarization pilot. This tentative schedule will be revised and made precise as part of defining the final guidelines. 1. Feb NIST sends out Call for Participation in TRECVID 2007 20. Feb Applications for participation in TRECVID 2007 due at NIST 1. Apr Guidelines complete May Download of feature/search development data Jun Download of feature/search development data 7. Jul Shot boundary test collection DVDs shipped by NIST Jul Download of feature/search test data 3. Aug Search topics available from TRECVID website. 7. Aug Shot boundary detection submissions due at NIST for evaluation. 10. Aug Feature extraction task submissions due at NIST for evaluation. 17. Aug Feature extraction donations available for active participants 17. Aug Results of shot boundary evaluations returned to participants 20. Aug - 5. Oct Search and feature assessment at NIST 14. Sep Results of feature extraction evaluations returned to participants 10. Sep Search task submissions due at NIST for evaluation 12. Oct Results of search evaluations returned to participants 15. Oct Speaker proposals due at NIST 22. Oct Notebook papers due at NIST ~7. Nov Workshop registration closes 8. Nov Copyright forms due back at NIST (see Notebook papers for instructions) 5,6 Nov TRECVID Workshop at NIST in Gaithersburg, MD 15. Dec Workshop papers publicly available (slides added as they arrive) 1. Mar 2008 Final versions of TRECVID 2007 papers due at NIST W o r k s h o p f o r m a t The workshop itself (a week earlier than usual!), November 5-6 at NIST in Gaithersburg, Maryland near Washington,DC, will be used as a forum both for presentation of results (including failure analyses and system comparisons), and for more lengthy system presentations describing retrieval techniques used, experiments run using the data, and other issues of interest to researchers in information retrieval. As there is a limited amount of time for these presentations, the evaluation coordinators and NIST will determine which groups are asked to speak and which groups will present in a poster session. Groups that are interested in having a speaking slot during the workshop will be asked to submit a short abstract before the workshop describing the experiments they performed. Speakers will be selected based on these abstracts. As some organizations may not wish to describe their proprietary algorithms, TRECVID defines two categories of participation: *Category A: Full participation* Participants will be expected to present full details of system algorithms and various experiments run using the data, either in a talk or in a poster session. *Category C: Evaluation only* Participants in this category will be expected to submit results for common scoring and tabulation. They will not be expected to describe their systems in detail, but will be expected to provide a general description and report on time and effort statistics in a notebook paper. H o w t o r e s p o n d t o t h i s c a l l Organizations wishing to participate in TRECVID 2007 should respond to this call for participation by submitting an application. An application consists of an email with four parts: contact information, a one-paragraph description of your technical approach, whether you will participate as a Category A or a Category C group, and a list of tasks that you are likely to participate in: - shot boundary determination - high-level feature extraction - search (specify: interactive, manually-assisted, fully automatic) - rushes summarization at ACM MM (23-29 Sep., Augsburg, Germany) - rushes summarization at TRECVID (iff workshop at ACM MM does not happen) **Please indicate one or BOTH rushes workshops depending on whether you could participate in September and/or November, even though only one will take place. Contact information includes a full regular address, voice and fax telephone numbers, and an email address of the one person in the organization who will be the main TRECVID contact. If desired, you may include a second email address that will be the address to which all TRECVID-related email will be sent. For example, this second email address may be the address of a local mailing list at your institution that distributes TRECVID mail internally to project participants. If only one email address is supplied, that address will be used for the TRECVID mailing list. Please note that email is the primary method of communication in TRECVID. Once you have applied, you will be subscribed to the trecvid2007 email discussion list, can participate in finalizing the guidelines, and sign up to get the data. The trecvid2007 email discussion list will serve as the main forum for such discussion and for dissemination of other information about TRECVID 2007. It accepts postings only from the email addresses used to subscribe to it. All applications must be submitted by *February 20, 2007* to Lori.Buckland at nist.gov. Any administrative questions about conference participation, application format, content, etc. should be sent to the same address. If you would like to contribute to TRECVID in one or more of the following ways, please contact Paul Over (info at bottom of page) directly as soon as possible: - create master shot reference for feature/search video - provide MT (Dutch speech or ASR, Dutch metadata) to English for feature/search video - agree to host video data (< 100 GB) for download by other participants on a fast, password-protected site. (Asian and European sites especially needed) Paul Over Alan Smeaton Wessel Kraaij