DUC 2005: Call for Participation

Over the past several years, we have witnessed a tremendous increase in interest in summarization research from both academia and industry. A DARPA program, Translingual Information Detection, Extraction, and Summarization (TIDES), specifically called for major advances in summarization technology, both in English and from other languages to English (cross-language summarization). In response, the National Institute of Standards and Technology (NIST) initiated the Document Understanding Conference (DUC) series to evaluate automatic text summarization. Its goal is to further progress in summarization and enable researchers to participate in large-scale experiments.

In DUC 2001 - 2004 a growing number of research groups (23 in 2004) have participated in the evaluation of generic and focused summaries of English newspaper and newswire data. Various target sizes (10 - 400 words) have been used and both single-document summaries and summaries of multiple documents (around 10 documents per set) have been evaluated. Summaries have been manually judged for their readability, and both manual and automatic evaluation of content coverage have been explored. In 2004 the output of Arabic-to-English MT systems was summarized and automatic scoring (ROUGE) played a major role.

DUC 2005 marks a major change in focus from previous years. The road mapping committee strongly recommended that new tasks be undertaken that are strongly tied to a clear user application. There has also been serious discussion in the program committee about working on new evaluation methodologies and metrics that would better address issues of variation in human-authored summaries. Therefore, the main thrust of DUC 2005 will be to have a single simpler (but still user-oriented) task that will allow the whole community to put some of their time/effort into helping with this new evaluation framework during 2005.

The system task in 2005 will be to synthesize from a set of 25-50 documents a brief, well-organized, fluent answer to a need for information that cannot be met by just stating a name, date, quantity, etc. This task will model real-world complex question answering and was suggested by "An Empirical Study of Information Synthesis Tasks" (Enrique Amigo, Julio Gonzalo, Victor Peinado, Anselmo Penas, Felisa Verdejo; {enrique,julio,victor,anselmo,felisa}@lsi.uned.es).

You are invited to participate in DUC 2005 and in the discussions concerning its definition, scheduling, etc. NIST expects to make the test data available on May 31, 2005, and participants are expected to submit their results by June 17, 2005. Preliminary information about DUC 2005 is available in the DUC 2005 guidelines.

Organizations interested in participating in DUC 2005 should submit an application as soon as possible to begin participating in the online discussions defining DUC 2005. Submitting an application does not commit you to participating in the DUC 2005 task. However, once you apply you will be subscribed to the DUC2005 email discussion list, which will be the means of discussing and communicating about DUC 2005. You are encouraged to bring up questions, concerns, and suggestions using this forum. Please email your application to Lori.Buckland@nist.gov. The application should include the following information:

1. Contact information (organization name, full mailing address, voice and fax phone numbers, email of a main DUC contact)
2. A short paragraph on the organization's summarization approach
3. An indication of whether this group has participated in DUC before

Dissemination of DUC work and results other than in the (publicly available) conference proceedings is welcomed, but the conditions of participation preclude specific advertising claims based on DUC results. All summarization results submitted to DUC will be published in the Proceedings and archived on the DUC web site.

Any questions concerning DUC should be sent to hoa.dang AT nist.gov. Late applications may be accepted if resources allow, but in no case will sample or test data be released to groups who have not applied.