DUC 2006: Call for Participation
Document Understanding Conference (DUC)
New York City, NY
June 8-9, 2006
National Institute of Standards and Technology (NIST)
As the amount of online text continues to grow, we are witnessing a tremendous
increase in interest in summarization research from both academia and
The Document Understanding Conference (DUC) is a series of
summarization evaluations that have been conducted by the National
Institute of Standards and Technology (NIST) since 2001. Its goal is
to further progress in automatic text summarization and enable
researchers to participate in large-scale experiments in both the development and evaluation of summarization systems.
DUC has evaluated summarization systems for generic and focused summaries of English newspaper and newswire data. Various target sizes (10 - 400 words) have been used and both single-document summaries and summaries of multiple documents have been evaluated. Summaries have been manually judged for their readability, and both manual and automatic evaluation of content coverage have been explored.
In 2004, the road mapping committee strongly recommended that new tasks be undertaken that are strongly tied to a clear user application.
Therefore, DUC 2005 had a single system task requiring informative summaries to be generated in response to a complex question (e.g., "What are the advantages and disadvantages of same-sex schools").
DUC has continued to grow since its inception, with 31 sites world-wide
participating in the DUC 2005 question-focused summarization task. The
system task for DUC 2006 will essentially be the same as the 2005 task
and will model real-world complex question answering, in which an
information need cannot be satisfied by simply stating a name, date,
quantity, etc. Given a topic (question) and a set of 25 relevant
documents, the task is to synthesize a fluent, well-organized 250-word
summary of the documents that answers the question(s) in the topic
statement. Successful performance on the task benefits from a combination of IR and NLP
capabilities, including passage retrieval, compression, and fusion of information. NIST expects to make the test data available on
March 6, 2006, and participants are expected to submit their
results by March 22, 2006. Summaries will be manually judged for
both fluency and responsiveness to the topic statement, and the results will be presented and discussed at the DUC 2006 Workshop to be held in conjunction with the HLT-NAACL 2006 Conference. Information about DUC 2006 will be updated and made available in the DUC 2006 guidelines.
You are invited to participate in the DUC 2006 system task.
Organizations interested in participating should submit an
application as soon as possible, but no later than January 18,
2006. Submitting an application does not commit you to
participating in the DUC 2006 task. However, once you apply you will
be subscribed to the duc2006 email list, which will be the
means of discussing and communicating about DUC 2006. You are
encouraged to bring up questions, concerns, and suggestions using this
applications may be accepted if resources allow, but in no case will
sample or test data be released to groups who have not applied.
Please email your application to Lori.Buckland@nist.gov. The
application should include the following information:
information (organization name, full mailing address, voice and fax
phone numbers, email of a main DUC contact)
- Names and email
addresses of group members to be included in the duc2006 mailing
- A short paragraph on the organization's summarization
- An indication of whether this group has participated
in DUC or TREC before
All summarization results submitted to DUC will be published in the
Proceedings and archived on the DUC web site. Dissemination of DUC
work and results other than in the conference proceedings is welcomed,
but the conditions of participation preclude specific advertising
claims based on DUC results.