Understanding
Conferences
Introduction
Publications
Data
Guidelines
DUC Roadmap 2005 - 2007
ROAD MAP
FOR DUC 05 - DUC 07 AUTOMATIC SUMMARISING EVALUATION PROGRAMME
Working Group, Spring 2004
Karen Sparck Jones, Hans van Halteren, Marie-Francine Moens, Guy Lapalme, Dragomir Radev, Bonnie Dorr, Paul Over, Ed Hovy, Kathy McKeown, Donna Harman
The Working Group reviewed the work done in DUC 01-04, noting both achievements and limitations. The consensus was that the programme so far had been reasonably successful in attracting participation and in developing and testing extractive summarising systems for news material, but that a significant step up to more challenging requirements was required, which would increase the range of source types, address more demanding summarising purposes, and tackle realistic extrinsic, rather than intrinsic, evaluation protocols. All these would encourage a shift from extracting to abstracting approaches to summarising.
At the DUC 04 Workshop on May 6-7, the Group presented its first suggestions for a new phase of DUC, from DUC 05-07. This would focus on purpose-oriented summarising requiring the fusion of information from a range of source types in the form of a resume as might be needed by a person in an organisation having to deal with eg a natural disaster or public health emergency like a major earthquake or an outbreak of SARS. In order to accommodate a range of participant interests and capabilities the Group suggested a structure for the evaluation programmes from 05-07 which would include tests both for individual new source types and combinations, with increasingly demanding specifications for input and output.
It was clear from the discussions of the Working Group's suggestions at the DUC 04 Workshop that the suggestions were attractive, and constructive detailed comments were made which helped to develop the basic idea. It was recognised that the identification of viable scenarios for resumes, ie for `situation topics', and of appropriate and available data sources for these was going to be difficult, and also that the programme would need help with this from the summarising community and likely participants. As a first step, NIST would take the lead in developing examples which would check both the viability of the basic proposal and help to firm it up in detail, which would be presented at the ACL Summarising Workshop at Barcelona in July 2004.
The documents amplifying this Note are as follows:
- Donna Harman's DUC 05 presentation (pdf) for lunchtime discussion at Barcelona ACL
- NIST's proposed Next Steps based on WG proposals from DUC 04 (to be discussed in Barcelona)
- KSJ's slides for presentation of WG proposals at DUC 04, extended to include main points made in discussion.
Background items:
- Table summarising original DUC 2000 Road Map.
- Very brief report on DUC 01 - DUC 04
- Table showing DUC01-04 instantiation of original Road Map
- Notes with main features of DUC evaluations so far, based on Paul Over's successive sets of overview slides
- Combined notes on the activities of some particular groups set up at DUC 03 to work on particular matters, also covering notes on Rouge.
- Summarising Factors (Karen Sparck Jones, 20 May 2004)
- Some of many disaster situation sites:
