DUC Roadmap 2005 - 2007
FOR DUC 05 - DUC 07 AUTOMATIC SUMMARISING EVALUATION PROGRAMME
Working Group, Spring 2004
Karen Sparck Jones,
Hans van Halteren, Marie-Francine Moens, Guy Lapalme, Dragomir Radev,
Bonnie Dorr, Paul Over, Ed Hovy, Kathy McKeown, Donna Harman
The Working Group reviewed the work done in DUC 01-04, noting both
achievements and limitations. The consensus was that the programme
so far had been reasonably successful in attracting participation
and in developing and testing extractive summarising systems for
news material, but that a significant step up to more challenging
requirements was required, which would increase the range of source
types, address more demanding summarising purposes, and tackle
realistic extrinsic, rather than intrinsic, evaluation protocols.
All these would encourage a shift from extracting to abstracting
approaches to summarising.
At the DUC 04 Workshop on May 6-7, the Group presented its first
suggestions for a new phase of DUC, from DUC 05-07. This would focus
on purpose-oriented summarising requiring the fusion of information
from a range of source types in the form of a resume as might be
needed by a person in an organisation having to deal with eg a natural
disaster or public health emergency like a major earthquake or an
outbreak of SARS. In order to accommodate a range of participant
interests and capabilities the Group suggested a structure for
the evaluation programmes from 05-07 which would include tests both
for individual new source types and combinations, with increasingly
demanding specifications for input and output.
It was clear from the discussions of the Working Group's suggestions
at the DUC 04 Workshop that the suggestions were attractive, and
constructive detailed comments were made which helped to develop
the basic idea. It was recognised that the identification of viable
scenarios for resumes, ie for `situation topics', and of appropriate
and available data sources for these was going to be difficult, and
also that the programme would need help with this from the summarising
community and likely participants. As a first step, NIST would take
the lead in developing examples which would check both the viability
of the basic proposal and help to firm it up in detail, which would
be presented at the ACL Summarising Workshop at Barcelona in July 2004.
The documents amplifying this Note are as follows:
- Table summarising original DUC 2000 Road Map.
- Very brief report on DUC 01 - DUC 04
- Table showing DUC01-04 instantiation of original Road Map
- Notes with main features of DUC evaluations so far, based on
Paul Over's successive sets of overview slides
- Combined notes on the activities of some particular groups
set up at DUC 03 to work on particular matters, also
covering notes on Rouge.
- Summarising Factors (Karen Sparck Jones, 20 May 2004)
- Some of many disaster situation sites: