Actual implementation of original RoadMap 2000 See Notes below for interpretation of KSJ's comments KSJ's comments Yr 1 Yr 2 Yr 3 Yr 4 Corpus development **** **** **** **** LIMITED Evaluation software development **** **** **** **** YES Extractive summary, intrinsic evaluation - single document - simple genre **** **** **** YES - sophisticated genre **** **** NO - multiple document - single genre, language **** **** **** YES - evolving (**) **** LIMITED (novelty) - multiple genres, languages **** NO Abstract summaries, <<<<< see below, 1 intrinsic evaluation - single document - simple genre **** **** **** YES - sophisticated genre **** **** NO - multiple document - single genre, language **** **** **** YES - evolving (**) **** LIMITED (novelty) - multiple genres, languages **** NO Abstract, multidoc summaries, extrinsic evaluation - TDT-specific evaluation **** **** NO question answering - single question (**) **** YES -multiple questions, single lang (**) NO -multiple questions, languages Notes ----- 1 The RoadMap distinguished extracts from non-extractive abstracts. DUC so far has only done abstracts in a very limited sense, basically at the level of selected sentence manipulation: this is perhaps best called quasi-extractive summarising. 2. In addition to the RoadMap specification, DUC has included some very limited (simulated) extrinsic evaluation, in Yr 3 and 4, by usefulness (single document) and responsiveness (multidocument). 3. YES only means that something has been done under this heading: it is not an indicator of the extent or value of that something.