- Evaluate the first 5 terms for each method.
- For each term, score it as ‘correct’ if either
- It is in the 400-word summary, or
- It is clearly relevant to the main subject
- For UTD, we evaluate the first 5 topic labels, taken from the first 2 topics
Topic Evaluations (for UTD only):
- Judge whether the topic is correct, based on the labels.
- If any label for a topic is wrong, the topic is judged wrong
Each method required approximately one hour to evaluate all 30 folders