Summing up …
Overall peer quality:
- “Grammaticality”(especially “All” choice) was too sensitive to low-level formatting.
- “Coherence” and “organization”, as defined, made little sense for very short summaries.
- “Coherence” was generally hard to distinguish from “organization”.
For the future:
- Address dependence of grammaticality on low-level formatting
- Reassess operational definitions of coherence/organization to better capture what researchers want to measure