Some feedback from NIST video summary assessors with respect to the process, instructions, assessment software, etc. On the whole it was easy to see what was going on; however, with the single frames, I couldn't tell whether there was zooming or not. The speeded-up bits were difficult, as were the four split screens and the ones with a lot of small screens at the bottom or sides. ======================================================================== I thought the second question about how many "nearly identical" segments was sometimes difficult to judge on the 5-point scale. ======================================================================== Obtaining the answers was always a challenge. Each summary went fairly quickly... I know that I set my own criteria for judging whether a scene in the summaries should be included in my answers. As an example: the answer said "a woman talking and her dress blowing in the wind." One scene/clip showed a woman 'possibly' talking with her dress blowing in the wind, and a second scene, a close up of her definitely talking, but her dress only apparently being moved by the wind. I chose the first scene but the second scene could have been OK also. I'm trying to say that many answers were subjective, and what role does subjectivity play? If a scene lasts a fraction of a fraction of a second, and you blink your eyes at that time --- you may have missed an answer. What is the purpose of the scene being this short? Also, too many of these type scenes could strain the eyes of the assessor. I had eyestrain when there were several to many frames to look at-at the same time. I noticed that when examining scenes to be selected (or not) as an answer, there were scenes/clips that were not included in the overall summary. These scenes/clips did not affect my answers, but why weren't they included in the overall summary? ======================================================================== Overall the [assessment] software was very user friendly. I liked the ability to click anywhere on the blue to pause the summary videos but found that when I clicked near the narrative sections I sometimes missed and inadvertently clicked on one of the narrative items because I was looking intently at the screen. I caught myself a couple of times and reversed the mistake. It might be useful, if possible, to limit clicking to the right of the narratives. When I invoked the button for a summary I was not able to pause the video in time for the first couple of scenes in the fast-moving summary videos. It would be helpful to have the SUMMARY button placed closer to the blue area in order to pause the video more quickly since there's no turning back in the summary. ======================================================================== I found that listening to the ear phones while watching the videos and summaries was not useful and even a distraction at times. The fast-forward of the video soundtrack reminded me of Donald Duck and was unintelligible. The summary narrative was not much better because the the summary narratives were snippets and there was little continuity. ======================================================================= 1. Quality of film was poor in some cases (dark or no contrast). 2. Film within a film hard to follow. 3. Speed of film, too fast in some cases to catch inclusions to be noted. By the time you saw it and paused, the scene was no longer visible. 4. Films with several small frames running at the same time--difficult to follow action in all of them. 5. Items on bottom for judging were arbitrary. I understood it to mean the the summary was easy to understand, not that it was representative of the whole film that we had viewed. Others in our group thought the criterion was whether it followed and told the story of the film. Which was correct? 6. Some of the films were good with many inclusions; others were poor and had only one or two inclusions. It was hard to imagine that the same system was used for all of them since the variation in quality and quantity of inclusions was so great.