[tv13.list] TV13 SIN localisation questoins

[Paul Over]

For participants in the semantic indexing task who plan to submit
localization results ....

Hi Koen,

I've tried to address your questions below.

- Paul

On 06/28/2013 10:05 AM, Koen van de Sande wrote:

> - are we allowed to submit multiple bounding boxes per i-frame for the
> same object?

Yes, but for the ground truth we plan to ask only for one per frame
and will just use that one to judge the submissions.  At least this
year.

> - what is the evaluation measure that will be used?

 From the Guidelines:

Measures (localization): Temporal and spatial localization will be 
evaluated using precision and recall based on the judged items at two 
levels - the frame and the pixel, respectively. NIST will then calculate 
an average for each of these values for each concept and for each run.

>      o in the case that multiple instances of the target object are
> present, how does the evaluation work? Do we get it 'right' if we put a
> box around one of the objects?

Yes, where "one of the objects" is the one the assessor marks.

>      o if we are allowed to submit multiple bounding boxes, what is the
> penalty for a mistake? E.g. we predict two boxes, but only one object is
> present

No penalty this year

>    - in the submission format (copied below), we need to specify a File#
> and a Frame #. Given that the filenames in the
> tv13.localization.iframes.tgz file are "28124/1/100.jpeg", what do a
> file# and frame# look like, as I have three numbers for two slots.

In your example, 28124 is the file number. 1 is the shot number (not
relevant to the submission). 100 is the frame number.

> - is it possible to save the iframes from tv13.localization.iframes.tgz
> using a lossless format like PNG? The current JPEG images have
> additional compression artifacts (not present in the video) due to the
> high compression rate used.

Perhaps next year but not now, a few days from the submission date :-)

> Thanks for your time.
>
> Best,
>
> Koen van de Sande
> University of Amsterdam
>
> Concept#
> |   File# Frame#
> |   |     |    UpperLeftX
> |   |     |    |   UpperLeftY
> |   |     |    |   |   LowerRightX
> |   |     |    |   |   |   LowerRightY
> |   |     |    |   |   |   |
> xxx xxxxx xxxx xxx xxx xxx xxx
>

-- 
Paul Over
National Institute of Standards and Technology
Gaithersburg, MD  20899-8940
USA
----------------------------------------------
Retrieval Group
Information Access Division
Information Technology Laboratory
Bldg. 225  Rm. A211  (Mailstop 8940)
Voice: 301 975-6784    Fax: 301 975-5287