From over at nist.gov Tue Jun 4 13:13:25 2013 From: over at nist.gov (Paul Over) Date: Tue, 4 Jun 2013 13:13:25 -0400 Subject: [tv13.list] EastEnders transcript replacement Message-ID: <51AE2035.7010506@nist.gov> Hi, A problem (repeated data, malformed xml) in the BBC EastEnders transcripts has been addressed and a new copy of those files is now on the DCU server. If you already have a copy, please download a new one. (You'll notice one file is incomplete - 5331593301391104626.xml. We'll just have to live with that fact.) Thanks to Herv? Bredin and Robin Aly and the BBC for finding and addressing the problem quickly. - Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Thu Jun 6 07:36:59 2013 From: over at nist.gov (Paul Over) Date: Thu, 6 Jun 2013 07:36:59 -0400 Subject: [tv13.list] TRECVID submissions for semantic indexing Message-ID: <51B0745B.8030509@nist.gov> Hi, This message is for participants in the semantic indexing task ... The SIN submission website is now open for testing. This year we have added automatic checking of SIN submissions and strongly encourage you to submit runs early to see that they can get past the checker. Official runs will not be accepted with errors. On the submission page in its current form, choose the "TEST" organization and submit as often as you wish. You'll get feedback from the checker but the submission will be discarded. On 24. June, the list of participating teams will be added and the website opened for final, official submissions. Please review the instructions on SIN submissions in the guidelines and look at the DTD (view source) and the sample submission before submitting any runs. The submission page requires the use of the active participant's userid/password that your team contact was given after applying to participate in TRECVID. http://www-nlpir.nist.gov/projects/tv2013/tv2013.html#sin Thanks, Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Mon Jun 10 07:24:39 2013 From: over at nist.gov (Paul Over) Date: Mon, 10 Jun 2013 07:24:39 -0400 Subject: [tv13.list] Question of Semantic Indexing task - progress In-Reply-To: <51b2988b.c786440a.6007.48fc@mx.google.com> References: <51b2988b.c786440a.6007.48fc@mx.google.com> Message-ID: <51B5B777.3050705@nist.gov> Hi, A progress submission is just the main (60 concept on IACC.2.A) task BUT run *separately* on just the IACC.2.B and then again *separately* on just the IACC.2.C. You will be running this year's system against "future" data so in 2013 and 2014 you could compare those future systems to this year's system on the same data, etc. Hope that clarifies things. - Paul On 06/07/2013 10:35 PM, changzhiluo wrote: > Hi, ... We encountered a problem when we read > the Submission types of Semantic Indexing task, that is what exactly we > should summit in the "progress" task. Does it mean we should summit the > results of 346 concepts using the testing data IACC.2.A, IACC.2.B and > IACC.2.C, or just summit the results of the 60 main task concepts using > IACC.2.A, IACC.2.B and IACC.2.C? -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Tue Jun 11 07:44:27 2013 From: over at nist.gov (Paul Over) Date: Tue, 11 Jun 2013 07:44:27 -0400 Subject: [tv13.list] ASR for IACC.2 videos Message-ID: <51B70D9B.3000600@nist.gov> Hi, Just a reminder, we still expect to have ASR for the IACC.2 videos and will make it available for download as soon as we get it from the generous folks at LIMSI. - Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Mon Jun 17 07:25:23 2013 From: over at nist.gov (Paul Over) Date: Mon, 17 Jun 2013 07:25:23 -0400 Subject: [tv13.list] ASR for IACC.2 available for download Message-ID: <51BEF223.5090308@nist.gov> Hi, For participants using the IACC.2 collections (A,B,C) the ASR, provided again by LIMSI, is now in the active participant's area for download: http://www-nlpir.nist.gov/projects/tv2013/active/ Thanks to LIMSI for providing this and to Georges Quenot for arranging it again. - Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Tue Jun 18 13:53:43 2013 From: over at nist.gov (Paul Over) Date: Tue, 18 Jun 2013 13:53:43 -0400 Subject: [tv13.list] Instance search topics Message-ID: <51C09EA7.90104@nist.gov> Hi, This message is for participants in the instance search task ... The instance search test topics will be available from the active participant's part of the website, as scheduled, on Monday. We've applied a new convention to the short piece of text associated with each topic. INS has always had an inconsistent definition of instance. In most test topics, (1) we are looking for instances/views of a singular specific object/person. But a second sort of topic has focused on (2) instances of any member of a set of objects, all created to be mutually indistinguishable, e.g., logos. In order to try to make clearer which interpretation to apply, we've revised the topic text using "this/these" for case (1) and "a/an" for case (2). Objects/locations/people known by their proper names are referred to with just some form of their name. For example: this fountain these stairs Angela Merkel a Ford Mustang logo Of course it is the example images that are decisive. We hope this makes each topic's intent clearer for software and any human searcher. - Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Wed Jun 19 13:57:23 2013 From: over at nist.gov (Paul Over) Date: Wed, 19 Jun 2013 13:57:23 -0400 Subject: [tv13.list] SIN submission problem Message-ID: <51C1F103.1080605@nist.gov> Hi, Sorry, but it seems we cannot correctly process submission files compressed with tar or zip - issues related to being able to predict the name of the contained file from the name of the compressed container. Please use gzip or bzip2 if possible. Email me directly (over at nist.gov) if this a real hardship. Thanks, Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Fri Jun 21 09:22:00 2013 From: over at nist.gov (Paul Over) Date: Fri, 21 Jun 2013 09:22:00 -0400 Subject: [tv13.list] SIN submissions; INS test topics Message-ID: <51C45378.10507@nist.gov> Hi, Two announcements: 1) Semantic indexing ... The SIN submission webpage now contains the list of team IDs and is open for final submissions. Please continue to use the TEST team ID until you get your submissions past the checker. The submission site now handles zip and tar compression. Note: please restrict your file names (for container and contained) to alphanumeric characters to avoid strange side effects in the downstream Perl code. 2) Instance search ... The test topics (30 and 24 resp.) for automatic and interactive runs along with the example images (masked and not) are now available for download from the active participant's area. This is test data so please make sure knowledge of it does not in any way contaminate your experimental results. - Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Fri Jun 28 06:53:48 2013 From: over at nist.gov (Paul Over) Date: Fri, 28 Jun 2013 06:53:48 -0400 Subject: [tv13.list] Concept Pair runs In-Reply-To: <51CD3568.5090002@uva.nl> References: <51CD3568.5090002@uva.nl> Message-ID: <51CD6B3C.6070703@nist.gov> On 06/28/2013 03:04 AM, Cees Snoek wrote:> hi Paul, for the concept pair task, we are allowed to submit two runs. > In addition a requirement wrt a baseline is mentioned, does this imply > one regular run and one baseline, or should the baseline be considered > as another additional runs, making the total for the pair task: three runs? > -Cees. > Hi Cees, Sorry this is not clear in the guidelines. We want the baseline run from each team but let's not count it against the maximum of 2. So you can submit 2 regular paired concept runs and 1 baseline. - Paul -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287 From over at nist.gov Fri Jun 28 10:24:56 2013 From: over at nist.gov (Paul Over) Date: Fri, 28 Jun 2013 10:24:56 -0400 Subject: [tv13.list] TV13 SIN localisation questoins In-Reply-To: References: Message-ID: <51CD9CB8.9010009@nist.gov> For participants in the semantic indexing task who plan to submit localization results .... Hi Koen, I've tried to address your questions below. - Paul On 06/28/2013 10:05 AM, Koen van de Sande wrote: > - are we allowed to submit multiple bounding boxes per i-frame for the > same object? Yes, but for the ground truth we plan to ask only for one per frame and will just use that one to judge the submissions. At least this year. > - what is the evaluation measure that will be used? From the Guidelines: Measures (localization): Temporal and spatial localization will be evaluated using precision and recall based on the judged items at two levels - the frame and the pixel, respectively. NIST will then calculate an average for each of these values for each concept and for each run. > o in the case that multiple instances of the target object are > present, how does the evaluation work? Do we get it 'right' if we put a > box around one of the objects? Yes, where "one of the objects" is the one the assessor marks. > o if we are allowed to submit multiple bounding boxes, what is the > penalty for a mistake? E.g. we predict two boxes, but only one object is > present No penalty this year > - in the submission format (copied below), we need to specify a File# > and a Frame #. Given that the filenames in the > tv13.localization.iframes.tgz file are "28124/1/100.jpeg", what do a > file# and frame# look like, as I have three numbers for two slots. In your example, 28124 is the file number. 1 is the shot number (not relevant to the submission). 100 is the frame number. > - is it possible to save the iframes from tv13.localization.iframes.tgz > using a lossless format like PNG? The current JPEG images have > additional compression artifacts (not present in the video) due to the > high compression rate used. Perhaps next year but not now, a few days from the submission date :-) > Thanks for your time. > > Best, > > Koen van de Sande > University of Amsterdam > > Concept# > | File# Frame# > | | | UpperLeftX > | | | | UpperLeftY > | | | | | LowerRightX > | | | | | | LowerRightY > | | | | | | | > xxx xxxxx xxxx xxx xxx xxx xxx > -- Paul Over National Institute of Standards and Technology Gaithersburg, MD 20899-8940 USA ---------------------------------------------- Retrieval Group Information Access Division Information Technology Laboratory Bldg. 225 Rm. A211 (Mailstop 8940) Voice: 301 975-6784 Fax: 301 975-5287