face recognition challenge: train/test sets

asked 2014-11-24 07:06:52 -0600

berak
32993 ●7 ●81 ●312

i noticed, that currently the benchmark works like this:

train once on pairsDevTrain.txt (1100 matching, 1100 non-matching pairs)
test against all 10 splits in pairs.txt (300 matching, 300 non-matching per split, 6000 all)

shouldn't pairsDevTrain.txt be tested against pairsDevTest.txt, like mentioned here (3.a), while each split from pairs.txt should be trained against the 9 other splits ?

(admittedly, training will take ages this way, but i'm having doubts, that it does not get enough traindata for the pairs.txt splits this way)

@Dmitriy Anisimov, as the maintainer of the datasets repo, maybe can you shed some light on this ?

edit retag flag offensive close merge delete

add a comment

answered 2015-02-20 03:21:01 -0600

Dmitriy Anisimov
101 ●2 ●3

For some reasons I didn't see your question earlier.

I double checked description here.

So pairsDevTrain.txt and pairsDevTest.txt should be used for parameters selection and as we don't choose parameters they shouldn't be used. And you are right: for reporting result each split from pairs.txt should be tested against the 9 other splits (train).

p.s. whole my idea with benchmarks was to implement some solution for each Challenge's building block using only OpenCV. I had time on implementation 3 benchmarks and I hoped that OpenCV community may implement some others :)

edit flag offensive delete link

Comments

" I didn't see your question earlier." - true, i couldn't find it either. it seemed to have vanished somehow during the spam-related qa-shutdown.
"each split from pairs.txt should be tested against the 9 other splits (train)" - an easy solution to this might be to add another option to the cmdline parser, so that users can switch between "dev" (the existing solution, fast) and "split" (more to the spec, also higher difficulty, but obviously higher memory/time requirements) mode. i'd try a pr, if that halfway meets your ideas.
some mentioning, that the lfw-benchmark is about face verification (same/not-same), while opencv's existing face-reco is all about identification (closest item from db)

ps. it's a nice and useful benchmark, even apart from the vision challenge !

berak ( 2015-02-20 03:45:56 -0600 )edit

"i'd try a pr, if that halfway meets your ideas." sure, the addition of such command option was the first thing I thought too.

Dmitriy Anisimov ( 2015-02-20 05:50:39 -0600 )edit

well, here 's my try.

berak ( 2015-02-20 08:39:00 -0600 )edit

I took a look on it, but I can't merge OpenCV PR's and looks like currently there are some higher priority tasks, as there are already several unmerged PRs related to Challenge's code.

Dmitriy Anisimov ( 2015-02-27 04:13:58 -0600 )edit

yea, ok. thanks for looking at it anyway.

low priority, i guess. did not expect this to happen overnight.

what should i do ? leave it as is ? close it for now (and try later? maybe even after the challenge is over ?)

berak ( 2015-02-27 04:20:40 -0600 )edit

yes, let leave it as is. One my PR from the end of December still waiting in open state too.

Dmitriy Anisimov ( 2015-02-27 06:41:04 -0600 )edit

add a comment

face recognition challenge: train/test sets

1 answer

Comments

Links

Question Tools

Stats

Related questions

face recognition challenge: train/test sets edit

1 answer

Comments

Links

Question Tools

Stats

Related questions

face recognition challenge: train/test sets