Check failed: a <= b (0 vs. -1.19209e-07)

[fnzhan]

I am run it to train, but error happened in this step: solver = caffe.SGDSolver(caffe_root + 'models/VGGNet/ICDAR2017/SSD_300x300/solver.prototxt')

the trace stack is :
F0708 18:54:26.167244 20822 # **math_functions.cpp:250] Check failed: a <= b (0 vs. -1.19209e-07)
*** Check failure stack trace: *****
@ 0x7f84859db5ad google::LogMessage::Fail()
@ 0x7f84859dd413 google::LogMessage::SendToLog()
@ 0x7f84859db13b google::LogMessage::Flush()
@ 0x7f84859dddfe google::LogMessageFatal::~LogMessageFatal()
@ 0x7f8485fdc0e2 caffe::caffe_rng_uniform<>()
@ 0x7f848600ac36 caffe::SampleBBox()
@ 0x7f848600af80 caffe::GenerateSamples()
@ 0x7f848600b1b4 caffe::GenerateBatchSamples()
@ 0x7f8485f8ec28 caffe::AnnotatedDataLayer<>::load_batch()
@ 0x7f8485e90f2a caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()

I have search a lot, still unresolved. Have you meet this problem? @chenxinpeng

[nszceta]

@fnzhan were you able to identify the problem?

[didulle]

Have you fixed that issue? I have exactly the same issue

[Gesalat]

@didulle
I had the same problem (with original caffe ssd from wei liu). What I tried:
Comment line 250 in src/caffe/util/math_functions.cpp:
// CHECK_LE(a, b);
and then recompile pycaffe with
make py

The idea comes from http://blog.csdn.net/gxb0505/article/details/73702451 (I don' speek Chinese but Google --> Translate this page does the job for me)

But I don't get the training started (Now the problem is "Data layer prefetch queue empty") and I am not sure if this is caused by the rude method I used to get rid of the previous error. Maybe you could try commenting&compiling and report if it works out.

[Gesalat]

Another idea is #669 of original ssd caffe.

[deblauwetom]

FYI, for me, this was the solution:
weiliu89/caffe#669 (comment)