new error at pruning step #82

vince62s · 2016-11-09T19:22:12Z

Dan,
I did not update with your last commit but my thought is that it is not related.
Here is a log of another error, popping at pruning time (step1, step0 being ok)

float-counts-prune 0.25 200000 data/lm/lm_4_prune4000000.pocolm/work/step0/float.all data/lm/lm_4_prune4000000.pocolm/work/step0/protected.all data/lm/lm_4_prune4000000.pocolm/work/step1/float.1 data/lm/lm_4_prune4000000.pocolm/work/step1/float.2 data/lm/lm_4_prune4000000.pocolm/work/step1/float.3 data/lm/lm_4_prune4000000.pocolm/work/step1/float.4 2>>data/lm/lm_4_prune4000000.pocolm/work/step1/log/float_counts_prune.log
float-counts-prune: float-counts-prune.cc:364: float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float): Assertion `count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount' failed.

danpovey · 2016-11-09T19:24:56Z

I think it could be related. Try the latest commit.

On Wed, Nov 9, 2016 at 2:22 PM, vince62s [email protected] wrote:

Dan,
I did not update with your last commit but my thought is that it is not
related.
Here is a log of another error, popping at pruning time (step1, step0
being ok)
float-counts-prune 0.25 200000 data/lm/lm_4_prune4000000.pocolm/work/step0/float.all
data/lm/lm_4_prune4000000.pocolm/work/step0/protected.all
data/lm/lm_4_prune4000000.pocolm/work/step1/float.1
data/lm/lm_4_prune4000000.pocolm/work/step1/float.2
data/lm/lm_4_prune4000000.pocolm/work/step1/float.3
data/lm/lm_4_prune4000000.pocolm/work/step1/float.4
2>>data/lm/lm_4_prune4000000.pocolm/work/step1/log/float_counts_prune.log

float-counts-prune: float-counts-prune.cc:364: float
pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float,
float): Assertion `count > 0.0 && discount > 0.0 && backoff_total >
backoff_count && backoff_total >= 0.99 * discount' failed.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#82, or mute the thread
https://github.com/notifications/unsubscribe-auth/ADJVuxPeUpCkAOOhbyADkhYeBngXg_nrks5q8h1kgaJpZM4Kt5Bg
.

vince62s · 2016-11-09T20:45:44Z

same thing.
I also have this in the console, that I did not mention before:
warning: in float-counts,1.29348e-11 != 1.29409e-11
warning: in float-counts,9.8279e-12 != 9.83022e-12
warning: in float-counts,8.16296e-12 != 8.16658e-12
warning: in float-counts,9.1713e-13 != 9.22712e-13
warning: in float-counts,7.59186e-13 != 7.63044e-13
warning: in float-counts,8.89627e-13 != 8.89721e-13

danpovey · 2016-11-09T20:56:59Z

Can you run the binary that failed in gdb?

On Wed, Nov 9, 2016 at 3:45 PM, vince62s [email protected] wrote:

same thing.
I also have this in the console, that I did not mention before:
warning: in float-counts,1.29348e-11 != 1.29409e-11
warning: in float-counts,9.8279e-12 != 9.83022e-12
warning: in float-counts,8.16296e-12 != 8.16658e-12
warning: in float-counts,9.1713e-13 != 9.22712e-13
warning: in float-counts,7.59186e-13 != 7.63044e-13
warning: in float-counts,8.89627e-13 != 8.89721e-13

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#82 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu3jg_rZdOukwNl6QXJ3aO4_UQkAeks5q8jD4gaJpZM4Kt5Bg
.

vince62s · 2016-11-09T21:11:45Z

I need some guidance, never done that before.
float-counts-prune is called in the pruning python script.

danpovey · 2016-11-09T21:26:38Z

Do

gdb --args float-counts-prune 0.25 200000
data/lm/lm_4_prune4000000.pocolm/work/step0/float.all
data/lm/lm_4_prune4000000.pocolm/work/step0/protected.all
data/lm/lm_4_prune4000000.pocolm/work/step1/float.1
data/lm/lm_4_prune4000000.pocolm/work/step1/float.2
data/lm/lm_4_prune4000000.pocolm/work/step1/float.3
data/lm/lm_4_prune4000000.pocolm/work/step1/float.4

(gdb) r
.. wait till it crashes
(gdb) up
keep doing 'up' till you get to the right stack frame, or 'down' if you go
too far
then
(gdb)  p count
(gdb) p discount
.. and any other relevant variables.

On Wed, Nov 9, 2016 at 4:11 PM, vince62s [email protected] wrote:

I need some guidance, never done that before.
float-counts-prune is called in the pruning python script.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#82 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu-hcR9whpcKLbiXWc-QWDbvaL9nSks5q8jcSgaJpZM4Kt5Bg
.

vince62s · 2016-11-09T21:38:20Z

(gdb) up
#4 0x00000000004038d0 in pocolm::FloatCountsPruner::PruningLogprobChange (
this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10,
backoff_count=0.500001132, backoff_total=0.500001132)
at float-counts-prune.cc:362
362 assert(count > 0.0 && discount > 0.0 &&
(gdb) p count
$1 = 0.500001132
(gdb) p discount
$2 = 2.55000576e-10
(gdb)

danpovey · 2016-11-09T21:47:21Z

Can you type 'bt' ? I want to see the whole backtrace.

On Wed, Nov 9, 2016 at 4:38 PM, vince62s [email protected] wrote:

(gdb) up
#4 #4 0x00000000004038d0 in
pocolm::FloatCountsPruner::PruningLogprobChange (
this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10,
backoff_count=0.500001132, backoff_total=0.500001132)
at float-counts-prune.cc:362
362 assert(count > 0.0 && discount > 0.0 &&
(gdb) p count
$1 = 0.500001132
(gdb) p discount
$2 = 2.55000576e-10
(gdb)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#82 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu3BjNFNg37PknvxZwH0TPcij16iUks5q8j1NgaJpZM4Kt5Bg
.

vince62s · 2016-11-09T21:50:38Z

(gdb) bt
#0 0x00007ffff7221c37 in GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff7225028 in __GI_abort () at abort.c:89
#2 0x00007ffff721abf6 in __assert_fail_base (
fmt=0x7ffff736b3b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x418220 "count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount",
file=file@entry=0x417d20 "float-counts-prune.cc", line=line@entry=364,
function=function@entry=0x418a80 <pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)::__PRETTY_FUNCTION> "float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)") at assert.c:92
#3 0x00007ffff721aca2 in GI___assert_fail (
assertion=0x418220 "count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount",
file=0x417d20 "float-counts-prune.cc", line=364,
function=0x418a80 <pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)::__PRETTY_FUNCTION> "float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float)") at assert.c:101
#4 0x00000000004038d0 in pocolm::FloatCountsPruner::PruningLogprobChange (
this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10,
backoff_count=0.500001132, backoff_total=0.500001132)
at float-counts-prune.cc:362
#5 0x0000000000404094 in pocolm::FloatCountsPruner::DoPruningForLmState (
---Type to continue, or q to quit---
this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:537
#6 0x0000000000403be6 in pocolm::FloatCountsPruner::FlushOutput (
this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:476
#7 0x0000000000403532 in pocolm::FloatCountsPruner::ProcessInput (
this=0x7fffffffd890) at float-counts-prune.cc:279
#8 0x0000000000402939 in pocolm::FloatCountsPruner::FloatCountsPruner (
this=0x7fffffffd890, argc=9, argv=0x7fffffffde58)
at float-counts-prune.cc:178
#9 0x0000000000401761 in main (argc=9, argv=0x7fffffffde58)
at float-counts-prune.cc:701
(gdb)

danpovey · 2016-11-09T21:56:34Z

I'm a little confused how the discount amount would be so tiny although the
count for the state was reasonable (0.5).
Can you show the metaparameters (final.metaparams)?
The only way the metaparameter optimization would have reached this result
is if your dev set was part of your training set.

On Wed, Nov 9, 2016 at 4:50 PM, vince62s [email protected] wrote:

(gdb) bt
#0 0x00007ffff7221c37 in

_GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 #1 0x00007ffff7225028 in
__GI_abort () at abort.c:89 #2
#2 0x00007ffff721abf6 in
__assert_fail_base ( fmt=0x7ffff736b3b8 "%s%s%s:%u: %s%sAssertion `%s'
failed.\n%n", assertion=assertion@entry=0x418220 "count > 0.0 && discount >
0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount",
file=file@entry=0x417d20 "float-counts-prune.cc", line=line@entry=364,
function=function@entry=0x418a80
<pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float,
float)::_PRETTY_FUNCTION> "float pocolm::FloatCountsPruner::PruningLogprobChange(float,
float, float, float)") at assert.c:92
#3 #3 0x00007ffff721aca2 in

_GI___assert_fail ( assertion=0x418220 "count > 0.0 && discount > 0.0 &&
backoff_total > backoff_count && backoff_total >= 0.99 * discount",
file=0x417d20 "float-counts-prune.cc", line=364, function=0x418a80
<pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float,
float)::_PRETTY_FUNCTION> "float pocolm::FloatCountsPruner::PruningLogprobChange(float,
float, float, float)") at assert.c:101
#4 #4 0x00000000004038d0 in
pocolm::FloatCountsPruner::PruningLogprobChange (
this=0x7fffffffd890, count=0.500001132, discount=2.55000576e-10,
backoff_count=0.500001132, backoff_total=0.500001132)
at float-counts-prune.cc:362
#5 #5 0x0000000000404094 in
pocolm::FloatCountsPruner::DoPruningForLmState (
---Type to continue, or q to quit---
this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:537
#6 #6 0x0000000000403be6 in
pocolm::FloatCountsPruner::FlushOutput (
this=0x7fffffffd890, history_length=3) at float-counts-prune.cc:476
#7 #7 0x0000000000403532 in
pocolm::FloatCountsPruner::ProcessInput (
this=0x7fffffffd890) at float-counts-prune.cc:279
#8 #8 0x0000000000402939 in
pocolm::FloatCountsPruner::FloatCountsPruner (
this=0x7fffffffd890, argc=9, argv=0x7fffffffde58)
at float-counts-prune.cc:178
#9 #9 0x0000000000401761 in main
(argc=9, argv=0x7fffffffde58)
at float-counts-prune.cc:701
(gdb)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#82 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVuymVWayXX7Qf5Ab7kQMz3FGjP33nks5q8kAvgaJpZM4Kt5Bg
.

vince62s · 2016-11-09T21:58:44Z

yes, the dev is in the training set

count_scale_1 0.500001125302356
order2_D1 0.724208510369568
order2_D2 0.297727220253901
order2_D3 0.127992215348989
order2_D4 0.055617652988870
order3_D1 0.055193203110626
order3_D2 0.013962291974136
order3_D3 0.005097823583461
order3_D4 0.001982142055876
order4_D1 0.000000000100000
order4_D2 0.000000000010000
order4_D3 0.000000000001000
order4_D4 0.000000000000100

vince62s · 2016-11-09T21:59:19Z

would that behave differently if I use the fold into train option ?

danpovey · 2016-11-09T22:12:21Z

You will get totally wrong discounting parameters if the dev is in the
training set. Yes, you should exclude it and do fold-into-dev.
Jian, could you modify optimize_metaparameters.py so that if the
highest-order D1 after optimization finishes is less than, say, 0.1, it
exits with error status after printing out a message saying "your dev set
is probably in your training set; this is not advisable"?

On Wed, Nov 9, 2016 at 4:59 PM, vince62s [email protected] wrote:

would that behave differently if I use the fold into train option ?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#82 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu5e5KIdLNm8QYcOfNioR1s1JaZs0ks5q8kI3gaJpZM4Kt5Bg
.

wantee · 2016-11-10T02:04:17Z

sure.

On Thu, Nov 10, 2016 at 6:12 AM, Daniel Povey [email protected]
wrote:

You will get totally wrong discounting parameters if the dev is in the
training set. Yes, you should exclude it and do fold-into-dev.
Jian, could you modify optimize_metaparameters.py so that if the
highest-order D1 after optimization finishes is less than, say, 0.1, it
exits with error status after printing out a message saying "your dev set
is probably in your training set; this is not advisable"?

On Wed, Nov 9, 2016 at 4:59 PM, vince62s [email protected] wrote:

would that behave differently if I use the fold into train option ?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#82 (comment),
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/
ADJVu5e5KIdLNm8QYcOfNioR1s1JaZs0ks5q8kI3gaJpZM4Kt5Bg>

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#82 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGphTPx6vdU83j8eMMHqv2swT7I1nUeUks5q8kVGgaJpZM4Kt5Bg
.

francisr · 2016-11-10T11:07:24Z

A related question: what should I do if I don't have a dev set?

wantee · 2016-11-10T11:30:08Z

A dev set is required for optimizing the metaparameters. You could extract part of your training set as dev set.

vince62s closed this as completed Nov 11, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new error at pruning step #82

new error at pruning step #82

vince62s commented Nov 9, 2016 •

edited

Loading

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

wantee commented Nov 10, 2016

francisr commented Nov 10, 2016

wantee commented Nov 10, 2016

new error at pruning step #82

new error at pruning step #82

Comments

vince62s commented Nov 9, 2016 • edited Loading

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

vince62s commented Nov 9, 2016

vince62s commented Nov 9, 2016

danpovey commented Nov 9, 2016

wantee commented Nov 10, 2016

francisr commented Nov 10, 2016

wantee commented Nov 10, 2016

vince62s commented Nov 9, 2016 •

edited

Loading