Kaldi decoding error with new release of CNTK about cntk HOT 14 CLOSED

Hi lahiruts,

We are aware of this issue. A fix will be on the way soon.

Thanks,
Yongqiang

from cntk.

I have the same problem....
Please, let us know when it is fixed! ;)

Thanks,
Mirco

from cntk.

Is there a chance you could run this in Debug mode? The call stack is missing a critical entry, between WriteOutput() and NotifyFunctionValuesMBSizeModified(). I guess that function got inlined in the Release build.

from cntk.

Actually, could you just try the latest? We did fix something related to this a few days ago.

Would you mind letting me know if it works now?

from cntk.

I tried with the latest downloadable sources but the error (see below) still persists.
Any idea about it?

Thank you!

Mirco

Allocating matrices for forward and/or backward propagation. evaluate: reading 250 frames of SimMalespk06-usphdevmalespk06snt3852 [CALL STACK] /home/mirco/CNTK-master/build/release/lib/libcntkmath.so ( Microsoft::MSR::CNTK::DebugUtil::PrintCallStack() + 0xbf ) [0x7f156f3061ff] cntk ( void Microsoft::MSR::CNTK::ThrowFormatted<std::logic_error>(char const*, ...) + 0xdd ) [0x53d41d] cntk ( Microsoft::MSR::CNTK::ComputationNode<float>::NotifyFunctionValuesMBSizeModified() + 0x496 ) [0x53e436] cntk ( ) [0x758857] cntk ( Microsoft::MSR::CNTK::SimpleOutputWriter<float>::WriteOutput(Microsoft::MSR::CNTK::IDataReader<float>&, unsigned long, Microsoft::MSR::CNTK::IDataWriter<float>&, std::vector<std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, std::allocator<std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > > > const&, unsigned long, bool) + 0x363 ) [0x75b9d3] cntk ( void DoWriteOutput<float>(Microsoft::MSR::CNTK::ConfigParameters const&) + 0x669 ) [0x7606f9] cntk ( void DoCommands<float>(Microsoft::MSR::CNTK::ConfigParameters const&) + 0xc07 ) [0x5942e7] cntk ( wmainOldCNTKConfig(int, wchar_t**) + 0x909 ) [0x535539] cntk ( wmain1(int, wchar_t**) + 0x68 ) [0x535b88] cntk ( main + 0xd8 ) [0x529548] /lib/x86_64-linux-gnu/libc.so.6 ( __libc_start_main + 0xf5 ) [0x7f156df7dec5] cntk ( ) [0x52d4e7] Closed Kaldi writer LOG (latgen-faster-mapped:main():latgen-faster-mapped.cc:163) Time taken 1.2866s: real-time factor assuming 100 frames/sec is inf LOG (latgen-faster-mapped:main():latgen-faster-mapped.cc:166) Done 0 utterances, failed for 0 LOG (latgen-faster-mapped:main():latgen-faster-mapped.cc:168) Overall log-likelihood per frame is -nan over 0 frames.

from cntk.

Hmm... I think the first would be to see the actual error. Somehow it gets masked.

Are you building your own binary? If so, could you run it in Debug, or if that does not work, enable the fprintf() call in Basics.h ThrowFormatted()?

template
__declspec_noreturn static inline void ThrowFormatted(const char* format, ...)
{
va_list args;
char buffer[1024];

va_start(args, format);
vsprintf(buffer, format, args);

#ifdef _DEBUG // print this to log before throwing, so we can see what the error is
fprintf(stderr, "\nAbout to throw exception '%s'\n", buffer);
#endif
Microsoft::MSR::CNTK::DebugUtil::PrintCallStack();
throw E(buffer);
};

from cntk.

If I enable those fprintf in the Basics.h, before the error reported in the previous post, I have:

`Allocating matrices for forward and/or backward propagation.
evaluate: reading 425 frames of SimFemalespk01-usphdevfemalespk01snt2717
About to throw exception 'NotifyFunctionValuesMBSizeModified: labels InputValue operation had its col dimension 425 changed by the reader to 1, but different from MBLayout.'

It seems that for some reasons the reader does something weird.

The reading part of the CNTK2_write.cntk is the following:

reader=[
# reader to use
readerType=Kaldi2Reader
readMethod=blockRandomize
frameMode=false
miniBatchMode=Partial
randomize=Auto
verbosity=0
features=[
dim=$featDim$
scpFile=$inputCounts$
rx=$inputFeats$
]

from cntk.

We have discovered a tricky problem which may be related to this. I will let you know when it lands.

from cntk.

Thank you! I'll stay tuned!

Mirco

On Fri, Feb 5, 2016 at 6:41 PM, Frank Seide [email protected]
wrote:

We have discovered a tricky problem which may be related to this. I will
let you know when it lands.

—
Reply to this email directly or view it on GitHub
#65 (comment).

from cntk.

Yongqiang tracked it down--in decoding, the labels should not be referenced, but they were, because the logLLs formally depend on them, indirectly through the priors. The fix is to skip PreComputeNodes that have already been computed while analyzing which inputs an output depends on.

This is in master now, but I do not have a positive test case for this. Would you mind trying it and letting me know whether this fixes it?

from cntk.

Sure, tomorrow I will try it ;)

On Fri, Feb 5, 2016 at 11:11 PM, Frank Seide [email protected]
wrote:

Yongqiang tracked it down--in decoding, the labels should not be
referenced, but they were, because the logLLs formally depend on them,
indirectly through the priors. The fix is to skip PreComputeNodes that have
already been computed while analyzing which inputs an output depends on.

This is in master now, but I do not have a positive test case for this.
Would you mind trying it and letting me know whether this fixes it?

—
Reply to this email directly or view it on GitHub
#65 (comment).

from cntk.

Hi Frank,
It seems fixed. My decoding is running as expected now.
Thanks,
Lahiru

from cntk.

I confirm that now it works.
Thank you!
Mirco

from cntk.

thank your for your verification and feedback

from cntk.