Comments (14)
Hi lahiruts,
We are aware of this issue. A fix will be on the way soon.
Thanks,
Yongqiang
from cntk.
I have the same problem....
Please, let us know when it is fixed! ;)
Thanks,
Mirco
from cntk.
Is there a chance you could run this in Debug mode? The call stack is missing a critical entry, between WriteOutput() and NotifyFunctionValuesMBSizeModified(). I guess that function got inlined in the Release build.
from cntk.
Actually, could you just try the latest? We did fix something related to this a few days ago.
Would you mind letting me know if it works now?
from cntk.
I tried with the latest downloadable sources but the error (see below) still persists.
Any idea about it?
Thank you!
Mirco
Allocating matrices for forward and/or backward propagation. evaluate: reading 250 frames of SimMalespk06-usphdevmalespk06snt3852 [CALL STACK] /home/mirco/CNTK-master/build/release/lib/libcntkmath.so ( Microsoft::MSR::CNTK::DebugUtil::PrintCallStack() + 0xbf ) [0x7f156f3061ff] cntk ( void Microsoft::MSR::CNTK::ThrowFormatted<std::logic_error>(char const*, ...) + 0xdd ) [0x53d41d] cntk ( Microsoft::MSR::CNTK::ComputationNode<float>::NotifyFunctionValuesMBSizeModified() + 0x496 ) [0x53e436] cntk ( ) [0x758857] cntk ( Microsoft::MSR::CNTK::SimpleOutputWriter<float>::WriteOutput(Microsoft::MSR::CNTK::IDataReader<float>&, unsigned long, Microsoft::MSR::CNTK::IDataWriter<float>&, std::vector<std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >, std::allocator<std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > > > const&, unsigned long, bool) + 0x363 ) [0x75b9d3] cntk ( void DoWriteOutput<float>(Microsoft::MSR::CNTK::ConfigParameters const&) + 0x669 ) [0x7606f9] cntk ( void DoCommands<float>(Microsoft::MSR::CNTK::ConfigParameters const&) + 0xc07 ) [0x5942e7] cntk ( wmainOldCNTKConfig(int, wchar_t**) + 0x909 ) [0x535539] cntk ( wmain1(int, wchar_t**) + 0x68 ) [0x535b88] cntk ( main + 0xd8 ) [0x529548] /lib/x86_64-linux-gnu/libc.so.6 ( __libc_start_main + 0xf5 ) [0x7f156df7dec5] cntk ( ) [0x52d4e7] Closed Kaldi writer LOG (latgen-faster-mapped:main():latgen-faster-mapped.cc:163) Time taken 1.2866s: real-time factor assuming 100 frames/sec is inf LOG (latgen-faster-mapped:main():latgen-faster-mapped.cc:166) Done 0 utterances, failed for 0 LOG (latgen-faster-mapped:main():latgen-faster-mapped.cc:168) Overall log-likelihood per frame is -nan over 0 frames.
from cntk.
Hmm... I think the first would be to see the actual error. Somehow it gets masked.
Are you building your own binary? If so, could you run it in Debug, or if that does not work, enable the fprintf() call in Basics.h ThrowFormatted()?
template
__declspec_noreturn static inline void ThrowFormatted(const char* format, ...)
{
va_list args;
char buffer[1024];
va_start(args, format);
vsprintf(buffer, format, args);
#ifdef _DEBUG // print this to log before throwing, so we can see what the error is
fprintf(stderr, "\nAbout to throw exception '%s'\n", buffer);
#endif
Microsoft::MSR::CNTK::DebugUtil::PrintCallStack();
throw E(buffer);
};
from cntk.
If I enable those fprintf in the Basics.h, before the error reported in the previous post, I have:
`Allocating matrices for forward and/or backward propagation.
evaluate: reading 425 frames of SimFemalespk01-usphdevfemalespk01snt2717
About to throw exception 'NotifyFunctionValuesMBSizeModified: labels InputValue operation had its col dimension 425 changed by the reader to 1, but different from MBLayout.'
It seems that for some reasons the reader does something weird.
The reading part of the CNTK2_write.cntk is the following:
reader=[
# reader to use
readerType=Kaldi2Reader
readMethod=blockRandomize
frameMode=false
miniBatchMode=Partial
randomize=Auto
verbosity=0
features=[
dim=$featDim$
scpFile=$inputCounts$
rx=$inputFeats$
]
from cntk.
We have discovered a tricky problem which may be related to this. I will let you know when it lands.
from cntk.
Thank you! I'll stay tuned!
Mirco
On Fri, Feb 5, 2016 at 6:41 PM, Frank Seide [email protected]
wrote:
We have discovered a tricky problem which may be related to this. I will
let you know when it lands.—
Reply to this email directly or view it on GitHub
#65 (comment).
from cntk.
Yongqiang tracked it down--in decoding, the labels should not be referenced, but they were, because the logLLs formally depend on them, indirectly through the priors. The fix is to skip PreComputeNodes that have already been computed while analyzing which inputs an output depends on.
This is in master now, but I do not have a positive test case for this. Would you mind trying it and letting me know whether this fixes it?
from cntk.
Sure, tomorrow I will try it ;)
On Fri, Feb 5, 2016 at 11:11 PM, Frank Seide [email protected]
wrote:
Yongqiang tracked it down--in decoding, the labels should not be
referenced, but they were, because the logLLs formally depend on them,
indirectly through the priors. The fix is to skip PreComputeNodes that have
already been computed while analyzing which inputs an output depends on.This is in master now, but I do not have a positive test case for this.
Would you mind trying it and letting me know whether this fixes it?—
Reply to this email directly or view it on GitHub
#65 (comment).
from cntk.
Hi Frank,
It seems fixed. My decoding is running as expected now.
Thanks,
Lahiru
from cntk.
I confirm that now it works.
Thank you!
Mirco
from cntk.
thank your for your verification and feedback
from cntk.
Related Issues (20)
- Value goes invalid when using TestMinibatch
- Request for a no-opencv dotnet release
- CNTK C# Crash when layer is deeper
- Convolution 1D CNTK C++ HOT 1
- Add support to release linux aarch64 wheels
- Error while deploying MS Teams Bot with SSO
- how to install in python3.8 HOT 1
- A model causes CNTK crash with cudnnSetPoolingNDDescriptor when invoking the pooling operator
- This repo is missing important files HOT 1
- API document issue due to syntax in source code HOT 1
- Error: could not find all specified 'to_nodes' in clone. Looking for ['relu5'], found [None]
- SequenceClassification.py does not work out of box HOT 1
- Microsoft has absolutely no potential
- development of a neural network for object search
- join vcpkg
- cntk crash and pycharm process finished with exit code -1066598274 (0xC06D007E)
- ConvolutionTranspose2D outputs normally when num_filters is tuple and dilation is 0
- program crash when get gradient of `ConvolutionTranspose2D`
- when shape contains negative integer, input_variable should throw an error HOT 1
- MAX_POOLING crash, when pooling_window_shape contains 0 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cntk.