key not found in checkpoint in distributed mode of tensorflow about benchmarks HOT 5 CLOSED

I had the same problem, how did you solve it?

from benchmarks.

@hangzh2012 I changed some code to adapt it ,here are some questions:
1、 different way of 'variable_update' will save model parameters from different variables scope (local、global 、trainable) ,but in function of 'eval' ,it only read from global scope
2、different way of 'variable_update' will save different parameter variables in checkpoint ,for example, the mode of ‘replicated’ will save num_gpus copies in checkpoints such as 'v0/xxx' 'v1/xxx' while the mode of 'distributed_replicated' only save variables starting with 'v/'

besides ,you can read my commit here for reference : pan463194277@3e98be2

from benchmarks.

I believe we fixed this issue. @hangzh2012 can you try with the latest version of tf_cnn_benchmarks? Note --eval only works in non-distributed mode.

from benchmarks.

@reedwm Yes, it worked fine when using (--variable_update replicated).

from benchmarks.

@pan463194277 Thank you for your reply. It worked well now. I used the eval method in a wrong way(--variable_update parameter_server).

from benchmarks.