Code Monkey home page Code Monkey logo

Comments (5)

dutran avatar dutran commented on July 22, 2024

It looks strange to me too. It seems that your solver failed when it was trying to save a snapshot. Can you check if you have any more space on disk?

from c3d.

whjxnyzh123 avatar whjxnyzh123 commented on July 22, 2024

oh, I just resume traing from iteration 9000,

[nlpr4@bnode01 c3d_finetuning]$ GLOG_logtostderr=1 ../../build/tools/train_net.bin \
> c3d_ucf101_finetuning_solver.prototxt c3d_ucf101_finetune_whole_iter_9000.solverstate

this time it worked......

I0606 10:37:48.255017 64324 solver.cpp:237] Iteration 10000, lr = 1e-05
I0606 10:37:48.255482 64324 solver.cpp:87] Iteration 10000, loss = 0.0224443
I0606 10:37:48.869892 64324 solver.cpp:159] Snapshotting to c3d_ucf101_finetune_whole_iter_                                                          10000
I0606 10:37:50.094324 64324 solver.cpp:166] Snapshotting solver state to c3d_ucf101_finetun                                                          e_whole_iter_10000.solverstate
I0606 10:39:26.923598 64324 solver.cpp:237] Iteration 10020, lr = 1e-05
I0606 10:39:26.924058 64324 solver.cpp:87] Iteration 10020, loss = 0.0513535
I0606 10:41:03.297607 64324 solver.cpp:237] Iteration 10040, lr = 1e-05
I0606 10:41:03.298079 64324 solver.cpp:87] Iteration 10040, loss = 0.0078975

I really don't know what happened last time. any way. it is working now. thank you

from c3d.

whjxnyzh123 avatar whjxnyzh123 commented on July 22, 2024

i will close this issue

from c3d.

dutran avatar dutran commented on July 22, 2024

Glad to hear!

from c3d.

avatar commented on July 22, 2024

Hi @whjxnyzh , thanks for posting out your training process. I also fine-tuning the UCF101 model these days, but for iteration 10000, I still got 0.05 to 0.2 loss, much bigger than your loss number. So I was wondering if there is any trick during training? Like you changed your lr to 0.00001, or something else? Thanks. BTW, if you could share your iteration 20000 model, I would greatly appreciated.

from c3d.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.