Code Monkey home page Code Monkey logo

Comments (8)

tqjack avatar tqjack commented on July 23, 2024 5

这个我问过transformers那边了，说目前deepspeed不支持4bit/8bit训练，所以目前只能ddp，zero optimization应该都是不行的

from firefly.

tqjack avatar tqjack commented on July 23, 2024

还有就是，readme里面qlora的训练命令使用的是train.py,这个是不是写错了，应该用train_qlora.py？

from firefly.

yangjianxin1 avatar yangjianxin1 commented on July 23, 2024

感谢提醒，确实应该是train_qlora.py，已经修复了。
目前还未尝试deepspeed+qlora，我们会进行尝试，看是否可行。

from firefly.

2018211801 avatar 2018211801 commented on July 23, 2024

蹲~deepspeed多卡+qlora

from firefly.

DumoeDss avatar DumoeDss commented on July 23, 2024

我看requirements里包含deepspeed，是不是可以不安装？Windows下安装失败。

from firefly.

yangjianxin1 avatar yangjianxin1 commented on July 23, 2024

目前qlora暂时不能结合deepspeed训练，需要用torchrun启动脚本

from firefly.

zhangjunyi111 avatar zhangjunyi111 commented on July 23, 2024

这个我问过transformers那边了，说目前deepspeed不支持4bit/8bit训练，所以目前只能ddp，zero optimization应该都是不行的

在哪问的。链接发一下可以吗

from firefly.

zhangjunyi111 avatar zhangjunyi111 commented on July 23, 2024

目前qlora暂时不能结合deepspeed训练，需要用torchrun启动脚本

torchrun启动脚本指的是什么？

from firefly.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.