Code Monkey home page Code Monkey logo

Comments (4)

yangapku avatar yangapku commented on May 13, 2024

您好，这里有一个相关的issue #36 有所解释，相关具体流程可以参考readme数据预处理部分和finetune dataset的代码实现。目前我们处理上，在预处理时会把多对多的情况，拆成多个一对一的图文对用于训练。由于训练中对图文对会进行shuffle，shuffle的随机性减缓了相同图像或文本的样例出现在一个batch。同时我们在多卡global batch上计算对比学习损失，batch如果开大些，能学习的负例也比较多，相对这个因素造成的因素就会小一些了。目前Chinese-CLIP在预训练和汇报的下游任务上，简单起见没有再进行更多这方面的特殊处理，您也可以在目前的基础上继续优化数据处理和采样流程，可能会有更好的提升~

from chinese-clip.

jerryl777 avatar jerryl777 commented on May 13, 2024

感谢回复。
冒昧多问一下，一个图对应多个文本的情况，“将这多个文本拼接成一个长文本”以及“拆成多个图文对样本”这两种处理方式的优劣是什么？

from chinese-clip.

yangapku avatar yangapku commented on May 13, 2024

您好！我们使用"多个文本拼接成一个长文本"这种情况还是比较少的，主要的考虑，一个是可能文本的长度会变长，有可能超出我们预训练模型52的文本长度限制，另一个是可能文本的形式会和下游应用产生一定的gap（下游一般是针对单条文本的任务）。最终采用了直接拆成多个图文对样本这种方案。

如果有更多问题，欢迎继续留言。如果觉得Chinese-CLIP代码库对您有帮助，请您为我们点点star⭐️并推荐给身边的朋友们！

from chinese-clip.

jerryl777 avatar jerryl777 commented on May 13, 2024

👌谢谢！

from chinese-clip.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.