Comments (4)
Hi @daegonYu ,
This is a hyperparameter for tuning. Empirically we observe that lower temperature will lead to better performance but might cause training instability under float16 precision for large models. A lower temperature allows the logits to vary in a wider range and thus has more flexibility.
from unilm.
“A lower temperature allows the logits to vary in a wider range and thus has more flexibility.” This can be interpreted as saying that embeddings make it easier to learn more diverse expressions. But in "https://huggingface.co/intfloat/multilingual-e5-base"
3. Why does the cosine similarity scores distribute around 0.7 to 1.0?
This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.
For text embedding tasks like text retrieval or semantic similarity, what matters is the relative order of the scores instead of the absolute values, so this should not be an issue.
If embeddings can be expressed in wider range, I think cosine similarity should be distributed over a wide range. Cosine similarity is distributed between 0.7 and 1.0. It's difficult to understand because it seems like something contradictory. Simply put, I wonder why lowering the temperature allows learning a wider range of logits.
from unilm.
The logits are calculated with cosine_similarity / t
. Therefore, the logits will fall in [-100, 100]
with t = 0.01
and [-50, 50]
with t=0.02
, etc.
However, this does not mean the learned cosine similarity will be in a wider range. On the contrary, the cosine similarity tends to concentrate as the temperature becomes lower.
from unilm.
All right. I understand what you said, but why does "the cosine similarity tends to concentrate as the temperature becomes lower." Can you tell if this is happening?
from unilm.
Related Issues (20)
- (BEIT-3) I can't find transformers.py HOT 3
- Kosmos-2.5 Dependencies Should Be Updated HOT 1
- The pretraining code and model of beit3
- TROCR model weight file cannot be downloaded.
- BEIT3 RuntimeError: Size mismatch when loading checkpoint for BEiT3ForRetrieval model
- Kosmos-2.5 - Deployment Challenges on a Windows 11 + RTX 3090 PC HOT 1
- [BEIT-3] error happens when I evaluate BEiT-3 finetuned model on VQAv2 HOT 4
- Missing Documentation
- Kosmos 2.5 dataset availability
- Kosmos-2.5 - Image-to-markdown generation for images outside the sample-set provided is almost entirely garbled - output markdown is completely unusable.
- Kosmos-2.5 - Python version 3.10.x (and any other confirmed working versions) should be mentioned as a requirement to deploy & infer the Kosmos-2.5 model
- Kosmos-2 batch modality and processing speed
- how to resume training for layoutlmv3-publaynet
- TextDiffuser - SD2.1 code and weights
- the problem when load Beats model
- kosmos-2.5 | trying to connect to `openaipublic.blob.core.windows.net`
- layoutlmv3 pretrain
- Compile LayoutLMv3 using Neuron SDK for AWS Inferentia (inf1)
- Beit-3 , clarification on classification task and associated pretrained weights
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unilm.