What does "unique tokens" mean (in the paper) ? about starcoder2 HOT 2 CLOSED

#3 The term "unique tokens" refers to the total count of distinct token instances in the dataset after tokenization, not the size of the vocabulary. When a dataset mentions "622B+ unique tokens," it means there are over 622 billion distinct occurrences of tokens identified using a specific tokenizer, like StarCoderV2's, after processing and deduplicating the dataset. It's about the total unique token occurrences, not the number of different words or symbols. 术语“唯一标记”是指标记化后数据集中不同标记实例的总数，而不是词汇表的大小。当数据集提到“622B+ 唯一标记”时，这意味着在处理和删除重复数据集后，使用特定标记生成器（例如 StarCoderV2 的标记生成器）识别出超过 6220 亿个不同的标记。它与唯一标记出现的总数有关，而不是不同单词或符号的数量。

Thanks for the clarification!

from starcoder2.

#3
The term "unique tokens" refers to the total count of distinct token instances in the dataset after tokenization, not the size of the vocabulary. When a dataset mentions "622B+ unique tokens," it means there are over 622 billion distinct occurrences of tokens identified using a specific tokenizer, like StarCoderV2's, after processing and deduplicating the dataset. It's about the total unique token occurrences, not the number of different words or symbols.

from starcoder2.