Reranker model fails to load (maidalun1020/bce-reranker-base_v1) - no max token length is set about infinity HOT 4 CLOSED

@Matheus-Garbelini haha, thanks a lot @michaelfeil. This model is indeed not mine, hence I could just assume it was some upstream config issue, but you confirmed that this was the case.

Currently I'm running infinity with embeddings + reranking models and it's works flawlessly.
Regards.

from infinity.

Hey @Matheus-Garbelini , thanks for opening the issue

Looks like there is no max_length attr in the config.json

Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

1. How does it work if you send a rerank request using from sentence_transformer import CrossEncoder? Is the max length respected for this model? (I am pretty sure the issue is upstream)
2. Given that max_length is not in the config, how would you expect the 514 token length to be handled? (model supports only 514)

from infinity.

Please check the usage of "maidalun1020/bce-reranker-base_v1" in: https://github.com/netease-youdao/BCEmbedding?tab=readme-ov-file#3-based-on-sentence_transformers

from sentence_transformers import CrossEncoder

# init reranker model
model = CrossEncoder('maidalun1020/bce-reranker-base_v1', max_length=512)

# calculate scores of sentence pairs
scores = model.predict(sentence_pairs)

max_length should be 512.

from infinity.

@shenlei1020 @Matheus-Garbelini Thanks for your comments - excited to see your responses here.

I would avoid overwriting the defaults of the authors model code - it depends on the person publishing the model. In this case, a wrong value was sent on purpose by the engineers behind https://huggingface.co/maidalun1020/bce-reranker-base_v1/discussions/4 . I encourage you to fix things in the future directly in the upstream repos - infinity just optimizes the inference .

https://huggingface.co/maidalun1020/bce-reranker-base_v1/discussions/4/files will solve it.

from infinity.