Ask about the model of the tokenizer.

[lx-Meteors]

Hello, I would like to ask if the tokenizers used for Qwen and Llama are both Qwen2.5-7B and Llama-3.1-8B?

Why not directly use the word lists of DeepSeek-R1-Distill-Qwen and DeepSeek-R1-Distill-Llama? Is there any design-related reason for this?

thank you.

[MikeDean2367]

Hi,

To answer your first question: Yes! We use the original tokenizers from Qwen and Llama.

As for your second question, there is no specific design-related reason. We initially tried loading the tokenizer from the Distill version using AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"), but it threw an error. As a workaround, we used the original tokenizers and found that they worked without any issues.

[zxlzr]

hi do you have any other questions?

[lx-Meteors]

Thank you. I have no other questions. Meteor ***@***.***