How to do the automatic safety evaluations

[tivon-x]

Hello!

This is very nice work!

I kindly want to know how to do the automatic safety evaluations. According to the paper, you use a safety critique llm for the evaluations. Will you release the safety critique llm in the future? Or are there any other methods for the automatic safety evaluations?

[IS2Lab]

Thanks for your recognition of our work. At present, we have no plans to release our safety critique llm in the short term. However, we are happy to provide support to help you with safety evaluations. If you require automatic safety evaluations, please provide a jsonl file including traceid, prompt, and the corresponding response and send it to xiaohanyuan@zju.edu.cn (if you have any questions about the required submission files, please contact this email first.) We will evaluate and return the results according to the order of requests.

In addition, for other automatic safety evaluations, we introduced and analyzed various methods in our paper. You can refer to the paper.

Thanks again for your understanding and support!

[krishnakanthnakkav2]

Hello Authors,

Any updates on the release of critic LLM (Qwen-14b-chat) that performs automatic evalutions?

[zggg1p]

We release Octopus, an automated LLM safety evaluator, to address the community’s need for accurate and reproducible safety assessment tools. You can download the model from HuggingFace or ModelScope. Thanks for your interest and support.