[Bug]: Evaluation fails using gpt-5 with ragas: temperature parameter not supported

[antoniomtz]

Version

1.3.0

Which installation method(s) does this occur on?

PyPi

Describe the bug.

Description

When running ragas accuracy evaluations using gpt-5 model, the evaluation fails with a 400 error indicating that the temperature parameter is not supported.

Error Message

ragas.metrics.nv_metrics:157 - An error occurred: Error code: 400 - {'error': {'message': "Unsupported value: 'temperature' does not support 0.1 with this model. Only the default (1) value is supported.", 'type': 'invalid_request_error', 'param': 'temperature', 'code': 'unsupported_value'}}. Skipping a sample by assigning it nan score.

As per openAI documentation https://platform.openai.com/docs/guides/latest-model#gpt-5-parameter-compatibility, temperature is not supported for GPT-5 models.

Minimum reproducible example

create evaluators using ragas:

judge_llm:
   _type: openai
   model_name: gpt-5

accuracy:
  _type: ragas
  metric: AnswerAccuracy
  llm_name: judge_llm

Relevant log output

Click here to see error details

WARNING  - ragas.metrics._nv_metrics:157 - An error occurred: Error code: 400 - {'error': {'message': "Unsupported value: 'temperature' does not support 0.1 with this model. Only the default (1) value is supported.", 'type': 'invalid_request_error', 'param': 'temperature', 'code': 'unsupported_value'}}. Skipping a sample by assigning it nan score.

Other/Misc.

No response

Code of Conduct

• I agree to follow the NeMo Agent toolkit Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[hsachdeva9]

Hey @antoniomtz, I am really interested in working on this. Could you please assign this to me?

[AnuradhaKaruppiah]

Hello @antoniomtz, thx for filing this issue. I believe this has been addressed in #1134. Can you please try with that fix?

[antoniomtz]

@AnuradhaKaruppiah , sure I can test it. However, the fix for that PR was related to Claude and not GPT models.
@willkill07 comments?

[sushank-rana-a30001665]

Any updates on this ?

Also, can report that when tried to update my eval llm to from gpt-4.1-mini to gpt-5-mini notice all my llm based metrics have started throwing NaN.

Do you have any recommendation which gpt model maybe best suited for Ragas?

[willkill07]

Sorry for the delay @sushank-rana-a30001665

The issue is with the version of ragas we currently depend on.

Upstream ragas has added support for newer models, but we haven't updated the version yet due to source incompatible changes required and a backlog of other things to do.

https://docs.ragas.io/en/v0.4.0/howtos/migrations/migrate_from_v03_to_v04/#overview-of-major-changes

CCing @AnuradhaKaruppiah for viz on this with eval work

[AnuradhaKaruppiah]

It is indeed time for refreshing ragas and other evaluators. I will scope this into NAT1.5 and update here as soon as it is available in develop.

[willkill07]

This has been updated/fixed with our migration to a newer version of ragas.