Skip to content

Conversation

@andrewtran117
Copy link
Contributor

Adds CustomAdapter for onboarding user defined custom evaluation tools. Provides defaults for required schema fields and supports JSON/JSONL/directory inputs.

Copy link

@borgr borgr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good, I mainly wonder about higher level stuff here (naming and usecases).
So currently this works specifically from jsons, maybe the naming should convey it?
It is not custom in the sense that you expect someone to change the code to read their files. Someone still needs to change the outputs to the jsons, when will people use it?
I am thinking for example now on the EvEvEv people, will they want to use this code somehow? (They parse leaderboards into our format). We need to add to README and documentation to lead people to use it when they have a relevant usecase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants