Feature/add ToolCallPrecisionRecallMatchGrader & rename: ToolCallSequenceMatchGrader ->ToolCallStepSequenceMatchGrader #68
Conversation
…uation - Add ToolCallSequenceMatchSimpleGrader supporting precision/recall metrics - Support flexible matching with/without arguments - Add comprehensive test suite with 21 test cases - Update documentation in overview.md and agent_graders.md - Emphasize ToolCallSequenceMatchGrader for multi-step complex scenarios
helloml0326
requested review from
XiaoBoAI,
XieLipeng0830,
chr6192,
ployts and
weizhang25
| RECALL = "recall" | ||
|
|
||
|
|
||
| class ToolCallSequenceMatchSimpleGrader(BaseGrader): |
There was a problem hiding this comment.
class ToolCallSequenceExpectationGrader ?
"simple" is not necessary in the class name if it describes the "simple" nature of this grader.
"Simple" does not help people understand this grader at all.
There was a problem hiding this comment.
ToolCallSequenceMatchSimpleGrader is a simplified version of ToolCallSequenceMatchGrader. The latter uses a multi-step concept, treating tool_call as a two-dimensional array, while the former simplifies this logic. Which naming approach do you prefer:
Keep the current names: ToolCallSequenceMatchGrader, ToolCallSequenceMatchSimpleGrader
Rename based on code logic: ToolCallSequenceMatchGrader → ToolCallStepSequenceMatchGrader, ToolCallSequenceMatchSimpleGrader → ToolCallPrecisionRecallMatchGrader
There was a problem hiding this comment.
Rename based on code logic: ToolCallSequenceMatchGrader → ToolCallStepSequenceMatchGrader, ToolCallSequenceMatchSimpleGrader → ToolCallPrecisionRecallMatchGrader
This looks better.
helloml0326
changed the title
helloml0326
changed the title
4 participants
OpenJudge Version
[The version of OpenJudge you are working on, e.g.
import openjudge; print(openjudge.__version__)]Description
[Please describe the background, purpose, changes made, and how to test this PR]
Checklist
Please check the following items before code is ready to be reviewed.
pre-commit run --all-filescommand