Skip to content

validate_tool_arguments does not enforce Literal/const constraints on tool fields #1106

@planetf1

Description

@planetf1

Background

Literal constraints on tool fields are silently dropped on the way to the
backend and on the way to validation. Two independent gaps combine to produce
the same end-user symptom — strict=True isn't actually strict for
Literal[...] fields.

This isn't introduced by #1104; both gaps predate it. PR #1104 made the
validator-side gap visible because discriminated-union tools now reach the
validator with Literal tag constraints in their schema for the first time.

End-user impact

strict=True isn't truly strict. Two reproducers, one for each gap:

Gap A — discriminated-union branches

class Cat(BaseModel):
    kind: Literal["cat"]
    name: str

class Dog(BaseModel):
    kind: Literal["dog"]
    name: str
    breed: str

def act(pet: Annotated[Cat | Dog, Field(discriminator="kind")]) -> str:
    ...

mt = MelleaTool.from_callable(act)
validate_tool_arguments(
    mt,
    {"pet": {"kind": "horse", "name": "Bob", "breed": "lab"}},
    strict=True,
)
# returns successfully — "horse" is not a valid kind

After #1104, the schema produced for pet correctly includes
{"const": "cat", "type": "string"} on each branch's kind field. The
validator receives that constraint but ignores it: _build_pydantic_type_from_schema
in mellea/backends/tools.py:549 maps everything with type: "string" to
plain str, never inspecting const or enum.

Gap B — plain Literal fields

def file_op(path: str, mode: Literal["read", "write"]) -> str:
    ...

mt = MelleaTool.from_callable(file_op)
validate_tool_arguments(
    mt,
    {"path": "/etc/passwd", "mode": "delete"},
    strict=True,
)
# returns successfully — "delete" is not in the Literal

For plain Literal fields the enum keyword is also lost in the schema
itself: Pydantic emits {"enum": ["read", "write"], "type": "string"} for
mode, but the simple-type flattening near the bottom of
convert_function_to_ollama_tool keeps only the type and discards the
enum array. So the constraint is invisible to both the backend (the
LLM doesn't see the allowed values either) and the validator.

When this matters in practice

Most of the time, LLMs emit values that are in the schema, so this isn't a
frequent runtime problem. It does matter when:

  • smaller models may hallucinate a tag value the schema forbids
  • you want defensive validation against malformed or hostile payloads
  • a test suite expects strict=True to actually reject every malformed shape
  • the model never saw the allowed values at all (Gap B), so it has no chance
    of knowing them

Workaround

Attach a field_validator to the Pydantic model, or post-validate arguments
manually before invoking the tool.

Success criteria

  1. The schema produced by convert_function_to_ollama_tool preserves
    enum and const keywords on simple-type fields. Plain Literal[...]
    fields should arrive at backends with the allowed values intact.
  2. validate_tool_arguments(..., strict=True) rejects any payload where a
    field constrained by Literal[...], Enum, or schema-level const /
    enum carries a value outside the allowed set. Both reproducers above
    should raise ValidationError.
  3. A negative test for each reproducer should land alongside the fix.

Suggested direction

Two small changes, ideally in one PR:

  • In the simple-type flattening branch of convert_function_to_ollama_tool,
    preserve enum and const keywords on the output property when present
    on the input.

  • In _build_pydantic_type_from_schema, recognise:

    • {"const": V}typing.Literal[V]
    • {"enum": [V1, V2, ...]}typing.Literal[V1, V2, ...]

    before falling through to the plain-type mapping.

Together this closes both gaps with a few branches of new logic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/toolsTool framework, Bash/Python tools, tool call lifecyclebugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions