Skip to content

Conversation

laulopezreal
Copy link

@laulopezreal laulopezreal commented Aug 20, 2025

Summary

Add a ToolAnswerResult wrapper and plumbing so tools that set result_as_answer=True return their raw string output as the final answer (skipping pydantic/json conversion and schema parsing).

Fix for Issue #3335: when a tool returns a plain string but sets result_as_answer=True (meaning the tool’s raw output should be used directly), the framework still attempted to convert/validate it into pydantic/json output. That caused incorrect parsing/validation and lost the intended raw-string answer. This PR prevents that conversion path for those cases.

##Key changes (files / behavior)

New type and behavior control

src/crewai/tools/tool_types.py
Adds ToolAnswerResult wrapper class (stores raw result and stringifies).

Agent / execution flow

src/crewai/agent.py
When iterating tools_results, if a tool result was marked result_as_answer the code now wraps the returned value with ToolAnswerResult before emitting AgentExecutionCompletedEvent.
src/crewai/task.py
Task._execute_core: when receiving agent result, capture raw_result from ToolAnswerResult so TaskOutput.raw holds the raw string.
Task._export_output: if result is a ToolAnswerResult, return (None, None) for pydantic/json outputs (skip conversion).

Tool usage

src/crewai/tools/tool_usage.py
Skip formatting when an available tool has result_as_answer True (so formatter/conversion paths are bypassed).

Tests

Added tests/tools/test_tool_result_as_answer.py with multiple unit tests validating:

  • a tool with result_as_answer=True bypasses conversion and leaves pydantic/json None,
  • normal conversion still works when result_as_answer=False,
  • last tool with result_as_answer=True wins,
  • ToolAnswerResult wrapper behaves as expected,
  • edge case outputs preserved,
  • reproduction test for issue [BUG] result_as_answer=True still triggers LLM STR→JSON conversion, altering final tool output #3335.
  • Small test cleanups across multiple test files (removed unused imports, made some tests commented/skipped).
  • Minor bugfixes / cleanups
  • src/crewai/agents/agent_adapters/langgraph/structured_output_converter.py
  • More specific exception handling (catch ValueError instead of broad except) when JSON decoding/extraction fails.
  • Several files: removed unused imports, cleaned typing imports, and small refactors (flow/path_utils, html_template_handler, etc.).

Behavioral impact

  • Tools that intentionally mark their outputs as final answers will now bypass conversion and be returned verbatim:
    TaskOutput.raw will contain the exact string returned by the tool.
  • TaskOutput.pydantic and TaskOutput.json_dict will be None for those outputs, preventing accidental schema validation or JSON extraction.
  • Existing tasks that rely on conversion remain unchanged (result_as_answer False).
  • Potential issues / review items

src/crewai/tools/tool_types.py: file ends with "No newline at end of file" in the diff — consider adding trailing newline for style.
src/crewai/agents/agent_adapters/langgraph/langgraph_adapter.py: commented out import (ToolMessage) — confirm intentional.
Ensure all callers that previously expected pydantic/json outputs handle the ToolAnswerResult case (Task._export_output now returns (None, None) — callers should expect that).
Run full test suite and CI to ensure no regressions; PR already introduces tests that validate the new behavior but double-check integration paths (delegation, agent events, and any consumers of TaskOutput.json_dict).

Note for reviewers

Suggested reviewers

Anyone responsible for tool execution and task output conversion (agents, tools, task serialization), and maintainers familiar with issue #3335.

Checklist

  • Confirm the goal: tools that set result_as_answer=True should return their raw string as the final answer, bypassing pydantic/json conversion and schema parsing.

  • Key files to review (changes & behavior)

src/crewai/tools/tool_types.py
Validate ToolAnswerResult implementation and str behavior.

src/crewai/agent.py
Confirm wrapping of tool result with ToolAnswerResult when result_as_answer True (lines added where AgentExecutionCompletedEvent emitted).

src/crewai/task.py
Verify Task._execute_core uses ToolAnswerResult to populate TaskOutput.raw and that Task._export_output returns (None, None) for ToolAnswerResult.
Ensure no callers assume pydantic/json are always present after this change.

src/crewai/agents/agent_adapters/langgraph/structured_output_converter.py
Confirm the narrower exception handling (catch ValueError) is correct for JSON decode/validate extraction.

Tests:

tests/tools/test_tool_result_as_answer.py
Review test coverage, edge cases, and mocking approach; ensure tests reflect intended behaviour and are not masking other issues.

Functional checks to perform locally

Run unit tests:

pytest tests/tools/test_tool_result_as_answer.py -q
pytest -q (or full suite) to detect regressions

Manually exercise a task where:

  • A tool returns a plain string and result_as_answer=True → expect TaskOutput.raw == exact string, pydantic/json None.
  • A tool returns JSON-like string but result_as_answer=True → ensure raw preserved (no parsing).
  • A tool returns a valid JSON and result_as_answer=False with output_pydantic set → ensure conversion still occurs.
  • Run any integration flows that previously relied on TaskOutput.json_dict/pydantic to ensure they gracefully handle None.

Edge cases & defensive questions

  • Multiple tools: confirm "last tool with result_as_answer True wins" behavior is intentional and documented.
  • Consumers: search for code paths that assume TaskOutput.pydantic or json_dict are non-None — these may need guard clauses.
  • Event consumers: verify listeners of AgentExecutionCompletedEvent and TaskStartedEvent handle ToolAnswerResult payloads correctly.
  • Type annotations: ensure new ToolAnswerResult type is properly typed/imported where needed to avoid mypy/typing issues.
  • Logging/observability: ensure the raw tool answers are not inadvertently leaked in logs (sensitive outputs).

Test & CI expectations

  • New unit tests must pass (tests/tools/test_tool_result_as_answer.py).
  • Run full test suite on CI and confirm no flakiness introduced by mocking or commented tests.

Suggested follow-ups (if approved)

  • Add a small integration test (optional) that simulates a real tool returning a string with result_as_answer True.
  • Add a short note in CHANGELOG / PR description describing the behavioral change (tools can now return raw answers).
  • Add unit tests for event listeners that may consume ToolAnswerResult.

@laulopezreal
Copy link
Author

@lorenzejay @tonykipkemboi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant