FIX: warn instead of raising on reasoning-model token truncation#2166
Open
romanlutz wants to merge 2 commits into
Open
FIX: warn instead of raising on reasoning-model token truncation#2166romanlutz wants to merge 2 commits into
romanlutz wants to merge 2 commits into
Conversation
When a reasoning model hits max_completion_tokens, the API returns finish_reason='length' with empty visible content. Previously this raised EmptyResponseException (misleadingly reported as 'Status Code: 204') and triggered the 10x retry storm. _validate_response now logs a warning explaining that reasoning models consume tokens on hidden reasoning, and returns a graceful empty response (error='empty') instead of raising, so runs continue. Non-length empty responses still raise as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jbolor21
approved these changes
Jul 11, 2026
Comment on lines
+338
to
+345
| if not (has_content or has_audio or has_tool_calls): | ||
| return construct_response_from_request( | ||
| request=request, | ||
| response_text_pieces=[""], | ||
| response_type="text", | ||
| error="empty", | ||
| ) | ||
| return None |
Contributor
There was a problem hiding this comment.
Is this the right behavior? based on how I understand this, if there is any content, we would just ignore it? Also, there is no test for this currently
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Reasoning models spend completion tokens on hidden reasoning before emitting any visible output. When
max_completion_tokensis set too low, the API can returnfinish_reason="length"with empty visible content.OpenAIChatTarget._validate_responsetreated that empty content as a hard failure and raisedEmptyResponseException(surfaced with a misleading "Status Code: 204"), which also triggered the full retry loop even though retrying with the same token budget cannot succeed.Since a low token budget can be a deliberate configuration choice, this changes the behavior to warn rather than raise:
finish_reason="length", log a warning explaining that reasoning models consume tokens on hidden reasoning and thatmax_completion_tokensmay need to be increased.error="empty") so the run continues instead of raising and retrying.EmptyResponseExceptionas before.The final stored result for the empty case is unchanged from today (an
error="empty"piece). The difference is no exception, no retry storm, and a clear warning pointing at the token limit.Tests and Documentation
test_validate_response_success_lengthto assert the truncation warning is emitted when content is present.test_validate_response_length_empty_returns_empty_responseasserting that a truncated-and-empty response returns aMessagewitherror="empty"and raises nothing.tests/unit/prompt_target/target/test_openai_chat_target.pysuite passes (101 tests).No documentation changes needed. JupyText not run (no doc or notebook changes).