Files
coder/coderd/x/chatd/encrypted_reasoning_fallback.go
Michael Suchacz b2c8798e0e fix: replay stateless OpenAI reasoning across chat turns (#30298)
OpenAI Responses chats default to `store=false`, and chatd only replayed
reasoning as an `item_reference`, which needs provider-side storage.
Reasoning was therefore dropped between tool steps and turns, so
reasoning models lost their chain of thought mid-task.

This pins coder/fantasy `coder_2_33` at 6ba8681, which includes fantasy
#63 (and #64, #65, and #66). #63 captures each completed reasoning item
(ID, `encrypted_content`, summary) and replays it in full regardless of
`store`. In chatd:

- Every Responses call requests `reasoning.encrypted_content`. The
`store` default and configured includes are unchanged, and Chat
Completions requests are untouched.
- Prompt rebuilds keep reasoning parts without visible text when they
carry finalized encrypted state, so reasoning survives durable history
and coderd restarts.
- The provider-switch sanitizer also strips OpenAI reasoning from
history produced by a different or unknown provider instance, since
reasoning IDs and encrypted content only resolve where they were issued.
- A compaction model override on a different provider instance, or with
a different model, gets a compaction prompt without the chat's OpenAI
reasoning.
- Each saved reasoning part records the provider instance, model, and
credential (the deployment's or the user's personal API key) that
produced it, and the sanitizer keeps it only when all three match the
current turn, because another model, or the organization behind another
credential, can reject the encrypted content. Switching models, adding
or removing a personal key, or editing a config's model or provider
drops the earlier reasoning; switching back restores it. The record is
never returned by the API. Reasoning saved before this change has no
record and follows its model config.
- If OpenAI rejects replayed reasoning it cannot decrypt
(`invalid_encrypted_content`), for example after the deployment's key is
replaced with one from another organization, the request is retried once
without reasoning, and the reasoning that step produces is saved with
the provider's next rejection epoch. Epochs are kept separately for each
provider, model, and credential. Later steps and turns leave out
reasoning from earlier epochs, on any replica and after restarts, so
only the first request after the change is rejected and newer reasoning
keeps replaying. A retried step that returns no reasoning saves its
epoch on its first part. The advisor's prompt never includes OpenAI
reasoning, as on `main`.

## Testing

Rebased onto main at 98332c739b as 5735acb36ed (same tree as merging
the pre-rebase head a378071862a with main; older SHAs below refer to the
pre-rebase history). 3c17d807a33 then adds the model to the reasoning
record, 21156b2b32e pins Fantasy d55a867, which treats a response that
echoes `store: false` as unstored, bf795daa923 adds the retry and the
advisor change, 7eb8b1d029a moves the pin to 88cffb5 (`coder_2_33`
merged into #63; its OpenAI provider code is identical to d55a867), and
f4f40734a41 pins `coder_2_33` at 6ba8681 once #63 merged. That adds
fantasy #66, which replays the Responses message phase on assistant
text. abccede81f7 makes the retry remember rejected reasoning in the
process. The branch was then rebased onto main at abc2faed60, so every
SHA above refers to the pre-rebase history. 5fb3ce1689 replaces that
memory with the credential and a rejection epoch saved on each reasoning
part, and 62b5cb532a keeps epochs per provider, model, and credential
and saves the epoch when the retried step returns no reasoning.
1057beb327 hides the four internal fields from the Swagger spec and API
reference, and 29a705e099 merges main at d68b2c8b25. Its only
conflicts were the fantasy pin (kept at 6ba8681, whose `coder_2_33`
history includes main's 3510552) and test imports.

- Tests in `chatprompt`, `chatprovider`, `chatsanitize`, the
provider-switch sanitizer, and `integration_responses_test.go` cover
tool-step and durable replay, legacy and incomplete metadata, provider
switches, and a web search chat switching from `store=false` to
`store=true`. Integration tests also cover the retry, a later tool step
after it, the next turn on a fresh server against the same database
(with and without reasoning in the retried step), a personal key added
mid-chat, and the advisor prompt. Red-green verified, including a config
whose model is edited in place. The compaction override's model check
has unit tests and UAT coverage but no integration test. `go vet`,
targeted race tests, and the pre-commit hook (gen, fmt, lint, build)
pass. Local pre-push hooks were skipped; CI covers them.
- Remote UAT on dogfood with real gpt-5-mini and gpt-5.4 behind a
logging proxy that checks every request body. Rounds 1 and 2 failed on
dangling item references (both fixed). Rounds 3 to 5 passed
(ee3c0bf3d60, 0580870b5ff, a378071862a): tool-step and durable replay,
restarts, interrupts, provider switches and config repoints, web search
history, and compaction overrides. Both test providers shared one
upstream account, so cross-provider checks cover request shape, not a
real rejection. Round 6 at 21156b2b32e (model record) with gpt-5-mini
and gpt-5-nano: PASS. 151 Responses requests, all 200; no request
carried reasoning from a different model except an older, unrecorded
part that follows its config by design. Same-model config switches,
switching back, edits to a config's model, a mid-turn switch, restarts,
and a same-provider compaction override with a different model all
behaved as described. Round 7 at bf795daa923 (retry and advisor): PASS.
Through AI Gateway, a corrupted saved blob is rejected with HTTP 400
`invalid_encrypted_content`. All 14 rejected requests were retried once
without reasoning and returned 200, a different 400 was not retried, and
advisor requests on the chat model, a same-provider override, and
another provider carried no reasoning.
- f4f40734a41 changes only the pin. Its new OpenAI code is fantasy #66,
which UAT rounds 1 to 7 did not run. A live check of fantasy 6ba8681
through AI Gateway, two turns with tool calls on gpt-5.5 and gpt-5-mini
with `store` false and true: all 18 requests returned 200, and the
gpt-5.5 requests replayed encrypted reasoning followed by assistant
messages with the `commentary` and `final_answer` phases. `go test
./coderd/x/chatd/... ./codersdk/` passes (21 packages) on the same
fantasy tree.
- UAT round 8 at abccede81f7 (remembered rejection, and fantasy #66
through chatd) with gpt-5.5, gpt-5-mini, and gpt-5-nano: PASS. 110
requests; the 6 rejections were 5 corrupted saved blobs, each retried
once without reasoning and answered 200, and one unrelated `temperature`
error that was not retried. After the first rejection, later steps and
turns in that chat were never rejected again and replayed newer
reasoning. A coderd restart cost one more rejection, which also dropped
reasoning saved after the first one, and a second chat was unaffected.
gpt-5.5 requests replayed assistant messages with the phases the model
returned, with `store` false and true.
- UAT round 9 at 62b5cb532a (saved rejection epoch, personal key
switch) with gpt-5.5, gpt-5-mini, and gpt-5-nano: PASS. 158 Responses
and 7 Chat Completions requests; the 6 rejections were corrupted saved
blobs, each retried once without reasoning and answered 200, and an
unrelated `temperature` error was not retried. After a coderd restart,
and on a second coderd against the same database, the chat was not
rejected again and kept replaying the reasoning saved after the
rejection. Adding a personal key left out the deployment key's reasoning
before sending, and removing it brought that reasoning back. A retried
step without reasoning, two chats rejected at once, compaction, a model
switch after a rejection, and a switch to Chat Completions and back all
behaved as described.

## Review record

- Independent full-branch test audit at bbe1a1b27a: PASS.
- Codex through a378071862: compaction overrides kept foreign reasoning
and configs could be repointed to another provider (both fixed); a
request for documentation TODOs and two behaviors that already exist on
`main` (declined).
- Codex on 5735acb36e: two findings. Reasoning replayed after a
same-provider model switch: fixed in 3c17d807a33. A provider's base URL
edited in place keeps its reasoning: covered by the retry in
bf795daa923.
- Codex on 21156b2b32, bf795daa92, and 7eb8b1d029: no major issues.
Codex on f4f40734a4: rejected reasoning stayed in the chat's history, so
each later request in that chat was rejected and retried again without
any reasoning. Fixed in abccede81f7. Codex on abccede81f: that memory
was lost on a restart or another replica, where the next rejection also
dropped newer reasoning, and it grew without bound. Fixed in
5fb3ce1689. Codex on 5fb3ce1689: a rejection advanced the epoch for
every model and credential on the provider, and a retried step without
reasoning lost the new epoch. Both fixed in 62b5cb532a. Codex on
62b5cb532a: keep reasoning out of a config switched from Responses to
Chat Completions (declined: the Chat Completions converter already skips
reasoning parts, confirmed with a probe test). Codex on 29a705e099: no
major issues.
- ethanndickson: retry without reasoning on `invalid_encrypted_content`,
and keep OpenAI reasoning out of the advisor prompt. Both fixed in
bf795daa923. Then: in-process memory is not enough because dogfood
restarts on every merge and runs several replicas. Fixed in 5fb3ce1689
with his proposed rejection epoch in chat history, plus the credential
record, and tested with the next turn on a fresh server. Then: hide the
internal fields from Swagger, fixed in 1057beb327. Approved on
62b5cb532a.
- coder/fantasy#63 review: honoring an echoed `store: false` is fixed in
d55a867; replaying web searches that had no preceding reasoning is a
pre-existing gap, tracked in #30528.
- CI on 5735acb36ed: green after rerunning two failures outside this
change (a `cli` `TestSSH` data race, and a `test-go-pg` job timeout that
also fails on `main`). CI on 21156b2b32e: all checks pass. CI on
bf795daa923: all checks pass. CI on 7eb8b1d029a: all checks pass. CI on
f4f40734a41: all checks pass. CI on abccede81f7 did not run, because the
branch conflicted with `main` (#30492 changed the provider-switch
sanitizer to return database errors). CI on 62b5cb532a: all checks
pass. CI on 29a705e099: all checks pass.

> Xum, an AI coding agent, implemented, tested, and opened this PR on
behalf of @ibetitsmike.
2026-10-13 13:49:12 +02:00

85 lines
2.7 KiB
Go

package chatd
import (
"context"
"errors"
"sync/atomic"
"charm.land/fantasy"
"github.com/google/uuid"
"github.com/openai/openai-go/v3"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/coderd/x/chatd/chatprovider"
)
func isInvalidEncryptedContentError(err error) bool {
var apiErr *openai.Error
return errors.As(err, &apiErr) && apiErr.Code == "invalid_encrypted_content"
}
// withEncryptedReasoningFallback wraps OpenAI Responses models in an
// encryptedReasoningFallbackModel that sets rejected when it retries.
func (p *Server) withEncryptedReasoningFallback(model chatprovider.Model, chatID uuid.UUID, rejected *atomic.Bool) chatprovider.Model {
if !model.Transport().UsesResponses() {
return model
}
return model.WithLanguageModel(&encryptedReasoningFallbackModel{
LanguageModel: model.LanguageModel(),
logger: p.logger.With(slog.F("chat_id", chatID)),
rejected: rejected,
})
}
// encryptedReasoningFallbackModel retries a request without OpenAI reasoning
// when the provider rejects its encrypted content. Encrypted reasoning only
// decrypts for the OpenAI organization that produced it, which can change
// without a provider, model, or credential change, for example when an admin
// replaces the deployment key with one from another organization. The
// rejected reasoning stays in the chat history, so the step stamps its new
// reasoning with the next epoch, and later steps leave out reasoning from
// earlier epochs.
type encryptedReasoningFallbackModel struct {
fantasy.LanguageModel
logger slog.Logger
rejected *atomic.Bool
}
// Stream retries only when the rejection arrives before any output.
func (m *encryptedReasoningFallbackModel) Stream(ctx context.Context, call fantasy.Call) (fantasy.StreamResponse, error) {
stream, err := m.LanguageModel.Stream(ctx, call)
if err != nil {
return nil, err
}
return func(yield func(fantasy.StreamPart) bool) {
var rejection error
started := false
for part := range stream {
if !started && part.Type == fantasy.StreamPartTypeError && isInvalidEncryptedContentError(part.Error) {
rejection = part.Error
break
}
started = started || part.Type != fantasy.StreamPartTypeWarnings
if !yield(part) {
return
}
}
if rejection == nil {
return
}
m.rejected.Store(true)
m.logger.Warn(ctx, "retrying model call without rejected encrypted reasoning", slogError(rejection))
call.Prompt = dropOpenAIReasoningParts(ctx, m.logger, call.Prompt)
retry, err := m.LanguageModel.Stream(ctx, call)
if err != nil {
yield(fantasy.StreamPart{Type: fantasy.StreamPartTypeError, Error: err})
return
}
for part := range retry {
if !yield(part) {
return
}
}
}, nil
}