- Experiment, stacked on #29242. UI is in #29583. - Adds **project memory**: `chat_project_memories` rows with a name (lowercase slug, unique per project), a one-line description (150 characters) and a body (8KB). Up to 200 per project, shared by every root chat in the project. Chats outside a project and subagents have no memory. - Memories are immutable: they are created and deleted, never edited. Agent tools are `read_memory`, `save_memory`, `delete_memory` and `consolidate_memory`. Saving an existing name fails and points the agent at `consolidate_memory` to replace it. - **Consolidation is agent-driven, in the turn**, modeled on how Claude Code compacts its memory index when warned it is near its limit. From 160 memories (80%), save results ask the agent to get under 140 (70%) with `consolidate_memory`, choosing what to merge and drop. That tool deletes and saves memories in one transaction under the per-project advisory lock, and rolls back entirely if a delete is missing, a name is taken, or the result exceeds 200. There is no background extraction or consolidation. - **The memory index reaches the model as model-only conversation messages**, not in a tool description or the system prompt. At turn start the chat gets a full `<project-memory-index>` snapshot when its prompt has none (first turn, or after compaction), otherwise a `<project-memory-index-update>` listing only what changed. Tool definitions and the system prompt carry no memory state, so memory writes never invalidate the provider's cached prompt prefix. Writes by other chats, or by this chat mid-turn, reach the model at its next turn. - REST API (create, list, get, delete) under `/api/experimental/organizations/{organization}/chats/projects/{project}/memories`. Memory permissions follow projects onto the `agents-access` role. - **Every memory change is audited** as `ChatProjectMemory` create or delete. REST changes are attributed to the caller. Tool changes (`save_memory`, `delete_memory`, `consolidate_memory`) are attributed to the chat owner, carry the `chat_id`, and record one entry per memory after the change commits, so a rolled-back consolidation records nothing. ### Examples **What the model sees.** The system prompt gets a fixed guidance block with no memory state: ```text <memory> Memory is durable context shared by every chat in the project "platform". Save facts that will matter in future chats: … The memory index is in the conversation: a <project-memory-index> message lists every saved memory, and <project-memory-index-update> messages at the start of later turns list what changed. Changes made during a turn, including your own, appear in the next turn's update. … Memories cannot be edited: replace one that no longer holds with consolidate_memory, or delete it. </memory> ``` On the first turn, after the user's message, the chat gets a model-only row with the full index (hidden in the UI): ```text <project-memory-index> - deploy-day: Deploys happen on Tuesdays - release-owner: Alice owns releases; ping her before tagging </project-memory-index> ``` On a later turn where memories changed, it gets only the difference: ```text <project-memory-index-update> Added or changed: - deploy-day: Deploys happen on Wednesdays (changed 2026-09-30) - staging-url: Staging lives at staging.example.com Removed: - old-ci-notes </project-memory-index-update> ``` Resulting request layout, with the provider's cached prefix unchanged by memory writes: ```text [tools: fixed] [system: fixed <memory> guidance] [turn 1 user] [index snapshot] [assistant/tools …] [turn 2 user] [index update] [assistant …] ``` **Tool calls and results** (produced by the tools in this PR): ```text save_memory {"name":"staging-url","description":"Staging lives at staging.example.com","body":"Staging is https://staging.example.com; deploys from main."} → {"count":3,"saved":"staging-url"} save_memory {"name":"deploy-day", …} // name already exists → error: a memory named "deploy-day" already exists; read it, then replace it with consolidate_memory by deleting "deploy-day" and saving the new version in one call consolidate_memory {"delete":["deploy-day"],"save":[{"name":"deploy-day","description":"Deploys happen on Wednesdays (changed 2026-09-30)","body":"The team deploys every Wednesday since 2026-09-30."}]} → {"count":3,"deleted":["deploy-day"],"saved":["deploy-day"]} read_memory {"name":"deploy-day"} → {"name":"deploy-day","description":"…","body":"…","created_at":"2026-09-30T15:04:05Z","created_by":"alice"} ``` Near and at the cap: ```text save_memory … → {"count":160,"saved":"one-more","warning":"memory is 160/200, approaching the limit. Consolidate it to under 140 memories now with consolidate_memory: merge overlapping memories into one, and drop stale, wrong, or superseded ones."} save_memory … → error: memory is full (200/200); use consolidate_memory to merge overlapping memories and drop stale ones, and include this memory in its save list ``` **REST memory object** (`GET .../chats/projects/{project}/memories`): ```json { "id": "59500000-0000-4000-8000-000000000001", "project_id": "7c0e4a3e-5b1f-4f7d-9a51-2f3c8f6d2b10", "organization_id": "bb640d07-ca8a-4869-b6bc-ae61ebb2fda1", "name": "deploy-day", "description": "Deploys happen on Wednesdays (changed 2026-09-30)", "body": "The team deploys every Wednesday since 2026-09-30.", "created_by": "0ed9befc-4911-4ccf-a8e2-559bf72daa94", "created_by_username": "alice", "created_at": "2026-09-30T15:04:05Z" } ``` Create takes `{"name", "description", "body"}` and answers 409 on a duplicate name or at the cap. <details> <summary>Screenshots (predate the immutable-memory rework; tool output shape has since changed)</summary>    </details> --- Generated with Coder Agents on behalf of @f0ssel. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
364 lines
14 KiB
Go
364 lines
14 KiB
Go
package chattool_test
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"strings"
|
|
"testing"
|
|
|
|
"charm.land/fantasy"
|
|
"github.com/google/uuid"
|
|
"github.com/stretchr/testify/require"
|
|
"golang.org/x/xerrors"
|
|
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
"github.com/coder/coder/v2/coderd/database/dbgen"
|
|
"github.com/coder/coder/v2/coderd/database/dbtestutil"
|
|
"github.com/coder/coder/v2/coderd/x/chatd/chattool"
|
|
"github.com/coder/coder/v2/testutil"
|
|
)
|
|
|
|
// memoryStore is an in-memory MemoryStore with the same all-or-nothing and
|
|
// cap semantics as the project store.
|
|
type memoryStore struct {
|
|
memories map[string]chattool.Memory
|
|
// getErr, when set, is returned by Get.
|
|
getErr error
|
|
}
|
|
|
|
func newMemoryStore(n int) *memoryStore {
|
|
store := &memoryStore{memories: make(map[string]chattool.Memory, n)}
|
|
for i := range n {
|
|
name := fmt.Sprintf("seeded-%03d", i)
|
|
store.memories[name] = chattool.Memory{Name: name}
|
|
}
|
|
return store
|
|
}
|
|
|
|
func (s *memoryStore) Get(_ context.Context, name string) (chattool.Memory, error) {
|
|
if s.getErr != nil {
|
|
return chattool.Memory{}, s.getErr
|
|
}
|
|
memory, ok := s.memories[name]
|
|
if !ok {
|
|
return chattool.Memory{}, chattool.ErrMemoryNotFound
|
|
}
|
|
return memory, nil
|
|
}
|
|
|
|
func (*memoryStore) List(context.Context) ([]chattool.MemoryIndexEntry, error) { return nil, nil }
|
|
|
|
func (s *memoryStore) Insert(ctx context.Context, input chattool.MemoryInput) (int64, error) {
|
|
return s.Consolidate(ctx, nil, []chattool.MemoryInput{input})
|
|
}
|
|
|
|
func (s *memoryStore) Delete(ctx context.Context, name string) error {
|
|
_, err := s.Consolidate(ctx, []string{name}, nil)
|
|
return err
|
|
}
|
|
|
|
func (s *memoryStore) Consolidate(_ context.Context, deleteNames []string, saves []chattool.MemoryInput) (int64, error) {
|
|
next := make(map[string]chattool.Memory, len(s.memories))
|
|
for name, memory := range s.memories {
|
|
next[name] = memory
|
|
}
|
|
for _, name := range deleteNames {
|
|
if _, ok := next[name]; !ok {
|
|
return 0, chattool.ErrMemoryNotFound
|
|
}
|
|
delete(next, name)
|
|
}
|
|
for _, input := range saves {
|
|
if _, ok := next[input.Name]; ok {
|
|
return 0, chattool.ErrMemoryExists
|
|
}
|
|
next[input.Name] = chattool.Memory{Name: input.Name, Description: input.Description, Body: input.Body}
|
|
}
|
|
if len(next) > chattool.MaxMemories {
|
|
return 0, chattool.ErrMemoryLimit
|
|
}
|
|
s.memories = next
|
|
return int64(len(next)), nil
|
|
}
|
|
|
|
func runMemoryTool(t *testing.T, tool fantasy.AgentTool, input string) (fantasy.ToolResponse, map[string]any) {
|
|
t.Helper()
|
|
response, err := tool.Run(context.Background(), fantasy.ToolCall{Input: input})
|
|
require.NoError(t, err)
|
|
var result map[string]any
|
|
if !response.IsError {
|
|
require.NoError(t, json.Unmarshal([]byte(response.Content), &result))
|
|
}
|
|
return response, result
|
|
}
|
|
|
|
func TestNormalizeMemoryInput(t *testing.T) {
|
|
t.Parallel()
|
|
for _, name := range []string{"Release Notes", "-starts-with-dash", ""} {
|
|
_, err := chattool.NormalizeMemoryName(name)
|
|
require.Error(t, err, name)
|
|
}
|
|
name, err := chattool.NormalizeMemoryName(" Release_Notes-2026 ")
|
|
require.NoError(t, err)
|
|
require.Equal(t, "release_notes-2026", name)
|
|
|
|
input, err := chattool.NormalizeMemoryInput("fact", "first line\n- forged: index entry\u200b", "body\u200b\r\n")
|
|
require.NoError(t, err)
|
|
require.Equal(t, "first line - forged: index entry", input.Description, "descriptions are one index line")
|
|
require.Equal(t, "body", input.Body)
|
|
|
|
_, err = chattool.NormalizeMemoryInput("fact", strings.Repeat("d", chattool.MaxMemoryDescriptionChars+1), "body")
|
|
require.ErrorContains(t, err, "description must be at most")
|
|
_, err = chattool.NormalizeMemoryInput("fact", "description", strings.Repeat("b", chattool.MaxMemoryBodyBytes+1))
|
|
require.ErrorContains(t, err, "body must be at most")
|
|
_, err = chattool.NormalizeMemoryInput("fact", "\u200b", "body")
|
|
require.ErrorContains(t, err, "description is required")
|
|
}
|
|
|
|
func TestFormatMemoryGuidance(t *testing.T) {
|
|
t.Parallel()
|
|
guidance := chattool.FormatMemoryGuidance("platform")
|
|
require.Contains(t, guidance, "<memory>")
|
|
require.Contains(t, guidance, `project "platform"`)
|
|
require.Contains(t, guidance, "people on this project")
|
|
require.Contains(t, guidance, "<project-memory-index>")
|
|
require.Contains(t, guidance, chattool.ConsolidateMemoryToolName)
|
|
}
|
|
|
|
func TestReadMemoryDescriptionIsFixed(t *testing.T) {
|
|
t.Parallel()
|
|
// Tool definitions lead the provider's cached prefix, so memory state
|
|
// must never reach them.
|
|
a := chattool.ReadMemory(newMemoryStore(0)).Info().Description
|
|
b := chattool.ReadMemory(newMemoryStore(chattool.MaxMemories)).Info().Description
|
|
require.Equal(t, a, b)
|
|
}
|
|
|
|
func TestReadMemory(t *testing.T) {
|
|
t.Parallel()
|
|
store := newMemoryStore(1)
|
|
_, result := runMemoryTool(t, chattool.ReadMemory(store), `{"name": "seeded-000"}`)
|
|
require.Equal(t, "seeded-000", result["name"])
|
|
|
|
response, _ := runMemoryTool(t, chattool.ReadMemory(store), `{"name": "missing"}`)
|
|
require.True(t, response.IsError)
|
|
require.Equal(t, "memory was not found", response.Content)
|
|
|
|
// Other failures must not tell the model the memory is gone.
|
|
store.getErr = xerrors.New("connection reset")
|
|
response, _ = runMemoryTool(t, chattool.ReadMemory(store), `{"name": "seeded-000"}`)
|
|
require.True(t, response.IsError)
|
|
require.Equal(t, "failed to read memory", response.Content)
|
|
}
|
|
|
|
func TestMemoryIndexReplay(t *testing.T) {
|
|
t.Parallel()
|
|
entry := func(name, description string) chattool.MemoryIndexEntry {
|
|
return chattool.MemoryIndexEntry{Name: name, Description: description}
|
|
}
|
|
|
|
_, ok := chattool.ReplayMemoryIndex([]string{"unrelated", chattool.FormatMemoryIndexUpdate([]chattool.MemoryIndexEntry{entry("a", "A")}, nil)})
|
|
require.False(t, ok, "an update without a snapshot has no base")
|
|
|
|
snapshot := chattool.FormatMemoryIndexSnapshot([]chattool.MemoryIndexEntry{entry("alpha", "First: with a colon"), entry("beta", "Second")})
|
|
seen, ok := chattool.ReplayMemoryIndex([]string{snapshot})
|
|
require.True(t, ok)
|
|
require.Equal(t, map[string]string{"alpha": "First: with a colon", "beta": "Second"}, seen)
|
|
|
|
current := []chattool.MemoryIndexEntry{entry("alpha", "First, corrected"), entry("gamma", "Third")}
|
|
changed, removed := chattool.DiffMemoryIndex(seen, current)
|
|
require.Equal(t, []chattool.MemoryIndexEntry{entry("alpha", "First, corrected"), entry("gamma", "Third")}, changed)
|
|
require.Equal(t, []string{"beta"}, removed)
|
|
|
|
// Replaying the snapshot and the update reproduces the current index,
|
|
// so the next diff is empty.
|
|
seen, ok = chattool.ReplayMemoryIndex([]string{snapshot, "unrelated", chattool.FormatMemoryIndexUpdate(changed, removed)})
|
|
require.True(t, ok)
|
|
changed, removed = chattool.DiffMemoryIndex(seen, current)
|
|
require.Empty(t, changed)
|
|
require.Empty(t, removed)
|
|
|
|
// A later snapshot replaces everything before it.
|
|
seen, ok = chattool.ReplayMemoryIndex([]string{snapshot, chattool.FormatMemoryIndexSnapshot(nil)})
|
|
require.True(t, ok)
|
|
require.Empty(t, seen)
|
|
}
|
|
|
|
func TestSaveMemory(t *testing.T) {
|
|
t.Parallel()
|
|
const input = `{"name":"durable-fact","description":"Durable fact","body":"Body"}`
|
|
t.Run("Saves", func(t *testing.T) {
|
|
t.Parallel()
|
|
store := newMemoryStore(0)
|
|
response, result := runMemoryTool(t, chattool.SaveMemory(store, "platform"), `{"name":"DURABLE-fact","description":"Durable","body":"Body"}`)
|
|
require.False(t, response.IsError, response.Content)
|
|
require.Equal(t, "durable-fact", result["saved"])
|
|
require.EqualValues(t, 1, result["count"])
|
|
require.NotContains(t, result, "warning")
|
|
require.Contains(t, store.memories, "durable-fact")
|
|
})
|
|
t.Run("ExistingNameIsNotOverwritten", func(t *testing.T) {
|
|
t.Parallel()
|
|
store := newMemoryStore(0)
|
|
store.memories["durable-fact"] = chattool.Memory{Name: "durable-fact", Body: "original"}
|
|
response, _ := runMemoryTool(t, chattool.SaveMemory(store, "platform"), input)
|
|
require.True(t, response.IsError)
|
|
require.Contains(t, response.Content, "already exists")
|
|
require.Contains(t, response.Content, chattool.ConsolidateMemoryToolName)
|
|
require.Equal(t, "original", store.memories["durable-fact"].Body)
|
|
})
|
|
t.Run("NearCapAsksToConsolidate", func(t *testing.T) {
|
|
t.Parallel()
|
|
store := newMemoryStore(chattool.MemoryConsolidateThreshold - 1)
|
|
response, result := runMemoryTool(t, chattool.SaveMemory(store, "platform"), input)
|
|
require.False(t, response.IsError, response.Content)
|
|
require.EqualValues(t, chattool.MemoryConsolidateThreshold, result["count"])
|
|
require.Equal(t, fmt.Sprintf(
|
|
"memory is %d/%d, approaching the limit. Consolidate it to under %d memories now with %s: merge overlapping memories into one, and drop stale, wrong, or superseded ones.",
|
|
chattool.MemoryConsolidateThreshold, chattool.MaxMemories, chattool.MemoryConsolidateTarget, chattool.ConsolidateMemoryToolName,
|
|
), result["warning"])
|
|
})
|
|
t.Run("FullPointsAtConsolidate", func(t *testing.T) {
|
|
t.Parallel()
|
|
store := newMemoryStore(chattool.MaxMemories)
|
|
response, _ := runMemoryTool(t, chattool.SaveMemory(store, "platform"), input)
|
|
require.True(t, response.IsError)
|
|
require.Contains(t, response.Content, "memory is full (200/200)")
|
|
require.Contains(t, response.Content, chattool.ConsolidateMemoryToolName)
|
|
require.Len(t, store.memories, chattool.MaxMemories)
|
|
})
|
|
}
|
|
|
|
func TestConsolidateMemory(t *testing.T) {
|
|
t.Parallel()
|
|
t.Run("MergesAndMakesRoom", func(t *testing.T) {
|
|
t.Parallel()
|
|
store := newMemoryStore(chattool.MaxMemories)
|
|
response, result := runMemoryTool(t, chattool.ConsolidateMemory(store, "platform"), `{
|
|
"delete": ["seeded-000", "seeded-001", "seeded-002"],
|
|
"save": [
|
|
{"name": "merged", "description": "Merged fact", "body": "Merged body"},
|
|
{"name": "new-fact", "description": "New fact", "body": "New body"}
|
|
]
|
|
}`)
|
|
require.False(t, response.IsError, response.Content)
|
|
require.EqualValues(t, chattool.MaxMemories-1, result["count"])
|
|
require.Contains(t, result, "warning", "still over the threshold")
|
|
require.NotContains(t, store.memories, "seeded-000")
|
|
require.Contains(t, store.memories, "merged")
|
|
require.Contains(t, store.memories, "new-fact")
|
|
})
|
|
t.Run("ReplacesByName", func(t *testing.T) {
|
|
t.Parallel()
|
|
store := newMemoryStore(1)
|
|
response, result := runMemoryTool(t, chattool.ConsolidateMemory(store, "platform"), `{
|
|
"delete": ["seeded-000"],
|
|
"save": [{"name": "seeded-000", "description": "Corrected", "body": "Corrected body"}]
|
|
}`)
|
|
require.False(t, response.IsError, response.Content)
|
|
require.EqualValues(t, 1, result["count"])
|
|
require.NotContains(t, result, "warning")
|
|
require.Equal(t, "Corrected body", store.memories["seeded-000"].Body)
|
|
})
|
|
t.Run("InvalidInputAppliesNothing", func(t *testing.T) {
|
|
t.Parallel()
|
|
for _, input := range []string{
|
|
`{}`,
|
|
`{"delete": ["Not Valid"]}`,
|
|
`{"delete": ["seeded-000"], "save": [{"name": "ok", "description": "", "body": "body"}]}`,
|
|
} {
|
|
store := newMemoryStore(1)
|
|
response, _ := runMemoryTool(t, chattool.ConsolidateMemory(store, "platform"), input)
|
|
require.True(t, response.IsError, input)
|
|
require.Contains(t, store.memories, "seeded-000", input)
|
|
}
|
|
})
|
|
}
|
|
|
|
// TestProjectMemoryStoreConsolidateIsAtomic runs against Postgres because the
|
|
// guarantee under test is the transaction rollback. It also checks that only
|
|
// committed changes are audited, one entry per memory created or deleted.
|
|
func TestProjectMemoryStoreConsolidateIsAtomic(t *testing.T) {
|
|
t.Parallel()
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
db, _ := dbtestutil.NewDB(t)
|
|
org := dbgen.Organization(t, db, database.Organization{})
|
|
user := dbgen.User(t, db, database.User{})
|
|
project := dbgen.ChatProject(t, db, database.ChatProject{OrganizationID: org.ID, OwnerID: user.ID})
|
|
var audited []string
|
|
store := chattool.NewProjectMemoryStore(db, project.ID, org.ID, user.ID, func(_ context.Context, action database.AuditAction, oldMemory, newMemory database.ChatProjectMemory) {
|
|
switch action {
|
|
case database.AuditActionCreate:
|
|
require.Equal(t, uuid.Nil, oldMemory.ID)
|
|
audited = append(audited, "create "+newMemory.Name)
|
|
case database.AuditActionDelete:
|
|
require.Equal(t, uuid.Nil, newMemory.ID)
|
|
audited = append(audited, "delete "+oldMemory.Name)
|
|
default:
|
|
t.Fatalf("unexpected audit action %q", action)
|
|
}
|
|
})
|
|
takeAudited := func() []string {
|
|
out := audited
|
|
audited = nil
|
|
return out
|
|
}
|
|
names := func() []string {
|
|
entries, err := store.List(ctx)
|
|
require.NoError(t, err)
|
|
out := make([]string, len(entries))
|
|
for i, entry := range entries {
|
|
out[i] = entry.Name
|
|
}
|
|
return out
|
|
}
|
|
input := func(name string) chattool.MemoryInput {
|
|
return chattool.MemoryInput{Name: name, Description: name + " description", Body: name + " body"}
|
|
}
|
|
|
|
for _, name := range []string{"alpha", "beta"} {
|
|
_, err := store.Insert(ctx, input(name))
|
|
require.NoError(t, err)
|
|
}
|
|
_, err := store.Insert(ctx, input("alpha"))
|
|
require.ErrorIs(t, err, chattool.ErrMemoryExists)
|
|
require.Equal(t, []string{"create alpha", "create beta"}, takeAudited())
|
|
|
|
// A missing delete rolls back the deletes and saves before it.
|
|
_, err = store.Consolidate(ctx, []string{"alpha", "missing"}, []chattool.MemoryInput{input("gamma")})
|
|
require.ErrorIs(t, err, chattool.ErrMemoryNotFound)
|
|
require.Equal(t, []string{"alpha", "beta"}, names())
|
|
|
|
// A duplicate save rolls back the deletes before it.
|
|
_, err = store.Consolidate(ctx, []string{"alpha"}, []chattool.MemoryInput{input("gamma"), input("gamma")})
|
|
require.ErrorIs(t, err, chattool.ErrMemoryExists)
|
|
require.Equal(t, []string{"alpha", "beta"}, names())
|
|
require.Empty(t, takeAudited(), "rolled-back consolidations are not audited")
|
|
|
|
// Deletes run first, so a save may reuse a deleted name.
|
|
count, err := store.Consolidate(ctx, []string{"alpha", "beta"}, []chattool.MemoryInput{input("alpha"), input("merged")})
|
|
require.NoError(t, err)
|
|
require.EqualValues(t, 2, count)
|
|
require.Equal(t, []string{"alpha", "merged"}, names())
|
|
require.Equal(t, []string{"delete alpha", "delete beta", "create alpha", "create merged"}, takeAudited())
|
|
|
|
// A result over the cap rolls back.
|
|
saves := make([]chattool.MemoryInput, chattool.MaxMemories-1)
|
|
for i := range saves {
|
|
saves[i] = input(fmt.Sprintf("fill-%03d", i))
|
|
}
|
|
_, err = store.Consolidate(ctx, nil, saves)
|
|
require.ErrorIs(t, err, chattool.ErrMemoryLimit)
|
|
require.Equal(t, []string{"alpha", "merged"}, names())
|
|
require.Empty(t, takeAudited())
|
|
count, err = store.Consolidate(ctx, []string{"merged"}, saves)
|
|
require.NoError(t, err)
|
|
require.EqualValues(t, chattool.MaxMemories, count)
|
|
require.Len(t, takeAudited(), 1+len(saves))
|
|
|
|
require.NoError(t, store.Delete(ctx, "alpha"))
|
|
require.ErrorIs(t, store.Delete(ctx, "alpha"), chattool.ErrMemoryNotFound)
|
|
require.Equal(t, []string{"delete alpha"}, takeAudited())
|
|
}
|