Extends #30882's `model_invocable` semantics to workspace skills and to stored skills created through the API. Claude Code and Xum let skill authors set `disable-model-invocation: true` in SKILL.md frontmatter, so agents never pick the skill on their own and it runs only when a person invokes it. Coder Agents honored the key only when someone imported a SKILL.md file while adding a project skill (#30884). ## What changes - **Workspace skills:** when the pinned SKILL.md sets the key to the YAML boolean `true`, chatd leaves the skill out of `<available-skills>` before merging sources, so a listed skill with the same name keeps its bare name, and `read_skill` reports it as not found. The skill stays in the `/` menu, and invoking it still loads its body. `read_skill_file` serves its supporting files under the bare name or `workspace/<name>` and refuses its `SKILL.md` (case-insensitive, after `path.Clean`). Skills without the key behave as on the base. - **Invoked workspace skills:** invoking a workspace skill now stores the pinned skill directory on the invocation part, and the prompt renders it as `dir` on `<invoked-skill>`, escaped as an attribute value. Before, only `read_skill` gave the model a skill's directory, and it refuses a hidden skill, so a hidden skill could not run the scripts it bundles. The part also stores the chat's workspace, and the prompt shows `dir` only while the chat stays bound to it, as it does for workspace file references. This changes #30861's invocation code. Hook payloads drop the directory and its workspace like the body, and API responses strip both. - **Stored skills:** a create request whose content sets the key and that omits `model_invocable` stores `model_invocable = false`. An explicit value wins, and PATCH is unchanged. This covers personal, organization, and project skills, which share `createSkill`. - `workspacesdk.SkillDisablesModelInvocation` reads the key. The `CreateSkillRequest.ModelInvocable` doc comment, the generated API docs, and `extending-agents.md` (the SKILL.md format, a new "Turn off automatic use" section, and the notes on the skill index, the skill tools, and aliases) describe the new behavior. ## Follow-ups - The personal and organization skill editors have no "Agents can use it automatically" switch yet. Only the project editor from #30884 has one. ## Testing - Go tests cover the parser, the turn's skill index and aliases, `read_skill`, `read_skill_file`, invocation of a hidden skill and the directory it carries (attribute escaping, no directory after the chat changes workspace, no directory in hook payloads), and the create default for personal and organization skills. Each new test fails when its code path is removed. - Remote dogfood UAT with a real model failed rounds 1 to 3. Rounds 1 and 2 found the hidden SKILL.md still readable through `read_skill_file` by symlinks and stale pins, plus a bare-name lookup bug and a docs gap that round 2 fixed. Round 3 found that the content check added against those bypasses refused a listed skill's file whose own frontmatter sets the key. After round 3 we narrowed the rule to the hidden skill's `SKILL.md` path. The key decides what agents are offered (the index and `read_skill`), not which files they can read: `read_file` and `execute` already open any workspace file, so a content check in `read_skill_file` protected nothing and kept producing edge cases. - Round 4 passed on `c9ff25dc29f`. Later commits added a behavior-neutral cleanup (`cd617533c2e`), one test case from the test audit (`044c1e6e23a`), and a comment fix (`9ec3768b18b`), which passed validation without another UAT round. Round 5 passed on `2b518c6c2b1`, which adds the skill directory. A real model ran hidden skills' scripts from the `dir` on `<invoked-skill>`, including directories whose paths contain `&`, `'`, `"`, `<` and `>`, and API responses carried no directory. Merging main after #30884 landed (`2d7d3cb9674`) did not change this PR's diff, so UAT was not rerun. `8f043b2a342`, which shows the directory only while the chat stays bound to the skill's workspace, has Go tests but no UAT round. - The doc-check skill ran in author self-check mode on `2d7d3cb9674`, and `27de7f5c6d7` fixes its findings: the skill index sentence said every skill is listed, the stored-skill sentence claimed any project editor import honors the key and presented the API default as fixed, the API path came before the UI path, and some wording broke the docs style rules. The fix is docs-only, so UAT was not rerun. - Polish and the test audit stayed inside the implementation diff. No new dependencies. - Codex review found no major issues on `9ec3768b18b`, then raised one P2 on the same head when the PR was marked ready: an invoked hidden skill gets no skill directory. A probe confirmed it, and `2b518c6c2b1` fixes it. Its earlier finding on that head (honor `CODER_AGENT_EXP_SKILL_META_FILE`) was refuted, because the pinned skill snapshot always reads `SKILL.md`. Both threads are resolved, and Codex found no major issues on `2b518c6c2b1` or on the merge `2d7d3cb9674`. - On `27de7f5c6d7`, Codex raised two P2s. One was confirmed: an invoked skill's directory stayed in the prompt after the chat moved to another workspace, which a prompt test showed on that head, and `8f043b2a342` fixes it. The other was declined: on Windows, `scripts\..\SKILL.md` gets past the `SKILL.md` check in `read_skill_file`, but that check is not an access boundary (`read_file` opens any path), and Windows has more aliases (`SKILL.md.`, `SKILL.md::$DATA`, `SKILL~1.MD`) that normalizing backslashes would not stop. On `8f043b2a342`, Codex asked to bind the directory to the workspace build instead. That was declined: every workspace start is a new build with new agent IDs, so the directory would drop after each stop and start, and the pinned body and the live files can diverge without a rebuild anyway, as they do for `read_skill`. A later request on that head asked to limit a hidden skill's `read_skill_file` access to invocations in the current workspace. That was declined too: `read_skill_file` resolves names against the current pin for listed skills on main as well, and the key decides which skills agents are offered, not which files they can read. All four threads are resolved. `a697dcd927f` adds the `format:"uuid"` tag that `TestSDKFieldsFormatted` requires on the new field, and Codex found no major issues on `a697dcd927f`. `ae70b2c533f` is docs-only: it applies the doc-check suggestion to say that an invoked skill's directory is included only while the chat stays in that workspace. The author doc-check self-check found no blocking findings on it, and Codex found no major issues on `ae70b2c533f`. > Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description. --------- Co-authored-by: Jaayden Halko <jaayden.halko@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
802 lines
30 KiB
Go
802 lines
30 KiB
Go
package coderd_test
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"encoding/json"
|
|
"fmt"
|
|
"net/http"
|
|
"strings"
|
|
"testing"
|
|
|
|
"github.com/google/uuid"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
|
|
"github.com/coder/coder/v2/coderd/audit"
|
|
"github.com/coder/coder/v2/coderd/coderdtest"
|
|
"github.com/coder/coder/v2/coderd/database"
|
|
"github.com/coder/coder/v2/coderd/database/dbauthz"
|
|
"github.com/coder/coder/v2/coderd/database/dbgen"
|
|
"github.com/coder/coder/v2/coderd/database/dbtestutil"
|
|
"github.com/coder/coder/v2/coderd/rbac"
|
|
"github.com/coder/coder/v2/coderd/util/ptr"
|
|
"github.com/coder/coder/v2/coderd/x/skills"
|
|
"github.com/coder/coder/v2/codersdk"
|
|
"github.com/coder/coder/v2/testutil"
|
|
)
|
|
|
|
// skillOwnerAPI drives the personal or organization skill routes through the
|
|
// same calls, so behavior both owners share is asserted for each of them.
|
|
type skillOwnerAPI struct {
|
|
path string
|
|
resourceType database.ResourceType
|
|
auditOrgID uuid.UUID
|
|
client *codersdk.ExperimentalClient
|
|
create func(context.Context, codersdk.CreateSkillRequest) (codersdk.Skill, error)
|
|
list func(context.Context) ([]codersdk.SkillMetadata, error)
|
|
get func(context.Context, string) (codersdk.Skill, error)
|
|
update func(context.Context, string, codersdk.UpdateSkillRequest) (codersdk.Skill, error)
|
|
delete func(context.Context, string) error
|
|
}
|
|
|
|
func personalSkillAPI(client *codersdk.ExperimentalClient) skillOwnerAPI {
|
|
return skillOwnerAPI{
|
|
path: "/api/experimental/users/me/skills",
|
|
resourceType: database.ResourceTypeUserSkill,
|
|
client: client,
|
|
create: func(ctx context.Context, req codersdk.CreateSkillRequest) (codersdk.Skill, error) {
|
|
return client.CreateUserSkill(ctx, codersdk.Me, req)
|
|
},
|
|
list: func(ctx context.Context) ([]codersdk.SkillMetadata, error) {
|
|
return client.UserSkills(ctx, codersdk.Me)
|
|
},
|
|
get: func(ctx context.Context, name string) (codersdk.Skill, error) {
|
|
return client.UserSkillByName(ctx, codersdk.Me, name)
|
|
},
|
|
update: func(ctx context.Context, name string, req codersdk.UpdateSkillRequest) (codersdk.Skill, error) {
|
|
return client.UpdateUserSkill(ctx, codersdk.Me, name, req)
|
|
},
|
|
delete: func(ctx context.Context, name string) error { return client.DeleteUserSkill(ctx, codersdk.Me, name) },
|
|
}
|
|
}
|
|
|
|
func organizationSkillAPI(client *codersdk.ExperimentalClient, orgID uuid.UUID) skillOwnerAPI {
|
|
return skillOwnerAPI{
|
|
path: fmt.Sprintf("/api/experimental/organizations/%s/skills", orgID),
|
|
resourceType: database.ResourceTypeOrganizationSkill,
|
|
auditOrgID: orgID,
|
|
client: client,
|
|
create: func(ctx context.Context, req codersdk.CreateSkillRequest) (codersdk.Skill, error) {
|
|
return client.CreateOrganizationSkill(ctx, orgID, req)
|
|
},
|
|
list: func(ctx context.Context) ([]codersdk.SkillMetadata, error) {
|
|
return client.OrganizationSkills(ctx, orgID)
|
|
},
|
|
get: func(ctx context.Context, name string) (codersdk.Skill, error) {
|
|
return client.OrganizationSkillByName(ctx, orgID, name)
|
|
},
|
|
update: func(ctx context.Context, name string, req codersdk.UpdateSkillRequest) (codersdk.Skill, error) {
|
|
return client.UpdateOrganizationSkill(ctx, orgID, name, req)
|
|
},
|
|
delete: func(ctx context.Context, name string) error { return client.DeleteOrganizationSkill(ctx, orgID, name) },
|
|
}
|
|
}
|
|
|
|
// skillOwnerCases each start a deployment and return a caller that manages
|
|
// one owner's skills, plus a peer that manages another owner of the same kind.
|
|
var skillOwnerCases = []struct {
|
|
name string
|
|
setup func(t *testing.T, opts *coderdtest.Options) (owner, peer skillOwnerAPI)
|
|
}{
|
|
{
|
|
name: "Personal",
|
|
setup: func(t *testing.T, opts *coderdtest.Options) (skillOwnerAPI, skillOwnerAPI) {
|
|
adminClient := coderdtest.New(t, opts)
|
|
firstUser := coderdtest.CreateFirstUser(t, adminClient)
|
|
ownerClient, _ := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
peerClient, _ := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
return personalSkillAPI(codersdk.NewExperimentalClient(ownerClient)), personalSkillAPI(codersdk.NewExperimentalClient(peerClient))
|
|
},
|
|
},
|
|
{
|
|
name: "OrgAdmin",
|
|
setup: func(t *testing.T, opts *coderdtest.Options) (skillOwnerAPI, skillOwnerAPI) {
|
|
ownerClient, db := coderdtest.NewWithDatabase(t, opts)
|
|
firstUser := coderdtest.CreateFirstUser(t, ownerClient)
|
|
orgAdminClient, _ := coderdtest.CreateAnotherUser(t, ownerClient, firstUser.OrganizationID, rbac.ScopedRoleOrgAdmin(firstUser.OrganizationID))
|
|
peerOrg := dbgen.Organization(t, db, database.Organization{})
|
|
return organizationSkillAPI(codersdk.NewExperimentalClient(orgAdminClient), firstUser.OrganizationID),
|
|
organizationSkillAPI(codersdk.NewExperimentalClient(ownerClient), peerOrg.ID)
|
|
},
|
|
},
|
|
{
|
|
name: "SiteOwner",
|
|
setup: func(t *testing.T, opts *coderdtest.Options) (skillOwnerAPI, skillOwnerAPI) {
|
|
ownerClient, db := coderdtest.NewWithDatabase(t, opts)
|
|
firstUser := coderdtest.CreateFirstUser(t, ownerClient)
|
|
client := codersdk.NewExperimentalClient(ownerClient)
|
|
peerOrg := dbgen.Organization(t, db, database.Organization{})
|
|
return organizationSkillAPI(client, firstUser.OrganizationID), organizationSkillAPI(client, peerOrg.ID)
|
|
},
|
|
},
|
|
}
|
|
|
|
func TestSkillsCRUD(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
for _, tc := range skillOwnerCases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
owner, _ := tc.setup(t, nil)
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
|
|
emptyList, err := owner.list(ctx)
|
|
require.NoError(t, err)
|
|
assert.NotNil(t, emptyList)
|
|
assert.Empty(t, emptyList)
|
|
rawEmptyList := requireRawSkillList(ctx, t, owner)
|
|
assert.NotNil(t, rawEmptyList)
|
|
assert.Empty(t, rawEmptyList)
|
|
|
|
content := userSkillMarkdown("crud-skill", "Initial description", "Use this skill for CRUD tests.")
|
|
created, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: content})
|
|
require.NoError(t, err)
|
|
assert.NotZero(t, created.ID)
|
|
assert.Equal(t, "crud-skill", created.Name)
|
|
assert.Equal(t, "Initial description", created.Description)
|
|
assert.Equal(t, content, created.Content)
|
|
assert.True(t, created.Enabled)
|
|
assert.True(t, created.ModelInvocable)
|
|
assert.False(t, created.Action)
|
|
assert.NotZero(t, created.CreatedAt)
|
|
assert.NotZero(t, created.UpdatedAt)
|
|
|
|
list, err := owner.list(ctx)
|
|
require.NoError(t, err)
|
|
require.Len(t, list, 1)
|
|
assert.Equal(t, created.ID, list[0].ID)
|
|
assert.Equal(t, "crud-skill", list[0].Name)
|
|
assert.Equal(t, "Initial description", list[0].Description)
|
|
assert.True(t, list[0].Enabled)
|
|
rawList := requireRawSkillList(ctx, t, owner)
|
|
require.Len(t, rawList, 1)
|
|
assert.NotContains(t, rawList[0], "content")
|
|
|
|
got, err := owner.get(ctx, "crud-skill")
|
|
require.NoError(t, err)
|
|
assert.Equal(t, created.ID, got.ID)
|
|
assert.Equal(t, content, got.Content)
|
|
|
|
updatedContent := userSkillMarkdown("crud-skill", "Updated description", "Updated body.")
|
|
updated, err := owner.update(ctx, "crud-skill", codersdk.UpdateSkillRequest{Content: ptr.Ref(updatedContent)})
|
|
require.NoError(t, err)
|
|
assert.Equal(t, created.ID, updated.ID)
|
|
assert.Equal(t, "Updated description", updated.Description)
|
|
assert.Equal(t, updatedContent, updated.Content)
|
|
assert.True(t, updated.Enabled)
|
|
|
|
disabled, err := owner.update(ctx, "crud-skill", codersdk.UpdateSkillRequest{Enabled: ptr.Ref(false)})
|
|
require.NoError(t, err)
|
|
assert.False(t, disabled.Enabled)
|
|
assert.Equal(t, updatedContent, disabled.Content)
|
|
assert.Equal(t, "Updated description", disabled.Description)
|
|
list, err = owner.list(ctx)
|
|
require.NoError(t, err)
|
|
require.Len(t, list, 1)
|
|
assert.False(t, list[0].Enabled)
|
|
got, err = owner.get(ctx, "crud-skill")
|
|
require.NoError(t, err)
|
|
assert.False(t, got.Enabled)
|
|
|
|
manual, err := owner.update(ctx, "crud-skill", codersdk.UpdateSkillRequest{ModelInvocable: ptr.Ref(false)})
|
|
require.NoError(t, err)
|
|
assert.False(t, manual.ModelInvocable)
|
|
assert.False(t, manual.Enabled)
|
|
list, err = owner.list(ctx)
|
|
require.NoError(t, err)
|
|
require.Len(t, list, 1)
|
|
assert.False(t, list[0].ModelInvocable)
|
|
|
|
require.NoError(t, owner.delete(ctx, "crud-skill"))
|
|
_, err = owner.get(ctx, "crud-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
|
|
// The frontmatter key sets the default only on create; an
|
|
// explicit model_invocable wins, and updates keep the stored value.
|
|
manualOnlyContent := func(name, body string) string {
|
|
return fmt.Sprintf("---\nname: %s\ndescription: Manual only\ndisable-model-invocation: true\n---\n\n%s\n", name, body)
|
|
}
|
|
flagged, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: manualOnlyContent("flagged-skill", "Body.")})
|
|
require.NoError(t, err)
|
|
assert.False(t, flagged.ModelInvocable)
|
|
explicit, err := owner.create(ctx, codersdk.CreateSkillRequest{
|
|
Content: manualOnlyContent("explicit-skill", "Body."),
|
|
ModelInvocable: ptr.Ref(true),
|
|
})
|
|
require.NoError(t, err)
|
|
assert.True(t, explicit.ModelInvocable)
|
|
explicit, err = owner.update(ctx, "explicit-skill", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(manualOnlyContent("explicit-skill", "Updated body.")),
|
|
})
|
|
require.NoError(t, err)
|
|
assert.True(t, explicit.ModelInvocable)
|
|
})
|
|
}
|
|
}
|
|
|
|
func requireRawSkillList(ctx context.Context, t *testing.T, owner skillOwnerAPI) []map[string]json.RawMessage {
|
|
t.Helper()
|
|
res, err := owner.client.Request(ctx, http.MethodGet, owner.path, nil)
|
|
require.NoError(t, err)
|
|
defer res.Body.Close()
|
|
require.Equal(t, http.StatusOK, res.StatusCode)
|
|
var raw []map[string]json.RawMessage
|
|
require.NoError(t, json.NewDecoder(res.Body).Decode(&raw))
|
|
return raw
|
|
}
|
|
|
|
func TestSkillValidationAndConflicts(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
for _, tc := range skillOwnerCases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
owner, peer := tc.setup(t, nil)
|
|
tests := []struct {
|
|
name string
|
|
content string
|
|
expectedMessage string
|
|
}{
|
|
{
|
|
name: "MissingFrontmatterDelimiters",
|
|
content: "name: missing-frontmatter\n\nBody.",
|
|
expectedMessage: "Invalid skill content.",
|
|
},
|
|
{
|
|
name: "MissingName",
|
|
content: "---\n" +
|
|
"description: Missing name\n" +
|
|
"---\n\nBody.",
|
|
expectedMessage: "Invalid skill name.",
|
|
},
|
|
{
|
|
name: "NonKebabCaseName",
|
|
content: userSkillMarkdown("NotKebab", "Invalid", "Body."),
|
|
expectedMessage: "Invalid skill name.",
|
|
},
|
|
{
|
|
name: "NameTooLong",
|
|
content: userSkillMarkdown(strings.Repeat("a", skills.MaxPersonalSkillNameBytes+1), "Invalid", "Body."),
|
|
expectedMessage: "Invalid skill name.",
|
|
},
|
|
{
|
|
name: "EmptyBody",
|
|
content: userSkillMarkdown("empty-body", "Invalid", " \n"),
|
|
expectedMessage: "Skill body is required.",
|
|
},
|
|
{
|
|
name: "TooLarge",
|
|
content: strings.Repeat("a", skills.MaxPersonalSkillSizeBytes+1),
|
|
expectedMessage: "Skill content is too large.",
|
|
},
|
|
}
|
|
for _, tt := range tests {
|
|
t.Run(tt.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: tt.content})
|
|
sdkErr := requireSDKErrorStatus(t, err, http.StatusBadRequest)
|
|
assert.Equal(t, tt.expectedMessage, sdkErr.Message)
|
|
})
|
|
}
|
|
|
|
t.Run("PatchEmptyBody", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown("patch-validation", "Valid", "Body.")})
|
|
require.NoError(t, err)
|
|
_, err = owner.update(ctx, "patch-validation", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown("patch-validation", "Invalid", " \n")),
|
|
})
|
|
sdkErr := requireSDKErrorStatus(t, err, http.StatusBadRequest)
|
|
assert.Equal(t, "Skill body is required.", sdkErr.Message)
|
|
})
|
|
|
|
t.Run("PatchNoFields", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown("patch-no-fields", "Valid", "Body.")})
|
|
require.NoError(t, err)
|
|
_, err = owner.update(ctx, "patch-no-fields", codersdk.UpdateSkillRequest{})
|
|
sdkErr := requireSDKErrorStatus(t, err, http.StatusBadRequest)
|
|
assert.Equal(t, "No skill fields to update.", sdkErr.Message)
|
|
})
|
|
|
|
// Only project skills can be actions, so personal and
|
|
// organization routes reject the field even when it is false.
|
|
t.Run("ActionRejected", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
content := userSkillMarkdown("action-rejected", "Valid", "Body.")
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: content, Action: ptr.Ref(true)})
|
|
sdkErr := requireSDKErrorStatus(t, err, http.StatusBadRequest)
|
|
assert.Equal(t, "Only project skills can be actions.", sdkErr.Message)
|
|
_, err = owner.create(ctx, codersdk.CreateSkillRequest{Content: content, ModelInvocable: ptr.Ref(false)})
|
|
require.NoError(t, err)
|
|
for _, action := range []bool{true, false} {
|
|
_, err = owner.update(ctx, "action-rejected", codersdk.UpdateSkillRequest{Action: ptr.Ref(action)})
|
|
sdkErr = requireSDKErrorStatus(t, err, http.StatusBadRequest)
|
|
assert.Equal(t, "Only project skills can be actions.", sdkErr.Message)
|
|
}
|
|
})
|
|
|
|
t.Run("PatchNameMismatch", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown("old-name", "Old", "Body.")})
|
|
require.NoError(t, err)
|
|
_, err = owner.update(ctx, "old-name", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown("new-name", "New", "Body.")),
|
|
})
|
|
sdkErr := requireSDKErrorStatus(t, err, http.StatusBadRequest)
|
|
assert.Equal(t, "Skill name in path does not match frontmatter name.", sdkErr.Message)
|
|
assert.Equal(t, `path has "old-name", frontmatter has "new-name"`, sdkErr.Detail)
|
|
})
|
|
|
|
t.Run("MissingSkill", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
_, err := owner.get(ctx, "missing-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
_, err = owner.update(ctx, "missing-skill", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown("missing-skill", "Missing", "Body.")),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
err = owner.delete(ctx, "missing-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
})
|
|
|
|
t.Run("SameNameConflictsOnlyWithinOwner", func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
sharedContent := userSkillMarkdown("shared-skill", "Shared", "Shared body.")
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: sharedContent})
|
|
require.NoError(t, err)
|
|
_, err = owner.create(ctx, codersdk.CreateSkillRequest{Content: sharedContent})
|
|
sdkErr := requireSDKErrorStatus(t, err, http.StatusConflict)
|
|
assert.Equal(t, "A skill with that name already exists.", sdkErr.Message)
|
|
_, err = peer.create(ctx, codersdk.CreateSkillRequest{Content: sharedContent})
|
|
require.NoError(t, err)
|
|
})
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestUserSkillLimit(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
adminClient := coderdtest.New(t, nil)
|
|
firstUser := coderdtest.CreateFirstUser(t, adminClient)
|
|
ownerClient, _ := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
owner := codersdk.NewExperimentalClient(ownerClient)
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
|
|
for i := range skills.MaxPersonalSkillsPerUser {
|
|
name := fmt.Sprintf("limit-skill-%03d", i)
|
|
_, err := owner.CreateUserSkill(ctx, codersdk.Me, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown(name, "Limit", "Body."),
|
|
})
|
|
require.NoError(t, err)
|
|
}
|
|
|
|
_, err := owner.CreateUserSkill(ctx, codersdk.Me, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown("limit-skill-overflow", "Limit", "Body."),
|
|
})
|
|
sdkErr := requireSDKErrorStatus(t, err, http.StatusConflict)
|
|
assert.Equal(t, "Personal skill limit reached.", sdkErr.Message)
|
|
assert.Equal(t,
|
|
fmt.Sprintf("Each user can have at most %d personal skills.", skills.MaxPersonalSkillsPerUser),
|
|
sdkErr.Detail,
|
|
)
|
|
}
|
|
|
|
func TestUserSkillLimitConcurrentCreates(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
adminClient := coderdtest.New(t, nil)
|
|
firstUser := coderdtest.CreateFirstUser(t, adminClient)
|
|
ownerClient, _ := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
owner := codersdk.NewExperimentalClient(ownerClient)
|
|
ctx := testutil.Context(t, testutil.WaitLong)
|
|
|
|
for i := range skills.MaxPersonalSkillsPerUser - 1 {
|
|
name := fmt.Sprintf("concurrent-limit-skill-%03d", i)
|
|
_, err := owner.CreateUserSkill(ctx, codersdk.Me, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown(name, "Limit", "Body."),
|
|
})
|
|
require.NoError(t, err)
|
|
}
|
|
|
|
const attempts = 8
|
|
start := make(chan struct{})
|
|
results := make(chan error, attempts)
|
|
for i := range attempts {
|
|
go func() {
|
|
<-start
|
|
name := fmt.Sprintf("concurrent-limit-overflow-%03d", i)
|
|
_, err := owner.CreateUserSkill(ctx, codersdk.Me, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown(name, "Limit", "Body."),
|
|
})
|
|
results <- err
|
|
}()
|
|
}
|
|
close(start)
|
|
|
|
successes := 0
|
|
for range attempts {
|
|
err := <-results
|
|
if err == nil {
|
|
successes++
|
|
continue
|
|
}
|
|
requireSDKErrorStatus(t, err, http.StatusConflict)
|
|
}
|
|
assert.Equal(t, 1, successes)
|
|
|
|
list, err := owner.UserSkills(ctx, codersdk.Me)
|
|
require.NoError(t, err)
|
|
assert.Len(t, list, skills.MaxPersonalSkillsPerUser)
|
|
}
|
|
|
|
func TestUserSkillRequestAllowsEscapedMaxSizeContent(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
adminClient := coderdtest.New(t, nil)
|
|
firstUser := coderdtest.CreateFirstUser(t, adminClient)
|
|
ownerClient, _ := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
owner := codersdk.NewExperimentalClient(ownerClient)
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
|
|
prefix := "---\nname: escaped-limit-skill\ndescription: Escaped\n---\n\n"
|
|
suffix := "\n"
|
|
bodyLen := skills.MaxPersonalSkillSizeBytes - len(prefix) - len(suffix)
|
|
require.Positive(t, bodyLen)
|
|
content := prefix + strings.Repeat(`"`, bodyLen) + suffix
|
|
require.Len(t, []byte(content), skills.MaxPersonalSkillSizeBytes)
|
|
|
|
raw, err := json.Marshal(codersdk.CreateSkillRequest{Content: content})
|
|
require.NoError(t, err)
|
|
require.Greater(t, len(raw), skills.MaxPersonalSkillSizeBytes+1024)
|
|
|
|
created, err := owner.CreateUserSkill(ctx, codersdk.Me, codersdk.CreateSkillRequest{
|
|
Content: content,
|
|
})
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "escaped-limit-skill", created.Name)
|
|
}
|
|
|
|
func TestUserSkillAuthorization(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
adminClient := coderdtest.New(t, nil)
|
|
firstUser := coderdtest.CreateFirstUser(t, adminClient)
|
|
ownerClient, ownerUser := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
otherClient, _ := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
userAdminClient, _ := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID, rbac.RoleUserAdmin())
|
|
admin := codersdk.NewExperimentalClient(adminClient)
|
|
owner := codersdk.NewExperimentalClient(ownerClient)
|
|
other := codersdk.NewExperimentalClient(otherClient)
|
|
userAdmin := codersdk.NewExperimentalClient(userAdminClient)
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
targetUser := ownerUser.Username
|
|
|
|
_, err := owner.CreateUserSkill(ctx, codersdk.Me, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown("auth-skill", "Auth", "Body."),
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
_, err = other.UserSkills(ctx, targetUser)
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
_, err = other.UserSkillByName(ctx, targetUser, "auth-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
_, err = other.CreateUserSkill(ctx, targetUser, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown("denied-create", "Denied", "Body."),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
_, err = other.UpdateUserSkill(ctx, targetUser, "auth-skill", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown("auth-skill", "Denied", "Body.")),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
err = other.DeleteUserSkill(ctx, targetUser, "auth-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
|
|
_, err = userAdmin.UserSkills(ctx, targetUser)
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
_, err = userAdmin.UserSkillByName(ctx, targetUser, "auth-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
_, err = userAdmin.CreateUserSkill(ctx, targetUser, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown("denied-admin-create", "Denied", "Body."),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusForbidden)
|
|
_, err = userAdmin.UpdateUserSkill(ctx, targetUser, "auth-skill", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown("auth-skill", "Denied", "Body.")),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusForbidden)
|
|
err = userAdmin.DeleteUserSkill(ctx, targetUser, "auth-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
|
|
_, err = admin.CreateUserSkill(ctx, targetUser, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown("admin-created", "Admin create", "Created by admin."),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusForbidden)
|
|
|
|
list, err := admin.UserSkills(ctx, targetUser)
|
|
require.NoError(t, err)
|
|
require.Len(t, list, 1)
|
|
assert.Equal(t, "auth-skill", list[0].Name)
|
|
got, err := admin.UserSkillByName(ctx, targetUser, "auth-skill")
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "auth-skill", got.Name)
|
|
_, err = admin.UpdateUserSkill(ctx, targetUser, "auth-skill", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown("auth-skill", "Admin update", "Updated by admin.")),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusForbidden)
|
|
require.NoError(t, admin.DeleteUserSkill(ctx, targetUser, "auth-skill"))
|
|
}
|
|
|
|
func TestUserSkillSoftDeleteCleanup(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
adminClient, _, api := coderdtest.NewWithAPI(t, nil)
|
|
firstUser := coderdtest.CreateFirstUser(t, adminClient)
|
|
ownerClient, ownerUser := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
owner := codersdk.NewExperimentalClient(ownerClient)
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
|
|
_, err := owner.CreateUserSkill(ctx, codersdk.Me, codersdk.CreateSkillRequest{
|
|
Content: userSkillMarkdown("soft-delete-skill", "Soft delete", "Body."),
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
require.NoError(t, adminClient.DeleteUser(ctx, ownerUser.ID))
|
|
readAuthzCtx := dbauthz.AsSystemRestricted(ctx)
|
|
_, err = api.Database.GetUserSkillByUserIDAndName(
|
|
readAuthzCtx,
|
|
database.GetUserSkillByUserIDAndNameParams{
|
|
UserID: ownerUser.ID,
|
|
Name: "soft-delete-skill",
|
|
},
|
|
)
|
|
require.ErrorIs(t, err, sql.ErrNoRows)
|
|
|
|
createAuthzCtx := dbauthz.As(ctx, rbac.Subject{
|
|
Type: rbac.SubjectTypeUser,
|
|
ID: ownerUser.ID.String(),
|
|
Roles: rbac.RoleIdentifiers{rbac.RoleMember()},
|
|
Scope: rbac.ScopeAll,
|
|
}.WithCachedASTValue())
|
|
_, err = api.Database.InsertUserSkill(
|
|
createAuthzCtx,
|
|
database.InsertUserSkillParams{
|
|
ID: uuid.New(),
|
|
UserID: ownerUser.ID,
|
|
Name: "after-soft-delete",
|
|
Description: "Soft delete",
|
|
Content: userSkillMarkdown("after-soft-delete", "Soft delete", "Body."),
|
|
},
|
|
)
|
|
require.True(t, database.IsCheckViolation(err, database.CheckConstraint("user_skill_user_deleted")))
|
|
require.ErrorContains(t, err, "Cannot create user_skill for deleted user")
|
|
}
|
|
|
|
func TestUserSkillDatabaseConstraints(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
adminClient, _, api := coderdtest.NewWithAPI(t, nil)
|
|
firstUser := coderdtest.CreateFirstUser(t, adminClient)
|
|
_, ownerUser := coderdtest.CreateAnotherUser(t, adminClient, firstUser.OrganizationID)
|
|
tests := []struct {
|
|
name string
|
|
params database.InsertUserSkillParams
|
|
constraint database.CheckConstraint
|
|
}{
|
|
{
|
|
name: "NameFormat",
|
|
params: database.InsertUserSkillParams{
|
|
ID: uuid.New(),
|
|
UserID: ownerUser.ID,
|
|
Name: "not kebab",
|
|
Description: "Invalid",
|
|
Content: userSkillMarkdown("not kebab", "Invalid", "Body."),
|
|
},
|
|
constraint: database.CheckSkillsNameFormat,
|
|
},
|
|
{
|
|
name: "NameSize",
|
|
params: database.InsertUserSkillParams{
|
|
ID: uuid.New(),
|
|
UserID: ownerUser.ID,
|
|
Name: strings.Repeat("a", skills.MaxPersonalSkillNameBytes+1),
|
|
Description: "Invalid",
|
|
Content: userSkillMarkdown("too-long-name", "Invalid", "Body."),
|
|
},
|
|
constraint: database.CheckSkillsNameSize,
|
|
},
|
|
{
|
|
name: "ContentSize",
|
|
params: database.InsertUserSkillParams{
|
|
ID: uuid.New(),
|
|
UserID: ownerUser.ID,
|
|
Name: "content-too-large",
|
|
Description: "Invalid",
|
|
Content: strings.Repeat("a", skills.MaxPersonalSkillSizeBytes+1),
|
|
},
|
|
constraint: database.CheckSkillsContentSize,
|
|
},
|
|
}
|
|
for _, tt := range tests {
|
|
t.Run(tt.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
authzCtx := dbauthz.As(ctx, coderdtest.AuthzUserSubject(ownerUser))
|
|
|
|
_, err := api.Database.InsertUserSkill(authzCtx, tt.params)
|
|
require.True(t, database.IsCheckViolation(err, tt.constraint), "expected %s, got %v", tt.constraint, err)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestUserSkillSchemaConstants(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
_, _, sqlDB := dbtestutil.NewDBWithSQLDB(t)
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
var triggerDef string
|
|
require.NoError(t, sqlDB.QueryRowContext(
|
|
ctx,
|
|
`SELECT pg_get_functiondef('enforce_skills_per_owner_limit'::regproc)`,
|
|
).Scan(&triggerDef))
|
|
assert.Contains(t, triggerDef, fmt.Sprintf("skill_limit constant int := %d;", skills.MaxPersonalSkillsPerUser))
|
|
|
|
constraints := map[database.CheckConstraint]string{
|
|
database.CheckSkillsNameSize: fmt.Sprintf("octet_length(name) <= %d", skills.MaxPersonalSkillNameBytes),
|
|
database.CheckSkillsNameFormat: "name ~ '^[a-z0-9]+(-[a-z0-9]+)*$'::text",
|
|
database.CheckSkillsContentSize: fmt.Sprintf("octet_length(content) <= %d", skills.MaxPersonalSkillSizeBytes),
|
|
}
|
|
for constraint, expected := range constraints {
|
|
t.Run(string(constraint), func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
var constraintDef string
|
|
require.NoError(t, sqlDB.QueryRowContext(
|
|
ctx,
|
|
`SELECT pg_get_constraintdef(oid) FROM pg_constraint WHERE conname = $1`,
|
|
constraint,
|
|
).Scan(&constraintDef))
|
|
assert.Contains(t, constraintDef, expected)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestSkillAudit(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
for _, tc := range skillOwnerCases {
|
|
//nolint:paralleltest,tparallel // Subtests share one auditor and run sequentially.
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
auditor := audit.NewMock()
|
|
owner, _ := tc.setup(t, &coderdtest.Options{Auditor: auditor})
|
|
ctx := testutil.Context(t, testutil.WaitMedium)
|
|
auditor.ResetLogs()
|
|
|
|
genName := func(t *testing.T) string {
|
|
return strings.ToLower(strings.ReplaceAll(t.Name(), "/", "-"))
|
|
}
|
|
requireLog := func(t *testing.T, log database.AuditLog, action database.AuditAction, status int, skill codersdk.Skill) {
|
|
t.Helper()
|
|
assert.Equal(t, action, log.Action)
|
|
assert.EqualValues(t, status, log.StatusCode)
|
|
assert.Equal(t, owner.resourceType, log.ResourceType)
|
|
assert.Equal(t, owner.auditOrgID, log.OrganizationID)
|
|
assert.Equal(t, skill.ID, log.ResourceID)
|
|
assert.Equal(t, skill.Name, log.ResourceTarget)
|
|
}
|
|
|
|
t.Run("CreateEmitsLog", func(t *testing.T) {
|
|
auditor.ResetLogs()
|
|
skill, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown(genName(t), "Audit", "Body.")})
|
|
require.NoError(t, err)
|
|
|
|
logs := auditor.AuditLogs()
|
|
require.Len(t, logs, 1)
|
|
requireLog(t, logs[0], database.AuditActionCreate, http.StatusCreated, skill)
|
|
})
|
|
|
|
t.Run("UpdateEmitsLog", func(t *testing.T) {
|
|
name := genName(t)
|
|
skill, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown(name, "Initial", "Body.")})
|
|
require.NoError(t, err)
|
|
auditor.ResetLogs()
|
|
_, err = owner.update(ctx, name, codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown(name, "Updated", "Updated body.")),
|
|
})
|
|
require.NoError(t, err)
|
|
|
|
logs := auditor.AuditLogs()
|
|
require.Len(t, logs, 1)
|
|
requireLog(t, logs[0], database.AuditActionWrite, http.StatusOK, skill)
|
|
})
|
|
|
|
t.Run("DeleteEmitsLog", func(t *testing.T) {
|
|
name := genName(t)
|
|
skill, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown(name, "Delete", "Body.")})
|
|
require.NoError(t, err)
|
|
auditor.ResetLogs()
|
|
require.NoError(t, owner.delete(ctx, name))
|
|
|
|
logs := auditor.AuditLogs()
|
|
require.Len(t, logs, 1)
|
|
requireLog(t, logs[0], database.AuditActionDelete, http.StatusNoContent, skill)
|
|
})
|
|
|
|
t.Run("ReadsDoNotEmitLogs", func(t *testing.T) {
|
|
name := genName(t)
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown(name, "Read", "Body.")})
|
|
require.NoError(t, err)
|
|
auditor.ResetLogs()
|
|
|
|
_, err = owner.list(ctx)
|
|
require.NoError(t, err)
|
|
_, err = owner.get(ctx, name)
|
|
require.NoError(t, err)
|
|
assert.Empty(t, auditor.AuditLogs())
|
|
})
|
|
|
|
t.Run("ValidationFailureDoesNotEmitLog", func(t *testing.T) {
|
|
auditor.ResetLogs()
|
|
_, err := owner.create(ctx, codersdk.CreateSkillRequest{Content: userSkillMarkdown("bad-name", "Invalid", " \n")})
|
|
requireSDKErrorStatus(t, err, http.StatusBadRequest)
|
|
assert.Empty(t, auditor.AuditLogs())
|
|
})
|
|
|
|
t.Run("MissingSkillFailuresDoNotEmitLogs", func(t *testing.T) {
|
|
auditor.ResetLogs()
|
|
_, err := owner.update(ctx, "missing-audit-skill", codersdk.UpdateSkillRequest{
|
|
Content: ptr.Ref(userSkillMarkdown("missing-audit-skill", "Missing", "Body.")),
|
|
})
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
err = owner.delete(ctx, "missing-audit-skill")
|
|
requireSDKErrorStatus(t, err, http.StatusNotFound)
|
|
assert.Empty(t, auditor.AuditLogs())
|
|
})
|
|
})
|
|
}
|
|
}
|
|
|
|
func userSkillMarkdown(name string, description string, body string) string {
|
|
return fmt.Sprintf("---\nname: %s\ndescription: %s\n---\n\n%s\n", name, description, body)
|
|
}
|
|
|
|
func requireSDKErrorStatus(t *testing.T, err error, status int, msgAndArgs ...any) *codersdk.Error {
|
|
t.Helper()
|
|
require.Error(t, err, msgAndArgs...)
|
|
sdkErr := coderdtest.SDKError(t, err)
|
|
require.Equal(t, status, sdkErr.StatusCode(), msgAndArgs...)
|
|
return sdkErr
|
|
}
|