Report template insights session usage per app instead of five fixed
family rows, with registry names and icons.
- Add per-app rows for reported session apps (VS Code, Cursor, GoLand,
Zed, etc.); unused apps are omitted
- Replace `type` (`builtin`/`app`) with `method` (`ssh`,
`reconnecting_pty`, `port_forwarding`, `workspace_app`); session
methods are derived from the app name until #31012 stores them
- Add `usage_total_seconds` to count each user's active minutes once,
and use it as the UI bar denominator so overlapping apps can exceed
100%
- Merge template apps sharing a slug into one row across templates,
using the newest display name and icon
- Rename Web Terminal slug from `reconnecting-pty` to
`reconnecting_pty`
- Cap the response at the 64 busiest unregistered apps and fold the
rest into "Other apps", flagged with `seconds_is_upper_bound`
("Up to" in the UI)
- Add `unknown` and `overflow` accounting rows with explanatory
tooltips, and a fallback icon for apps without one
- Remove the unused `codersdk.UnionByFamily` helper from the Go SDK
BREAKING CHANGE: `report.apps_usage` in `GET /api/v2/insights/templates`
no longer returns the fixed five rows, family slugs, or previous display
names. Consumers must update.
Part of CLIENT-642
Prometheus metrics handled separately in #29906
Track individual applications across deployment statistics, Prometheus, telemetry, and the dashboard instead of exposing only fixed family totals. Preserve existing fields and gauges so current consumers can migrate without breaking.
- Add per-app counts with normalized identifiers and registry-backed display names, families, and bundled icons. Unrecognized apps retain their identifiers.
- Limit each statistics report to 64 distinct app names, combining excess names into an explicit overflow bucket.
- Group dashboard connections by family, including an Other category, with alphabetical per-app tooltips and distinct loading and empty states.
- Export app-keyed Prometheus counts separately from current family metadata, allowing queries to enrich historical counts without rewriting stored samples.
- Document family queries with unknown fallback, deployment isolation, and replica handling, alongside collection behavior and migration guidance.
- Include per-app counts in telemetry, update API schemas and documentation, and add regression coverage across collection, metrics, and dashboard behavior.
Closes#27411
- Make all six raw session-count queries return a session_counts JSONB
object keyed by app name instead of fixed per-family columns. Fold the
results into existing family fields in Go using the shared SDK registry,
including unrecognized names under unknown.
- Remove the registry parameters and associated dbauthz validation from
these queries, which no longer perform family attribution in SQL.
Preserve existing deployment stats, Prometheus, and telemetry outputs.
- Select session maps directly in the two latest-row queries instead of
decomposing and reaggregating a single row. Keep byte, connection, and
latency aggregates independent of per-app expansion.
- Add regressions for latest eligible rows, empty session maps, distinct
same-family app names, labels, and unchanged traffic aggregates.
- Centralize app-to-family attribution in the SDK registry so recognized
app aliases contribute to their family's session counts and usage
minutes without duplicating name lists across SQL queries.
- Pass the registry into agent stats queries, insights, and usage rollups.
- Validate its structure in dbauthz so missing or renamed families fail
explicitly instead of silently producing zero counts.
- Decompose session maps once per row for per-agent sums while retaining
the key-presence checks used by insights. Update reporting consumers
and add coverage for alias attribution and registry validation.
Replace fixed session_count_* columns with a sparse session_counts
JSONB object so new session types require no schema changes.
Convert retained and unrolled-up activity, warning on backlogs spanning
more than 24 hours. Update batching, queries, and rollups while preserving
existing API shapes and compatibility with older agents.
Add migration, query, and batcher coverage.
Fixes all our Go file imports to match the preferred spec that we've _mostly_ been using. For example:
```
import (
"context"
"time"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/xerrors"
"gopkg.in/natefinch/lumberjack.v2"
"cdr.dev/slog/v3"
"github.com/coder/coder/v2/codersdk/agentsdk"
"github.com/coder/serpent"
)
```
3 groups: standard library, 3rd partly libs, Coder libs.
This PR makes the change across the codebase. The PR in the stack above modifies our formatting to maintain this state of affairs, and is a separate PR so it's possible to review that one in detail.
Upgrades to slog v3 which includes a small, but backward incompatible API change to the acceptible call arguments when logging. This change allows us to verify via compile time type checking that arguments are correct and won't cause a panic, as was possible in slog v1, which this replaces (v2 was tagged but never used in coder/coder).
It also updates dependencies that also use slog and were updated.
I've left the `aibridge` dependency as a commit SHA, under the assumption that the team there (cc @pawbana @dannykopping ) will tag and update the dependency soon and on their own schedule.
Other dependencies, I pushed new tags.
The metrics cache to calculate and expose build time metrics for
templates currently calls `GetTemplates`, which returns all templates
even if they are deleted. We can use the `GetTemplatesWithFilter` query
to easily filter out deleted templates from the results, and thus not
call `GetTemplateAverageBuildTime` for those deleted templates. Delete
time for workspaces for non-deleted templates is still calculated.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Closes https://github.com/coder/internal/issues/961
Likely the same deal as in #19599, the body of `require.Eventually` now fires immediately, when it used to fire after 250ms (the interval). Presumably, the deployment stats become ready before the vs code session count gets incremented. This was never an issue with the 250ms delay, as this flake has only cropped up after the testify version bump.
We'll fix the issue by making it possible to wait for a full metrics cache refresh, i.e. removing `require.Eventually` in this test altogether.
This PR should resolve https://github.com/coder/internal/issues/719 by
limiting the `workspace_builds` rows selected by the query to the most
recent 100 builds of a template, as opposed to all builds in the last
30d. For our own internal templates with the most builds (1700-2000 in a
30d period) this should cut the query execution time by about 80%.
Unless we have some restriction on keeping the 30d period, contract
related or otherwise, this seems like a safe change to make. In addition
to the execution speed improvements it also means the memory for the
query is bounded as well.
If we want to keep a 30d time period for the avg build time value I
think it's worth exploring a purpose built solution such as histogram
structures where the build times could be bucketized by template ID as
they're observed.
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
metricscache_test has been running tests against dbmem only, instead of
against postgres. Unfortunately the implementations of
GetTemplateAverageBuildTime have diverged between dbmem and postgres.
This change gets the tests working on Postgres and test for the
behaviour postgres provides.
Refactors our use of `slogtest` to instantiate a "standard logger" across most of our tests. This standard logger incorporates https://github.com/coder/slog/pull/217 to also ignore database query canceled errors by default, which are a source of low-severity flakes.
Any test that has set non-default `slogtest.Options` is left alone. In particular, `coderdtest` defaults to ignoring all errors. We might consider revisiting that decision now that we have better tools to target the really common flaky Error logs on shutdown.
Joins in fields like `username`, `avatar_url`, `organization_name`,
`template_name` to `workspaces` via a **view**.
The view must be maintained moving forward, but this prevents needing to
add RBAC permissions to fetch related workspace fields.
* chore: remove InsertWorkspaceAgentStat query
InsertWorkspaceAgentStats (batch) exists. We only used the singular in
a single unit test place. Removing the single for the batch, reducing
the interface size.
* fix: strip timezone information from a date in dau response
Timezone information is lost, so do not forward it to the client.
* fix: timezone offset should be flipped
* Make tests deterministic
* chore(Makefile): use golangci-lint version from dogfood Dockerfile
* chore(dogfood/Dockerfile): update golangci-lint to latest version
* chore(coderd): address linter complaints
* chore: add /v2 to import module path
go mod requires semantic versioning with versions greater than 1.x
This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```
Migrate generated files to import /v2
* Fix gen
* fix: don't make session counts cumulative
This made for some weird tracking... we want the point-in-time
number of counts!
* Add databasefake query for getting agent stats
* Add deployment stats endpoint
* The query... works?!?
* Fix aggregation query
* Select from multiple tables instead
* Fix continuous stats
* Increase period of stat refreshes
* Add workspace counts to deployment stats
* fmt
* Add a slight bit of responsiveness
* Fix template version editor overflow
* Add refresh button
* Fix font family on button
* Fix latest stat being reported
* Revert agent conn stats
* Fix linting error
* Fix tests
* Fix gen
* Fix migrations
* Block on sending stat updates
* Add test fixtures
* Fix response structure
* make gen
* chore: convert workspace agent stats from json to table
* chore: convert agent stats to use a table
Backwards compatibility becomes hard when all agent stats are in a JSON blob.
We also want to query this table for new agents that are failing health checks
so we can display it in the UI.
* Fix migration using default values
- rbac: export rbac.Permissions
- dbauthz: move GetDeploymentDAUs, GetTemplateDAUs,
GetTemplateAverageBuildTime from querier.go to system.go
and removes auth checks
- dbauthz: remove AsSystem(), add individual roles for
autostart, provisionerd, add restricted system role for
everything else
feat: Add initial AuthzQuerier implementation
- Adds package database/dbauthz that adds a database.Store implementation where each method goes through AuthZ checks
- Implements all database.Store methods on AuthzQuerier
- Updates and fixes unit tests where required
- Updates coderd initialization to use AuthzQuerier if codersdk.ExperimentAuthzQuerier is enabled
* chore: Use dbgen in unit test
- organizationparam_test
- templateparam_test
* Use dbgen in all unit tests vs insert methods
* fixup! Use dbgen in all unit tests vs insert methods
---------
Co-authored-by: Cian Johnston <cian@coder.com>
* chore: Update sqlc to v1.16.0
* chore: Fix cases where types became Null-types
* chore: Set parameter_schemas default_destination_scheme and default_source_scheme to NOT NULL
* chore: Add enum validation to database fake
* chore: Fix all tests that skipping enum values
* fix: Use correct err in providionerdserver audit log failure log