Speeds up `GetTemplateParameterInsights`, which the insights collector
runs every 5 minutes on every replica. The `@>` array join could only
hash on `name`, so every run scanned all of
`workspace_build_parameters`. Unnesting the build IDs and joining on
`(workspace_build_id, name)` lets the existing unique index serve it,
and the results don't change since each build ID shows up at most once
per group.
On 8.4M synthetic build parameters the collector's 5 minute window went
from 5,193 ms to about 31 ms. `TestGetTemplateParameterInsights` is new
and passes on both the old and new SQL. This also fixes the Prometheus
docs example, which summed per-replica copies of the same metric.
I split the `workspace_app_stats` index into #31766 so this one has no
migration and can be backported to 2.34 through 2.39 (2.34 to 2.38
without the docs hunk).
The pre-push hook skipped tests on this push because of a stale
`origin/main` in my checkout. An earlier full run only failed in
`agent/` and `site/`.
Part of [CORE-439](https://linear.app/codercom/issue/CORE-439).
The session banner now labels the VS Code family "VS Code Family", so it isn't confused with the VS Code editor. coderd_insights_applications_usage_seconds now reports built-in apps per app instead of per family.
- Generate the frontend's family labels from the Go table in codersdk.
- Add the connection_method label: workspace_app for template apps, and ssh, reconnecting_pty, or port_forwarding for built-in apps.
- Rename the application_name label to display_name. For built-in apps it holds the registry name, "Unknown" for unknown, and "Other apps" for overflow.
- Built-in apps get one series per app name, only for apps used in the window. The series limit and the fold rule are shared with the template insights API.
- Breaking: selectors with a value for application_name, or with slug="", match nothing. The Prometheus docs map each old series to its new one.
Closes#29940
Closes CLIENT-1034
Closes CLIENT-1036
Report template insights session usage per app instead of five fixed
family rows, with registry names and icons.
- Add per-app rows for reported session apps (VS Code, Cursor, GoLand,
Zed, etc.); unused apps are omitted
- Replace `type` (`builtin`/`app`) with `method` (`ssh`,
`reconnecting_pty`, `port_forwarding`, `workspace_app`); session
methods are derived from the app name until #31012 stores them
- Add `usage_total_seconds` to count each user's active minutes once,
and use it as the UI bar denominator so overlapping apps can exceed
100%
- Merge template apps sharing a slug into one row across templates,
using the newest display name and icon
- Rename Web Terminal slug from `reconnecting-pty` to
`reconnecting_pty`
- Cap the response at the 64 busiest unregistered apps and fold the
rest into "Other apps", flagged with `seconds_is_upper_bound`
("Up to" in the UI)
- Add `unknown` and `overflow` accounting rows with explanatory
tooltips, and a fallback icon for apps without one
- Remove the unused `codersdk.UnionByFamily` helper from the Go SDK
BREAKING CHANGE: `report.apps_usage` in `GET /api/v2/insights/templates`
no longer returns the fixed five rows, family slugs, or previous display
names. Consumers must update.
Part of CLIENT-642
Prometheus metrics handled separately in #29906
coderd_insights_applications_usage_seconds summed each family's per-app
minutes, so a user with VS Code and Cursor open in the same minute was
counted as two VS Code minutes. GetTemplateInsightsByTemplate now takes
the registry's app-to-family mapping as jsonb, groups the raw agent
stats by family, and counts each minute once per user and family.
- Keep the metric's four built-in series (Visual Studio Code, JetBrains,
Web Terminal, SSH) and their labels; only their values change. The
HELP text now says built-in apps are grouped by family.
- Add AppFamilyName.DisplayName() and remove the duplicate
TemplateBuiltinAppDisplayName* constants from codersdk. The template
insights handler returns the same display names.
- Pin how the migration 000596 rollback approximates family minutes
from per-app rows.
Family values can drop wherever the old per-app sum double-counted
overlapping apps. No migration or storage changes.
Store per-app session minutes in a child table instead of fixed family-minute columns, so adding an IDE no longer needs a rollup schema change. API, Prometheus, and telemetry output are unchanged.
- Add `template_usage_stats_session_apps`, keyed by start time, template, user, and app name, replacing the `ssh_mins`, `sftp_mins`, `reconnecting_pty_mins`, `vscode_mins`, and `jetbrains_mins` columns.
- Group app names into families at read time through the `codersdk` registry, so the registry never reaches SQL and attribution covers all history rather than only the buckets the rollup still revisits.
- Group `GetTemplateInsightsByTemplate` by app name so that Prometheus and the API report a family identically.
- Drop the `@app_families` query parameter and its dbauthz validation.
- Report a family total as the sum of its apps rather than the distinct minutes any of them was active. Numbers are unchanged today, because every agent reports one canonical name per family.
- Keep the child table window index-usable and count minutes with `COUNT(DISTINCT)` in the grouping, so a dashboard-sized range uses an index scan and a large deployment does not spill the sort to disk.
Part of #27410
- Centralize app-to-family attribution in the SDK registry so recognized
app aliases contribute to their family's session counts and usage
minutes without duplicating name lists across SQL queries.
- Pass the registry into agent stats queries, insights, and usage rollups.
- Validate its structure in dbauthz so missing or renamed families fail
explicitly instead of silently producing zero counts.
- Decompose session maps once per row for per-agent sums while retaining
the key-presence checks used by insights. Update reporting consumers
and add coverage for alias attribution and registry validation.
Replace fixed session_count_* columns with a sparse session_counts
JSONB object so new session types require no schema changes.
Convert retained and unrolled-up activity, warning on backlogs spanning
more than 24 hours. Update batching, queries, and rollups while preserving
existing API shapes and compatibility with older agents.
Add migration, query, and batcher coverage.
closes https://github.com/coder/internal/issues/464
# Summary
This PR resolves a flaky test that was sensitive to DST transitions in
various time zones. The root of the flake was:
* a bug; the query and its tests assume 24 hours per day
* the tests used local system time, which resulted in failures for dates
proximal to DST transitions
# Changes
Query:
The original query assumed 24 hour intervals between each day, which is
not a valid assumption. It now increments `1 day` at a time.
Database tests:
Database level tests for the query all assumed 24 hour days. They now
increment in DST-aware days instead. Instead of using time.Now() as a
base for testing, the test uses a series of dates over the course of an
entire year, to ensure that DST transition dates are present in every
test run.
# API Endpoint
The endpoint that delivers the user status chart now accepts an IANA
timezone name as a parameter and passes it, keeping the existing offset
as a fallback, to the database query.
API level tests were added to ensure the correct response form and error
behaviour. Correctness of content is tested at the database level.
In this PR we're optimizing the `GetTemplateAppInsightsByTemplate` query
by pre-filtering out apps which do not have an active session during the
start/end time window.
---------
Signed-off-by: Callum Styan <callumstyan@gmail.com>
RE: https://github.com/coder/coder/issues/15740,
https://github.com/coder/coder/issues/15297
In order to add a graph to the coder frontend to show user status over
time as an indicator of license usage, this PR adds the following:
* a new `api.insightsUserStatusCountsOverTime` endpoint to the API
* which calls a new `GetUserStatusCountsOverTime` query from postgres
* which relies on two new tables `user_status_changes` and
`user_deleted`
* which are populated by a new trigger and function that tracks updates
to the users table
The chart itself will be added in a subsequent PR
---------
Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
This PR updates the `*ByTempalte` insights queries used for generating Prometheus metrics to behave the same way as the new rollup query and re-written insights queries that utilize the rolled up data.
Add `template_usage_stats` table for aggregating tempalte usage data.
Data is rolled up by the `UpsertTemplateUsageStats` query, which fetches
data from the `workspace_agent_stats` and `workspace_app_stats` tables.