Commit Graph
39 Commits
Author SHA1 Message Date
dylanhuff-at-coder 3728e7a4e1 perf: use the build parameter index in GetTemplateParameterInsights (#31765)
Speeds up `GetTemplateParameterInsights`, which the insights collector
runs every 5 minutes on every replica. The `@>` array join could only
hash on `name`, so every run scanned all of
`workspace_build_parameters`. Unnesting the build IDs and joining on
`(workspace_build_id, name)` lets the existing unique index serve it,
and the results don't change since each build ID shows up at most once
per group.

On 8.4M synthetic build parameters the collector's 5 minute window went
from 5,193 ms to about 31 ms. `TestGetTemplateParameterInsights` is new
and passes on both the old and new SQL. This also fixes the Prometheus
docs example, which summed per-replica copies of the same metric.

I split the `workspace_app_stats` index into #31766 so this one has no
migration and can be backported to 2.34 through 2.39 (2.34 to 2.38
without the docs hunk).

The pre-push hook skipped tests on this push because of a stale
`origin/main` in my checkout. An earlier full run only failed in
`agent/` and `site/`.

Part of [CORE-439](https://linear.app/codercom/issue/CORE-439).
2026-11-10 09:30:21 -08:00
Ehab Younes 28192213ab feat!: label the VS Code family and report built-in app usage per app (#29906)
The session banner now labels the VS Code family "VS Code Family", so it isn't confused with the VS Code editor. coderd_insights_applications_usage_seconds now reports built-in apps per app instead of per family.

- Generate the frontend's family labels from the Go table in codersdk.
- Add the connection_method label: workspace_app for template apps, and ssh, reconnecting_pty, or port_forwarding for built-in apps.
- Rename the application_name label to display_name. For built-in apps it holds the registry name, "Unknown" for unknown, and "Other apps" for overflow.
- Built-in apps get one series per app name, only for apps used in the window. The series limit and the fold rule are shared with the template insights API.
- Breaking: selectors with a value for application_name, or with slug="", match nothing. The Prometheus docs map each old series to its new one.

Closes #29940
Closes CLIENT-1034
Closes CLIENT-1036
2026-10-27 09:58:36 +00:00
Ehab Younes f4cd0deccb feat!: report template insights session usage per app (#29796)
Report template insights session usage per app instead of five fixed
family rows, with registry names and icons.

- Add per-app rows for reported session apps (VS Code, Cursor, GoLand,
  Zed, etc.); unused apps are omitted
- Replace `type` (`builtin`/`app`) with `method` (`ssh`,
  `reconnecting_pty`, `port_forwarding`, `workspace_app`); session
  methods are derived from the app name until #31012 stores them
- Add `usage_total_seconds` to count each user's active minutes once,
  and use it as the UI bar denominator so overlapping apps can exceed
  100%
- Merge template apps sharing a slug into one row across templates,
  using the newest display name and icon
- Rename Web Terminal slug from `reconnecting-pty` to
  `reconnecting_pty`
- Cap the response at the 64 busiest unregistered apps and fold the
  rest into "Other apps", flagged with `seconds_is_upper_bound`
  ("Up to" in the UI)
- Add `unknown` and `overflow` accounting rows with explanatory
  tooltips, and a fallback icon for apps without one
- Remove the unused `codersdk.UnionByFamily` helper from the Go SDK

BREAKING CHANGE: `report.apps_usage` in `GET /api/v2/insights/templates`
no longer returns the fixed five rows, family slugs, or previous display
names. Consumers must update.

Part of CLIENT-642
Prometheus metrics handled separately in #29906
2026-10-19 17:59:18 +03:00
Ehab Younes 2f2169ed19 fix: deduplicate template insights family metrics (#30497)
coderd_insights_applications_usage_seconds summed each family's per-app
minutes, so a user with VS Code and Cursor open in the same minute was
counted as two VS Code minutes. GetTemplateInsightsByTemplate now takes
the registry's app-to-family mapping as jsonb, groups the raw agent
stats by family, and counts each minute once per user and family.

- Keep the metric's four built-in series (Visual Studio Code, JetBrains,
  Web Terminal, SSH) and their labels; only their values change. The
  HELP text now says built-in apps are grouped by family.
- Add AppFamilyName.DisplayName() and remove the duplicate
  TemplateBuiltinAppDisplayName* constants from codersdk. The template
  insights handler returns the same display names.
- Pin how the migration 000596 rollback approximates family minutes
  from per-app rows.

Family values can drop wherever the old per-app sum double-counted
overlapping apps. No migration or storage changes.
2026-10-12 23:58:35 +03:00
Ehab Younes 817fc77612 feat: store session usage per app and family in template usage stats (#29109)
Store per-app session minutes in a child table instead of fixed family-minute columns, so adding an IDE no longer needs a rollup schema change. API, Prometheus, and telemetry output are unchanged.

- Add `template_usage_stats_session_apps`, keyed by start time, template, user, and app name, replacing the `ssh_mins`, `sftp_mins`, `reconnecting_pty_mins`, `vscode_mins`, and `jetbrains_mins` columns.
- Group app names into families at read time through the `codersdk` registry, so the registry never reaches SQL and attribution covers all history rather than only the buckets the rollup still revisits.
- Group `GetTemplateInsightsByTemplate` by app name so that Prometheus and the API report a family identically.
- Drop the `@app_families` query parameter and its dbauthz validation.
- Report a family total as the sum of its apps rather than the distinct minutes any of them was active. Numbers are unchanged today, because every agent reports one canonical name per family.
- Keep the child table window index-usable and count minutes with `COUNT(DISTINCT)` in the grouping, so a dashboard-sized range uses an index scan and a large deployment does not spill the sort to disk.

Part of #27410
2026-09-18 15:26:47 +02:00
Ehab Younes da45488208 feat: attribute session counts by app family (#28337)
- Centralize app-to-family attribution in the SDK registry so recognized
app aliases contribute to their family's session counts and usage
minutes without duplicating name lists across SQL queries.
- Pass the registry into agent stats queries, insights, and usage rollups.
- Validate its structure in dbauthz so missing or renamed families fail
explicitly instead of silently producing zero counts.
- Decompose session maps once per row for per-agent sums while retaining
the key-presence checks used by insights. Update reporting consumers
and add coverage for alias attribution and registry validation.
2026-09-10 13:54:11 +02:00
Ehab Younes 90c7533204 perf: store workspace agent session counts as JSONB (#28126)
Replace fixed session_count_* columns with a sparse session_counts
JSONB object so new session types require no schema changes.

Convert retained and unrolled-up activity, warning on backlogs spanning
more than 24 hours. Update batching, queries, and rollups while preserving
existing API shapes and compatibility with older agents.

Add migration, query, and batcher coverage.
2026-09-07 22:31:42 +02:00
Sas Swart cfcb81fb0f fix: user status change chart accommodates DST (#22191)
closes https://github.com/coder/internal/issues/464

# Summary

This PR resolves a flaky test that was sensitive to DST transitions in
various time zones. The root of the flake was:
* a bug; the query and its tests assume 24 hours per day
* the tests used local system time, which resulted in failures for dates
proximal to DST transitions

# Changes

Query:

The original query assumed 24 hour intervals between each day, which is
not a valid assumption. It now increments `1 day` at a time.

Database tests:

Database level tests for the query all assumed 24 hour days. They now
increment in DST-aware days instead. Instead of using time.Now() as a
base for testing, the test uses a series of dates over the course of an
entire year, to ensure that DST transition dates are present in every
test run.

# API Endpoint

The endpoint that delivers the user status chart now accepts an IANA
timezone name as a parameter and passes it, keeping the existing offset
as a fallback, to the database query.

API level tests were added to ensure the correct response form and error
behaviour. Correctness of content is tested at the database level.
2026-03-04 12:54:39 +02:00
Callum Styan a59a84b2a7 perf: optimize GetTemplateAppInsightsByTemplate by pre-filtering on start/end times (#20669)
In this PR we're optimizing the `GetTemplateAppInsightsByTemplate` query
by pre-filtering out apps which do not have an active session during the
start/end time window.

---------

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2025-12-09 15:21:16 -08:00
Sas Swart 50bf5ca8fe fix(coderd/database): aggregate user engagement statistics by interval (#16150)
This PR addresses the TODO comment here:

https://github.com/coder/coder/pull/16134/files#diff-1844f895bb005f036da11d800fe2a76b54bfddd481c5d8cb15f210c64679caa5R47

The backend now backfills entries for dates with no status changes.
2025-01-16 17:34:53 +02:00
Sas Swart ab880904e2 feat(coderd/database): order user engagement statistics by date (#16142)
This PR orders user engagement statistics by date as mentioned
[here](https://github.com/coder/coder/pull/16134/files#diff-1844f895bb005f036da11d800fe2a76b54bfddd481c5d8cb15f210c64679caa5R47)
2025-01-15 13:19:25 +02:00
Sas SwartandMathias Fredriksson 4543b21b7c feat(coderd/database): track user status changes over time (#16019)
RE: https://github.com/coder/coder/issues/15740,
https://github.com/coder/coder/issues/15297

In order to add a graph to the coder frontend to show user status over
time as an indicator of license usage, this PR adds the following:

* a new `api.insightsUserStatusCountsOverTime` endpoint to the API
* which calls a new `GetUserStatusCountsOverTime` query from postgres
* which relies on two new tables `user_status_changes` and
`user_deleted`
* which are populated by a new trigger and function that tracks updates
to the users table

The chart itself will be added in a subsequent PR

---------

Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
2025-01-13 13:08:16 +02:00
Garrett Delfosse 5f640eb219 fix: correct connection_median_latency_ms in query (#15086)
Closes https://github.com/coder/coder/issues/14805
2024-10-17 12:22:26 -04:00
Mathias Fredriksson a0fce363cd feat(coderd): add times_used to coder_apps in insights API (#13292)
For now, only applied to `coder_app`s, same logic can be implemented for
VS Code, SSH, etc.

Part of #13099
2024-05-16 16:53:01 +03:00
Mathias Fredriksson 47fd190064 fix(coderd/database): improve perf of GetTemplateInsightsByInterval (#12773)
Refs #12122
2024-03-27 14:10:46 +02:00
Mathias Fredriksson 0da29d74ac fix(coderd/database): improve query perf of GetTemplateAppInsights (#12767)
Refs #12122
2024-03-27 12:28:36 +02:00
Mathias Fredriksson ae0ee622bb fix(coderd/database): improve data exclusion in UpsertTemplateUsageStats (#12764)
The PostgreSQL query analyzer wasn't able to eliminate the agent stats without re-introducing this filter.

Before: https://explain.dalibo.com/plan/21h7gb4f4bef391g
After: https://explain.dalibo.com/plan/721ec1cccee91egc
2024-03-26 17:21:05 +02:00
Mathias Fredriksson b183236482 feat(coderd/database): use template_usage_stats in *ByTemplate insights queries (#12668)
This PR updates the `*ByTempalte` insights queries used for generating Prometheus metrics to behave the same way as the new rollup query and re-written insights queries that utilize the rolled up data.
2024-03-25 17:42:02 +02:00
Mathias Fredriksson 2332d8197a feat(coderd/database): use template_usage_stats in GetUserActivityInsights query (#12672)
This PR updates the `GetUserActivityInsights` query to use rolled up `template_usage_stats` instead of raw agent and app stats.
2024-03-25 16:16:41 +02:00
Mathias Fredriksson a8ed689bda feat(coderd/database): use template_usage_stats in GetUserLatencyInsights query (#12671)
This PR updates the `GetUserLatencyInsights` query to use rolled up `template_usage_stats` instead of raw agent and app stats.
2024-03-25 16:07:40 +02:00
Mathias Fredriksson 5738a03930 feat(coderd/database): use template_usage_stats in GetTemplateAppInsights query (#12669)
This PR updates the `GetTemplateAppInsights` query to use rolled up `template_usage_stats` instead of raw agent and app stats.
2024-03-25 15:58:37 +02:00
Mathias Fredriksson 5f3be62c83 feat(coderd/database): use template_usage_stats in GetTemplateInsightsByInterval query (#12667)
This PR updates the `GetTemplateInsightsByInterval` query to use rolled up `template_usage_stats` instead of raw agent and app stats.
2024-03-25 15:45:49 +02:00
Mathias Fredriksson 35d08434a9 feat(coderd/database): use template_usage_stats in GetTemplateInsights query (#12666)
This PR updates the `GetTemplateInsights` query to use rolled up `template_usage_stats` instead of raw agent and app stats.
2024-03-25 15:33:31 +02:00
Mathias Fredriksson 04f0510b09 feat(coderd/database): add template_usage_stats table and rollup query (#12664)
Add `template_usage_stats` table for aggregating tempalte usage data.
Data is rolled up by the `UpsertTemplateUsageStats` query, which fetches
data from the `workspace_agent_stats` and `workspace_app_stats` tables.
2024-03-22 18:33:34 +02:00
Marcin Tojek 0a550815e9 feat: expose app insights as Prometheus metrics (#10346) 2023-11-07 17:14:59 +01:00
Marcin Tojek c4f590581e feat: expose template insights as Prometheus metrics (#10325) 2023-10-19 08:45:12 +00:00
Marcin Tojek 4c3b579f58 feat: expose insights into user activity (#9807) 2023-09-26 18:42:16 +02:00
Marcin Tojek d0d64bbdca refactor: define insights interval (#9693) 2023-09-15 12:01:00 +00:00
Mathias Fredriksson af939d1e94 fix(coderd): optimize template app insights query for speed and decrease intervals (#9302) 2023-08-24 14:34:38 +00:00
Mathias Fredriksson 04d5e3f54f fix(coderd): rewrite template insights query for speed and fix intervals (#9300) 2023-08-24 14:38:32 +03:00
Mathias Fredriksson d00817ea4a perf(coderd): simplify insights queries to speed them up (#9299) 2023-08-24 10:52:20 +00:00
Mathias Fredriksson 6b69abfec7 fix(coderd): use stable sorting for insights and improve test coverage (#9250)
Fixes #9213
2023-08-24 13:36:40 +03:00
Mathias Fredriksson 03453b1e02 feat(coderd): add template app usage to insights (#9138)
Fixes #8658
2023-08-21 15:08:58 +03:00
Marcin Tojek e8627195a2 feat(coderd): expose parameter description and type (#8944) 2023-08-07 16:11:44 +00:00
Mathias Fredriksson d3991fac26 feat(coderd): add parameter insights to template insights (#8656) 2023-08-03 14:43:23 +00:00
Bruno Quaresma fbb2a6a434 refactor(coderd): add avatar URL to the users' latency endpoint (#8701) 2023-07-25 12:15:46 +02:00
Marcin Tojek b7ced94f0f fix: prevent repetition of template IDs in template_usage_by_day (#8693) 2023-07-24 15:07:01 +02:00
Mathias Fredriksson e0c1aacac1 fix(coderd): fix template insight intervals (#8662) 2023-07-21 20:51:35 +00:00
Mathias Fredriksson 30fe153296 feat(coderd): add user latency and template insights endpoints (#8519)
Part of #8514
Refs #8109
2023-07-21 18:00:19 +00:00