feat(examples/templates/kubernetes-podman-rootless): persist home and fail on Podman startup errors

- Mount a persistent volume at /home/coder (fs_group_change_policy
  OnRootMismatch) so repos, images and build cache survive restarts
- Fail the startup script with the Podman log when the API socket
  doesn't start
- Use kubernetes_pod_v1 and pin nodes to amd64
- Docs: fail-fast snippet and the templates init link

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Cedric FitzgeraldandClaude Opus 5.5 committed 2026-11-02 10:36:32 -08:00
1 parent 882b98974b
commit 0227e04155
4 files changed
+73 -41

No files matched your search

@@ -352,29 +352,7 @@ spec:
user.max_user_namespaces: 65536
```
Reference the `NodeClass` from a `NodePool`:
```yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: podman
spec:
template:
spec:
nodeClassRef:
group: eks.amazonaws.com
kind: NodeClass
name: podman
requirements:
# The codercom/example-podman image is published for amd64 only.
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
```
Auto Mode labels each node with `eks.amazonaws.com/nodeclass` set to the name of its `NodeClass`.
Schedule Podman workspaces onto the pool with the node selector `eks.amazonaws.com/nodeclass: podman`.
Reference the `NodeClass` from a `NodePool` and schedule Podman workspaces onto that pool with a node selector.
Refer to [Create a Node Class for Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-node-class.html) for the full `advancedCompute` reference.
If you can't use `advancedCompute`, run Podman workspaces on a managed node group instead.
@@ -392,19 +370,16 @@ The value must be a multiple of 65536, and `idsPerPod` requires `user.max_user_n
idsPerPod: 1048576
```
### Use native overlay storage instead of `fuse-overlayfs`
### Use native overlay storage instead of fuse-overlayfs
Kernel overlayfs cannot stack on the workspace container's own overlay root filesystem.
Without extra configuration, Podman falls back to `fuse-overlayfs`, which requires the `/dev/fuse` device and the smarter-device-manager setup above.
On nodes with kernel 5.13 or later, you can skip the FUSE device entirely:
1. Put the Podman graph root on a real filesystem such as ext4 or xfs.
If your template mounts a persistent volume at `/home/coder`, like the `kubernetes` starter, the default graph root already qualifies and keeps images across workspace restarts.
For ephemeral workspaces, mount an `emptyDir` volume at `/home/coder/.local/share/containers` instead.
Pulled images and build cache are lost when the workspace stops.
For ephemeral workspaces, mount an `emptyDir` volume at `/home/coder/.local/share/containers` instead; pulled images and build cache are lost when the workspace stops.
1. Clear the `mount_program` setting, because some Podman images pin it to `fuse-overlayfs` in `/etc/containers/storage.conf`.
Write the override from the agent `startup_script`, which can also start the Docker-compatible API socket:
```tf
@@ -426,8 +401,7 @@ On nodes with kernel 5.13 or later, you can skip the FUSE device entirely:
`docker buildx` commands still require BuildKit and aren't supported.
With both settings, Podman uses native kernel overlay, which performs better than `fuse-overlayfs` and removes the device plugin dependency.
When you use native overlay, skip the smarter-device-manager steps above and remove the `github.com/fuse` limit from the pod spec.
Without the device plugin, that limit keeps pods in `Pending`.
When you use native overlay, skip the smarter-device-manager steps above and remove the `github.com/fuse` limit from the pod spec; without the device plugin, that limit keeps pods in `Pending`.
For a complete reference template, run `coder templates init --id kubernetes-podman-rootless` or refer to [`examples/templates/kubernetes-podman-rootless`](https://github.com/coder/coder/tree/main/examples/templates/kubernetes-podman-rootless).
+1 -1
View File
@@ -209,7 +209,7 @@
"kubernetes",
"podman"
],
"markdown": "\n# Remote development on Kubernetes with rootless Podman\n\nProvision workspaces that run containers with rootless Podman, without privileged pods or a custom container runtime.\n\nThe workspace pod runs as a non-root user.\nPodman uses native kernel overlay storage, so the `/dev/fuse` device plugin is not required on nodes with kernel 5.13 or later.\nThe agent starts a Docker-compatible API socket for tools such as Testcontainers and Docker Compose.\n\n## Prerequisites\n\n- Nodes with `user.max_user_namespaces` set above `0`.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#rootless-podman) for per-platform instructions, including EKS Auto Mode.\n If only some nodes qualify, set the `node_selector` variable to schedule workspaces onto them.\n- The Coder provisioner needs permission to create pods in the target namespace.\n\n## Notes\n\n- The Podman graph root is an `emptyDir` volume, so pulled images are lost when the workspace stops.\n Mount a persistent volume at `/home/coder` to keep them instead.\n- One level of container nesting is supported.\n For dev containers, use the `docker-outside-of-docker` feature.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#nested-container-engines).\n"
"markdown": "\n# Remote development on Kubernetes with rootless Podman\n\nProvision workspaces that run containers with rootless Podman, without privileged pods or a custom container runtime.\n\nThe workspace pod runs as a non-root user.\nPodman uses native kernel overlay storage, so the `/dev/fuse` device plugin is not required on nodes with kernel 5.13 or later.\nThe agent starts a Docker-compatible API socket for tools such as Testcontainers and Docker Compose.\n\n## Prerequisites\n\n- Nodes with `user.max_user_namespaces` set above `0`.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#rootless-podman) for per-platform instructions, including EKS Auto Mode.\n If only some nodes qualify, set the `node_selector` variable to schedule workspaces onto them.\n- The Coder provisioner needs permission to create pods in the target namespace.\n\n## Notes\n\n- The home directory is a persistent volume, so cloned repositories, pulled images, and build cache survive workspace restarts.\n The volume must be ext4 or xfs for native overlay storage, which most default storage classes provide.\n- If Podman can't start its API socket, for example because user namespaces are disabled on the node, the startup script fails and prints the Podman error to the startup logs.\n- One level of container nesting is supported.\n For dev containers, use the `docker-outside-of-docker` feature.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#nested-container-engines).\n"
},
{
"id": "nomad-docker",
@@ -24,8 +24,9 @@ The agent starts a Docker-compatible API socket for tools such as Testcontainers
## Notes
- The Podman graph root is an `emptyDir` volume, so pulled images are lost when the workspace stops.
Mount a persistent volume at `/home/coder` to keep them instead.
- The home directory is a persistent volume, so cloned repositories, pulled images, and build cache survive workspace restarts.
The volume must be ext4 or xfs for native overlay storage, which most default storage classes provide.
- If Podman can't start its API socket, for example because user namespaces are disabled on the node, the startup script fails and prints the Podman error to the startup logs.
- One level of container nesting is supported.
For dev containers, use the `docker-outside-of-docker` feature.
Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#nested-container-engines).
@@ -31,19 +31,40 @@ provider "kubernetes" {
data "coder_workspace" "me" {}
data "coder_workspace_owner" "me" {}
data "coder_parameter" "home_disk_size" {
name = "home_disk_size"
display_name = "Home disk size"
description = "The size of the home disk in GB, including Podman images and build cache"
default = "20"
type = "number"
icon = "/emojis/1f4be.png"
mutable = false
validation {
min = 1
max = 99999
}
}
resource "coder_agent" "main" {
os = "linux"
arch = "amd64"
startup_script = <<-EOT
#!/bin/bash
# Seed shell dotfiles that the empty home volume hides on first start.
[ -f ~/.bashrc ] || cp -rT /etc/skel ~ || true
# Use native kernel overlay storage. Some Podman images pin
# mount_program to fuse-overlayfs in /etc/containers/storage.conf.
mkdir -p ~/.config/containers
printf '[storage]\ndriver = "overlay"\n\n[storage.options.overlay]\nmount_program = ""\n' > ~/.config/containers/storage.conf
# Docker-compatible API socket for Testcontainers, Compose, and
# the dev container docker-outside-of-docker feature.
(podman system service --time=0 unix:///tmp/podman.sock >/dev/null 2>&1 &)
(podman system service --time=0 unix:///tmp/podman.sock >/tmp/podman-service.log 2>&1 &)
for i in $(seq 50); do [ -S /tmp/podman.sock ] && break; sleep 0.1; done
if [ ! -S /tmp/podman.sock ]; then
echo "Podman API socket didn't start:" >&2
cat /tmp/podman-service.log >&2
exit 1
fi
EOT
env = {
@@ -54,7 +75,37 @@ resource "coder_agent" "main" {
}
}
resource "kubernetes_pod" "main" {
resource "kubernetes_persistent_volume_claim_v1" "home" {
metadata {
name = "coder-${data.coder_workspace.me.id}-home"
namespace = var.namespace
labels = {
"app.kubernetes.io/name" = "coder-pvc"
"app.kubernetes.io/instance" = "coder-pvc-${data.coder_workspace.me.id}"
"app.kubernetes.io/part-of" = "coder"
//Coder-specific labels.
"com.coder.resource" = "true"
"com.coder.workspace.id" = data.coder_workspace.me.id
"com.coder.workspace.name" = data.coder_workspace.me.name
"com.coder.user.id" = data.coder_workspace_owner.me.id
"com.coder.user.username" = data.coder_workspace_owner.me.name
}
annotations = {
"com.coder.user.email" = data.coder_workspace_owner.me.email
}
}
wait_until_bound = false
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "${data.coder_parameter.home_disk_size.value}Gi"
}
}
}
}
resource "kubernetes_pod_v1" "main" {
count = data.coder_workspace.me.start_count
metadata {
name = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}"
@@ -66,11 +117,15 @@ resource "kubernetes_pod" "main" {
}
}
spec {
node_selector = var.node_selector
# The codercom/example-podman image is published for amd64 only.
node_selector = merge({ "kubernetes.io/arch" = "amd64" }, var.node_selector)
security_context {
run_as_user = 1000
fs_group = 1000
run_as_non_root = true
# Skip the recursive chgrp on every start, which would rewrite the
# subuid-owned image layers in the Podman graph root.
fs_group_change_policy = "OnRootMismatch"
}
container {
name = "dev"
@@ -90,15 +145,17 @@ resource "kubernetes_pod" "main" {
memory = "4Gi"
}
}
# The Podman graph root lives under home on this ext4 or xfs volume,
# so kernel overlay works and images persist across restarts.
volume_mount {
name = "container-storage"
mount_path = "/home/coder/.local/share/containers"
name = "home"
mount_path = "/home/coder"
}
}
volume {
name = "container-storage"
empty_dir {
size_limit = "20Gi"
name = "home"
persistent_volume_claim {
claim_name = kubernetes_persistent_volume_claim_v1.home.metadata[0].name
}
}
}