feat(examples/templates/kubernetes-podman-rootless): persist home and fail on Podman startup errors
- Mount a persistent volume at /home/coder (fs_group_change_policy OnRootMismatch) so repos, images and build cache survive restarts - Fail the startup script with the Podman log when the API socket doesn't start - Use kubernetes_pod_v1 and pin nodes to amd64 - Docs: fail-fast snippet and the templates init link Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
882b98974b
commit
0227e04155
4 files changed
+73
-41
No files matched your search
@@ -352,29 +352,7 @@ spec:
|
||||
user.max_user_namespaces: 65536
|
||||
```
|
||||
|
||||
Reference the `NodeClass` from a `NodePool`:
|
||||
|
||||
```yaml
|
||||
apiVersion: karpenter.sh/v1
|
||||
kind: NodePool
|
||||
metadata:
|
||||
name: podman
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
nodeClassRef:
|
||||
group: eks.amazonaws.com
|
||||
kind: NodeClass
|
||||
name: podman
|
||||
requirements:
|
||||
# The codercom/example-podman image is published for amd64 only.
|
||||
- key: kubernetes.io/arch
|
||||
operator: In
|
||||
values: ["amd64"]
|
||||
```
|
||||
|
||||
Auto Mode labels each node with `eks.amazonaws.com/nodeclass` set to the name of its `NodeClass`.
|
||||
Schedule Podman workspaces onto the pool with the node selector `eks.amazonaws.com/nodeclass: podman`.
|
||||
Reference the `NodeClass` from a `NodePool` and schedule Podman workspaces onto that pool with a node selector.
|
||||
Refer to [Create a Node Class for Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-node-class.html) for the full `advancedCompute` reference.
|
||||
|
||||
If you can't use `advancedCompute`, run Podman workspaces on a managed node group instead.
|
||||
@@ -392,19 +370,16 @@ The value must be a multiple of 65536, and `idsPerPod` requires `user.max_user_n
|
||||
idsPerPod: 1048576
|
||||
```
|
||||
|
||||
### Use native overlay storage instead of `fuse-overlayfs`
|
||||
### Use native overlay storage instead of fuse-overlayfs
|
||||
|
||||
Kernel overlayfs cannot stack on the workspace container's own overlay root filesystem.
|
||||
Without extra configuration, Podman falls back to `fuse-overlayfs`, which requires the `/dev/fuse` device and the smarter-device-manager setup above.
|
||||
On nodes with kernel 5.13 or later, you can skip the FUSE device entirely:
|
||||
|
||||
1. Put the Podman graph root on a real filesystem such as ext4 or xfs.
|
||||
|
||||
If your template mounts a persistent volume at `/home/coder`, like the `kubernetes` starter, the default graph root already qualifies and keeps images across workspace restarts.
|
||||
For ephemeral workspaces, mount an `emptyDir` volume at `/home/coder/.local/share/containers` instead.
|
||||
Pulled images and build cache are lost when the workspace stops.
|
||||
For ephemeral workspaces, mount an `emptyDir` volume at `/home/coder/.local/share/containers` instead; pulled images and build cache are lost when the workspace stops.
|
||||
1. Clear the `mount_program` setting, because some Podman images pin it to `fuse-overlayfs` in `/etc/containers/storage.conf`.
|
||||
|
||||
Write the override from the agent `startup_script`, which can also start the Docker-compatible API socket:
|
||||
|
||||
```tf
|
||||
@@ -426,8 +401,7 @@ On nodes with kernel 5.13 or later, you can skip the FUSE device entirely:
|
||||
`docker buildx` commands still require BuildKit and aren't supported.
|
||||
|
||||
With both settings, Podman uses native kernel overlay, which performs better than `fuse-overlayfs` and removes the device plugin dependency.
|
||||
When you use native overlay, skip the smarter-device-manager steps above and remove the `github.com/fuse` limit from the pod spec.
|
||||
Without the device plugin, that limit keeps pods in `Pending`.
|
||||
When you use native overlay, skip the smarter-device-manager steps above and remove the `github.com/fuse` limit from the pod spec; without the device plugin, that limit keeps pods in `Pending`.
|
||||
|
||||
For a complete reference template, run `coder templates init --id kubernetes-podman-rootless` or refer to [`examples/templates/kubernetes-podman-rootless`](https://github.com/coder/coder/tree/main/examples/templates/kubernetes-podman-rootless).
|
||||
|
||||
|
||||
@@ -209,7 +209,7 @@
|
||||
"kubernetes",
|
||||
"podman"
|
||||
],
|
||||
"markdown": "\n# Remote development on Kubernetes with rootless Podman\n\nProvision workspaces that run containers with rootless Podman, without privileged pods or a custom container runtime.\n\nThe workspace pod runs as a non-root user.\nPodman uses native kernel overlay storage, so the `/dev/fuse` device plugin is not required on nodes with kernel 5.13 or later.\nThe agent starts a Docker-compatible API socket for tools such as Testcontainers and Docker Compose.\n\n## Prerequisites\n\n- Nodes with `user.max_user_namespaces` set above `0`.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#rootless-podman) for per-platform instructions, including EKS Auto Mode.\n If only some nodes qualify, set the `node_selector` variable to schedule workspaces onto them.\n- The Coder provisioner needs permission to create pods in the target namespace.\n\n## Notes\n\n- The Podman graph root is an `emptyDir` volume, so pulled images are lost when the workspace stops.\n Mount a persistent volume at `/home/coder` to keep them instead.\n- One level of container nesting is supported.\n For dev containers, use the `docker-outside-of-docker` feature.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#nested-container-engines).\n"
|
||||
"markdown": "\n# Remote development on Kubernetes with rootless Podman\n\nProvision workspaces that run containers with rootless Podman, without privileged pods or a custom container runtime.\n\nThe workspace pod runs as a non-root user.\nPodman uses native kernel overlay storage, so the `/dev/fuse` device plugin is not required on nodes with kernel 5.13 or later.\nThe agent starts a Docker-compatible API socket for tools such as Testcontainers and Docker Compose.\n\n## Prerequisites\n\n- Nodes with `user.max_user_namespaces` set above `0`.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#rootless-podman) for per-platform instructions, including EKS Auto Mode.\n If only some nodes qualify, set the `node_selector` variable to schedule workspaces onto them.\n- The Coder provisioner needs permission to create pods in the target namespace.\n\n## Notes\n\n- The home directory is a persistent volume, so cloned repositories, pulled images, and build cache survive workspace restarts.\n The volume must be ext4 or xfs for native overlay storage, which most default storage classes provide.\n- If Podman can't start its API socket, for example because user namespaces are disabled on the node, the startup script fails and prints the Podman error to the startup logs.\n- One level of container nesting is supported.\n For dev containers, use the `docker-outside-of-docker` feature.\n Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#nested-container-engines).\n"
|
||||
},
|
||||
{
|
||||
"id": "nomad-docker",
|
||||
|
||||
@@ -24,8 +24,9 @@ The agent starts a Docker-compatible API socket for tools such as Testcontainers
|
||||
|
||||
## Notes
|
||||
|
||||
- The Podman graph root is an `emptyDir` volume, so pulled images are lost when the workspace stops.
|
||||
Mount a persistent volume at `/home/coder` to keep them instead.
|
||||
- The home directory is a persistent volume, so cloned repositories, pulled images, and build cache survive workspace restarts.
|
||||
The volume must be ext4 or xfs for native overlay storage, which most default storage classes provide.
|
||||
- If Podman can't start its API socket, for example because user namespaces are disabled on the node, the startup script fails and prints the Podman error to the startup logs.
|
||||
- One level of container nesting is supported.
|
||||
For dev containers, use the `docker-outside-of-docker` feature.
|
||||
Refer to [Docker in workspaces](https://coder.com/docs/admin/templates/extending-templates/docker-in-workspaces#nested-container-engines).
|
||||
@@ -31,19 +31,40 @@ provider "kubernetes" {
|
||||
data "coder_workspace" "me" {}
|
||||
data "coder_workspace_owner" "me" {}
|
||||
|
||||
data "coder_parameter" "home_disk_size" {
|
||||
name = "home_disk_size"
|
||||
display_name = "Home disk size"
|
||||
description = "The size of the home disk in GB, including Podman images and build cache"
|
||||
default = "20"
|
||||
type = "number"
|
||||
icon = "/emojis/1f4be.png"
|
||||
mutable = false
|
||||
validation {
|
||||
min = 1
|
||||
max = 99999
|
||||
}
|
||||
}
|
||||
|
||||
resource "coder_agent" "main" {
|
||||
os = "linux"
|
||||
arch = "amd64"
|
||||
startup_script = <<-EOT
|
||||
#!/bin/bash
|
||||
# Seed shell dotfiles that the empty home volume hides on first start.
|
||||
[ -f ~/.bashrc ] || cp -rT /etc/skel ~ || true
|
||||
# Use native kernel overlay storage. Some Podman images pin
|
||||
# mount_program to fuse-overlayfs in /etc/containers/storage.conf.
|
||||
mkdir -p ~/.config/containers
|
||||
printf '[storage]\ndriver = "overlay"\n\n[storage.options.overlay]\nmount_program = ""\n' > ~/.config/containers/storage.conf
|
||||
# Docker-compatible API socket for Testcontainers, Compose, and
|
||||
# the dev container docker-outside-of-docker feature.
|
||||
(podman system service --time=0 unix:///tmp/podman.sock >/dev/null 2>&1 &)
|
||||
(podman system service --time=0 unix:///tmp/podman.sock >/tmp/podman-service.log 2>&1 &)
|
||||
for i in $(seq 50); do [ -S /tmp/podman.sock ] && break; sleep 0.1; done
|
||||
if [ ! -S /tmp/podman.sock ]; then
|
||||
echo "Podman API socket didn't start:" >&2
|
||||
cat /tmp/podman-service.log >&2
|
||||
exit 1
|
||||
fi
|
||||
EOT
|
||||
|
||||
env = {
|
||||
@@ -54,7 +75,37 @@ resource "coder_agent" "main" {
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_pod" "main" {
|
||||
resource "kubernetes_persistent_volume_claim_v1" "home" {
|
||||
metadata {
|
||||
name = "coder-${data.coder_workspace.me.id}-home"
|
||||
namespace = var.namespace
|
||||
labels = {
|
||||
"app.kubernetes.io/name" = "coder-pvc"
|
||||
"app.kubernetes.io/instance" = "coder-pvc-${data.coder_workspace.me.id}"
|
||||
"app.kubernetes.io/part-of" = "coder"
|
||||
//Coder-specific labels.
|
||||
"com.coder.resource" = "true"
|
||||
"com.coder.workspace.id" = data.coder_workspace.me.id
|
||||
"com.coder.workspace.name" = data.coder_workspace.me.name
|
||||
"com.coder.user.id" = data.coder_workspace_owner.me.id
|
||||
"com.coder.user.username" = data.coder_workspace_owner.me.name
|
||||
}
|
||||
annotations = {
|
||||
"com.coder.user.email" = data.coder_workspace_owner.me.email
|
||||
}
|
||||
}
|
||||
wait_until_bound = false
|
||||
spec {
|
||||
access_modes = ["ReadWriteOnce"]
|
||||
resources {
|
||||
requests = {
|
||||
storage = "${data.coder_parameter.home_disk_size.value}Gi"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_pod_v1" "main" {
|
||||
count = data.coder_workspace.me.start_count
|
||||
metadata {
|
||||
name = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}"
|
||||
@@ -66,11 +117,15 @@ resource "kubernetes_pod" "main" {
|
||||
}
|
||||
}
|
||||
spec {
|
||||
node_selector = var.node_selector
|
||||
# The codercom/example-podman image is published for amd64 only.
|
||||
node_selector = merge({ "kubernetes.io/arch" = "amd64" }, var.node_selector)
|
||||
security_context {
|
||||
run_as_user = 1000
|
||||
fs_group = 1000
|
||||
run_as_non_root = true
|
||||
# Skip the recursive chgrp on every start, which would rewrite the
|
||||
# subuid-owned image layers in the Podman graph root.
|
||||
fs_group_change_policy = "OnRootMismatch"
|
||||
}
|
||||
container {
|
||||
name = "dev"
|
||||
@@ -90,15 +145,17 @@ resource "kubernetes_pod" "main" {
|
||||
memory = "4Gi"
|
||||
}
|
||||
}
|
||||
# The Podman graph root lives under home on this ext4 or xfs volume,
|
||||
# so kernel overlay works and images persist across restarts.
|
||||
volume_mount {
|
||||
name = "container-storage"
|
||||
mount_path = "/home/coder/.local/share/containers"
|
||||
name = "home"
|
||||
mount_path = "/home/coder"
|
||||
}
|
||||
}
|
||||
volume {
|
||||
name = "container-storage"
|
||||
empty_dir {
|
||||
size_limit = "20Gi"
|
||||
name = "home"
|
||||
persistent_volume_claim {
|
||||
claim_name = kubernetes_persistent_volume_claim_v1.home.metadata[0].name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user