Kubernetes (Helm)

The chart is published as an OCI artifact at oci://ghcr.io/procoduck/charts/shepherd — nothing to clone, no helm repo add. Images are published alongside it to ghcr.io/procoduck/shepherd (and ghcr.io/procoduck/shepherd-simulator for the sandbox) for linux/amd64 and linux/arm64.

Shepherd needs a PostgreSQL and two secret values. Either can be yours or the cluster’s: point it at an existing database, or let CloudNativePG provision one; supply the encryption key and bootstrap password, or let External Secrets generate them. The database is the only thing worth backing up — Shepherd itself holds no state.

Installing

helm install shepherd oci://ghcr.io/procoduck/charts/shepherd --version 0.22.0 \
  --namespace shepherd --create-namespace \
  --set existingSecret=shepherd-secrets

The secret it names must exist first — see Secrets for what goes in it, or how to have it generated for you. The image needs no --set: image.registry defaults to ghcr.io/procoduck and the tag falls back to the chart’s appVersion, so each chart version installs the Shepherd release it was published for.

Chart version is not the app version. --version 0.22.0 is the chart’s; Shepherd’s is the chart’s appVersion (0.18.0 for chart 0.22.0). They move independently, so a chart-only change bumps only the chart. Run helm show chart oci://ghcr.io/procoduck/charts/shepherd to see which Shepherd release a chart version installs.

What the chart renders

Conditional on values: Ingress/shepherd or HTTPRoute/shepherd, HorizontalPodAutoscaler/shepherd, NetworkPolicy/shepherd, ServiceMonitor/shepherd, Cluster/shepherd-db, and the ExternalSecrets with their Password generators (two for Shepherd's secret, one more for the sandbox token). With receiver.enabled: the receiver tier’s Deployment, ConfigMap, Service, ServiceAccount and NetworkPolicy, all named shepherd-receiver, and — while receiver.applyTenantRoutes is on, the default — a Role/RoleBinding shepherd-tenant-routes on HTTPRoutes in the release namespace plus a ClusterRole/ClusterRoleBinding that reads only the HTTPRoute CRD, with the service-account token mounted on the Shepherd pods. That is the only Kubernetes API access Shepherd ever has.

Three more objects appear only during an install or upgrade and delete themselves when the migration succeeds: ConfigMap/shepherd-migrate, ServiceAccount/shepherd-migrate and, if you use an inline secrets: block, Secret/shepherd-migrate-secrets. They exist because Helm runs every hook before any normal resource, so the migration Job cannot see the runtime ConfigMap yet. If a migration fails you will find them sitting in the namespace next to the failed Job; that is expected, and the next attempt clears them.

Nothing is exposed outside the cluster unless you ask for it. To see exactly what an install would create before creating it, render it locally: helm template shepherd oci://ghcr.io/procoduck/charts/shepherd --version 0.22.0.

Migrations run before the new code

The migration Job is a Helm hook with hook-weight: -5, so on both install and upgrade the schema is brought current before the Deployment is touched. That ordering is the point: a rolling update that started new pods against an old schema would fail in a way that looks like an application bug. If the Job fails, the release fails and the running Deployment is left alone.

If this namespace has a ResourceQuota or LimitRange requiring every pod to declare resources, set migrations.job.resources. The Job is a pre-install hook, so a pod the namespace refuses to admit fails the whole install — before a single tracked resource exists, and surfacing minutes later as a --wait timeout rather than as the admission error it is.

Exposing Shepherd

Both ingress paths are off by default, and deliberately: an HTTPRoute needs the Gateway API CRDs, which a vanilla cluster does not have, so defaulting it on made a plain helm install fail with no matches for kind HTTPRoute.

Pick the one your cluster actually runs:

# Gateway API
--set route.enabled=true \
--set "route.hostnames[0]=shepherd.example.internal" \
--set "route.parentRefs[0].name=my-gateway"

# or a classic Ingress. Each host takes a list of paths -- a bare
# string is rejected, with the offending path named.
--set ingress.enabled=true \
--set ingress.className=nginx \
--set "ingress.hosts[0].host=shepherd.example.internal" \
--set "ingress.hosts[0].paths[0].path=/"

# or neither, for a look before you commit
kubectl -n shepherd port-forward svc/shepherd 8080:8080

In a values file the same Ingress is easier to read, and this is where TLS and a non-default pathType (Prefix, Exact or ImplementationSpecific) go. Wildcard hosts are fine:

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: "*.shepherd.example.internal"
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: shepherd-tls
      hosts: [shepherd.example.internal]

Set config.server.base_url to the URL collectors can reach. It is the address every claimed collector's beacon pipeline writes its self-monitoring metrics to, and the Shepherd URL Connect a cluster writes into the k8s-monitoring values, so leaving it at the placeholder means every collector reports to an address that does not exist. It plays no part in single sign-on: the OIDC redirect URL comes from config.oidc.redirect_url when the provider is declared in the chart, and from the browser's own origin when it is set up in the admin UI.

Set config.server.trusted_proxies if people sign in with local accounts. Without it, every browser behind the Ingress or Gateway shares one per-IP sign-in throttle bucket, which anyone can exhaust. See Configuration for ingress-nginx and NGINX Gateway Fabric examples. It is safe only with externalTrafficPolicy: Local on the proxy’s Service, and with nothing but your proxy able to connect from inside the list — that page explains both.

Upgrading

helm upgrade shepherd oci://ghcr.io/procoduck/charts/shepherd --version <new> \
  --namespace shepherd --reuse-values

Read the changelog first: each release since v0.9.0 opens with an Upgrade: block, and a chart minor that needs operator action has its own section in UPGRADING.md. Migrations run ahead of the new pods automatically.

Upgrading to chart 0.22.0: local sign-in and removed git files

An ordinary helm upgrade. Migration 0034 adds a column, a trigger and a one-row-per-link backfill on repo_links, and takes moments. Shepherd no longer trusts X-Forwarded-For, X-Real-IP or True-Client-IP from any client, so behind an Ingress or Gateway every browser shares one per-IP local sign-in throttle bucket until you set config.server.trusted_proxies (OIDC sign-in is unaffected). Git sync now deletes the pipeline of a file removed from a linked repository: pipelines whose files were removed before the upgrade go on the link’s next new commit, unless that backlog trips the mass-delete guard, which holds it back and shows the link as warning. Details in UPGRADING.md.

Upgrading to chart 0.21.0: one fleet reload

An ordinary helm upgrade; no migration. Every collector reloads its config once after the upgrade, including collectors nothing matched: the served config’s first line is now // Generated by Shepherd (it read // Generated by Shepherd prod on every deployment) and two spacing artefacts are gone, so every collector’s served bytes change. The new pods recompute every cached config an older pod computed before serving it, so the change reaches quiet organisations too. Plan it as a fleet-wide config push. While old and new pods both serve, a visual-builder save can be refused once; a wizard pipeline can read Not following wizard until you reload after the rollout; and a save served by an old pod skips the Save a hand edit? question, so avoid hand-editing wizard pipelines until the rollout finishes. The new graph view fetches a 1.6 MB Web Worker script, served uncompressed unless a proxy in front compresses it. Details in UPGRADING.md.

Upgrading to chart 0.20.0: one fleet reload, then decide on tenant IDs

An ordinary helm upgrade. Every collector serving a pipeline reloads its config once after the upgrade: the served config’s header now shows each pipeline’s real revision, and migrations 0032 and 0033 make the new pods recompute every served config an older pod computed — including one rewritten while the rollout was still running — so the change reaches quiet organisations too. A collector whose organisation is edited mid-rollout may reload one extra time. Destinations’ tenant IDs are now sent as X-Scope-OrgID; one stored before this release starts being sent only when its pipelines are regenerated, so list them first with shepherd admin rerender-destinations --all from a server pod (a dry run) and add --apply when the list is right. Details in UPGRADING.md.

Upgrading to chart 0.19.0: Kubernetes 1.32, then one command

Chart 0.19.0 needs Kubernetes 1.32 or newer; on an older cluster helm upgrade stops with a kubeVersion error before changing anything. After the upgrade, run shepherd admin rerender-destinations once from a server pod (first with --dry-run): it regenerates the wizard pipelines generated by earlier releases, so they write to their destination’s URL with its auth and pick up this release’s wizard fixes. Collectors served one of them reload once. Database Metrics pipelines need a separate step to read their connection string from a Secret. Details in UPGRADING.md.

Upgrading from before chart 0.18.0: a few seconds without endpoints

Chart 0.18.0 labels the server pods app.kubernetes.io/component: server and points the shepherd Services, the PodDisruptionBudget and the NetworkPolicies at that label, so they no longer catch the sandbox simulator. The running pods do not have the label yet, so the Services have no endpoints until the first new pod is Ready — typically a few seconds; collectors keep their last config meanwhile. To avoid even that, label the running server pods right before the upgrade:

kubectl -n shepherd label pod \
  -l 'app.kubernetes.io/instance=shepherd,app.kubernetes.io/name=shepherd,!app.kubernetes.io/component' \
  app.kubernetes.io/component=server
helm upgrade shepherd oci://ghcr.io/procoduck/charts/shepherd --version 0.22.0 \
  --namespace shepherd --reuse-values

Reach a server pod through the Service (kubectl exec svc/shepherd), not deploy/shepherd, whose selector cannot change and still matches the simulator. Details in UPGRADING.md.

Upgrading to chart 0.9.0 needs one manual step

Up to 0.8.x the runtime ConfigMap, Secret and ServiceAccount were rendered as Helm hooks, to solve an ordering problem: Helm runs every hook before any normal resource, so the migration Job could not otherwise see them. In 0.9.0 the Job gets its own short-lived copies and those three go back to being ordinary tracked resources.

Helm creates hook resources without ownership metadata, so your existing objects cannot be adopted and the upgrade stops. Hand them to Helm once — this changes no data, restarts nothing, and is safe to run while Shepherd is serving:

REL=shepherd; NS=shepherd

for obj in configmap/$REL secret/$REL-secrets serviceaccount/$REL; do
  kubectl get $obj -n $NS >/dev/null 2>&1 || continue
  kubectl label   $obj -n $NS app.kubernetes.io/managed-by=Helm --overwrite
  kubectl annotate $obj -n $NS \
    meta.helm.sh/release-name=$REL \
    meta.helm.sh/release-namespace=$NS --overwrite
done

The secret/ line applies only if you used an inline secrets: block; the loop skips whatever is not there. If you forget, the chart stops with a message naming the exact objects and repeating these commands rather than letting Helm fail one object at a time. The full note is in UPGRADING.md.

Three consequences of that change are worth knowing, two of them improvements:

• helm rollback now reverts your config. It did not before: rollback runs pre-rollback hooks, never pre-upgrade ones, so the ConfigMap stayed at the new content while the Deployment went back to the old image.
• helm uninstall now removes them — including the chart-rendered shepherd-secrets, which used to be left behind. If you relied on uninstall/reinstall preserving the encryption key, copy it out first.
• Values files are now validated strictly. Unknown keys and quoted booleans are rejected with the offending path named, so a file carrying an old typo — simulater:, replicaCount:, config.tracing.endpont: — that installed silently before will now fail. That is the point: quoting "false" used to turn a feature on.

High availability

The Deployment runs two replicas by default and holds no state, so replicas are free to add and safe to lose. A PodDisruptionBudget keeps one pod up through voluntary disruption such as a node drain — it is rendered only when more than one pod will ever be running, since a budget over a single pod deadlocks the drain it was meant to survive.

To hand the replica count to an autoscaler instead:

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80

# Required. Not optional.
resources:
  requests: { cpu: 100m, memory: 128Mi }
  limits:   { memory: 512Mi }

autoscaling.enabled requires resources.requests.cpu, and the chart refuses to install without it. A CPU-utilization HPA targets a percentage of the request; with no request there is nothing to take a percentage of, so the autoscaler sits in FailedGetResourceMetric forever and never scales — indistinguishable from one that has simply decided not to. Since resources is empty by default, the two defaults together produced exactly that. With autoscaling on, replicas is no longer rendered at all (otherwise every upgrade would snap the count back), and the PodDisruptionBudget follows minReplicas instead.

nodeSelector, tolerations, affinity and priorityClassName are all exposed. Nothing spreads replicas across zones for you — use affinity for that.

Verifying the install

kubectl -n shepherd rollout status deploy/shepherd
kubectl -n shepherd get pods -l app.kubernetes.io/component=server
kubectl -n shepherd logs job/shepherd-migrate     # if the release failed

Then sign in as the administrator created on first boot and continue with Agent tokens. Every value the chart accepts is listed in Helm values.