Destinations

A destination is where an organisation’s telemetry is written: a Prometheus remote-write endpoint (Mimir), a Loki push endpoint, or an OTLP endpoint. Org admins manage them on the Destinations page. Wizards ship to them by name: the destination a wizard step picks becomes that pipeline’s writer (prometheus.remote_write or loki.write), including its authentication.

The URL

Enter the full endpoint URL the collector writes to, for example https://mimir.example.com/api/v1/push or https://loki.example.com/loki/api/v1/push. Shepherd puts it into the writer exactly as written and adds no path.

Tenant

For a multi-tenant Mimir or Loki, set the destination’s Tenant ID. Wizard pipelines that ship there send it as X-Scope-OrgID: as a header on prometheus.remote_write, and as loki.write’s own tenant_id. Leave it empty to send no tenant. It follows Mimir’s rule: letters, digits and ! - _ . * ' ( ), at most 150 characters, no slashes or spaces.

Any tenant your backend expects is allowed except one that belongs to another organisation in Shepherd (the tenant ID an application administrator gave it). That is refused when you save, and refused again when a wizard pipeline is generated, in case the other organisation was given that tenant ID later. The other way round, an administrator cannot give an organisation a tenant ID another organisation’s destination already sends.

This guards a destination’s tenant only. A pipeline you write by hand can set any header, X-Scope-OrgID included; Shepherd does not inspect it.

A pipeline picks up a tenant change when it is regenerated: saving a destination regenerates every wizard pipeline that names it. Until then the configuration collectors are served keeps the tenant it was generated with. A pipeline that ships to more than one destination is regenerated as a whole, so saving any one of them also starts sending a tenant already stored on the others.

Authentication from a Kubernetes Secret

Shepherd never holds a telemetry backend’s credentials. For an authenticated destination you name a Kubernetes Secret that already exists on every cluster whose collectors ship there, and the collector reads it at runtime with remote.kubernetes.secret. The served config contains the Secret’s namespace, its name and the key names below, never the values.

OAuth2 scopes are set on the destination itself (the OAuth2 scopes field, stored as extra.oauth2_scopes), not in the Secret. They are not sensitive, and because Shepherd never reads the Secret it could not tell whether an optional key is there.

Create the Secret in the namespace you name on the destination:

kubectl -n monitoring create secret generic mimir-credentials \
  --from-literal=username=<user> \
  --from-literal=password=<password>

A basic-auth destination named prom-prod pointing at that Secret gives a wizard pipeline this writer:

remote.kubernetes.secret "metrics_auth" {
  namespace = "monitoring"
  name      = "mimir-credentials"
}

prometheus.remote_write "metrics" {
  endpoint {
    name = "prom-prod"
    url  = "https://mimir.example.com/api/v1/push"

    basic_auth {
      username = convert.nonsensitive(remote.kubernetes.secret.metrics_auth.data["username"])
      password = remote.kubernetes.secret.metrics_auth.data["password"]
    }
  }
}

TLS from a Secret or ConfigMap

For a backend behind a private CA, or one that requires a client certificate, open TLS options on the destination. As with authentication, the certificates stay on the spoke clusters: you name the objects, and the collector reads them at runtime with remote.kubernetes.secret or remote.kubernetes.configmap. TLS options need an https:// URL.

• The client certificate keys are those of a kubernetes.io/tls Secret, which is what cert-manager writes. A cert-manager Secret also carries ca.crt, so it can serve as both CA and client certificate; the collector then reads it once.
• Name a different CA key for a bundle published under another key, for example a trust-manager Bundle target.
• The kube-root-ca.crt ConfigMap in every namespace works as the CA when the backend’s certificate was issued by the cluster CA.
• There is no option to skip certificate verification: it would let the Secret-backed credentials above be sent to anyone on the path. Trust the backend’s CA instead.
• Check the CA key name. If the Secret or ConfigMap has no data key by that name, Alloy treats the CA as unset: the collector loads and reports healthy, then trusts only the system roots and fails at send time with x509 unknown authority. A missing tls.crt or tls.key is caught earlier — the collector refuses the whole config.
• These options apply to the writer’s endpoint only. An OAuth2 destination’s token URL is fetched without them, so a token endpoint behind a private CA still needs a CA the collector’s system trust store already has.

The served config holds only the references and key names:

remote.kubernetes.configmap "metrics_tls_ca" {
  namespace = "monitoring"
  name      = "backend-ca"
}

prometheus.remote_write "metrics" {
  endpoint {
    name = "prom-prod"
    url  = "https://mimir.example.com/api/v1/push"

    tls_config {
      ca_pem      = remote.kubernetes.configmap.metrics_tls_ca.data["ca.crt"]
      server_name = "mimir.internal.example"
    }
  }
}

What the collector needs to read Secrets and ConfigMaps

The collector’s service account needs get on Secrets in the Secret’s namespace: remote.kubernetes.secret reads the one named Secret with a GET when the config loads and again every minute, and never lists or watches. The Grafana Alloy Helm chart, and so k8s-monitoring’s collectors, grants Secret access cluster-wide by default; check what your chart version grants for ConfigMaps. If your collectors run with narrower RBAC, grant it in that one namespace. A destination whose CA is in a ConfigMap needs get on configmaps in that namespace (remote.kubernetes.configmap reads the same way) — the second rule below. The first rule also grants list and watch, which are not needed:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: alloy-read-destination-secrets
  namespace: monitoring
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alloy-read-destination-secrets
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: alloy-read-destination-secrets
subjects:
  - kind: ServiceAccount
    name: <collector service account>
    namespace: <collector namespace>

Shepherd’s Kubernetes test suite runs a real Alloy with this Role’s Secret rule and checks that its remote-write requests carry the Secret’s credentials.

When a change takes effect

The writer, auth block included, is generated when a wizard is previewed or committed. It is stored as part of the pipeline, so it passes the same validation as any other line and shows in the editor and its revision history. A wizard that names a destination the organisation does not have, or one of the wrong type, is refused with a message saying which.

Saving a change to a destination regenerates every wizard pipeline in the organisation that ships to it, from the answers stored with the pipeline, in the same step. Each regenerated pipeline passes the full validation gate, gets a new revision and an audit entry under your name, and collectors receive it on their next poll; the confirmation says how many were regenerated (Destination updated — 3 pipelines regenerated). Renaming a destination renames it in those pipelines too. If any of them would fail validation, nothing is saved and the message names the pipeline, the reason and what to do; each pipeline it names links to its page. The same happens when one of them was edited by hand after its wizard generated it: Shepherd never overwrites the edit. Such a pipeline is badged Not following wizard in the pipelines list and Hand-edited in its header, and the editor asks before the save that makes it one (Save a hand edit?). Open the pipeline and do one of three things, then save the destination again:

• Restore last wizard version drops the hand edit. The pipeline page offers it whenever a wizard pipeline no longer follows its wizard and has a revision its wizard wrote; the revision history marks those revisions wizard. It restores that revision as a new one, so the edit stays in the history, and puts back that revision’s matchers and enabled state too (the confirmation says so when they differ). A wizard revision from before Shepherd 0.15.0 that it cannot vouch for may not re-attach the pipeline; if the destination change is still refused, detach or delete it.
• Detach from wizard keeps the edit: the pipeline keeps its text, matchers, owner and history, becomes an ordinary pipeline, and no longer follows destination changes.
• Delete the pipeline.

Running the wizard again does not help: it creates a new pipeline.

Shepherd recognises a hand edit by a fingerprint of the text the wizard last wrote, so a newer version of a wizard does not make older pipelines look edited. A wizard pipeline generated before that fingerprint existed is compared with what its wizard would generate today instead, until its next regeneration records one. If that wizard has changed since, such a pipeline reads Not following wizard although nobody edited it, and Restore last wizard version offers its current revision, which re-attaches it.

A destination a wizard pipeline still ships to cannot be deleted. The refusal lists the pipelines, each linked to its page; detach them from the wizard, or delete them, first.

Pipelines generated before destination auth

Wizard pipelines committed before destination auth was rendered read their URL from sys.env("SHEPHERD_DEST_<NAME>_URL"), which nothing sets, and carry no auth. Convert them once with shepherd admin rerender-destinations, run with the server's own configuration, --dry-run first:

kubectl -n shepherd exec svc/shepherd -- /usr/local/bin/shepherd admin rerender-destinations \
  --config /etc/shepherd/shepherd.yaml --dry-run

It lists every pipeline it converts and every one it cannot (for example, one naming a destination that was deleted), and is safe to run again. Saving a destination also regenerates the pipelines that use it.

Tenant IDs stored before they were sent

A destination’s tenant ID was stored but not sent before Shepherd rendered it. Such a tenant reaches a pipeline when the pipeline is next regenerated, which moves that data into the named backend tenant. To see which destinations carry a tenant and which pipelines would change, run the same command with --all; it writes nothing until you add --apply, and never overwrites a pipeline edited by hand:

kubectl -n shepherd exec svc/shepherd -- /usr/local/bin/shepherd admin rerender-destinations \
  --config /etc/shepherd/shepherd.yaml --all