Receiver tier

The receiver tier is an OTLP/HTTP endpoint for applications — Lambdas, services, jobs — that sits behind your Gateway API gateway and forwards each tenant's data to your tenant-aware backends (Mimir, Loki, Tempo) carrying that tenant's X-Scope-OrgID. One receiver serves every tenant.

It is off by default (receiver.enabled: false): an opt-in feature. Its security review (containment, a tested off-switch, the gateway as the only ingress, a real-Alloy tenancy proof) is signed off, and it stays opt-in. It sets no CORS headers, so a browser sending OTLP from another origin needs that handled in front of it.

How it works

client  -->  https://gw.example/otlp/<route-segment>/v1/traces
        -->  your gateway's HTTPRoute
               strips the /otlp/<route-segment> prefix
               SETS X-Scope-OrgID: <tenant>   (replacing anything the client sent)
        -->  the receiver (Alloy, OTLP/HTTP)
               forwards the request's X-Scope-OrgID with its data
        -->  your backend

The tenant comes from the route, not the client: a client that sends its own X-Scope-OrgID has it overwritten by the gateway. The route segment in the URL is an identifier, not a secret — URLs leak through logs and browser history. Protect the endpoint with the gateway's own controls (origin allowlists, rate limits) and rotate segments when they leak.

Because the tenant header is only trustworthy when the gateway set it, the receiver accepts traffic only from your gateway's pods. Its NetworkPolicy admits the peers in receiver.networkPolicy.gatewayFrom and nothing else, so a pod elsewhere cannot reach it and claim a tenant.

Enable it

receiver:
  enabled: true
  exporters:
    traces:
      enabled: true
      endpoint: "http://tempo-distributor.observability:4318"
    metrics:
      enabled: true
      endpointFromSecret: {name: mimir-otlp, key: endpoint}
      authorizationFromSecret: {name: mimir-otlp, key: authorization}
  networkPolicy:
    # Your gateway's data-plane pods -- a podSelector is REQUIRED.
    gatewayFrom:
      - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: gateways}}
        podSelector: {matchLabels: {gateway.networking.k8s.io/gateway-name: edge}}
    # Where the receiver may send data. Egress is otherwise denied (DNS aside).
    egress:
      - to:
          - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: observability}}

Gateway API implementations label their data-plane pods gateway.networking.k8s.io/gateway-name: <gateway>, which is the selector shown. Credentials go in a Secret and reach the receiver only as environment variables — never in its ConfigMap.

What the chart refuses

• An empty gatewayFrom or egress — a receiver no gateway can reach, or one that cannot deliver, fails helm template with a message.
• A gatewayFrom peer without a non-empty podSelector — a namespace alone would admit any pod in it, and any such pod could assert any tenant.
• A gRPC listener — only OTLP/HTTP is served, because the tenant header is set by an HTTPRoute, which cannot front gRPC.
• A request-body limit left unset, or a batch cap below the batch size.

The receiver's config is rendered and validated when its pod starts. A config the renderer refuses stops the new pod at Init:Error — the receiver already running keeps serving — and kubectl logs <pod> -c render says why.

Tenant routes

Create a tenant route on Delivery → Tenant routes (or through TenantRouteService), naming the Gateway it attaches to. Shepherd then creates the route's HTTPRoute in its own namespace, pointing at the receiver, and waits for the gateway to report it attached. Only org admins create, rotate and revoke tenant routes, and an organisation needs a tenant ID before it can have any. The In cluster column shows the result:

• applied — the gateway accepted the route; traffic to its path reaches the receiver with the org's tenant.
• refused — the gateway will not attach it, and says why underneath. Usually the Gateway's listeners do not admit routes from Shepherd's namespace: set their allowedRoutes.namespaces to include it. Shepherd keeps checking, with a backoff of up to 30 minutes, so the route turns applied after the Gateway is fixed.
• error — anything else (the error is shown); retried with backoff.
• pending — not applied yet; the reconciler applies it on its next pass.
• not applied — Shepherd will not apply this route, and says why underneath: route apply is not running on this server (the receiver tier is disabled, receiver.applyTenantRoutes is false, or Shepherd is not running in a cluster), or it is a Faro route, which has no receiver yet.
• not routed — the route was revoked and its HTTPRoute is not removed yet.
• removed — the route was revoked, or its rotation overlap ended, and its HTTPRoute has been deleted.

Rotating a route keeps the old path working until its overlap ends, then Shepherd revokes it and deletes its HTTPRoute. Revoking deletes it straight away. Shepherd never creates or changes a Gateway, and never touches HTTPRoutes it did not create (it finds its own by the shepherd.io/tenant-route-id label).

Connect an app

Each active OTLP route has a Connect an app button (any org member can use it). Enter the service name and Shepherd shows the endpoint to send to, plus ready-to-paste configuration: a .env file, a Kubernetes container env block, Lambda environment JSON, Terraform, SAM and CDK fragments, and notes for OpenTelemetry SDKs. Most SDKs need no code change — the standard OTEL_EXPORTER_OTLP_* variables are enough.

The endpoint uses your gateway's public address. Set it once with receiver.publicBaseURL (it must be https://); otherwise users type it in the dialog. Only OTLP over HTTP is offered: the gateway finds the tenant from the URL path, which a gRPC client cannot set.

receiver:
  publicBaseURL: https://telemetry.example.com

To do this, the chart gives Shepherd's ServiceAccount a Role on httproutes in the release namespace and permission to read the one httproutes.gateway.networking.k8s.io CRD (to check the Gateway API version), and mounts its token on the Shepherd pod. None of it exists with the receiver off. To manage the HTTPRoutes yourself instead, set receiver.applyTenantRoutes: false and write each one in this shape:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: acme-otlp
  namespace: shepherd            # the receiver's namespace
spec:
  parentRefs:
    - name: edge
      namespace: gateways
  rules:
    - matches:
        - path: {type: PathPrefix, value: /otlp/acme-7f3k2}
      filters:
        - type: URLRewrite
          urlRewrite:
            path: {type: ReplacePrefixMatch, replacePrefixMatch: ""}
        - type: RequestHeaderModifier
          requestHeaderModifier:
            set: [{name: X-Scope-OrgID, value: acme}]
      backendRefs:
        - name: shepherd-receiver      # <release>-receiver
          port: 4318