Logs

Every process writes to stdout and stderr; shpyrd names the instance (web.1, worker.2), streams the lines live, keeps them bounded on the node, and forwards them wherever you keep your logs.

Live logs

shpyrd logs --project shop              # last 200 lines of every instanceshpyrd logs --project shop -f -p worker # follow one process type

The dashboard's Logs tab streams the same lines with a process filter, a text filter and level highlighting. There is nothing to set up: on shpyrd cloud and on a cluster you run yourself alike, this path reads from the Kubernetes API with nothing enabled.

The log agent

The log agent runs Vector on every node. It reads the container logs of project instances and turns every line into a structured event:

{"time": "...", "project": "shop", "process": "web", "instance": "web.2", "stream": "stdout", "level": "info", "msg": "request", "fields": {"method": "GET", "path": "/", "status": 200}}

Lines that are JSON get level and msg promoted and the rest kept in fields; plain lines keep msg as the text. Build output and the pods of attached databases and caches are not part of this stream.

Bounded on the node. Each container keeps at most 20 MiB of logs on disk (two files of 10 MiB, rotated by the kubelet), so an application logging at full speed cannot fill a node. History beyond that lives wherever you drain it.

Fenced in. A network policy lets the agent reach the drains' receivers (on the internet or in a platform namespace), DNS and the API server, and nothing in a project; only the platform reads its counters. On a local cluster the enriched stream is also written to the agent's own pod log (kubectl logs -n logs-system daemonset/vector), to see the pipeline at work; the cloud profiles leave that out.

Log drains

A drain forwards lines as they are written. Two kinds of receiver:

And three scopes:

• a project drain receives that project's lines; project admins add them on the project page or with --project;
• a workspace drain receives every project's lines of the workspace, labelled with the project; workspace admins add them on the workspace's Log drains page or with --workspace;
• a cluster drain (self-hosted) receives every project's lines of the platform; the operator of a cluster you run yourself adds them over a kubeconfig with --cluster.

shpyrd drains add https://in.logs.betterstack.com/ --header "Authorization: Bearer ..." --project shopshpyrd drains add https://http-intake.logs.datadoghq.com/api/v2/logs --header "DD-API-KEY: ..." --processes web --project shopshpyrd drains add syslog+tls://logs.papertrailapp.com:6514 --workspace acmeshpyrd drains list --project shopshpyrd drains remove in-logs-betterstack-com --project shop

The name defaults to the receiver's host. --processes limits a drain to some process types. Header values are stored in the cluster and never shown again, in the CLI or the dashboard. The pages show each drain's delivery status (Pending, Active with the last delivery time and line count, Failing with the error) refreshed every 30 seconds; a receiver that keeps failing raises a DrainFailing warning event on the drain, and after five minutes of failed checks a drain.failing entry in the project's activity (the cluster's, for a workspace or cluster drain).

Providers