Skip to content

Container Orchestration

Containerised sinks are the recommended approach for cloud and Kubernetes environments. The official Helm chart includes sink deployments alongside the rest of the Nexomatic platform.


The Nexomatic Helm chart (helm/nexomatic) includes sub-charts for the built-in sink node kinds. Each sub-chart deploys a Kubernetes Deployment that auto-scales based on CPU.

  • Kubernetes 1.24+ cluster
  • Helm 3
  • The Nexomatic Helm chart and its dependencies
  • A node created on the Nodes screen for each sink deployment, with its Client ID and Client Secret (see Node Credentials)

The sink sub-charts share a common configuration pattern. Environment variables come in two shapes: a plain env list (name/value), and the secret-backed envFromSecrets block — CLIENT_ID and CLIENT_SECRET are pulled from a Kubernetes Secret via secretKeyRef:

# Python sink (LOCAL:BASE)
internal-python-sink:
replicaCount: 2
image:
repository: localhost:32000/nexomatic-sink-local
tag: latest
env:
- name: HUB_HOST
value: "nexomatic-hub" # Kubernetes service name
- name: HUB_PORT
value: "50051"
- name: HUB_TLS
value: "" # Empty for in-cluster plaintext
- name: AUTH_METHOD
value: "client_credentials"
envFromSecrets:
clientId:
secret: nexomatic-sink-secrets
key: client-id
clientSecret:
secret: nexomatic-sink-secrets
key: client-secret
# PowerShell sink (LOCAL:POWERSHELL)
internal-pwsh-sink:
replicaCount: 1
image:
repository: localhost:32000/nexomatic-sink-powershell-local
tag: latest
env:
- name: HUB_HOST
value: "nexomatic-hub"
- name: HUB_PORT
value: "50051"
- name: HUB_TLS
value: ""
- name: AUTH_METHOD
value: "client_credentials"
envFromSecrets:
clientId:
secret: nexomatic-sink-secrets
key: pwsh-client-id
clientSecret:
secret: nexomatic-sink-secrets
key: pwsh-client-secret

The chart renders SERVER_URL itself (pointing at the release’s controller service) — any SERVER_URL entry you add to env is ignored.

Terminal window
kubectl create secret generic nexomatic-sink-secrets \
--namespace nexomatic \
--from-literal=client-id='<python-sink-client-id>' \
--from-literal=client-secret='<python-sink-client-secret>' \
--from-literal=pwsh-client-id='<powershell-sink-client-id>' \
--from-literal=pwsh-client-secret='<powershell-sink-client-secret>'
Terminal window
helm upgrade --install nexomatic ./helm/nexomatic \
--namespace nexomatic \
--create-namespace \
-f values.yaml

If your sinks need to access Azure resources (Key Vault, Storage, etc.) and you are running in AKS, you can use Workload Identity instead of storing credentials as secrets.

internal-pwsh-sink:
serviceAccount:
azureWorkloadIdentity: "<managed-identity-client-id>"
env:
- name: AUTH_METHOD
value: "entra_id"
- name: AZURE_TENANT_ID
value: "<azure-tenant-id>"
- name: AZURE_CLIENT_ID
value: "<managed-identity-client-id>"
- name: AZURE_FEDERATED_TOKEN_FILE
value: "/var/run/secrets/azure/tokens/azure-identity-token"

For smaller deployments or local development, Docker Compose is a convenient alternative.

# docker-compose.yml (sink only — assumes Hub is already running)
services:
sink-python:
image: your-registry/nexomatic-sink-local:latest
restart: unless-stopped
environment:
SERVER_URL: https://controller.example.com
HUB_HOST: hub.example.com
HUB_PORT: "50051"
HUB_TLS: "true"
CLIENT_ID: LOCAL:BASE
CLIENT_SECRET: "${SINK_CLIENT_SECRET}"
AUTH_METHOD: client_credentials
sink-powershell:
image: your-registry/nexomatic-sink-powershell-local:latest
restart: unless-stopped
environment:
SERVER_URL: https://controller.example.com
HUB_HOST: hub.example.com
HUB_PORT: "50051"
HUB_TLS: "true"
CLIENT_ID: LOCAL:POWERSHELL
CLIENT_SECRET: "${PWSH_CLIENT_SECRET}"
AUTH_METHOD: client_credentials

Store secrets in a .env file (not committed to source control):

.env
SINK_CLIENT_SECRET=<secret>
PWSH_CLIENT_SECRET=<secret>

Start the sinks:

Terminal window
docker compose up -d

Sinks are stateless and horizontally scalable. The Hub distributes tasks across all connected sinks that match the step’s placement and constraints.

  • Kubernetes: Adjust replicaCount or enable HPA (Horizontal Pod Autoscaler) in the Helm values.
  • Docker Compose: Use docker compose up --scale sink-python=4.

The sink containers serve HTTP health endpoints: the chart’s defaults are a liveness probe on /healthz and a readiness probe on /readyz, both on port 8006.

If a sink pod crashes, Kubernetes restarts it automatically. The Hub detects the disconnection and re-queues any in-flight tasks.


VariableRequiredDescription
SERVER_URL(chart-rendered)Controller API URL — the chart sets it to the release’s controller service
HUB_HOSTYesHub hostname (use the Kubernetes service name inside the cluster)
HUB_PORTYesHub port (default 50051)
HUB_TLSYestrue for TLS, empty for in-cluster plaintext
CLIENT_IDYesNode client id — via envFromSecrets.clientId
CLIENT_SECRETYesNode client secret — via envFromSecrets.clientSecret
AUTH_METHODYesclient_credentials, entra_id, google, github, or none
MAX_LOG_BYTESNoMaximum raw log size per task (default 5 MB)
TASK_TIMEOUTNoSeconds before a task is aborted (default 3600)
BACKPRESSURE_THRESHOLDNoQueue depth that triggers back-pressure signalling (default 10)