For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Tune the database, controller replicas, and the node pools that host Substrate Workers before you run kagent in production.

A default kagent installation runs one controller replica against the development database that kagent install creates, and schedules agents onto any node in the cluster. Each default works for a trial installation, and each needs a decision before you run kagent in production: which database to run, how many controller replicas to keep, and how to treat the nodes that host Workers.

Controller configuration

Controller configuration fields are set in the config section of the kagent Helm chart values. The chart renders these settings into a configuration file at /etc/kagent/controller.yaml, which the kagent controller reads.

Settings that describe how Kubernetes runs the controller stay in the Helm values outside of config. These include images, probes, volumes, Services, and Secret references. A Secret reference belongs to both groups. The chart projects the Secret into the controller pod, then writes the path of the projected file into the configuration file.

config:
  logging:
    level: info
  server:
    grpcReflection: false
    auth:
      mode: insecure
  sessions:
    idleTtl: 604800s
  runtimeDefaults:
    logging:
      level: info

The controller enforces the following rules when it reads the settings at start time.

• Durations are in seconds. A duration takes protobuf JSON notation. The value is a number with an s suffix, as in 604800s or 0.5s. A Go duration string such as 168h fails validation.
• The document is strict. The controller rejects an unknown field, a duplicate key, a null value, a YAML alias or merge, and a file with more than one document.
• Settings load once. The controller validates the whole file at startup, passes the result to its owners, and reads nothing from the environment afterward. A change therefore needs a restart. The chart triggers that restart by checksumming the ConfigMap into the pod.

To check your settings before you apply them, render the document that the chart builds from your Helm values.

helm template kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 1.0.0-alpha13 \
  --namespace kagent \
  --values kagent.yaml \
  --show-only templates/controller-configmap.yaml \
  | yq '.data."controller.yaml"' > controller.yaml

Then run kagent config validate against the rendered document.

kagent config validate --component controller --config controller.yaml

After you change these values, you can verify their validity by checking the controller’s logs, as an invalid field prevents the controller from starting.

Runtime defaults

The only section of settings that the controller does not apply to itself is config.runtimeDefaults. The controller compiles those settings into each agent runtime that it starts. If you want a setting to apply to both the controller and the agents, such as the log level, the telemetry endpoints, or the content capture decision, you must configure that setting in both config and config.runtimeDefaults.

Each config.runtimeDefaults setting is the installation’s default. A Harness can replace most of these settings for the agents that run on it by naming the equivalent variable in its spec.env field. A small set of names is reserved instead, such as the certificate trust variables that let a runtime reach the egress gateway. For the reserved names and the per-harness differences, see Environment variables on an Actor.

Choose a database

kagent stores SessionsSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more and compiled revisionsRevisionThe compiled, immutable output of one Agent, identified by a content digest. A Session runs the revision it was created from for its whole life, so editing the Agent affects only sessions created afterward. in PostgreSQL. The kagent chart does not deploy a database of its own. It reads a connection string from a Kubernetes Secret, and database.postgres.connectionStringSecretRef names that Secret and the key inside it.

database:
  postgres:
    connectionStringSecretRef:
      name: kagent-postgres
      key: connectionString

kagent install creates a development database in the cluster, so an installation does not need an external database to start. Run your own PostgreSQL instance for production, so that the data outlives the cluster that kagent runs on.

The development database

kagent install creates a database for you unless you pass --skip-database-setup. It deploys a single-replica kagent-postgresql Deployment backed by a 500Mi PersistentVolumeClaim, creates the roles that kagent and Agent Substrate connect as, and writes the connection Secrets for both: kagent-postgres in the kagent namespace, and substrate-postgres-readwrite and substrate-postgres-owner in ate-system.

Connections to this database do not use a password. When database.postgres.clientCertificate.enabled is set to true, which kagent install does, each client authenticates with a Pod Certificate projected into its pod under /run/postgres.podcert.ate.dev. The connection string points its sslcert and sslkey parameters at those files.

Use an external PostgreSQL instance

For production, run PostgreSQL outside the cluster’s lifecycle so that a kagent uninstall cannot delete the data.

1. Create a Secret that holds the connection string.

   kubectl create secret generic my-postgres \
     --namespace kagent \
     --from-literal=connectionString="postgresql://user:password@postgres.example.com:5432/kagent?sslmode=verify-full"

2. Point the chart at your Secret in your Helm values file.

   database:
     postgres:
       connectionStringSecretRef:
         name: my-postgres
         key: connectionString
       clientCertificate:
         enabled: false
   config:
     database:
       role: kagent_owner
       schema: kagent
       vectorEnabled: true
       skipMigrations: false
       pool:
         maxConns: 20
         minConns: 2
   controller:
     replicas: 3

   database.postgres configures the Kubernetes packaging, including the Secret reference and the client certificate. config.database sets what the controller reads from its configuration file. For more information about these settings, see Controller configuration.

   Setting	Description
   database.postgres.connectionStringSecretRef	The Secret and key that hold the connection string. Keeping the string in a Secret keeps the credentials out of Helm values. The controller restarts to pick up a changed Secret.
   database.postgres.clientCertificate	Set enabled to true only for a database that authenticates clients with Pod Certificates, such as the one that kagent install creates. The connection string must then point its sslcert and sslkey parameters at the files under /run/postgres.podcert.ate.dev. An external instance that authenticates with a password, such as the one in the preceding example, leaves enabled at false.
   config.database.role	The role that kagent assumes on each connection. Give each installation that shares one database a distinct role.
   config.database.schema	The schema that holds kagent’s tables. config.database.vectorSchema names the schema that holds the shared pgvector extension, and kagent needs USAGE on it.
   config.database.vectorEnabled	Set to true only when the instance has the pgvector extension installed. The setting enables the vector migration, and features that depend on it, such as long-term memory, fail without the extension. The development database does not include pgvector.
   config.database.pool	Connection pool sizing, through maxConns, minConns, maxConnIdleTime, and maxConnLifetime. The two durations take protobuf seconds notation, as in 1800s. Omit the fields to keep the pgx library defaults.
   config.database.skipMigrations	Set to true to stop the controller from running migrations at startup. The controller then verifies that the database is already migrated and fails if it is not. Apply the migrations from a pipeline before you install or upgrade.

3. Apply the values to the kagent Helm release.

   helm upgrade --install kagent \
     oci://ghcr.io/kagent-dev/kagent/helm/kagent \
     --version 1.0.0-alpha13 \
     --namespace kagent \
     --values kagent.yaml

Run multiple controller replicas

So that a controller failure does not stop reconciliation, set controller.replicas to a number higher than 1.

controller:
  replicas: 3

Leader election keeps the replicas from conflicting. One replica takes a Kubernetes lease and performs reconciliation, garbage collection, and scheduled runs. The other replicas stay ready and take over when the leader’s lease expires.

Sign push notification callbacks

Agent-to-Agent (A2A) push notifications are unsigned by default. A callback carries no Authorization header unless you supply an Ed25519 signing key, and the JSON Web Key Set (JWKS) endpoint returns 404. Push notifications still work without a key, so sign them only where a receiver must verify that a callback came from your installation.

To sign callbacks, generate one unencrypted PKCS#8 Ed25519 key and store it in a Secret in the kagent namespace.

1. Generate the signing key.

   openssl genpkey -algorithm ED25519 -out private-key.pem

2. Store the key in a Secret. The Secret must hold the key under the private-key.pem key.

   kubectl create secret generic kagent-push-signing -n kagent \
     --from-file=private-key.pem

Then name that Secret in your kagent Helm values.

controller:
  push:
    signing:
      secretName: kagent-push-signing
config:
  gateway:
    push:
      issuer: https://kagent.example.com

With a signing key configured, each callback attempt carries a fresh five-minute Ed25519 JWT in an Authorization: Bearer header, and the /.well-known/jwks.json endpoint publishes the matching public key.

Every controller replica must load the same key. After you rotate the key, restart all controller pods together, because replicas that run on different keys publish different JSON Web Key Sets.

Node operations

A WorkerWorkerA pre-started, sandboxed pod that hosts at most one Actor at a time. Actors are multiplexed across a pool of Workers over time.Learn more is an ordinary Kubernetes pod, so every node event reaches the ActorsActorThe sandboxed unit of compute, provided by Agent Substrate, that runs a Session's conversation loop. Every Session is backed by one.Learn more that this Worker hosts. What the event costs depends on what each Actor was doing when the node went away.

• A suspended Actor costs nothing. Its state is already written to snapshot storage, and it occupies no Worker while it waits, so the next message resumes it onto whichever Worker in the pool has availability.
• An awake Actor is lost. The Session’s in-flight state lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends.

A lost Actor moves to ACTOR_STATE_CRASHED, and Agent Substrate refuses resume and suspend on an Actor in this state. Through kagent, deleting the Actor and starting a new SessionSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more is the only available option, so the Session ends with the node.

Many Actors on a node are suspended at any moment, so losing the node crashes only the ones that are mid-turn. Those Actors still lose their Sessions, and a node pool that runs Workers therefore needs stricter rules than the rest of the cluster. A Worker pod that Kubernetes deletes gives its Actors time to suspend. Node auto-upgrade, a spot reclaim, and a pool scale-down do not.

Graceful pod deletion

Deleting a Worker pod is safe. Kubernetes forwards SIGTERM into the Actor’s containers, the Worker drains for up to 30 minutes inside a pod termination grace period of 3600 seconds, and an Actor that suspends inside this window keeps its state and stays resumable. Rolling a workerImage change through a pool therefore finishes any in-flight turns rather than cutting them off.

Node auto-upgrade

A reclaimed node is not a graceful deletion, and node auto-upgrade is the most likely way to hit one. Google Kubernetes Engine (GKE) enables auto-upgrade by default and runs it on Google’s maintenance schedule, not yours.

gcloud container node-pools update "${NODE_POOL}" \
  --cluster "${CLUSTER_NAME}" --location "${CLUSTER_LOCATION}" \
  --no-enable-autoupgrade

Spot and preemptible nodes

A provider reclaims spot and preemptible nodes on its own schedule, and each reclaim crashes every Actor that was awake on the node. A pool on reclaimable capacity therefore trades a lower bill for Sessions that end without warning. Use these nodes only where the agents on this pool can afford to lose a turn.

WorkerPool scale-down

When you scale a serving WorkerPool down, it removes pods without suspending the Actors on them. This strands Sessions in the same way that a reclaimed node does. For pool sizing and the rest of the Substrate runtime settings, see Tune Agent Substrate.

Expire idle Sessions

Every SessionSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more holds database records and pins a runtime revision for as long as it exists, including a Session that is never opened again. The kagent controller deletes idle Sessions on a timer. Deleting idle sessions bounds the records that a kagent installation stores as well as the revisions that it holds open.

A leader-only worker sweeps Sessions once a minute and deletes every Session that has been idle past the limit, using the same deletion workflow that a client delete uses. To change this sweep interval, you can configure config.sessions.expirationInterval.

To choose how long a Session can be idle before the worker deletes it, set config.sessions.idleTtl in your kagent Helm values. The value is a duration in seconds with an s suffix, and it defaults to 604800s, or seven days. Setting it to 0s turns the worker off, retries included, and a negative value is rejected.

config:
  sessions:
    idleTtl: 2592000s

Durations in the config block follow protobuf JSON notation. Seconds are the only supported unit, and a Go duration string such as 720h fails validation and stops the controller from starting.

This value governs every Session in the installation. kagent does not provide a per-agent override, and it does not cap the value.

Idle time runs from whichever is later: the Session’s creation, or its most recently stored A2A event. Reads, renames, lifecycle calls, and retried writes do not reset it. A Session that is only ever listed still expires.

A fork is the case where the creation time decides. A fork replays its source Session’s events and keeps their original timestamps, so its most recently stored event is older than the fork itself. Idle time therefore runs from the fork’s own creation, which gives it a full interval rather than an almost expired one.

Work in progress is never deleted. A Session with a running task survives past the interval, and so does one with a turn waiting at INPUT_REQUIRED or AUTH_REQUIRED. A pending lifecycle operation or a checkpoint being captured also holds a Session open.

A Session share carries its own lifetime, separate from the Session’s idle limit. A client sets CreateSessionShareRequest.ttl when it creates a share, and config.sessions.shareMaxTtl caps what a client may ask for. A positive cap rejects any request that exceeds it, and supplies the lifetime for a request that omits ttl. The default 0s leaves a share unbounded unless the client sets ttl, and a negative value is rejected at startup.

config:
  sessions:
    shareMaxTtl: 86400s

To watch the sweep, scrape kagent_session_expired_total, which counts the Sessions that it removed. Each removal also logs expired idle session at debug level with the Session’s ID and how long it had been idle.

How configuration changes reach agents

kagent watches the Secrets and ConfigMaps that a HarnessHarnessA Kubernetes custom resource defining how an agent is allowed to run: which harness executes the agent, its workload image, and its WorkerPool and snapshot storage. An Agent pairs it with the AgentTemplate that it runs.Learn more and AgentTemplateAgentTemplateA Kubernetes custom resource defining what an agent does: its model, system prompt, tools, skills, and plugins. It runs only once an Agent pairs it with a Harness.Learn more reference, such as the API keys and TLS certificates in a ModelConfigModelConfigA Kubernetes custom resource naming one model at one provider, along with the credentials to reach it. An AgentTemplate references one by name, and every agent compiled from that template calls the model that it names.Learn more. An edit to one of them recompiles the pair into a new revision.

A new revision does not reach the Sessions that are already running. A Session is pinned to the revision that it was created from and keeps that revision for life. To move an existing conversation onto new configuration, create a new Session.

The model provider API key is the exception. Agents do not hold the key. Instead, the Agent Substrate egress gateway reads the key from the Secret and adds it to each model request, so a rotated key reaches running Sessions after you restart the egress gateway. See Rotate the model provider API key.

This behavior differs from kagent 0.x, where an agent ran as a Deployment and a secret change restarted its pods.

Rotate the model provider API key

To rotate the model provider API key, update the Secret that holds the key. Then you can restart the Agent Substrate egress gateway so that running Sessions use the new value.

Manage the Secret yourself in a production installation. You then rotate the key at its source, and kagent reads the current value.

1. Update the key.

   You manage the SecretHelm manages the Secret
   When providers.<provider>.apiKey is empty and apiKeySecretRef names a Secret that you create, such as one that an ExternalSecret syncs, the chart does not render the Secret. Update the Secret wherever you manage it.

   When you set providers.<provider>.apiKey in your Helm values, the chart creates the Secret and Helm owns it. Rotate the key through Helm rather than by editing the Secret, because a direct edit does not last. With Helm 3, the next helm upgrade --reuse-values writes the previous key back. With Helm 4, which applies changes on the server side, the next upgrade fails with a field ownership conflict on the Secret.

   Upgrade the kagent release with the new key. The following example uses OpenAI. For another provider, replace openAI with the provider’s key in the providers values.

   helm upgrade kagent \
     oci://ghcr.io/kagent-dev/kagent/helm/kagent \
     --version 1.0.0-alpha13 \
     --namespace kagent \
     --reuse-values \
     --set-string providers.openAI.apiKey="${NEW_OPENAI_API_KEY}"

2. Restart the Agent Substrate egress gateway. The egress gateway can keep sending the previous key after the Secret changes, and a restart makes it use the new key right away.

   kubectl rollout restart deployment/atenet-egress -n ate-system

Route agent traffic through a proxy

A proxy puts agent-to-agent and agent-to-MCP calls on the same egress path as the rest of your cluster. Your existing inspection, logging, and policy then cover them too. When agents and MCP servers sit behind an API gateway or proxy, point kagent at the proxy endpoint.

proxy:
  url: "http://proxy.kagent.svc.cluster.local:8080"

The controller rewrites internally built Kubernetes URLs to the proxy and sets the x-kagent-host header so that the proxy routes each request to the correct backend. The rewrite covers one agent invoking another as a tool, and an agent calling a RemoteMCPServer at an internal URL. An external URL, such as a RemoteMCPServer at https://external.example.com, is left alone.

Scrape controller metrics

The controller serves a Prometheus /metrics endpoint, turned off by default. To enable the endpoint, grant Prometheus access to it, and review the metrics that the controller reports, see Metrics.

Next steps