KV Store and Persistence

On the platform, the runtime's object store is served by the orchestrator: a deployment-scoped, versioned key/value store backed by Postgres. It is what makes state survive pod restarts and stay consistent across a deployment's replicas. Everything built on the runtime object store (object-read/object-write, agent memory, runtime-written secrets) rides on it in clustered runs.

The model

Every entry is keyed by (deploymentId, namespace, key) and carries a monotonically increasing version.

The store is deployment-scoped: a deployment's replicas share one store, and different deployments, even of the same integration, never see each other's data. The scope key is OCTO_DEPLOYMENT_ID, injected into every pod.

Namespaces isolate system state from user data (system vs user), and secret namespaces are separate again (below). Picking a namespace also picks a durability tier.

Writes are compare-and-swap: a write carries the version it expects (0 to create) and fails with a conflict when the entry changed underneath it, so concurrent replicas never silently overwrite each other. See Clustering.

Values are opaque bytes, capped at 1 MiB per entry. Keys may contain /.

How the runtime talks to it

The runtime's k8s services module (selected by RUNTIME_SERVICES_MODULE=k8s) implements the object store as an HTTP client of the orchestrator, using the ORCHESTRATOR_URL injected at deploy time:

GET/PUT/DELETE /deployments/{id}/kv/{namespace}/{key}

The version travels in the X-Object-Version header both ways: reads return the current version, writes send the expected version and get the new one back. A mismatch is a 409 the runtime surfaces as its version-conflict error. In standalone runs the same object-store interface is served locally, so flow YAML doesn't change between the two.

Durability tiers

An aggregator's in-flight group state must survive a restart, while a memoized response body with a 60-second TTL need not cost a database round-trip. You choose a tier, persistent or volatile, by choosing a namespace, and the _volatile suffix on its name says which you get:

The suffix is the only thing that routes. A key is never rewritten and never carries the namespace: the two travel separately and meet only in the Redis key layout, octo:kv:<deployment>:<namespace>:<key>, where they stay separate segments. The deployment comes first so one deployment's keys can be swept together, and the key comes last so it may contain a colon while a namespace may not, which is refused rather than escaped: namespace a:b with key k and namespace a with key b:k would otherwise be the same Redis key, and two callers who believed they were in separate keyspaces would share one, versions included.

Volatile makes no durability promise at all. Your value may vanish on a restart, and on the platform it may be evicted while the process is still running, because the bundled Redis runs with maxmemory-policy allkeys-lru and no on-disk persistence. Use it for state whose loss costs you a recompute, never for state whose loss costs you correctness.

Both tiers are versioned and compare-and-swap in exactly the same way, so you move code between them without changing it.

How a pod reaches each tier

Integration pods reach Redis directly, exactly as they reach NATS: the orchestrator injects REDIS_URL alongside NATS_URL at deploy time, and the runtime dials it itself. Redis must therefore be reachable from the integration pods, not only from the orchestrator. With the bundled Redis it is: same namespace, no credentials. With a managed one behind externalRedis.existingSecret, the orchestrator binds the pods' REDIS_URL to the same Secret by reference, so the password never enters a Deployment manifest.

The persistent tier still goes through the orchestrator, because that is where the database and the encryption key are. That is also why secrets are never volatile: a secret always takes the orchestrator path and is always encrypted at rest.

Watching the tiers

The Storage health tab beside the object browser reports what a reachability check cannot: Redis memory against its ceiling, the hit rate, how many keys have been evicted, and on the database side how much of the observability service's connection pool is in use and how large kv_store has grown. Eviction counts are the answer to "my volatile object is not there": the tier working as designed, not a fault.

Standalone persistence

Run octo run and it serializes your persistent namespaces into --storage-dir (default ./octo-store, or $OCTO_STORAGE_DIR), one file per namespace, so your objects survive Ctrl-C the way a deployment's survive a rollout. Writes are coalesced and flushed shortly after they happen, and a graceful stop flushes synchronously; a kill -9 can lose the last fraction of a second. Set the directory to an empty value to keep the whole store in memory and leave nothing behind.

Two kinds of namespace never reach that directory. Volatile namespaces do not, because that is what volatile means. Secret namespaces do not either, because a standalone runtime has no encryption key and writing an OAuth refresh token into your working directory in cleartext would be the wrong trade. Standalone secrets keep a memory-only lifetime, so expect a restart to re-authenticate.

Secret namespaces

Namespaces ending in _secrets (e.g. user_secrets) are encrypted at rest: the orchestrator encrypts values with AES-GCM before they reach Postgres and decrypts on read, using the key configured as KV_ENCRYPTION_KEY (the Helm chart's kv.encryptionKey, or kv.existingSecret naming a Secret that holds it). The key is base64-encoded and must decode to 16, 24 or 32 bytes; use 32 for AES-256. This is what backs the runtime secret store in clustered runs, so a flow that writes a secret writes to a secret namespace of this same table.

Plain namespaces are stored as-is, so ordinary KV traffic pays no encryption cost. When no key is configured, secret-namespace operations are rejected (a 503) while plain KV keeps working.

A namespace that is both is refused with a 400. user_secrets_volatile asks the store to hold a credential somewhere that keeps it in the clear and may evict it, which is the one combination that must never exist. The check strips one suffix and re-tests rather than reading the last one, because composing them hides one behind the other: user_secrets_volatile does not end in _secrets, and user_volatile_secrets does not end in _volatile.

The object browser

The platform UI's object view uses a JSON facade over the same store, fixed to the user namespace family and adding the listing the raw KV routes lack:

GET  /deployments/{id}/namespaces
GET  /deployments/{id}/objects
GET/PUT/DELETE /deployments/{id}/objects/{key}

Secret namespaces are filtered out of the browser; their values are never listed or displayed.

Persistence guarantees

• Persistent-tier storage is Postgres (the kv_store table), so entries survive pod restarts, rollouts, and scale changes. The store's lifetime is the deployment's, not a pod's.
• Writes are transactional and serialized per key (a row lock plus the CAS check), so a successful write is durable and its version unique. Losers of a concurrent race get a clean conflict.
• Cleanup on undeploy is best-effort and sweeps both tiers. Entries have no foreign key to the deployment; undeploying deletes them after the workload is gone, and a failure in either tier is logged while the other is still attempted, leaving harmless orphans rather than failing the undeploy. Data does not survive undeploy by contract: redeploying an integration creates a new deployment id and therefore an empty store.
• Durability follows your Postgres setup. The store is exactly as durable as the platform database; see Deploying with Helm for storage configuration.
• The volatile tier guarantees you nothing, by design. See Durability tiers.