letsweb.biz
EN/PL Let's talk

Blog

Change a setting without a deploy: a scoped key in ACL'd Consul

Tomasz Kuśmierczyk & Claude ·

Sometimes you need one small piece of configuration that is (a) the same across every running instance of your app, (b) survives restarts, and (c) can change at runtime — without rebuilding or redeploying. A feature flag. A maintenance switch. A site-wide default. Reaching for a database or a config-file rebuild is overkill.

If you already run Consul, its KV store is the natural home — and this post shows how to do it safely when Consul’s ACLs are enabled (as they should be).

Why Consul KV

  • Shared. Every node’s agent talks to the same KV, so all replicas read one value.
  • Persistent. It lives in Consul’s state — survives restarts.
  • Watchable. Blocking queries push changes to your app in near real-time — no polling.
  • Already there. If you use Consul for service discovery, the KV store comes for free.

The ACL wrinkle (and why it’s a good thing)

A well-run Consul has ACLs enabled with default_policy = deny. Your app can’t read or write KV until you hand it a token — and that’s a feature, not a chore. The right move is least privilege: a token that can touch exactly the one key prefix it needs, and nothing else.

Step 1 — a narrowly-scoped policy

A policy is a set of rules. Grant write on only your prefix:

# myapp-config.hcl
key_prefix "myapp/config/" {
  policy = "write"   # "write" implies read
}
consul acl policy create -name myapp-config -rules @myapp-config.hcl

The prefix ends with / and is specific to your app. Don’t grant "" — that’s the whole KV tree.

consul acl policy create is a one-time upload, not a config file you place somewhere. The .hcl is just input you hand to the CLI once — the same way a .sql file is input to a database. The flow:

  you author   myapp-config.hcl       (rules, in a text file — anywhere)
       │
       ▼
  consul acl policy create -rules @myapp-config.hcl
       │   (the CLI reads the file and SENDS the rules to Consul over its HTTP API)
       ▼
  Consul stores the policy in its OWN internal state
       (its Raft database on the Consul *server* nodes — replicated, not your file)

After that command runs, the policy is data inside Consul, identified by a name and an ID; the file has done its job. So:

  • Where do I place the file so Consul uses it? Nowhere. Consul isn’t watching a folder for policy files — don’t drop it in /etc/consul.d/ expecting it to load. The file is throwaway input, the same way a .sql file is input to psql: the database stores the row in its own tables and never reads your .sql again.
  • How does Consul know where it is? It doesn’t look for a file. When a token (with this policy attached) makes a request, Consul looks the policy up by ID in its own state — no file lookup ever happens.
  • Which server has it? All the Consul server nodes, as part of their replicated Raft state — as Consul’s internal data, not as a copy of your .hcl.

So keep the .hcl in version control as source, and you can delete it from disk anytime — the live policy won’t notice.

Naming the key

A useful shape is <scope>/<app>/<name> (e.g. prod/billing/rate-limit): an environment/project scope, the owning app, then the setting. Lowercase, with hyphens inside a segment.

Consul KV is a flat store — there are no real folders. When you write myapp/config/feature-x, Consul doesn’t create a myapp folder containing a config folder; it stores one entry whose key is the literal string myapp/config/feature-x. The / is just a character in that string — no more special to the storage engine than a - or a letter (you could name a key myapp:config:x and Consul wouldn’t care). The hierarchy is a convention: the UI, kv get -recurse (the command that lists every key under a path), and key_prefix rules (the ACL rules that grant access to a whole group of keys at once) all treat / as a separator and match on string prefixes — so consistent /-naming makes related keys group, list, and get scoped together as if they lived in folders.

One distinction matters. A path ending in / (myapp/config/) is a prefix: it doesn’t hold a value itself — it stands for the whole group of keys that begin with it, e.g. myapp/config/feature-x, myapp/config/theme, myapp/config/rate-limit. You point a policy or a recursive list at it to act on all of them at once — grant a permission to the whole group, or read every key under it in one call. A path without a trailing slash (myapp/config/feature-x) is a leaf key: it’s the entry that actually holds a value. This maps onto Consul’s two ways of addressing keys in a policy — collectively or one at a time: key_prefix "myapp/config/" grants on every key under the prefix (present and future), while key "myapp/config/feature-x" grants on just that one exact key. Reach for the prefix rule and you scope the policy once — then add as many keys under it later as you like without touching the policy again, all covered by that single grant.

The same key path actually shows up in more than one place: the ACL policy — which has to grant the token access to that exact path — plus the one-off seed command (consul kv put …) and your app’s reads, writes, and watch. They all have to spell the path identically, and a mismatch fails quietly, not loudly.

The policy is the easy one to get wrong, because its prefix and the key are the same string in two roles: myapp/config/ (the policy’s prefix from Step 1) is what the token may touch; myapp/config/feature-x (the app’s key) is what the app touches. The prefix has to cover the key. Misspell either, so it no longer covers, and Consul returns access denied. Misspell only the app’s read and you instead get an empty result — a key that doesn’t exist isn’t an error. Either way nothing crashes; the setting just silently never takes effect, and you’re left hunting for the typo.

Step 2 — mint a token for that policy

Both this and consul acl policy create in Step 1 are administrative commands — they change Consul’s ACL system itself, so they need a management token, not the scoped token you’re about to mint. Without one, under default_policy = deny, they return 403.

A management token is one attached to Consul’s built-in global-management policy — full, unrestricted access to the whole cluster (it’s where acl = "write" comes from). You don’t create it for this task; it already exists. The very first one is the bootstrap token that consul acl bootstrap prints when ACLs are first enabled on the cluster — an operator keeps it in a secrets store (and can mint further management tokens from that same policy). It’s the operator credential used for setup, not something you ship with your app.

The consul CLI authenticates its own request to the cluster with the token in the CONSUL_HTTP_TOKEN environment variable (or a -token= flag) — it’s simply the credential the CLI presents, the same role the X-Consul-Token header plays for the HTTP API later. Export your management token first:

export CONSUL_HTTP_TOKEN="$MGMT_TOKEN"   # a management token (acl = "write"); Step 1 needed this too
consul acl token create -description "myapp config" -policy-name myapp-config

This prints two ids:

  • AccessorID — a public handle, used to inspect or revoke the token later.
  • SecretID — the actual token. Treat it like a password.

Capture the SecretID straight into your secrets store. Never echo it to a log or commit it to a repo.

Step 3 — store the secret where your app reads secrets

Put the SecretID wherever your app already gets secrets — a secrets manager, your orchestrator’s variable/secret store, an injected env var. Not in the repository. The app reads it (e.g. CONSUL_TOKEN) at startup.

Step 4 — seed a default

With ACLs on, this write needs a token too — a bare consul kv put returns 403. It comes from the same CONSUL_HTTP_TOKEN as Step 2, but here it can be any token that writes the prefix: your management token, or the scoped token you just minted (it already has write on myapp/config/).

export CONSUL_HTTP_TOKEN="$SECRET_ID"   # the SecretID from Step 2 (or an operator token)
consul kv put myapp/config/feature-x on

Step 5 — read and write from the app

You don’t need a heavy client library — the HTTP API is plenty:

# read
curl -s -H "X-Consul-Token: $TOKEN" \
  http://127.0.0.1:8500/v1/kv/myapp/config/feature-x?raw

# write
curl -s -X PUT -H "X-Consul-Token: $TOKEN" \
  --data 'off' http://127.0.0.1:8500/v1/kv/myapp/config/feature-x

If Consul’s HTTP API is served over TLS, the only changes are the scheme and the port (HTTPS defaults to 8501), plus telling curl which CA to trust:

# read over HTTPS
curl -s -H "X-Consul-Token: $TOKEN" --cacert /etc/consul.d/tls/consul-agent-ca.pem \
  https://127.0.0.1:8501/v1/kv/myapp/config/feature-x?raw

Point --cacert at the CA that signed the agent’s certificate (not -k — skipping verification defeats the TLS). The token header and the rest of the URL are identical.

The local agent on 127.0.0.1 proxies to the cluster, so any node works.

Step 6 — watch for changes (the good part)

Polling is wasteful. Consul’s blocking queries let your app long-poll: pass the last index and a wait time, and the request hangs until the value changes (or the wait elapses). On change, every replica’s watch wakes up and converges:

# blocks up to 5 minutes until the key changes past ModifyIndex 42
curl -s -H "X-Consul-Token: $TOKEN" \
  "http://127.0.0.1:8500/v1/kv/myapp/config/feature-x?index=42&wait=300s"

In your app: keep a goroutine/thread looping on blocking reads, updating an in-memory value that the request path reads with zero latency.

Security checklist

  • Least privilege — the token writes one prefix, nothing else.
  • Secret hygiene — the SecretID is never printed, logged, or committed; it lives only in your secrets store.
  • Revocable — delete the token by its AccessorID anytime (consul acl token delete -id <AccessorID>) and the app loses access immediately.
  • Validate on read — treat the KV value as untrusted input: accept only known-good values, and fall back to a safe default for anything else.

What it’s good for

  • Feature flags — flip behavior without a deploy.
  • Maintenance mode — one key your edge/app reads to show a banner or return 503.
  • Tunable knobs — rate limits, thresholds, sizes, adjusted live.
  • Site-wide defaults — the use case that prompted this post.

A real example: an admin-set, site-wide default

We recently wired exactly this for a site. The goal: let an admin pick a setting from a dashboard and have every visitor see it — not a per-browser preference, but a true global default, changeable live.

The shape:

  1. A single KV key holds the current value.
  2. The app has a write-scoped token (as above) plus a watch loop, so all replicas share one value and converge instantly when it changes.
  3. The app serves the site’s HTML, and injects the current value into the page on the way out — so visitors get it at parse time, with no flash and no client-side fetch.
  4. A dashboard behind login (admin-gated) is the only thing that writes the key.

The result: an admin changes one control, the KV key updates, the watch fans it out to every replica, and the next page every visitor loads reflects it — no rebuild, no redeploy, identical for everyone, and it survives restarts. All from one ACL-scoped key.

Takeaway

For one small, shared, live-changeable setting, you don’t need a database or a deploy pipeline. A single ACL-scoped Consul KV key — least-privilege token, blocking-query watch, validate-on-read — gives you cluster-wide runtime config with a tiny footprint. The discipline that makes it safe is the same discipline that makes it boring: scope the token tightly, keep the secret out of your code, and treat the value as untrusted input.

← Back to blog

Comments