{
    "product": "Cenvero Stratum",
    "generated_at": "2026-08-03T07:17:51+00:00",
    "format": "cenvero-docs-v1",
    "document_count": 1,
    "documents": [
        {
            "slug": "operations",
            "title": "Operations & Troubleshooting",
            "category": null,
            "url": "https://www.stratum.cenvero.com/docs/operations",
            "headings": [
                {
                    "level": 1,
                    "text": "Operations & Troubleshooting"
                },
                {
                    "level": 2,
                    "text": "Systemd units"
                },
                {
                    "level": 2,
                    "text": "Logs"
                },
                {
                    "level": 2,
                    "text": "Network services: on / off"
                },
                {
                    "level": 2,
                    "text": "Changing settings"
                },
                {
                    "level": 2,
                    "text": "Updates"
                },
                {
                    "level": 2,
                    "text": "Health checks"
                },
                {
                    "level": 2,
                    "text": "Troubleshooting"
                },
                {
                    "level": 3,
                    "text": "The agent won't start"
                },
                {
                    "level": 3,
                    "text": "`cenvero-str-ctl` says the agent is unreachable (exit code 3)"
                },
                {
                    "level": 3,
                    "text": "License is not active"
                },
                {
                    "level": 3,
                    "text": "TLS certificate is pending"
                },
                {
                    "level": 3,
                    "text": "Node is not registered"
                },
                {
                    "level": 3,
                    "text": "\"REST API disabled\" — set an API token"
                },
                {
                    "level": 3,
                    "text": "Where to look first"
                },
                {
                    "level": 2,
                    "text": "Next steps"
                }
            ],
            "word_count": 1943,
            "markdown": "# Operations & Troubleshooting\n\nDay-to-day operation of a Stratum node: managing its systemd units, finding the\nlogs, turning individual network services on and off, changing settings, applying\nupdates, and a runbook for the issues you are most likely to hit.\n\nEverything here runs **on the node itself** as `root`. The operator tool,\n`cenvero-str-ctl`, talks to the running agent over a local socket — see the\n[CLI Reference](/docs/cli) for the full command surface.\n\n## Systemd units\n\nA node runs three systemd units, installed and enabled by the installer:\n\n| Unit | What it does |\n|------|--------------|\n| `cenvero-stratum` | The agent itself — control plane, APIs, and the in-kernel data plane. |\n| `cenvero-str-watchdog` | A small separate watchdog that monitors the agent and restarts it if it stops responding. It is bound to the agent (`BindsTo`), so it follows the agent's lifecycle. |\n| `cenvero-str-network` | A one-shot boot unit that brings up the host uplinks and the two bridges (`cnv-mgmt-br0`, `cnv-user-br0`) **before** the agent starts, so the interfaces are present when the agent attaches to them. |\n\nManage them with the usual `systemctl` verbs:\n\n```bash\n# Status and recent log lines for the agent\nsystemctl status cenvero-stratum\n\n# Start / stop / restart the agent (the watchdog follows it)\nsudo systemctl restart cenvero-stratum\nsudo systemctl stop cenvero-stratum\nsudo systemctl start cenvero-stratum\n\n# The watchdog and the boot-time network unit\nsystemctl status cenvero-str-watchdog\nsystemctl status cenvero-str-network\n```\n\nRestarting the agent is safe: because the data plane runs in the kernel and the\ninterfaces stay held across the swap, the brief restart does **not** tear down\nexisting traffic. Stopping the agent leaves the kernel data plane and the bridges\nin place.\n\n> Do not disable `cenvero-str-network` while the node is in service — it owns the\n> host networking the agent attaches to.\n\n## Logs\n\nThe agent logs structured JSON to standard out, which under systemd is captured\nby the **journal**. That is the first place to look:\n\n```bash\n# Follow the agent's live log\njournalctl -u cenvero-stratum -f\n\n# Last 200 lines\njournalctl -u cenvero-stratum -n 200\n\n# Since a time, or only errors and worse\njournalctl -u cenvero-stratum --since \"1 hour ago\"\njournalctl -u cenvero-stratum -p err\n\n# The watchdog has its own unit log\njournalctl -u cenvero-str-watchdog -n 100\n```\n\nOn-disk files live under **`/var/log/cenvero-str/`** — most notably the installer\nlog written during install. The agent's self-healing service prunes old rotated\nlog files (older than 7 days) from this directory when disk space runs low, so it\nis also where any rotated logs accumulate.\n\nTurn up the detail when you need it (see *Changing settings* below):\n\n```bash\nsudo cenvero-str-ctl config set log_level debug\nsudo systemctl restart cenvero-stratum\n```\n\nValid levels are `debug`, `info`, `warn`, and `error`. Set it back to `info` when\nyou are done — `debug` is noisy.\n\n## Network services: on / off\n\nThe agent exposes several **network services** you can independently enable or\ndisable. View them and their live state with `service status`, and flip each one\nwith `service <name> on|off`:\n\n```bash\n# See every service: its switch, its address:port, and whether it is listening\ncenvero-str-ctl service status\n\n# Turn services off / on\nsudo cenvero-str-ctl service rest off\nsudo cenvero-str-ctl service metrics on\n```\n\nThe toggleable services are:\n\n| Service | Name | What it is |\n|---------|------|------------|\n| REST API | `rest` | The HTTPS management & operator/billing API. |\n| gRPC API | `grpc` | The gRPC management API. |\n| WebSocket API | `websocket` | The live-events / streaming API. |\n| Metrics | `metrics` | The Prometheus `/metrics` endpoint. |\n| DNS | `dns` | The built-in DNS server. |\n| DHCP | `dhcp` | The built-in DHCP server. |\n\nThe verb-first form works too — `service off rest`, `service on dns` — and `-h`\non the group shows the help.\n\nA few notes:\n\n- **A change takes effect on the next agent restart.** After toggling a service,\n  apply it with `sudo systemctl restart cenvero-stratum`, then re-check with\n  `service status`.\n- **The local control socket can never be disabled** — it is how `cenvero-str-ctl`\n  reaches the agent (including the command you just ran). Trying to turn it off is\n  refused with an explanation.\n- **Disabling the REST API cuts off remote callers.** The REST API also serves the\n  operator/billing API (tenant suspend/resume/limit). Turning it off is allowed —\n  it is your call — but the command warns you first; the local socket and the panel\n  are unaffected.\n\nUnder the hood, `service <name> on|off` writes the matching off-switch into the\nsame operator local-overrides file that `config set` uses (see below), so a panel\nre-sync never clobbers your choice.\n\n## Changing settings\n\n`config set` adjusts a node's **operational settings** on a running node. It\nrecords your change in an operator local-override file that takes precedence over\nthe panel-delivered config and **survives a re-sync** — it never edits the signed\nnode config itself.\n\n```bash\n# Inspect the active configuration (paths, ports, API settings, services)\ncenvero-str-ctl config show\n\n# Change a setting, then restart to apply it\nsudo cenvero-str-ctl config set api_rate_limit 2000\nsudo systemctl restart cenvero-stratum\n```\n\nThe key names match what you see in `config show`, so the same name round-trips\nbetween the two. Commonly-set keys include:\n\n| Key | Meaning |\n|-----|---------|\n| `log_level` | Agent log level: `debug`, `info`, `warn`, `error` |\n| `api_bind_address` | IP the REST/gRPC/WebSocket APIs listen on (e.g. `0.0.0.0` or a management IP) |\n| `api_rate_limit` / `api_rate_burst` | API requests per minute per key, and the burst allowance (`0` rate disables the limiter) |\n| `api_allowed_ips` | Comma-separated IPs/CIDRs allowed to reach the API (empty = allow all) |\n| `port_rest`, `port_grpc`, `port_websocket` | API listen ports |\n| `metrics_bind_addr` | Prometheus metrics bind address `host:port` |\n| `dns_listen_addr`, `dns_allowed_clients` | Built-in DNS server bind address and allowed clients |\n\nRun `cenvero-str-ctl config set -h` for the full whitelist and per-key help.\n\n> **What you cannot set locally.** Identity, signing, and cluster-secret fields —\n> `node_id`, `license_server`, `api_token`, `gateway_shared_key`, and TLS/cluster\n> material — are **not** settable with `config set` by design. They are\n> panel-assigned or provisioned. Reaching for one returns a clear message telling\n> you where it is managed instead (e.g. TLS material is handled by\n> `cenvero-str-ctl tls …`, and the API token is rotated via the installer/panel).\n>\n> **Every `config set` change applies on the next restart.** It is saved\n> immediately, but the running agent only picks it up when it restarts.\n\n## Updates\n\nUpdates are **pull-based**: the agent polls a signed manifest from the panel and,\nwhen a newer fully-verified release for its channel is available, downloads,\nverifies, and applies it on its own. You can check or trigger this manually:\n\n```bash\n# What's installed now\ncenvero-str-ctl status\n\n# Ask the agent to check the manifest for a newer release (no download)\ncenvero-str-ctl update check\n\n# Apply an available update now (otherwise it applies on the next poll)\nsudo cenvero-str-ctl update apply\n```\n\n`update check` reports the current version, the latest on offer for your license\nchannel, and whether an update is available — without downloading anything. If the\npanel is unreachable it says so plainly rather than failing hard. `update apply`\nperforms the pull, checksum + publisher-signature verification, an atomic\nin-place swap, and a watchdog-supervised restart, with self-rollback if the\npost-restart health check fails. See [Upgrades](/docs/upgrades) for the full\nmodel (channels, downgrade protection, and rollback).\n\nA node in the **Frozen** license state keeps running its current version but\ncannot pull updates until the license is renewed.\n\n## Health checks\n\nThe agent continuously self-checks core subsystems (disk space, memory, the\ndatabase, the bridges, and the in-kernel data plane) and attempts an automatic\nrepair when one fails. Inspect or force a run:\n\n```bash\n# Latest result for every health check\ncenvero-str-ctl heal status\n\n# Force an immediate run now\nsudo cenvero-str-ctl heal check\n```\n\n---\n\n## Troubleshooting\n\n### The agent won't start\n\n**1. Read the journal** — it almost always names the cause:\n\n```bash\njournalctl -u cenvero-stratum -n 100 --no-pager\nsystemctl status cenvero-stratum\n```\n\n**2. Confirm you are running as `root`** and the host network came up:\n\n```bash\nsystemctl status cenvero-str-network\n```\n\nThe network unit must succeed first — it creates the bridges the agent attaches\nto. If it failed, fix the host networking and `sudo systemctl restart\ncenvero-str-network`.\n\n**3. Inspect the config the agent is loading:**\n\n```bash\nsudo cenvero-str-ctl config show\n```\n\nA config that fails to decode falls back to defaults with a note in the output.\nA *signed* config that fails verification is **rejected** — re-fetch a fresh\npanel-signed config rather than hand-editing the file.\n\n**4. Check disk space** and the data directory (`/var/lib/cenvero-str/`). A full disk\nstops the agent from writing its state; the health check reclaims rotated logs\nbut a genuinely full volume needs operator action.\n\n### `cenvero-str-ctl` says the agent is unreachable (exit code 3)\n\nThe CLI talks to the agent over its local socket. If commands report the agent is\nunreachable, the agent process is not running — start it and re-check:\n\n```bash\nsudo systemctl start cenvero-stratum\ncenvero-str-ctl status\n```\n\n### License is not active\n\nCheck the installed license and this machine's hardware identity:\n\n```bash\nsudo cenvero-str-ctl license status\n```\n\n- **Expired / in grace / frozen** — renew it. Existing traffic keeps running even\n  when frozen; only new or changing operations are blocked. Renew with\n  `sudo cenvero-str-ctl license renew` (or fetch a fresh one after renewing in\n  your account). See [Licensing](/docs/licensing) for the warn → grace → freeze\n  model.\n- **Not activated** — activate this machine with `sudo cenvero-str-ctl license activate CNVR-XXXX-XXXX-XXXX-XXXX`, then confirm it in your account.\n- **Hardware-ID unavailable** — `license status` shows the machine's hardware ID.\n  The binding requires firmware/hardware identifiers that are not exposed inside\n  generic virtual machines, so a node must be **bare metal**. If the ID shows as\n  unavailable, you are likely running in an unsupported virtualized environment.\n- **Wrong release channel** — a stable license runs only stable builds and a\n  pre-release license runs only beta/RC builds; a mismatch fails closed. Make sure\n  the build you installed matches your license's channel.\n\n### TLS certificate is pending\n\nAfter install the agent generates its own certificate; if a customer-approval or\ndomain step is outstanding the certificate can show as pending. Confirm what the\nagent is using and re-issue if needed:\n\n```bash\nsudo cenvero-str-ctl config show          # shows the TLS cert/key paths in use\nsudo cenvero-str-ctl tls -h               # certificate management commands\n```\n\nTLS material is managed by the certificate manager (`cenvero-str-ctl tls …`), not\n`config set`. Adding or removing a SAN domain re-generates the certificate.\n\n### Node is not registered\n\nRegistration binds the node to its license and mints its node token. If a node\nnever registered, re-check connectivity to the panel and the license, then let it\nre-register:\n\n```bash\ncenvero-str-ctl status                    # shows license + registration state\nsudo cenvero-str-ctl license activate CNVR-XXXX-XXXX-XXXX-XXXX\n```\n\nRegistration needs outbound HTTPS to your management/license server and a valid\nlicense. If the panel is unreachable, the agent keeps retrying — fix connectivity\nand it completes on its own.\n\n### \"REST API disabled\" — set an API token\n\nIf a client gets a *REST API disabled* (or 401/unauthorized) response, two\nsettings govern access:\n\n**1. Is the service enabled?** Confirm with `cenvero-str-ctl service status`. If\n`rest` is **off**, turn it back on and restart:\n\n```bash\nsudo cenvero-str-ctl service rest on\nsudo systemctl restart cenvero-stratum\n```\n\n**2. Is an API token set?** The REST/gRPC/WebSocket APIs require the bearer token\nto be configured. `config show` reports **Token set: yes/no**. The token is a\n**credential**, so it is *not* set with `config set` — it is provisioned by the\ninstaller (which generates one and prints it) or rotated via the panel. If no\ntoken is set, re-run the installer / fetch a panel-signed config to provision\none, then restart the agent.\n\n### Where to look first\n\n| Symptom | First check |\n|---------|-------------|\n| Agent down / crash-looping | `journalctl -u cenvero-stratum -n 100` |\n| CLI \"agent unreachable\" | `systemctl status cenvero-stratum` |\n| License problems | `cenvero-str-ctl license status` |\n| A service not answering | `cenvero-str-ctl service status` |\n| Settings not taking effect | Did you `systemctl restart cenvero-stratum`? |\n| Overall health | `cenvero-str-ctl heal status` |\n\n## Next steps\n\n- **[CLI Reference](/docs/cli)** — the full command surface.\n- **[Configuration](/docs/configuration)** — the node config and what it controls.\n- **[Licensing](/docs/licensing)** — the enforcement state machine.\n- **[Upgrades](/docs/upgrades)** — how updates are pulled and verified.\n"
        }
    ]
}