Docs / Load Balancer

CENVERO/DOCS

// Load Balancer

Load Balancer

Stratum ships an L4 load balancer designed to run in the in-kernel data plane: it distributes TCP and UDP flows across a set of backends using a per-VIP algorithm, maintains a stateful connection table so established flows stay pinned to their backend, and performs health checks to take unhealthy backends out of rotation automatically.

Status: the L4 load balancer's control surface (lb commands, VIP definitions, backend pools, health checks) and its in-kernel data plane are both in place, and the data plane loads and attaches on a real Linux kernel. Validate the L4 balancer against your own workload profile before production rollout. The Layer-7 HTTP proxy described below is a separate userspace load balancer and does not depend on it.

A VIP can serve traffic from outside the fabric or stay internal for service-mesh style balancing between endpoints — that is a matter of how you route and announce it, not of what kind of node you are on.

Two load balancers, and which one you want

Stratum has two, and they solve different problems. Choosing the wrong one is the most common mistake, so start here.

	L4 (`lb`)	L7 HTTP (`l7lb`)
Works on	TCP and UDP — any protocol	HTTP and HTTPS only
Decides using	Addresses and ports	Hostname, URL path, headers
Handles TLS	No — traffic passes through untouched	Yes — terminates it for you
Use it for	Databases, game servers, SMTP, gRPC, anything not HTTP, or HTTPS you want passed straight through	Websites and APIs where you want one address serving several apps, or TLS handled centrally

They are independent. You can run both — an L4 VIP in front of a database and an L7 frontend for your web tier — and neither depends on the other.

What the L4 balancer requires: its own interface

This is the requirement people actually hit, and it has nothing to do with node mode.

The L4 balancer must be given a dedicated interface to receive VIP traffic on — one it does not share with your workload bridge or with a Gateway node's WAN or LAN interface. Each interface can carry only one of these fast-path attachments, so they cannot be stacked on the same NIC.

Until that interface is configured, VIPs are configuration only. Every lb command works, your VIPs and backends are stored and listed back correctly, and health checks report — but no traffic is balanced, because nothing is attached to receive it. If your VIP looks perfectly configured and simply does not pass traffic, this is the first thing to check.

The interface is named in the node's configuration and applied at startup — see Configuration. The L7 balancer has no such requirement: it is an ordinary listener, so it needs a free port and nothing else.

Switching a node's mode

Mode is not a runtime switch. It is set in the node's signed configuration — at install, or delivered from the panel — and applied when the agent starts. There is deliberately no command that flips it live, because changing which interfaces forward traffic mid-flight would cut the traffic already crossing the node.

To change it: update the node's configuration, let the node pick it up, and restart the agent. To read the current mode:

cenvero-str-ctl node info

Every node can host workloads and forward traffic — one node doing both is a normal setup. What cannot be shared is an interface, not a role.

How the L4 balancer moves a packet

Understanding this matters because one deployment detail depends on it.

A client connects to the VIP. As far as the client is concerned it is

talking to one address, and it will keep talking to that address for the lifetime of the connection.

A backend is chosen, once, using the VIP's algorithm.
The destination address is rewritten to that backend, and the packet is

forwarded on. The backend can be anywhere the node can route to — it does not have to be on the same subnet or the same machine.

The connection is remembered, so every later packet in that conversation

goes to the same backend. The choice is made once, not per packet.

The reply comes back through the node, which puts the VIP address back on

it before it reaches the client — so the client only ever sees the address it connected to.

Two consequences follow, and they surprise people:

Your backends see the real client address. Only the destination is rewritten, never the source. Your application logs show actual client IPs with no extra configuration and no X-Forwarded-For to parse.

Replies must come back through the balancing node. Step 5 is where the VIP address is restored. If a backend answers the client directly — because it has its own route out — the client receives a reply from an address it never contacted and discards it. The usual symptom is connections that hang instead of failing. Make sure your backends route back through the node holding the VIP.

One VIP is one address, one port, one protocol

A VIP carries a single frontend_port and a single protocol. To serve a service on two ports, create two VIPs:

# HTTP and HTTPS for the same service = two VIPs, same backends
sudo cenvero-str-ctl lb create '{"id":"web-80","frontend_ip":"10.0.0.100","frontend_port":80,"protocol":"tcp","algorithm":"round-robin"}'
sudo cenvero-str-ctl lb create '{"id":"web-443","frontend_ip":"10.0.0.100","frontend_port":443,"protocol":"tcp","algorithm":"round-robin"}'

Each VIP has its own backends, its own algorithm and its own health checks, so they can differ where that is useful — for example checking a different health path per port.

TLS at layer 4

The L4 balancer does not read or terminate TLS. Encrypted traffic passes through exactly as it arrives, and each backend must present its own certificate. That is the right choice when you want end-to-end encryption with no middle box, or when the protocol is not HTTP at all.

If you want certificates handled in one place instead, that is the L7 balancer — see Terminating TLS below.

Direct server return

dsr_enabled is accepted on a VIP but is not yet in effect: a VIP with it set is served through the normal path described above, with replies returning through the node. Plan capacity on that basis.

Creating a VIP

lb create takes one argument: a JSON object describing the virtual IP. A VIP and its backends are created separately — first the VIP, then each backend with lb add-backend.

Field	Type	Description
`id`	string	Required. The VIP's name, used by every other `lb` command.
`frontend_ip`	string	Required. The virtual IP address clients connect to.
`algorithm`	string	Required. One of the algorithms below.
`frontend_port`	number	The port clients connect to.
`protocol`	string	`tcp` or `udp`.
`dsr_enabled`	boolean	Enable direct server return.
`health_check`	object	Optional active health check — see Health checks below.

sudo cenvero-str-ctl lb create '{"id":"web-lb","frontend_ip":"10.20.0.10","frontend_port":80,"protocol":"tcp","algorithm":"least-conn"}'

List your VIPs:

cenvero-str-ctl lb list

Show detail for one VIP, including per-backend health and connection counts:

cenvero-str-ctl lb show web-lb

{
  "data": {
    "load_balancer": {
      "id": "web-lb",
      "frontend_ip": "10.20.0.10",
      "frontend_port": 80,
      "protocol": "tcp",
      "algorithm": "least-conn",
      "dsr_enabled": false,
      "backends": [
        {
          "id": "web-1",
          "ip": "10.20.0.50",
          "port": 80,
          "weight": 1,
          "healthy": true,
          "active_conns": 0
        }
      ]
    }
  },
  "status": "ok"
}

Algorithms

Set one of these as the algorithm field. An unrecognized value is rejected when the VIP is created.

Algorithm	Value	Behaviour
Round-robin	`round-robin`	Distributes new connections evenly in turn. Good default for stateless services.
Least connections	`least-conn`	Sends each new connection to the backend with the fewest active connections. Handles variable-cost requests well.
Weighted round-robin	`weighted`	Like round-robin but each backend has a relative weight (see below). Use when backends have unequal capacity.
Source-hash persistence	`source-hash`	Hashes the client source IP to a backend. The same client always reaches the same backend as long as the backend is healthy — useful for session-affinity without shared state.
Maglev	`maglev`	Consistent hashing with Maglev's lookup table, which keeps flow-to-backend mapping stable when the backend set changes.
Consistent hash	`consistent-hash`	Hash-ring backend selection, minimizing remapping when backends are added or removed.

Setting weights

Weight is a property of each backend, so choose the weighted algorithm on the VIP and give each backend its relative weight as you add it:

sudo cenvero-str-ctl lb create '{"id":"api-lb","frontend_ip":"10.20.0.20","frontend_port":443,"protocol":"tcp","algorithm":"weighted"}'

sudo cenvero-str-ctl lb add-backend '{"vip_id":"api-lb","id":"api-1","ip":"10.20.0.50","port":443,"weight":3}'
sudo cenvero-str-ctl lb add-backend '{"vip_id":"api-lb","id":"api-2","ip":"10.20.0.51","port":443,"weight":1}'

The weights 3 and 1 are relative — api-1 receives three times the new connections of api-2.

Health checks

The agent probes backends at a configurable interval. A backend that fails enough consecutive checks is marked unhealthy and removed from the connection pool. It re-enters the pool automatically when it passes the same number of consecutive checks.

A health check is configured as part of the VIP, in the optional health_check object passed to lb create:

sudo cenvero-str-ctl lb create '{"id":"api-lb","frontend_ip":"10.20.0.20","frontend_port":443,"protocol":"tcp","algorithm":"round-robin","health_check":{"type":"http","interval_sec":5,"timeout_sec":2,"threshold":3,"http_path":"/healthz"}}'

Field	Description	Notes
`type`	`tcp` (connection probe) or `http` (expects a 2xx response)	Omit the whole object, or leave `type` empty, for no active check
`interval_sec`	Seconds between probes
`timeout_sec`	Probe timeout in seconds
`threshold`	Consecutive checks before flipping a backend's state	One value covers both directions
`http_path`	Path to request for an `http` check, e.g. `/healthz`	Ignored for `tcp` checks

Overriding a backend's health manually

lb set-health is a manual override, not the check configurator — it forces one backend up or down, which is useful for draining a backend before maintenance. It takes three positional arguments: the VIP id, the backend id, and up or down.

# Take a backend out of rotation
sudo cenvero-str-ctl lb set-health web-lb web-1 down

# Put it back
sudo cenvero-str-ctl lb set-health web-lb web-1 up

If an active health check is configured for the VIP, the next probe result will overwrite a manual override.

Adding and removing backends live

Backend changes take effect immediately without dropping established connections. The connection table keeps existing flows on their current backend until the flow closes naturally.

lb add-backend takes a JSON object identifying the VIP (vip_id) plus the backend's own id, ip, port, and weight. vip_id, id, and ip are required.

# Add a backend
sudo cenvero-str-ctl lb add-backend '{"vip_id":"web-lb","id":"web-3","ip":"10.20.0.52","port":80,"weight":1}'

Retiring a backend without dropping traffic

lb remove-backend takes effect immediately, which cuts the connections currently on that backend. To retire one cleanly, drain it first: the load balancer stops sending it new connections while the existing ones finish, and reports how many are still active.

# Stop new connections; existing ones keep being served
sudo cenvero-str-ctl lb drain web-lb web-1

# Re-run to watch the count fall, then remove it once it reaches zero
sudo cenvero-str-ctl lb remove-backend web-lb web-1

Draining the only healthy backend of a load balancer is refused, since that would take the service down.

lb remove-backend takes two positional arguments — the VIP id and the backend id:

# Remove a backend
sudo cenvero-str-ctl lb remove-backend web-lb web-1

To drain a backend gracefully, mark it down first and let its existing flows finish before removing it:

sudo cenvero-str-ctl lb set-health web-lb web-1 down
# ...wait for active_conns on that backend to reach 0 in `lb show web-lb`...
sudo cenvero-str-ctl lb remove-backend web-lb web-1

List a VIP's backends at any time:

cenvero-str-ctl lb backends web-lb

North-south VIPs on Gateway nodes

On a Gateway node, VIPs in the fabric subnet are reachable from outside the cluster automatically because the Gateway node announces those routes via BGP. If you want a VIP to have a publicly routable address, assign it from a prefix your BGP session announces:

sudo cenvero-str-ctl lb create '{"id":"external-web","frontend_ip":"203.0.113.10","frontend_port":443,"protocol":"tcp","algorithm":"round-robin"}'

sudo cenvero-str-ctl lb add-backend '{"vip_id":"external-web","id":"web-1","ip":"10.20.0.50","port":443,"weight":1}'
sudo cenvero-str-ctl lb add-backend '{"vip_id":"external-web","id":"web-2","ip":"10.20.0.51","port":443,"weight":1}'

See BGP Edge Routing for how prefixes are advertised.

Deleting a VIP

sudo cenvero-str-ctl lb delete web-lb

Active connections are torn down immediately. Drain traffic to backends first if zero disruption is required.

Layer-7 HTTP load balancer

Separately from the in-kernel L4 balancer above, Stratum ships a userspace Layer-7 (HTTP/HTTPS) reverse proxy. Where the L4 balancer distributes raw TCP/UDP flows, the L7 proxy understands HTTP: it terminates TLS, routes by Host header and URL path prefix, and load-balances across backend pools. The two are orthogonal — reach for the L4 balancer for raw flow distribution and the L7 proxy when you need HTTP-aware routing or TLS termination. The L7 proxy stays off until you configure a frontend.

Its two building blocks are:

Pools — a named set of HTTP backends with a selection algorithm (round-robin or least-connections) and health checks.
Frontends — a listener, optionally terminating TLS, that routes each request to a pool by Host and path prefix, with a default pool as fallback.

How a request travels

A request arrives at a frontend — a listener on one address and port.
If the frontend has TLS, it is decrypted here. From this point on the

proxy is working with a plain HTTP request.

The routes are checked in order and the first match wins. A route matches

on hostname, on a path prefix, or both. Anything matching no route goes to the frontend's default_pool; a frontend with no default and no match returns an error rather than guessing.

The pool picks a backend — round-robin, or the one with the fewest active

requests.

The request is forwarded over plain HTTP and the response passed back.

Because the proxy is the one talking to the client, it adds X-Forwarded-For (the real client address), X-Forwarded-Proto and X-Forwarded-Host so your application can still see who asked and how. WebSocket and other upgrade connections pass through unchanged.

Routing is per request, not per connection — two requests on the same keep-alive connection can land on different backends, and on different pools.

Terminating TLS

This is where most questions come up, so plainly:

The certificate goes on the frontend, once. Your backends do not need one.

A frontend with a tls block accepts HTTPS from clients, decrypts, and then speaks plain HTTP to the backends. So:

You need one certificate per frontend, not one per backend and not one per

port. Ten backends behind an HTTPS frontend need zero certificates between them.

The backend link is unencrypted. That is fine when the backends are on a

network you control — the usual case, since they are your own endpoints on your own fabric. If you need encryption all the way to the application, do not terminate here: use an L4 VIP and let each backend hold its own certificate.

A frontend with no tls block serves plain HTTP. Mixing is normal — run a

plain frontend on :80 and a TLS one on :443.

The certificate itself is either a PEM certificate and key you supply, or the node's own managed certificate. The minimum accepted version is TLS 1.2.

Serving several ports

Unlike an L4 VIP, one frontend is one listener — but you can add as many frontends as you need, and they can share pools:

# Plain HTTP on 80 and TLS on 443, both serving the same backends
sudo cenvero-str-ctl l7lb frontend add '{"name":"http","listen":":80","default_pool":"web"}'
sudo cenvero-str-ctl l7lb frontend add '{"name":"https","listen":":443","tls":{"use_managed":true},"default_pool":"web"}'

One frontend can also serve many sites on a single port by routing on hostname — that is the usual reason to choose L7 over L4:

sudo cenvero-str-ctl l7lb frontend add '{"name":"edge","listen":":443","tls":{"use_managed":true},
  "routes":[
    {"host":"api.example.com","pool":"api"},
    {"host":"www.example.com","path_prefix":"/static","pool":"assets"},
    {"host":"www.example.com","pool":"web"}
  ],"default_pool":"web"}'

Order matters: the /static rule sits before the general www rule, because the first match wins.

Backend pools

A pool is created from a single JSON object:

sudo cenvero-str-ctl l7lb pool add '{"name":"web","algorithm":"round-robin","backends":["10.20.0.50:8080","10.20.0.51:8080"]}'

List pools, and add or remove a pool's backends live:

cenvero-str-ctl l7lb pool list
cenvero-str-ctl l7lb backend list web
sudo cenvero-str-ctl l7lb backend add web 10.20.0.52:8080
sudo cenvero-str-ctl l7lb backend remove web 10.20.0.50:8080

The proxy runs active health checks (periodic HTTP probes with configurable healthy and unhealthy thresholds) and also ejects a backend passively when it returns transport errors while proxying, re-admitting it once it recovers.

TLS-terminating frontends

A frontend binds a listener and routes to pools. This one terminates TLS with an operator-provided certificate and routes one host and path prefix to the web pool, falling back to it by default:

sudo cenvero-str-ctl l7lb frontend add '{"name":"https","listen":":8443","tls":{"cert_file":"/etc/cenvero-str/tls/lb.crt","key_file":"/etc/cenvero-str/tls/lb.key"},"default_pool":"web","routes":[{"host":"api.example.com","path_prefix":"/v1","pool":"web"}]}'

TLS termination accepts either an operator-supplied PEM certificate and key (as above) or the node's own managed certificate; the minimum TLS version is 1.2. A frontend with no tls block serves plain HTTP. The proxy sets the X-Forwarded-For, X-Forwarded-Proto, and X-Forwarded-Host headers on proxied requests and passes WebSocket and other upgrade connections through unchanged.

List or remove frontends, and show the full picture — every frontend and pool with per-backend health and active connection counts:

cenvero-str-ctl l7lb frontend list
sudo cenvero-str-ctl l7lb frontend remove https
cenvero-str-ctl l7lb status

When a pool has no healthy backend the proxy returns 503; when a chosen backend fails mid-request it returns 502 without leaking internal error detail.