Palomar: Cluster Controller

Cluster Controller is the Palomar component that manages the cluster state for a given site. Think of it as the "brain" of your Palomar cluster — it continuously tracks the health and activity of all mail Backends, and makes decisions about where users should be located, automatically responding to problems. While it uses proxying statistics to assess system health, it does not directly monitor the backends themselves.

Each Cluster site has to have one Cluster Controller running. The Controller provides health checks, load balancing, evacuation automation, and API access to GeoDB. If the Cluster Controller goes down, there is no immediate impact to the cluster (mail continues to flow normally), but it needs to be brought back as soon as possible to maintain automation and monitoring capabilities.

WARNING

Cluster Controller will not provide automation to more than its home site. Every Palomar site requires a Cluster Controller to be running.

Installation

See Cluster Controller Installation.

Architecture

The Cluster Controller consists of several components that work together to manage your mail cluster.

mermaid

%%{init: {'theme': 'dark'}}%%
flowchart TD
  subgraph Legend [Legend]
    direction LR
    L1[User Interface] ~~~ L2[Compute Layer] ~~~ L3[Data Layer] ~~~ L4[Backends Layer]
  end

  Legend ~~~ CLUSTER_ADMIN

  CLUSTER_ADMIN([Dovecot Cluster Administrator]) --> FRONTEND[Frontend]
  CLUSTER_ADMIN --> REST_API

  subgraph ClusterController [Cluster Controller]
      direction TB
      FRONTEND
      REST_API[REST API]
      CELERY_TASKS[Celery Tasks]
      REDIS[(Redis)]
      PROMETHEUS[(Prometheus)]

      FRONTEND --> REST_API
      REST_API --> CELERY_TASKS
      REST_API --> REDIS
      CELERY_TASKS --> REDIS
      CELERY_TASKS --> PROMETHEUS
  end

  REST_API --> CASSANDRA[(Cassandra)]
  REST_API --> DOVECOT_BACKENDS[Dovecot Backends]
  CELERY_TASKS --> CASSANDRA
  CELERY_TASKS --> DOVECOT_BACKENDS
  PROMETHEUS --> DOVECOT_BACKENDS

  style FRONTEND fill:#a8d5ba,stroke:#333,color:#000
  style REST_API fill:#87ceeb,stroke:#333,color:#000
  style CELERY_TASKS fill:#87ceeb,stroke:#333,color:#000
  style CASSANDRA fill:#f4a460,stroke:#333,color:#000
  style REDIS fill:#f4a460,stroke:#333,color:#000
  style PROMETHEUS fill:#f4a460,stroke:#333,color:#000
  style DOVECOT_BACKENDS fill:#dda0dd,stroke:#333,color:#000

  style L1 fill:#a8d5ba,stroke:#333,color:#000
  style L2 fill:#87ceeb,stroke:#333,color:#000
  style L3 fill:#f4a460,stroke:#333,color:#000
  style L4 fill:#dda0dd,stroke:#333,color:#000

What Each Component Does

Component	What It Does	Why It Matters
Frontend (Web UI)	Browser-based dashboard showing cluster status, Backend health, and controls for manual operations	Allows administrators to monitor the cluster and perform operations without using the API directly
REST API	HTTP endpoints for all Controller operations	Enables automation, scripting, and integration with other systems
Celery (Scheduler & Workers)	Triggers background tasks on a schedule (every 5-60 seconds depending on the task). Execute the actual work: checking health, moving users, calculating load scores.	Ensures continuous monitoring even when no administrator is watching. The "muscle" that carries out all automated operations

Data Stores

The Cluster Controller requires one external service and includes two bundled services:

External Service (Customer-Provided):

Store	What It Stores	What Happens If It's Down
Cassandra (GeoDB)	Permanent cluster state: which sites exist, which Backends are in each site, user locations, feature flag settings	Controller cannot function - all state is lost. Must be highly available.

Bundled Services (Included with Controller Deployment):

Store	What It Stores	What Happens If It's Down
Redis	Temporary data: task queue, cached metrics, ongoing move tracking	Tasks stop executing, UI shows stale data. Controller recovers automatically when Redis returns.
Prometheus	Time-series metrics from all Backends: CPU, memory, login success/failure rates	Health checking and load balancing become blind - no automatic decisions can be made. Manual operations still work.

Redis and Prometheus are provided via the Helm chart or Docker Compose deployment and shouldn't be replaced with external instances.

Site Reachability Monitoring

Site reachability monitoring is performed by the cluster plugin running in Dovecot proxies.

WARNING

The reachability feature is currently only visible in the UI and can be accessed via the API — no automatic action is taken.

Background Tasks

The Controller runs several automated tasks that keep your cluster healthy. These tasks run continuously in the background without administrator intervention. The background task lifecycle is briefly described by below diagram.

mermaid

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#861BE4', 'primaryTextColor': '#FFFFFF', 'primaryBorderColor': '#a855f7', 'lineColor': '#6b21a8', 'secondaryColor': '#1a0a2e', 'tertiaryColor': '#0E0D10', 'actorBkg': '#861BE4', 'actorBorder': '#a855f7', 'actorTextColor': '#FFFFFF', 'signalColor': '#6b21a8', 'signalTextColor': '#FFFFFF', 'noteBkgColor': '#1a0a2e', 'noteTextColor': '#FFFFFF', 'activationBorderColor': '#a855f7', 'activationBkgColor': '#9833FF', 'labelBoxBkgColor': '#1a0a2e', 'labelTextColor': '#FFFFFF'}}}%%
  sequenceDiagram
      participant Beat as Celery Beat<br/>(Scheduler)
      participant Redis as Redis<br/>(Queue)
      participant Worker as Celery Worker
      participant Prom as Prometheus
      participant DB as Cassandra
      participant Backend as Dovecot Backend

      Beat->>Redis: Schedule task (every Xs)
      Redis->>Worker: Dequeue task
      Worker->>DB: Check feature flags

      alt Feature Enabled
          Worker->>Prom: Query metrics
          Worker->>DB: Read/Update state
          Worker->>Backend: Execute action (doveadm)
      else Feature Disabled
          Worker->>Worker: Skip (raise Ignore)
      end

Scheduled / automatically triggered tasks:

Task	How Often	What It Does
`cache_metrics`	Every 5 seconds	Fetches fresh metrics from Prometheus and caches them in Redis. This makes the Web UI responsive and ensures other tasks have current data.
`scrape_stats`	On demand	Automatically triggered each time Prometheus scrapes the Controller API - `/metrics` endpoint, ensuring up-to-date stats are collected.
`check_user_moves`	Every 5 seconds	Monitors ongoing user migrations. If a move gets stuck (Backend not responding), it retries or escalates to a force-move.
`rebalance_sites`	Every 60 seconds	Analyzes load across all Backends. If one Backend is significantly more loaded than others, moves some users to balance the load.
`check_Backend_health`	Every 60 seconds	Checks if Backends are healthy by looking at login and mail delivery success rates. Automatically evacuates users from failing Backends.
`evacuate_zero_load_factor_hosts`	Every 60 seconds	Moves users off Backends that have been marked for decommissioning (load_factor = 0).
`check_site_reachability`	Every 60 seconds	Tests connectivity to remote sites in multi-site deployments.
`cache_proxies`	Every 5 minutes	Refreshes the cached list of the site's proxies. See Controller Proxy List.
`delete_stale_data`	Every 24 hours	Cleans up orphaned records in the database (e.g., statistics for Backends that no longer exist).

What if a task fails?

Individual task failures are logged but don't kill the Cluster Controller. The task will be retried on the next scheduled run. The Controller continues running because these failures are often temporary (e.g., network, brief service outages). Persistent failures usually indicate a problem with an external service (Prometheus, Cassandra, or a Backend). Check the Cluster Controller logs for details.

Important for health checking: Just because the Cluster Controller process is up doesn't mean it's successfully executing the tasks you expect. Monitor the logs to verify that scheduled tasks are completing successfully.

Feature Flags

All automated behavior is controlled by feature flags. For example, health checking can be enabled or disabled independently. This gives you fine-grained control over what the Controller does automatically versus what requires manual intervention.

Why Feature Flags?

When you first deploy a Controller or make significant changes, you may want to:

Observe what the Controller would do without actually doing it (DryRun mode)
Disable automation for specific Backends during maintenance
Enable automation gradually, starting with less critical sites

Flag Hierarchy

Feature flags are defined at three levels: Global, Site, and Backend. Precedence is top-down — once a feature is disabled at a higher level, lower-level settings cannot re-enable it.

mermaid

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#6366f1', 'primaryTextColor': '#f8fafc', 'primaryBorderColor': '#818cf8', 'lineColor': '#94a3b8', 'secondaryColor': '#1e293b', 'tertiaryColor': '#0f172a', 'background': '#0f172a', 'mainBkg': '#1e293b', 'nodeBorder': '#475569', 'clusterBkg': '#1e293b', 'clusterBorder': '#475569', 'titleColor': '#f8fafc', 'edgeLabelBackground': '#1e293b'}}}%%
  flowchart LR
      Global["GLOBAL
  (default for entire cluster)"]
      Site["SITE
  (override for one site)"]
      Backend["BACKEND
  (override for one Backend)"]

      Global -->|overridden by| Site
      Site -->|overridden by| Backend

      classDef primary fill:#6366f1,stroke:#818cf8,stroke-width:2px,color:#f8fafc
      classDef secondary fill:#3b82f6,stroke:#60a5fa,stroke-width:2px,color:#f8fafc
      classDef accent fill:#8b5cf6,stroke:#a78bfa,stroke-width:2px,color:#f8fafc

      class Global primary
      class Site secondary
      class Backend accent

⚠️ TODO

API endpoint for Backend feature flags configuration doesn't exist yet

Example: You enable load balancing globally, but disable it for a specific Backend that you're troubleshooting. The rest of the cluster continues to balance automatically while that one Backend is excluded.

Available Features

Feature	What It Controls	When to Disable
`MetricsExport`	Whether the Controller collects and exports metrics. Required for all other features.	Rarely - disabling this blinds the Controller
`BackendHealthAutoHandling`	Automatic detection of failing Backends and user evacuation	During planned maintenance when you expect temporary failures
`BackendLoadBalancing`	Automatic movement of users to balance load across Backends	When you want manual control over user placement

Feature States

State	What Happens
Enabled	Full automation - the Controller acts on its decisions
Disabled	No automation - the feature doesn't run at all (default)
DryRun	The Controller makes decisions and logs them, but doesn't execute. Perfect for testing.

Recommended Workflow

Start with only the MetricsExport feature enabled, and verify metrics are flowing
Enable features in DryRun mode and review logs
Once you're confident, switch to Enabled

Feature flags can be managed via the Controller API or the Controller Web UI under "Site Features".

Load Balancing

Load balancing ensures users are distributed evenly across your Backends. Without it, some Backends might become overloaded while others sit idle - leading to poor performance for some users and wasted capacity. When BackendLoadBalancing feature is disabled on the Backend level then this Backend is not a part of (load)balancing.

How It Works

Every 60 seconds, the Controller:

Retrieves metrics: The Controller queries usage metrics (CPU, memory, metacache pressure) from Prometheus. Prometheus is populated by the scrape_stats task, which scrapes GeoDB rather than the Backends directly.
Calculates a score for each Backend indicating how loaded it is relative to others
Finds the most imbalanced pair - the most loaded and least loaded Backends
Moves users from the overloaded Backend to the underloaded one (if the imbalance exceeds a threshold)

mermaid

flowchart TD
    Start([Every 60 seconds])
    GetScores["Calculate load score<br/>for each Backend"]
    FindPair["Find most loaded and<br/>least loaded Backends"]
    CheckDelta{"Is the difference<br/>significant enough?"}
    MoveUsers["Move some users from<br/>overloaded → underloaded"]
    Done([Wait for next cycle])
    Skip([No action needed])

    CheckPeak{"Is any Backend extremely<br/>loaded on a single metric?"}

    Start --> GetScores
    GetScores --> FindPair
    FindPair --> CheckDelta
    CheckDelta -->|"Yes (delta > threshold)"| MoveUsers
    CheckDelta -->|"No"| CheckPeak
    CheckPeak -->|"Yes (single metric > cap)"| MoveUsers
    CheckPeak -->|"No"| Skip
    MoveUsers --> Done
    Skip --> Done

The Z-Score: Measuring "How Loaded"

The Cluster Controller uses a statistical measure called Z-score to compare Backends fairly. The Z-score tells you how far a Backend is from the average:

Z-score > 0: Backend is more loaded than average
Z-score < 0: Backend is less loaded than average
Z-score = 0: Backend is exactly at average

The formula combines multiple metrics (memory, CPU, metacache) into a single score, so a Backend with high memory but low CPU is compared fairly against one with low memory but high CPU. Each per-metric Z-score is clamped to the [-10, 10] range before they are combined, so one extreme metric (for example the memory score right after a Backend restart) cannot dominate the total.

Single-Metric Overload Detection

Because the combined score is a sum, a Backend that is extremely loaded on just one metric (for example, CPU pinned at 100% while memory is average) can still end up with an unremarkable total and be missed by the comparison above.

To catch this, the Controller also tracks each Backend's highest single-metric Z-score. When the regular comparison finds no pair to balance, a Backend whose highest single-metric Z-score exceeds HOST_LOAD_BALANCE_SINGLE_AXIS_ZSCORE_CAP is treated as overloaded anyway, and some of its users are moved to the least loaded Backend that is not itself over the cap on any metric.

When Load Balancing Skips a Backend

A Backend is excluded from load balancing if:

It's not online (status is offline or standby)
It has a user move already in progress
It was recently involved in a move (HOST_LOAD_BALANCE_MIN_COOL_TIME_SECS)
There isn't enough historical data to calculate a reliable score (HOST_LOAD_BALANCE_MIN_SAMPLES)
It's failing health checks (if lmtp or login failure > HOST_FAILURE_RATIO)

Configuration

Setting	What It Controls
`HOST_LOAD_BALANCE_SCORE_DELTA_THRESHOLD_RATIO`	How different two Backends must be before moving users. Higher = less sensitive.
`HOST_LOAD_BALANCE_MIN_COOL_TIME_SECS`	How long to wait before moving the same user again. Prevents thrashing.
`HOST_LOAD_BALANCE_MIN_SAMPLES`	Minimum data points needed before trusting a Backend's score. New Backends need time to gather data.
`HOST_LOAD_BALANCE_SINGLE_AXIS_ZSCORE_CAP`	Single-metric Z-score above which a Backend is drained even if its combined score looks normal.

Health Checking

Health checking automatically detects and responds to failing Backends. When a Backend starts failing (e.g., rejecting logins, failing to deliver mail), the Controller moves users away before they're significantly impacted.

How It Works

Every 60 seconds, the Controller:

Fetches statistics for each Backend: how many logins succeeded vs failed, how many LMTP mail deliveries succeeded vs failed
Calculates failure rates: What percentage of operations are failing?
Takes action based on how severe the failures are

mermaid

flowchart TD
    Start([Every 60 seconds])
    FetchMetrics["Get login and delivery<br/>statistics from Prometheus"]
    ForEach["Check each Backend"]
    CalcRate["Calculate failure rate:<br/>failures ÷ total attempts"]

    CheckSevere{"Failure rate<br/>> 90%?"}
    CheckHigh{"Failure rate<br/>> 10%?"}

    Evacuate["CRITICAL: Evacuate all users, when no users then set Backend OFFLINE"]
    MovePartial["WARNING: Move some users<br/>to healthy Backends"]
    Healthy["Backend is healthy<br/>No action needed"]

    Start --> FetchMetrics
    FetchMetrics --> ForEach
    ForEach --> CalcRate
    CalcRate --> CheckSevere
    CheckSevere -->|Yes| Evacuate
    CheckSevere -->|No| CheckHigh
    CheckHigh -->|Yes| MovePartial
    CheckHigh -->|No| Healthy

Failure Thresholds

Condition	What Happens
> 90% failures	Backend is critically failing. All users are evacuated. When no users are left, status is set to OFFLINE.
> 10% failures (configurable)	Backend is degraded. Some users are moved away to reduce load and see if it recovers.
< 10% failures (configurable)	Backend is healthy. No action taken.

WARNING

Cluster Controller for the critical failure case try to move 20% of original number of users instead of current amount to avoid taking too long for the Backend to be evacuated.

Recovery

While a Backend is offline, proxies are periodically checking when it comes back online. When the health check succeeds, the Controller automatically brings the Backend back online. Users can then be moved back by normal load balancing.

Catastrophe Protection

What if many Backends fail at once (e.g., a network issue affecting half your data center)? Moving all users to the remaining Backends could overload them, making the situation worse.

The Controller has built-in protection:

If more than 30% (configurable) of Backends would be taken offline, automatic evacuations stop
A critical alert is raised for manual intervention
The administrator can then assess the situation and decide how to proceed

When Catastrophe Protection Triggers

Check the Controller logs immediately. This usually indicates a serious infrastructure problem (network outage, storage failure, etc.) rather than individual Backend issues.

Configuration

Setting	What It Controls
`HOST_FAILURE_RATIO`	Failure rate threshold for moving users. Lower = more sensitive.
`HOST_FAILURE_MIN_LOGINS`	Minimum attempts before making decisions. Prevents overreacting to small sample sizes.
`HOST_FAILURE_COOL_TIME_SECS`	Minimum time in seconds between moving users from a host with failing logins.
`HOST_FAILURE_BACKEND_NUM_THRESHOLD`	Maximum percentage of Backends that can fail before catastrophe protection kicks in.

Manual Operations

While the Controller automates most operations, sometimes you need manual control - for planned migrations, emergency evacuations, or testing.

Batch Move Users

Gradually migrates a percentage of users from one site to other sites. The Cluster Controller sends a request to the Backend to move users (doveadm doveadm cluster user batch move backend command). If HOST_FAILURE_COOL_TIME_SECS is set the min-last-moved parameter is added to doveadm command. This is used to avoid moving the same users too often.

⚠️ TODO

Add reference to the OpenAPI batch move users post

Use cases:

Planned data center migration over several days
Gradually shifting load to a new site
Reducing capacity at one site before maintenance

Force Move Users

Immediately moves a percentage of users in a single operation. Faster than batch move but more disruptive.

⚠️ TODO

Add reference to the OpenAPI force move users post

Use cases:

Emergency evacuation when a site is failing
Quick capacity rebalancing
Testing migration procedures

Manual Rebalance

Triggers (via REST API request) an immediate load rebalancing cycle without waiting for the 60-second schedule.

REST API

See Palomar REST APIs for the complete API reference.

Use cases:

After adding new Backends that need users
After maintenance when you want immediate redistribution
Testing load rebalancing behavior

Forwarding doveadm Commands

The Controller can run a doveadm HTTP command on one host of its own site on an administrator's behalf and return the reply unchanged. This saves opening a shell on the host, or exposing the doveadm HTTP port beyond the Controller, when you only need to ask a Backend or a proxy something.

The command name and its arguments are passed through as given, so anything the doveadm HTTP API on that host understands can be run. Note that the API spells command names in camelCase (stats dump on the command line is statsDump over HTTP).

WARNING

The command runs with the Controller's doveadm credentials and is not limited to read-only commands. Anyone allowed to call the Controller API can therefore run anything doveadm offers on any Backend or proxy of the site.

For that reason forwarding is disabled by default. Set DOVEADM_HTTP_FORWARD_ENABLED to enable it, and leave it off otherwise. While it is off, the endpoint answers 403 without looking at the request at all, so it also reveals nothing about which hosts the site has. Listing the proxies is not gated: like listing Backends, it is inventory rather than an action.

Why a setting and not a feature flag

Feature flags are changed through the Controller API, which is the same API that would then run the commands, so a flag would not keep anyone out who can already reach the API. A setting lives in controller.cfg and can only be changed by whoever deploys the Controller, which is the boundary this capability needs. The cost is that turning it on or off is a configuration change and takes a restart, rather than a toggle in the UI.

Only hosts the site already knows are accepted, so the endpoint cannot be used to reach an arbitrary address with those credentials. A request always targets exactly one host:

Target	Host
Backend	Required, and must be a Backend of this site.
Proxy	Optional. Given, it must be a known proxy; left out, the Controller picks one of the site's proxies at random and tries up to three.

Proxies are interchangeable for running a command, which is why the Controller may pick one for you. It moves on to the next proxy only when the command could not be delivered at all (name resolution, connection or timeout error). Once a host has answered, its reply is final even when doveadm reports an error, because the command may already have had an effect and must not run a second time somewhere else.

How long the Controller waits for a reply is set by DOVEADM_HTTP_FORWARD_TIMEOUT_SECS.

Use cases:

Inspecting the state of a single Backend or proxy while debugging
Running a doveadm command on a host you have no shell access to
Scripting an occasional command across hosts without distributing the doveadm password

How the Controller Knows the Proxies

Unlike Backends, proxies are not registered anywhere in Palomar GeoDB. Every proxy does write its own host name into the proxy_host column of the proxy_dest_stats table when it exports statistics, so the Controller takes the distinct values of that column as the site's proxy list. The cache_proxies task refreshes it every 5 minutes into Redis, and the API serves that cached copy.

Two consequences are worth knowing:

A newly added proxy is usable as a target only after it has exported statistics and the next refresh has run, so allow a few minutes.
Nothing deletes the rows of a proxy that was removed from the site, so a proxy counts as gone once it has not exported a statistic for an hour. Until then it can still be picked. If it is unreachable, the Controller simply moves on to another proxy.

The current list is available through the API, which is the quickest way to check whether a proxy is visible to the Controller at all.

API Endpoint

The Controller API provides programmatic access to all Controller functions. You can:

Create, read, update, and delete sites and Backends
Query cluster statistics and metrics
Trigger manual operations
Manage feature flags

The API also exposes an OpenMetrics endpoint for monitoring systems to scrape Controller metrics.

REST API

See Palomar REST APIs for the complete API reference.

Web UI

The Controller includes a browser-based administration interface. Access it by navigating to your Controller's HTTP endpoint (default port 8080).

The UI provides:

Dashboard: Overview of all sites and their health status
Backend list: Detailed status of each Backend with real-time metrics
Feature flags: Toggle automation features on/off
Operations: Trigger manual moves and rebalancing
Site reachability: View connectivity status between sites

Administration

See Palomar Administration.

Databases

Mechanisms

Users

Extensions

Palomar: Cluster Controller ​

Installation ​

Architecture ​

What Each Component Does ​

Data Stores ​

Site Reachability Monitoring ​

Background Tasks ​

Feature Flags ​

Why Feature Flags? ​

Flag Hierarchy ​

Available Features ​

Feature States ​

Load Balancing ​

How It Works ​

The Z-Score: Measuring "How Loaded" ​

Single-Metric Overload Detection ​

When Load Balancing Skips a Backend ​

Configuration ​

Health Checking ​

How It Works ​

Failure Thresholds ​

Recovery ​

Catastrophe Protection ​

Configuration ​

Manual Operations ​

Batch Move Users ​

Force Move Users ​

Manual Rebalance ​

Forwarding doveadm Commands ​

How the Controller Knows the Proxies ​

API Endpoint ​

Web UI ​

Administration ​

Palomar: Cluster Controller

Installation

Architecture

What Each Component Does

Data Stores

Site Reachability Monitoring

Background Tasks

Feature Flags

Why Feature Flags?

Flag Hierarchy

Available Features

Feature States

Load Balancing

How It Works

The Z-Score: Measuring "How Loaded"

Single-Metric Overload Detection

When Load Balancing Skips a Backend

Configuration

Health Checking

How It Works

Failure Thresholds

Recovery

Catastrophe Protection

Configuration

Manual Operations

Batch Move Users

Force Move Users

Manual Rebalance

Forwarding doveadm Commands

How the Controller Knows the Proxies

API Endpoint

Web UI

Administration