Dovecot Pro
Search K

Appearance

Palomar: Administration ​

REST API Reference ​

Site Operations ​

Adding a New Site ​

Use doveadm cluster site add to create a site.

New sites start in online status. Provide a site name (required), and optionally a tag and load-balancer host used for inter-site proxying.

The UUID/ID should only be supplied when re-adding a previously removed site (otherwise, it is autogenerated by default).

Parameter Rules & Guidance ​

  • Name (required): Must match the cluster_local_site used by your Proxies/Backends and the controller’s CLUSTER_SITE. Mismatches will break routing for that site.
  • Load balancer (optional): Required only if the site exposes an external LB for cross-site connections; it’s the hostname/IP where other sites’ proxies connect.
  • Tag (optional): Used to shard/match users and backends to the site.
  • ID/UUID (optional): Auto-generated; specify only when re-adding the same site.

Typical Workflow ​

  1. Ensure the controller is running and connected to Cassandra (GeoDB) and that the keyspace/tables exist or auto-init is enabled.
  2. Add the site: doveadm cluster site add --load-balancer lb.dc1.example.com DC1 (tag optional)

Important

Palomar requires a front LB for proxy ingress; it must be transparent or speak HAProxy PROXY v2.

Modifying a Site ​

You can update status, load balancer, and (if necessary) name.

WARNING

If you rename, it must continue to match cluster_local_site on all servers in that site or the site stops operating.

  • Update status/LB: doveadm cluster site update --status {online|offline|standby|evacuate} [--load-balancer HOST] NAME (or --self). Changing status also clears any active force-move.
  • Status meanings: online (accepts connections), offline (unreachable), standby (reachable, doesn’t accept connections), evacuate (migrating users out).

Site Rebalance ​

When Backend Load Balancing is enabled, the controller automatically rebalances load using a Z-score-based index. It evaluates backend load once per minute and moves users in batches; each user stays on a backend for at least one hour to avoid thrashing.

Automatic balancing waits until enough Prometheus samples have been collected before beginning to move users.

For emergency or planned rebalances between sites, you can use site force-move to drain a percentage of users from one site to one or more destination sites (hash-based selection). Start/update/stop are supported, and the state is tracked in GeoDB.

Setting Site Features ​

Feature states can be toggled per-site via the controller UI or API.

States: on, dryrun, off.

Settings are persisted in GeoDB ([settings,cluster_settings]]).

By default features are disabled; if the table doesn’t exist, all features are treated as enabled for backward compatibility.

UI changes affect only the local site.

Backend Health Auto Handling ​

If enabled, the controller automatically offlines broken hosts and/or moves users away based on failure ratios and cooldowns (e.g., HOST_FAILURE_RATIO, HOST_FAILURE_MIN_LOGINS, HOST_FAILURE_COOL_TIME_SECS).

If disabled, the controller will not take backends offline or move users due to health issues.

Backend Load Balancing ​

If enabled, the controller moves users to even load using Z-scores and thresholds like HOST_LOAD_BALANCE_SCORE_DELTA_THRESHOLD_RATIO, HOST_LOAD_BALANCE_MIN_SAMPLES, and HOST_LOAD_BALANCE_MIN_COOL_TIME_SECS.

If disabled, no automatic moves occur.

Metrics Export ​

If enabled, the controller exports Prometheus metrics; if disabled, the /metrics endpoint won’t serve data.

Proxy Operations ​

Adding a New Proxy ​

Proxies within the same site have identical configuration. Adding a new proxy can be done by just launching a new proxy server and adding it into the load balancer's proxy pool.

Dovecot doesn't explicitly keep track of the proxies. However, they usually show up in the proxy_dest_stats table in GeoDB. It may be useful to manually remove those rows eventually.

Removing a Proxy ​

Remove the proxy server from the load balancer's proxy pool. Wait for a while so at least some of the connections finish on their own. Shutdown the server to forcibly disconnect the rest of the connections.

Backend Operations ​

Adding a New Backend ​

doveadm cluster backend add adds a new Backend.

On the Backend host, --self usually suffices; cluster_local_site and cluster_backend_name are taken from config and the backend is added as standby.

Use UUID only when re-adding. (Backend ID is auto-generated otherwise.)

Additionally, the backend must be added by a resolvable hostname (not raw IP).

Ensure all Proxies/Backends share the same doveadm TCP settings (doveadm_port, doveadm_password) and that monitoring users (e.g., probe-%{backend_host}) are configured on both Proxy and Backend as documented.

Changing Backend Status ​

doveadm cluster backend update --status changes a backend status between offline/standby/online.

  • After controller determines that an offline backend is online again, it changes the status automatically to online.
  • standby backends are not automatically touched.
  • Neither offline nor standby backends receive any new connections. Existing connections are not kicked/moved, so normally backends shouldn't be set standby before the users are moved out.

Changing Load-Factor for Backend ​

doveadm cluster backend update --load-factor changes load-factor for a backend.

This can be used to (temporarily) reduce load from backend. Normally, load should always be set to 100. This is primarily used by the controller.

This setting is not accurate, so setting it to 99 or 78 will not have the expected effect.

Recommended values are 100, 50, and 0 for draining the host.

Evacuating Backend ​

You can evacuate a backend by setting its load-factor to 0 with doveadm cluster backend update --load-factor 0. Controller will notice this and start moving users to other backends. There is also the evacuate button to trigger immediate move of all users to other backends.

If you need a backend evacuation without controller, you can use doveadm cluster user batch move backend --count 1000000 to immediately move all users to other backend(s).

WARNING

This command exits without waiting for the moves to complete.

WARNING

Immediate evacuation methods can potentially increase the load on the source backend significantly if many backends are pulling metacache from it at the same time.

After evacuation is complete, shutdown Dovecot to make sure all sessions are gone.

Finally, set the status of the backend to standby to signify that the backend is not usable for connections.

Removing a Backend ​

Backend Overloaded ​

Controller attempts to prevent backend from becoming overloaded by automatically moving users out, but this may not always happen fast enough. To make it happen faster, you can change backend load-factor to a smaller number or even to 0 to remove all users from the backend.

Disabling Automation ​

Disabling Controller Automation ​

Outside Kubernetes (e.g., Docker Compose), use site features to disable automation:

  • Set Backend Health Auto Handling and/or Backend Load Balancing to off (or dryrun) via the UI/API.
  • The controller UI is available on the controller-api service to make these changes.

User Operations ​

Checking User Status ​

doveadm cluster user status shows the current status for a user.

Accessing User ​

doveadm cluster user access accesses a backend as a given user and prints which backend it was.

This command can also cause a connection to the other site. This command is mainly used internally, but may be helpful also for debugging.

Moving Users to Another Backends ​

doveadm cluster user move backend moves the given user to the given backend.

doveadm cluster user batch move backend moves a number of users to the given backends.

WARNING

This command exits without waiting for the move to complete.

Moving User to Another Site ​

doveadm cluster user move site moves the given user to the given site.

WARNING

This command exits without waiting for the move to complete.

Autoscaler ​

Controller supports autoscaling. There are two autoscalers to choose from:

StaticAutoScaler ​

This is the default scaler, and returns number of backends in site.

LuaAutoScaler ​

This scaler calls method get_desired_number_of_backends(site) and expects it to return number of desired backends.

Managing Controller Features ​

Controller offers feature control via the admin UI or HTTP API. The features that can be switched 'on', 'dryrun' or 'off' are:

  • "Backend Health Auto Handling": Automatically take backends down or bring them back up based on backend Z-score explained in Health Check
  • "Backend Load Balancing": Automatically move users between backends to distribute load as explained in Load Balancing
  • "Metrics Export": Collect dovecot statistics and export them to Prometheus. If this feature is disabled, not metric is exported in response to Prometheus scraping of /metrics HTTP endpoint.

Feature settings are recorded in a table (cluster_settings) in GeoDB and are disabled by default. If the table is not present, all features are enabled for backward-compatibility.

To manage feature settings via the admin page, navigate to Site features page from the left pane. Note that you can only change feature settings of the local site via controller admin UI. To manage the features via the controller API, see OpenAPI documentation.

Monitoring ​

  • Controller → Prometheus: Metrics export must be enabled at the site level
  • Readiness for auto-balancing: Automatic balancing waits until HOST_LOAD_BALANCE_MIN_SAMPLES are collected to avoid premature balancing decisions.
  • What to collect: Follow the Recommended Metrics for Backends (mail/FTS/index I/O, metacache, cluster user-move counters, etc.).

Logging ​

  • Controller: JSON logs from API, scheduler/workers, Prometheus poller, Redis, and init-actions (e.g., redis-wait), which aid startup/operational visibility.
  • Proxy/Backend: Structured logs covering auth/routing (Proxy) and storage/IMAP/POP/LMTP operations (Backend).

Examples ​

Examples ​

Create a Site ​

shell
# create site with cross-site LB and shard tag
doveadm cluster site add --load-balancer lb.dc1.example.com --tag dc1-shard-a DC1

Gracefully Evacuate a Site (Planned Maintenance) ​

shell
# start moving a controlled percentage of users away from DC1 into DC2/DC3
doveadm cluster site force-move start DC1 25 DC2 DC3
# later, increase percentage (e.g., 60%)
doveadm cluster site force-move update DC1 60
# stop when done
doveadm cluster site force-move stop DC1

Add a Backend from a Host ​

shell
# run on the backend host after configuring cluster_local_site and cluster_backend_name
doveadm cluster backend add --self

Doveadm Commands ​

cluster user batch move status

Man Pagedoveadm-cluster-user(1)

Output the status of the currently running user batch move.

CLI

Revision: 09949e1

Last updated:

Copyright © Open-Xchange GmbH