Palomar: Administration

Site Operations

Adding a New Site

doveadm cluster site add adds a new site.

This will add a new site, in online status. You can specify site load balancer hostname, which is used when connections are routed from site to site. Tag is optional and will be used to match users and backends to a site shard. UUID should only be used when re-adding a site.

Initializing Groups in a New Site

Generally speaking, there is no need to initialize user groups. If a group does not exist yet, and a user of that groups logs in, the group is created automatically.

In case the need to manually create user groups arises, the doveadm cluster site init command can be used.

This command initializes a site by creating cluster_default_group_count user groups in GeoDB.

If --update is passed, this command removes existing groups and recreates them. It can thereby shrink or grow the number of groups depending on cluster_default_group_count. Users also get their groups unassigned when running with --update.

Note

Running with --update deletes existing groups and can take a while to run, so do NOT use this in a production environment.

Proxy Operations

Adding a New Proxy

Proxies within the same site have identical configuration. Adding a new proxy can be done by just launching a new proxy server and adding it into the load balancer's proxy pool.

Dovecot doesn't explicitly keep track of the proxies. However, they usually show up in the proxy_dest_stats table in GeoDB. It may be useful to manually remove those rows eventually.

Removing a Proxy

Remove the proxy server from the load balancer's proxy pool. Wait for a while so at least some of the connections finish on their own. Shutdown the server to forcibly disconnect the rest of the connections.

Backend Operations

Adding a New Backend

doveadm cluster backend add adds a new backend.

You can usually use just --self when adding the host you are on. Site (cluster_local_site) and hostname (cluster_backend_name) are inferred from config. The host will be added in standby mode. UUID should only be used when re-adding a backend.

Changing Backend Status

doveadm cluster backend update --status changes a backend status between offline/standby/online.

• After controller determines that an offline backend is online again, it changes the status automatically to online.
• standby backends are not automatically touched.
• Neither offline nor standby backends receive any new connections. Existing connections are not kicked/moved, so normally backends shouldn't be set standby before the users are moved out.

Changing Load-Factor for Backend

doveadm cluster backend update --load-factor changes load-factor for a backend.

This can be used to (temporarily) reduce load from backend. Normally, load should always be set to 100. This is primarily used by the controller, which starts the group moves.

This setting is not accurate, so setting it to 99 or 78 will not have the expected effect.

Recommended values are 100, 50, and 0 for draining the host.

Evacuating Backend

doveadm cluster backend evacuate evacuates a backend with full speed:

• Set backend load factor to 0
• Set backend status to standby
• Assign new random backends to groups currently using this backend, and start the moves immediately.

If you're not in a hurry to evacuate the backend immediately, it's better to only set the load factor to 0 and let the controller move the groups out of the backend slowly.

NOTE

This command exits without waiting for the moves to complete.

Removing a Backend

• Start evacuating the server
• Wait for no users to exist in the backend (doveadm who)
• Optionally remove the server completely (doveadm cluster backend remove)

Backend Overloaded

Controller attempts to prevent backend from becoming overloaded by automatically moving groups out, but this may not always happen fast enough. To make it happen faster, you can change backend load-factor to a smaller number or even to 0 to remove all users from the backend. If this still isn't fast enough, use doveadm cluster backend evacuate command.

Disabling Automation

Disabling Controller Automation

⚠️ TODO

Provide non-k8s information here

Group Operations

Moving Group to Another Backend

doveadm cluster group move moves a group within a site.

The move is started by running the doveadm cluster group access command on the source backend.

NOTE

This command exits without waiting for the move to complete.

Listing All Groups

doveadm cluster group list lists all groups.

Checking Status of Group

doveadm cluster group status shows the current status of the group for a site.

User Operations

Checking User Status

doveadm cluster user status shows the current status for a user.

Accessing User

doveadm cluster user access accesses a backend as a given user and prints which backend it was.

This command can also cause a connection to the other site. This command is mainly used internally, but may be helpful also for debugging.

Moving User to Another Group

doveadm cluster user move moves the given user to the given group, and optionally to a different site.

The move is started by running the doveadm cluster user access command on the source backend.

NOTE

This command exits without waiting for the move to complete.

Autoscaler

Controller supports autoscaling. There are two autoscalers to choose from:

StaticAutoScaler

This is the default scaler, and returns number of backends in site.

LuaAutoScaler

This scaler calls method get_desired_number_of_backends(site) and expects it to return number of desired backends.

REST API Reference

• Browse OpenAPI specification
• Download OpenAPI specification

Managing Controller Features

Controller offers feature control via the admin UI or HTTP API. The features that can be switched on or off are:

• "Backend Health Auto Handling": Automatically take backends down or bring them back up based on backend Z-score explained in Health Check
• "Backend Load Balancing": Automatically move groups between backends to distribute load as explained in Load Balancing
• "Metrics Export": Collect dovecot statistics and export them to Prometheus. If this feature is disabled, not metric is exported in response to Prometheus scraping of /metrics HTTP endpoint.

Feature settings are recorded in a table (cluster_settings) in GeoDB and are disabled by default. If the table is not present, all features are enabled for backward-compatibility.

To manage feature settings via the admin page, navigate to Site features page from the left pane. Note that you can only change feature settings of the local site via controller admin UI. To manage the features via the controller API, see OpenAPI documentation.

Monitoring

⚠️ TODO

Logging

⚠️ TODO

Doveadm Commands

cluster backend add

Man Page	doveadm-cluster-backend(1)

Adds a new backend to cluster. New backends are added in standby state. ID should not be used unless you are re-adding the same backend.

CLI

• Usage: doveadm cluster backend add [--self self] [--site site] [--id id] [--host host]

Argument(s)	Type	Description
--self		If set, targets the local site. (host will be ignored)
--site	string
--id	string
--host	string

HTTP API

Parameter	Type	Description
self		If set, targets the local site. (host will be ignored)
site	string
id	string
host	string

cluster backend evacuate

Man Page	doveadm-cluster-backend(1)

Start moving all user groups to other backends. Will not wait for the moves to finish. Also sets the backend to standby and load factor to 0.

CLI

• Usage: doveadm cluster backend evacuate [--self self] [--host host]

Argument(s)	Type	Description
--self		If set, targets the local site. (host will be ignored)
--host	string

HTTP API

Parameter	Type	Description
self		If set, targets the local site. (host will be ignored)
host	string

cluster backend list

Man Page	doveadm-cluster-backend(1)

Lists all backends in this cluster, including other sites.

CLI

• Usage: doveadm cluster backend list [--site site]

Argument(s)	Type	Description
--site	string	Only list specific site given.

HTTP API

Parameter	Type	Description
site	string	Only list specific site given.

cluster backend remove

Man Page	doveadm-cluster-backend(1)

Remove a backend from cluster.

CLI

• Usage: doveadm cluster backend remove [--self self] [--host host]

Argument(s)	Type	Description
--self		If set, targets the local site. (host will be ignored)
--host	string

HTTP API

Parameter	Type	Description
self		If set, targets the local site. (host will be ignored)
host	string

cluster backend status

Man Page	doveadm-cluster-backend(1)

Shows the current status of the backend.

CLI

• Usage: doveadm cluster backend status [--self self] [--host host]

Argument(s)	Type	Description
--self		If set, targets the local site. (host will be ignored)
--host	string

HTTP API

Parameter	Type	Description
self		If set, targets the local site. (host will be ignored)
host	string

cluster backend update

Man Page	doveadm-cluster-backend(1)

CLI

• Usage: doveadm cluster backend update [--self self] [--status status] [--load-factor load-factor] [--host host]

Argument(s)	Type	Description
--self		If set, targets the local site. (host will be ignored)
--status	string	online: Host is online and accepts connections standby: Host is online but is not usable for connections offline: Host is offline and will be turned back online when it recovers
--load-factor
--host	string

HTTP API

Parameter	Type	Description
self		If set, targets the local site. (host will be ignored)
status	string	online: Host is online and accepts connections standby: Host is online but is not usable for connections offline: Host is offline and will be turned back online when it recovers
loadFactor
host	string

cluster geodb refresh

Man Page	doveadm-cluster-geodb(1)

Refreshes local caches from GeoDB. This is mainly intended for testing purposes. Usually this is not necessary to run, because cluster has been designed to work correctly even when caches aren’t up-to-date.

CLI

• Usage: doveadm cluster geodb refresh

cluster group access

Man Page	doveadm-cluster-group(1)

Perform cluster lookup for the cluster_backend_test_username user as if it belonged to the specified group. This command must be run from a backend, not a proxy. For now this command is mainly used internally to finish user group moves.

CLI

• Usage: doveadm cluster group access <group>

Argument(s)	Type	Description
group	string

HTTP API

Parameter	Type	Description
group	string

cluster group list

Man Page	doveadm-cluster-group(1)

List all user groups.

CLI

• Usage: doveadm cluster group list [--site site] <mask>

Argument(s)	Type	Description
--site	string
mask	string

HTTP API

Parameter	Type	Description
site	string
mask	string

cluster group move

Man Page	doveadm-cluster-group(1)

Initiate a user group move, which finishes on background. The move is initialized by updating the moving and alt_backed_id fields in GeoDB. The next time a user in the group is accessed triggers the actual moving by the original backend. This user access is also automatically triggered by running doveadm cluster group access command in the group’s original backend.

CLI

• Usage: doveadm cluster group move <site> <group> <host>

Argument(s)	Type	Description
site	string
group	string
host	string

HTTP API

Parameter	Type	Description
site	string
group	string
host	string

cluster group status

Man Page	doveadm-cluster-group(1)

Display current status of a user group.

CLI

• Usage: doveadm cluster group status <site> <group>

Argument(s)	Type	Description
site	string
group	string

HTTP API

Parameter	Type	Description
site	string
group	string

cluster group update

Man Page	doveadm-cluster-group(1)

Updates group settings.

CLI

• Usage: doveadm cluster group update [--sticky-backend sticky-backend] [--sticky-users sticky-users] <site> <group>

Argument(s)	Type	Description
--sticky-backend		Whether this user group is eligible to be moved automatically to another backend in load balancing.
--sticky-users		Whether users in this user group are eligible to be moved automatically to other groups.
site	string
group	string

HTTP API

Parameter	Type	Description
stickyBackend		Whether this user group is eligible to be moved automatically to another backend in load balancing.
stickyUsers		Whether users in this user group are eligible to be moved automatically to other groups.
site	string
group	string

cluster kick

Man Page	doveadm-cluster-kick(1)

Disconnect user’s connections from Palomar

CLI

• Usage: doveadm cluster kick <user>

Argument(s)	Type	Description
user	string

HTTP API

Parameter	Type	Description
user	string

cluster site add

Man Page	doveadm-cluster-site(1)

Adds a new site to cluster. Load balancer is the host or IP address where another site’s proxy connects to do site to site connections. New sites are added in online state. UUID should not be used unless you are re-adding the same site.

CLI

• Usage: doveadm cluster site add [--id id] [--tag tag] [--load-balancer load-balancer] <name>

Argument(s)	Type	Description
--id	string
--tag	string
--load-balancer	string
name	string

HTTP API

Parameter	Type	Description
id	string
tag	string
loadBalancer	string
name	string

cluster site init

Man Page	doveadm-cluster-site(1)

Initialise site by creating user groups.

CLI

• Usage: doveadm cluster site init [--update update] [--self self] <name>

Argument(s)	Type	Description
--update
--self		If set, targets the local site. (name will be ignored)
name	string

HTTP API

Parameter	Type	Description
update
self		If set, targets the local site. (name will be ignored)
name	string

cluster site list

Man Page	doveadm-cluster-site(1)

List all sites.

CLI

• Usage: doveadm cluster site list

cluster site remove

Man Page	doveadm-cluster-site(1)

Remove a site. This currently has no safety checks, so be sure to remove first all of the site’s backends and remove it from all users’ preferred sites.

CLI

• Usage: doveadm cluster site remove <name>

Argument(s)	Type	Description
name	string

HTTP API

Parameter	Type	Description
name	string

cluster site status

Man Page	doveadm-cluster-site(1)

Site status.

CLI

• Usage: doveadm cluster site status <name>

Argument(s)	Type	Description
name	string

HTTP API

Parameter	Type	Description
name	string

cluster site update

Man Page	doveadm-cluster-site(1)

Update site status and load balancer.

CLI

• Usage: doveadm cluster site update [--self self] [--load-balancer load-balancer] [--status status] <name>

Argument(s)	Type	Description
--self		If set, targets the local site. (name will be ignored)
--load-balancer	string
--status	string
name	string

HTTP API

Parameter	Type	Description
self		If set, targets the local site. (name will be ignored)
loadBalancer	string
status	string
name	string

cluster tag create

Man Page	doveadm-cluster-tag(1)

Creates a new tag for cluster. UUID can be provided, if not, it will be generated.

CLI

• Usage: doveadm cluster tag create [--id id] <name>

Argument(s)	Type	Description
--id	string
name	string

HTTP API

Parameter	Type	Description
id	string
name	string

cluster tag delete

Man Page	doveadm-cluster-tag(1)

Delete a tag. This currently has no safety checks, so be sure to remove first all of the tag’s sites.

CLI

• Usage: doveadm cluster tag delete <id>

Argument(s)	Type	Description
id	string	Name or ID.

HTTP API

Parameter	Type	Description
id	string	Name or ID.

cluster tag list

Man Page	doveadm-cluster-tag(1)

List all tags.

CLI

• Usage: doveadm cluster tag list

cluster tag update

Man Page	doveadm-cluster-tag(1)

Change tag name.

CLI

• Usage: doveadm cluster tag update [--name name] <id>

Argument(s)	Type	Description
--name	string
id	string

HTTP API

Parameter	Type	Description
name	string
id	string

cluster user access

Man Page	doveadm-cluster-user(1)

Perform cluster lookup for the user (or users). Displays the host name or IP address of the backend which the user ended up. This command can be run from either a proxy or a backend. This command is also used internally to finish user moves.

CLI

• Usage: doveadm cluster user access [-i ip] [-A] [-S socket-path] [-u user] [-F user-file]

Argument(s)	Type	Description
-i		Display backend's IP address rather than hostname.
-A	boolean	Apply operation to all users.
-S	string	Path to doveadm socket.
-u	string	UID of user to apply operation to.
-F	string	A filename. If set, fetch usernames from file. One username per line.

HTTP API

Parameter	Type	Description
ip		Display backend's IP address rather than hostname.
allUsers	boolean	Apply operation to all users.
socketPath	string	Path to doveadm socket.
user	string	UID of user to apply operation to.
userFile	string	A filename. If set, fetch usernames from file. One username per line.

cluster user delete

Man Page	doveadm-cluster-user(1)

Deletes a record from GeoDB.

CLI

• Usage: doveadm cluster user delete <user>

Argument(s)	Type	Description
user	string

HTTP API

Parameter	Type	Description
user	string

cluster user move

Man Page	doveadm-cluster-user(1)

Initiates moving user to a different user group which will finish in background. The move is initialized by updating the moving_* fields in GeoDB. The next time the user is accessed triggers the actual moving by the original backend. This user access is also automatically triggered by running doveadm cluster user access command.

CLI

• Usage: doveadm cluster user move [--site site] <user> <group> [failover group]

Argument(s)	Type	Description
--site	string
user	string
group	string
failover-group	string

HTTP API

Parameter	Type	Description
site	string
user	string
group	string
failoverGroup	string

cluster user status

Man Page	doveadm-cluster-user(1)

Displays status information of a user.

CLI

• Usage: doveadm cluster user status <user>

Argument(s)	Type	Description
user	string

HTTP API

Parameter	Type	Description
user	string