Palomar: Cluster Controller Settings

This page lists settings used by the Cluster Controller.

For Dovecot Core Palomar settings, see Palomar Configuration Settings.

Configuration Notes

Default Configuration

All deployment methods provide a default controller.cfg that should get you started and allow you to tweak settings based on your needs (e.g, configuration of an external Cassandra).

Database

Palomar GeoDB is automatically populated with local site information during installation for all deployment methods.

You can also manually add new sites using Controller's admin UI or by making API calls to the Controller.

Backend Initialization

This is automated ONLY in Kubernetes deployments.

For Docker Compose setups, please refer to Palomar Administration section for details on how to populate backends into GeoDB. Alternatively, you can use Controller API linked above to add the backends.

Monitoring

There is no strict monitoring for Cluster Controller, but using standard methods to track the state of your containers or pods is recommended.

Additionally, a basic UI is available on the controller-api container, allowing graphical interaction with Cluster Controller.

Settings

BACKEND_REACHABILITY_NAME_RESOLUTION_FAILURE_DURATION_ALLOWED_SECS

Default	10
Value	Python Integer
Changes	Added: 3.2.0

Used to determine the duration of time in seconds the backend host name fails to resolve within the cluster to mark it as an unreachable backend. Cluster Controller periodically tries to query each backend on its doveadm HTTP API and if any error is detected, it will not choose the backend with errors for user moves.

To not be too sensitive to transient name resolution errors, a period
of time is allowed before the backend is deemed unreachable from
controller. The backend is unmarked as soon as there is a successful
query.

BACKEND_REACHABILITY_NETWORK_FAILURE_DURATION_ALLOWED_SECS

Default	10
Value	Python Integer
Changes	Added: 3.2.0

Similar to DNS errors, this setting determines how long generic network errors are ignored before they are treated as error.

Note that only name resolution and network error errors have these
grace periods. If querying a backend fails due to authentication error
(e.g. wrong doveadm password set in Controller settings) the backend
is immediately marked as unreachable.

CASSANDRA_GEODB_KEYSPACE

Default	"d8s_cluster"
Value	Python String
Changes	Added: 3.0.1

Cassandra keyspace used for dovecot cluster.

CASSANDRA_KEYSPACE

Default	"d8s_cluster"
Value	Python String
Changes	Deprecated: 3.0.1 Use CASSANDRA_GEODB_KEYSPACE instead.

Cassandra keyspace to use.

CASSANDRA_LOAD_BALANCING_POLICY

Default	"DCAwareRoundRobin"
Value	Python String
Allowed Values	"RoundRobin""DCAwareRoundRobin"
Changes	Added: 3.0.1

Policy to use for cassandra load balancing. Determines how requests are distributed among nodes in the cluster.

• "RoundRobin": evenly distributes queries across all nodes in the cluster, regardless of what datacenter the nodes may be in.
• "DCAwareRoundRobin": Similar to "RoundRobinPolicy", but prefers hosts in the local datacenter and only uses nodes in remote datacenters as a last resort. If this policy is selected, number of remote hosts to use can be configured with CASSANDRA_USED_HOSTS_PER_REMOTE_DC.

CASSANDRA_LOCAL_DC

Default	"d8s_cluster"
Value	Python String
Changes	Added: 3.0.1

Name of the local datacenter in Cassandra cluster.

CASSANDRA_PORT

Default	9042
Value	Python Integer

The port Cassandra servers listen on.

CASSANDRA_PROTOCOL_VERSION

Default	4
Value	Python Integer

Version of the cassandra protocol to be used.

CASSANDRA_SERVERS

Default	"localhost"
Value	Python String

Comma separated list of Cassandra server endpoints.

CASSANDRA_SITE

Default	"d8s_cluster"
Value	Python String
Changes	Deprecated: 3.0.1 Use CASSANDRA_LOCAL_DC instead.

Name of the local datacenter in Cassandra cluster.

CASSANDRA_TLS_ENABLED

Default	False
Value	Python Boolean

Whether to use TLS in Cassandra connections.

CASSANDRA_USED_HOSTS_PER_REMOTE_DC

Default	0
Value	Python Integer
Changes	Added: 3.0.1

If CASSANDRA_LOAD_BALANCING_POLICY is set to "DCAwareRoundRobin", controls how many nodes in each remote datacenter will have connections opened against them.

CELERY_RESULT_EXPIRES_SECS

Default	60
Value	Python Integer

Time to live of task results in Redis (in seconds).

CLUSTER_SITE

Default	"dc1a"
Value	Python String

Palomar site this controller is responsible for. Must be the same site name configured in dovecot proxies with cluster_local_site.

CONTROLLER_API_URL

Default	"http://localhost:8000"
Value	Python String

URL controller API runs on. The server will bind to this address and the admin UI uses this address to send the HTTP requests.

DRY_RUN

Default	200
Value	Python Integer
Changes	Deprecated: 3.0.2 Use feature state 'dryrun'.

Whether to enable DRY_RUN mode to log but not perform controller worker actions, such as: set_host_offline, set_host_online and move users.

GROUP_MOVE_FINISH_TIMEOUT_SECS

Default	600
Value	Python Integer
Changes	Added: 3.0.1 Deprecated: 3.1.0

Maximum time for a group move to finish in seconds. Controller tries to calculate the finish time based on the trend of moved users so far. If it seems that backends won't be able to finish the move in time, it will be forced. A forced move is done by directly setting group's backend ID in GeoDB.

GROUP_MOVE_START_TIMEOUT_SECS

Default	180
Value	Python Integer
Changes	Added: 3.0.1 Deprecated: 3.1.0

Maximum allowed time of a group move to start in seconds. If backends don't start moving group after this time, a force move will be done. A forced move is done by directly setting group's backend ID in GeoDB.

HOST_FAILURE_BACKEND_NUM_THRESHOLD

Default	0.3
Value	Python Float

Percentage of backends in a site that are allowed to become offline before stopping backend evacuations within the site.

HOST_FAILURE_COOL_TIME_SECS

Default	3600
Value	Python Integer

Minimum time in seconds a user will not be moved to a new backend because of backend health decisions. The users' move timestamps are tracked by backends, so users that have been moved for any reason by either Controller or doveadm commands will not be moved again until this many seconds have passed.

HOST_FAILURE_FORCE_MOVE_PERCENTAGE_INCREASE_INTERVAL_SECONDS

Default	60
Value	Python Integer
Changes	Added: 3.1.0

Time interval between increases in percentage of users when backend is being evacuated in a force-move operation. If a backend is slow in moving users to another backend (due to admin's decision to set load factor to 0 or for health handling by controller) a force-move is started in which proxies are instructed to move a portion of users away to other healthy backends. The amount of users being moved to other backends starts at 10% then after a duration of time, configurable by this setting, it is increased to 40% then 70% and finally 100%. In all these percentage increases the time interval applies as well.

HOST_FAILURE_MIN_LOGINS

Default	10
Value	Python Integer

Minimum number of logins needed in past 5 minutes to start processing host's health (i.e. change backend status if necessary or move users from it if there is high failure rate).

HOST_FAILURE_RATIO

Default	0.1
Value	Python Float

Ratio of failed logins or mail deliveries to trigger user moves. Must be in (0, 1) range exclusive.

HOST_LOAD_BALANCE_GROUP_MIN_COOL_TIME_SECS

Default	3600
Value	Python Integer
Changes	Added: 3.0.1 Deprecated: 3.1.0 Use HOST_LOAD_BALANCE_MIN_COOL_TIME_SECS instead.

Minimum time in seconds a group will not be moved to a new backend. If a group needs to be moved for load balancing, this period is honored and groups that have been moved recently will not be moved again. Must be larger than 0.

HOST_LOAD_BALANCE_MIN_COOL_TIME_SECS

Default	3600
Value	Python Integer

Used to determine if a user or a backend has been subject to user moves recently and should be skipped for load balancing.

For users, it determines the minimum time in seconds a user will not be moved to a new backend because of load balancing decisions. The users' move timestamps are tracked by backends, so users that have been moved for any reason by either Controller or doveadm commands will not be moved again until this many seconds have passed.

For backends, it determines the minimum time in seconds no move will be done from or to the backend. The timestamp of last user move from or to the backend is tracked by Controller and at each load balancing iteration, backends that have had a user move within this period will be skipped.

HOST_LOAD_BALANCE_MIN_SAMPLES

Default	3000
Value	Python Integer

Number of samples over the last 24 hours needed for all the Z-scores for a host to do load balancing.

HOST_LOAD_BALANCE_SCORE_DELTA_THRESHOLD_RATIO

Default	0.5
Value	Python Float

Minimum load score difference between backends to initiate user move. See Cluster Controller Load Balancing.

LOG_COLOR_ENABLED

Default	False
Value	Python Boolean
Changes	Added: 3.2.0

Whether coloured logging is enabled for controller when non-structured logging is selected (coloured logging is always disabled for structured logging).

LOG_LEVEL

Default	info
Value	Python String
Allowed Values	criticalerrorwarninginfodebug
Changes	Added: 3.1.2

Log level for controller.

LOG_STRUCTURED

Default	False
Value	Python Boolean
Changes	Added: 3.1.2

Whether structured logging is enabled for controller. If set to True, logs are emitted as a json payload.

SITE_USER_MOVE_BATCH_INTERVAL_SECONDS

Default	1
Value	Python Integer
Changes	Added: 3.2.0

The number of seconds to wait between each move command for a batch of users when moving users out of a site. Can also be set to 0 to not wait at all.

SITE_USER_MOVE_MAX_BATCH_SIZE

Default	200
Value	Python Integer
Changes	Added: 3.2.0

Maximum number of users that can be moved in each batch of user site moves. User site moves are performed in batch of users and there is a small delay between each request to individual backends. This ensures that destination site(s) will not have load spikes. Note that this limit is for each backend move. There can be as many parallel moves from each backend in the site with the user count of this value. If the backend has user count below the value for this setting, all parameter will be used instead of giving the user count.

USER_MOVE_FINISH_TIMEOUT_SECS

Default	600
Value	Python Integer
See Also	USER_MOVE_START_TIMEOUT_SECS
Changes	Added: 3.1.0

Maximum time for non load-balancing user move operations to finish in seconds. Controller tries to calculate the finish time based on the trend of moved users so far. If it seems that backends won't be able to finish the move in time, the operation will be retried once and if still not successful, it will be forced. A forced move is done by writing to force_move_percentage column in backends table.

USER_MOVE_START_TIMEOUT_SECS

Default	180
Value	Python Integer
See Also	USER_MOVE_FINISH_TIMEOUT_SECS
Changes	Added: 3.1.0

Maximum allowed time of a non load-balancing user move operation to start in seconds. If backends don't start moving user after this time, the operation will be retried once. If the move is still not successful on the second try, a force move will be triggered. A forced move is done by populating force_move_percentage on backend's table in GeoDB.