Palomar: Load Rebalancing

If enabled, Cluster Controller can perform automatic load balancing based on collected data from backends.

Briefly, a load index is assigned to all backends called Z-score. If backends have too big of a variation in load, a group move between the backend with the highest load and the backend with the lowest load is triggered. The difference in load that triggers the balancing is set by HOST_LOAD_BALANCE_SCORE_DELTA_THRESHOLD_RATIO.

A group move is an operation where controller decides that all users of a group should be routed to a different dovecot backend. When this decision is made, controller updates the group in GeoDB with the new backend. The actual moving starts the next time a user that belongs to the group logs in or receives an email via LMTP.

The process of monitoring and adjusting backend load is a continuous operation that is periodically done by controller. By default 1 group per hour is moved. At the start and end of group move Dovecot emits cluster_user_group_move_started and cluster_user_group_move_finished respectively.

To safeguard the site from making too early decisions about load, automatic load balancing is deferred until a sufficient amount of data is collected by Prometheus (on the controller). The value of this minimum number of data samples can be tweaked with HOST_LOAD_BALANCE_MIN_SAMPLES. By default, it's set to 3000 samples which would roughly take 12 hours to collect.