Palomar: Group Size Rebalancing

Cluster Controller can normalize the size of groups assigned to backends by moving users between them.

If enabled (GROUP_BALANCE_ENABLED), this will result in groups on each backend to converge to be roughly the same size.

Group rebalancing is performed in two stages.

First Stage

User groups for each backend is analyzed and if any group size difference is higher than GROUP_BALANCE_GROUP_SIZE_RATIO_SLACK a plan is generated and stored on redis for later execution.

This step is done once per day at midnight (UTC timezone). At this stage, it is decided which specific users should be moved to which groups. The number of users moved is capped to the value of GROUP_BALANCE_MAX_USER_MOVE_BETWEEN_GROUPS.

Second Stage

Done throughout the day, the plan is read periodically and users are reassigned to new groups based on the plan.

This step is done every 30 minutes and at each round a maximum number of users set by GROUP_BALANCE_MAX_USER_MOVES_PER_PASS is moved to new groups.