Palomar: Configuration

This page describes the Dovecot config necessary to implement the Palomar Architecture.

WARNING

If you are upgrading from OX Dovecot Pro 2.3.x, see Transition from OX Dovecot Pro 2.3.x Architecture.

Load Balancers

Palomar requires a layer of load balancers to balance traffic ingress to the Dovecot Proxy.

Requirements

The load balancer MUST either be transparent (keeps the original client IP visible) or it MUST support HAProxy PROXY V2 protocol.

Round-robin distribution of incoming connections to the pool of available site proxies is a common setup. More advanced traffic-management can be done as well, although specific configuration details for the load balancer is site-dependent and is out-of scope for this documentation.

Example

load-balancer: HAproxy configuration example

# haproxy.conf
#
# Sample Configuration for a Palomar load balancing setup using software
# (HAproxy) load balancing
#
# Sample assumptions:
#   - Incoming traffic will be balanced to 2 proxies, located at 192.168.1.10
#     and 192.168.1.11

defaults
    mode                    tcp
    log                     global
    option                  dontlognull
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 128000

listen lmtp
   bind 0.0.0.0:24
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:24 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:24 check send-proxy-v2

listen pop3
   bind 0.0.0.0:110
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:110 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:110 check send-proxy-v2

listen imap4
   bind 0.0.0.0:143
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:143 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:143 check send-proxy-v2

listen sieve
   bind 0.0.0.0:4190
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:4190 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:4190 check send-proxy-v2

listen submission
   bind 0.0.0.0:587 accept-proxy
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:587 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:587 check send-proxy-v2

Dovecot Proxy

Dovecot Proxies authenticate the user, convert to internal user ID (if needed), and redirect to the Dovecot Backend.

Installation

A Dovecot Proxy requires the dovecot-pro-cluster package.

Configuration

Documentation on cluster settings can be found below.

# Name of the proxy's local site
cluster_local_site = DC1

# Include all the required settings for running in cluster proxy mode
@cluster_defaults = proxy

GeoDB / LocalDB

See Palomar GeoDB.

The GeoDB tables need to be created. See Cassandra.

⚠️ TODO

Add links to Cassandra configuration pages

dict_server {
  dict cluster-localdb {
    sqlite_path = /dev/shm/cluster-localdb.sqlite
  }
}

# Cassandra is used for GeoDB
#cassandra_hosts = ...

Trusted Networks

If HAProxy (or another non-transparent load balancer) is used, its IPs must be trusted:

haproxy_trusted_networks = 192.168.1.0/24

See haproxy_trusted_networks.

Dovecot Proxy IPs/networks that must be trusted (via login_trusted_networks):

Dovecot Proxies in other sites. If user connection is going to a wrong site, the proxy will connect to the correct site's proxy (via the load balancer).
cluster-backend-test service connects to the local proxy. This is based on the first IP address in service imap-login { inet_listener }, which defaults to the listen. Typically this is 127.0.0.1 or ::1.

login_trusted_networks = 127.0.0.1 ::1 192.168.1.0/24

Doveadm

The Doveadm TCP service must be configured for all Proxies and Backends using the same doveadm_port and doveadm_password.

doveadm_port = 2300
doveadm_password = # shared password

service doveadm {
  inet_listener tcp {
    port = 2300 # same as doveadm_port
  }
}

Monitoring Backends/Users

Palomar requires "test" accounts to be configured so that backends can be monitored.

The test username is recommended to use %{backend_host}, which expands to the tested backend. This way if there is a problem with the test user, it affects only a single backend.

The proxy needs a passdb listing the test users. They could be in the same passdb as the real users, or they can have a separate passdb. In the latter case, it should be placed as the first passdb to catch the test users.

cluster_backend_test_username = probe-%{backend_host}
cluster_backend_test_password = # shared password

# Passdb configuration for monitoring users first:
passdb test-accounts {
  args = /etc/dovecot/cluster-test-accounts.passwd
  fields {
    proxy = y
    password = # shared password
  }
  driver = passwd-file
}

# Passdb for the real users:
#passdb ldap {
#  ...
#}

See:

Dovecot Backend

Dovecot Backends perform all work related to protocol commands and interfaces with the storage.

Installation

A Dovecot Backend requires the dovecot-pro-cluster package.

Configuration

Documentation on cluster settings can be found below.

# Name of the backend (same as in geodb)
cluster_backend_name = hac-dc1-be3
# Name of the backend's local site
cluster_local_site = DC1

# Include all the required settings for running in cluster backend mode
@cluster_defaults = backend

GeoDB / LocalDB

See Palomar GeoDB.

The GeoDB tables need to be created. See Cassandra.

⚠️ TODO

Add links to Cassandra configuration pages

dict_server {
  dict cluster-localdb {
    sqlite_path = /dev/shm/cluster-localdb.sqlite
  }
}

# Cassandra is used for GeoDB
#cassandra_hosts = ...

Doveadm

The Doveadm TCP service must be configured for all Proxies and Backends using the same doveadm_port and doveadm_password.

doveadm_port = 2300
doveadm_password = # shared password

service doveadm {
  inet_listener tcp {
    port = 2300 # same as doveadm_port
  }
}

Monitoring Users

Each backend must receive test user logins from proxies. The users can be in the same passdb as the real users, or they can have a separate passdb. In the latter case, it should be placed as the first passdb to catch the test users.

# Passdb configuration for monitoring users first:
passdb {
  args = /etc/dovecot/cluster-test-accounts.passwd
  driver = passwd-file
}

# Passdb for the real users:
#passdb ldap {
#  ...
#}

# Backends use this in "doveadm cluster group access" command:
cluster_backend_test_username = probe-%{backend_host}

See:

Userdb Fields

These settings are necessary to allow for user listing and metacache pulling.

The userdb on backends must provide the user_cluster_group and metacache_last_host_dict fields. To do that we are returning user_cluster_group=%{passdb:forward_cl_user_group} and metacache_last_host_dict=%{passdb:forward_meta_last_host_dict} from userdb.

⚠️ TODO

Shouldn't user_cluster_group and metacache_last_host_dict be documented in settings?

Also forward_cl_user_group and forward_meta_last_host_dict.

I understand they are not "settings" as set by a user, but they have defined behavior.

Alternatively, these fields could be returned by passdb as long as they are prefixed with userdb_, i.e. userdb_user_cluster_group and userdb_metacache_last_host_dict. In any case, the passdb/userdb must succeed for all the valid logins and other user accesses so the fields are always added. This configuration applies only to login-based protocols (imap, pop3, submission, managesieve). For lmtp and doveadm they are hardcoded.

Cluster process sets forward_cl_user_group and forward_meta_last_host_dict when applicable so backends can use them. The following example is using a static userdb (must be adapted to the actually used userdb or passdb):

userdb static {
  fields {
    user_cluster_group = %{passdb:forward_cl_user_group}
  }
}

Suggested metric configuration:

metric metacache_pull_finished {
    filter = event=metacache_pull_finished and error=""
    group_by type {
    }
}
metric metacache_pull_finished_failure {
   filter = event=metacache_pull_finished and not error=""
   group_by type {
   }
}

Cassandra

You need to setup a Cassandra namespace for Palomar.

Cassandra Setup

cql

create table if not exists tags (
  id uuid,
  tag text,
  primary key ((id))
);

create table if not exists sites (
  id uuid,
  name text,
  tag uuid,
  load_balancer text,
  status text,
  primary key ((id))
);

create table if not exists site_reachability (
  src_site_id uuid,
  dest_site_id uuid,
  reachable int,
  primary key ((src_site_id), dest_site_id)
);

create table if not exists backends (
  id uuid,
  site_id uuid,
  load_factor int,
  host text,
  status text,
  status_reason text,
  last_moved_from text,
  last_moved_to text,
  primary key ((site_id), id)
);

create table if not exists backend_stats (
  site_id uuid,
  id uuid,
  key text,
  value double,
  type text,
  primary key ((site_id), id, key)
);

create table if not exists proxy_dest_stats (
  proxy_site_id uuid,
  proxy_host text,
  dest_host text,
  key text,
  value double,
  type text,
  primary key ((proxy_site_id), dest_host, proxy_host, key)
);

create table if not exists user_groups (
  name text,
  site_id uuid,
  backend_id uuid,
  alt_backend_id uuid,
  sticky_users int,
  moving text,
  refresh_after timestamp,
  sticky_backend int,
  primary key ((site_id), name),
);

create table if not exists group_sites (
  group_name text,
  site_id uuid,
  status text,
  failover_site_id uuid,
  last_update timestamp,
  primary key ((group_name), site_id),
);

create table if not exists users (
  username text,
  preferred_site_id uuid,
  group_name text,
  failover_site_group_name text,
  metacache_last_host text,
  moving_preferred_site_id uuid,
  moving_group_name text,
  moving_failover_site_group_name text,
  last_update timestamp,
  primary key ((username))
);


create table if not exists cluster_settings (
  section text,
  site_id uuid,
  backend_id uuid,
  setting_name text,
  setting_value text,
  primary key ((section), site_id, backend_id, setting_name),
);

Settings

Palomar Core Configuration

`cluster_backend_name`

Default	[None]
Value	string

Host name of the backend. This must be the same name as used by doveadm cluster-backend commands.

Cluster: Applies only to Backend.

`cluster_backend_test_password`

Default	[None]
Value	string

Password used for logging into backends to see if it's up or down.

Cluster: Applies only to Proxy.

`cluster_backend_test_username`

Default	[None]
Value	string

This setting is used for two purposes:

Username used by proxy for logging into backends to see if it's up or down. %{backend_host} variable expands to the hostname of the backend.
Username used by backend for running doveadm cluster group access command. This command just needs any existing user to work - it doesn't matter that it's not actually in the accessed group.

Cluster: Applies to both Proxy and Backend.

`cluster_default_group_count`

Default	[None]
Value	unsigned integer

Number of user groups that may be automatically created. This is used for creating a group for a user that doesn't yet have one. The group will be named default-N where N is between 1 and cluster_default_group_count.

Cluster: Applies only to Proxy.

`cluster_geodb`

Default	[None]
Value	string

Dictionary URI used for the globally shared GeoDB. This typically points to Cassandra.

Cluster: Applies to both Proxy and Backend.

`cluster_local_site`

Default	[None]
Value	string

Name of the local site. This must be the same name as used by doveadm cluster-site commands.

Cluster: Applies to both Proxy and Backend.

`cluster_localdb`

Default	[None]
Value	string

Dictionary URI used for the server-specific local database. This typically points to SQLite under /dev/shm.

Cluster: Applies to both Proxy and Backend.

`cluster_proxy_check_backends`

Default	`yes`
Value	string

If enabled, this proxy runs checks to see whether backends are up or down.

If disabled, this proxy never sets any backends offline.

Cluster: Applies only to Proxy.

`cluster_user_move_timeout`

Default	`31secs`
Value	time

If user moving hasn't finished by this timeout, just assume it finished and continue to the next user.

Cluster: Applies only to Proxy.

Cluster Controller

See Cluster Controller Settings.

Databases

Mechanisms

Users

Extensions

Palomar: Configuration ​

Load Balancers ​

Example ​

Dovecot Proxy ​

Installation ​

Configuration ​

GeoDB / LocalDB ​

Trusted Networks ​

Doveadm ​

Monitoring Backends/Users ​

Dovecot Backend ​

Installation ​

Configuration ​

GeoDB / LocalDB ​

Doveadm ​

Monitoring Users ​

Userdb Fields ​

Cassandra ​

Settings ​

Palomar Core Configuration ​

cluster_backend_name

cluster_backend_test_password

cluster_backend_test_username

cluster_default_group_count

cluster_geodb

cluster_local_site

cluster_localdb

cluster_proxy_check_backends

cluster_user_move_timeout

Cluster Controller ​

Palomar: Configuration

Load Balancers

Example

Dovecot Proxy

Installation

Configuration

GeoDB / LocalDB

Trusted Networks

Doveadm

Monitoring Backends/Users

Dovecot Backend

Installation

Configuration

GeoDB / LocalDB

Doveadm

Monitoring Users

Userdb Fields

Cassandra

Settings

Palomar Core Configuration

`cluster_backend_name`

`cluster_backend_test_password`

`cluster_backend_test_username`

`cluster_default_group_count`

`cluster_geodb`

`cluster_local_site`

`cluster_localdb`

`cluster_proxy_check_backends`

`cluster_user_move_timeout`

Cluster Controller