Palomar: Configuration

This page describes the Dovecot config necessary to implement the Palomar Architecture.

WARNING

If you are upgrading from OX Dovecot Pro 2.3.x, see Transition from OX Dovecot Pro 2.3.x Architecture.

Load Balancers

Palomar requires a layer of load balancers to balance traffic ingress to the Dovecot Proxy.

Requirements

The load balancer MUST either be transparent (keeps the original client IP visible) or it MUST support HAProxy PROXY V2 protocol.

Round-robin distribution of incoming connections to the pool of available site proxies is a common setup. More advanced traffic-management can be done as well, although specific configuration details for the load balancer is site-dependent and is out-of scope for this documentation.

Example

load-balancer: HAproxy configuration example

# haproxy.conf
#
# Sample Configuration for a Palomar load balancing setup using software
# (HAproxy) load balancing
#
# Sample assumptions:
#   - Incoming traffic will be balanced to 2 proxies, located at 192.168.1.10
#     and 192.168.1.11

defaults
    mode                    tcp
    log                     global
    option                  dontlognull
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 128000

listen lmtp
   bind 0.0.0.0:24
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:24 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:24 check send-proxy-v2

listen pop3
   bind 0.0.0.0:110
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:110 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:110 check send-proxy-v2

listen imap4
   bind 0.0.0.0:143
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:143 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:143 check send-proxy-v2

listen sieve
   bind 0.0.0.0:4190
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:4190 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:4190 check send-proxy-v2

listen submission
   bind 0.0.0.0:587 accept-proxy
   mode tcp
   balance roundrobin
   server hac-dc1-proxy2 192.168.1.11:587 check send-proxy-v2
   server hac-dc1-proxy1 192.168.1.10:587 check send-proxy-v2

Dovecot Proxy

Dovecot Proxies authenticate the user, convert to internal user ID (if needed), and redirect to the Dovecot Backend.

Installation

A Dovecot Proxy requires the dovecot-pro-cluster package.

Configuration

Documentation on cluster settings can be found below.

# Name of the proxy's local site
cluster_local_site = DC1

# Include all the required settings for running in cluster proxy mode
@cluster_defaults = proxy
@metric_defaults = proxy

GeoDB / LocalDB

See Palomar GeoDB.

WARNING

The GeoDB tables MUST be manually created. See Cassandra.

cassandra_hosts = cassandra-host-3 cassandra-host-2 cassandra-host-1
dict_server {
  dict cluster-geodb {
    cassandra_keyspace = cluster
  }
}

HAProxy

If HAProxy (or compatible) load balancer is used in front of Proxies:

haproxy_trusted_networks needs to trust the HAProxy servers' IP addresses.
inet_listener_haproxy = yes needs to be set for all listeners that accept HAProxy connections, possibly in a different port. For example:
```
service imap-login {
  inet_listener imap-haproxy {
    haproxy = yes
    port = 1143
  }
}
```

Trusted Networks

Dovecot Proxy IPs/networks that must be trusted (via login_trusted_networks):

Dovecot Proxies in other sites. If user connection is going to a wrong site, the proxy will connect to the correct site's proxy (via the load balancer).
cluster-backend-test service connects to the local proxy. This is based on the first IP address in service imap-login { inet_listener }, which defaults to the listen. Typically this is 127.0.0.1 or ::1.

login_trusted_networks = 127.0.0.1 ::1 192.168.1.0/24

Doveadm

The Doveadm TCP service must be configured for all Proxies and Backends:

doveadm_password must be set to the same value in both the Proxy and the Backend
doveadm_port in the Proxy must be set to the same port as the where the backend listens
doveadm_ssl can be used to enable SSL/TLS for doveadm proxying. In this case also set inet_listener_ssl = yes for the doveadm listener.
For multi-site Palomar operation the doveadm listener must be enabled also on the Proxies.

Example:

doveadm_port = 2300
#doveadm_ssl = yes # If enabled, set also ssl=yes in the inet_listener
doveadm_password = # shared password between all Proxies and Backends

service doveadm {
  inet_listener tcp {
    port = 2300 # same as doveadm_port
    #ssl = yes
  }
}

Monitoring Backends/Users

Palomar requires "test" accounts to be configured so that backends can be monitored.

The test username is recommended to use %{backend_host}, which expands to the tested backend. This way if there is a problem with the test user, it affects only a single backend.

Both the proxy and the backend need a passdb, which can authenticate the test users. They could be in the same passdb as the real users, or they can have a separate passdb. In the latter case, it should be placed as the first passdb to catch the test users.

cluster_backend_test_username = probe-%{backend_host}
cluster_backend_test_password = # shared password

# Passdb configuration for monitoring users first:
passdb test-accounts {
  args = /etc/dovecot/cluster-test-accounts.passwd
  fields {
    proxy = y
    password = # shared password
  }
  driver = passwd-file
}

# Passdb for the real users:
#passdb ldap {
#  ...
#}

See:

Dovecot Backend

Dovecot Backends perform all work related to protocol commands and interfaces with the storage.

Installation

A Dovecot Backend requires the dovecot-pro-cluster package.

Configuration

Documentation on cluster settings can be found below.

# Name of the backend (same as in geodb)
cluster_backend_name = hac-dc1-be3
# Name of the backend's local site
cluster_local_site = DC1

# Include all the required settings for running in cluster backend mode
@cluster_defaults = backend
@metric_defaults = backend

GeoDB / LocalDB

See Palomar GeoDB.

WARNING

The GeoDB tables MUST be manually created. See Cassandra.

cassandra_hosts = cassandra-host-3 cassandra-host-2 cassandra-host-1
dict_server {
  dict cluster-geodb {
    cassandra_keyspace = cluster
  }
}

Doveadm

The Doveadm TCP service must be configured for all Proxies and Backends:

doveadm_password must be set to the same value in both the Proxy and the Backend
doveadm_port in the Proxy must be set to the same port as the where the backend listens
doveadm_ssl can be used to enable SSL/TLS for doveadm proxying. In this case also set inet_listener_ssl = yes for the doveadm listener.
For multi-site Palomar operation the doveadm listener must be enabled also on the Proxies.

Example:

doveadm_port = 2300
#doveadm_ssl = yes # If enabled, set also ssl=yes in the inet_listener
doveadm_password = # shared password between all Proxies and Backends

service doveadm {
  inet_listener tcp {
    port = 2300 # same as doveadm_port
    #ssl = yes
  }
}

Monitoring Users

Each backend must receive test user logins from proxies. The users can be in the same passdb as the real users, or they can have a separate passdb. In the latter case, it should be placed as the first passdb to catch the test users.

# Passdb configuration for monitoring users first:
passdb {
  args = /etc/dovecot/cluster-test-accounts.passwd
  driver = passwd-file
}

# Passdb for the real users:
#passdb ldap {
#  ...
#}

# Backends use this in "doveadm cluster group access" command:
cluster_backend_test_username = probe-%{backend_host}

Depending on the configuration, the test users may also need explicit userdb configuration so they can successfully log in.

See:

Userdb Fields

Backends require user_cluster_group userdb extra field, which specifies the user group of the logging in user. For lmtp and doveadm protocols this happens automatically, but it must be configured explicitly for other protocols.

This is done by returning user_cluster_group=%{passdb:forward_cl_user_group} extra field from userdb. Alternatively, it can be returned by passdb by adding userdb_ prefix, i.e. returning userdb_user_cluster_group extra field. The forward_cl_user_group is automatically added by the cluster service running on a Proxy.

Example using a static userdb (must be adapted to the actually used userdb or passdb):

userdb static {
  fields {
    user_cluster_group = %{passdb:forward_cl_user_group}
  }
}

Cassandra

You need to setup a Cassandra namespace for Palomar.

Cassandra Setup

cql

create table if not exists tags (
  id uuid,
  tag text,
  primary key ((id))
);

create table if not exists sites (
  id uuid,
  name text,
  tag uuid,
  load_balancer text,
  status text,
  primary key ((id))
);

create table if not exists site_reachability (
  src_site_id uuid,
  dest_site_id uuid,
  reachable int,
  primary key ((src_site_id), dest_site_id)
);

create table if not exists backends (
  id uuid,
  site_id uuid,
  load_factor int,
  host text,
  status text,
  status_reason text,
  last_moved_from text,
  last_moved_to text,
  primary key ((site_id), id)
);

create table if not exists backend_stats (
  site_id uuid,
  id uuid,
  key text,
  value double,
  type text,
  primary key ((site_id), id, key)
);

create table if not exists proxy_dest_stats (
  proxy_site_id uuid,
  proxy_host text,
  dest_host text,
  key text,
  value double,
  type text,
  primary key ((proxy_site_id), dest_host, proxy_host, key)
);

create table if not exists user_groups (
  name text,
  site_id uuid,
  backend_id uuid,
  alt_backend_id uuid,
  sticky_users int,
  moving text,
  refresh_after timestamp,
  sticky_backend int,
  primary key ((site_id), name),
);

create table if not exists group_sites (
  group_name text,
  site_id uuid,
  status text,
  failover_site_id uuid,
  last_update timestamp,
  primary key ((group_name), site_id),
);

create table if not exists users (
  username text,
  preferred_site_id uuid,
  group_name text,
  failover_site_group_name text,
  metacache_last_host text,
  moving_preferred_site_id uuid,
  moving_group_name text,
  moving_failover_site_group_name text,
  last_update timestamp,
  primary key ((username))
);


create table if not exists cluster_settings (
  section text,
  site_id uuid,
  backend_id uuid,
  setting_name text,
  setting_value text,
  primary key ((section), site_id, backend_id, setting_name),
);

Settings

Palomar Core Configuration

`@cluster_defaults`

Default	[None]
Value	Groups Includes
Allowed Values	`proxybackend`

Group that expands to various settings that are required for cluster to run in a proxy or backend.

`cluster_backend_name`

Default	[None]
Value	string

Host name of the backend. This must be the same name as used by doveadm cluster-backend commands.

Cluster: Applies only to Backend.

`cluster_backend_test_password`

Default	[None]
Value	string

Password used for logging into backends to see if it's up or down.

Cluster: Applies only to Proxy.

`cluster_backend_test_username`

Default	[None]
Value	string

This setting is used for two purposes:

Username used by proxy for logging into backends to see if it's up or down. %{backend_host} variable expands to the hostname of the backend.
Username used by backend for running doveadm cluster group access command. This command just needs any existing user to work - it doesn't matter that it's not actually in the accessed group.

Cluster: Applies to both Proxy and Backend.

`cluster_default_group_count`

Default	[None]
Value	unsigned integer

Number of user groups that may be automatically created. This is used for creating a group for a user that doesn't yet have one. The group will be named default-N where N is between 1 and cluster_default_group_count.

Cluster: Applies only to Proxy.

`cluster_geodb`

Default	[None]
Value	string

Dictionary URI used for the globally shared GeoDB. This typically points to Cassandra.

Cluster: Applies to both Proxy and Backend.

`cluster_local_site`

Default	[None]
Value	string

Name of the local site. This must be the same name as used by doveadm cluster-site commands.

Cluster: Applies to both Proxy and Backend.

`cluster_localdb`

Default	[None]
Value	string

Dictionary URI used for the server-specific local database. This typically points to SQLite under /dev/shm.

Cluster: Applies to both Proxy and Backend.

`cluster_proxy_check_backends`

Default	`yes`
Value	string

If enabled, this proxy runs checks to see whether backends are up or down.

If disabled, this proxy never sets any backends offline.

Cluster: Applies only to Proxy.

`cluster_user_move_timeout`

Default	`31secs`
Value	time

If user moving hasn't finished by this timeout, just assume it finished and continue to the next user.

Cluster: Applies only to Proxy.

Cluster Controller

See Cluster Controller Settings.

Databases

Mechanisms

Users

Extensions

Palomar: Configuration ​

Load Balancers ​

Example ​

Dovecot Proxy ​

Installation ​

Configuration ​

GeoDB / LocalDB ​

HAProxy ​

Trusted Networks ​

Doveadm ​

Monitoring Backends/Users ​

Dovecot Backend ​

Installation ​

Configuration ​

GeoDB / LocalDB ​

Doveadm ​

Monitoring Users ​

Userdb Fields ​

Cassandra ​

Settings ​

Palomar Core Configuration ​

@cluster_defaults

cluster_backend_name

cluster_backend_test_password

cluster_backend_test_username

cluster_default_group_count

cluster_geodb

cluster_local_site

cluster_localdb

cluster_proxy_check_backends

cluster_user_move_timeout

Cluster Controller ​

Palomar: Configuration

Load Balancers

Example

Dovecot Proxy

Installation

Configuration

GeoDB / LocalDB

HAProxy

Trusted Networks

Doveadm

Monitoring Backends/Users

Dovecot Backend

Installation

Configuration

GeoDB / LocalDB

Doveadm

Monitoring Users

Userdb Fields

Cassandra

Settings

Palomar Core Configuration

`@cluster_defaults`

`cluster_backend_name`

`cluster_backend_test_password`

`cluster_backend_test_username`

`cluster_default_group_count`

`cluster_geodb`

`cluster_local_site`

`cluster_localdb`

`cluster_proxy_check_backends`

`cluster_user_move_timeout`

Cluster Controller