Upgrading from Dovecot Pro 3.2.x to Dovecot Pro 3.3.x

Dovecot Pro

• Boolean settings no longer accept y or 1 as a value in the configuration. They were never documented, and only their true side ever worked - the matching n and 0 have always been errors. Use yes and no instead. Values that don't come from the configuration still accept them, so e.g. a userdb lookup can keep returning mail_debug=y.

• shutdown_clients was replaced by service_shutdown_clients_timeout, which also allows everything in between its two endpoints: shutdown_clients=yes becomes service_shutdown_clients_timeout=0 and shutdown_clients=no becomes service_shutdown_clients_timeout=infinite. The conversion is done automatically as long as dovecot_config_version is older than this release. A reload keeps the existing client sessions running for that long, e.g. to take new SSL certificates into use without disconnecting anyone.

• When the timeout expires - or right away, with the default 0 - the processes disconnect all their remaining clients, also the ones that are in the middle of a command, and the login processes abort the logins that are still in progress. Previously shutdown_clients=yes kept an IMAP session running until the client had been idle for 10 seconds, or for 30 seconds if it never was, and the other services waited for up to 30 seconds for the clients to disconnect by themselves.

• Only the processes of service_type = client and service_type = login services keep serving their clients across a reload. The internal services are always replaced by it, so their old processes are stopped regardless of the timeout - otherwise they would pile up with every reload.

• The services whose processes serve externally visible client connections have service_type client now: imap, pop3, submission, imap-hibernate, imap-urlauth, imap-urlauth-worker, managesieve and doveadm. lmtp is left out on purpose, because an MTA can keep an idling LMTP connection open for a long time, which would then keep the old generation's processes running. The previous default isn't preserved for older dovecot_config_version values, because the type only tells the master process how the service's processes are to be treated.

• A custom service { .. } block that relied on shutdown_clients=no needs type = client added to it. Without it the reload stops the service's old processes in spite of the converted timeout.

The Pro services whose processes serve client connections have service_type client now as well: intercept-api, intercept-proxy and utimaco-x1. utimaco-x3-kafka is left out on purpose, because only one of its processes may consume from the Kafka topic at a time, so the reload has to stop the old process.

Changed Setting Defaults

These changes don't take effect until dovecot_config_version is changed to 3.3.0.

Setting	Old Default	New Default	Notes
mail_temp_dir	/dev/shm/dovecot	/tmp	The temporary files exist to keep large mails out of memory, which a tmpfs defeats. /dev/shm is also commonly far too small (e.g. 64 MB in containers). Installations that have a large enough tmpfs and want to avoid the disk IO can still set this explicitly.

Added Features

Metacache Pull Skipped Metric

A metacache pull that doesn't run at all, because another process is already pulling the same user, is no longer counted as a failure. The metacache_pull_finished event gained a skip_reason_code field for this, the default metacache_pull_finished metric now excludes such pulls, and a new default metacache_pull_skipped metric counts them.

Feature	Notes
metacache_pull_skipped metric	Metric was added

New Cassandra Settings

New settings were added to configure the Cassandra driver. The defaults are the same as the driver's previous behavior, except that the client now sends cassandra_application_name and cassandra_application_version to the server.

If the Cassandra cluster has multiple datacenters, it's recommended to set cassandra_local_datacenter. Otherwise the local datacenter is the datacenter of whichever cassandra_hosts host answers first.

Setting	Notes
cassandra_local_datacenter	Setting was added.
cassandra_connections_per_host	Setting was added.
cassandra_reconnect_policy	Setting was added.
cassandra_reconnect_base_delay	Setting was added.
cassandra_reconnect_max_delay	Setting was added.
cassandra_tcp_keepalive	Setting was added.
cassandra_token_aware_routing	Setting was added.
cassandra_token_aware_shuffle_replicas	Setting was added.
cassandra_latency_aware_exclusion_threshold	Setting was added.
cassandra_latency_aware_scale	Setting was added.
cassandra_latency_aware_retry_period	Setting was added.
cassandra_latency_aware_update_rate	Setting was added.
cassandra_latency_aware_min_measured	Setting was added.
cassandra_request_queue_size	Setting was added.
cassandra_application_name	Setting was added.
cassandra_application_version	Setting was added.
cassandra_client_id	Setting was added.
cassandra_source_ip	Setting was added.

FTS Direct Autoindexing

fts_autoindex was changed from a boolean to no, yes or direct. The existing yes and no values work as before. The new direct value indexes the newly added mails directly in the same process, instead of asynchronously via the indexer service. This is mainly useful when migrating mails with doveadm sync or doveadm backup.

Setting	Notes
fts_autoindex	Setting value direct was added.

Unauthenticated Client Limit

A new login_unauthenticated_client_limit setting limits the number of unauthenticated client connections in each login process, separately from service_client_limit. When the limit is reached, the oldest unauthenticated connection is disconnected. This is mainly useful with high performance login mode. The default is unlimited, so the behavior doesn't change unless the setting is configured.

Setting	Notes
login_unauthenticated_client_limit	Setting was added.

Language Detection Uses Only the Configured Languages

FTS language detection now uses only the textcat fingerprints of the languages listed in language, instead of all the languages in the textcat configuration file. This makes language detection much faster: e.g. with two configured languages the detection is about 10 times faster than with the 180 fingerprints in the default libexttextcat configuration. Creating a minimal textcat configuration file with textcat_config_path is no longer necessary.

Text in a language that isn't configured is now detected as the closest configured language. Previously such text was usually detected as an unknown language, which used the default language (see language_default). Set textcat_filter_languages = no to keep the old behavior. The setting defaults to no as long as dovecot_config_version is older than this release.

Setting	Notes
textcat_filter_languages	Setting was added.

Reload Without Disconnecting Clients

Feature	Notes
service_shutdown_clients_timeout	Keep the existing sessions running for a while after doveadm reload, e.g. to take new SSL certificates into use without disconnecting anyone. See reloading the configuration.
doveadm reload --kick-timeout	Override service_shutdown_clients_timeout for a single reload.
doveadm process status and doveadm service status generation and kill_time columns	Tell which configuration generation a process belongs to and when the master process is going to signal it next. doveadm service status -a lists also the older generations.

Changed Features

Auth SQL Bind Parameters

• Auth SQL queries (passdb_sql_query, userdb_sql_query and userdb_sql_iterate_query) now pass %{variable} values to the database as bind parameters instead of expanding them into the query text. Queries that relied on the old text expansion fail at lookup time with an error:
  • A %{variable} must produce a whole value on its own. Wrapping a single variable in single quotes still works ('%{user}'), but combining it with other text does not: '%{user | username}@example.com' and '%{user}%' must be rewritten with the var-expand concat filter, e.g. %{user | username | concat('@example.com')} and %{user | concat('%')}.
  • A backslash is no longer allowed inside a single-quoted string. Escape a quote as '', not \'.
  • SQL comments (--, /*, // and #) are no longer allowed outside a quoted string, and neither is PostgreSQL dollar quoting ($$...$$) or a positional parameter ($1, $2, ...).
  • A literal ? outside a quoted string can no longer appear in the query. This affects PostgreSQL's jsonb ?, ?| and ?& operators.
• Logged queries no longer show the %{password} value unless auth_debug_passwords = yes is set.

See Variables in Queries for details.

• A local_name filter nested inside another local_name filter must now be a hostname that matches the outer filter's name, the same way as a nested local or remote network must be inside the outer network. For example local_name *.example.com { local_name imap.example.com { .. } } is allowed, but local_name *.example.com { local_name imap.example.org { .. } } or an inner name with a wildcard now fails the config parsing. Previously such blocks were accepted, but their settings applied only to connections matching both names, which was usually nothing. See Connection Filters.

• Some settings can now be used only where they have an effect. Using one inside a filter that doesn't use it is a configuration error instead of being silently ignored. For example quota_mail_size applies to the user rather than to a quota root, so it can't be used inside quota { .. }.

• The global default_* settings and protocols can be used only globally. Setting a default_* inside service { .. } used to change that service's derived settings, because their defaults expand $SET:default_internal_user and similar within the filter. Use the per-service setting instead, e.g. service_user rather than service foo { default_internal_user = .. }. Referring to the global settings with $SET: inside a filter keeps working.

• The listen setting is now called inet_listener_listen. The old name still works as an alias, so configurations don't need changes, but doveconf writes the new name outside of inet_listener { .. } blocks.

• source_location in log_debug and log_core_filter is now matched against each log line's own source location. Previously info, warning and error lines of events with a raised minimum log level (e.g. auth lines with auth_verbose = no) were matched against an internal source location, and the result matched for one log line was reused for the event's later log lines. Because the filter results can't be cached anymore when source_location is used, every debug log call becomes several times slower. Use source_location only temporarily while debugging. See Performance With source_location.

• The `normalizer-icu` filter no longer uses libicu for the default language_filter_normalizer_icu_id or its variants without NFC and/or [\x20] Remove. Other IDs now require the lang_filter_normalizer_icu module (liblang_filter_normalizer_icu.so in the Dovecot module directory), which links with libicu. Dovecot processes no longer load libicu and libstdc++ unless such an ID is used, which saves about 220 kB of memory per process. Packages may ship the module separately. With mail_chroot, such an ID requires loading the module via mail_plugins: mail_plugins { lang_filter_normalizer_icu = yes }

Cassandra Uncertain Writes

More Cassandra write errors are now handled as uncertain writes, because the write may have been applied: Cassandra server internal and overloaded errors, the driver running out of hosts to try, replies that couldn't be decoded and any unrecognized server error. Previously these were handled as definite failures, and dictmap deleted the storage object even though the write may have been applied, which could lose mails. Now these are handled as described in Uncertain Writes, so more success is uncertain and related log messages may be seen.

Removed Features

Feature	Notes
shutdown_clients setting	Replaced by service_shutdown_clients_timeout.