Dovecot Pro
2.3
main

Appearance

Note: This is pre-release documentation.
Please access https://doc.dovecotpro.com/latest/ for documentation on released versions.

Dovecot Pro: Product Description ​

Dovecot Pro Version 3.x ("Pro") is a highly-available, distributed, standards compliant email platform designed to handle both message storage and remote message access, that scales to tens of millions of active users.

The platform is highly available, can work across multiple physical and/or virtual sites, and provides SLA support options.

Pro’s design allows any platform component to be lost, taken offline for maintenance, or upgraded without affecting the overall service availability for an individual end-user.

Pro supports distributed storage technologies to securely and reliably provide cost-effective message and mailbox metadata storage.

Pro implements the following open email network protocols:

See Dovecot RFC Support for a non-exhaustive list of additional standards that Dovecot implements.

Scope ​

This page describes Pro in detail, and defines how the software must be used in order to be eligible for SLA support.

This document incorporates all content from the Dovecot Pro website (https://doc.dovecotpro.com/) for support descriptions of any features or concepts that do not specifically appear on this page. If there is a conflict between the information in this page, and the documentation on the rest of this site, this document takes precedence.

Dovecot Pro Email Platform (Palomar) ​

Pro's exclusive mail platform architecture, the Dovecot Pro Palomar Architecture ("Palomar"), contains the various components (both Dovecot software and external services) necessary to operate the platform.

The customer MUST operate Palomar in the design outlined in this product definition. Dovecot/OX cannot support operation of Pro software via custom configuration and designs.

Palomar Overview ​

The Palomar architecture comprises Dovecot Pro Proxies ("Proxy") and Dovecot Pro Backends ("Backend"), Palomar Cluster Controller ("Controller"), GeoDB, a highly-available shared storage with access to a shared Dictmap server, OX Abuse Shield (optional), and integration with a customer's identity and authentication databases (passdb/userdb) and external load balancers.

Palomar's stateless design enables a highly-available service, in which any of the individual components can be lost, taken offline for maintenance, or upgraded (up to the high availability capacity limitations determined by a customer) without affecting the overall service availability. This high-availability design maximizes both uptime and operational flexibility, allowing for dynamic scalability and zero-downtime maintenance.

At any given point in time, a user is routed to a single Backend node in the system, from any connection point at the external boundary of the platform, due to performance and efficiency reasons. However, the user does not "live" on this Backend and may be serviced from any other Backend as time passes.

A user's mails should not be accessed simultaneously from multiple Backends in this architecture; the GeoDB and cluster services prevent this from happening. However, even if simultaneous access on multiple Backends did occur, the obox plugin mailbox format was designed not to lose any mails or metadata changes in such situation. This self-healing feature introduces some loss of performance when it occurs, due to the need to continually merge index changes between the multiple Backend accesses, which is why Palomar directs a user to a single Backend, no matter the number of incoming connections that a user spawns.

In Palomar, users are packaged into "groups". Generally, Palomar actions (such as site rebalancing) are performed on these user groups as opposed to individual users.

The Palomar architecture supports multiple sites. "Sites" can be virtual sites hosted within the same data center. Multi-site operations are supported ONLY IF using a multi-site capable storage solution -- currently, only Scality (sproxyd). For all intents and purposes, in Palomar, "cluster" is synonym to "site".

The Palomar architecture enables both horizontal and vertical scaling. The required network topology is:

  • Public (No user data): load balancers for network ingress
  • Private (no mail data on nodes; requires access to GeoDB): Proxy pool(s) with cluster service enabled; Cluster Controller node
  • Private (temporary user mail data): Backend pool(s) with cluster service enabled; Authentication source(s) - accessed by Proxy and Backends, but is customer specific; GeoDB (Cassandra/CQL)
  • Storage Network (permanent user mail data): Mail Storage

In Palomar, Proxy and Backend nodes run the same core Pro software; local configuration defines the role each system takes.

It is HIGHLY recommended to configure the entire platform as a zero-trust environment; however, the Palomar architecture does not currently require this.

End Users ​

End users are defined by provisioning their details into userdb/passdb.

End users do not live on Palomar machines; they are exclusively virtual users. End users connect to Palomar via mail APIs; there is no shell access to the systems and users are not provisioned on Palomar machines.

Palomar Components ​

External Load Balancer ​

Incoming connections to publicly available services exposed on Proxies should be distributed among all available nodes. Each incoming connection is independent at this level, so there is no requirement to route connections from the same user or external source to a single proxy.

Palomar does not require a specific solution at this level, so the choice is up to the customer. Both hardware (e.g. F5) and software (e.g. haproxy) load balancers have been used successfully in customer installations.

The load balancers must either be transparent or they must implement the HAProxy Proxy Protocol v2. This is so Pro can correctly log and track the end-user's connection telemetry as opposed to that of the load balancer.

For software load balancers, this process MUST live on a separate system than Proxies. Palomar does not support running load balancers and Proxies on the same system.

Dovecot Pro Proxy ​

The Proxy's main function is initial authentication/authorization and user identity normalization. Dovecot supports proxying IMAP, POP3, Submission, LMTP, ManageSieve, and doveadm connections to other hosts.

Proxies directly expose publicly available services and handle initial client connections. They make a user database lookup to the customer's identity management or authentication system (e.g., LDAP) to authenticate and authorize the user, and to lookup user-specific routing parameters.

Proxies are usually configured to handle SSL/TLS encryption, including the SSL certificate management. This may also be done by the external load balancer in front of the Proxies, but STARTTLS commands cannot be used in that case.

Authentication and user identity normalization MUST be done at this layer. Palomar assumes the user has been authorized in all layers below the Proxy.

After authentication, sessions are routed to the site where the user is currently assigned.

The cluster service, which runs on the Proxies, makes sure that a user’s data is not concurrently accessed by multiple Backends. This is required to optimize performance and to avoid seeing stale user data.

Proxies are stateless allowing any of them to be removed or become unavailable without end-user impact. A user's session may be terminated on the client, but this is an expected event in mail access protocols and transparent client reconnection will re-enable the session.

Proxies are connected to:

  • the customer's authentication systems (passdb/userdb) to authenticate the platform and (optionally) enforce routing decisions.
  • (optionally) OX Abuse Shield to apply authentication policy.
  • Palomar GeoDB to track user routing inside the Palomar platform.
  • Backends in order to route authenticated users to the system that will handle the mail session.
  • external load-balancer layers in foreign sites to route connecting users that are not being serviced by the local site.

The Proxies do NOT need to connect to mail storage, backends in foreign sites, or any other Proxies directly.

Submission Proxy ​

Pro/Palomar provides an optional submission proxy service that acts as a frontend for any full-featured Mail Transfer Agent (MTA), adding all the necessary functionality for an SMTP Submission service, also known as a Mail Submission Agent (MSA).

The submission proxy can fully handle the required SMTP authentication, which allows a single authentication system to be defined in Dovecot for use with all mail storage and delivery protocols.

Submission proxying for the submission service works by proxying to a Dovecot Pro Backend, which then relays to a MTA (which is not provided as part of Pro). Proxying directly from the proxy to an MTA is not supported.

Authentication Systems (passdb/userdb) ​

Pro supports the following systems for authentication/user information:

Operation, administration, and system performance of the passdb/userdb is the responsibility of the customer; Pro only supports maintaining stable API access to the external systems.

LDAP is recommended, as experience has shown it has the necessary performance to scale to millions of connections.

SQL is supported functionally, but is NOT recommended. At scale, SQL servers cannot handle login loads and will cause noticeable authentication latencies. Dovecot/OX cannot provide support or optimizations for the slow performance.

Customer-specific authentication solutions should be developed using Pro's Lua auth framework. Dovecot/OX Professional Services can be engaged to assist in developing custom solutions. If Lua auth is used to connect to an external HTTP-based service, it must use Pro's Lua http functionality.

Static (local) files are mainly useful to inject local configuration into the Pro session. It should not be used for specific customer data. It is recommended that all variables should be injected into configuration via Lua (if used) instead of using separate static files, if possible.

OX Abuse Shield ​

OX Abuse Shield is an optional component of Palomar that applies authentication policy during user login, via the Proxies.

OX Abuse Shield is recommended to be operated in a high-availability setup; it is not required for login though, although authentication policy will be skipped if the service is not available.

OX Abuse Shield is not provided as part of the base Pro license; separate licensing is needed for the product.

Dovecot Pro Backend ​

The Backend does the primary work of reading and writing mails to storage and handling the bulk of the mail protocol interaction with the client.

The Backends are organized as a pool of independent nodes. A user is not permanently assigned to a specific Backend. However, due to the performance and load reasons, the platform is designed to allow users to move between Backends.

Backends are connected to:

The Backends do NOT need to connect to Backends on foreign sites, or any Proxies.

Backend states ​

Each backend can be in one of the pre-defined states at each point in time:

  • online: Backend is functional and active in the site. Users are routed to the backend.
  • offline: Backend is active in the site but has been temporarily deactivated by the automation based on the outcome of Health Check. Offline backends can still serve users but proxies don't route actual users to them. Instead, Dovecot proxies probe them with test accounts to check if they are healthy again and are automatically set to online again if the test passes.
  • standby: Backend is manually taken out of service and won't automatically change status to online. This mode is mostly intended for admins to perform maintenance or fleet management operations.

Dovecot Pro Palomar Cluster Controller ​

The Controller is a non-highly available component that performs administrative tasks and automated tasks on cluster state. These tasks include load balancing, health monitoring, and statistics gathering.

One Controller node is needed per site. The Controller does not need to be highly available, but platform-level actions will not occur unless the Controller is active.

The Controller provides a graphical user interface to the admin API. It also exposes an API to other Palomar nodes for maintenance tasks and user administration.

The Controller is connected to the GeoDB to track user routing inside the Palomar platform.

The Controller is installed and instantiated via either Kubernetes and a helm chart or via docker-compose.

WARNING

Cluster Controller is distributed via containers. These containers MUST be used as-is. The contents of the containers MUST NOT be altered. All interaction with the services in the container MUST be performed via documented APIs only.

Cluster Controller UI Experimental ​

Cluster Controller UI is an optional component providing a web-based GUI for the Cluster Controller API.

Cluster Controller UI is currently marked as experimental. The service is provided as assistance to the operator, but as it is not required for Palomar production usage, no SLA support is currently provided.

GeoDB ​

The GeoDB contains the metadata used by Palomar to route users and groups, maintain the platform health and performance, and allow autodiscovery and bootstrapping of nodes in the platform.

Proxies, Backends, and Controllers access GeoDB to perform these actions.

GeoDB requires a server that supports CQL version 4 or higher. GeoDB otherwise shares the same Cassandra requirements as Dictmap.

Mail Storage (Supported API/Systems) ​

Storage is generally the most expensive and technically difficult component of an email system, due to the volume and storage capacities needed. As such, it is critically important that the underlying technology is proven to be stable and secure through extensive development, testing, and QA. Therefore, Palomar/Pro only supports a small, defined list of storage technologies.

Palomar requires a highly available, distributed storage solution, independent of the Backends, as users must be able to move between nodes in the platform.

Data replication MUST be handled by the storage system; Pro does not do data replication itself.

Storage Systems ​

Premium Support ​

Premium support indicates that these storage systems are actively tested and developed against by Dovecot and OX.

Each Pro release is certified to work on at least one version/installation of the system which are identified in Release Notes for that specific Pro release. This means that release testing has been directly performed using at least one version of the storage system for any given Dovecot Pro release.

Storage options within this level are guaranteed to have compatibility with the following:

  • Authentication (at least 1 solution).
  • Specific API behavior including necessary request parameters needed to interact with the system.

Additional features of these storage systems may also be supported (i.e. AWS IAM). These additional features are noted below.

Execution, operation and configuration of the storage system is NOT directly supported by OX. However, these solutions are classified as Premium support options because:

  • Similar installations SHOULD behave similarly - the storage experience has been - by design - commodified by the vendor.
  • Vendor support is available.
  • Cooperative engineering effort with the vendor has been conducted to maximize performance and stability.
Scality sproxyd ​

See Scality (sproxyd).

  • Multi-site (geo-redundant) capable solution.
  • Sizing, setup, and server support is handled by Scality.
  • Release Notes: specific version of RING tested against will be provided.
AWS S3 ​

See AWS S3.

  • Single AWS region only (multi-availability zones within a single region is OK).
  • Platform support is handled by AWS.
  • Customer is responsible for working with AWS to ensure sufficient operational limits (e.g. IOps and ingress/egress limits).
  • IAM is supported; v2 and v4 signatures.
  • Documentation: The full list of tested S3 API calls is provided.
  • Release Notes: The month & year when the AWS S3 service was tested and the active documentation link at that time.
Basic Support ​

Basic support indicates that Dovecot Pro maintains access via a specified API that can be used to interact with the storage system, but it does not certify actual storage vendors and does not test directly against these systems prior to any release version.

Dovecot/OX will not support any additional features other than basic API compatibility defined in the Product Definition.

Dovecot/OX cannot guarantee or troubleshoot storage performance or behavior.

Execution, operation and configuration of the storage system is NOT supported by Dovecot/OX. Vendor support is storage dependent.

The customer assumes the risk if a storage vendor's API changes in the future.

OX/Dovecot Assistance

If something breaks in an existing production system, OX will use best effort to work with the customer and fix the issue. However, this is not an SLA event and the customer should understand that quick and efficient fixes may be difficult as the Dovecot team does not have local and direct access to the storage systems for testing and debugging. In contract, Premium storage systems are locally available for the team to test and debug.

"AWS S3 compatible" ​

See S3 Compatible Storage.

  • Uses AWS's S3 implementation as the reference behavior.
  • "AWS S3 compatible API" is defined as a storage system that complies with the behavior identified and tested as part of Dovecot Pro "AWS S3" Premium Support for that release.
Azure Blob Storage ​
  • Microsoft Azure blob storage API.
  • Documentation: The full list of Azure Blob Storage API calls is provided.
  • Authentication: User SAS (with Entra IDM).
  • Release Notes: The month & year when the Azure Blob storage service was last tested and the active documentation link at that time.
NFS ​

WARNING

NFS SLA Support is not provided as part of the base Dovecot Pro License.

A separate SLA Support agreement needs to be negotiated for customers that need NFS support.

  • Site replication is NFS vendor-specific.
  • NFSv3 = known to work; NFSv4 = no production knowledge.
  • NFS system is customer provided and not supported by Dovecot/OX.
    • Storage/filesystem performance is not guaranteed.
    • No sizing information on NFS can be provided. If a customer wants sizing information, they should work with a storage vendor that has expertise.

Mailbox Format ​

Pro's obox mailbox format is the ONLY mailbox format supported for production use.

No other mailbox is supported for production use. Software support for non-obox mailbox formats is limited to migration, backups, and/or archiving use-cases.

Customers using sdbox/mdbox on Pro 2.3 MUST migrate users to obox on Pro 3. There is no support for direct physical mailbox conversion of sdbox/mdbox to obox.

fs-dictmap ​

For all object storage installations, fs-dictmap is REQUIRED. fs-dictmap is not required for NFS.

Dovecot stores and retrieves fs-dictmap information using CQL (Cassandra Query Language). Dovecot/OX recommends and tests Pro releases with CQL protocol version 4.

The fs-dictmap database must be configured to be multi-node and highly available.

Apache Cassandra ​

Dovecot/OX tests at least one version of Apache Cassandra for every Pro release.

Dovecot/OX does not support customer configuration or operation of Cassandra. Dovecot/OX may provide Cassandra recommendations, but these are not binding. Support for Cassandra can be pursued through 3rd parties.

As part of the Release Notes, the specific version of Cassandra that was tested and confirmed to be working will be provided.

Other Options ​

Customer can use other solutions that claim compatibility with CQL, but the customer is responsible for determining this compatibility.

Note

Compatibility means full support for the feature set provided by the Apache Cassandra version(s) identified in the Release Notes.

For example, AWS Keyspaces does NOT implement certain features provided by Apache Cassandra so it cannot be used with Dovecot Pro.

Dovecot/OX cannot provide support for customer configuration or operation of these databases.

Mail Quota ​

Pro only supports user quota tracking via the "count" driver of the quota plugin. The count quota backend tracks the quota internally within Dovecot's index files.

Mail Data Access ​

Dovecot/OX can only support direct mail data access if the customer uses Pro provided APIs (i.e. doveadm commands; scripts shipped with Pro packages).

Dovecot/OX CANNOT support direct modification of mail storage, neither for NFS nor object storage solutions.

doveadm-fs(1) commands can be used for debugging and fixing as instructed by OX/Dovecot, but not as a method to access mails in production.

Mail Delivery ​

Palomar only supports mail delivery via LMTP.

Dovecot does not provide or directly support any AV/AS solution. Such a solution must deliver mails via LMTP when processing is completed.

Mail (IMAP) Searching ​

Palomar only supports searching messages via the IMAP protocol if the fts-dovecot plugin is used.

Pro Features ​

Dovecot Pro Object Storage Mailbox Format (obox) ​

"obox" is Pro's exclusively supported mailbox format for Palomar.

For obox, a user’s mail data is retrieved to a Backend, index and metadata ("metacache") changes occur on that Backend, and this changed data is uploaded back to object storage as needed. In case of a Backend failure, another Backend can continue servicing the user mailbox by downloading the metacache locally onto its server.

Although the system is designed to allow users to move between multiple backends, and there is code to support accidental access of the mailbox from two servers at once, this behavior comes with a performance penalty. Thus, Palomar requires that a user must be accessed from a single server and the platform is designed to enable and support this behavior.

obox is optimized for cloud technologies by minimizing I/O with the storage. obox tracks which index files have been altered or are needed locally and uploads / downloads them to object storage only as necessary. This usage pattern most efficiently leverages the object storage paradigm, as opposed to a more traditional black storage strategy.

While using object storage, a user’s mail indexes are fetched from storage and cached locally. The mail indexes are periodically updated to object storage while the session is active. Once the session either expires or the user logs off, any updated indexes are uploaded to storage. By working with local, cached indexes, Pro provides fast access to the user’s mailboxes while leveraging the advantages that object storage provides for long-term storage needs.

obox consists of three major components. The first component is a block-storage native mailbox format. Each message is stored in its own “file” (a discrete object). Mailbox indexes, and other Dovecot user data files, are bundled into separate discrete objects. The second component is a collection of drivers that implement support for various storages, such as S3 and sproxyd. There is additionally a "fscache" driver that implements a local filesystem cache for mail objects. The third component is metadata storage for index files and other metadata, such as Sieve scripts. It synchronizes these files between a local cache and the object storage.

Dovecot Pro Full Text Search (FTS) ​

When messages are not pre-indexed, IMAP searches fall back on slow sequential searches through all message headers or text. This strategy is slow on block storage and becomes performance prohibitive in a distributed object storage architecture.

At the same time, mobile clients are the fastest growing segment for mailbox access. These clients have additional bandwidth limitations like monetary cost and network latency that emphasize the need for an efficient, feature rich server-based search solution.

To assist in addressing these concerns, a Pro exclusive indexing and search architecture has been developed. This design is built-in to the Pro software, provides better performance and scaling for large mail volumes, and uses the same storage pool as mail objects.

Full text search has the following features:

  • Indexes can be natively stored in object storage (built-in interoperability with the obox plugin)
  • Smaller index sizes, compared to legacy search drivers, to improve search performance
  • Avoid duplication of data by using word stemming and normalization and skipping bad characters
  • Native integration with Pro software; no third-party software needed
  • No extra compute nodes needed for search string word stemming and normalization
  • No separate storage or search nodes needed; scaling and administration can be done at a single level (Backends) instead of across multiple, disparate server farms

Pro's standard IMAP SEARCH TEXT/BODY (RFC 3501) parameters use the FTS indexes. Searches of message headers already benefit from Pro's fast message index cache implementation but can also optionally be done from FTS indexes.

Tika Support Experimental ​

Apache Tika enables Dovecot to index text in various mail attachments, to improve the user's search experience.

Additionally, Tika supports Optical Character Recognition (OCR) which enables certain types of image data (e.g. scanned PDFs, JPEG) to be indexed as well.

Tika support in Dovecot Pro is currently marked as experimental. The feature is tested as part of the release process, but no SLA support is provided.

Shared Mailboxes ​

Dovecot Pro supports multiple users connecting to the same underlying mailbox. This is done through configuration of the software.

All users with access to a defined mailbox connect to a distinct mailbox on a single Backend, so there may be performance penalties if an excessive number of users concurrently access the same mailbox.

Pro does not provide support for administration of user access rights. This is the responsibility of the Customer’s identity management system.

OX App Suite Integrations ​

For customers that hold both a Pro license and an OX App Suite license, the following cross-stack features are additionally supported as part of Pro.

Virtual Attachment View ​

This is a proprietary Dovecot Pro plugin (virtual-attachments plugin) to list all attachments in a mailbox; OX App Suite UI will display these attachments in a Drive-like format.

Dumpster ​

Uses Pro's lazy-expunge plugin; OX App Suite UI can display this special mailbox to provide customer self-help assistance in retrieving accidentally deleted emails.

Unified Quota ​

A single quota can be used for both the mail store (Pro) and Drive (OX App Suite).

iMIP Notifications ​

Pro push notifications (via the push-notification-chronos plugin) to OX App Suite middleware when iMIP (iCalendar Message-Based Interoperability, RFC 6047) attachments are detected in incoming messages.

ImapTest ​

The ImapTest tool is provided to Pro customers as a courtesy. This package/software has absolutely NO support, warranty, or SLA.

Pro Distribution Channels ​

Pro, with the exception of the Cluster Controller component, is currently provided as packages built for a specified list of operating systems.

Packages: Operating System Support ​

Pro is only supported on Linux distributions on x86 hardware.

The list of distributions supported is listed in the Release Notes for a given Pro release.

The product rules for when Operating System support is added and dropped can be found at the OS Distribution Support page.

Cluster Controller ​

The Cluster Controller is distributed either via helm charts (for use with Kubernetes) or via execution through docker-compose.

Kubernetes ​

Other than Cluster Controller, Pro is not currently supported as a Kubernetes deployment.

Separately Licensed Plugins ​

The following Pro plugins are available for Pro customers, but they require separate/additional licensing:

intercept ​

The Lawful Interception Framework implements lawful interception (LI) facilities, in which targeted users can be monitored by their interaction with the various Dovecot services.

Pro supports several lawful interception vendor APIs, including Utimaco and PineApp.

fts-jp-kuromoji ​

An additional tokenizer for the "fts-dovecot" plugin which utilizes the Atilika Kuromoji tokenizer library to tokenize Japanese text.

See Dovecot FTS Kuromoji Tokenizer.

proxyauth ​

Implements the PROXYAUTH extension for IMAP.

See proxyauth plugin.

vault ​

The vault plugin stores incoming mail first to a configurable, read-only mailbox location (e.g., "ARCHIVE") and, if that succeeds, to the user’s INBOX. It also adds the IMAP "\Seen" flag to the message.

sieve-zimbra-compat ​

This package extends Pro's Sieve support for the vendor-specific language extensions used by the Zimbra implementation of Sieve.

See sieve-zimbra-compat plugin.

🖨️ View Print-Optimized Version

Revision: 618b521

Last updated:

Copyright © Open-Xchange GmbH