Obox Storage Support: AWS S3

This page describes support for the S3 service directly provided by Amazon Web Services (AWS).

Note

For details on "S3 Compatible" storage support from non-AWS providers, see S3 Compatible Storage.

Technical Requirements

Note

A Dovecot Pro/Palomar installation is only supported in a single AWS region.

This is because AWS S3 only offers asynchronous data replication across regions, which is not supported by obox.

Palomar can be setup using multiple Availability Zones (AZs) within a region. Each AZ is considered a "site" for purposes of Palomar.

Note

Availability Zones must be treated as separate Palomar "sites". It is not supported to simply randomly distribute proxy and backend nodes between various AZs.

Dictmap

Palomar with AWS S3 requires dictmap.

Dovecot Pro does NOT support AWS DynamoDB (as it uses a different protocol than CQL).

Managed services exist that provide the necessary CQL infrastructure on AWS, such as AWS Keyspaces, DataStax Astra DB, or ScyllaDB Cloud. OX does not support configuration or operation of these managed services, and cannot provide recommendations or operational advice.

Configuration

Using the aws-s3 scheme is the recommended way to configure the S3 driver for AWS.

Currently it's the same as using the generic S3 Compatible Storage Config scheme with the following additional URL parameters automatically added:

Parameter	Description
`addhdrvar=x-amz-security-token:%{auth:token}`	Enable using security token if returned by IAM lookup.
`loghdr=x-amz-request-id` `loghdr=x-amz-id-2`	Include the these headers' values in all log messages related to the request. This additional information helps when troubleshooting.

Example debug log message, which shows how the x-amz-* headers are included:

Debug: http-client: conn 1.2.3.4:443 [1]: Got 200 response for request [Req1: GET https://test-mails.s3-service.com/?prefix=user%2Fidx%2F]: OK (x-amz-request-id:AABBCC22BB7798869, x-amz-id-2:DeadBeefanXBapRucWGAD1+aWwYMfwmXydlI0mHSuh4ic/j8Ji7gicTsP7xpMQz1IR9eydzeVI=) (took 63 ms + 140 ms in queue)

URL Formatting

An HTTP URL to specify how the object storage is accessed.

The parameters are specified as URL-style parameters, such as http://url/?param1=value1&param2=value2.

URL escaping is used, so if password is foo/bar the URL is http://user:foo%2fbar@example.com/.

Additionally, because Dovecot expands %variables inside the plugin section, the % needs to be escaped. So the final string would be e.g.:

plugin {
  obox_fs = s3:https://user:foo%%2fbar@example.com/ # password is foo/bar
}

Configuration Parameters

Parameter	Description	Default
`absolute_timeout=<time_msecs>`	Maximum total time for an HTTP request to finish. Overrides all timeout configuration.	None
`addhdr=<name>:<value>`	Add the specified header to all HTTP requests.	None
`addhdrvar=<name>:<variable>`	Add the specified header to all HTTP requests and set the value to the expanded variables value.	None
`auth_host=<value>`	IAM hostname and port. The default is 169.254.169.254:80. Normally there is no reason to change this. This is mainly intended for testing.	None
`auth_role=<value>`	Perform AWS IAM lookup using this role name.	None
`bucket=<n>`	Used by some backends to specify the bucket to save the data into.	None
`bulk_delete_limit=<n>`	Number of deletes supported within the same bulk delete request. 0 disables bulk deletes.	`1000`
`connect_timeout=<time_msecs>`	Timeout for establishing a TCP connection.	`<timeout> parameter`
`delete_max_retries=<n>`	Max number of HTTP request retries for delete actions.	`<max_retries> parameter`
`delete_timeout=<time_msecs>`	Timeout for sending a delete HTTP response.	`<timeout> parameter`
`loghdr=<name>`	Headers with the given name in HTTP responses are logged as part of any error, debug or warning messages related to the HTTP request. These headers are also included in the `http_request_finished` event as fields prefixed with `http_hdr_`. Can be specified multiple times.	None
`max_connect_retries=<n>`	Number of connect retries.	`2`
`max_retries=<n>`	Max number of HTTP request retries. Retries happen for 5xx errors as well as for 423(locked) with sproxyd. There is a wait between attempting next retry. The initial retry is done after 50ms. The following retries are done after waiting ten times as long as the previous attempt, so 50ms -> 500 ms -> 5s ->10s. The maximum wait time per attempt before retry is limited to 10 seconds. Please note that if the overall request time exceeds the configured `absolute_timeout` it takes precedence, emits an error and prevents further retries. While the configured `timeout` value determines how long HTTP responses are allowed to take before an error is emitted.	`4`
`no_trace_headers=1`	Set to 1 to not add `X-Dovecot-User` or `X-Dovecot-Session` headers to HTTP request. These headers are useful to correlate object storage requests to App Suite/Dovecot sessions. If not doing correlations via log aggregation, this is safe to disable.	`0`
`read_max_retries=<n>`	Max number of HTTP request retries for read actions.	`<max_retries> parameter`
`read_timeout=<time_msecs>`	Timeout for a receiving read HTTP response.	`<timeout> parameter`
`reason_header_max_length=<n>`	Maximum length for X-Dovecot-Reason HTTP header. If header is present, it contains information why obox operation is being done.	`0`
`region=<value>`	Specify region name for AWS S3 bucket. When this is specified, Dovecot starts using v4 signatures. (The default is to use v2 signatures.)	None
`slow_warn=<time_msecs>`	Log a warning about any HTTP request that takes longer than this time.	`5s`
`timeout=<time_msecs>`	Default timeout for HTTP responses, unless overwritten by other parameters.	`10s`
`write_max_retries=<n>`	Max number of HTTP request retries for write actions.	`<max_retries> parameter`
`write_timeout=<time_msecs>`	Timeout for a write HTTP response.	`<timeout> parameter`

Bucket Name

There are two ways to specify the bucket name in the configuration.

Path Style

S3 requests' path begins with the bucket parameter. This is configured with the bucket parameter.

For example: https://s3.example.com/?bucket=BUCKETNAME will result in requests looking like https://s3.example.com/BUCKETNAME/object-path

Virtual-Hosted Style

The first subdomain in the URL specifies the bucket.

NOTE

AWS S3 supports only this style for new buckets.

For example: https://BUCKETNAME.s3.example.com

Bulk Deletes

The S3 schemes support bulk-delete requests.

The bulk-delete option is enabled by default to delete up to 1000 keys with one request.

To change this behavior refer to bulk_delete_limit.

To actually delete that many mails in a single request, you must also set obox_max_parallel_deletes:

obox_max_parallel_deletes = 1000

This value should be the same as bulk_delete_limit or lower.

AWS Signature Version

The obox S3 driver uses the AWS signature version 2 method by default, but version 4 can be used by adding the region parameter to the S3 URL:

plugin {
  obox_index_fs = aws-s3:https://ACCESSKEY:SECRET@BUCKETNAME.s3.eu-central-1.amazonaws.com/?region=eu-central-1
}

Example Configuration

WARNING

All text indicated by {{VARIABLE NAME}} in the examples below MUST be replaced with your local configuration value(s).

Note

Dictmap must also be configured to use this storage driver.

mail_location = obox:%8Mu/%u:INDEX=~/:CONTROL=~/
plugin {
  # BUCKET_NAME: Storage bucket name to use.
  # REGION: AWS region to use for storage.
  # S3ACCESS: IAM role to use for storage access.

  # Examples use data compression with zstd (level 3)

  # Without lazy_expunge plugin:
  obox_fs = fscache:2G:/var/cache/mails/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; aws-s3:https://{{BUCKET_NAME}}.s3.{{REGION}}.amazonaws.com/?region={{REGION}}&auth_role={{S3ACCESS}}&reason_header_max_length=200 ; refcounting-table:lockdir=/tmp:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11:storage-objectid-prefix=%u/mail-storage/
  # With lazy_expunge plugin:
  #obox_fs = fscache:2G:/var/cache/mails/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; aws-s3:https://{{BUCKET_NAME}}.s3.{{REGION}}.amazonaws.com/?region={{REGION}}&auth_role={{S3ACCESS}}&reason_header_max_length=200 ; refcounting-table:bucket-size=10000:bucket-cache=%h/buckets.cache:nlinks-limit=3:delete-timestamp=+10s:bucket-deleted-days=11:storage-objectid-prefix=%u/mail-storage/

  obox_index_fs = compress:zstd:3:dictmap:proxy:dict-async:cassandra ; aws-s3:https://{{BUCKET_NAME}}.s3.{{REGION}}.amazonaws.com/?region={{REGION}}&auth_role={{S3ACCESS}}&reason_header_max_length=200 ; diff-table:storage-passthrough-paths=full
  fts_dovecot_fs = fts-cache:fscache:2G:/var/cache/fts/%4Nu:compress:zstd:3:dictmap:proxy:dict-async:cassandra ; aws-s3:https://{{BUCKET_NAME}}.s3.{{REGION}}.amazonaws.com/?region={{REGION}}&auth_role={{S3ACCESS}}&reason_header_max_length=200 ; dict-prefix=%u/fts/:storage-passthrough-paths=full

  obox_max_parallel_deletes = 1000
}

# If IAM is used, this configuration is also needed
#mail_uid = vmail
#service fs-auth {
#  unix_listener fs-auth {
#    user = vmail
#  }
#}

Authentication Options

IAM Authentication

Note

This is the recommended way of authentication with AWS S3.

Dovecot supports AWS Identity and Access Management (IAM) for authenticating requests to AWS S3 using the AWS EC2 Instance Metadata Service (IMDS). Version 2 of IMDS (IMDSv2) is supported.

Using IAM allows running Dovecot with S3 Storage while not keeping the credentials in the configuration.

A requirement for using IMDSv2 is that Dovecot is running on an AWS EC2 instance, otherwise the IMDS will not be reachable. Additionally an IAM role must be configured which allows trusted entities, EC2 in this case, to assume that role. The role (for example s3access) that will be assumed must have the AmazonS3FullAccess policy attached.

The auth_role can be configured as a URL parameter which specifies the IAM role to be assumed. If no auth_role is configured, no IAM lookup will be done.

When using IAM you must ensure that the fs-auth service has proper permissions/owner.

Configure the user for the fs-auth listener to be the same as for mail_uid.

mail_uid = vmail

service fs-auth {
  unix_listener fs-auth {
    user = vmail
  }
}

More IAM Information

Manual Authentication

Get ACCESSKEY and SECRET from "AWS -> My account -> Security credentials -> Access credentials".

Create the BUCKETNAME "from AWS Management Console -> S3 -> Create Bucket".

If the ACCESSKEY or SECRET contains any special characters, they can be %hex-encoded.

Note

dovecot.conf handles %variable expansion internally as well, so % needs to be escaped as %% and ':' needs to be escaped as %%3A. For example if the SECRET is "foo:bar" this would be encoded as https://ACCESSKEY:foo%%3Abar:s3.example.com/. This double-%% escaping is needed only when the string is read from dovecot.conf - it doesn't apply, for example, if the string comes from a userdb lookup.

Partition Prefix

AWS S3 can internally shard data more efficiently by including a partition prefix in all S3 paths. Without this prefix, the S3 bucket may not scale above a certain limit in the number of S3 requests/second. (See Best Practices Design Patterns: Optimizing Amazon S3 Performance).

Note

In Dovecot Pro, "partition prefixes" were referred to as "dispersion prefixes". Those terms are interchangable.

We recommend implementing the dispersion prefix by using the first 8 characters of the hex representation of the MD5 hash of the username at the beginning of each object path.

mail_location = obox:%8Mu/%u:INDEX=~/:CONTROL=~/

Partition Behavior

When a S3 bucket is created, AWS creates a single shared partition for the bucket with a default limit of 3,500 requests/second for PUTs/DELETEs/POSTs and 5,500 requests/second for GETs.

This 3,500 TPS limit is generally too small and quickly surpassed by Dovecot obox installations, which results in a spike of 503: Slow Down log events.

By using a partition prefix in the S3 path, you can automatically spread the load between internal S3 partitions (and their associated limits), so there should be no 503 errors related to storage action rate limiting when using obox.

DNS

AWS instances are known to react badly when high packets per second network traffic is generated. DNS lookups for S3 storage access can generate large numbers of requests. A local DNS caching system should be used in order to reduce the network load.

AWS S3 API

AWS S3 REST API Calls

URL	Notes
`HEAD <path>`	Metadata read operation on S3 object
`GET <path>`	Read operation on S3 object
`PUT <path>`	Write operation on S3 object
`DELETE <path>`	Delete operation on S3 object
`POST /?delete`	Bulk-delete operation on up to 1000 Objects per request, bulk_delete_limit
`PUT /latest/api/token` (always sent to `169.254.169.254`)	Lookup operation for IMDSv2 token for IAM authentication
`GET /latest/meta-data/iam/security-credentials/<role>`	Lookup operation for IAM credentials

AWS S3 Object ID Format

URI Path we write to:

/<key prefix>/<bucketname>/<dispersion prefix>/<dovecot internal path>

Key	Description
`<key prefix>`	From URL config (optional; empty if not specified)
`<bucketname>`	Either from URL hostname or URL `bucket=` query parameter
`<dispersion prefix>`	From `mail_location setting`. Recommended value (see XXX) gives two levels of dispersion of the format: `[0-9a-f]{2}/[0-9a-f]{3}`
`<dovecot internal path>`	Dovecot internal path to file. Example: `$user/mailboxes/$mailboxguid/$messageguid`

Internal Path Variables:

Variable	Description
`$user`	Dovecot unique username (installation defined)
`$mailboxguid`	32 byte randomly generated UID defining a mailbox
`$messageguid`	32 byte randomly generated UI defining a message blob

Migration From Dovecot Pro 2.3.x

Dovecot Pro 3.0 supports using AWS S3 only with Dictmap. To allow the in-place migration of user data without the need to actually copy all user data fs-dictmap provides options and scripts to do so. See Path Based Object Storages for more details on how to use path based object storages, like Amazon S3 with dictmap.

The in-place migration can be done using the storage-objectid-migrate setting and the scripts storage-objectid-migrate-mails.sh, storage-objectid-migrate-index.sh to migrate indexes and mails. For more details refer to Migrating Path Based Object Storages to Dictmap.

Debug Headers

Dovecot sends the following HTTP headers towards storage. They should be logged for troubleshooting purposes:

X-Dovecot-Username
X-Dovecot-Session-Id
X-Dovecot-Reason

Storage-Side Object Metadata

When saving data to object storage, Dovecot stores metadata associated with each blob for data recovery purposes.

This data is written to the HTTP endpoint by adding Dovecot metadata headers to the request. When retrieving a message from object storage, this data is returned in the received headers (only parsed by Dovecot if needed).

For S3, the header names are: x-amz-meta-dovecot-<key>.

Email Objects

Key	Description	Max Length (in bytes)	Other Info
`fname`	Dovecot filename	N/A (installation dependent; username component to naming)
`guid`	Message GUID	32
`origbox`	Folder GUID of first folder where stored	32	Copying does not update
`pop3order`	POP3 message order	10	Only if needed by migration
`pop3uidl`	POP3 UIDL	N/A (depends on source installation)	Only if message was migrated
`received`	Received data	20 (in theory; rarely more than 10)	UNIX timestamp format
`saved`	Saved data	20 (in theory; rarely more than 10)	UNIX timestamp format
`size`	Message size	20 (in theory; rarely more than 10)	Size in bytes
`username`	Dovecot unique username	N/A (installation dependent)

Index Objects

Key	Description	Max Length (in bytes)	Other Info
`fname`	Dovecot filename	N/A (installation dependent; username component to naming)
`mailbox-guid`	Mailbox GUID the index refers to	32
`size`	Message size	20 (in theory; rarely more than 10)	Size in bytes
`username`	Dovecot unique username	N/A (installation dependent)

FTS Index Objects

Key	Description	Max Length (in bytes)	Other Info
`fname`	Dovecot filename	N/A (installation dependent; username component to naming)
`username`	Dovecot unique username	N/A (installation dependent)

Databases

Mechanisms

Formats

Extensions

Obox Storage Support: AWS S3 ​

Technical Requirements ​

Dictmap ​

Configuration ​

URL Formatting ​

Configuration Parameters ​

Bucket Name ​

Path Style ​

Virtual-Hosted Style ​

Bulk Deletes ​

AWS Signature Version ​

Example Configuration ​

Authentication Options ​

IAM Authentication ​

Manual Authentication ​

Partition Prefix ​

Partition Behavior ​

DNS ​

AWS S3 API ​

AWS S3 REST API Calls ​

AWS S3 Object ID Format ​

Migration From Dovecot Pro 2.3.x ​

Debug Headers ​

Storage-Side Object Metadata ​

Email Objects ​

Index Objects ​

FTS Index Objects ​

Obox Storage Support: AWS S3

Technical Requirements

Dictmap

Configuration

URL Formatting

Configuration Parameters

Bucket Name

Path Style

Virtual-Hosted Style

Bulk Deletes

AWS Signature Version

Example Configuration

Authentication Options

IAM Authentication

Manual Authentication

Partition Prefix

Partition Behavior

DNS

AWS S3 API

AWS S3 REST API Calls

AWS S3 Object ID Format

Migration From Dovecot Pro 2.3.x

Debug Headers

Storage-Side Object Metadata

Email Objects

Index Objects

FTS Index Objects