Obox Storage Support: AWS S3

This page describes support for the S3 service directly provided by Amazon Web Services (AWS).

WARNING

For details on "S3 Compatible" storage support from non-AWS providers, see S3 Compatible Storage.

Technical Requirements

DANGER

A Dovecot Pro/Palomar installation is only supported in a single AWS region.

This is because AWS S3 only offers asynchronous data replication across regions, which is not supported by obox.

Palomar can be setup using multiple Availability Zones (AZs) within a region. Each AZ is considered a "site" for purposes of Palomar.

DANGER

Availability Zones must be treated as separate Palomar "sites". It is not supported to simply randomly distribute proxy and backend nodes between various AZs.

Dictmap

Palomar with AWS S3 requires dictmap.

Dovecot Pro does NOT support AWS DynamoDB (as it uses a different protocol than CQL).

Managed services exist that provide the necessary CQL infrastructure on AWS, such as DataStax Astra DB or ScyllaDB Cloud. OX does not support configuration or operation of these managed services, and cannot provide recommendations or operational advice.

UNSUPPORTED

AWS Keyspaces cannot be used with Dovecot Pro as it lacks support for certain Dovecot-required Cassandra features.

Configuration

Using the aws-s3 driver is a simpler way to configure the S3 driver for AWS. Currently it's the same as using the fs-s3 driver with the following default settings:

Currently it's the same as using the generic S3 Compatible Storage Config scheme with the following additional settings automatically added:

Parameter	Description
`fs_http_add_headers`/`x-amz-security-token` = `%{auth:token}`	Enable using security token if returned by IAM lookup.
`fs_http_log_headers`/`x-amz-request-id` = `yes` `fs_http_log_headers`/`x-amz-id-2` = `yes`	Include these headers' values in all log messages related to the request. This additional information helps when troubleshooting.

Example debug log message, which shows how the x-amz-* headers are included:

Debug: http-client: conn 1.2.3.4:443 [1]: Got 200 response for request [Req1: GET https://test-mails.s3-service.com/?prefix=user%2Fidx%2F]: OK (x-amz-request-id:AABBCC22BB7798869, x-amz-id-2:DeadBeefanXBapRucWGAD1+aWwYMfwmXydlI0mHSuh4ic/j8Ji7gicTsP7xpMQz1IR9eydzeVI=) (took 63 ms + 140 ms in queue)

Bucket Name

There are two ways to specify the bucket name in the configuration.

Path Style

S3 requests' path begins with the fs_s3_bucket.

For example:

fs_s3_bucket = BUCKETNAME

will result in requests to https://s3.example.com/BUCKETNAME/object-path.

Virtual-Hosted Style

The first subdomain in the URL specifies the bucket.

TIP

AWS S3 supports only this style for new buckets.

For example: https://BUCKETNAME.s3.example.com

Bulk Deletes

The S3 schemes support bulk-delete requests.

The bulk-delete option is enabled by default to delete up to 1000 keys with one request.

To change this behavior refer to bulk_delete_limit.

To actually delete that many mails in a single request, you must also set obox_max_parallel_deletes:

obox_max_parallel_deletes = 1000

This value should be the same as bulk_delete_limit or lower.

AWS Signature Version

The obox S3 driver uses the AWS signature version 4 method by default, but version 2 can be used by adding the fs_s3_signing setting:

fs_s3_signing = v2

Example Configuration

WARNING

All text indicated by {{VARIABLE NAME}} in the examples below MUST be replaced with your local configuration value(s).

TIP

Dictmap must also be configured to use this storage driver.

mail_driver = obox

# BUCKET_NAME: Storage bucket name to use.
# REGION: AWS region to use for storage.
# S3ACCESS: IAM role to use for storage access.
fs_s3_url = https://{{BUCKET_NAME}}.s3.{{REGION}}.amazonaws.com/
fs_s3_region = {{REGION}}
fs_s3_auth_role = {{S3ACCESS}}
fs_compress_write_method = zstd
obox {
  fs fscache {
    size = 512M
    path = /var/cache/mails/%{user | sha1 % 4}
  }
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
    storage_objectid_prefix = %{user}/mails/
    #lock_path = /tmp # Set only without lazy_expunge plugin
  }
  fs aws-s3 {
  }
}
metacache {
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
    storage_passthrough_paths = full
  }
  fs aws-s3 {
  }
}
fts dovecot {
  fs fts-cache {
  }
  fs fscache {
    size = 512M
    path = /var/cache/fts/%{user | sha1 % 4}
  }
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
    storage_passthrough_paths = full
  }
  fs aws-s3 {
  }
}

Without IAM use the settings:

fs_s3_access_key = {{ACCESSKEY}}
fs_s3_secret = {{SECRET}}

Settings

`fs_aws_s3`

Value	Named Filter
See Also	`fs_s3`

Filter for AWS S3-specific settings. fs_s3 filter is also used.

`fs_s3`

Value	Named Filter

Filter for S3-specific settings.

`fs_s3_access_key`

Default	[None]
Value	string

S3 access key. Not needed when AWS IAM is used.

`fs_s3_auth_host`

Default	`169.254.169.254`
Value	string
Advanced Setting; this should not normally be changed.

AWS IAM hostname. Normally there is no reason to change this. This is mainly intended for testing.

`fs_s3_auth_port`

Default	`80`
Value	Port Number
Advanced Setting; this should not normally be changed.

AWS IAM port. Normally there is no reason to change this. This is mainly intended for testing.

`fs_s3_auth_role`

Default	[None]
Value	string
See Also	IAM Authentication

If not empty, perform AWS IAM lookup using this role.

`fs_s3_bucket`

Default	[None]
Value	string

S3 bucket name added to the request path.

`fs_s3_bulk_delete_limit`

Default	`1000`
Value	unsigned integer

Number of deletes supported within the same bulk delete request. 0 disables bulk deletes.

`fs_s3_region`

Default	[None]
Value	string
See Also	`fs_s3_signing`

Specify region name for AWS S3 bucket. Only needed when using v4 signing.

`fs_s3_secret`

Default	[None]
Value	string

S3 secret. Not needed when AWS IAM is used.

`fs_s3_signing`

Default	`v4`
Value	string
Allowed Values	`v4v2`
See Also	`fs_s3_region`

AWS s3 signing version to use. It is recommended to keep the default v4 signing which also requires fs_s3_region to be set. The AWS v2 signing is deprecated.

`fs_s3_url`

Default	[None]
Value	string

URL for accessing the S3 storage. For example: https://BUCKETNAME.s3.example.com

`fs_http_add_headers`

Default	[None]
Value	String List

Headers to add to HTTP requests.

`fs_http_log_headers`

Default	[None]
Value	Boolean List

Headers with the given name in HTTP responses are logged as part of any error, debug or warning messages related to the HTTP request. These headers are also included in the http_request_finished event as fields prefixed with http_hdr_.

`fs_http_log_trace_headers`

Default	`yes`
Value	boolean

If yes add X-Dovecot-User: and X-Dovecot-Session: headers to HTTP request. The session header is useful to correlate object storage requests to AppSuite/Dovecot sessions.

`fs_http_reason_header_max_length`

Default	[None]
Value	unsigned integer

If non-zero add X-Dovecot-Reason: header to the HTTP request. The value contains a human-readable string why the request is being sent.

`fs_http_slow_warning`

Default	`5s`
Value	time (milliseconds)

Log a warning about any HTTP request that takes longer than this time.

Default HTTP Settings

fs-s3 overrides some of the default HTTP client settings:

You can override these and any other HTTP client or SSL settings by placing them inside fs_s3 named filter.

Authentication Options

IAM Authentication

INFO

This is the recommended way of authentication with AWS S3.

Dovecot supports AWS Identity and Access Management (IAM) for authenticating requests to AWS S3 using the AWS EC2 Instance Metadata Service (IMDS) when using Amazon EC2 or IAM Roles for Service Accounts (IRSA) when using Amazon Elastic Kubernetes Service (Amazon EKS). Version 2 of IMDS (IMDSv2) is supported.

Using IAM allows running Dovecot with S3 Storage while not keeping the credentials in the configuration.

When using IAM you must ensure that the fs-auth service has proper permissions/owner.

Configure the user for the fs-auth listener to be the same as for mail_uid.

mail_uid = vmail

service fs-auth {
  unix_listener fs-auth {
    user = vmail
  }
}

IAM Authentication in EC2

A requirement for using IMDSv2 is that Dovecot is running on an AWS EC2 instance, otherwise the IMDS will not be reachable. Additionally an IAM role must be configured which allows trusted entities, EC2 in this case, to assume that role. The role (for example s3access) that will be assumed must have the AmazonS3FullAccess policy attached.

The auth_role can be configured as a URL parameter which specifies the IAM role to be assumed. If no auth_role is configured, no IAM lookup will be done.

More Information for IAM on EC2

IAM Authentication in EKS

When using EKS and IRSA, Dovecot is using environment variables provided to the backend pods to acquire temporary credentials using the AssumeRoleWithWebIdentity API by the AWS Security Token Service (STS). To allow Dovecot access to these variables, import_environment must be configured accordingly:

import_environment = {
  AWS_REGION
  AWS_ROLE_ARN
  AWS_WEB_IDENTITY_TOKEN_FILE
}

To allow the AssumeRoleWithWebIdentity call to successfully assume a Role and fetch temporary credentials an IAM role with the "AssumeRole" policy from a Service Account must be configured. The Service Account must be assigned to the backend pods that will call the STS API. The auth_role is used as RoleSessionName, it can be freely chosen when running in EKS. The role name can be viewed by administrators to help identify who performed an action in AWS. See sts:RoleSessionName for more details on the RoleSessionName.

More Information for IAM on EKS

Manual Authentication

Get ACCESSKEY and SECRET from "AWS -> My account -> Security credentials -> Access credentials".

Create the BUCKETNAME "from AWS Management Console -> S3 -> Create Bucket".

If the ACCESSKEY or SECRET contains any special characters, they can be %hex-encoded.

Partition Prefix

AWS S3 used to recommend having explicit partition prefixes for better distributing the S3 load. However, this is no longer necessary. AWS automatically creates the prefixes as needed and rebalances the data. (See Best Practices Design Patterns: Optimizing Amazon S3 Performance).

Partition Behavior

When a S3 bucket is created, AWS creates a single shared partition for the bucket with a default limit of 3,500 requests/second for PUTs/DELETEs/POSTs and 5,500 requests/second for GETs.

This 3,500 TPS limit is generally too small and quickly surpassed by Dovecot obox installations, which results in a spike of 503: Slow Down log events.

AWS automatically eventually splits the partitions when reaching request limits. However, it's better to ask AWS support to split the partition early on before users see any errors.

DNS

AWS instances are known to react badly when high packets per second network traffic is generated. DNS lookups for S3 storage access can generate large numbers of requests. A local DNS caching system should be used in order to reduce the network load.

AWS S3 API

AWS S3 REST API Calls

URL	Notes
`HEAD <path>`	Metadata read operation on S3 object.
`GET <path>`	Read operation on S3 object.
`PUT <path>`	Write operation on S3 object.
`DELETE <path>`	Delete operation on S3 object.
`POST /?delete`	Bulk-delete operation on up to 1000 Objects per request, bulk_delete_limit.
`PUT /latest/api/token` (always sent to `169.254.169.254`)	Lookup operation for IMDSv2 token for IAM authentication.
`GET /latest/meta-data/iam/security-credentials/<role>`	Lookup operation for IAM credentials.

AWS S3 Object ID Format

URI Path we write to:

/<key prefix>/<bucketname>/<dovecot internal path>

Key	Description
`<key prefix>`	From URL config (optional; empty if not specified)
`<bucketname>`	Either from URL hostname or `fs_s3_bucket`
`<dovecot internal path>`	Dovecot internal path to file. Example: `$user/mailboxes/$mailboxguid/$messageguid`

Internal Path Variables:

Variable	Description
`$user`	Dovecot unique username (installation defined)
`$mailboxguid`	32 byte randomly generated UID defining a mailbox
`$messageguid`	32 byte randomly generated UI defining a message blob

Migration From Dovecot Pro 2.3.x

Dovecot Pro 3.0 supports using AWS S3 only with Dictmap. To allow the in-place migration of user data without the need to actually copy all user data fs-dictmap provides options and scripts to do so. See Path Based Object Storages for more details on how to use path based object storages, like Amazon S3 with dictmap.

The in-place migration can be done using the storage-objectid-migrate setting and the scripts storage-objectid-migrate-mails.sh, storage-objectid-migrate-index.sh to migrate indexes and mails. For more details refer to Migrating Path Based Object Storages to Dictmap.

Debug Headers

Dovecot sends the following HTTP headers towards storage. They should be logged for troubleshooting purposes:

X-Dovecot-Username
X-Dovecot-Session-Id
X-Dovecot-Reason

Storage-Side Object Metadata

When saving data to object storage, Dovecot stores metadata associated with each blob for data recovery purposes.

This data is written to the HTTP endpoint by adding Dovecot metadata headers to the request. When retrieving a message from object storage, this data is returned in the received headers (only parsed by Dovecot if needed).

For S3, the header names are: x-amz-meta-dovecot-<key>.

Email Objects

Key	Description	Max Length (in bytes)	Other Info
`fname`	Dovecot filename	N/A (installation dependent; username component to naming)
`guid`	Message GUID	32
`origbox`	Folder GUID of first folder where stored	32	Copying does not update
`pop3order`	POP3 message order	10	Only if needed by migration
`pop3uidl`	POP3 UIDL	N/A (depends on source installation)	Only if message was migrated
`received`	Received data	20 (in theory; rarely more than 10)	UNIX timestamp format
`saved`	Saved data	20 (in theory; rarely more than 10)	UNIX timestamp format
`size`	Message size	20 (in theory; rarely more than 10)	Size in bytes
`username`	Dovecot unique username	N/A (installation dependent)

Index Objects

Key	Description	Max Length (in bytes)	Other Info
`fname`	Dovecot filename	N/A (installation dependent; username component to naming)
`mailbox-guid`	Mailbox GUID the index refers to	32
`size`	Message size	20 (in theory; rarely more than 10)	Size in bytes
`username`	Dovecot unique username	N/A (installation dependent)

FTS Index Objects

Key	Description	Max Length (in bytes)	Other Info
`fname`	Dovecot filename	N/A (installation dependent; username component to naming)
`username`	Dovecot unique username	N/A (installation dependent)

Databases

Mechanisms

Users

Extensions

Obox Storage Support: AWS S3 ​

Technical Requirements ​

Dictmap ​

Configuration ​

Bucket Name ​

Path Style ​

Virtual-Hosted Style ​

Bulk Deletes ​

AWS Signature Version ​

Example Configuration ​

Settings ​

fs_aws_s3

fs_s3

fs_s3_access_key

fs_s3_auth_host

fs_s3_auth_port

fs_s3_auth_role

fs_s3_bucket

fs_s3_bulk_delete_limit

fs_s3_region

fs_s3_secret

fs_s3_signing

fs_s3_url

fs_http_add_headers

fs_http_log_headers

fs_http_log_trace_headers

fs_http_reason_header_max_length

fs_http_slow_warning

Default HTTP Settings ​

Authentication Options ​

IAM Authentication ​

IAM Authentication in EC2 ​

IAM Authentication in EKS ​

Manual Authentication ​

Partition Prefix ​

Partition Behavior ​

DNS ​

AWS S3 API ​

AWS S3 REST API Calls ​

AWS S3 Object ID Format ​

Migration From Dovecot Pro 2.3.x ​

Debug Headers ​

Storage-Side Object Metadata ​

Email Objects ​

Index Objects ​

FTS Index Objects ​