Obox Storage Support: S3 Compatible

WARNING

Dovecot Pro/obox only directly supports the S3 service as provided directly by Amazon Web Services (AWS).

Dovecot Pro has a lower-tier of SLA support for other "S3-compatible" systems, but it is the customer's responsibility to ensure that system adequately implements the same behavior as AWS's S3 service.

S3 is not defined as an official protocol or API. "S3-compatible" systems attempt to replicate AWS's service offering.

Dictmap

Palomar with S3 Compatible storage requires dictmap.

Managed services exist that provide the necessary CQL infrastructure on AWS, such as DataStax Astra DB or ScyllaDB Cloud. OX does not support configuration or operation of these managed services, and cannot provide recommendations or operational advice.

UNSUPPORTED

AWS Keyspaces cannot be used with Dovecot Pro as it lacks support for certain Dovecot-required Cassandra features.

Configuration

S3 Compatible storage uses the s3 scheme for configuration:

fs_s3_url = https://s3.example.com/
fs_s3_access_key = ACCESSKEY
fs_s3_secret = SECRET

Bucket Name

There are two ways to specify the bucket name in the configuration.

Path Style

S3 requests' path begins with the fs_s3_bucket.

For example:

fs_s3_bucket = BUCKETNAME

will result in requests to https://s3.example.com/BUCKETNAME/object-path.

Virtual-Hosted Style

The first subdomain in the URL specifies the bucket.

TIP

AWS S3 supports only this style for new buckets.

For example: https://BUCKETNAME.s3.example.com

Bulk Deletes

The S3 schemes support bulk-delete requests.

The bulk-delete option is enabled by default to delete up to 1000 keys with one request.

To change this behavior refer to bulk_delete_limit.

To actually delete that many mails in a single request, you must also set obox_max_parallel_deletes:

obox_max_parallel_deletes = 1000

This value should be the same as bulk_delete_limit or lower.

S3 Example Configuration

WARNING

All text indicated by {{VARIABLE NAME}} in the examples below MUST be replaced with your local configuration value(s).

TIP

Dictmap must also be configured to use this storage driver.

mail_driver = obox
mail_path = %{user}

fs_s3_url = https://{{S3_STORAGE_URL}}/
fs_s3_access_key = {{ACCESSKEY}}
fs_s3_secret = {{SECRET}}
fs_s3_bucket = mails
fs_s3_region = region
fs_s3_auth_role = s3access
fs_compress_write_method = zstd
obox {
  fs fscache {
    size = 512M
    path = /var/cache/mails/%{user | sha1 % 4}
  }
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
    storage_objectid_prefix = %{user}/mails/
    #lock_path = /tmp # Set only without lazy_expunge plugin
  }
  fs s3 {
  }
}
metacache {
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
    storage_passthrough_paths = full
  }
  fs s3 {
  }
}
fts dovecot {
  fs fts-cache {
  }
  fs fscache {
    size = 512M
    path = /var/cache/fts/%{user | sha1 % 4}
  }
  fs compress {
  }
  fs dictmap {
    dict proxy {
      name = cassandra
      socket_path = dict-async
    }
    storage_passthrough_paths = full
  }
  fs s3 {
  }
}

Settings

fs_s3

Value	Named Filter

Filter for S3-specific settings.

fs_s3_access_key

Default	[None]
Value	string

S3 access key. Not needed when AWS IAM is used.

fs_s3_auth_host

Default	169.254.169.254
Value	string
Advanced Setting; this should not normally be changed.

AWS IAM hostname. Normally there is no reason to change this. This is mainly intended for testing.

fs_s3_auth_port

Default	80
Value	Port Number
Advanced Setting; this should not normally be changed.

AWS IAM port. Normally there is no reason to change this. This is mainly intended for testing.

fs_s3_auth_role

Default	[None]
Value	string
See Also	IAM Authentication

If not empty, perform AWS IAM lookup using this role.

fs_s3_bucket

Default	[None]
Value	string

S3 bucket name added to the request path.

fs_s3_bulk_delete_limit

Default	1000
Value	unsigned integer

Number of deletes supported within the same bulk delete request. 0 disables bulk deletes.

fs_s3_region

Default	[None]
Value	string
See Also	fs_s3_signing

Specify region name for AWS S3 bucket. Only needed when using v4 signing.

fs_s3_secret

Default	[None]
Value	string

S3 secret. Not needed when AWS IAM is used.

fs_s3_signing

Default	v4
Value	string
Allowed Values	v4v2
See Also	fs_s3_region

AWS s3 signing version to use. It is recommended to keep the default v4 signing which also requires fs_s3_region to be set. The AWS v2 signing is deprecated.

fs_s3_url

Default	[None]
Value	string

URL for accessing the S3 storage. For example: https://BUCKETNAME.s3.example.com

fs_http_add_headers

Default	[None]
Value	String List

Headers to add to HTTP requests.

fs_http_log_headers

Default	[None]
Value	Boolean List

Headers with the given name in HTTP responses are logged as part of any error, debug or warning messages related to the HTTP request. These headers are also included in the http_request_finished event as fields prefixed with http_hdr_.

fs_http_log_trace_headers

Default	yes
Value	boolean

If yes add X-Dovecot-User: and X-Dovecot-Session: headers to HTTP request. The session header is useful to correlate object storage requests to AppSuite/Dovecot sessions.

fs_http_reason_header_max_length

Default	[None]
Value	unsigned integer

If non-zero add X-Dovecot-Reason: header to the HTTP request. The value contains a human-readable string why the request is being sent.

fs_http_slow_warning

Default	5s
Value	time (milliseconds)

Log a warning about any HTTP request that takes longer than this time.

S3 API

S3 Compatible servers must match the API behavior of AWS S3 API.

Debug Headers

Dovecot sends the following HTTP headers towards storage. They should be logged for troubleshooting purposes:

• X-Dovecot-Username
• X-Dovecot-Session-Id
• X-Dovecot-Reason

Storage-Side Object Metadata

When saving data to object storage, Dovecot stores metadata associated with each blob for data recovery purposes.

This data is written to the HTTP endpoint by adding Dovecot metadata headers to the request. When retrieving a message from object storage, this data is returned in the received headers (only parsed by Dovecot if needed).

For S3, the header names are: x-amz-meta-dovecot-<key>.

Email Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
guid	Message GUID	32
origbox	Folder GUID of first folder where stored	32	Copying does not update
pop3order	POP3 message order	10	Only if needed by migration
pop3uidl	POP3 UIDL	N/A (depends on source installation)	Only if message was migrated
received	Received data	20 (in theory; rarely more than 10)	UNIX timestamp format
saved	Saved data	20 (in theory; rarely more than 10)	UNIX timestamp format
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
mailbox-guid	Mailbox GUID the index refers to	32
size	Message size	20 (in theory; rarely more than 10)	Size in bytes
username	Dovecot unique username	N/A (installation dependent)

FTS Index Objects

Key	Description	Max Length (in bytes)	Other Info
fname	Dovecot filename	N/A (installation dependent; username component to naming)
username	Dovecot unique username	N/A (installation dependent)